Grooper Infrastructure - Revision history

Dgreenwood: /* Thread processing guidance */

2026-02-06T15:52:56Z

Thread processing guidance

← Older revision		Revision as of 10:52, 6 February 2026
Line 235:		Line 235:
	== Thread processing guidance ==		== Thread processing guidance ==

	Grooper Activity Processing services distribute processing resources to automated tasks in a Batch Process~~. These services use~~ your system's processing threads ~~to do this~~. Other Grooper services (such as Import Watcher ~~services)~~ also ~~use~~ threads to perform their work.		Grooper Activity Processing services distribute processing resources to automated tasks in a Batch Process using your system's processing threads. Other Grooper services, such as Import Watcher, also consume threads to perform their work.

	~~Please visit~~ the [[Thread Processing Guidance]] article ~~for more information. This article explains how thread processing works and outlines best practices for using threads effectively~~.		For detailed guidance on thread usage and best practices, please see the [[Thread Processing Guidance]] article.

	== Cloud Deployment Guidance ==		== Cloud Deployment Guidance ==

Dgreenwood: /* About Thread Allocation for Grooper Services */

2026-02-06T15:50:14Z

About Thread Allocation for Grooper Services

@@ Line 233: / Line 233: @@
 </div>
-== About Thread Allocation for Grooper Services ==
+== Thread processing guidance ==
-=== Understand the "n minus one" rule ===
+Grooper Activity Processing services distribute processing resources to automated tasks in a Batch Process. These services use your system's processing threads to do this. Other Grooper services (such as Import Watcher services) also use threads to perform their work.
-When Grooper services are installed, each service is assigned a number of CPU threads.
+Please visit the [[Thread Processing Guidance]] article for more information. This article explains how thread processing works and outlines best practices for using threads effectively.
 == Cloud Deployment Guidance ==

Dgreenwood: /* About Thread Allocation for Grooper Services */

2026-02-06T15:43:15Z

About Thread Allocation for Grooper Services

← Older revision		Revision as of 10:43, 6 February 2026
Line 239:		Line 239:
	When Grooper services are installed, each service is assigned a number of CPU threads.		When Grooper services are installed, each service is assigned a number of CPU threads.

	* ~~Some~~ services (for example, '''Import Watcher''') always run on a '''single thread'''.		* Certain services (for example, '''Import Watcher''') always run on a '''single thread'''.
	* '''Activity Processing''' services can use '''multiple threads'''.		* '''Activity Processing''' services can use '''multiple threads'''.

	~~Your~~ machine ~~has~~ a finite number of processing threads~~, and~~ '''~~over~~-allocating ~~them will~~ cause errors'''.		The machine provides a finite number of processing threads. '''Over-allocating threads may cause errors'''.
			The operating system must always have at least '''one thread available'''. Therefore, the total number of threads assigned to Grooper services must not exceed the number of available threads minus one.

	~~The operating system must always have at least~~ '''one ~~thread available~~'''~~. For that reason, the total number of threads assigned to Grooper services must never exceed the number of available threads minus one.~~		This principle is known as the '''"n minus one" rule''':

	~~This is known as the '''"n minus one" rule''':~~		* If '''n''' is the total number of threads available on the machine, the maximum number of threads that may be assigned to Grooper services is '''n − 1'''.

	* If '''n''' is the total number of threads available on the machine, the maximum number of threads ~~you~~ may ~~assign~~ to Grooper services is '''n − 1'''.

	=== Consider the "n minus x" rule ===		=== Consider the "n minus x" rule ===

	In ~~real-world~~ environments, Grooper is rarely the only ~~software~~ running.		In production environments, Grooper is rarely the only application running. Other software requires CPU threads, so additional reservations may be necessary:

	Other ~~applications also require~~ CPU threads, so additional reservations may be necessary:

	* '''SQL installed on the same machine'''		* '''SQL installed on the same machine'''
	** Follow an '''n − 2''' rule		** Follow an '''n − 2''' rule: 1 thread for the OS, 1 for SQL.
	** (1 thread for the OS, 1 for SQL)

	* '''SQL and IIS installed on the same machine'''		* '''SQL and IIS installed on the same machine'''
	** Follow an '''n − 3''' rule		** Follow an '''n − 3''' rule: 1 thread for the OS, 1 for SQL, 1 for IIS.
	** (1 thread for the OS, 1 for SQL, 1 for IIS)

	* '''Other background applications''' (for example, antivirus software)		* '''Other background applications''' (for example, antivirus software)
	* Reserve additional threads as needed		** Reserve additional threads as needed.

	~~This~~ is ~~one of the key reasons we recommend~~ '''~~distributing~~ SQL, IIS, and Grooper processing services across multiple machines''' whenever possible.		Therefore, it is recommended to '''distribute SQL, IIS, and Grooper processing services across multiple machines''' whenever possible.

	=== Bottom line ===		=== Bottom line ===

	'''Do not over-allocate ~~your~~ available threads.'''		* '''Do not over-allocate available threads.'''
	* If ~~you do~~, '''Grooper ~~can~~ behave erratically or fail unexpectedly'''.		* If threads are over-allocated, '''Grooper may behave erratically or fail unexpectedly'''.
	* Always leave sufficient CPU resources for the operating system and any supporting services.		* Always leave sufficient CPU resources for the operating system and any supporting services.

Dgreenwood: /* About thread allocation for Grooper services */

2026-02-06T15:28:09Z

About thread allocation for Grooper services

← Older revision		Revision as of 10:28, 6 February 2026
Line 233:		Line 233:
	</div>		</div>

	== About ~~thread allocation~~ for Grooper ~~services~~ ==		== About Thread Allocation for Grooper Services ==

	~~'''Be aware of~~ the "n minus one" rule~~!'''~~		=== Understand the "n minus one" rule ===

	~~Services~~ are assigned a number of CPU ~~threads when you install them. For some services, like Import Watcher, the will always run using a single thread. Activity Processing services can run using multiple~~ threads.		When Grooper services are installed, each service is assigned a number of CPU threads.

	~~Keep in mind~~, ~~your machine only has~~ a ~~certain number of processing threads available~~. ~~You~~ '''''~~will~~''''' ~~run into errors if you over-allocate your available threads~~.		* Some services (for example, '''Import Watcher''') always run on a '''single thread'''.
			* '''Activity Processing''' services can use '''multiple threads'''.

	~~Remember too, the operating system itself must always have~~ a ~~single thread available to run. So, the absolute maximum~~ number of threads ~~you can assign to all your services '~~''''~~should not~~''~~''' go beyond the total number of threads available minus one reserved for the operating system. Hence, the "n minus one" rule~~.		Your machine has a finite number of processing threads, and '''over-allocating them will cause errors'''.

	The ~~"n minus~~ one~~" rule is as follows:~~		The operating system must always have at least '''one thread available'''. For that reason, the total number of threads assigned to Grooper services must never exceed the number of available threads minus one.
	* If "n" is the maximum number of threads available ~~on your machine~~, the ~~maximum~~ number of threads ~~you can distribute~~ to Grooper services ~~is "n"~~ minus one.

	'''~~Be aware of the~~ "n minus x" rule!'''		This is known as the '''"n minus one" rule''':

	~~Other programs running in the background will need threads to run as well.~~		* If '''n''' is the total number of threads available on the machine, the maximum number of threads you may assign to Grooper services is '''n − 1'''.
	* If ~~SQL is installed on the same machine as your Grooper services, you should follow an "~~n ~~minus two" rule, reserving one for the OS and one for SQL.~~
	* If IIS ''~~and~~'~~' SQL are installed~~ on the ~~same~~ machine ~~as your~~ Grooper services~~, you should follow an "n minus three" rule, reserving one for the OS, one for SQL, and one for IIS.~~
	* If ''~~other~~'' ~~applications, such as anti-virus software, are running in the background you will need to reserve threads for those applications as well.~~
	* ''~~This is one of the many reasons why we recommend distributing SQL, IIS, and processing services across multiple machines~~.''

	'''''~~THE LONG STORY SHORT HERE IS DO NOT OVER~~-~~ALLOCATE YOUR AVAILABLE THREADS!~~ ~~GROOPER CAN BEHAVE ERRATICALLY IF YOU DO!~~'''''		=== Consider the "n minus x" rule ===

			In real-world environments, Grooper is rarely the only software running.

			Other applications also require CPU threads, so additional reservations may be necessary:

			* '''SQL installed on the same machine'''
			** Follow an '''n − 2''' rule
			** (1 thread for the OS, 1 for SQL)

			* '''SQL and IIS installed on the same machine'''
			** Follow an '''n − 3''' rule
			** (1 thread for the OS, 1 for SQL, 1 for IIS)

			* '''Other background applications''' (for example, antivirus software)
			* Reserve additional threads as needed

			This is one of the key reasons we recommend '''distributing SQL, IIS, and Grooper processing services across multiple machines''' whenever possible.

			=== Bottom line ===

			'''Do not over-allocate your available threads.'''
			* If you do, '''Grooper can behave erratically or fail unexpectedly'''.
			* Always leave sufficient CPU resources for the operating system and any supporting services.

	== Cloud Deployment Guidance ==		== Cloud Deployment Guidance ==

Randallkinard: /* Grooper Repository details */

2026-01-22T19:10:45Z

Grooper Repository details

← Older revision		Revision as of 14:10, 22 January 2026
Line 290:		Line 290:

	But what makes up that database and file store? Here, you will find more details on the tables contained in a Grooper Database and the file structure of a Grooper File Store.		But what makes up that database and file store? Here, you will find more details on the tables contained in a Grooper Database and the file structure of a Grooper File Store.
			<div style="padding-left: 1.5em;">
	===The Database===		===The Database===
	The Grooper Repository's database (aka "Grooper Database") is the core component of Grooper. All aspects of Grooper configuration are stored has metadata in the Grooper Database's tables. The processing status of Batches and their tasks are stored in the Grooper Database. Statistical information and logs are stored in the Grooper Database. There is no Grooper without the Grooper Database.		The Grooper Repository's database (aka "Grooper Database") is the core component of Grooper. All aspects of Grooper configuration are stored has metadata in the Grooper Database's tables. The processing status of Batches and their tasks are stored in the Grooper Database. Statistical information and logs are stored in the Grooper Database. There is no Grooper without the Grooper Database.
Line 297:		Line 297:

	''Please note the Grooper database tables are not listed here in alphabetical order. They are listed in terms of how impactful they are to Grooper's overall architecture.''		''Please note the Grooper database tables are not listed here in alphabetical order. They are listed in terms of how impactful they are to Grooper's overall architecture.''
			<div style="padding-left: 1.5em;">
	====dbo.TreeNode====		====dbo.TreeNode====

Line 762:		Line 762:

	Using a SQL database as the core of Grooper is part of what makes it so efficient. Every property of every node in Grooper is a simple entry in a table. As a result, these objects and their properties can be loaded into memory and accessed nearly instantly. As file I/O is one of the slowest aspects of modern computing, this would not be the case otherwise (For example: If Grooper Repositories were defined by something like a project file). This also allows objects to be managed discretely on an individual basis, which allows multiple users to work in one environment and prevent overlapping work by locking objects.		Using a SQL database as the core of Grooper is part of what makes it so efficient. Every property of every node in Grooper is a simple entry in a table. As a result, these objects and their properties can be loaded into memory and accessed nearly instantly. As file I/O is one of the slowest aspects of modern computing, this would not be the case otherwise (For example: If Grooper Repositories were defined by something like a project file). This also allows objects to be managed discretely on an individual basis, which allows multiple users to work in one environment and prevent overlapping work by locking objects.
			</div>
	===The File Store===		===The File Store===
	The [[File Store]] in Grooper is a file share in a Windows environment. It houses the files associated with nodes in Grooper that have information that would otherwise be inefficient to store in a cell in a database table.		The [[File Store]] in Grooper is a file share in a Windows environment. It houses the files associated with nodes in Grooper that have information that would otherwise be inefficient to store in a cell in a database table.
Line 773:		Line 773:

	Although the majority of files in the File Store relate to Batch objects (a page's image or imported files), some files are the result of other "in-Grooper" processes, such as layout data, OCR character data, extracted index data, and more.		Although the majority of files in the File Store relate to Batch objects (a page's image or imported files), some files are the result of other "in-Grooper" processes, such as layout data, OCR character data, extracted index data, and more.
			</div>

	== Diagram: Grooper's Core and Supporting Architecture ==		== Diagram: Grooper's Core and Supporting Architecture ==

Randallkinard: /* Cloud Deployment Guidance */

2026-01-22T19:09:24Z

Cloud Deployment Guidance

← Older revision		Revision as of 14:09, 22 January 2026
Line 261:		Line 261:

	However, in practice, we have found not all cloud services are created equal.		However, in practice, we have found not all cloud services are created equal.
			<div style="padding-left: 1.5em;">
	=== SQL Guidance ===		=== SQL Guidance ===
	For high volume environments, we recommend hosting SQL on a VM rather than using a “SQL as a Service” solution (for example Azure SQL Database).		For high volume environments, we recommend hosting SQL on a VM rather than using a “SQL as a Service” solution (for example Azure SQL Database).
Line 279:		Line 279:
	** A common scenario would be a Split Pages step creating a page in the file store then a subsequent Image Processing step immediately attempting to process the created page in the next step. The page is saved to the file store in such a way that Grooper may “know” a file exists, but is not able to immediately access it for the Image Processing activity to process it, resulting in an error being thrown.		** A common scenario would be a Split Pages step creating a page in the file store then a subsequent Image Processing step immediately attempting to process the created page in the next step. The page is saved to the file store in such a way that Grooper may “know” a file exists, but is not able to immediately access it for the Image Processing activity to process it, resulting in an error being thrown.
	** This will be more common in high volume environments with large amounts of threads allocated to processing tasks. However, it will likely affect even small environments.		** This will be more common in high volume environments with large amounts of threads allocated to processing tasks. However, it will likely affect even small environments.
			</div>

	==Grooper Repository details==		==Grooper Repository details==

Randallkinard: /* System Architecture and Requirements */

2026-01-22T19:08:45Z

System Architecture and Requirements

← Older revision		Revision as of 14:08, 22 January 2026
Line 46:		Line 46:

	==System Architecture and Requirements==		==System Architecture and Requirements==
			<div style="padding-left: 1.5em;">
	===The Grooper Database: SQL Server===		===The Grooper Database: SQL Server===
	This refers to any machine installed with Microsoft SQL Server hosting a '''Grooper Database'''.		This refers to any machine installed with Microsoft SQL Server hosting a '''Grooper Database'''.
Line 83:		Line 83:
	**<li class="attn-bullet">'''''BE AWARE:''''' The Grooper application (GrooperInstall_##.##.####.exe) ''must'' be installed on this server. The Grooper Web Client application (GrooperWebClient_##.##.####.exe) leverages Grooper DLLs to function.		**<li class="attn-bullet">'''''BE AWARE:''''' The Grooper application (GrooperInstall_##.##.####.exe) ''must'' be installed on this server. The Grooper Web Client application (GrooperWebClient_##.##.####.exe) leverages Grooper DLLs to function.
	**<li class="fyi-bullet"> ''For smaller installs, the Grooper Web Server and Grooper SQL Server may be the same machine. For larger installs, this is not recommended. We recommend a dedicated server for the Grooper Web Server.''		**<li class="fyi-bullet"> ''For smaller installs, the Grooper Web Server and Grooper SQL Server may be the same machine. For larger installs, this is not recommended. We recommend a dedicated server for the Grooper Web Server.''
			<div style="padding-left: 1.5em;">
	==== Grooper App Pool User ====		==== Grooper App Pool User ====

Line 151:		Line 151:
	* Required if also using the application pool identity as a service account running Grooper services		* Required if also using the application pool identity as a service account running Grooper services
	\|}		\|}
			</div>
	=== Grooper Processing Servers ===		=== Grooper Processing Servers ===
	This refers to any machine in the Grooper "processing environment". A processing environment will be composed of the Grooper web server ''and'' any machine dedicated to hosting Grooper's Activity Processing services.		This refers to any machine in the Grooper "processing environment". A processing environment will be composed of the Grooper web server ''and'' any machine dedicated to hosting Grooper's Activity Processing services.
Line 161:		Line 161:
	**<li class="attn-bullet"> Be aware this is a ''minimum''. High volume scenarios will need more cores to process content efficiently.		**<li class="attn-bullet"> Be aware this is a ''minimum''. High volume scenarios will need more cores to process content efficiently.
	* The port for licensing is 13900 by default.		* The port for licensing is 13900 by default.
			<div style="padding-left: 1.5em;">
	==== Grooper Service Users ====		==== Grooper Service Users ====

Line 204:		Line 204:

	Other Grooper services may require additional permissions. For example, an Import Watcher's user may need access to directories used for importing file content.		Other Grooper services may require additional permissions. For example, an Import Watcher's user may need access to directories used for importing file content.
			</div>
	=== Client Computer ===		=== Client Computer ===
	This refers to a machine connecting to a Grooper Repository by opening the Grooper web app in a browser. The client computer does ''not'' need either the main Grooper application or the Grooper Web Client application installed. It just needs to be able to connect to the Grooper Repository using a supported browser.		This refers to a machine connecting to a Grooper Repository by opening the Grooper web app in a browser. The client computer does ''not'' need either the main Grooper application or the Grooper Web Client application installed. It just needs to be able to connect to the Grooper Repository using a supported browser.
Line 211:		Line 211:
	* User logs in using Windows Account (Domain or Local) on Grooper Web Server.		* User logs in using Windows Account (Domain or Local) on Grooper Web Server.
	* In version 2025, users may also log in using OAuth credentials. See the [[OAuth Setup]] article for more information.		* In version 2025, users may also log in using OAuth credentials. See the [[OAuth Setup]] article for more information.
			<div style="padding-left: 1.5em;">
	==== Additional requirements for scanning users only ====		==== Additional requirements for scanning users only ====
	This refers to a client machine hooked up to a scanner, used to scan documents into Grooper using the web client. These machines need to have '''Grooper Desktop''' installed and running.		This refers to a client machine hooked up to a scanner, used to scan documents into Grooper using the web client. These machines need to have '''Grooper Desktop''' installed and running.
Line 218:		Line 218:
	* A Windows 10 or Windows 11 PC is required for Grooper Desktop.		* A Windows 10 or Windows 11 PC is required for Grooper Desktop.
	**<li class="fyi-bullet"> ''For more information visit the [[Desktop Scanning in Grooper]] article.''		**<li class="fyi-bullet"> ''For more information visit the [[Desktop Scanning in Grooper]] article.''
			</div>
	===Windows Prerequisites===		===Windows Prerequisites===
	The following are included in the installation package of Grooper and are required for installation.		The following are included in the installation package of Grooper and are required for installation.
Line 231:		Line 231:
	:Microsoft Access Database Engine contains essential components that make it possible for Microsoft Office programs to interact with software services and applications that do not belong to Microsoft. For instance, if you are working in a program that gives you the possibility to export tables directly to Microsoft Access, then you need the database engine to make that happen.		:Microsoft Access Database Engine contains essential components that make it possible for Microsoft Office programs to interact with software services and applications that do not belong to Microsoft. For instance, if you are working in a program that gives you the possibility to export tables directly to Microsoft Access, then you need the database engine to make that happen.
	:The Microsoft Access Database Engine is available in multiple editions, which mainly depend on which Microsoft Office product you’re using. The hint is in the year. As such, if you use Microsoft Office 2016, then you need Microsoft Access Database Engine 2016.		:The Microsoft Access Database Engine is available in multiple editions, which mainly depend on which Microsoft Office product you’re using. The hint is in the year. As such, if you use Microsoft Office 2016, then you need Microsoft Access Database Engine 2016.
			</div>

	== About thread allocation for Grooper services ==		== About thread allocation for Grooper services ==

Randallkinard: /* Diagram: Common types of Grooper environments */

2026-01-22T19:04:06Z

Diagram: Common types of Grooper environments

← Older revision		Revision as of 14:04, 22 January 2026
Line 808:		Line 808:

	[[File:2025 GrooperInfrastructure DistributedEnvironment.png]]		[[File:2025 GrooperInfrastructure DistributedEnvironment.png]]
			</div>
	<section end = "centralized vs distributed" />		<section end = "centralized vs distributed" />
	~~</div>~~

Randallkinard: /* Diagram: Common types of Grooper environments */

2026-01-22T19:01:54Z

Diagram: Common types of Grooper environments

← Older revision		Revision as of 14:01, 22 January 2026
Line 786:		Line 786:
	* Centralized environment - All components are installed on a single machine. This setup is good for a test Grooper environment but not generally suitable for production environments.		* Centralized environment - All components are installed on a single machine. This setup is good for a test Grooper environment but not generally suitable for production environments.
	* Distributed environment - Components are installed on multiple networked machines. An environment is "fully distributed" if each core component is isolated to its own server.		* Distributed environment - Components are installed on multiple networked machines. An environment is "fully distributed" if each core component is isolated to its own server.
			<div style="padding-left: 1.5em;">
	=== Centralized environment (suitable for testing only) ===		=== Centralized environment (suitable for testing only) ===

Line 809:		Line 809:
	[[File:2025 GrooperInfrastructure DistributedEnvironment.png]]		[[File:2025 GrooperInfrastructure DistributedEnvironment.png]]
	<section end = "centralized vs distributed" />		<section end = "centralized vs distributed" />
			</div>

Dgreenwood: /* Fully distributed environment */

2026-01-08T17:24:57Z

Fully distributed environment

← Older revision		Revision as of 12:24, 8 January 2026
Line 802:		Line 802:
	** Example: The Grooper Database and Grooper File Store are often hosted on their own servers.		** Example: The Grooper Database and Grooper File Store are often hosted on their own servers.
	*** Databases are resource-intensive. They need consistent CPU, RAM, and disk I/O for query execution and indexing.		*** Databases are resource-intensive. They need consistent CPU, RAM, and disk I/O for query execution and indexing.
	*** File storage operations (uploads/downloads) can be very I/O heavy and compete for the same resources. If both ~~run~~ on the same machine, large file transfers can slow down database queries, causing latency spikes.		*** File storage operations (uploads/downloads) can be very I/O heavy and compete for the same resources.
			*** If both the Grooper database and file store operate on the same machine, large file transfers can slow down database queries, causing latency spikes.
	* Distributing components allows for better scaling as well. Grooper services (Activity Processing mainly) can be installed on any number of machines to increase throughput.		* Distributing components allows for better scaling as well. Grooper services (Activity Processing mainly) can be installed on any number of machines to increase throughput.
	* Environments may also be ''partially'' distributed with multiple Grooper components hosted on one or more machine. However, for larger installs we recommend setting up a fully distributed environment.		* Environments may also be ''partially'' distributed with multiple Grooper components hosted on one or more machine. However, for larger installs we recommend setting up a fully distributed environment.