Grooper Infrastructure (Concept)

⚠	This article was migrated from an older version and has not been updated for the current version of Grooper. This tag will be removed upon article review and update.

This article is about the current version of Grooper. Note that some content may still need to be updated.
2024	2023.1

Grooper Infrastructure refers to the computing underpinnings of what makes up a Grooper Repository and the software that allows the Grooper platform to automate tasks and users to interface with it.

Technically speaking, what is Grooper? Grooper is a repository of information made up by 12 tables (11 as of Grooper v2021) in a SQL Server Database and associated files in a Windows file share. This information is displayed to the user via an application(s) that displays a 1 to 1 relationship of what exists in that database and fileshare. Gaining an understanding of how to properly create using Grooper begins by first understanding the structure of its architecture, and why it is built the way it is. This sets a foundational groundwork for knowing how to think and interact with the application.

Grooper infrastructure

System Architecture and Requirements

End User PC (Thick Client)

This refers to a machine with a "thick client" Grooper install connected to a Grooper host server.

• Windows 10 or later is required.
• Network latency should be (ideally) less than 50ms.
• Network throughput should be (ideally) greater than 50MB.
• "Thick client" refers to Grooper software installed locally on a machine.
  • This allows for the thick client Grooper applications (Grooper Design Studio, Grooper Dashboard and Grooper Config) to be run locally on that machine (version "2023.1" and older).
  • Users would include scan operators and document review operators running Grooper Dashboard or Grooper designers and admins using Grooper Design Studio to design solutions and Grooper Config to install Grooper Services.

Client Computer (Web Client Only)

This refers to a machine connecting to a web client Grooper install via a browser. The client computer will not need a thick client Grooper installation.

• Google Chrome, Microsoft Edge or Apple Safari are required.
  • Other browsers may work but are not supported
• User logs in using Windows Account (Domain or Local) on Grooper Web Server.

Additional requirements for remote scanning users only

This refers to a machine hooked up to a scanner, used to scan documents into Grooper using the web client.

• A Windows 10 or later PC is required for Grooper Desktop.
  • Grooper Desktop is a lightweight application installed separately from the main installation of Grooper. It is used to allow a computer with a physical document scanner to upload scanned documents directly to the Grooper Web Server.
• For more information visit the Desktop Scanning in Grooper article.

Grooper Processing Server

This refers to any machine in the Grooper "processing environment", including the primary Grooper host server and any machine dedicated to hosting Grooper Activity Processing services.

• Windows Server 2016 or later is required.
• Server RAM should be 16GB or more.
• The CPU should consist of 4 or more cores.
  • Be aware this is a minimum. High volume scenarios will need more cores to process content efficiently.
• The port for licensing is 13900 by default.
• A processing environment can have as many servers as is necessary to meet business requirements.

Grooper SQL Server

This refers to any machine installed with Microsoft SQL Server hosting the Grooper Database.

• Windows Server 2016 or later is required.
• SQL Server 2014 or later is required.
• The Grooper Database should allow for a size of 20GB or more.
• The scaling of the Grooper Database should be set to percentage, ideally, not the default of a few MB. This will reduce frequent scaling that hinders performance.
• The SQL Port is 1433.
• For smaller installs, the Grooper SQL Server may be the same machine as the primary Grooper host server. For larger installs, we recommend a dedicated server for the Grooper SQL Server.

⚠	Be aware. We do not recommend using SQL as a Service (SQLaaS) solutions to sand up the Grooper Database for most scenarios. Please refer to our Cloud Deployment Guidance below for more information.

Grooper Filestore Share

This refers to the machine hosting the Grooper Filestore folders.

• Windows Server 2016 or later is required
• The ideal file size would accommodate for roughly 150GB for small to medium installs.
• The authentication requires CIFS or SMB.
• For smaller installs, the Grooper Filestore may be located on the primary Grooper host server. For larger installs, the Grooper FileStore should be located on its own server.

Grooper Web Server (Web Client Only)

This refers to the server hosting the Grooper Web Client application.

• Windows Server 2016 or later is required
• Server RAM should be 16GB or more.
• The CPU should consist of 4 or more cores.
• The port for licensing is 13900 by default.
• Internet Information Services (IIS) must be installed and properly configured.
  • For all configuration requirements visit the Installing IIS portion of the "Install and Setup" article.
• The Grooper app pool identity configured in IIS must have read/write access to the Grooper Filestore and Database.
  • For all app pool user permission requirements visit the App Pool Identity Permissions portion of the "Install and Setup" article.
• Requires an SSL Certificate for encrypted connections.
• BE AWARE: Grooper must also be installed on this server. The Grooper Web Client app leverages Grooper DLLs to function.
• For smaller installs, the Grooper Web Server and Grooper SQL Server may be the same machine. For larger installs, this is not recommended. We recommend a dedicated server for the Grooper Web Server.

Windows Prerequisites

The following are included in the installation package of Grooper and are required for installation (listed are x64 packages.)

• vc_redist_2012_update4_x64

  Microsoft Visual C++ 2012 Redistributable (x64) - 11.0.61030

• vc_redist_2010_sp1_x64

  Microsoft Visual C++ 2010 x64 Redistributable Setup

• vc_redist_2013_x64

  Microsoft Visual C++ 2013 Redistributable (x64) - 12.0.30501

• vc_redist.x64

  Microsoft Visual C++ 2017 Redistributable (x64) - 14.10.25017

Visual C++ Redistributable Packages install runtime components of Visual C++ Libraries on a computer that does not have Visual C++ installed. The libraries are required to run applications that are developed by using the corresponding version of Visual C++.

• NDP462-DevPack-KB3151934-ENU

  Microsoft .NET Framework 4.6.2 Developer Pack

.NET is an integral part of many apps running on Windows and provides common functionality for those apps to run. For developers, the .NET Framework provides a comprehensive and consistent programming model for building apps that have visually stunning user experiences and seamless and secure communication.

• SQLSysClrTypes

In LINQ to SQL, the data model of a relational database maps to an object model that is expressed in the programming language of your choice. When the application runs, LINQ to SQL translates the language-integrated queries in the object model into SQL and sends them to the database for execution. When the database returns the results, LINQ to SQL translates the results back to objects that you can work with in your own programming language.

In order to translate data between the object model and the database, a type mapping must be defined. LINQ to SQL uses a type mapping to match each common language runtime (CLR) type with a particular SQL Server type. You can define type mappings and other mapping information, such as database structure and table relationships, inside the object model with attribute-based mapping. Alternatively, you can specify the mapping information outside the object model with an external mapping file.

• AccessDatabaseEngine_x64

Microsoft Access Database Engine contains essential components that make it possible for Microsoft Office programs to interact with software services and applications that do not belong to Microsoft. For instance, if you are working in a program that gives you the possibility to export tables directly to Microsoft Access, then you need the database engine to make that happen.

The Microsoft Access Database Engine is available in multiple editions, which mainly depend on which Microsoft Office product you’re using. The hint is in the year. As such, if you use Microsoft Office 2016, then you need Microsoft Access Database Engine 2016.

• ReportViewer

Used for viewing built in Grooper Reports, which uses a Microsoft Reporting framework.

Click here for a "light mode" version of the above image

About thread allocation for Grooper services

Be aware of the "n minus one" rule!

Services are assigned a number of CPU threads when you install them. For some services, like Import Watcher, the will always run using a single thread. Activity Processing services can run using multiple threads.

Keep in mind, your machine only has a certain number of processing threads available. You will run into errors if you over-allocate your available threads.

Remember too, the operating system itself must always have a single thread available to run. So, the absolute maximum number of threads you can assign to all your services should not go beyond the total number of threads available minus one reserved for the operating system. Hence, the "n minus one" rule.

The "n minus one" rule is as follows:

• If "n" is the maximum number of threads available on your machine, the maximum number of threads you can distribute to Grooper services is "n" minus one.

Be aware of the "n minus x" rule!

Other programs running in the background will need threads to run as well.

• If SQL is installed on the same machine as your Grooper services, you should follow an "n minus two" rule, reserving one for the OS and one for SQL.
• If IIS and SQL are installed on the same machine as your Grooper services, you should follow an "n minus three" rule, reserving one for the OS, one for SQL, and one for IIS.
• If other applications, such as anti-virus software, are running in the background you will need to reserve threads for those applications as well.
• This is one of the many reasons why we recommend distributing SQL, IIS, and processing services across multiple machines.

THE LONG STORY SHORT HERE IS DO NOT OVER-ALLOCATE YOUR AVAILABLE THREADS! GROOPER CAN BEHAVE ERRATICALLY IF YOU DO!

Cloud Deployment Guidance

Grooper can be deployed using a variety of cloud-based resources. At its core, Grooper needs access to a SQL database and a Windows file system to set up a Grooper Repository. Whether you’re deploying using Amazon Web Services or Microsoft Azure or some other cloud computing platform, you should be able to deploy Grooper as long as you have those two things.

However, in practice, we have found not all cloud services are created equal.

SQL Guidance

For high volume environments, we recommend hosting SQL on a VM rather than using a “SQL as a Service” solution (for example Azure SQL Database).

• High volume environments have a high transaction rate with the Grooper Database which can present several issues when using a SQLaaS solution.
• High volume environments will experience better performance using “Simple” recovery mode as opposed to full. If the SQLaaS does not allow you to use “Simple” recovery, do not use it.
• If you already have SQL licensing, it may also be more cost effective for your organization to host SQL on a VM.

For all environments, you may experience time out issues when using a SQLaaS solution over a VM.

• For small (mostly user directed) environments, this will be less of an issue. For larger (more automated and larger volume) environments, this issue will be more apparent.

Even when hosting SQL on a VM, we recommend configuring autogrow by fairly small increments.

• In a recent high-volume deployment using Microsoft Azure, we found an autogrow setting of 2048 Mb to be problematic. The VM could not grow the database in time to keep up with Grooper’s transaction rate. However, dropping autogrow down to 256 Mb resolved the issue.

File Store Guidance

We do not recommend using Azure Files currently. Users should use Azure Disk Storage instead.

• Azure Files creates files in such a way that we have experienced multithreading issues when an item is created/modified in the file store by one thread and then (nearly) immediately accessed by another thread.
  • A common scenario would be a Split Pages step creating a page in the file store then a subsequent Image Processing step immediately attempting to process the created page in the next step. The page is saved to the file store in such a way that Grooper may “know” a file exists, but is not able to immediately access it for the Image Processing activity to process it, resulting in an error being thrown.
  • This will be more common in high volume environments with large amounts of threads allocated to processing tasks. However, it will likely affect even small environments.

The Five Layers of Grooper

Grooper consists of five main components that constantly interact with one another:

1. Database
2. File Store
3. Application
4. Licensing
5. Web Server

The Database

This is the core component of Grooper as all aspects of the functionality and configuration of a Grooper repository are stored here as metadata in tables. There are 15 tables (as of Grooper version 2024) that make up the Grooper database:

Grooper 2023 and 2023.1Deprecated Tables

dbo.CustomStats

• This table contains a name / value pair of processing activity statistics. The name is the name, and the value will be a whole integer if the statistic indicates a quantity of actions, and a floating point decimal if the value represents a cumulative time span.

Column Name	Data Type	Size	Description
SessionId	int	4
Name	Varchar	64
Value	Float	8

dbo.FileStoreEntry

• This table contains the information necessary for associating filestore objects with Grooper nodes. Note: a node with associated files will also contain those files’ filestore object locations, names, and mimetypes in the “files” column of TreeNode.

Column Name	Data Type	Size	Description
Id	int	4	sequential number assigned to object in row
FileId	uniqueidentifier	16	guid assigned as the name of the object (stored in the filestore with this number followed by .grp)
NodeId	uniqueidentifier	16	guid associated with node from Grooper tree
FileStoreId	uniqueidentifier	16	guid given to the node referencing the file store

dbo.ImportJob

This table is not present prior to Grooper 2023

• The entries in this table are related to the creation of Batches in Grooper via the submission of Import Jobs from an Import Provider. Each row is an Import Job to be processed.

Column Name	Data Type	Size	Description
Id	uniqueidentifier	4	unique row identifier
Description	varchar	256	description of the job as written by the end user
Provider	varchar	(-1)	the import provider leveraged by this job
Priority	int	4	numeric value (1-5) given to sort batch priority
Status	int	4	status as indicated numerically (ready, working, completed, paused)
SubmittedBy	varchar	256	name of account that submitted the import job
Submitted	datetime	8	date/time when import job was submitted
Started	datetime	8	date/time when import job started processing
Completed	datetime	8	date/time when import job completed
Files	int	4	number of files imported with job
Bytes	bigint	8	size of files in batch created on import

dbo.Lock

• This table contains one row per locked node; prevents overlapping access to various resources.

Column Name	Data Type	Size	Description
NodeId	uniqueidentifier	16	guid of node from Grooper hierarchy tree
UserName	varchar	64	windows user that applied the unlock
MachineName	varchar	64	hostname of the machine where the unlock occurred
ModuleName	varchar	64	Grooper module that executed the unlock
ProcedureName	varchar	64	the procedure executed to apply the unlock
CreateDate	smalldatetime	4	date and time the unlock occurred

dbo.Log

• This table contains the Grooper log, which is represented in the Event Viewer of Grooper Design Studio.

Column Name	Data Type	Size	Description
Id	int	4	unique row identifier
TimeStamp	datetime	8	date and time the event was logged
Type	int	4	type of event logged
ProcName	varchar(MAX)	-1	name of procedure associated with the event
Message	varchar(MAX)	-1	message displayed in and defining, in general, the event
UserAccount	varchar	64	windows user under which the event triggered
MachineName	varchar	64	hostname of the machine under which the event triggered
NodeId	uniqueidentifier	16	guid of parent node associated with ‘NodeName’
NodeName	varchar	256	Item Name, or name of node associated with the triggering of the event
NodeType	varchar	64	type of Grooper Node defining the ‘NodeName’

dbo.NodeReference

• This table contains a list of all referenced nodes, and what other nodes they are referenced by. Used to protect referenced nodes from deletion, and in determining what nodes are necessary on exporting.

Column Name	Data Type	Size	Description
NodeId	uniqueidentifier	16	guid given to node in Grooper hierarchy tree
ReferencingNodeId	uniqueidentifier	16	guid of parent node associated with ‘NodeId’

dbo.ProjecssingJob

This table is not present prior to Grooper 2023

• This table contains jobs submitted from activities of a Batch Process. A single Processing Job may consist of many individual Processing Tasks, so this table is related to dbo.ProcessingTask.

Column Name	Data Type	Size	Description
Id	int	4	unique row identifier
ActivityType	varchar	256	name of automated activity, or task, to be completed
BatchProcess	varchar	256	name of Batch Process that submitted activities belong to
StepId	uniqueidentifier	16	GUID of Batch Process Step configured as the activity being submitted for processing
StepNo	int	4	numeric position of the Batch Process Step within its parent Batch Process
StepName	varchar	256	name given to Batch Process Step
Priority	int	4	numeric value (1-5) given to sort batch priority
Status	int	4	status as indicated numerically (ready, working, completed, paused, error)
SubmittedBy	varchar	256	name of user that submitted job
BatchId	uniqueidentifier	16	GUID of batch related to running process
BatchName	varchar	256	name of batch in processing
ThreadPoolId	int	4	numeric id of associated Processing Queue
Completed	datetime	8	date/time job was completed

dbo.ProcessingTask

• This table contains tasks submitted for activity processing as part of production batches. Multiple similar Processing Tasks make up the whole of a Processing Job, so this table is related to dbo.ProcessingJob.

Column Name	Data Type	Size	Description
Id	int	4	unique row identifier
BatchId	uniqueidentifier	16	guid of batch related to running process
NodeId	uniqueidentifier	16	guid of node related to batch in running process
Settings	varchar(MAX)	-1	json properties of running node
ModuleName	varchar	256	module running current task
Status	int	4	status of current task (i.e. running, error…)
UserName	varchar	256	windows user that submitted task
MachineName	varchar	256	hostname of machine running task
PageCount	int	4	number of pages in batch being processed
Submitted	datetime	8	date/time when processing batch was submitted
Started	datetime	8	date/time when processing batch started
Completed	datetime	8	date/time when batch process completed
Priority	int	4	numeric value (1-5) given to sort batch priority
Scope	int	4	the level of the batch that a step is processing
StepNo	int	4	current step of batch process
BatchName	varchar	256	name of batch in processing
BatchProcessName	varchar	256	name of process associated with batch in processing
StepName	varchar	256	name of step of process associated with batch in processing

dbo.ServiceInstance

• This table contains a list of installed services. Note: If an installed service does not have a reference in this table, it will not show up in Grooper Config.

Column Name	Data Type	Size	Description
Id	int	4	unique row identifier
RepositoryId	uniqueidentifier	16
MachineName	varchar	64	hostname of pc running Grooper service
ServiceName	varchar	256	name of Grooper service running
TypeName	varchar	256	typer of Grooper service running
PropertiesJson	varchar	2048	json information defining /Node Properties/Properties
UserName	varchar	64	windows user running Grooper service
Password	varchar	64	password of windows user running Grooper service
InstanceNo	int	4	number of a specific service on a machine
ThreadPirority	int	4	numeric value given to service thread to determine processor priority

dbo.SessionStats

• This table contains statistical information regarding batches in Grooper.

Column Name	Data Type	Size	Description
Id	int	4	unique row identifier
BatchId	uniqueidentifier	16	guid for batch node
BatchName	varchar	256	name of batch
BatchProcessId	uniqueidentifier	16	guid of process node related to the batch
BatchProcessName	varchar	256	name given to process by end user
StepName	varchar	256	name of step in the process
ProcessingScope	in	4	the level of the batch that a step is processing
ModuleName	varchar	256	name of Grooper module
ModuleDisplayName	varchar	256	name of Grooper module given to batch process step
MachineName	varchar	256	hostname of pc running batch
UserName	varchar	256	windows user under which the batch is running
StartTime	datetime	8	time batch was initiated
EndTime	datetime	8	time batch ended or was terminated
TasksProcessed	int	4	number of tasks processed in batch

dbo.Setting

• This table contains only the Database version, including build number.

Column Name	Data Type	Size	Description
Name	varchar	64	DatabaseVersion
Value	varchar	1024	Current installed Grooper Database version

dbo.TreeNode

• The main Grooper table, TreeNode contains one entry for every object in the Grooper node tree. Composed of the following columns:

Column Name	Data Type	Size	Description
Id	uniqueidentifier	16	The node’s unique ID. A GUID autogenerated by Grooper.
RowId	int	4	Identity Column for the table. Generated by SQL server.
RowVersion	timestamp	8	A timestamp column, updated when the node is changed. Used by certain processes to determine if the node has been updated.
Name	varchar	256	The name of the node.
TypeName	varchar	64	The name of the object type of the node
ParentId	uniqueidentifier	16	ID of the node’s parent. The Grooper Root node has a parent ID with all zeros.
NodeIndex	int	4	Index of the node in its containing object (usually a folder)
Attributes	int	4	A flags attribute indicating whether the node has the following attributes: read-only, fixed contents (no children may be added), sorted (children will always show up in alphabetical order), has ACL, is a system object (may not be changed).
NumChildren	int	4	Number of children.
Properties	varchar(MAX)	-1	Column containing the JSON properties for this node.
NodeValues	varchar(MAX)	-1	metadata about the node that pertains to Grooper modules
Files	varchar(MAX)	-1	objects in the Grooper file store associated with the node

These tables may exist in previous versions of Grooper but have been deprecated as the platform evolved.

dbo.AccessControlEntry

This table has been deprecated as of Grooper version 2023

• This table contains the information necessary for implementing Grooper’s node-level security architecture.

Column Name	Data Type	Size	Description
Id	int	10	unique number referencing the table's row
NodeId	uniqueidentifier	16	guid associated with node from Grooper node tree
PrincipalId	uniqueidentifier	16
Permissions	in	4

Using a SQL database as the core of Grooper allows for great efficiency. Every property of every object in Grooper, as a result of being a simple entry in a table, can be loaded into memory and accessed nearly instantly. This would not be the case otherwise (if for example Grooper repositories were defined by something like a project file), as file i/o is one of the slowest aspects of modern computing. This also allows the discrete management of objects on an individual basis to allow multiple users to work in one environment and prevent work overlap by locking objects.

The File Store

The File Store in Grooper is a file share in a Windows environment. It houses the files associated with objects in Grooper that have information that would otherwise be inefficient to store in (a cell in) a database table.

The Grooper Filestore exists at a user-specified location. This may be a local or an network path, but if a filestore is given a local path, computers connecting to that repository remotely will not be able to access it. If you want to set up a repository so that other computers can connect to it, make sure you reference the filestore using a UNC path!

The filestore contains three levels of directories. A typical filestore entry will exist on disk as, e.g. 00 > 00 > 00 > [guid].grp. Each of the lowest-level folders in the filestore will have a maximum of 256 files, at which point a new folder at that level will be created. If the lowest level contains 256 folders, a new folder will be created at the level above; this gives the Grooper filestore a limit of 256 ^ 4 = 4,294,967,296 files stored on disk.

While the filestore entries are all given .grp extensions, the contents of the file are unaltered from their “actual” form. If you navigate, for example, to the .grp file associated with an pdf imported using full import, you can open it and view it with a pdf viewer. The files in the filestore are intentionally obfuscated to prevent users from interacting with them outside of Grooper, as they are essentially Grooper-internal objects.

Although the majority of files in the filestore relate to batch objects (split pages, imported documents, image processing undo objects, etc.), some files are the result of other in-Grooper processes such as layout data, OCR character data, etc.

The Application

This is the most visible aspect of Grooper as it is the software you interact with that displays the currently active repository. It consists of several pieces:

1. Grooper Design Studio
   • This is the "main" Grooper application and where all the building and testing of extractors, models, profiles, and processes is done.
2. Grooper Config
   • This is the Grooper application that allows the creation of Grooper repositories and the management of Grooper services.
3. Grooper Dashboard
   • This application is meant to facilitate ease of management of Batches currently in production in Grooper. You can start, stop, and otherwise adjust Batches from here. You can accomplish everything Grooper Dashboard can do from Grooper Design Studio, but Grooper Dashboard gives a simplified interface for ease of management by stripping all the complexity of Grooper Design Studio and leaving just a list-view of the Batches.
4. Grooper Attended Client
   • This application handles all review modules in Grooper, such as Data Review or Classification Review.
5. Grooper Unattended Client
   • This application allows for the completion of tasks submitted by a Batch Process that a human would not interact with, such as Recognize or Extract. Typically tasks submitted would be picked up by an Activity Processing Service to be completed, but in the case one isn't running and you need to complete one of these "unattended" tasks, this application will launch and complete the work.

It is worth noting that these are the applications that can be run from the Grooper server. If you are interacting with Grooper via a web browser you will not interact with these applications. However, the vast majority of Grooper's functionality can be performed via the web interface. The only real case in which you need to interact with a Grooper application on the server is for Grooper Config. You cannot perform the functions of Grooper Config via a web browser.

Every object in the Grooper Node Tree is an object, or row, in a specific table, the dbo.TreeNode table. The GUI of Grooper is essentially wrapping information from the Grooper database, and associated files from the Grooper File Store, into a series of grids and windows that allow you to directly interact with that database, and its related Windows file share, without writing SQL queries.

Licensing

A Grooper license number is provided to you after confirmation of purchase. The license key provided can be used in a few different ways.

GROOPER HOSTED LICENSE

In Grooper Design Studio on the Grooper Root Node there is a License Serial# or URL property. If the provided license is put into this property it will provide licensing to that Grooper repository. This license can be used on any Grooper repository, so keep it secure.

TRADITIONAL ONLINE ACTIVATION

The license key can also be entered in Grooper Config via the Activate License command using the Online option of the Activation Method property. The installed license is fixed to the machine it was installed on but can be hosted remotely via a Grooper Licensing service. A hosted license can be rehosted to a different machine if need be. If you are launching Grooper Design Studio from the machine the license is hosted on, you will automatically have licensing. If, however, you are launching Grooper Design Studio on a machine that is not currently hosting the license, you can still attain the license by entering the URL of the hosted license on the License Serial# or URL property of the Grooper Root Node in Grooper Deisgn Studio.

OFFLINE ACTIVATION

If you are running Grooper on a machine that will not be connected to the internet and whish to activate a license on that machine you can use the Offline option of the Activation Method property of the Activate License menu of Grooper Config. When using this option you will supply the license key like normal but when the command is executed you will be given a URL you can take to a machine that is connected to the internet. From there you can download a license file that you can then use on the offline computer.

Web Server

If IIS is installed and properly configured on your Windows Server you can install the Grooper Web Server. This will create a website within the default site of IIS that allows Grooper Design Studio and Review to be accessed via a web browser.