2023.1:Separation (Concept)

This article is about an older version of Grooper.

Information may be out of date and UI elements may have changed.

2025

2023.1

2023

Separation is the process of taking an unorganized inventory_2 Batch of loose contract Batch Pages and organizing them into documents represented by folder Batch Folders in Grooper. This is done so Grooper can later assign a description Document Type to each document folder in a process known as "classification".

About

Let's revisit the Five Phases of Grooper.

Acquire
- Either physical pages are scanned into Grooper or digital files are imported into a Batch in Grooper.
Condition
- This involves running Recognize and OCR on the Batch to allow Grooper to read the text and clean up the pages if needed.
Organize
- This is where you separate the pages in the Batch into individual document folders.
- After the pages have been separated, then the document folders are classified.
Collect
- Data is extracted from the documents.
Deliver
- The extracted data is exported from Grooper to the destination of your choice.

What is Separation?

Imagine you have a bunch of pages and you put them in a box. This is similar to importing pages into a Batch.

Now, let's say you need to look for a specific type of document in the box. It would be difficult to just go through all of the loose pages to find the documents you are looking for. It's difficult to determine where one document ends and another begins.

So, you take those pages and you sort them into folders. Each folder contains one document which is comprised of one or more pages. Now, it is much easier to tell the documents apart!

This is how separation works. It is organizing the pages so that Grooper can identify one document from another.

Why do we need to separate?

The point of separation is so Grooper can later Classify the documents. Classification is the process of assigning Document Types to the document folders. Classification can only be applied to a document folder, not loose pages. So, even if you are only bringing in a single document into Grooper, you would need to make sure the loose pages are contained within a document folder in order to classify.

The point of document classification is to let Grooper know what to do with the documents.

For example, Grooper won't know what data to extract from a document without a Data Model. Where does that Data Model come from? A Document Type. How does a document folder get a Document Type? Classification!
For more information on classification, please take a look at our Classification wiki article.

When do we need to separate?

There are two ways to bring documents into Grooper. You can scan physical copies of the pages directly into a Batch or you can import digital documents into Grooper. After you have your documents in your Batch, whether or not you need to separate depends first on whether your documents are scanned or imported. If imported, whether or not they need to be separated depends on if they are "Discreet" or "Packeted" documents.

Scanned Documents

Documents scanned into Grooper always need to be separated. Documents are scanned page by page into a Batch. Separation takes those loose pages and turns them into documents by creating a folder for each document and placing their pages inside.

Imported Documents: Discreet Documents

"Discreet" documents are digital documents where each file contains one document. When a file is imported into Grooper, it is automatically put in its own document folder. If the file only contains one individual document, then it will already be in its own document folder and there is no need to separate.

Imported Documents: Packeted Documents

Sometimes when you import digital documents, each file might contain multiple documents. These are considered "Packeted" documents. In this case, Grooper will bring the file in as a document folder, but it will need to be separated so that each document is contained within its own document folder.

How do we separate?

Separation is a process that happens as part of a Batch Process. You will need to add a Separate step (a Batch Process Step with the Activity property set to Separate) to your Batch Process and configure it with a Separation Provider. A Separation Provider is a property that tells Grooper how we want to run separation on the pages. For example, if each of our documents are four pages each, we might want to tell Grooper to separate every four pages. We could also separate every time Grooper finds a title at the top of a page.

There are 8 different Separation Providers you can configure. Here we are going to give a brief explanation of the provider, but for a deeper understanding you will need to visit each of their articles individually:

Change in Value Separation - Grooper will separate when it detects a value (that you configure) changes from one page to another, such as an invoice number.
Control Sheet Separation - During the scanning process, you can make sure to place a Grooper Control Sheet in between each document. Grooper will separate at the point it detects a Control Sheet.
EPI Separation - Grooper will separate based on extracted page numbers and will detect a new document when the page number resets or when a lower page number comes up in the Batch.
ESP Auto Separation - One of the more complicated separation techniques involving Lexical training.
- Since ESP Auto Separation uses a Content Model's training data (as well as classification rules set on its Document Types), it both separates and classifies documents during the Separate activity.
Event-Based Separation - Grooper will separate based on an "event" such as after X number of pages or any time Grooper encounters a blank page. The following are events that can be configured for this provider:
- Blank Page - A blank page will trigger a new folder.
- Barcode - A scanned barcode will trigger a new folder.
- Content Type - This Separation Event uses Lexical or Visual training examples to trigger folder creation. Whenever a page confidently matches a trained example document's first page, a new folder is created.
- Page Count - This is for fixed page separation. A new folder is created by a set number of pages for a document.
- Shape - A new folder is created every time a "shape feature" is detected.
Multi Separator - This provider allows you to use multiple Separation Providers at once.
Pattern-Based Separation - Grooper will separate based on text patterns, such as a document title or label.
Undo Separation - This provider actually turns separated documents back into loose pages.

Text-Based vs. Scan Supported

It is important to note that the majority of Separation Providers are "Text-Based" providers. That means they require readable data from your pages to work. OCR must be performed prior to any Text-Based provider being used. The following are Text-Based providers:

Change in Value Separation
EPI Separation
ESP Auto Separation
Pattern-Based Separation

"Scan Supported" providers do not need readable data to work. That means it is possible to run these providers as early as when you are scanning in physical pages. OCR is not required prior to running a Scan Supported provider. The following are Scan Supported Providers:

Control Sheet Separation
Event-Based Separation

Glossary

Activity: Activity is a property on edit_document Batch Process Steps. Activities define specific document processing operations done to a inventory_2 Batch, folder Batch Folder, or contract Batch Page. Batch Process Steps configured with specific Activities are frequently referred by the name of the Activity followed by the word "step". For example: Classify step.

Batch Page: contract Batch Page objects represent individual pages within a inventory_2 Batch. The Batch Page object is the most granular unit in the hierarchy of Batch Objects in Grooper.

Batch Pages are frequently referred to simply as "pages".

Batch Process Step: edit_document Batch Process Step objects are specific actions within the sequence defined by a settings Batch Process. A Batch Process Step plays a critical role in automating and managing the flow of documents through the various stages of processing within Grooper.

Batch Process Steps are frequently referred to as simply "steps".
Because a single Batch Process Step executes a single Activity configuration, they are often referred to by their referenced Activity as well. For example, a "Recognize step".

Batch Process: settings Batch Process objects are crucial components in Grooper's architecture. A Batch Process orchestrates the document processing strategy and ensures each inventory_2 Batch of documents is managed systematically and efficiently.

Batch Processes by themselves do nothing. Instead, the workflows they execute are designed by adding child edit_document Batch Process Steps.
A Batch Process is often referred to as simply a "process".

Batch: inventory_2 Batch objects are fundamental in Grooper's architecture as they are the containers of documents that get moved through Grooper's workflow mechanisms known as settings Batch Processes.

Change in Value Separation: The Change in Value Separation Provider creates a new folder and separates every time an extracted value changes from one contract Batch Page to another.

Classification: Classification is the process of identifying and organizing documents into categorical types based on their content or layout. Classification is key for efficient document management and data extraction workflows. Grooper has different methods for classifying documents. These include methods that use machine learning and text pattern recognition. In a Grooper Batch Process, the Classify Activity will assign a Content Type to a folder Batch Folder.

Classify: unknown_document Classify is an Activity that "classifies" folder Batch Folders in a inventory_2 Batch by assigning them a Content Type (e.g. a description Document Type) using patterns, lexical understanding, or rules as defined by a stacks Content Model.

Content Model: stacks Content Model node objects define the taxonomy of document sets in terms of the description Document Type they contain. They also house the Data Elements that appear on each collections_bookmark Content Category and Document Type within them. Content Models serve as the root of a Content Type hierarchy and are crucial for organizing the different types of documents that Grooper can recognize and process.

Content Type: Content Type refers to objects in Grooper used to classify folder Batch Folders. These include: stacks Content Models, collections_bookmark Content Categories, and description Document Types.

Control Sheet Separation: Control Sheet Separation is a Separation Provider that uses Grooper document_scanner Control Sheets to separate documents.

Data Model: data_table Data Model node objects serve as the top-tier structure defining the taxonomy for Data Elements and are leveraged during the Extract Activity to extract data from a folder Batch Folders. They are a hierarchy of Data Elements that sets the stage for the extraction logic and review of data collected from documents.

Document Type: description Document Type objects represent a distinct type of document, like an invoice or contract. Document Types are created as children of a stacks Content Model or a collections_bookmark Content Category and are used to classify individual folder Batch Folders. Each Document Type in the hierarchy defines the Data Elements and Behaviors that apply to Batch Folders of that specific classification.

EPI Separation: The EPI Separation Separation Provider uses embedded page information ("EPI") to Separate loose pages into document folders. A Data Extractor is used to find page numbers from the text on a page and Grooper uses this information to separate the pages.

ESP Auto Separation: ESP Auto Separation is a Separation Provider used for document separation. It is unique in that it both separates and classifies documents at the same time. It uses page-level classification training examples (among other things) to determine where to insert document folders in a inventory_2 Batch.

Event-Based Separation: Event-Based Separation is a Separation Provider that Separates documents using one or more "Separation Events". Each Separation Event triggers the creation of a new folder.

Five Phases of Grooper: The "Five Phases of Grooper" is a conceptual term that seeks to build understanding of how documents are processed through Grooper.

Lexical: The Lexical Classification Method classifies folder Batch Folders based on the text content of trained document examples. This is achieved through the statistical analysis of word frequencies that identify description Document Types.

Multi Separator: The Multi Separator Separation Provider performs separation using multiple Separation Providers. It allows users to create a list of any of the other Separation Providers. If the first provider on the list fails to separate a page (or, as more often is the case, a series of pages), the next one will be applied. If that fails, the next, and so on.

OCR: OCR is stands for Optical Character Recognition. It allows text on paper documents to be digitized, in order to be searched or edited by other software applications. OCR converts typed or printed text from digital images of physical documents into machine readable, encoded text.

Pattern-Based Separation: Pattern-Based Separation is a Separation Provider that creates a new document folder every time a value returned by a defined pattern is encountered on a page.

Pattern-Based: Pattern-Based is a Collation Provider option for pin Data Type extractors. Pattern-Based uses regular expressions to sequence returned results into a final result set.

Recognize: format_letter_spacing_wide Recognize is an Activity that obtains machine-readable text from contract Batch Pages and folder Batch Folders. Recognize will selectively perform OCR for images and native-text extraction for digital text in PDFs. Recognize can also be configured to collect "layout data" like lines, checkboxes, and barcodes. Various other Activities then use this machine-readable text and layout data for document analysis and data extraction.

Separate: insert_page_break Separate is an Activity that sorts contract Batch Pages into individual folder Batch Folders. This distinguishes "loose pages" from the documents formed by those pages. Once loose pages are separated into Batch Folder documents, they can be further processed by unknown_document Classify, export_notes Extract, output Export and other Activities that need to run on the folder (i.e. document) level.

Separation Provider: The Provider property of the Separate Activity defines the type of separation to be performed at the designated Scope.

Separation: Separation is the process of taking an unorganized inventory_2 Batch of loose contract Batch Pages and organizing them into documents represented by folder Batch Folders in Grooper. This is done so Grooper can later assign a description Document Type to each document folder in a process known as "classification".

Undo Separation: Undo Separation is a Separation Provider. Instead of putting loose contract Batch Pages into folder Batch Folders, this Separation Provider removes Batch Folders, leaving only loose pages.

Visual: The Visual Classification Method uses image analysis instead of text data to determine the description Document Type assigned to a folder Batch Folder during classification. Instead of using text-based extractors, an "Extract Features" IP Command in an perm_media IP Profile is used to collect image-based data from a Batch Folder's image(s). This image-based data is compared against that of previously trained document examples of each Document Type to classify the Batch Folder.