2023:Data Table: Difference between revisions

From Grooper Wiki
← Older edit
No edit summary
Tag: Reverted
m Dgreenwood moved page Data Table to 2023:Data Table
 
(5 intermediate revisions by the same user not shown)
Line 1: Line 1:
{|cellpadding="10" cellspacing="5" style="margin:12px"
<onlyinclude>
|-style="background-color:#ed2330; color:white"
<blockquote style="font-size:14pt">
|style="font-size:14pt"|'''WIP'''||This article is a work-in-progress. This article is subject to change and/or expansion.  It may be incomplete or stop abruptly.  Notably, this article is missing screenshots.
'''Data Tables''' are '''[[Data Element]]s''' in a '''[[Data Model]]''' used to extract data from tables in documents.  Each column of data is represented by a '''Data Column''' object created as the '''Data Table's''' children.
 
This tag will be removed upon draft completion.
|}
 
<blockquote style="font-size:125%">
'''''Expressions''''' (not to be confused with regular expressions) are snippets of VB.Net code that expand Grooper’s core functionality.
</blockquote>
</blockquote>


There are four types of expressions:
Data is extracted using one of the following [[Table Extraction]] methods. Each of these methods take a different approach in modeling a table's structure:
* '''''Data Model Expressions''''' modify the behavior of '''Data Fields''' and '''Data Columns''' and their values.
* '''''Batch Process Expressions''''' modify the behavior of '''Batch Process''' steps.
* '''''IP Profile Expressions''''' modify the behavior of image processing steps ('''IP Steps''') in an '''IP Profile'''.
* '''''Mapping Expressions''''' modify the behavior of '''''Import/Export Mappings'''''.


{|cellpadding="10" cellspacing="5" style="margin:12px"
* '''''[[Row Match]]'''''
|-
* '''''[[Header-Value]]'''''
|style="font-size:125%; background-color:#36b0a7; color:white"|'''FYI'''
* '''''[[Grid Layout]]'''''  
|style="border: 4px solid #36b0a7"|
* '''''[[Tabular Layout]]'''''
This article has general information on what the various Grooper expressions are and how they work.  If you are looking for more specific examples of expressions, check out our Expressions Cookbook and LINQ articles.
* '''''[[Fluid Layout]]'''''
* [[Expressions Cookbook]]
* '''''[[Delimited Extract]]'''''
* [[LINQ to Grooper Objects]]
* '''''[[Fixed Width]]'''''
|}
</onlyinclude>


{|cellpadding="10" cellspacing="5" style="margin:12px"
== About ==
|-
|style="background-color:#662d91; font-size:125%"|'''&#128526;'''
|style="border: 4px solid #662d91"|
''Special thanks to BIS team member Dave Hanon for contributing this article!''
|}


== Data Model Expressions ==
Many documents contain data in a table, presented on the page as some kind of grid of information divided into rows and columns.  The '''Data Table''' object's purpose is to define the processing logic to model and collect tabular data. 


'''''Data Model Expressions''''' are VB.Net code snippets that modify the behavior of '''Data Fields''' and '''Data Columns''' and their values.  These expressions are commonly used to validate and/or manipulate extracted data, populate fields with system data, and sometimes even document metadata.
A '''Data Table''' can be added to your '''Data Model''' to extract data from the table's cells.  Once added to your '''Data Model''', you will add '''Data Columns''' to the '''Data Table'''.  You can add as many columns as you need to collect data from all (or only some) of the table's columns on the document.  '''Data Columns''' will exist as children of the '''Data Table''' in the node hierarchy.
* '''Data Columns''' also allow for additional configuration, such as assigning the '''''Value Type''''' for the extracted data in that column (decimal, string, Boolean, etc).
* There are several table extraction methods. Some will require configuration of a '''Data Table's''' '''Data Columns'''.  Others will not (or allow for optional configurations).  Please visit the [[Table Extraction]] article for more information on tabular data extraction in general.


There are four types of data model expressions:
{| class="fyi-box"
* '''''Default Value Expressions'''''
|-  
* '''''Calculated Value Expressions'''''
* '''''Is Valid Expressions'''''
* '''''Is Required Expressions'''''
 
{|cellpadding="10" cellspacing="5" style="margin:12px"
|-
|style="font-size:125%; background-color:#36b0a7; color:white"|'''FYI'''
|style="border: 4px solid #36b0a7"|
'''''Data Model Expressions''''' are configurable for '''Data Field''' and '''Data Column''' objects in a '''Data Model'''.  However, for the purposes of brevity, we will often use the generic term "field" throughout this article, referring both to '''Data Fields''' and '''Data Column''' cells.
|}
 
=== Default Value Expressions ===
 
A '''''Default Value Expression''''' is a VB.Net code snippet that determines the default value for a '''Data Field''' or '''Data Column'''. This can be as simple as a default text string, or it could be an expression that returns system variables or '''Batch''' object information.
* If the '''Data Field''' or '''Data Column''' is configured with an extractor, and that extractor returns a result during extraction, the default value will be overwritten.
 
{|cellpadding="10" cellspacing="5" style="margin:12px"
|-
|style="font-size:200%; background-color:#f89420; color:white"|&#9888;
|style="border: 4px solid #f89420"|
You ''cannot'' reference '''Data Fields''' or '''Data Columns''' by name using a '''''Default Value''''' expression.
 
If you do need to reference '''Data Fields''' or '''Data Columns''' in some kind of expression that populates a field with a default value, you need to use a '''''Calculated Value Expression''''' set in ''Set If Empty'' mode.
* Refer to the [[#Calculated Value Expressions|Calculated Value Expressions]] portion of this article for more information.
|}
 
==== <span style="color: #662d91; font-size: 115%">Return Type</span> ====
 
''''' Default Value Expressions''''' must return a result that is compatible with the data type of the '''Data Field''' or '''Data Column''' on which it is configured (set using their '''''Data Type''''' property).
* Ex: If a '''Data Field's''' type is set to ''String'', the expression must return a string value, or the field will throw an error.
 
==== Example Default Value Expression ====
 
The following expression would generate a random globally unique identifier (GUID):
* <code>Guid.NewGuid</code>
 
=== Calculated Value Expressions ===
 
A '''''Calculated Value Expression''''' is a VB.Net code snippet that calculates a value of a field based on the values of other fields, much like how a formula defines a relationship between various cells in a spreadsheet. In addition to mathematical operations and text string manipulation, these expressions can inspect the Grooper node tree, batch, and environment variables or file paths to calculate the desired value.
 
These expressions can be used in two ways:
* To populate empty fields with calculated (or manipulated) values
* To validate existing field values
 
==== <span style="color:#662d91; font-size:115%">Return Type</span> ====
 
'''''Calculated Value Expressions''''' must return a result that is compatible with the '''Data Field/Data Column's''' data type (set using their '''''Data Type''''' property).
* Ex: If a '''Data Field's''' data type is set to ''Integer'', its '''''Calculated Value Expression''''' must evaluate to an integer, or else an error will occur.
 
==== <span style="color:#662d91; font-size:115%">Referencing Data Fields and Data Columns by Name</span> ====
 
Peer fields ('''Data Fields''' at the same level of the '''Data Model''') and peer columns (sibling '''Data Columns''' of a '''Data Table''') can be referenced by name in '''''Calculated Value Expressions'''''.
* Ex: A '''Data Field's''' expression could reference a peer '''Data Field''' named "Subtotal" simply by writing <code>Subtotal</code> into the expression.
 
{|cellpadding="10" cellspacing="5" style="margin:12px"
|-
|style="font-size:200%; background-color:#f89420; color:white"|&#9888;
|style="border: 4px solid #f89420"|
You must substitute underscores for spaces and special characters in '''Data Column''' and '''Data Field''' names. 
* Ex:  A '''Data Field''' named "Social Security Number" would need to be typed out <code>Social_Security_Number</code>
 
Multiple spaces or special characters in a row must be substituted with a single underscore.
* Ex:  A '''Data Column''' named "Expenses - Sch. A" would need to be typed out <code>Expenses_Sch_A</code>
 
If a '''Data Field''' or '''Data Column's''' name begins with a number, you must prepend an underscore to the name.
* Ex: A '''Data Field''' named "911" would need to be typed out <code>_911</code>
 
 
If you’re unsure how to format a field’s name, simply begin typing it and Grooper’s built-in IntelliSense menu will show you the correct way.
|}
 
'''Data Fields''' in peer ''single instance'' '''Data Sections''' may be referenced, using dot notation.
* Ex: You could access the "DOB" '''Data Field''' inside a "Employee" '''Data Section''' with the expression <code>Employee.DOB</code>
* Be aware, this expression will fail if the '''Data Section's''' '''''Miss Disposition''''' property is set to ''No Instance'' and no section instance is extracted.
 
 
'''Data Fields''' in non-peer ''single instance'' '''Data Sections''' can also be referenced using dot notation.  However, you must reference the '''Data Model's''' parent '''Content Type''' ''first''.
* Ex: Given a '''Data Model''' for a '''Document Type''' named "Student Records":
**  A '''Data Field''' named "Average" in a '''Data Section''' named "Math" could access the "GPA" '''Data Field''' inside a "Totals" '''Data Section''' with the expression <code>Student_Records.Totals.GPA</code>
* Be aware, the expression will fail if the referenced '''Data Section's''' '''''Miss Disposition''''' property is set to ''No Instance'' and no section instance is extracted.
 
 
'''Data Fields''' in multi-instance '''Data Sections''' ''cannot'' be referenced whatsoever. 
* If you try to reference fields in multi-instance sections, you will find Grooper's IntelliSense will not show the field's name and the expression will throw an error if you manually type in the name.
 
==== <span style="color:#662d91; font-size:115%">Calculate Modes</span> ====
 
'''''Calculated Value Expressions''''' can execute using one of three modes, set by configuring the '''''Calculate Mode''''' property:
* ''Validate''
* ''Set If Empty''
* ''Always Set''
 
{|cellpadding=10 cellspacing=5
|style="width:10%"|
'''Mode'''
|
|
'''Description'''
'''FYI'''
|-
|valign=top|
''Validate''
|
|
''Validate'' will check that the field's value mathematically satisfies the '''''Calculated Value Expression'''''.
Generally, the first '''Data Column''' underneath your '''Data Table''' will correspond to the leftmost column and the last will be the rightmost.  The top '''Data Column''' lines up with the first column, the second '''Data Column''' the second column, and so on.   
* If it does not, puts the field in an error state. The error message on the field will show the difference between the field’s value and the expected result of the expression.
* As this mode pertains to mathematical validation only, it only works with numerical data types (Int, Decimal, Double).
** For non-mathematical validation, use an '''''Is Valid Expression'''''.
|-
|valign=top|
''Set If Empty''
|
''Set If Empty'' will only populate the field with the '''''Calculated Value Expression's''''' result if no value was collected during the '''Extract''' step of a '''Batch Process'''.
* In other words, if the field is still blank after extraction, the expression will run and fill the field with its result.  If there's anything at all in the field, the expression does nothing.
* Some Grooper users think of a '''''Calculated Value Expression''''' in ''Set If Empty'' mode as a more robust version of a '''''Default Value Expression'''''.
** '''''Calculated Value Expressions''''' can reference '''Data Elements''' (whereas '''''Default Value Expressions''''' cannot) and more methods than '''''Default Value Expressions'''''.
|-
|valign=top|
''Always Set''
|
''Always Set'' will always populate the field with the '''''Calculated Value Expression's''''' result (unless the expression fails to produce a result).
* When you use these types of expressions, you may not even configure the '''Data Field''' or '''Data Column''' with an extractor, instead generating the calculated field's value using using other field values in the '''Data Model''', system or document metadata or a combination of thereof.
** If an extractor ''is'' configured you may be using the expression to manipulate the extracted value to get a desired result (such as performing substring matching or some kind of mathematical operation).
* If any of the component values that make up the expression are modified, the '''''Calculated Value Expression''''' will update automatically.
** Ex: Imagine a calculated field's result is populated using a '''''Calculated Value Expression''''' that simply adds the values of two other fields together.  If one of those referenced field's value is changed manually during user review, the calculated field's value will automatically be updated by the '''''Calculated Value Expression'''''.
|}
 
==== <span style="color:#662d91; font-size:115%">Example Calculated Value Expression</span> ====
 
'''''!!EXAMPLE FORTHCOMING!!'''''
 
=== Is Valid Expression ===
 
An '''''Is Valid Expression''''' is a snippet of VB.Net code that determines whether a '''Data Field''' or '''Data Column’s''' value is valid.
 
==== <span style="color:#662d91; font-size:115%">Return Type</span> ====


'''''Is Valid Expressions''''' must return a Boolean (True/False) value:
However, this is not strictly necessary.   You can re-order the column order how you see fit.
* The expression must evaluate to “True” to be considered valid.
* If the expression returns “False,” an error is thrown stating "Validation Expression failed". The field's background color also changes to red to visually indicate the error.
** You may write a custom error message by configuring the '''''Validate Message''''' property.


==== <span style="color:#662d91; font-size:115%">Referencing Data Fields and Data Columns by Name</span> ====
You can change the order of columns within your '''Data Table''' by right clicking a '''Data Column''' in the Node Tree and choosing either "Move Up" or "Move Down".  Keyboard shortcuts are also available.  "Move Up" is <code>Ctrl + Up</code> and "Move Down" is <code>Ctrl + Down</code>.  
 
Peer fields ('''Data Fields''' at the same level of the '''Data Model''') and peer columns (sibling '''Data Columns''' of a '''Data Table''') can be referenced by name in '''''Is Valid Expressions'''''.
* Ex: A '''Data Field's''' expression could reference a peer '''Data Field''' named "Subtotal" simply by writing <code>Subtotal</code> into the expression.
 
{|cellpadding="10" cellspacing="5" style="margin:12px"
|-
|style="font-size:200%; background-color:#f89420; color:white"|&#9888;
|style="border: 4px solid #f89420"|
You must substitute underscores for spaces and special characters in '''Data Column''' and '''Data Field''' names.
* Ex:  A '''Data Field''' named "Social Security Number" would need to be typed out <code>Social_Security_Number</code>
 
Multiple spaces or special characters in a row must be substituted with a single underscore.
* Ex: A '''Data Column''' named "Expenses - Sch. A" would need to be typed out <code>Expenses_Sch_A</code>
 
If a '''Data Field''' or '''Data Column's''' name begins with a number, you must prepend an underscore to the name.
* Ex: A '''Data Field''' named "911" would need to be typed out <code>_911</code>
 
 
If you’re unsure how to format a field’s name, simply begin typing it and Grooper’s built-in IntelliSense menu will show you the correct way.
|}
|}


'''Data Fields''' in peer ''single instance'' '''Data Sections''' may be referenced, using dot notation.
=== Table Extract Methods ===
* Ex: You could access the "DOB" '''Data Field''' inside a "Employee" '''Data Section''' with the expression <code>Employee.DOB</code>
* Be aware, this expression will fail if the '''Data Section's''' '''''Miss Disposition''''' property is set to ''No Instance'' and no section instance is extracted.


There are six different extraction methods available in Grooper.  Using the '''Data Table's''' '''''Extract Method''''' property you will select and configure one of the following:


'''Data Fields''' in non-peer ''single instance'' '''Data Sections''' can also be referenced using dot notation.  However, you must reference the '''Data Model's''' parent '''Content Type''' ''first''.
* '''''[[Row Match]]''''' - This uses an extractor to match each row.  You could reference a '''Data Type''' extractor that returns each whole row in the table to populate the rows in the '''Data Table'''.
* Ex: Given a '''Data Model''' for a '''Document Type''' named "Student Records":
* '''''[[Header-Value]]''''' - This method detects the layout of a table by analyzing results from header extractors and value extractors defined on each '''Data Column'''.
**  A '''Data Field''' named "Average" in a '''Data Section''' named "Math" could access the "GPA" '''Data Field''' inside a "Totals" '''Data Section''' with the expression <code>Student_Records.Totals.GPA</code>
* '''''[[Grid Layout]]''''' - This method creates a grid from header positions, using extractors to match column and (sometimes optionally) row headers.  Once the grid is created ("inferred" from the column and row header positions) it extracts the corresponding text data from the cells within the grid.
* Be aware, the expression will fail if the referenced '''Data Section's''' '''''Miss Disposition''''' property is set to ''No Instance'' and no section instance is extracted.
* '''''[[Tabular Layout]]''''' - This method is an improvement upon the '''''Header-Value''''' method. It also detects a table's layout using a table's column headers and value extractors defined on the '''Data Column''' objects.  However, in general, there is much less configuration required up front with more ability to fine tune configuration according to your needs. This method also can make efficient use of [[Label Sets]] to aid in table extraction.
* '''''[[Fluid Layout]]''''' - This method requires [[Label Sets]] in order to function.  It can be configured in a way to use either the '''''Row Match''''' or the '''''Tabular Layout''''' method based on how a '''Document Type's''' labels are collected.
* '''''[[Delimited Extract]]''''' - This method allows for efficient extraction of character delimited text files, such as CSV files.
* '''''[[Fixed Width]]''''' - This method reads tabular data from "fixed width" formatted text files.


== Property Details ==


'''Data Fields''' in multi-instance '''Data Sections''' ''cannot'' be referenced whatsoever. 
This section expands on the Grooper documentation for various '''Data Table''' properties.
* If you try to reference fields in multi-instance sections, you will find Grooper's IntelliSense will not show the field's name and the expression will throw an error if you manually type in the name.


==== <span style="color:#662d91; font-size:115%">Example Is Valid Expression</span> ====
=== Maximum Display Rows ===
 
'''''!!SCREENSHOTS FORTHCOMING!!'''''
 
For this example, we’ll be using the <code>Regex.IsMatch</code> method. This method compares a string input to a regular expression, returning “True” if they match.
 
The field “OKDL” is meant to capture Oklahoma Driver’s License Numbers.
* OK driver license have a strict pattern, consisting of one capital letter followed by 9 numerical digits.
* We could use the following '''''Is Valid Expression''''' to verify the result matches the driver license pattern (or in other words, is valid data).
** <code>Regex.IsMatch(OKDL, "[A-Z][0-9]{9}")</code>
 
If the field’s value does not match the regular expression, Grooper will mark it as invalid and the field turns red:
 
Change the value to a valid ID number, and viola!  The field is no longer in error:
 
=== Is Required Expressions ===
 
You can set a field to be required in Grooper by setting the '''''Required''''' property to true.  If the field is not extracted during the '''Extract''' step (in other words, "blank"), the field will be flagged as invalid.  However, what if whether or not a field is required is based on the value of some other field?  You would want that field to be ''conditionally'' required, based on some set criteria or parameter.
 
That's what '''''Is Required Expressions''''' are for.  An '''''Is Required Expression''''' is a snippet of VB.Net code that sets a field’s “Required” status conditionally.
 
==== <span style="color:#662d91; font-size:115%">Return Type ====
 
'''''Is Required Expressions''''' must return a Boolean (True/False) value:
* If the expression returns “True,” the field becomes required.
* If the expression returns “False,” it remains optional.
 
==== <span style="color:#662d91; font-size:115%">Referencing Data Fields and Data Columns by Name</span> ====
 
Peer fields ('''Data Fields''' at the same level of the '''Data Model''') and peer columns (sibling '''Data Columns''' of a '''Data Table''') can be referenced by name in '''''Is Required Expressions'''''.
* Ex: A '''Data Field's''' expression could reference a peer '''Data Field''' named "Subtotal" simply by writing <code>Subtotal</code> into the expression.
 
{|cellpadding="10" cellspacing="5" style="margin:12px"
|-
|style="font-size:200%; background-color:#f89420; color:white"|&#9888;
|style="border: 4px solid #f89420"|
You must substitute underscores for spaces and special characters in '''Data Column''' and '''Data Field''' names. 
* Ex:  A '''Data Field''' named "Social Security Number" would need to be typed out <code>Social_Security_Number</code>
 
Multiple spaces or special characters in a row must be substituted with a single underscore.
* Ex:  A '''Data Column''' named "Expenses - Sch. A" would need to be typed out <code>Expenses_Sch_A</code>
 
If a '''Data Field''' or '''Data Column's''' name begins with a number, you must prepend an underscore to the name.
* Ex: A '''Data Field''' named "911" would need to be typed out <code>_911</code>
 
 
If you’re unsure how to format a field’s name, simply begin typing it and Grooper’s built-in IntelliSense menu will show you the correct way.
|}
 
'''Data Fields''' in peer ''single instance'' '''Data Sections''' may be referenced, using dot notation.
* Ex: You could access the "DOB" '''Data Field''' inside a "Employee" '''Data Section''' with the expression <code>Employee.DOB</code>
* Be aware, this expression will fail if the '''Data Section's''' '''''Miss Disposition''''' property is set to ''No Instance'' and no section instance is extracted.
 
 
'''Data Fields''' in non-peer ''single instance'' '''Data Sections''' can also be referenced using dot notation.  However, you must reference the '''Data Model's''' parent '''Content Type''' ''first''.
* Ex: Given a '''Data Model''' for a '''Document Type''' named "Student Records":
**  A '''Data Field''' named "Average" in a '''Data Section''' named "Math" could access the "GPA" '''Data Field''' inside a "Totals" '''Data Section''' with the expression <code>Student_Records.Totals.GPA</code>
* Be aware, the expression will fail if the referenced '''Data Section's''' '''''Miss Disposition''''' property is set to ''No Instance'' and no section instance is extracted.
 
 
'''Data Fields''' in multi-instance '''Data Sections''' ''cannot'' be referenced whatsoever. 
* If you try to reference fields in multi-instance sections, you will find Grooper's IntelliSense will not show the field's name and the expression will throw an error if you manually type in the name.
 
==== <span style="color:#662d91; font-size:115%">Example Is Required Expression</code> ====
 
'''''!!SCREENSHOTS FORTHCOMING!!'''''
 
In the following example, there are two fields.
* “Marital Status” is a Boolean field with a true value of “Married” and a false value of “Single.”
* “Spouse Name” is a string field.  It has the the following '''''Is Required Expression''''':
** <code>Marital_Status = True</code>
 
When the "Marital Status" field is set to "Single" (therefore "False"), the "Spouse Name" field is not required.
* The condition set by the '''''Is Required Expression''''' <code>Marital_Status = True</code> is met.  The "Marital Status" field's value equates to "False".  So, the expression returns "False"   
* The field remains an optional field.  No value is required to be entered.
 
If “Marital Status” is “Married” (therefore "True"), the “Spouse Name” field then becomes required. 
* The condition set by the '''''Is Required Expression''''' <code>Marital_Status = True</code> is met.  The "Marital Status" field's value equates to "True".  So, the expression returns "True"   
* The field will be in error and turn red until a value is entered.
 
== Batch Process Expressions ==
 
'''''Batch Process Expressions''''' are snippets of VB.Net code that alter the behavior of '''Batch Process Steps'''.  Normally, steps in a '''Batch Process''' follow a sequential order.  One step finishes, the next starts up, and so on until all steps are completed.  With '''''Batch Process Expressions''''' you can conditionally apply steps or route or re-route '''Batch''' content to defined steps based on certain processing criteria.
 
There are two types of Batch Process Expressions:
* '''''Should Submit Expressions'''''
** A '''Batch Process Step''' with a '''''Should Submit Expression''''' will only execute if the expression evaluates to “True” for the current '''Batch''' object, skipping documents or pages that do not satisfy the expression.
* '''''Next Step Expressions'''''
** '''''Next Step Expressions''''' determine which step of the '''Batch Process''' to advance to once the current step is completed.
 
=== Should Submit Expressions ===
 
A '''''Should Submit Expression''''' is a snippet of VB.net code that determines whether a '''Batch Process Step''' should execute. These expressions can inspect image properties, '''Batch''' object attributes, or the Grooper Node Tree. These properties can be parameters to apply conditional logic to the '''Batch Process'''.
* '''''Should Submit Expressions''''' are a great solution for applications where only some--but not ''all''--documents in a '''Batch''' need to be processed by a given '''Batch Process Step'''.
 
==== <span style="color:#662d91; font-size:115%">Return Type</span> ====
 
'''''Should Submit Expressions''''' are Boolean.
* If the expression evaluates to “True,” the '''Batch Process Step''' will execute.
* If it evaluates to “False,” the step will be skipped.
 
==== <span style="color:#662d91; font-size:115%">Example Should Submit Expression</code> ====
 
'''''!!EXAMPLE FORTHCOMING!!'''''
 
=== Next Step Expressions ===
 
A '''''Next Step Expression''''' is a snippet of VB.net code that determines which step in the '''Batch Process''' to advance the '''Batch''' to once the current step is completed.
* '''''Next Step Expressions''''' can be a conditional workflow tool, routing '''Batches''' to different steps based on certain processing parameters.
* Ex: If a step flagged any of the documents with an error, you might send that '''Batch''' to a '''Review''' step.  If no documents are flagged, you could bypass the '''Review''' step, instead sending it to the next automated step.
 
==== <span style="color:#662d91; font-size:115%">Return Type</span> ====
 
'''''Next Step Expressions''''' must evaluate to the name of a '''Batch Process Step''' or "Nothing".
* Commonly, <code>If</code> statements are used to return the name of one step if the expression is true, and another step if it is false.
* If an expression returns a value of <code>Nothing</code>, no further steps are processed and '''Batch''' will complete.
 
==== <span style="color:#662d91; font-size:115%">Using If Statements to Route Steps</span> ====
 
<code>If</code> statements take the following structure:
* <code>If(''Expression'', ''TrueValue'', ''FalseValue'')</code>
** <code>''Expression''</code> is the expression to be evaluated.
** <code>''TrueValue''</code> is the name of the '''Batch Process Step''' to advance to if the expression returns "True".
** <code>''FalseValue''</code> is the name of the '''Batch Process Step''' to advance to if the expression returns "False".
 
 
The names of all steps in the '''Batch Process''' will be accessible in the '''''Next Step Expression''''' IntelliSense menu.
 
{|cellpadding="10" cellspacing="5" style="margin:12px"
|-
|style="font-size:200%; background-color:#f89420; color:white"|&#9888;
|style="border: 4px solid #f89420"|
You must substitute underscores for spaces and special characters in '''Batch Process Step''' names. 
* Ex:  A step named "Data Review" would need to be typed out <code>Data_Review</code>
 
Multiple spaces or special characters in a row must be substituted with a single underscore.
* Ex:  A step named "Recognize - OCR" would need to be typed out <code>Recognize_OCR</code>
 
If a '''Batch Process Step's''' name begins with a number, you must prepend an underscore to the name.
* Ex: A step named "2nd OCR Pass" would need to be typed out <code>_2nd_OCR_Pass</code>
 
 
If you’re unsure how to format a step's name, simply begin typing it and Grooper’s built-in IntelliSense menu will show you the correct way.
|}
 
==== <span style="color:#662d91; font-size:115%">Example Next Step Expression</span> ====
 
'''''!!SCREENSHOTS FORTHCOMING!!'''''
 
Picture an '''Export''' step in a '''Batch Process'''.  One of two things is going to happen at the end of that step:
* Either all documents in the '''Batch''' will be extracted successfully, with no errors.
* Or, one or more fields in one or more of the documents will throw an error and need to be reviewed.
 
For '''Batches''' without any errors, you may just want to export all their documents in an '''Export''' step.  For '''Batches''' that had extraction errors you may want a user to review them in a '''Review''' step.  This is exactly what '''''Next Step Expressions''''' are for.  The following expression:
* <code>If(Batch.HasDataErrors, Review, Export)</code>
 
 
<code>Batch.HasDataErrors</code> is a Boolean attribute of a '''Batch'''.  It returns "True" if the '''Batch''' contains invalid index data. So in this case, if the '''Batch''' contains invalid index data, it moves the batch to the '''Review''' step. If the batch does not contain data errors, it moves ahead to the '''Export''' step.
 
== IP Profile Expressions ==
 
'''''IP Profile Expressions''''' allow for conditional handling of image processing, allowing single '''IP Profiles''' to accommodate multiple contingencies without redundant processing or the need to create multiple '''IP Profiles'''.  These expressions are configured for the '''IP Steps''' (or '''IP Groups''') in an '''IP Profile'''
 
 
'''IP Profiles''' can employ two types of expressions (similar to '''Batch Processes''' and their expressions):
* '''''Should Execute Expressions'''''
** These expressions determine ''if'' an '''IP Step''' should be executed on an image at all.
* '''''Next Step Expressions'''''.
** These expressions determine which '''IP Step''' in the '''IP Profile''' to advance the image to after the current step is completed.
 
=== Should Execute Expressions ===
 
An '''IP Step's''' (or '''IP Group's''') '''''Should Execute Expression''''' determines whether or not the step should be executed on the input image (or whether the group of steps should be executed in the case of '''IP Groups''').
 
==== <span style="color:#662d91; font-size:115%">Return Type</code> ====
 
'''''Should Execute Expressions''''' must return a Boolean (True/False) value.
* If the expression evaluates to "True", the step processes the image.
* If it evaluates to "False", no processing occurs and the image is advanced to the following step.
 
==== <span style="color:#662d91; font-size:115%">Referencing Prior IP Steps By Name</code> ====
 
Names of all '''IP Steps''' that occur before the step in question will be accessible in the '''''Should Execute Expression''''' IntelliSense menu.
* Commonly, you will conditionally execute a step based on the results of a previous step.  You'd use the <code>Results</code> method to do this, using <code>Results.''IPStepName''</code> as part of the expression.
* FYI: Technically speaking IntelliSense will show you steps after the step whose '''''Should Execute Expression''''' you are configuring.  However, it won't make any sense to use them in the expression if they have not been applied in the '''IP Profile''' up to that point.
 
{|cellpadding="10" cellspacing="5" style="margin:12px"
|-
|style="font-size:200%; background-color:#f89420; color:white"|&#9888;
|style="border: 4px solid #f89420"|
You must substitute underscores for spaces and special characters in '''IP Step''' names. 
* Ex:  A step named "Line Removal" would need to be typed out <code>Line_Removal</code>
 
Multiple spaces or special characters in a row must be substituted with a single underscore.
* Ex:  A step named "Line Removal - Vert. Only" would need to be typed out <code>Line_Removal_Vert_Only</code>
 
If an '''IP Step's''' name begins with a number, you must prepend an underscore to the name.
* Ex: A step named "2nd Pass Line Removal" would need to be typed out <code>_2nd_Pass_Line_Removal</code>
 
 
If you’re unsure how to format a step's name, simply begin typing it and Grooper’s built-in IntelliSense menu will show you the correct way.
|}
 
==== <span style="color:#662d91; font-size:115%">Example Should Execute Expression</code> ====
 
'''''!!SCREENSHOTS FORTHCOMING!!'''''
 
For this example, I’ve set up a '''Binarize''' step in my '''IP Profile''', which will convert color images to black and white. I will use a  '''''Should Execute Expression''''' to ''only'' execute the '''Binarize''' step on non-binary images (color, greyscale, anything not a true bitonal black and white image).
 
We can use the <code>Image.IsBinary</code> method to inspect the image’s color scheme and negating it with <code>NOT</code> to prevent the step from firing on already-binarized images.
 
When the '''IP Step''' encounters a color image, the expression evaluates to "True" and the step executes.
 
When the '''IP Step''' encounters a black and white image, the expression evaluates to "False" and the step does not execute.
 
=== Next Step Expressions ===
 
An '''IP Step’s''' '''''Next Step Expression''''' determines which '''IP Step''' (or '''IP Group''') to advance to upon completion of the current step. These expressions commonly consist of an <code>If</code> statement, which consists of an expression, one '''IP Step''' to advance to when that expression returns true, and another '''IP Step''' if the expression returns false.
 
==== <span style="color: #662d91; font-size:115%">Return Type</span> ====
 
'''''Next Step Expressions''''' must evaluate to the name of an '''IP Step''', '''IP Group''' or "Nothing".
* Commonly, <code>If</code> statements are used to return the name of one step if an expression is true, and another if it is false.
* If the expression returns a value of <code>Nothing</code>, the '''IP Profile''' will cease execution entirely.
 
==== <span style="color: #662d91; font-size:115%">Using If Statements to Route IP Steps</span> ====
 
<code>If</code> statements take the following structure:
* <code>If(''Expression'', ''TrueValue'', ''FalseValue'')</code>
** <code>''Expression''</code> is the expression to be evaluated.
** <code>''TrueValue''</code> is the name of the '''IP Step''' to advance to if the expression returns "True".
** <code>''FalseValue''</code> is the name of the '''IP Step''' to advance to if the expression returns "False".
 
==== <span style="color: #662d91; font-size:115%">Referencing IP Steps By Name</code> ====
 
The names of '''IP Steps''' and '''IP Groups''' in the '''IP Profile''' will be accessible in the '''''Next Step Expression''''' IntelliSense menu.
* You can reference other '''IP Steps''' via the <code>Steps</code> namespace.
** Ex: <code>Steps.Speck_Removal</code>
 
{|cellpadding="10" cellspacing="5" style="margin:12px"
|-
|style="font-size:200%; background-color:#f89420; color:white"|&#9888;
|style="border: 4px solid #f89420"|
You must substitute underscores for spaces and special characters in '''IP Step''' names. 
* Ex:  A step named "Line Removal" would need to be typed out <code>Line_Removal</code>
 
Multiple spaces or special characters in a row must be substituted with a single underscore.
* Ex:  A step named "Line Removal - Vert. Only" would need to be typed out <code>Line_Removal_Vert_Only</code>
 
If an '''IP Step's''' name begins with a number, you must prepend an underscore to the name.
* Ex: A step named "2nd Pass Line Removal" would need to be typed out <code>_2nd_Pass_Line_Removal</code>
 
 
If you’re unsure how to format a step's name, simply begin typing it and Grooper’s built-in IntelliSense menu will show you the correct way.
|}
 
==== <span style="color: #662d91; font-size:115%">Example Next Step Expression</span> ====
 
'''''!!SCREENSHOTS FORTHCOMING!!'''''
 
Here, we have an '''IP Profile''' with three steps:
* '''Auto Border Crop'''
* '''Binarize'''
* '''Speck Removal'''
 
I've entered the following '''''Next Step Expression''''' for the first step ('''Auto Border Crop'''):
* <code>If(Image.IsBinary, Steps.Speck_Removal, Steps.Binarize)</code>
 
Once the '''Auto Border Crop''' step finishes, it will do the following according to the '''''Next Step Expression'''''.
# Evaluate the expression.
#* Here, it will inspect the image properties to see if the image is binary (black and white) using the <code>Image.IsBinary</code> method.
# If the expression evaluates true, move the image to the '''Speck Removal''' step.
#* So, if the image is already binary (in other words, if <code>Image.IsBinary = True</code>), then the image will be advanced to the '''Speck Removal''' step (skipping the '''Binarize''' step entirely).
# If the expression evaluates to false, move the image to the '''Binarize''' step.
#* If the image is not binary (color, greyscale, etc), it will be advanced to the ''Binarize''' step first.
 
== Mapping Expressions ==
 
'''''Mapping Expressions''''' are snippets of VB.Net code that calculate, format or generate field values as they are imported into Grooper or exported to a database.
 
=== <span style="color: #662d91; font-size:115%">Import Mapping Expressions</span> ===
 
'''''Import Mapping Expressions''''' are snippets of VB.Net code that live on '''''Import Behavior Definitions'''''.  These expressions can calculate, format, or generate field values as the document is imported into Grooper, using document metadata from the import source.
* Ex: You can use '''''Import Mapping Expressions''''' to parse the username from an email's "Sender" property directly to a field before extraction ever takes place.
 
==== <span style="color: #662d91; font-size:115%">Return Type</span> ====
 
'''''Import Mapping Expressions''''' must return a value of the same data type as the '''Data Field''' or '''Data Column''' they are populating.
 
==== <span style="color: #662d91; font-size:115%">Example Import Mapping Expression</span> ====
 
'''''!!EXAMPLE FORTHCOMING!!'''''
 
=== <span style="color: #662d91; font-size:115%">Export Mapping Expressions</span> ===
 
'''''Export Mapping Expressions''''' are snippets of VB.Net code that live on '''''Export Behavior Definitions''''' and can calculate, format, or generate field values at the time of the document’s export. 
 
Previously in Grooper, placeholder fields would need to be created in a '''Data Model''' and '''''Data Model Expressions''''' would calculate and/or generate values at the time of data extraction.  This created unnecessary overhead, making the '''Data Model''' more difficult to manage, and could increase the time the '''Extract''' step took to process. 
 
'''''Export Mapping Expressions''''' give users a simple way to manipulate Grooper extracted data upon export, before it lands in a database table or content management system (CMS) field or metadata property.  No need to bloat your '''Data Model''' with unnecessary placeholder fields!
 
{|cellpadding="10" cellspacing="5" style="margin:12px"
|-
|style="font-size:200%; background-color:#f89420; color:white"|&#9888;
|style="border: 4px solid #f89420"|
'''''Export Mapping Expressions''''' must evaluate to the same data type as the table column (in the case of database exports) or field or metadata property (in the case of exports to content management systems) they are being exported to.
|}


==== <span style="color: #662d91; font-size:115%">Return Type</span> ====
The '''''Maximum Display Rows''''' property specifically has to do with how rows are displayed in the '''Data Viewer''' when a user executes the '''Review''' activity.


'''''Export Mapping Expressions''''' must return a value compatible with the data type of the export destination.
Imagine you have a data dense document with a table with several hundred rows. Grooper extracts the table and now it's time to present it to a data entry clerk in '''Review'''.  It's going to take Grooper a while to load that '''Data Table'''.  This can be an unnecessary lag point in the user's review experience.
* In the case of database exports (using a '''''Data Export'''''), the expression's data type must match the table column's data type.
** Ex: If a SQL column has a data type of DATE, the expression must return a date value in an acceptable format.
* In the case of content management system exports (using a '''''CMIS Export'''''), the expression's data type must match the field or metadata property's data type.
** Ex: If a SharePoint Document Library column has a column type of "Number", the expression must return a decimal or integer value.


==== <span style="color: #662d91; font-size:115%">Example Export Mapping Expression</span> ====
The '''''Maximum Display Rows''''' property allows you to dynamically load rows.  Instead of loading them all at once, you can restrict this to only load, say 50 at a time.  After you scroll to the bottom of the first 50 loaded rows, the next 50 will load, then the next, and so on until you reach the end of the table.  This way, the user doesn't have to wait for the entire table to load up front and can start reviewing the extracted data quicker.


'''''!!EXAMPLE FORTHCOMING!!'''''
[[Category:Articles]]

Latest revision as of 13:26, 28 December 2023

Data Tables are Data Elements in a Data Model used to extract data from tables in documents.  Each column of data is represented by a Data Column object created as the Data Table's children.

Data is extracted using one of the following Table Extraction methods. Each of these methods take a different approach in modeling a table's structure:


About

Many documents contain data in a table, presented on the page as some kind of grid of information divided into rows and columns.  The Data Table object's purpose is to define the processing logic to model and collect tabular data.

A Data Table can be added to your Data Model to extract data from the table's cells.  Once added to your Data Model, you will add Data Columns to the Data Table.  You can add as many columns as you need to collect data from all (or only some) of the table's columns on the document.  Data Columns will exist as children of the Data Table in the node hierarchy.

  • Data Columns also allow for additional configuration, such as assigning the Value Type for the extracted data in that column (decimal, string, Boolean, etc).
  • There are several table extraction methods. Some will require configuration of a Data Table's Data Columns. Others will not (or allow for optional configurations). Please visit the Table Extraction article for more information on tabular data extraction in general.

FYI

Generally, the first Data Column underneath your Data Table will correspond to the leftmost column and the last will be the rightmost. The top Data Column lines up with the first column, the second Data Column the second column, and so on.

However, this is not strictly necessary. You can re-order the column order how you see fit.

You can change the order of columns within your Data Table by right clicking a Data Column in the Node Tree and choosing either "Move Up" or "Move Down".  Keyboard shortcuts are also available.  "Move Up" is Ctrl + Up and "Move Down" is Ctrl + Down.

Table Extract Methods

There are six different extraction methods available in Grooper. Using the Data Table's Extract Method property you will select and configure one of the following:

  • Row Match - This uses an extractor to match each row.  You could reference a Data Type extractor that returns each whole row in the table to populate the rows in the Data Table.
  • Header-Value - This method detects the layout of a table by analyzing results from header extractors and value extractors defined on each Data Column.
  • Grid Layout - This method creates a grid from header positions, using extractors to match column and (sometimes optionally) row headers.  Once the grid is created ("inferred" from the column and row header positions) it extracts the corresponding text data from the cells within the grid.
  • Tabular Layout - This method is an improvement upon the Header-Value method. It also detects a table's layout using a table's column headers and value extractors defined on the Data Column objects. However, in general, there is much less configuration required up front with more ability to fine tune configuration according to your needs. This method also can make efficient use of Label Sets to aid in table extraction.
  • Fluid Layout - This method requires Label Sets in order to function. It can be configured in a way to use either the Row Match or the Tabular Layout method based on how a Document Type's labels are collected.
  • Delimited Extract - This method allows for efficient extraction of character delimited text files, such as CSV files.
  • Fixed Width - This method reads tabular data from "fixed width" formatted text files.

Property Details

This section expands on the Grooper documentation for various Data Table properties.

Maximum Display Rows

The Maximum Display Rows property specifically has to do with how rows are displayed in the Data Viewer when a user executes the Review activity.

Imagine you have a data dense document with a table with several hundred rows. Grooper extracts the table and now it's time to present it to a data entry clerk in Review. It's going to take Grooper a while to load that Data Table. This can be an unnecessary lag point in the user's review experience.

The Maximum Display Rows property allows you to dynamically load rows. Instead of loading them all at once, you can restrict this to only load, say 50 at a time. After you scroll to the bottom of the first 50 loaded rows, the next 50 will load, then the next, and so on until you reach the end of the table. This way, the user doesn't have to wait for the entire table to load up front and can start reviewing the extracted data quicker.

Retrieved from "https://wiki.grooper.com/index.php?title=2023:Data_Table&oldid=17147"