2021:Data Rule (Node Type)

WIP	This article is a work-in-progress. It was written using a beta version of 2021. This article is subject to change and/or expansion as it is updated to the release version of 2021. This tag will be removed upon draft completion.

The Data Rule object allows for complex validation and manipulation of a Data Model's Data Elements (Data Fields, Data Sections, and Data Tables) in Grooper.

This allows users to create a conditional hierarchy of actions to take if certain conditions met. These conditions are configured using .NET, LINQ and/or lambda expressions. When the expression is "triggered", either evaluating to "true" or "false", certain actions can be made. These include:

• Calculate Value - This action sets the value of a Data Field or cells a Data Column, using calculate expressions to perform mathematical or concatenation operations of Data Elements.
• Clear Item - This action clears the value of a Data Element.
• Append - This action can add the value of a Data Element to a another, allowing you to add entries to a table, multi-instance section or multi-cardinality field.
• Copy - This action copies or moves the value of a Data Element.
• Parse Value - This action uses a regular expression pattern to return part of a Data Field's value or cell in a Data Column's value.
• Raise Issue - This action adds an issue to the issue log, used for validating a Data Element. This action can also be used to flag the Data Element.

These trigger conditions and subsequent actions set on the Data Rules objects are executed through the Apply Rules activity after data is extracted from an Extract activity. Optionally, Data Model, Data Section, and/or Data Table elements may be configured to execute a Data Rule during a user attended review activity (i.e. Data Review), by setting the object's Validate Rule.

About

You may download and import the file below into your own Grooper environment (version 2021). This contains a Batch with the example document(s) discussed in this article and a Content Model configured according to its instructions. Media:Data Rule (v2021).zip

Some Basics About Expressions

Grooper makes use of expressions to calculate and validate extracted data (or otherwise populate/set data if not otherwise extracted) and use extracted data to populate fields in a Data Model. Traditionally, this is configured on a Data Field object in a Data Model (or in the case of validating or calculating cells in a table, the Data Column object), using the Default Value, Calculated Value, or Is Valid properties.

For example, let's say we have several documents in a Batch. Each one contains W-2 wage reporting forms for various individuals and we want to do some basic tax filing calculation. In order to find someone's total income, it may not be quite as simple as pulling the listed wages from a single W-2. An individual might have multiple W-2s from multiple employers.

If an individual worked for three different employers over the course of a year, their total income would be the wages from all three W-2s added together. This is where expressions come in handy in Grooper. There is no extractible "Total Wages" field on the document. It's just three pages, each page a different W-2. There is no text data for an extractor to return that corresponds to all three W-2's wages field added together.

But we could create a Data Section in our Data Model to return the wages from each individual W-2 form, by adding a "Wages" Data Field and configuring its extraction. Then, we could create a "Total Wages" Data Field and use a Calculated Value expression to add up the results of each "Wages" Data Field in each section (each W-2 in this case).

Here, we have a simple Content Model set up to solve the problem described above. It has a Data Section named "W2 Info" It has two child Data Fields: "Fed Wages" and "Employee SSN". The Data Section is configured to create one section for each W2 or each page in this case. This document has three W2s (or three pages). Hence, we have three sections. The "W2 Info" Data Section's child "Fed Wages" Data Field is configured to return the "Wages, tips, other compensation" field for the W-2s. This "Total Income" Data Field will be configured to add up the results of the "Fed Wages" Data Field from each section.
Here, we've selected the "Total Income" Data Field. Calculated Value property is configured with the expression W2_Info.SumOf("Fed Wages") This will add the results of each "Fed Wages" Data Field in the "W2 Info" Data Section together. The Calculate Mode is set to Always Set. This will always run the expression regardless of whether the Data Field is populated by an extractor (It's not in this case. The Value Extractor property is blank.). FYI If you were to press the "Test Extraction" button at this point, the Calculated Value expression would appear to fail. However, it's actually configured correctly to do what we want. This has to do with the scope we're testing extraction in our Data Model. With the "Total Wages" Data Field selected in the node tree, we've narrowed down the scope of our Data Model to just test within this Data Field's scope. But we're actually looking for information outside the "Total Wages" Data Field's scope. We're using information in the "W2 Info" Data Section's scope to add up the total wages for each W2 in its child "Fed Wages" Data Fields. Once we select the parent Data Model in the node tree, we will be at a higher scope in the data hierarchy. At that point, the expression will run against the full scope and successfully execute.
Upon extraction... The expression executes, adding up the "Fed Wages" Data Field values for each section.

Conditional Expressions and Data Rules

You can do a lot with expressions, even applying some conditional logic to their execution. If the condition is met, the expression executes. If not, it doesn't or some other expression executes.

In our example of documents containing W-2 forms we make some assumptions about the document. We assume each document contains a W-2 for a single individual. Each individual should only have one social security number. It would be problematic if their were multiple social security numbers extracted from the W-2 forms. This could indicate there are multiple W-2s for multiple individuals in a single document.

To account for this, you could use a more complex Calculated Value expression to only add up the "Fed Wages" Data Fields if the social security number was the same for each document. If the condition of their only being one social security number for each W-2 is met, the expression to add up the wages would execute. If not, it wouldn't.

This is basic conditional logic. If "x" condition is met, then do "y". If there's only one social security number, then add up all the wages. Otherwise, do nothing (or something else). You could go another step further and add an Is Valid expression to flag the document if the social security numbers didn't match, as well.

However, the more complex a Data Model's data hierarchy (the more Data Sections and Data Tables it has), generally the more complex these conditions tend to be. The more conditions you add for an expression to execute, the more complex the expression becomes. This can result in very cumbersome expressions that are difficult to form and manage.

This is where the Data Rule object really shines. Data Rules allow you to use Trigger expressions to determine one or multiple subsequent Actions to take if that expressions evaluates to true (or false). This is also basic conditional logic. If the trigger expression is true, do the action. Otherwise, do nothing (or a different action). Furthermore, you can more easily create a complex hierarchy of conditions by adding child Data Rules to parent Data Rules. If the trigger expression evaluates to true, the child Data Rules will execute, with their own triggers and even own child Data Rules. This allows for simpler set up, execution, and management of more complex conditional expressions as well as some actions that fall outside normal expressions you can set up in a Data Field or Data Column.

A Basic Example Data Rule

Here, we've added a Data Rule object to our Content Model's Local Resources folder. As it is configured now, this Data Rule (when executed) will add up "W2 Info" Data Section's "Fed Wages" Data Field values for each document, populating the "Total Income" Data Field. As is currently, it does essentially the exact same thing as the Calculated Value property configuration of the "Total Income" Data Field discussed above. The first thing you'll always configure for a Data Rule is the Scope property. This determines at what level in a Content Model's Data Model hierarchy the Data Rule executes. This will determine which Data Elements are accessible. In this case, we set the Scope to the full Data Model. This gives the most access to the most Data Elements in the Data Model. If we instead selected the "W2 Info" Data Section as the Scope instead, we would only have access to the Data Elements in that Data Section. This would not include the "Total Income" Data Field, which is what we're trying to populate here. The Trigger property controls the condition determining whether or not the action executed by the Data Rule is taken or not. The Trigger can be a .NET, LINQ or lambda expression. If left blank, as is the case here, the Trigger defaults to "true". In effect, the Data Rule will always execute its action. We will revisit this Trigger soon. The True Action property determines what happens if the Trigger evaluates to "true". Or put another way, if the Data Rule's condition is met. There are many possible actions detailed in the Actions section of this article. Each action type has its own set of configuration properties. In this case, we have selected Calculate Value. This action type's configuration is very similar to how a Data Field's Calculated Value property is configured. The Target Field property controls what Data Field (or Data Columns in the case of cells in a Data Table) is affected by the action. The Value Expression property controls what populates the Target Field's Data Field. It is what is calculated (or maybe better put how it's calculated). In this case we're using the exact same expression to add up the "Fed Wages" Data Fields in each of the "W2 Info" Data Section's sections as discussed above.

Data Rules are executed by the Apply Rules activity. After data is extracted by the Extract activity, any Data Rule referenced by the Apply Rules activity will alter the document's index data according to the Data Rule's configuration. You can test the Data Rule's results in Grooper Design Studio when the Data Rule is selected in the node tree. This will help you verify its configuration, giving you a preview of what would happen if that Data Rule was executed by the Apply Rules activity. Press the "Test Rule" button to execute the selected Data Rule. You can see in our case, the Calculate Value action is executed, using the Value Expression W2_Info.SumOf("Fed Wages") to populate the Target Field, "Total Income". Effectively, this adds up all the wage values for each W-2 in the document and returns the result to the "Total Income" Data Field. ⚠ Data Rules differ from expressions configured on Data Field (or Data Column) objects in one major way. Index data for documents must be extracted before executing a Data Rule. Documents must be processed by an Extract activity before testing a Data Rule or running the Apply Rules activity. The Data Model's results must be present and stored on the document before the Data Rule can manipulate them. Expressions configured on the Data Field (or Data Column) objects themselves execute during extraction when the Extract activity runs.

The Trigger

The Trigger property serves the purpose of establishing the condition that must be met in order for the Data Rule's action to be taken. These triggering conditions are also set using expressions. These expressions must return a Boolean "true" or "false" value. If the Trigger expression evaluates to "true", the True Action configuration is executed. If the Trigger expression returns "false", the False Action configured is executed (If it is configured. If left blank, no further action will be taken.)

In our case, something is wrong with our documents if the W2 forms have more than one social security number. Individuals should only have one social security number. If we added up all the wages for multiple W-2s with mismatched social security numbers, we would not be adding up the total income for an individual correctly. We'd end up with inaccurate data. This is the case for this document. It is comprised of two W-2 froms. For the first one, in the first section, the social security number extracted is "987‑56‑4321" For the second one, in the second section, the social security number extracted is "987‑65‑4321"

Luckily, there's an expression we could use to determine if our "W2 Info" Data Section has multiple social security numbers in its sections. This is a good opportunity for a LINQ expression. LINQ (or Language INtegrated Query) expressions are particularly helpful when navigating a Data Model's hierarchical structure to pull information from Data Sections and Data Tables.

Writer's Note: You aren't limited to just LINQ expressions for the Trigger. You can use any expression that returns a Boolean value, including standard.NET expressions, LINQ expressions, and lambda expressions. A LINQ expression just works well for this particular example.

(From sec In W2_Info Select sec.Employee_SSN).Distinct().Count() = 1

Let's break down this expression to understand what's going on.

(From sec In W2_Info Select sec.Employee_SSN).Distinct().Count() = 1

This is the "LINQ-iest" part of the expression. It's querying the extracted instances of our Data Model's objects, to return multiple results.

LINQ expressions always start with From (This indicates the data source from where are you querying the data). Next we declare a type variable we've named sec. We'll use later in the expression to return multiple instances of a Data Field in a Data Section (What you name it doesn't matter, just that you use the same name when you reference the variable later on in the query). The In clause determines the query's scope. We're looking for the "Employee SSN" Data Field in the "W2 Info" Data Section. The "W2 Info" Data Section is our scope. In W2_Info will only query instances (the results of its children Data Fields) in the sections produced by the "W2 Info" Data Section. The Select clause determines what values the query returns (or "selects"). We want information about the "Employee SSN" Data Fields. So, we've entered sec.Employee_SSN. Note, we've referenced the variable we declared at the start of the query, sec to do this.

Now the expression has some information it can work with. In this case, the social security numbers for each W-2 in the document (as returned by the "Employee SSN" Data Field for each section produced by the "W2 Info" Data Section).

(From sec In W2_Info Select sec.Employee_SSN).Distinct().Count() = 1

This part of the expression is counting the number of distinct values returned by the query (Technically, the .Distinct() expression is returning a subset of distinct values in the query's results. Then, the .Count() expression is counting the values in that subset). If the social security number is the same for each W2, there's only one distinct value. This should evaluate to "1". If not, it will be a larger number.

(From sec In W2_Info Select sec.Employee_SSN).Distinct().Count() = 1

This is just an equivalency argument to give us a Boolean "true" or "false" value. If the left side of the argument (the expression (From sec In W2_Info Select sec.Employee_SSN).Distinct().Count()) counts a single unique social security number in each section is equivalent to the right side of the argument (i.e. "1 = 1") it will return "true", otherwise "false".

If we use this expression as the Data Rule's Trigger, it will conditionally execute the True Action configured above only if it evaluates as true. Effectively, it will only add up all the wages for each W-2 only when the social security numbers for each W-2 are the same. Using the Trigger property, we've entered the LINQ expression described above. (From sec In W2_Info Select sec.Employee_SSN).Distinct().Count() = 1 If we press the "Test Rule" button to execute the Data Rule, it will only apply the rule's 'True Action configuration if the Trigger Expression is true. As you can see here, this document extracted multiple different social security numbers. So, nothing happens. The "Total Income" Data Field remains blank.

However, in the case of this document, its three W-2 forms do have the same social security number. So the "Employee SSN" field is the same for all three section instances of the "W2 Info" Data Section. Therefore, the Trigger expression returns "true". The True Action then executes, which is set to Calculate Value populating the Target Field "Total Income" with the results of the Value Expression W2_Info.SumOf("Fed Wages") In other words, the three "Fed Wages" Data Field results are added together and returned to the "Total Income" Data Field. Conditional logic!

Actions

Once a Data Rule is triggered, what happens next is determined by the True Action and False Action properties. When the Trigger expression evaluates to true, the True Action is executed. When the Trigger expression returns false, the False Action is executed. This determines what action is taken once the trigger condition is met or not met.

This can be one of six choices:

• Calculate Value
• Raise Issue
• Clear Item
• Copy Item
• Parse Value
• Action List

Each action has its own configuration to execute the action, detailed below. If you would like to follow along with this tutorial in Grooper, you can download the zip below and import it into your Grooper environment. It contains the Content Model and Batch used in these examples.

You may download and import the file below into your own Grooper environment (version 2021). This contains a Batch with the example document(s) discussed in this article and a Content Model configured according to its instructions. Media:Data Rule (v2021).zip

Calculate ValueRaise IssueClear ItemCopy ItemParse ValueAction List

Calculate Value

The Calculate Value action will use a .NET, LINQ or lambda expression to populate a field with the expression's result. The possibilities here are as endless as the capabilities of these expressions. We can perform mathematical operations on numerical data. We can concatenate multiple string fields. We can perform incremental additions to date values. The Calculate Value action allows you to use any configurable expression to manipulate extracted data into a desired result. We've already seen one example of the Calculate Value action in the section of this article above. But let's look at another one.

In this example, we have a fairly simple report detailing costs of intangible services related to an oil drilling operation. As is necessary prior to the execution of a Data Rule, these documents have already been classified and their data has been extracted. In other words, they have been processed by the Classify activity (according to how the "Data Rules - Intangibles Table" Content Model is configured) and then by the Extract activity (using the data hierarchy and extraction configuration set on its associated Data Model). All of the table information has already been extracted, including the "Total" column which adds up the "Dry Hole" and "Completion" costs for each row. However, we might want to know the total cost associated with this table. We will use the Calculate Value action to populate this "Grand Total" Data Field.
We have added a Data Rule to the Local Resources folder of this Content Model. The Scope is set to Content Model's Data Model The Scope property determines what Data Elements in the Data Model's hierarchy the Data Rule has access to. In this case, we need a fairly broad scope. We need access to the Data Table object "Intangibles Table" and the Data Field object "Grand Total". We would not in this case want to scope down to the Data Table, for example. Then, we would only have access to its child Data Elements, namely its Data Column objects. The "Grand Total" Data Field lies outside the "Intangible Table" Data Table's scope, but both are within their parent Data Model's scope. Hence, we choose the parent Data Model. We're leaving the Trigger property blank for the time being. This means the Trigger will default to "true" and the True Action will always execute. We've chosen Calculate Value for the True Action. With Calculate Value selected, we must indicate which Data Field is populated. The whole point of this action is to manipulate data to come up with some new value. That value has to go somewhere. This property points the calculated value to the desired location in the Data Model. Here, we've selected the "Grand Total" Data Field. The Value Expression property determines how the value is calculated. The result of the expression entered here is what is ultimately returned. In our case, we're adding up all the values in the "Total" column of the table. The expression Intangibles_Table.SumOf("Total") does just that.
Press the "Test Rule" button to test the Data Rule's execution. The Calculate Value action's Value Expression configuration executes. In this case, the expression adds up all the values in the "Total" column. And, its resulting value populates the assigned Target Field.

A Word of Caution: Overwriting Results

There is one important thing to note about the Calculate Value action. The Value Expression's calculated value will overwrite any existing data in a field. For example, take this document. This document has a "Grand Total" value listed on the document. That means an extractor can find it and return it to a Data Field. Such is the case here. This value is the extracted value found in the document's text data. However, this isn't actually an accurate total. The document is wrong. The grand total of all the values in the "Total" column should add up to "$1,048,050.00" and not what we see here, "$1,111,000.00"
Press the "Test Rule" button to test the Data Rule's execution. See that the extracted value is replaced by the Value Expression's result.
Now, this may be what you want to do, but it may not be what you want to do. What if you don't want to overwrite the "Grand Total" Data Field it it's already populated? What if you only want to use the Calculate Value action to populate the field if it's blank? That's a great opportunity for a Trigger expression! If this were the case, we would only want to execute this Data Rule if the "Grand Total" Data Field is not there. We could use that as the condition to execute the True Action. All we need to do is figure out an expression that would evaluate to true or false if that's the case. The expression Grand_Total = 0 would return true if the field isn't populated and false if it was. We configure the conditional execution of the Data Rule using the Trigger property. Here, we've used the expression Grand_Total = 0 When the Data Rule executes, it will only apply the True Action if that Trigger expression is true. In this case, it is false. There's already a number in the "Grand Total" Data Field that is not "0". With no False Action configured, nothing happens. The True Action is not applied. The data remains intact.

Back to the top

Raise Issue

The Raise Issue action is useful for data validation. You may want to ensure two fields add up to a third field. You may want to ensure a date on the document is a date in the past or within a day range in the future. You may want to check if two fields are equal to each other. This is the realm of data validation. The Raise Issue action can log information in an issue log if conditions like these are not met.

The Raise Issue action will work in concert with the Trigger expression to log issues. If the Trigger expression returns true, and the Raise Issue action is selected as the True Action it will log a defined message in an issue log. You optionally have the capability to add a message category for issue message as well.

In this example, we have a fairly simple report detailing costs of intangible services related to an oil drilling operation. We expect the "Total" column to be the cells in the "Dry Hole" and "Completion" columns added together for each row. We will use the Raise Issue action to verify this. This document has some problems. The extracted numbers for the highlighted rows do not add up. The "Dry Hole" values and the "Completion" values do not add up to the "Total" column's values. We will configure this Data Rule with the Raise Issue action to validate this documents data, logging any issues where the Trigger expression we configure returns as true. In this case we will check if the "Dry Hole" column value added with the "Completion" column value does not equal the "Total" column's value for each row.
You always must assign the Scope for the Data Rule. Here, we've elected to scope down to the level of the "Intangibles Table" Data Table object in the Data Model. Because we're accessing individual cell results in the Data Table's children Data Columns, this scope provides us the necessary increased to check if the individual results on each row add up correctly. The Trigger expression will determine if an issue is raised or not. Using Raise Issue as a True Action, we need this expression to return true if there is a problem. In this case, the following expression will return true if the "Dry Hole" Data Column and "Completion" Data Column values to not add up to the "Total" Data Column's value in a row in the table: Dry_Hole + Completion <> Total FYI: <> is the inequality operator for .NET expressions. The True Action is set to Raise Issue logging issues when the Trigger evaluates true. As far as configuration goes, the Raise Issue action's Message property must be configured. Some message has to go in the issue log to identify it. This is also an expression based message, returning a string value. For simple messages, just type what you want in quotation marks (We'll talk about not-so-simple messages soon). Here, we've entered "Wrong Total", which will simply output "Wrong Total" as the log message.
When configuring a Data Rule, the "Diagnostics" tab will give you some more information on what's going on. Press the "Test Rule" button to test the Data Rule's execution. Navigate to the "Diagnostics" tab. Select the "Execution Log" to get some information on the Data Rule's execution. Since we are at a Data Table object's scope, the Data Rule's Trigger expression is checked row by row. For rows like this one, the "Dry Hole" column's value added with the "Completion" column's value adds up to the "Total" column's value correctly. The Trigger expression therefore returns false, and nothing happens. For rows like this one, the "Dry Hole" column's value added with the "Completion" column's value does not add up to the "Total" column's value correctly. The Trigger expression therefore returns true, and the Raise Issue action is applied, returning the message we configured "Wrong Total".
The "Issue Log" diagnostic will show you the issue log the Raise Issue action generates. Each issue the Data Rule locates will be added to the log. The name of the Data Rule, the offending Data Element on which the issue is raised, and the Log Message will be recorded. Optionally, if configured, the Category will be listed here as well. You may notice the message "Wrong Total" is a little generic. It doesn't give us much information about why the issue was raised. This is why the Log Message property is expression based. It allows you to access some additional information to populate the error message.
We could use an expression like this for our Log Message instead. This strings together some information from the columns in the row in which the issue is raised. The resulting message, has some more information than just a generic error message. Namely, it shows us the row's code value and why the issue was raised (ie x + y does not equal z) This could help us more easily identify which row has the invalid data. Note: The Log Message must evaluate to a string value. This is why we've used the Dry_Hole.ToString instead of just Dry_Hole, for example. The "Dry Hole" Data Column is configured to return decimal values, not string values. The .ToString portion of the expression just converts the decimal value to a string so it can be used in the message.

Back to the top

Clear Item

The Clear Item action will clear the data in a Data Field if the Trigger condition is met. Clear Item will also clear a Data Column's data if a Data Table is selected as the Scope. This can provide Grooper users a method of removing data from a field or table column if certain conditions are met. For instance, if you know the data is invalid based of the Trigger expression's true/false evaluation, you may prefer to remove the index data rather than keep the invalid data. This could also be a method of redacting sensitive index data after it is exported to a secure database. (Note: For complete redaction, you would probably also want to use the Redact activity to black-bar or white out the document's image and a Correct activity to remove the data from the document's text data as well)

For instance, we've seen already situations where this intangibles table's "Grand Total" field on the document does not actually add up to the summation of the "Total" column. We could use a Trigger expression to check if the "Total" column adds up to the "Grand Total" field and clear the extracted "Grand Total" Data Field if it does not add up correctly. For this document, the extracted values in the "Total" column are themselves accurate. However, the extracted "Grand Total" is inaccurate. It does not add up to the summation of all the values in the "Total" Data Column. We will configure this Data Rule to check if the values in the "Total" Data Column add up to the value in the "Grand Total" Data Field and clear it if it does not.
We have the Scope set to the Content Model's Data Model level. We need access to both the "Intangibles" Data Table and the "Grand Total" Data Field, both of which are children in the same level of the Data Model. For our Trigger condition, we've used an expression to check if the summation of the "Total" Data Column's values are not equal to the "Grand Total" Data Field. Intangibles_Table.SumOf("Total") <> Grand_Total FYI: <> is the "not equal to" operator in .NET. We've set the True Action property to Clear Item. All you need to configure for the Clear Item action is what field (or Data Column if scoped at a Data Table level) should be cleared. In this case, we've selected the "Grand Total" Data Field using the Element property.
Press the "Test Rule" button to test the Data Rule's execution. In this case, the Trigger expression evaluates to true. As the True Action, the Clear Item action is applied, and the "Grand Total" Data Field is cleared.

Back to the top

Copy Item

The Copy Item action will copy a field's value and paste it into another field. Optionally, the value can be moved as well (like a cut and paste operation). This can be useful for situations when you need to move data around in a Data Model's hierarchy. This provides an easy way to move a Data Field's value into a Data Section's single or multiple section instances.

For example, let's say you need to move the "Grand Total" Data Field into each section instance of a multi-section Data Section. We've configured this Data Section to return 11 section instances. Using the Copy Item action, we can copy a single Data Field. And then paste it into a Data Field for each of these 11 sections. We will configure this Data Rule to do this.
We've set the Data Rule's Scope property to the Content Model's Data Model. We need access to both the Data Model's child Data Field "Grand Total" and its child Data Section's own child Data Field "Grand Total Copy". The Content Model's Data Model includes all of these Data Elements. So it is the appropriate data hierarchy scope to use. We're leaving the Trigger property blank and going straight to the True Action property. We've set this to Copy Item. All you need to configure for the Copy Item action is the Data Field you're copying and what Data Field you're pasting the value to. The Source Element property is the Data Field whose value you are copying. In this case the "Grand Total" Data Field. The Target Element property is for the Data Field you are pasting the value into, populating it with the copied value. In this case the "Grand Total Copy" Data Field of the "Copy Section" Data Section.
Press the "Test Rule" button to test the Data Rule's execution. The Source Element is copied. And it is pasted into the Target Element. If copying into a Data Section, as is the case here, the copied Data Field will populated the targeted Data Field in each section instance established by the Data Section.

Back to the top

Parse Value

The Parse Value action allows you to use regular expression to parse an extracted Data Field's text data into another or multiple other Data Fields. This gives you the capability of using a regex pattern to match part of an extracted value's text and populate a different Data Field with its result. In many ways, this is much easier than expression based methods of parsing text data configured on the Data Field object itself.

Before detailing an example of the Parse Value action below, be aware the regex pattern must make use of "Named Groups" to populate the parsed text into Data Fields. The Named Groups' names in the regular expression will need to match the names of the Data Fields they are populating. How to do this will be covered in the example below.

In this example, we will parse out an extracted date into component month, day and year fields. You may have a full date on a document, but the database where this data ultimately lives may expect the date's day, month and year separated out into their own table columns. A Data Rule using the Parse Value action can quickly and easily do this for you before the data is exported. We have a "Full Date" Data Field already extracted. We will use the Parse Value action to populate the three Data Fields here, "Month", "Day", and "Year". This will be configured using this Data Rule.

We have selected the Content Model's Data Model for this Data Rule's Scope. We have left the Trigger property blank, going straight to the True Action, set to Parse Value. The Source Field property defines what Data Field's text value is parsed. Here, we've selected the "Full Date" Data Field, which has extracted the full date on the document. Using the Pattern property, you will write a regular expression to parse the Source Field's' text value. The regex pattern you write has some specific requirements in order to populate fields with parsed results. Press the ellipsis button at the end of the property to bring up a "Pattern" editor window. This "Pattern" editor window will allow you to enter a regular expression to parse out the desired values. We've started things out with a simple regex pattern to match the full extracted date. See highlighted the portions of the regex that correspond to the desired Data Fields we want to populate. We're halfway there, but Grooper needs a way of knowing what parts of the regex you want to use to populate which Data Fields. We do this with Named Groups.

If you're not familiar with Named Groups, they are a method of calling out portions of a regular expression in ways Grooper can manipulate them elsewhere. They create sub-instances of an extraction instance named according to however you named the group. Once Grooper has a name it can associate with the sub-instance, it can reference it somewhere else, calling on it to return just that portion of the regular expression.

Groups in regular expression are created with open and closed parenthesis. For example, take the regular expression \d{2}/\d{2}/\d{4} which will match some simple date formats. Bracketing the portion of the regular expression in parenthesis will place it in a group. For example placing parenthesis here, (\d{2})/\d{2}/\d{4} would create a group out of the portion of the regex capturing the month part of the date. (\d{2})/\d{2}/\d{4}

You name the group using the syntax ?<NAME> after the open parenthesis. If you wanted to name this group "Month", you would insert the name tag like so: (?<Month>\d{2})/\d{2}/\d{4}

Now we have a group that is named "Month" for just the two-digit month portion of the date. code>(?<Month>\d{2})/\d{2}/\d{4} Furthermore, since this creates a sub-instance of this regex pattern's extraction result, Grooper has a way it call out just these two digits. In other words, it has a way it can parse the month portion of the full date.

We could type this out in our regex pattern, but there are some shortcuts to making Named Groups, in Grooper. Select the portion of the regular expression you want to place in the Named Group with your cursor, and right-click it. Select "Create Group". Note: There's also a keyboard shortcut you can use Ctrl + G instead of right-clicking and pressing the "Create Group" button.
This will insert parenthesis around the text selection, insert the name tag ?<> and place your cursor between the angle brackets to start typing the name for the group.
As far as Named Groups and Parse Value is concerned, there's a couple things to keep in mind. Whatever you name the group determines what Data Field is populated. If the names do match, the Data Field will be populated with the portion of the regex pattern contained in the group. In this case the two-digit month matched by the "Month" group will return to the "Month" Data Field. Grooper's intellisense will also try and help you out. Once you start typing the group's name, Grooper will pop-up a window of available Data Fields as you type. Here, we would want to select the "Year" Data Field, which would name the group "Year". ⚠ There are some restrictions on what you can name a Named Group. Named Groups cannot begin with a number. Spaces and special characters (non-alphanumeric characters like #, $, -, @ etc) must be replaced with underscores _ Note: Grooper will make these substitutions for you if you select a Data Element using intellisense. Because special characters and spaces must be replaced with underscores, you can get into trouble if Data Fields share similar names with only the difference of a special character. For example, "Invoice #" and "Invoice $" would both need to be referenced by the same Named Group name "Invoice_". Grooper will likely throw you an error because two different Data Fields share the same Named Group identifier.
Now that we have the Parse Value action's Pattern configured, we can test it out. Press the "Test Rule" button to test the Data Rule's execution. If the Pattern property's regex matches the text data from the Source Field, the Named Groups in the regex will be parsed out to their correspondingly named Data Fields in the Data Model. Substrings of the source Data Field's value then populate the Data Fields whose names match Named Groups in the regular expression. ⚠ The source Data Field and the parsed Data Fields must be peers in the Data Model's hierarchy. You could not, for example, use a Data Field outside of a Data Section to populate Data Fields within the Data Section.

Back to the top

Action List

The Action List action gives you a way to execute multiple actions for a single Data Rule. This way, if multiple actions share the same trigger condition, you can just execute all of them in a list, one after the other without creating multiple Data Rule objects. The actions are executed in sequential order, one after the other in the list. This means order of operations can be important. If one action mutates the data in a way that affects another action, you may need to move an action up or down the list.

For example, in the previous action examples, we saw three Data Rule configurations that did not use any Trigger configuration whatsoever. So, they share the same trigger condition, just in this case that condition is always "true". Rather than executing three separate Data Rules, we could create a single Data Rule that executes all three actions. We will use the "no trigger" configurations of these three Data Rules. So, our Action List action will itself contain three actions a Calculate Value action, a Copy Item action, and a Parse Value action. We will configure this Data Rule with the Action List action.
As well as all actions sharing the same Trigger condition (here left blank), they also must share the same Scope. Here, all three actions use the Content Model's Data Model scope. The True Action is set to Action List. To add the actions, use select the Actions property and press the ellipsis button at the end. This will bring up an "Actions" collection editor window. Press the "Add" button to add an action to the list. Select the action you want to add.
As you add actions to the list, they will appear on the left panel of the collection editor. Use the right panel to configure each added action. Press the "OK" button when finished adding and configuring actions.
If the Trigger condition is met, each of the actions in the Action list executes. In our case the Trigger property is blank. So the actions execute by default. We have three actions in our list as indicated by the "Count=3" listed here. First, the Calculate Value action adds up the values in the "Intangibles Table" Data Table's "Total" Data Column, populating the "Grand Total" Data Field. Next, the Copy Item action copies the "Grand Total" Data Field's value to the "Grand Total Copy" Data Field in the "Copy Section" Data Section's sections. Note this is a situation where order of operations could be important. If the Copy Item action came before the Calculate Value action, this field would be empty (for the selected document). The Copy Item action would just copy blank data in that that case, effectively doing nothing. Last, the Parse Value action parses out the extracted "Full Date" Data Field into component "Month", "Day", and "Year" Data Fields.

Data Rule Hierarchy

One of the greatest strengths of the Data Rule object is the ability to create a hierarchy of multiple actions with different conditions. Data Rules can have other Data Rules as their own children. If the parent Data Rule's Trigger expression returns true, not only will its True Action' be applied, but its children Data Rules will also execute. These children Data Rules may have their own Trigger conditions for their actions. Their own child Data Types may even have their own Trigger' expressions. And on down the hierarchy.

This allows you to execute multi-conditional Data Rules with their own conditions, themselves executing only if their parent Data Type's trigger conditions are met. As long as the trigger expression returns true, the next level in the Data Rule hierarchy will execute. If the first Data Rule's Trigger is true, its child Data Rule will execute. If the child Data Rule's Trigger expression is true, its child Data Rule will execute. If its child Data Rule's Trigger is true, its own child Data Rule will execute. And so on down the line.

Let's look at a simple example using some of the actions in our Actions section above. We've seen these actions applied one-by-one, but we're going to create a single Data Type using a hierarchy of conditions to execute multiple actions.

Planning ahead, let's think about what we want to do.

1. If the "Dry Hole" column and the "Completion" column doesn't add up to the "Total" column, there's probably something drastically wrong with the data in (or extracted from) the intangibles table. If that is the case, the "Grand Total" extracted is probably wrong. We might just want to stop at that point to prevent ourselves from using incorrect data in one way or another. That document might need a human reviewer to check why the math doesn't add up.
2. If the column values do add up right, we probably want to add up the "Total" column to populate for the documents that don't have the Grand Total listed on the page.
   • We've already got a multi-conditional Data Rule! The first condition will form the first level in our Data Rule hierarchy, checking to see if the "Dry Hole" column and the "Completion" column add up to the "Total" column. The second condition will form the second level in our Data Rule hierarchy, checking to see if the "Grand Total" Data Field is empty.
3. For documents that do have a Grand Total field on the document, we want to check and see if the listed value adds up to all the values in the "Total" column added together. If it doesn't, we'll say we just want to clear the "Grand Total" field entirely.
4. But again, we only want to do this assuming the "Dry Hole" and "Completion" columns add up to the "Total" column properly. So, we have another multi-conditional Data Rule, but the second condition will be different than the condition used to check if the "Grand Total" Data Field is empty.
5. What if we still want to parse out the date value into day, month and year fields regardless of what's going on in the table? As we will see, this will be no problem for us. We can still use a single Data Rule to execute this action as well, outside of the multi-conditional action execution described above.

The first thing we're going to do is add a "blank" Data Rule. This only has the Scope property configured and nothing else. Here, set to the highest level in the Content Model available, the Data Model. You might think of this kind of Data Rule configuration as a container for subsequent Data Rules. No Trigger or Action has been configured, only the Scope at which this rule executes. By itself, this Data Rule will do nothing. But once we add child Data Rules to it and configure them, they surely will. This can be a way of organizing complex Data Rule hierarchies. In our case, it will also serve the purpose of creating a single parent Data Rule which will be used to apply all its child Data Rule configurations.
Next we need to add a child Data Rule that does actually do something. To do this, right-click the Data Rule. Select "Add" and "Data Rule..." We'll name this Data Rule "Total Column Validation" Press the "OK" button to add the Data Rule.
This will add a child Data Rule the next level down in the node tree. For the time being, we've set this Data Rule's Scope down to the "Intangibles Table" Data Table. And we've used the same Trigger expression in the Calculate Value action description described above. This isn't actually going to work out for us. But it brings up the importance of Scope when creating a hierarchy of multiple Data Rules.
Once you select a narrower scope in a Data Model's hierarchy, you limit your access to the Data Elements within that scope. This can cause problems when attempting to configure subsequent child Data Types Trigger expressions if they require access to Data Fields, Data Tables, or Data Sections outside of this more limited scope. If we are now to add a child Data Rule to the "Total Column Validation" Data Rule... And remember the "Total Column Validation" Data Rule is scoped to the "Intangibles Table" Data Table. Selecting the Scope property, you'll see we're limited to the "Intangibles Table" Data Table's level. When adding children Data Rules you can scope down the Data Model's hierarchy but you can't scope up. Since we ultimately will need access to Data Elements like the "Grand Total" Data Field which are outside this Data Table's scope, we need to enlarge the scope of the parent "Total Column Validation" Data Rule.
Now that that's out of the way, let's look at how we should have configured the "Total Column Validation" Data Rule. Rewind! We're back to the "Total Column Validation" Data Rule in our node tree. The Scope is set wider, to the full Data Model, giving us fuller access to its Data Elements. We do have to use a slightly different Trigger Expression however. We can't check if the individual cells in each row add up correctly. But what we can check is if the sum total of the "Dry Hole" column added with the sum total of the "Completion" column adds up to the sum total of the "Total" column, using the expression below. Intangibles_Table.SumOf("Dry Hole") + Intangibles_Table.SumOf("Completion") = Intangibles_Table.SumOf("Total") Functionally, this is what we're trying to check. If even one row's data doesn't add up, the sum total isn't going to add up either. This will still work as that first condition to check if our table data is ok. In this case we will not configure an action. This is just the first condition upon which the next actions second condition are dependent. Adding a child Data Rule to this Data Rule will still require this Trigger expression to be true in order for it to execute.
To create the next level of our Data Rule's hierarchy, we will add child Data Rule to the "Total Column Validation" Data Rule. This child Data Rule will only execute if its parent Data Rule's trigger condition is satisfied (if it returns "true"). This will require two triggers for the child Data Rule's action to be applied, both its parent's trigger and its own. This is the basic idea around multi-conditional execution of Data Rules. You effectively daisy-chain trigger conditions as you go down the parent-child node levels. Here, we have added the child Data Rule named "Blank Grand Total Calc" We've kept the scope at the largest level, the Content Model's Data Model. Our Trigger expression is set to Grand_Total = 0 We want this Data Rule to add up the values in the "Total" column if the "Grand Total" Data Field is unextracted (which will make it evaluate to "0" as a decimal field). Note, this Data Rule as a child of its parent Data Rule actually has two conditions for the action's application. First, its parent Data Rule checking if the "Dry Hole" and "Completion" values add up to the "Total" column's values. Then, its own trigger condition, checking to see if the "Grand Total" Data Field is "0", must also be true. Since we are manipulating extracted data, instead of just validating it or copying it or clearing it or parsing out portions of the extracted text, we've set the True Action to Calculate Value. The Target Field property determines which Data Field is populated with the calculated value. Here, set to the "Grand Total" Data Field. The Value Expression property determines how the value is calculated. Here, set to the expression Intangibles_Table.SumOf("Total")
When it comes time to testing this Data Rule, be sure to execute the right level of the Data Rule hierarchy. In order to ensure all conditions above the Data Rule are also executed, be sure to test (and ultimately execute the Data Rule with the Apply Rules activity) at the highest parent level. The top level of the hierarchy is the Data Rule that contains all child Data Rules in the node levels below it. In our case, it is this Data Rule named "Data Hierarchy Example". Press the "Test Rule" button to test the Data Rule's execution, including all its children Data Rules. First the top level of the hierarchy is executed at the selected Data Rule, then the next level, then the next level, and so on. In this case, our parent Data Rule has no Trigger condition or Action. So, the next level in the hierarchy executes. This Data Rule checks to see if the "Dry Hole" and "Completion" columns add up to the "Total" column in the table. If the trigger condition is satisfied (if the expression evaluates to "true"), the next level of the hierarchy executes. In this case, executing the Data Rule we just added to determine if the "Grand Total" field is blank. Here, both conditions in the hierarchy are met, applying the True Action configuration. In this case, the Calculate Value action's configuration to populate the "Grand Total" Data Field with the values of the "Total" column added together.
For the next Data Rule added to our hierarchy, we will add the Data Rule configured to recalculate the "Grand Total" Data Field if the extracted "Total" column's values all added together does not match the extracted "Grand Total" on the document. However, we also want this rule to only execute if the "Dry Hole" and "Completion" column values added together equal the "Total" column's values added together. After all, if the "Grand Total" doesn't match up with the summation of the "Total" column, as long as the "Dry Hole" column added with the "Completion" column does add up to the "Total" column, we can just use the values in the table to calculate the "Grand Total" Data Field instead of the wrong value on the document. So, it also needs the trigger condition configured on the "Total Column Validation" Data Rule, but will have its own trigger condition, distinct from the "Total Column Validation" Data Rule's trigger. To do this we will add a Data Rule as another child of the "Total Column Validation" Data Rule. As both children of the "Total Column Validation" Data Rule they are peers or "siblings" in the hierarchy. They will share their parent's trigger condition. If that parent condition is met, they will subsequently execute with their own trigger condition configurations. Here, we've added a Data Rule named "Grand Total Recalc" as a child of the "Total Column Validation" Data Rule. We've kept the scope at the largest level, the Content Model's Data Model. Our Trigger expression is set to Intangibles_Table.SumOf("Total") <> Grand_Total Since we are using data in the documents extracted table data to come up with a new value, we've set the Target Field property to Calculate Value. The Target Field property determines which Data Field is populated with the calculated value. Note, this Data Rule as a child of its parent Data Rule actually has two conditions for the action's application. First, its parent Data Rule checking if the "Dry Hole" and "Completion" values add up to the "Total" column's values. Then, its own trigger condition, checking to see if the extracted "Grand Total" Data Field value is different from the summation of the "Grand Total" Data Column's values. Here, set to the "Grand Total" Data Field. The Value Expression property determines how the value is calculated. Here, set to the expression Intangibles_Table.SumOf("Dry Hole") + Intangibles_Table.SumOf("Completion")
When the parent Data Rule executes, its children execute as well, going to the next level in the hierarchy, as long as the Trigger expressions keep evaluating to "true". Press the "Test Rule" button to test the Data Rule's execution, including all its children Data Rules. The parent Data Rule does not have its Trigger property configured. So, it evaluates as true by default. Then, the next Data Rule in the hierarchy executes. In this case the "Total Column Validation" Data Rule's trigger returns "true". Then, the next Data Rule in the hierarchy executes. In this case we have multiple sibling Data Rules. The first child "Blank Grand Total Calc" does not trigger in this case because the "Grand Total" Data Field is populated (Grand_Total = 0 evaluates to false). So, the next child Data Rule executes.
What if we also want to do something that doesn't really have anything to do with this conditional execution of Data Rules we've set up here? What if we want to parse out the month, day, and year values from the document's extracted date using the Parse Value action? That doesn't have anything to do with the information in the table values. Whether the column values match up correctly shouldn't impact that action whatsoever. That is not a problem whatsoever. We could easily add a Parse Value Data Rule. We'd just need to make sure it was added to the right level of the hierarchy. Since the "Total Column Validation" Data Rule's execution doesn't impact whether or not we want to parse out the date value, we want to make sure it is at a level above or at least at the same level in the hierarchy. This will make it execute independent from its Trigger configurations. Here, we've added a "Parse Date" Data Rule to the same level as the "Total Column Validation" Data Rule. They are now siblings in the hierarchy, both children of the parent "Data Hierarchy Example" Data Rule. They will now both execute when the "Data Hierarchy Exmaple" Data Rule executes. First the "Total Column Validation" Data Rule and its children. Then, once all that's done, the "Parse Date" Data Rule will execute. Note, this Data Rule uses the exact same configuration as discussed in the Parse Date example above As a sibling in the hierarchy, when the parent Data Rule executes, the "Parse Date" Data Rule will execute regardless of what happens with its sibling "Total Column Validation" Data Rule's execution.

An Aside on False Actions

In this article, we've almost exclusively focused on True Actions. If the Trigger expression is true, the configured action is applied. False Actions bear some mention as well. You may want to do something if the trigger condition is not met instead or in combination with an action if it is met.

However, when it comes to Data Rule execution in a hierarchy, whether a True or False Action is configured does not impact whether the next rung of the hierarchy executes. The Trigger expression solely determines this. The next level of the Data Rule's hierarchy will only execute if the Trigger expression returns true. The True or False Action only determines what action is taken for the specific Data Rule. Another way of getting at this, an action is not necessary to continue executing child Data Rules, only that the trigger conditions keep returning true.

Let's revisit our "Total Column Validation" Data Rule, configured above. Previously, we didn't perform any action on this Data Rule. Instead, it was used as a trigger condition for its two child Data Rules. Here, we've configured a False Action set to clear the "Grand Total" Data Field using the Clear Field action. If the Trigger expression evaluates to "false", this action will be applied. If it evaluates "true", its child Data Rules will execute. For this document, the "Code" and "Dry Hole" columns do not property add up to the "Total" column. The expresion Intangibles_Table.SumOf("Dry Hole") + Intangibles_Table.SumOf("Completion") = Intangibles_Table.SumOf("Total") is false. So, upon executing this Data Rule, the False Action clears the "Grand Total" Data Field. Note the Data Rule stops executing at this point. Since the Trigger expression is false, the child Data Rules do not execute, and the field is no longer manipuated.