2021:Data Rule (Node Type)

2021	This article is in development for the upcoming version of Grooper, Grooper 2021. The Value Reader is a new data extraction object in 2021. This information is incomplete and/or may change by the time of release.

The Data Rule object allows for complex validation and manipulation of a Data Model's Data Elements (Data Fields, Data Sections, and Data Tables) in Grooper.

This allows users to create a conditional hierarchy of actions to take if certain conditions met. These conditions are configured using .NET, LINQ and/or lambda expressions. When the expression is "triggered", either evaluating to "true" or "false", certain actions can be made. These include:

• Calculate Value - This action sets the value of a Data Field or cells a Data Column, using calculate expressions to perform mathematical or concatenation operations of Data Elements.
• Clear Item - This action clears the value of a Data Element.
• Copy Item - This action copies or moves the value of a Data Element.
• Parse Value - This action uses a regular expression pattern to return part of a Data Field's value or cell in a Data Column's value.
• Raise Issue - This action adds an issue to the issue log, used for validating a Data Element. This action can also be used to flag the Data Element.

These trigger conditions and subsequent actions set on the Data Rules objects are executed through the Apply Rules activity after data is extracted from an Extract activity.

About

Some Basics About Expressions

Grooper makes use of expressions to validate extracted data and use extracted data to populate fields in a Data Model. Traditionally, this is configured on a Data Field object in a Data Model (or in the case of validating or calculating cells in a table, the Data Column object), using the Default Value, Calculated Value, or Is Valid properties.

For example, let's say we have several documents in a Batch. Each one contains W-2 wage reporting forms for various individuals and we want to do some basic tax filing calculation. In order to find someone's total income, it may not be quite as simple as pulling the listed wages from a single W-2. An individual might have multiple W-2s from multiple employers.

If an individual worked for three different employers over the course of a year, their total income would be the wages from all three W-2s added together. This is where expressions come in handy in Grooper. There is no extractible "Total Wages" field on the document. It's just three pages, each page a different W-2. There is no text data for an extractor to return that corresponds to all three W-2's wages field added together.

But we could create a Data Section in our Data Model to return the wages from each individual W-2 form, by adding a "Wages" Data Field and configuring its extraction. Then, we could create a "Total Wages" Data Field and use a Calculated Value expression to add up the results of each "Wages" Data Field in each section (each W-2 in this case).

Here, we have a simple Content Model set up to solve the problem described above. It has a Data Section named "W2 Info" It has two child Data Fields: "Fed Wages" and "Employee SSN". The Data Section is configured to create one section for each W2 or each page in this case. This document has three W2s (or three pages). Hence, we have three sections. The "W2 Info" Data Section's child "Fed Wages" Data Field is configured to return the "Wages, tips, other compensation" field for the W-2s. This "Total Income" Data Field will be configured to add up the results of the "Fed Wages" Data Field from each section.
Here, we've selected the "Total Income" Data Field. Calculated Value property is configured with the expression W2_Info.SumOf("Fed Wages") This will add the results of each "Fed Wages" Data Field in the "W2 Info" Data Section together. The Calculate Mode is set to Always Set. This will always run the expression regardless of whether the Data Field is populated by an extractor (It's not in this case. The Value Extractor property is blank.). FYI If you were to press the "Test Extraction" button at this point, the Calculated Value expression would appear to fail. However, it's actually configured correctly to do what we want. This has to do with the scope we're testing extraction in our Data Model. With the "Total Wages" Data Field selected in the node tree, we've narrowed down the scope of our Data Model to just test within this Data Field's scope. But we're actually looking for information outside the "Total Wages" Data Field's scope. We're using information in the "W2 Info" Data Section's scope to add up the total wages for each W2 in its child "Fed Wages" Data Fields. Once we select the parent Data Model in the node tree, we will be at a higher scope in the data hierarchy. At that point, the expression will run against the full scope and successfully execute.
Upon extraction... The expression executes, adding up the "Fed Wages" Data Field values for each section.

Conditional Expressions and Data Rules

You can do a lot with expressions, even applying some conditional logic to their execution. If the condition is met, the expression executes. If not, it doesn't or something other expression executes.

In our example of documents containing W-2 forms we make some assumptions about the document. We assume each document contains a W-2 for a single individual. Each individual should only have one social security number. It would be problematic if their were multiple social security numbers extracted from the W-2 forms. This could indicate there are multiple W-2s for multiple individuals in a single document.

To account for this, you could use a more complex Calculated Value expression to only add up the "Fed Wages" Data Fields if the social security number was the same for each document. If the condition of their only being one social security number for each W-2 is met, the expression to add up the wages would execute. If not, it wouldn't.

This is basic conditional logic. If "x" condition is met, then do "y". If there's only one social security number, then add up all the wages. Otherwise, do nothing (or something else). You could go another step further and add an Is Valid expression to flag the document if the social security numbers didn't match, as well.

However, the more complex a Data Model's data hierarchy (the more Data Sections and Data Tables it has), generally the more complex these conditions tend to be. The more conditions you add for an expression to execute, the more complex the expression becomes. This can result in very cumbersome expressions that are difficult to form and manage.

This is where the Data Rule object really shines. Data Rules allow you to use Trigger expressions to determine one or multiple subsequent Actions to take if that expressions evaluates to true (or false). This is also basic conditional logic. If the trigger expression is true, do the action. Otherwise, do nothing (or a different action). Furthermore, you can more easily create a complex hierarchy of conditions by adding child Data Rules to parent Data Rules. If the trigger expression evaluates to true, the child Data Rules will execute, with their own triggers and even own child Data Rules. This allows for simpler set up, execution, and management of more complex conditional expressions as well as some actions that fall outside normal expressions you can set up in a Data Field or Data Column.

A Basic Example Data Rule

Here, we've added a Data Rule object to our Content Model's Local Resources folder. As it is configured now, this Data Rule (when executed) will add up "W2 Info" Data Section's "Fed Wages" Data Field values for each document, populating the "Total Income" Data Field. As is currently, it does essentially the exact same thing as the Calculated Value property configuration of the "Total Income" Data Field discussed above. The first thing you'll always configure for a Data Rule is the Scope property. This determines at what level in a Content Model's Data Model hierarchy the Data Rule executes. This will determine which Data Elements are accessible. In this case, we set the Scope to the full Data Model. This gives the most access to the most Data Elements in the Data Model. If we instead selected the "W2 Info" Data Section as the Scope instead, we would only have access to the Data Elements in that Data Section. This would not include the "Total Income" Data Field, which is what we're trying to populate here. The Trigger property controls the condition determining whether or not the action executed by the Data Rule is taken or not. The Trigger can be a .NET, LINQ or lambda expression. If left blank, as is the case here, the Trigger defaults to "true". In effect, the Data Rule will always execute its action. We will revisit this Trigger soon. The True Action property determines what happens if the Trigger evaluates to "true". Or put another way, if the Data Rule's condition is met. There are many possible actions detailed in the Actions section of this article. Each action type has its own set of configuration properties. In this case, we have selected Calculate Value. This action type's configuration is very similar to how a Data Field's Calculated Value property is configured. The Target Field property controls what Data Field (or Data Columns in the case of cells in a Data Table) is affected by the action. The Value Expression property controls what populates the Target Field's Data Field. It is what is calculated (or maybe better put how it's calculated). In this case we're using the exact same expression to add up the "Fed Wages" Data Fields in each of the "W2 Info" Data Section's sections as discussed above.

Data Rules are executed by the Apply Rules activity. After data is extracted by the Extract activity, any Data Rule referenced by the Apply Rules activity will alter the document's index data according to the Data Rule's configuration. You can test the Data Rule's results in Grooper Design Studio when the Data Rule is selected in the node tree. This will help you verify its configuration, giving you a preview of what would happen if that Data Rule was executed by the Apply Rules activity. Press the "Test Rule" button to execute the selected Data Rule. You can see in our case, the Calculate Value action is executed, using the Value Expression W2_Info.SumOf("Fed Wages") to populate the Target Field, "Total Income". Effectively, this adds up all the wage values for each W-2 in the document and returns the result to the "Total Income" Data Field. ⚠ Data Rules differ from expressions configured on Data Field (or Data Column) objects in one major way. Index data for documents must be extracted before executing a Data Rule. Documents must be processed by an Extract activity before testing a Data Rule or running the Apply Rules activity. The Data Model's results must be present and stored on the document before the Data Rule can manipulate them. Expressions configured on the Data Field (or Data Column) objects themselves execute during extraction when the Extract activity runs.

The Trigger

The Trigger property serves the purpose of establishing the condition that must be met in order for the Data Rule's action to be taken. These triggering conditions are also set using expressions. These expressions must return a Boolean "true" or "false" value. If the Trigger expression evaluates to "true", the True Action configuration is executed. If the Trigger expression returns "false", the False Action configured is executed (If it is configured. If left blank, no further action will be taken.)

In our case, something is wrong with our documents if the W2 forms have more than one social security number. Individuals should only have one social security number. If we added up all the wages for multiple W-2s with mismatched social security numbers, we would not be adding up the total income for an individual correctly. We'd end up with inaccurate data. This is the case for this document. It is comprised of two W-2 froms. For the first one, in the first section, the social security number extracted is "987‑56‑4321" For the second one, in the second section, the social security number extracted is "987‑65‑4321"

Luckily, there's an expression we could use to determine if our "W2 Info" Data Section has multiple social security numbers in its sections. This is a good opportunity for a LINQ expression. LINQ (or Language INtegrated Query) expressions are particularly helpful when navigating a Data Model's hierarchical structure to pull information from Data Sections and Data Tables.

Writer's Note: You aren't limited to just LINQ expressions for the Trigger. You can use any expression that returns a Boolean value, including standard.NET expressions, LINQ expressions, and lambda expressions. A LINQ expression just works well for this particular example.

(From sec In W2_Info Select sec.Employee_SSN).Distinct().Count() = 1

Let's break down this expression to understand what's going on.

(From sec In W2_Info Select sec.Employee_SSN).Distinct().Count() = 1

This is the "LINQ-iest" part of the expression. It's querying the extracted instances of our Data Model's objects, to return multiple results.

LINQ expressions always start with From (This indicates the data source from where are you querying the data). Next we declare a type variable we've named sec. We'll use later in the expression to return multiple instances of a Data Field in a Data Section (What you name it doesn't matter, just that you use the same name when you reference the variable later on in the query). The In clause determines the query's scope. We're looking for the "Employee SSN" Data Field in the "W2 Info" Data Section. The "W2 Info" Data Section is our scope. In W2_Info will only query instances (the results of its children Data Fields) in the sections produced by the "W2 Info" Data Section. The Select clause determines what values the query returns (or "selects"). We want information about the "Employee SSN" Data Fields. So, we've entered sec.Employee_SSN. Note, we've referenced the variable we declared at the start of the query, sec to do this.

Now the expression has some information it can work with. In this case, the social security numbers for each W-2 in the document (as returned by the "Employee SSN" Data Field for each section produced by the "W2 Info" Data Section).

(From sec In W2_Info Select sec.Employee_SSN).Distinct().Count() = 1

This part of the expression is counting the number of distinct values returned by the query (Technically, the .Distinct() expression is returning a subset of distinct values in the query's results. Then, the .Count() expression is counting the values in that subset). If the social security number is the same for each W2, there's only one distinct value. This should evaluate to "1". If not, it will be a larger number.

(From sec In W2_Info Select sec.Employee_SSN).Distinct().Count() = 1

This is just an equivalency argument to give us a Boolean "true" or "false" value. If the left side of the argument (the expression (From sec In W2_Info Select sec.Employee_SSN).Distinct().Count()) counts a single unique social security number in each section is equivalent to the right side of the argument (i.e. "1 = 1") it will return "true", otherwise "false".

If we use this expression as the Data Rule's Trigger, it will conditionally execute the True Action configured above only if it evaluates as true. Effectively, it will only add up all the wages for each W-2 only when the social security numbers for each W-2 are the same. Using the Trigger property, we've entered the LINQ expression described above. (From sec In W2_Info Select sec.Employee_SSN).Distinct().Count() If we press the "Test Rule" button to execute the Data Rule, it will only apply the rule's 'True Action configuration if the Trigger Expression is true. As you can see here, this document extracted multiple different social security numbers. So, nothing happens. The "Total Income" Data Field remains blank.

However, in the case of this document, its three W-2 forms do have the same social security number. So the "Employee SSN" field is the same for all three section instances of the "W2 Info" Data Section. Therefore, the Trigger expression returns "true". The True Action then executes, which is set to Calculate Value populating the Target Field "Total Income" with the results of the Value Expression W2_Info.SumOf("Fed Wages") In other words, the three "Fed Wages" Data Field results are added together and returned to the "Total Income" Data Field. Conditional logic!

Actions

Once a Data Rule is triggered, what happens next is determined by the True Action and False Action properties. When the Trigger expression evaluates to true, the True Action is executed. When the Trigger expression returns false, the False Action is executed. This determines what action is taken once the trigger condition is met or not met.

This can be one of six choices:

• Calculate Value
• Raise Issue
• Clear Item
• Copy Item
• Parse Value
• Action List

Each action has its own configuration to execute the action, detailed below.

Calculate ValueRaise IssueClear ItemCopy ItemParse ValueAction List

Calculate Value

The Calculate Value action will use a .NET, LINQ or lambda expression to populate a field with the expression's result. The possibilities here are as endless as the capabilities of these expressions. We can perform mathematical operations on numerical data. We can concatenate multiple string fields. We can perform incremental additions to date values. The Calculate Value action allows you to use any configurable expression to manipulate extracted data into a desired result. We've already seen one example of the Calculate Value action in the section of this article above. But let's look at another one.

In this example, we have a fairly simple report detailing costs of intangible services related to an oil drilling operation. As is necessary prior to the execution of a Data Rule, these documents have already been classified and their data has been extracted. In other words, they have been processed by the Classify activity (according to how the "Data Rules - Intangibles Table" Content Model is configured) and then by the Extract activity (using the data hierarchy and extraction configuration set on its associated Data Model). All of the table information has already been extracted, including the "Total" column which adds up the "Dry Hole" and "Completion" costs for each row. However, we might want to know the total cost associated with this table. We will use the Calculate Value action to populate this "Grand Total" Data Field.
We have added a Data Rule to the Local Resources folder of this Content Model. The Scope is set to Content Model's Data Model The Scope property determines what Data Elements in the Data Model's hierarchy the Data Rule has access to. In this case, we need a fairly broad scope. We need access to the Data Table object "Intangibles Table" and the Data Field object "Grand Total". We would not in this case want to scope down to the Data Table, for example. Then, we would only have access to its child Data Elements, namely its Data Column objects. The "Grand Total" Data Field lies outside the "Intangible Table" Data Table's scope, but both are within their parent Data Model's scope. Hence, we choose the parent Data Model. We're leaving the Trigger property blank for the time being. This means the Trigger will default to "true" and the True Action will always execute. We've chosen Calculate Value for the True Action. With Calculate Value selected, we must indicate which Data Field is populated. The whole point of this action is to manipulate data to come up with some new value. That value has to go somewhere. This property points the calculated value to the desired location in the Data Model. Here, we've selected the "Grand Total" Data Field. The Value Expression property determines how the value is calculated. The result of the expression entered here is what is ultimately returned. In our case, we're adding up all the values in the "Total" column of the table. The expression Intangibles_Table.SumOf("Total") does just that.
When the Data Rule executes... The Calculate Value action's Value Expression configuration executes. In this case, the expression adds up all the values in the "Total" column. And, its resulting value populates the assigned Target Field.

A Word of Caution: Overwriting Results

There is one important thing to note about the Calculate Value action. The Value Expression's calculated value will overwrite any existing data in a field. For example, take this document. This document has a "Grand Total" value listed on the document. That means an extractor can find it and return it to a Data Field. Such is the case here. This value is the extracted value found in the document's text data. However, this isn't actually an accurate total. The document is wrong. The grand total of all the values in the "Total" column should add up to "$1,048,050.00" and not what we see here, "$1,111,000.00"
If we execute the Data Rule on this document... See that the extracted value is replaced by the Value Expression's result.
Now, this may be what you want to do, but it may not be what you want to do. What if you don't want to overwrite the "Grand Total" Data Field it it's already populated? What if you only want to use the Calculate Value action to populate the field if it's blank? That's a great opportunity for a Trigger expression! If this were the case, we would only want to execute this Data Rule if the "Grand Total" Data Field is not there. We could use that as the condition to execute the True Action. All we need to do is figure out an expression that would evaluate to true or false if that's the case. The expression Grand_Total = 0 would return true if the field isn't populated and false if it was. We configure the conditional execution of the Data Rule using the Trigger property. Here, we've used the expression Grand_Total = 0 When the Data Rule executes, it will only apply the True Action if that Trigger expression is true. In this case, it is false. There's already a number in the "Grand Total" Data Field that is not "0". With no False Action configured, nothing happens. The True Action is not applied. The data remains intact.

Back to the top

Raise Issue

Clear Item

Copy Item

Copy Item

Copy Item