2021:Data Rule (Node Type)

2021	This article is in development for the upcoming version of Grooper, Grooper 2021. The Value Reader is a new data extraction object in 2021. This information is incomplete and/or may change by the time of release.

The Data Rule object allows for complex validation and manipulation of a Data Model's Data Elements (Data Fields, Data Sections, and Data Tables) in Grooper.

This allows users to create a conditional hierarchy of actions to take if certain conditions met. These conditions are configured using .NET, LINQ and/or lambda expressions. When the expression is "triggered", either evaluating to "true" or "false", certain actions can be made. These include:

• Calculate Value - This action sets the value of a Data Field or cells a Data Column, using calculate expressions to perform mathematical or concatenation operations of Data Elements.
• Clear Item - This action clears the value of a Data Element.
• Copy Item - This action copies or moves the value of a Data Element.
• Parse Value - This action uses a regular expression pattern to return part of a Data Field's value or cell in a Data Column's value.
• Raise Issue - This action adds an issue to the issue log, used for validating a Data Element. This action can also be used to flag the Data Element.

These trigger conditions and subsequent actions set on the Data Rules objects are executed through the Apply Rules activity after data is extracted from an Extract activity.

About

Some Basics About Expressions

Grooper makes use of expressions to validate extracted data and use extracted data to populate fields in a Data Model. Traditionally, this is configured on a Data Field object in a Data Model (or in the case of validating or calculating cells in a table, the Data Column object), using the Default Value, Calculated Value, or Is Valid properties.

For example, let's say we have several documents in a Batch. Each one contains W-2 wage reporting forms for various individuals and we want to do some basic tax filing calculation. In order to find someone's total income, it may not be quite as simple as pulling the listed wages from a single W-2. An individual might have multiple W-2s from multiple employers.

If an individual worked for three different employers over the course of a year, their total income would be the wages from all three W-2s added together. This is where expressions come in handy in Grooper. There is no extractible "Total Wages" field on the document. It's just three pages, each page a different W-2. There is no text data for an extractor to return that corresponds to all three W-2's wages field added together.

But we could create a Data Section in our Data Model to return the wages from each individual W-2 form, by adding a "Wages" Data Field and configuring its extraction. Then, we could create a "Total Wages" Data Field and use a Calculated Value expression to add up the results of each "Wages" Data Field in each section (each W-2 in this case).

Here, we have a simple Content Model set up to solve the problem described above. It has a Data Section named "W2 Info" It has two child Data Fields: "Fed Wages" and "Employee SSN". The Data Section is configured to create one section for each W2 or each page in this case. This document has three W2s (or three pages). Hence, we have three sections. The "W2 Info" Data Section's child "Fed Wages" Data Field is configured to return the "Wages, tips, other compensation" field for the W-2s. This "Total Income" Data Field will be configured to add up the results of the "Fed Wages" Data Field from each section.
Here, we've selected the "Total Income" Data Field. Calculated Value property is configured with the expression W2_Info.SumOf("Fed Wages") This will add the results of each "Fed Wages" Data Field in the "W2 Info" Data Section together. The Calculate Mode is set to Always Set. This will always run the expression regardless of whether the Data Field is populated by an extractor (It's not in this case. The Value Extractor property is blank.). FYI If you were to press the "Test Extraction" button at this point, the Calculated Value expression would appear to fail. However, it's actually configured correctly to do what we want. This has to do with the scope we're testing extraction in our Data Model. With the "Total Wages" Data Field selected in the node tree, we've narrowed down the scope of our Data Model to just test within this Data Field's scope. But we're actually looking for information outside the "Total Wages" Data Field's scope. We're using information in the "W2 Info" Data Section's scope to add up the total wages for each W2 in its child "Fed Wages" Data Fields. Once we select the parent Data Model in the node tree, we will be at a higher scope in the data hierarchy. At that point, the expression will run against the full scope and successfully execute.
Upon extraction... The expression executes, adding up the "Fed Wages" Data Field values for each section.

Conditional Expressions and Data Rules

You can do a lot with expressions, even applying some conditional logic to their execution. If the condition is met, the expression executes. If not, it doesn't or something other expression executes.

In our example of documents containing W-2 forms we make some assumptions about the document. We assume each document contains a W-2 for a single individual. Each individual should only have one social security number. It would be problematic if their were multiple social security numbers extracted from the W-2 forms. This could indicate there are multiple W-2s for multiple individuals in a single document.

To account for this, you could use a more complex Calculated Value expression to only add up the "Fed Wages" Data Fields if the social security number was the same for each document. If the condition of their only being one social security number for each W-2 is met, the expression to add up the wages would execute. If not, it wouldn't.

This is basic conditional logic. If "x" condition is met, then do "y". If there's only one social security number, then add up all the wages. Otherwise, do nothing (or something else). You could go another step further and add an Is Valid expression to flag the document if the social security numbers didn't match, as well.

However, the more complex a Data Model's data hierarchy (the more Data Sections and Data Tables it has), generally the more complex these conditions tend to be. The more conditions you add for an expression to execute, the more complex the expression becomes. This can result in very cumbersome expressions that are difficult to form and manage.

This is where the Data Rule object really shines. Data Rules allow you to use Trigger expressions to determine one or multiple subsequent Actions to take if that expressions evaluates to true (or false). This is also basic conditional logic. If the trigger expression is true, do the action. Otherwise, do nothing (or a different action). Furthermore, you can more easily create a complex hierarchy of conditions by adding child Data Rules to parent Data Rules. If the trigger expression evaluates to true, the child Data Rules will execute, with their own triggers and even own child Data Rules. This allows for simpler set up, execution, and management of more complex conditional expressions as well as some actions that fall outside normal expressions you can set up in a Data Field or Data Column.

A Basic Example Data Rule

Here, we've added a Data Rule object to our Content Model's Local Resources folder. As it is configured now, this Data Rule (when executed) will add up "W2 Info" Data Section's "Fed Wages" Data Field values for each document, populating the "Total Income" Data Field. As is currently, it does essentially the exact same thing as the Calculated Value property configuration of the "Total Income" Data Field discussed above. The first thing you'll always configure for a Data Rule is the Scope property. This determines at what level in a Content Model's Data Model hierarchy the Data Rule executes. This will determine which Data Elements are accessible. In this case, we set the Scope to the full Data Model. This gives the most access to the most Data Elements in the Data Model. If we instead selected the "W2 Info" Data Section as the Scope instead, we would only have access to the Data Elements in that Data Section. This would not include the "Total Income" Data Field, which is what we're trying to populate here. The Trigger property controls the condition determining whether or not the action executed by the Data Rule is taken or not. The Trigger can be a .NET, LINQ or lambda expression. If left blank, as is the case here, the Trigger defaults to "true". In effect, the Data Rule will always execute its action. We will revisit this Trigger soon. The True Action property determines what happens if the Trigger evaluates to "true". Or put another way, if the Data Rule's condition is met. There are many possible actions detailed in the Actions section of this article. Each action type has its own set of configuration properties. In this case, we have selected Calculate Value. This action type's configuration is very similar to how a Data Field's Calculated Value property is configured. The Target Field property controls what Data Field (or Data Columns in the case of cells in a Data Table) is affected by the action. The Value Expression property controls what populates the Target Field's Data Field. It is what is calculated (or maybe better put how it's calculated). In this case we're using the exact same expression to add up the "Fed Wages" Data Fields in each of the "W2 Info" Data Section's sections as discussed above.

Data Rules are executed by the Apply Rules activity. After data is extracted by the Extract activity, any Data Rule referenced by the Apply Rules activity will alter the document's index data according to the Data Rule's configuration. You can test the Data Rule's results in Grooper Design Studio when the Data Rule is selected in the node tree. This will help you verify its configuration, giving you a preview of what would happen if that Data Rule was executed by the Apply Rules activity. Press the "Test Rule" button to execute the selected Data Rule. You can see in our case, the Calculate Value action is executed, using the Value Expression W2_Info.SumOf("Fed Wages") to populate the Target Field, "Total Income". Effectively, this adds up all the wage values for each W-2 in the document and returns the result to the "Total Income" Data Field. ⚠ Data Rules differ from expressions configured on Data Field (or Data Column) objects in one major way. Index data for documents must be extracted before executing a Data Rule. Documents must be processed by an Extract activity before testing a Data Rule or running the Apply Rules activity. The Data Model's results must be present and stored on the document before the Data Rule can manipulate them. Expressions configured on the Data Field (or Data Column) objects themselves execute during extraction when the Extract activity runs.

The Trigger

The Trigger property serves the purpose of establishing the condition that must be met in order for the Data Rule's action to be taken. These triggering conditions are also set using expressions. These expressions must return a Boolean "true" or "false" value. If the Trigger expression evaluates to "true", the True Action configuration is executed. If the Trigger expression returns "false", the False Action configured is executed (If it is configured. If left blank, no further action will be taken.)

In our case, something is wrong with our documents if the W2 forms have more than one social security number. Individuals should only have one social security number. If we added up all the wages for multiple W-2s with mismatched social security numbers, we would not be adding up the total income for an individual correctly. We'd end up with inaccurate data. This is the case for this document. It is comprised of two W-2 froms. For the first one, in the first section, the social security number extracted is "987‑56‑4321" For the second one, in the second section, the social security number extracted is "987‑65‑4321"

Luckily, there's an expression we could use to determine if our "W2 Info" Data Section has multiple social security numbers in its sections. This is a good opportunity for a LINQ expression. LINQ (or Language INtegrated Query) expressions are particularly helpful when navigating a Data Model's hierarchical structure to pull information from Data Sections and Data Tables.

Writer's Note: You aren't limited to just LINQ expressions for the Trigger. You can use any expression that returns a Boolean value, including standard.NET expressions, LINQ expressions, and lambda expressions. A LINQ expression just works well for this particular example.

(From sec In W2_Info Select sec.Employee_SSN).Distinct().Count() = 1

Let's break down this expression to understand what's going on.

(From sec In W2_Info Select sec.Employee_SSN).Distinct().Count() = 1

This is the "LINQ-iest" part of the expression. It's querying the extracted instances of our Data Model's objects, to return multiple results.

LINQ expressions always start with From (This indicates the data source from where are you querying the data). Next we declare a type variable we've named sec. We'll use later in the expression to return multiple instances of a Data Field in a Data Section (What you name it doesn't matter, just that you use the same name when you reference the variable later on in the query). The In clause determines the query's scope. We're looking for the "Employee SSN" Data Field in the "W2 Info" Data Section. The "W2 Info" Data Section is our scope. In W2_Info will only query instances (the results of its children Data Fields) in the sections produced by the "W2 Info" Data Section. The Select clause determines what values the query returns (or "selects"). We want information about the "Employee SSN" Data Fields. So, we've entered sec.Employee_SSN. Note, we've referenced the variable we declared at the start of the query, sec to do this.

Now the expression has some information it can work with. In this case, the social security numbers for each W-2 in the document (as returned by the "Employee SSN" Data Field for each section produced by the "W2 Info" Data Section).

(From sec In W2_Info Select sec.Employee_SSN).Distinct().Count() = 1

This part of the expression is counting the number of distinct values returned by the query (Technically, the .Distinct() expression is returning a subset of distinct values in the query's results. Then, the .Count() expression is counting the values in that subset). If the social security number is the same for each W2, there's only one distinct value. This should evaluate to "1". If not, it will be a larger number.

(From sec In W2_Info Select sec.Employee_SSN).Distinct().Count() = 1

This is just an equivalency argument to give us a Boolean "true" or "false" value. If the left side of the argument (the expression (From sec In W2_Info Select sec.Employee_SSN).Distinct().Count()) counts a single unique social security number in each section is equivalent to the right side of the argument (i.e. "1 = 1") it will return "true", otherwise "false".

If we use this expression as the Data Rule's Trigger, it will conditionally execute the True Action configured above only if it evaluates as true. Effectively, it will only add up all the wages for each W-2 only when the social security numbers for each W-2 are the same. Using the Trigger property, we've entered the LINQ expression described above. (From sec In W2_Info Select sec.Employee_SSN).Distinct().Count() = 1 If we press the "Test Rule" button to execute the Data Rule, it will only apply the rule's 'True Action configuration if the Trigger Expression is true. As you can see here, this document extracted multiple different social security numbers. So, nothing happens. The "Total Income" Data Field remains blank.

However, in the case of this document, its three W-2 forms do have the same social security number. So the "Employee SSN" field is the same for all three section instances of the "W2 Info" Data Section. Therefore, the Trigger expression returns "true". The True Action then executes, which is set to Calculate Value populating the Target Field "Total Income" with the results of the Value Expression W2_Info.SumOf("Fed Wages") In other words, the three "Fed Wages" Data Field results are added together and returned to the "Total Income" Data Field. Conditional logic!

Actions

Once a Data Rule is triggered, what happens next is determined by the True Action and False Action properties. When the Trigger expression evaluates to true, the True Action is executed. When the Trigger expression returns false, the False Action is executed. This determines what action is taken once the trigger condition is met or not met.

This can be one of six choices:

• Calculate Value
• Raise Issue
• Clear Item
• Copy Item
• Parse Value
• Action List

Each action has its own configuration to execute the action, detailed below.

Calculate ValueRaise IssueClear ItemCopy ItemParse ValueAction List

Calculate Value

The Calculate Value action will use a .NET, LINQ or lambda expression to populate a field with the expression's result. The possibilities here are as endless as the capabilities of these expressions. We can perform mathematical operations on numerical data. We can concatenate multiple string fields. We can perform incremental additions to date values. The Calculate Value action allows you to use any configurable expression to manipulate extracted data into a desired result. We've already seen one example of the Calculate Value action in the section of this article above. But let's look at another one.

In this example, we have a fairly simple report detailing costs of intangible services related to an oil drilling operation. As is necessary prior to the execution of a Data Rule, these documents have already been classified and their data has been extracted. In other words, they have been processed by the Classify activity (according to how the "Data Rules - Intangibles Table" Content Model is configured) and then by the Extract activity (using the data hierarchy and extraction configuration set on its associated Data Model). All of the table information has already been extracted, including the "Total" column which adds up the "Dry Hole" and "Completion" costs for each row. However, we might want to know the total cost associated with this table. We will use the Calculate Value action to populate this "Grand Total" Data Field.
We have added a Data Rule to the Local Resources folder of this Content Model. The Scope is set to Content Model's Data Model The Scope property determines what Data Elements in the Data Model's hierarchy the Data Rule has access to. In this case, we need a fairly broad scope. We need access to the Data Table object "Intangibles Table" and the Data Field object "Grand Total". We would not in this case want to scope down to the Data Table, for example. Then, we would only have access to its child Data Elements, namely its Data Column objects. The "Grand Total" Data Field lies outside the "Intangible Table" Data Table's scope, but both are within their parent Data Model's scope. Hence, we choose the parent Data Model. We're leaving the Trigger property blank for the time being. This means the Trigger will default to "true" and the True Action will always execute. We've chosen Calculate Value for the True Action. With Calculate Value selected, we must indicate which Data Field is populated. The whole point of this action is to manipulate data to come up with some new value. That value has to go somewhere. This property points the calculated value to the desired location in the Data Model. Here, we've selected the "Grand Total" Data Field. The Value Expression property determines how the value is calculated. The result of the expression entered here is what is ultimately returned. In our case, we're adding up all the values in the "Total" column of the table. The expression Intangibles_Table.SumOf("Total") does just that.
Press the "Test Rule" button to test the Data Rule's execution. The Calculate Value action's Value Expression configuration executes. In this case, the expression adds up all the values in the "Total" column. And, its resulting value populates the assigned Target Field.

A Word of Caution: Overwriting Results

There is one important thing to note about the Calculate Value action. The Value Expression's calculated value will overwrite any existing data in a field. For example, take this document. This document has a "Grand Total" value listed on the document. That means an extractor can find it and return it to a Data Field. Such is the case here. This value is the extracted value found in the document's text data. However, this isn't actually an accurate total. The document is wrong. The grand total of all the values in the "Total" column should add up to "$1,048,050.00" and not what we see here, "$1,111,000.00"
Press the "Test Rule" button to test the Data Rule's execution. See that the extracted value is replaced by the Value Expression's result.
Now, this may be what you want to do, but it may not be what you want to do. What if you don't want to overwrite the "Grand Total" Data Field it it's already populated? What if you only want to use the Calculate Value action to populate the field if it's blank? That's a great opportunity for a Trigger expression! If this were the case, we would only want to execute this Data Rule if the "Grand Total" Data Field is not there. We could use that as the condition to execute the True Action. All we need to do is figure out an expression that would evaluate to true or false if that's the case. The expression Grand_Total = 0 would return true if the field isn't populated and false if it was. We configure the conditional execution of the Data Rule using the Trigger property. Here, we've used the expression Grand_Total = 0 When the Data Rule executes, it will only apply the True Action if that Trigger expression is true. In this case, it is false. There's already a number in the "Grand Total" Data Field that is not "0". With no False Action configured, nothing happens. The True Action is not applied. The data remains intact.

Back to the top

Raise Issue

The Raise Issue action is useful for data validation. You may want to ensure two fields add up to a third field. You may want to ensure a date on the document is a date in the past or within a day range in the future. You may want to check if two fields are equal to each other. This is the realm of data validation. The Raise Issue action can log information in an issue log if conditions like these are not met.

The Raise Issue action will work in concert with the Trigger expression to log issues. If the Trigger expression returns true, and the Raise Issue action is selected as the True Action it will log a defined message in an issue log. You optionally have the capability to add a message category for issue message as well.

In this example, we have a fairly simple report detailing costs of intangible services related to an oil drilling operation. We expect the "Total" column to be the cells in the "Dry Hole" and "Completion" columns added together for each row. We will use the Raise Issue action to verify this. This document has some problems. The extracted numbers for the highlighted rows do not add up. The "Dry Hole" values and the "Completion" values to not add up to the "Total" column's values. We will configure this Data Rule with the Raise Issue action to validate this documents data, logging any issues where the Trigger expression we configure returns as true. In this case we will check if the "Dry Hole" column value added with the "Completion" column value does not equal the "Total" column's value for each row.
You always must assign the Scope for the Data Rule. Here, we've elected to scope down to the level of the "Intangibles Table" Data Table object in the Data Model. Because we're accessing individual cell results in the Data Table's children Data Columns, this scope provides us the necessary increased to check if the individual results on each row add up correctly. The Trigger expression will determine if an issue is raised or not. Using Raise Issue as a True Action, we need this expression to return true if there is a problem. In this case, the following expression will return true if the "Dry Hole" Data Column and "Completion" Data Column values to not add up to the "Total" Data Column's value in a row in the table: Dry_Hole + Completion <> Total FYI: <> is the inequality operator for .NET expressions. The True Action is set to Raise Issue logging issues when the Trigger evaluates true. As far as configuration goes, the Raise Issue action's Message property must be configured. Some message has to go in the issue log to identify it. This is also an expression based message, returning a string value. For simple messages, just type what you want in quotation marks (We'll talk about not-so-simple messages soon). Here, we've entered "Wrong Total", which will simply output "Wrong Total" as the log message.
When configuring a Data Rule, the "Diagnostics" tab will give you some more information on what's going on. Press the "Test Rule" button to test the Data Rule's execution. Navigate to the "Diagnostics" tab. Select the "Execution Log" to get some information on the Data Rule's execution. Since we are at a Data Table object's scope, the Data Rule's Trigger expression is checked row by row. For rows like this one, the "Dry Hole" column's value added with the "Completion" column's value adds up to the "Total" column's value correctly. The Trigger expression therefore returns false, and nothing happens. For rows like this one, the "Dry Hole" column's value added with the "Completion" column's value does not add up to the "Total" column's value correctly. The Trigger expression therefore returns true, and the Raise Issue action is applied, returning the message we configured "Wrong Total".
The "Issue Log" diagnostic will show you the issue log the Raise Issue action generates. Each issue the Data Rule locates will be added to the log. The name of the Data Rule, the offending Data Element on which the issue is raised, and the Log Message will be recorded. Optionally, if configured, the Category will be listed here as well. You may notice the message "Wrong Total" is a little generic. It doesn't give us much information about why the issue was raised. This is why the Log Message property is expression based. It allows you to access some additional information to populate the error message.
We could use an expression like this for our Log Message instead. This strings together some information from the columns in the row in which the issue is raised. The resulting message, has some more information than just a generic error message. Namely, it shows us the row's code value and why the issue was raised (ie x + y does not equal z) This could help us more easily identify which row has the invalid data. Note: The Log Message must evaluate to a string value. This is why we've used the Dry_Hole.ToString instead of just Dry_Hole, for example. The "Dry Hole" Data Column is configured to return decimal values, not string values. The .ToString portion of the expression just converts the decimal value to a string so it can be used in the message.

Clear Item

The Clear Item action will clear the data in a Data Field if the Trigger condition is met. Clear Item will also clear a Data Column's data if a Data Table is selected as the Scope. This can provide Grooper users a method of removing data from a field or table column if certain conditions are met. For instance, if you know the data is invalid based of the Trigger expression's true/false evaluation, you may prefer to remove the index data rather than keep the invalid data. This could also be a method of redacting sensitive index data after it is exported to a secure database. (Note: For complete redaction, you would probably also want to use the Redact activity to black-bar or white out the document's image and a Correct activity to remove the data from the document's text data as well)

For instance, we've seen already situations where this intangibles table's "Grand Total" field on the document does not actually add up to the summation of the "Total" column. We could use a Trigger expression to check if the "Total" column adds up to the "Grand Total" field and clear the extracted "Grand Total" Data Field if it does not add up correctly. For this document, the extracted values in the "Total" column are themselves accurate. However, the extracted "Grand Total" is inaccurate. It does not add up to the summation of all the values in the "Total" Data Column. We will configure this Data Rule to check if the values in the "Total" Data Column add up to the value in the "Grand Total" Data Field and clear it if it does not.
We have the Scope set to the Content Model's Data Model level. We need access to both the "Intangibles" Data Table and the "Grand Total" Data Field, both of which are children in the same level of the Data Model. For our Trigger condition, we've used an expression to check if the summation of the "Total" Data Column's values are not equal to the "Grand Total" Data Field. Intangibles_Table.SumOf("Total") <> Grand_Total FYI: <></nowiki> We've set the True Action property to Clear Item. All you need to configure for the Clear Item action is what field (or Data Column if scoped at a Data Table level) should be cleared. In this case, we've selected the "Grand Total" Data Field using the Element property.
Press the "Test Rule" button to test the Data Rule's execution. In this case, the Trigger expression evaluates to true. As the True Action, the Clear Item action is applied, and the "Grand Total" Data Field is cleared.

Copy Item

The Copy Item action will copy a field's value and paste it into another field. Optionally, the value can be moved as well (like a cut and paste operation). This can be useful for situations when you need to move data around in a Data Model's hierarchy. This provides an easy way to move a Data Field's value into a Data Section's single or multiple section instances.

For example, let's say you need to move the "Grand Total" Data Field into each section instance of a multi-section Data Section. We've configured this Data Section to return 11 section instances. Using the Copy Item action, we can copy a single Data Field. And then paste it into a Data Field for each of these 11 sections. We will configure this Data Rule to do this.
We've set the Data Rule's Scope property to the Content Model's Data Model. We need access to both the Data Model's child Data Field "Grand Total" and its child Data Section's own child Data Field "Grand Total Copy". The Content Model's Data Model includes all of these Data Elements. So it is the appropriate data hierarchy scope to use. We're leaving the Trigger property blank and going straight to the True Action property. We've set this to Copy Item. All you need to configure for the Copy Item action is the Data Field you're copying and what Data Field you're pasting the value to. The Source Element property is the Data Field whose value you are copying. In this case the "Grand Total" Data Field. The Target Element property is for the Data Field you are pasting the value into, populating it with the copied value. In this case the "Grand Total Copy" Data Field of the "Copy Section" Data Section.
Press the "Test Rule" button to test the Data Rule's execution. The Source Element is copied. And it is pasted into the Target Element. If copying into a Data Section, as is the case here, the copied Data Field will populated the targeted Data Field in each section instance established by the Data Section.

Parse Value

Action List