2023:Data Context (Concept)

Data without context is meaningless.

About

Context is critical to understanding and modeling the relationships between pieces of information on a document. Without context, it’s impossible to distinguish one data element from another. Context helps us understand what data refers to or “means”.

This allows us to build an extraction logic using Data Type and Field Class extractors in order to build and populate a Data Model.

There are three fundamental data context relationships:

• Syntactic - Context given by the syntax of data.
• Semantic - Context given by the lexical content associated with the data.
• Spatial - Context given by where the data exists on the page, in relationship with other data.

Using the context these relationships provide allows us to understand how to target data with extractors.

Syntactic

All data follows some kind of syntax. Characters, their positions in a text string, and the order in which they appear inform what that data is. For example, US currency is easily identified by its syntax. Once you see a dollar sign ("$") followed by some digits ("0" through "9"), you instantly know that piece of information is referring to currency values.

Not only is the dollar sign itself important, but its position is important as well. If you see "$100", you instantly know you're talking about a hundred dollars. But, if you see "100$", it's more confusing. Maybe this refers to a currency value, but maybe it doesn't. It does not have that agreed upon understanding of currency values provided by the syntactic context of a dollar sign at the beginning of the numbers.

Similarly, dates follow some agreed upon syntactic structures.

	07/20/1969
	07.20.1969
	07-20-1969

You may not have thought a lot about it, but you actually know quite a bit just from the syntax of a date. First, you know months, dates, and years are separated by certain characters, the slashes, hyphens, and dots. You know (for certain parts of the world) the first series of digits refers to a month, the middle to a day, and the last to a year. This is purely understood by the syntax used, the kinds of characters used and their order in the text string.

The string below uses the same characters as the dates below but a different configuration. This is no longer a date! The syntactic context is no longer there.

20-1969/07

Even English language follows a syntax! It has an alphabet, a list of characters used ("A" through "Z"). It has certain rules, such as adding an "s" (or sometimes "es") to the end of a noun will (usually) make it a plural noun. The apostrophe is a special character used to combine two words like "do not" and "don't". How characters are used and in what way informs how we read the written word.

Syntactic Context and Extraction

The great thing about syntax is it is highly structured. This structure allows us to capture data based on the syntax's pattern. Take our example of the various dates (and the not date) above.

07/20/1969 ✔ 07.20.1969 ✔ 07‑20‑1969 ✔ 20‑1969/07 ✘

The three dates follow a pattern: Two digits, then a slash, hyphen or a dot, then two digits, then another slash, hyphen or dot, and last, four digits. The non-date string "20-1969/07" does not follow this pattern. Targeting these syntactic contexts is quite easy with regular expression pattern matching. If you know the syntax informing what a piece of data is, you can write a regex pattern to match that syntax. In this case, the regex pattern \d{2}[/-.]\d{2}[/-.]\d{4} would return the top three dates but not the "non-date" at the bottom.

Syntactic Context and Regular Expression

Syntactic Context and Regular Expression

Data Type and Data Format extractors use regular expression pattern matching to do just this. Here, the Value Pattern is set to a regular expression pattern matching dates: \d{2}[/-.]\d{2}[/-.]\d{4} This regex pattern matches all the dates using the three date syntaxes on the page ("##/##/####" "##.##.####" and "##-##-####"). The matches are highlighted in green in the Document Viewer and returning in the Results Viewer. The unmatching "non-date" doesn't follow any of the date syntaxes. So, it does not match the regular expression and is not returned.

Semantic

More often than not, syntax alone doesn't provide enough context to identify a value. For example, take a social security number. Typically, an SSN uses a standard syntax.

441‑12‑1234

We can easily identify this number as a social security number just from its syntax.

441121234

However, without the context the dashes provide, it's much more ambiguous. Maybe the number below is an SSN, but it could be something else. We don't have enough context to tell what this piece of data is.

In cases like this, or for data that does not have a unique syntax, something else needs to provide context. We simply need more information to determine what this data is.

By far one of the most common ways of identifying data on the page is with language. If we stick a label in front of that number, it becomes much easier to tell what it is.

Social Security Number: 441121234

Now we know we're talking about a social security number due to the semantic context of the text label. In a very simple way, because we know what the words mean, we know what the data means.

Not only can semantic context identify data, but it can distinguish it as well.

Order Date: 01/01/2020 Delivery Date: 01/15/2020 Due Date: 01/30/2020

The data here are all dates, which we can easily infer from the date syntax. However, each label further identifies each date and distinguishes them from one another. The order date from the delivery date from the due date. At the end of the day, how useful is it that what you're seeing is a date? You want to know what that date refers to. What makes it different from another date. What it means. That is what semantic context helps you do.

Semantic Context and Extraction

Semantic context can also be targeted by regular expression. A word or phrase providing context can be explicitly matched. You just need to know what word or phrase is providing the context!

Order Date: 01/01/2020 ✔ Delivery Date: 01/15/2020 ✘ Due Date: 01/30/2020 ✘

While the regular expression \d{2}[/.-]\d{2}[/.-]\d{4} would match all three dates, what if we only wanted to match the line containing the order date? The regular expression Order Date: \d{2}[/.-]\d{2}[/.-]\d{4} would match the only the line containing order date, but not any of the others. We use the semantic context of what the phrase "order date" means in combination with the syntactic context of how dates are patterned to narrow our result down to what we want.

Semantic Context and Regular ExpressionSemantic Context and Key-Value Pair Collation

Semantic Context and Regular Expression

For the example above, ultimately we just want the date. We don't want the whole string "Order Date: 01/01/2020". We just want the date value "01/01/2020". We still want to use the semantic context of our label, when it comes to identifying the date. But when it comes time to returning values, we don't actually want it. There's a lot of ways to do this in Grooper. The most basic way to do this is with the Prefix Pattern of Data Types and Data Formats. For simple cases, like this one, this approach can be very effective. Prefix Patterns match a regular expression before the Value Pattern in the text flow. Essentially, we can break up our longer pattern Order Date: \d{2}[/.-]\d{2}[/.-]\d{4} matching the full line into two. The value we want to match \d{2}[/.-]\d{2}[/.-]\d{4} will comprise the Value Pattern. This is what we want to return. The label before the value Order Date: will comprise the Prefix Pattern. This is the context for the value we want to return. The Value Pattern \d{2}[/.-]\d{2}[/.-]\d{4} The value is highlighted in green in the Document Viewer. The Prefix Pattern Order Date:\s The prefix is highlighted in blue in the Document Viewer. Only the Value Pattern's text results are returned.

Semantic Context and Key-Value Pair Collation

A Key-Value Pair extractor refers to a Data Type whose Collation property has been set to Key-Value Pair. This is easily the most common way semantic context is used to target data in Grooper.

A Key-Value Pair extractor consists of a parent Data Type and two child extractors: one for finding the "key" and one for finding the "value". These children can be Data Types or Data Formats The first child extractor is always the Key Extractor (regardless of its name). The second child extractor is always the (Paired) Value Extractor (regardless of its name).

The Key Extractor will locate the text label for a particular value, in our case using the semantic context of the "order date". Then, Grooper will look for a result returned by the (Paired) Value Extractor that is nearby. If one is found, it will pass that value up to the parent Data Type as the ultimately returned value.

The Key Extractor The Key Extractor must be the first child of the parent Data Type. Here, it is named "KEY - Order Date". It doesn't matter what the name is. It just needs to be the first extractor that fires. When you as a human tries to find the "order date" in this list of dates, you first look for the phrase "order date" to identify what date you're looking for. A Key-Value Pair does the exact same thing. First, find the key (the label in this case). Once you know where that is, you can find the value associated with it. For the value pattern, we explicitly matched the phrase Order Date This is the semantic context for the date we want. If the key is found, it returns in the Results List, like a typical extractor result.
The (Paired) Value Extractor The (Paired) Value Extractor must be the second child of the parent Data Type. Again, the name doesn't matter, just the order. For the value pattern, we match the actual value we want to return, here a regex to match dates \d{2}[/.-]\d{2}[/.-]\d{4} The extractor returns all values matching the regex pattern, like a typical extractor result. For course we want to narrow our results down to just the order date, "01/01/2020". That's coming up next.
The Collation Provider Collation Providers manipulate and re-order extraction results. As is, this extractor is returning four results, the phrase "order date" and the three dates. To make this extractor a Key-Value Pair the Collation property must be set to Key-Value Pair on the parent Data Type. Select the Collation property. Choose Key-Value Pair from the dropdown list.
The Key-Value Pair Layout The critical part of the Key-Value Pair setup is its Layout setting. This determines where the extractor "looks" for the value in relation to the key. This can be "Horizontal", "Vertical", or "Flow" "Horizontal" will look for values to the right or left of a key. "Vertical" will look for values below or above a key. "Flow" will look for values before or after a key in a text flow (Often used to find values in a sentence or paragraph). Here, the value is to the right of the key. So the Horizontal Layout property is appropriate. Enable this layout by changing the property from Disabled to Enabled This will collate the results using the Key-Value Pair Collation Provider. The closest date to the right of the key is returned. In the Document Viewer, the Key Extractor's result is outlined in blue and the returned value is highlighted in green. Note: A Key-Value Pair extractor will always return a maximum of one result for each key located.

FYI

The Key and (Paired) Value Extractors can also be the parent Data Type's internal Pattern extractor results as well as a Referenced Extractor. However, there are some specific rules for which counts as the "key" and which counts as the "value" when you use these properties instead of child extractor objects. It all matters what value is returned "first". The "key" extractor must always execute first in the order of operations. The basic order of extractor execution is this: Pattern > Children > References If an internal Pattern on the parent Data Type is set, it will always be the Key Extractor. The (Paired) Value Extractor can be either a child extractor or a referenced extractor at that point. If only referenced extractors are used, the first in the list will be the Key Extractor and the second will be the (Paired) Value Extractor. If one child extractor and one referenced extractor is used, the child extractor will be the Key Extractor and the referenced extractor will be the (Paired) Value Extractor. However, to avoid confusion, most users will use two child extractors, even if one or both of those children are Data Types that reference other extractors.