Data Section - 2.90
Data Sections are Data Elements of a Data Model. They allow a document's content to be subdivided into smaller portions (or "sections") for further processing, yielding the extraction process higher efficiency and accuracy.
Often, they are used to extract repeating sections of a document. For example, if a document had several sections of data for different customers, a Data Section could be used to pull data for each customer. This is especially useful for situations where the data within the section is predictable, but the number of sections in the document is not (i.e. if one document has one customer's data listed but the next has five, the next has two, and so on and so on).
Data Sections can also be used to:
- Organize data from complex documents
- Make a hierarchical representation of a document's structure, or
- Reorder content from multiple columns on a page.
Data Sections may have, as its children:
- Data Fields
- Data Tables
- Their own Data Sections
Contents
About
Sometimes a Data Field by itself just doesn't cut it when it comes time to extract data. Data Fields are the smallest building blocks of your Data Models. They are designed to return a single piece of data. For example, a most report stlye documents will have a single date the report was made. A single "Report Date" Data Field is well suited for this data.
|
|||
This is exactly what Data Sections are for! Data Sections allow you to divide a document's content into smaller sections for further processing.
Data Sections subdivide the larger document into smaller data instances. Data instances are an encapsulation of text data within the document. The largest data instance would be the document itself. Individual pages would be smaller sub-instances of the document level data instance. If you want to execute an extractor on page and not the whole document, you effectively execute it on the page instance of the document instance. Data Sections allow Grooper users to define how the document is subdivided to execute an extractor on a section instance of the document instance. Rather than the Data Field (or other Data Element objects) executing against the whole document, it executes each data instance. It's like it creates smaller sub-documents or document chunks, ignoring all the text data outside of that chunk. Extractors used to populate Data Elements added to the Data Section will only execute against the text data contained in the Data Section. The rest of the document's text data is filtered out, narrowing the Data Elements' field of vision. You can even subdivide a Data Section's data instance with another Data Section. This way you can create a hierarchy of data instances by adding child Data Sections to parent Data Sections in a Data Model. The parent Data Section is a subdivision of the document's data instance. The child Data Section is a subdivision of the parent Data Section's data instance. A child Data Section of the child's Data Section would be a subdivision of the child's Data Section. It's like making a Russian nesting doll out of the document's text data. |
As with other Data Elements, Data Sections are created by adding them to a Data Model in a Content Model.
|
Section Extract Methods
How Grooper subdivides the document into the smaller data instances (or "section instances") is controlled by the Data Section's Extract Method property. Each Extract Method works a little differently to section out the document for subsequent extraction. They are as follows:
- Full Page - This method subdivides the document into full pages. You can use a page filter to define which page or pages establish the section instances (ie the first page, or the second and fourth pages, or the fifth through the last pages). You can also use an extractor to extract a page or pages where the extractor returns a result.
- Fixed - With this method, you establish the section instances by drawing a rectangular region on the document. Any text falling inside this rectangular zone forms the section instance. This method is useful for highly structured documents where you want to limit extraction to a specific area of a specific page of the document. This method will only ever return a single section instance.
- Divider - This method uses an extractor similar to the Split Collation Method to establish section instances. A Divider Extractor is used to anchor the sections to an extractible result. The results the extractor returns can be used as the beginning point of the section or ending point. For example, a section header line may be used to indicate where one section begins. If the next section also uses that same section header, another section would be established. Sections can also be established between the Divider Extractor's results or (less commonly) around the results.
- Geometric - This method uses a combination of extractors, positional adjustments, and line detection to establish rectangular regions for the section instances. Similar to the Fixed method, any text falling inside the rectangular zones forms the section instances. However, the Geometric method can produce multiple sections where the Fixed method only produces one. Furthermore, the Geometric method is always anchored to at least one extractor's result (the Main Extractor). The zone is expanded (or contracted) by adjusting the left, right, top and bottom edges of the zone using extractors or manually adjusting the length in inches or another unit. This method is useful for establishing sections from structured and semi-structured documents using columnar or atypical layouts.
- Simple - This method uses a single extractor to return the section instances. One section is created for each result the extractor returns. This method is only "simple" in that it uses a single extractor to return the section. The extractor used to populate the section instances can be as complex as you create it, using any configuration of a Data Type extractor with the multitude of possibilities to return instances using any of the Collation Providers available. This method is also commonly used in unstructured document processing using Field Class extractors to create sections out of targeted paragraphs in a document's text.
To choose the Extract Method
|
Fixed
In this example, we will demonstrate how to make a Data Section that returns a section for the highlighted portion of this document. This will limit the Data Section's Data Elements to return only data falling within this region. To accomplish this we will use the Fixed method. In many ways, this sectioning method is the most basic. You simply draw a rectangular box around the portion of the document you want to form the section. All the text falling within this rectangular region will form the Data Section's section instance. Furthermore, the Fixed method is the most basic in that only one section is established per document. |
Here, we have selected a Data Section with the Extract Method set to Fixed.
|
|
|
|
The Fixed extract method also requires you to indicate one which page the zone falls.
|
Grooper also gives you ways to verify the section instance (or instances) established by the Data Section. For more information on viewing the Data Sections section instances, visit the How To: Viewing the Section Instances section of this article.
For information on how to add Data Elements to a Data Section and how their extraction differs from standard full document extraction, visit the How To: Adding Data Elements to a Data Section section of this article.
Full Page
As its name implies, the Full Page section extraction method creates section instances out of full pages in a document. This can be useful to limit data extraction to a single page.
This can be as simple as indicating what page number you wish to create a section instance out of. You can also create multiple sections using the Full Page method by indicating multiple pages or a span of pages. This would create a section instance for every full page in the span (Note this means the Full Page section extraction method cannot create sections that span pages. You get a single full page per section instance. Top to bottom. No more, no less).
In this example, we will demonstrate how to create a Data Section returning the first page of this "Gross Production Monthly Tax Report" as the section instance.
Here, we have selected a Data Section with the Extract Method set to Full Page.
And that's it for the most basic Full Page section extraction method. In this case, a single section instance will be created encapsulating the first page of each document |
FYI | The Full Page section extraction method can also create multiple sections if multiple pages are listed in the Page Filter. Below are some examples of page filters you may use to create multiple sections. One section will be established for each page.
|
Very straightforward. However, this configuration assumes your documents are both highly structured where you know what your looking for is on a particular page (or pages) and the document's pages are in order.
What if this document was scanned in out of order? The first page would be last and last would be first. Using the method described above, a section instance would be created from the wrong page.
There is a potential solution using the Full Page method's Extractor property. This allows you to target a page or pages with an extractor. If the extractor produces a result on a page, a section instance will be created out of the full page. This could be a referenced extractor (a Data Type or less commonly a Field Class) or an internal text pattern local to the Extractor property.
We can easily solve the page order problem described above with a simple extractor looking for the document's title "Gross Production Monthly Tax Report". Configured correctly, the regex will only match the actual first page of the document, even if its out of order.
|
The Divider, Geometric, and Simple section extraction methods get into the "meat and potatoes" functionality of Data Sections. As well as being able to target single-instance sections, they have increased functionality to target multiple repeating sections containing the same data. We will target the repeating sections on this "Gross Production Monthly Tax Report" (henceforth called "Reporting Sections"). These sections can be targeted in different ways using any of these three section extraction methods. Their configurations are a little different, but at the end of the day, each of them can easily intuit the five reporting sections and their general boundaries. DividerThe Divider method uses a functionality similar (identical even) to how a Data Type using the Split Collation Provider returns results to establish the section instances. There are two parts of the Divider method's operation.
For example, each section here starts with a letter followed by "8. Production Unit Number". Once you see that piece of text, you can keep on going down the document until you see another letter and "8. Production Unit Number" at which point a new section should start. We will use an extractor to match this text anchoring the start of each section and the Begin split position to indicate this text indicates where a section begins. |
Here, we have selected a Data Section with the Extract Method set to Divider.
|
|
This is the pattern we've configured for our Divider Extractor.
|
Now that we have the Divider Extractor configured, we need to decide how to configure the Split Position property. This can be one of four options:
- Begin - The extractor's result marks the beginning of each section. Starting at the extractors result, the section will consume all text in the document until the next result.
- End - The extractor's result marks the end of each section. Starting at the top of the document, the section will consume all text until the extractors result. The next section will consume all text after until the next result.
- Between - This split position requires at least two results from the extractor. The section consumes all text between the extractor's results. Importantly, the Begin and End split positions are inclusive of the extractor's result where Between is exclusive. That means the extractor's results will not be included in the section when using the Between split position.
- Around - This is a less common split position. It will create sections on either side of the extractor's results. Imagine you had a single result extracting a line in the dead middle of the document. You would end up with two sections, one encapsulating everything before the result and one everything after. If your extractor produces two results, you'll end up with three sections: one from the top of the document up to the start of the first result, one from the end of the first result to the start of the second result, and one from the end of the second result to the end of the document. You will always end up with the number of results your extractor returns plus one. The Around split position is also exclusive. Results will be excluded from the sections produced.
|
|
For more information on viewing the Data Sections section instances, visit the How To: Viewing the Section Instances section of this article. |
|
Is this a big deal? Maybe. Maybe not. This will largely depend on your particular documents and their structure. In this case, it's probably not going to have much of an impact. We can still extract the data in the actual section. Having the extra text data doesn't really do anything one way or the other for us. However, if it does impact your extraction results, you may need to use a different Split Position and/or Divider Extractor configuration or even section extraction method. |
The Divider, Geometric, and Simple section extraction methods get into the "meat and potatoes" functionality of Data Sections. As well as being able to target single-instance sections, they have increased functionality to target multiple repeating sections containing the same data. We will target the repeating sections on this "Gross Production Monthly Tax Report" (henceforth called "Reporting Sections"). These sections can be targeted in different ways using any of these three section extraction methods. Their configurations are a little different, but at the end of the day, each of them can easily intuit the five reporting sections and their general boundaries. GeometricThe Geometric method uses a variety of tools in Grooper's toolbox to create a logical rectangular region around the sections you wish to extract. Any and all text falling within these rectangular zones form the section instances. The basic process to draw these zones is two-fold.
Optionally, if your sections are encased in lines, such as they are on this document, you can use Grooper's line detection to expand the zone's boundaries to nearby lines. |
Here, we have selected a Data Section with the Extract Method set to Geometric.
|
|
We will expand out the zones' sizes using the various Adjustment properties. There are three options for the borders' Adjustments:
All of these options also have a Manual Adjustment property to expand or contract the border even further by a set length in inches or another unit.
This is better, but we still need to expand out the sections' bottom edges. |
|
To expand the bottom edges, we're going to use the Anchor Adjustment option.
|
|
We just need the bottom edge to be a little bit larger to fully capture each of these sections. Using the Manual Adjustment property, we can manually expand the bottom edge by a set length after its been expanded to the anchor's location.
|
The Divider, Geometric, and Simple section extraction methods get into the "meat and potatoes" functionality of Data Sections. As well as being able to target single-instance sections, they have increased functionality to target multiple repeating sections containing the same data. We will target the repeating sections on this "Gross Production Monthly Tax Report" (henceforth called "Reporting Sections"). These sections can be targeted in different ways using any of these three section extraction methods. Their configurations are a little different, but at the end of the day, each of them can easily intuit the five reporting sections and their general boundaries. SimpleA lot of the time, you don't need anything fancier than an extractor's results to encapsulate sections on a document. The Simple method simply uses the results of an extractor (either a Data Type, a Field Class or an Internal regex pattern) to form a Data Section's section instances. Using this method, each result returned by the extractor will create one section instance. This ends up giving you a lot of flexibility in how you return section instances based on your documents need. Data Type extractors give you a host of different ways to return and manipulate text data through its Collation Providers. If you can configure a Data Type to return the full text of each section you want, you've done the job of finding and returning the sections. The Data Section can just use the Data Type's results to form each section instance. Furthermore, in unstructured document processing, you may want to use full paragraphs as a section. The Field Class extractor is one of the tools in Grooper's natural language processing toolkit. It centers around using machine learning to return data, such as training the extractor to find certain types of paragraphs. Once you've found the paragraph you're looking for, you can through it in a Data Section using the Field Class as the Simple method's extractor and parse through it to find the Data Elements you want. You could actually capture all five of the sections on this "Gross Production Monthly Tax Report" with a single regex pattern. This means we can create a Data Section using the Simple Extract Method using this regex pattern to populate all five sections instances. |
[A-Z] 8. Production Unit Number.*? 17\. Taxable Volume.*? \d{1,3}(,\d{3}){0,2}\.\d{2}\r\n
Since this pattern successfully returns each section, we can use it to configure a Data Section! |
|
Here, we have selected a Data Section with the Extract Method set to Simple.
And... That's it! This Data Section will now successfully return five section instances from the five results the extractor returns, one for each reporting section on the document. |
How To
Viewing the Section Instances
When configuring any of the section extraction methods, it can be useful to verify what section instances are created. Where they are physically on the document and what text data they contain. The "Instance View" tab is extremely helpful when testing out your Data Section configurations to do just this.
|
|
|
Adding Data Elements to a Data Section
Data Sections can have Data Fields, Data Tables, and even Data Sections as their child Data Elements. You add these Data Elements to the Data Section just like you do with a Data Model For this example, we will add a Data Field for the "Company Reporting Number" located in the Data Section we created.
|
|
|
|
We're going use a very general pattern to illustrate this point. You can see here the results of the configured Text Pattern extractor used for the Data Field
|
|
Now that the Data Field is added and configured to return a result, we can verify it only executes against the section instance created by the Data Section.
|