2.80:Paragraph Marking (Property)

From Grooper Wiki

Paragraph Marking alters the normal text data in a document by placing the carriage return and new line feed pairs at the end of each paragraph, instead of the end of each line. This allows users to break up a document's text flow into segments of paragraphs instead of segments of lines.

The Paragraph Marking property is enabled in the Preprocessing Options of the Pattern Editor. The are several paragraph detection settings to determine what qualifies as a paragraph.

About

Paragraph Marking is part of Grooper's Natural Language Processing (NLP) solution. Normally, after text data is obtained from the Recognize activity, the carriage return and new line feed characters \r\n are inserted at the end of each line. For structured documents, such as forms and reports, this is extremely useful. Information is typically conveyed line by line. These characters can be helpful anchors to locate and parse data.

For structured documents, like this report, important information is laid out line by line.

However, unstructured documents, such as contracts and correspondence, convey information differently. Instead of information being broken up line by line, it is broken up paragraph by paragraph.

For unstructured documents, like this contract, the same information is there. However, it is embedded in paragraphs.


Paragraph Marking allows Grooper to alter the \r\n pairs within the text data. Instead of being placed at the end of each line, they are placed at the end of each paragraph.

Instead of \r\n pairs marking the end of each line... ...they mark the end of each paragraph.

Enabling the Paragraph Marking property first detects paragraphs in a document, then removes all \r\n pairs except the pair at the last line of the paragraph. Furthermore, the removed \r\n pairs are replaced with a single space character to keep the normal text flow intact. That way the word at the end of a line will have a space between it and the word at the beginning of the next line.


This allows you to do two very important things when extracting data from unstructured documents.

1. You can easily target data existing between a line break.
Imagine you want to extract the highlighted address in this paragraph. It starts on one line and ends on another.
Under normal circumstances, there is a \r\n pair between the "18" and "18th" of "18 18th Street"
If you're extracting data from a paragraph, there's a good chance it could be broken up between lines like this. Furthermore, there's no telling at what point in the string it jumps to the next line. You could potentially plan for this in your extractor's regular expression pattern, including the option for every space character to also be a \r\n pair. However, your pattern will get bulky and needlessly complicated.
Instead, Paragraph Marking formats the text in a way that accounts for this. By removing the \r\n pairs at the end of each line, it turns the whole paragraph into one big line of text.

This allows a simpler data extractor to return the address. The \r\n that would normally interfere with a match are no longer there.

2. You can section out a document into paragraphs.
Imagine a series of contracts. All of them contain some information you want to extract from a certain clause.



If you can locate that clause's paragraph, you can limit extraction to match just the text within that paragraph, rather than the full document.



What Is a Paragraph?

Paragraph Marking detects paragraphs within a document in many different ways. In order to understand how Grooper does this, ask yourself "How do you know what a paragraph is?"

You don't even necessarily need to have text to distinguish between one paragraph and another. Even without real text, you can probably figure out what separates each of these "paragraphs" for these "documents".

The space between paragraphs is larger than the space between lines.
Paragraphs are often indented at the beginning.
Sometimes, bullets are used as paragraph markers.

Use Cases

Paragraph Marking is part of Grooper's Natural Language Processing (NLP) solution. It aids in data extraction from unstructured documents.

Imagine you have a collection of Non-Disclosure Agreements. In these contracts, there is a certain clause you're looking for with certain information. For example, a "Governing Law" clause that states which state has jurisdiction should there be a dispute. You'll need to find the paragraph (or paragraphs) that make up the clause and extract which state has jurisdiction from that clause. You'll use Paragraph Marking in a few ways to do this:

  1. To break up the contract into paragraphs instead of lines
  2. To use as a Value Extractor on a Field Class classifying the paragraph appropriately as a Governing Law clause.
  3. To use the processed text data (with \r\n pairs removed) to find the state holding jursdiction.

This is one of many examples. Any time you need to section a document into paragraphs and extract data from those paragraphs, you will likely take advantage of Paragraph Marking

Version Differences

Look forward to improvements in Paragraph Marking in version 2.9!

How To

Configuring Paragraph Marking is all about configuring how it detects paragraphs. So much so that Paragraph Marking is often referred to as paragraph detection. As mentioned above, this is done similarly to how you as a human reader breaks up paragraphs, mostly using space between paragraphs, indentations or other cues to mark a new paragraph. These configuration settings are detailed bellow.

Paragraph Marking is a Preprocessing Option for any regex pattern extractor in Grooper, including the Pattern property of Data Types, Data Formats, and any Internal Extractors

For this example, a Content Model named "Paragraph Marking" was added to the Node Tree with its Local Resources Folder and Data Model.

1. Add a Data Type to the Local Resources Folder and name it "Paragraphs".

2. Add a Data Format to the "Paragraphs" extractor and name it "Detection Settings".

3. For the value pattern, enter [^\r\n]+. This will match every character until a \r\n pair is found. For now, this captures every single line of text. We will use this pattern to visualize paragraph marking, to see if Grooper successfully removes all \r\n pairs except for the end of each paragraph. Notice without Paragraph Marking enabled, this pattern returns each line of text.



4. To enable Paragraph Marking switch to the Properties tab.

5. Expand the Preprocessing Options property.

6. Select the Paragraph Marking property and change it from Disabled to Enabled. (You may do this by either selecting Enabled from the dropdown list or double-clicking the Paragraph Marking property.




Now that we know how to turn Paragraph Marking on, the next tabs will look at how it detects paragraphs through its configurable properties.

7. Expand the Paragraph Marking property to see all its configurable properties.


The first thing Paragraph Marking does behind the scenes is look for lines of text after another that have the same (or nearly the same) line spacing between them. It is generally the case that a single paragraph will have the same amount of space between lines of text.

From there, Paragraph Marking's properties alter how these lines are grouped together and when a new paragraph should start.

The Maximum Line Spacing property sets the maximum height of line spacing inside a paragraph, in inches. Note, this includes the height of the white-space bellow a line as well as the text line's height itself.

In other words, Line Spacing = Text Height + Gap Height



So, if a line of text measures 0.2 inches and the gap bellow measures 0.1 inches, its line spacing height is 0.3 inches. Now, if the gap between the final line of text on one paragraph and starting line of another is 0.2 inches. That line spacing height is 0.4 inches. Setting the Maximum Line Spacing property higher than 0.4 inches would allow both paragraphs to be counted as one. Setting it bellow 0.4 inches will give paragraph detection an awareness that that gap is too big, and Paragraph Marking will start a new paragraph.

You can see this in our example in Grooper. All properties besides 'Maximum Line Spacing have been set to 0 or False to illustrate only how line spacing affects paragraph detection. The default property of 0.4 is too large. All paragraphs are detected as one big paragraph.



Changing that property to 0.3 will start a new paragraph if the line spacing height is larger than 0.3 inches. This will break up the text into its component paragraphs.


With a large document set however, there's no guarantee the maximum line spacing will always be the same literal height. Even within a single document, the line spacing between lines in a paragraph may change.

For example, this document switches between single and double spaced paragraphs. You can see the last paragraph is broken up in two paragraphs, when it should be one.



This is resolved by the Paragraph Spacing Ratio property. Instead of using a specific value for the height, it looks for the percentage by which the line spacing increases after a final line. Using the default of 125% marks a new paragraph when the line spacing height exceeds 25% the normal height.

Setting this property back to 125% will keep this paragraph as a single paragraph instead of two. First, the line spacing between the first two lines is measured. Then the line spacing between the third and fourth are found to be less than 125% of that height. So, a new paragraph is not created.



However, do note this can cause some issues. Grooper looks at the space between two lines to determine the baseline for the line spacing height. Then, it looks the the spacing between the second and third to determine if its within the set ratio.

First, this setting only impacts paragraphs that contain at least two lines of text. If a single line is detected as a paragraph, it won't have the context of the second line to figure out what the line spacing should be.

Second, if you have a paragraph with smaller spacing following a paragraph with larger spacing, the paragraph with smaller spacing can be consumed by the first. Since, the average line spacing was detected first the paragraphs with the larger line spacing, the paragraph with smaller spacing may not be separated by any space larger than the first paragraphs normal spacing. Furthermore, the space between the lines of the second paragraph are smaller then those of the first. So, a new paragraph won't be detected until it finds a gap exceeding the Paragraph Spacing Ratio of those lines.

You can see this in action below. The second paragraph gets combined with the first because the space between it and the third paragraph are the same as the space between the lines in the second paragraph. And a new paragraph isn't detected until the fourth paragraph, since the space between the lines of the third paragraph and the fourth does exceed the set ratio.



In an ideal world, there would be a little extra space between the second and third paragraph. However, the world of document processing is not always ideal. There is a solution for this particular case, demonstrated by the next property, Minimum Line Width.

One common feature of paragraphs is the last line tends to be smaller than the rest.

This is not always the case, but it can serve as another breaking point for where to stop and start a paragraph.

In our poorly spaced example, Paragraph Spacing Ratio caused the second and third paragraph to be detected as a single paragraph. However, the last line of the second paragraph is only about 1.75 inches long.

If we reset the Minimum Line Width to the previous default of 3, Paragraph Marking will start a new paragraph if the last line of a paragraph is less than 3 inches. This will give us the result we want.



However, the last line of a paragraph is not always the shortest line of the bunch. Setting this property too high can cause paragraph detection to break up single paragraphs into multiple lines or paragraphs.

You can see the results below setting the Minimum Line Width to 6.


Retrieved from "https://wiki.grooper.com/index.php?title=2.80:Paragraph_Marking_(Property)&oldid=2574"