2.80:Paragraph Marking (Property)

Paragraph Marking alters the normal text data in a document by placing the carriage return and new line feed pairs at the end of each paragraph, instead of the end of each line. This allows users to break up a document's text flow into segments of paragraphs instead of segments of lines.

The Paragraph Marking property is enabled in the Preprocessing Options of the Pattern Editor. The are several paragraph detection settings to determine what qualifies as a paragraph.

About

Paragraph Marking is part of Grooper's Natural Language Processing (NLP) solution. Normally, after text data is obtained from the Recognize activity, the carriage return and new line feed characters \r\n are inserted at the end of each line. For structured documents, such as forms and reports, this is extremely useful. Information is typically conveyed line by line. These characters can be helpful anchors to locate and parse data.

For structured documents, like this report, important information is laid out line by line.

However, unstructured documents, such as contracts and correspondence, convey information differently. Instead of information being broken up line by line, it is broken up paragraph by paragraph.

For unstructured documents, like this contract, the same information is there. However, it is embedded in paragraphs.

Paragraph Marking allows Grooper to alter the \r\n pairs within the text data. Instead of being placed at the end of each line, they are placed at the end of each paragraph.

Instead of \r\n pairs marking the end of each line...	...they mark the end of each paragraph.

Enabling the Paragraph Marking property first detect paragraphs in a document then remove all \r\n pairs except the pair at the last line of the paragraph. Furthermore, the removed \r\n pairs are replaced with a single space character to keep the normal text flow intact. That way the word at the end of a line will have a space between it and the word at the beginning of the next line.

This allows you to do two very important things when extracting data from unstructured documents.

1. You can easily target data existing between a line break.

   
   

   Imagine you are trying to extract the address highlighted below. It starts on one line and ends on another.

   
   

   

   
   

   Under normal circumstances, there is a \r\n pair between the "18" and "18th" of "18 18th Street"

   
   

   

   
   

   If you're extracting data from a paragraph, there's a good chance it could be broken up between lines like this. Furthermore, there's no telling at what point in the string it jumps to the next line. You could potentially plan for this in your extractor's regular expression pattern, including the option for every space character to also be a \r\n pair. However, your pattern will get bulky and needlessly complicated.

   
   

   Instead, Paragraph Marking formats the text in a way that accounts for this. By removing the \r\n pairs at the end of each line, it turns the whole paragraph into one big line of text. This allows a simpler data extractor to return the address. The \r\n that would normally interfere with a match are no longer there.

   
   

   

2. Section stuff