2.72:IP Profile (Object)

From Grooper Wiki

This article is about an older version of Grooper.

Information may be out of date and UI elements may have changed.

202520232.72

perm_media IP Profile node objects detail the operations and parameters for image enhancement and cleanup. These operations improve the accuracy of further processing steps, like the Recognize and Classify Activities.

These operations generally fall into three categories:

  1. Archival Adjustments - These are permanent adjustments to the exported document's image.
    • Permanent image adjustments are performed when an IP Profile is executed during the Image Processing activity.
  2. OCR Cleanup - Image cleanup can dramatically improve OCR results.
    • However, they can also drastically alter the document's image. Image adjustments are temporarily applied to a document prior to OCR when an IP Profile is executed during the Recognize activity. This is useful for non-destructive image clean up to improve OCR results, keeping the document's pages as their original image to preserve their archival images upon export.
  3. Layout Data Collection - This includes visual information used for data extraction purposes (such as table line locations, barcode information, OMR checkbox states) as well as image features used for Visual classification.
    • Layout Data can be collected either during the Image Processing or the Recognize activities.

Permanent vs. Temporary Image Processing

The Image Processing (Activity) activity permanently alters a document's image by applying an IP Profile. However, it is possible to temporarily clean up document images and revert back to the original document image. This is done during the Recognize activity.

For example, you may have a document where table lines are getting in the way of accurate OCR. However, if you remove these lines during the Image Processing activity, they will be permanently removed, making it difficult to review the documents in Data Review and changing the archival image stored later to something that no longer looks like the original document.

Instead, you can use an OCR Profile which references an IP Profile that has a Line Removal step during Recognize. The image will be temporarily changed according to the IP Profile. Then, OCR will run on the altered image. Last, the image will revert back to its original form, retaining the OCR results from the pre-processed image as well as the original image.

  • Furthermore, any image based data targeted by the IP Profile (such as the table line locations for this example) will still be saved to the Batch Page for later use.

For more information on both permanent and temporary image processing as a concept, visit the Image Processing (Concept) article.

Anatomy of the IP Profile Tab

Upon selecting an IP Profile in the Node Tree, you will edit it using the IP Profile tab. This is what the screen will look like. As you can see, there are several windows that make up this screen.


Here IP Steps in the profile are listed, selected, and added. IP Profiles are a sequential list of IP Steps, each one performing a image processing operation called an IP Command. This IP Profile is very simple. It only has one IP Step, using the Auto Deskew IP Command.

IP Steps are added to the list using the "Add" button, deleted using the "Delete" button, and you can change the order in which they process using the "Move Up" and "Move Down" buttons.

Here, you can select a Test Batch to help you configure your IP Profile. All alterations to the documents in the Test Batch are done in memory when configuring an IP Profile. They will retain their original form unless the IP Profile is applied using the Image Processing activity.


Here, you will see a list of processing results for each step in the IP Profile. Each step will be listed, with the time it took that step to run, whether or not the image was modified, and if the image was flagged by the step. It also contains an "End Result" containing the sum total run time for the whole IP Profile, it the profile modified the image, and if it has a flag by the end of it.


Each IP Command has its own set of configurable properties. Here, you can adjust them as needed to fit the demands of your document set.



Furthermore, using the "Selected Step" tab, you can create some conditional logic around if and when to apply certain IP Steps, using a snippet of Microsoft.Net code. This is done using the "Should Submit Expression" and "Next Step Expression" properties.



Lastly, you can use the "IP Profile" tab to add a description for the profile for other users to get more information as to what the profile does and is used for.


The Diagnostics Panel is extremely helpful when configuring IP Command Properties and verifying steps are processing a document as intended. It contains a number of images for each IP Step related to how its IP Command is altering the image, including a before "Input Image" and after "Output Image"


Last but not least is the Document Viewer. This allows you to view the document selected in the Batch Selector. This window will also show you the selected image in the Diagnostics Panel.


How To

Create a new IP Profile

Before you create an IP Profile, you will likely want a Test Batch to verify its results. Be sure to create a Test Batch before creating an IP Profile

Add a New IP Profile

IP Profiles may be created and stored in a Content Model's local resources folder or in the IP Profiles folder in the Node Tree (which is found in the Global Resources folder). However, the most common place to create an IP Profile is in the IP Profiles folder.

  1. Navigate to the "IP Profiles" folder via this path in the Node Tree: Root Node > Global Resources > IP Profiles
  2. Right click the "IP Profiles folder and mouse over "Add" and select "IP Profile..."
  3. Name the IP Profile whatever you like and select "OK" to create it.
    • For this exercise we just named ours "IP Profile Example"

This will create a blank IP Profile in the IP Profiles folder.

Before going any further, the first thing you will want to do is select a Test Batch.

  2. Using the Batch selector, expand the dropdown window to select a Test Batch.
  3. Select a Test Batch from the dropdown window.
    • The one we've selected here is named "00 Source Set".

This will give you something to work with when testing out your IP Profile. You can use the "Batch Viewer" window to select a document from the Test Batch and view that document in the "Document Viewer " window. Once we start adding IP Steps to modify the image, you can visually verify the results in the Document Viewer.

Add IP Steps

IP Steps are the individual elements of an IP Profile. The IP Profile will execute each step, one after other, altering the image according to whatever IP Command the IP Step uses.

  1. To add a new step, press the "Add" button. This will bring up a menu of all the different IP Commands available, organized by category.
  2. Choose the IP Command you wish to use.
    • For this example, we are adding an "Auto Deskew" step, which is part of the "Image Transforms" category. This will accomplish two things. One, it will straighten up the image, making the final output document more visually appealing (an Archival Adjustment). Two, we will get better OCR results (an OCR Cleanup). All the text will be nicely aligned line by line, instead of potentially jumbled on different lines.

Verify the Results

  1. To verify the results of the IP Profile, press the "Execute" button.

In the "Document Viewer" window, you can see the image was de-skewed by the single IP Step in our IP Profile. Now it's nice and straight, which is both visually appealing (an Archival Adjustment) and give us better OCR results (an OCR Cleanup). Notice a couple other things happened after we hit that "Execute" button as well.

  1. You will also see a set of diagnostic images and an "Execution Log" text file for each IP Step in the IP Profile, seen in the highlighted "Diagnostics Panel" window. These images and logs can be useful for configuring the step's properties.
    • There will also be a before "Input Image" and after "Output Image" both for each individual step as well as the entire IP Profile.
  2. The "Processing Results" window will give you some additional information as well. For each step, it will let you know, the time (in milliseconds) it took for the step to run, whether or not the image was modified, and if the IP Step threw any flags on the page, indicating an error.
    • This information is present for the whole IP Profile as well, indicated by the "END RESULT". If we had more steps in this IP Profile, it would have the total run time, whether any of the IP Steps altered the image, and whether or not a flag is present.


! Pressing the "Execute" button does not actually modify the selected document.

All alterations to the documents in the Test Batch are done in memory when configuring an IP Profile. Furthermore, any time you navigate to another document the IP Profile will "execute" on the temp batch (In other words, there's no need to press the Execute button unless you want to verify the results of the IP Profile on the currently selected document.


FYI If you do want to permanently apply the IP Profile to a page, you can do so in an ad hoc manner by selecting a page and pressing the "Save Processed Page" button. This will permanently apply the IP Profile's steps to the page in the Test Batch.

Sample Configuration of an IP Profile for Permanent Image Processing

Permanent IP makes alterations to the archival version of the exported document. As permanent image processing is, after all, permanent, you must be careful about which commands you use. Most often, IP Profiles for Permanent IP are fairly small, performing only moderate adjustments to the image. More drastic alterations that improve OCR but dramatically alter the image are left for Temporary IP Profiles.

Before anything, you need a sample document set of the kinds of documents targeted by your IP Profile. Once you have these documents ready in a Test Batch, it's a good idea to evaluate them, getting an idea about the kinds of issues the document set has. Below are the documents we will be looking at and some of the issues involved with them.

One of the most common issues resolved by a permanently applied IP Profile is a skewed image. Not only will you improve the image visually for human readers, but deskewing an image is also critically important for OCR to run correctly. This document does not look too bad, but the background isn't totally white. Some of the paper's texture came through when the document was scanned. A lot of scanners have built in image processing functions that will clean this up, but this functionality is somewhat basic and can cause problems down the road. Grooper's IP capabilities are much more robust, and there may be cases where not cleaning up the image actually produces a better result (more on that later).


This image is upside down. As a rule of thumb, if a document is hard for a human to read, it'll be hard for an OCR Reader to recognize the document's characters. There's not too much to fix on this document, but keep in mind an IP Profile will be applied to all documents. Part of the IP Profile creation will be to make sure what positively impacts one document doesn't negatively impact another.


Border cropping is very common when permanently applying an IP Profile. Black borders around images are typically a result of scanning the image and not actually part of the document. Border Crop or Border Fill can get rid of that border, giving an image that is more representative of the actual document. All those black pixels around the edge of the document are also most definitely getting in the way of OCR. However, you might have to ask yourself the question: Is the border part of the original document? If so should we keep the border for the archival export of the document? Take a look at the transcript on the left. At first glance, this border looks like it's a border from scanning the image, but it may actually be part of the document.

The first step we will add is an "Auto Border Crop" command. This command will only crop an image if it detects a border in an established zone. So, only those documents with borders should be affected.

Press the "Add" button and select "Auto Border Crop" from the "Border Cleanup" category.



For our document, the basic settings did not alter the image. You can see this viewing the "Processing Results". The "Image Modified" column has a value of "False". Also there is no "Output Image" in the Diagnostics Panel. Furthermore, you can see clearly there is still a black border on the image in the Document Viewer.



This is where the step's property panel can come in very helpful. The border is quite larger than normal. Changing the "Border Region Size" to 0.75 in on all four sides will properly crop this document. The diagnostic images are useful when configuring any IP Step's properties. In the case of Auto Border Crop, you'll want to use the "Zoning" image to configure the zone where the border falls.

The Border Region zone is shown by the thin red rectangle. It encapsulates some, but not all of the border (from the edge of the document to the edge of the Border Region) using the default of 0.25 in.


By increasing the Border Region Size to 0.75 in you can see the zone now entirely encapsulates the border.


Now, you can see the step removes (most of) the border.



The remaining part of the border can be removed using the "Border Fill Settings" or adding an "Border Fill" step.

Next, we need to take care of the upside down image. This can be done with an "Auto Orient" step. This step will automatically detect the orientation of the image based on the text on the page. If the page is upright, Auto Orient will detect the text reads like normal and do nothing. If the command detects the text is upside down, it will re-orient the page, turning it right-side up.

Press the "Add" button and select "Auto Orient" in the "Image Transforms" category.



Upon execution, the upside down page has been righted.


Next we need to take care of the skewed text. For this, we will use the Auto Deskew command.

Press the "Add" button and choose "Auto Deskew" from the "Image Transforms" category.



This will resolve issues around skewed documents.


Further Decision Making

The previous three IP Commands are very commonly found in Permanent IP Profiles. Since they determine automatically if a page is skewed, has a border, or is oriented incorrectly, there is significantly less risk that these commands will negatively impact the rest of the document set (less risk, but not none!). For any other IP Commands, you have one major question to ask.

How much do you want to alter the original image?

Anything you do with a Permanent IP Profile will affect the final document you send out during export. Keep in mind, there are ways to temporarily apply an IP Profile to improve OCR Results later.

When making your choices on what other IP Commands to add to a Permanent IP Profile you should focus on altering the document for human readability without negatively impacting machine readability (i.e. OCR results).


For example, the "Brightness Contrast" Color Adjustment command can be very helpful to increase both the human and machine readability for some documents. It modifies the brightness or contrast of an image (or both).

Increasing the Brightness property to "10" and the Contrast property to "40" cleans up the first page of this document nicely. The background has been removed while preserving the text.


However, you be sure to verify it does not negatively impact other pages.

On the second page, this command also starts to remove the table lines on the page. Grooper can use the positions of these lines for certain data extraction methods, such as Table Extraction. However, if you get rid of them with a Permanent IP Profile, you won't be able to find those lines later.


Furthermore, the IP Profile runs on all documents in the batch. Be sure it isn't adversely affecting other documents.

This page is fairly light to begin with. While the typewritten text is still there, the handwritten text is starting to fade. OCR Engines may not do well recognizing handwritten text, but if you ever want a human to look at this document and be able to read the handwritten text, increasing the contrast is not well suited for this document.

For situations like this, you may need to find a happy middle ground.

The "Contrast Stretch" command is often used to help improve the image quality of documents. It works to normalize the image's contrast. It adjust the contrast so that the lightest pixels are turned pure white and darkest are turned pure black.

It doesn't do quite as good of a job as the "Brightness Contrast" command but it does brighten up the whites and darken the black parts of the image a bit.
And it does so without losing the handwritten text on this document.

Just as you need to think about how one command will adversely effect other documents in the set, you can take advantage of documents that are very different from others in the set.

This document is the only color document and has some significant issues with how it was scanned. Likely it's actually a picture taken of a screen. Portions of the image are light portions are dark. It's white balance is all off. Since all the other documents are black and white or greyscale, we can reliably use an "Auto White Balance" command to make it look a little better.
FYI There are some ways to leverage image based information to create conditional logic around what steps to execute in an IP Profile. Visit the Conditional IP section of this article for more information.

Make Adjustments

It's very rare when you make an IP Profile and everything works perfectly without doing some unit testing and adjusting some properties on a step or two. Take our "Auto Border Crop" step and our two documents with borders. As we configured the step, this is the results of our two transcripts.



Depending on what route we want to take, there's still some cleaning up on these borders we can do. The easiest thing to do at this point would be to add a "Border Fill" command to clean up the black edges on the left side of these documents.

Press the "Add" button and select "Border Fill" under the "Border Cleanup" category.



One of the main adjustments you will make when using a Border Fill command is setting the "Method" property. By default, it is set to "Exclusive". This means anything fully outside the border zone (seen in the image below in red) will be filled with the selected "Fill" color.



Note the red zone intersects the black border for this document. If we set the method to "Inclusive" it will include borders that overlap the border zone, dropping them out.



But what if we start looking at these documents and don't actually want to remove the border for the transcript on the left but do want to remove the border for the right?


Keep this border Remove this border


For this document set, there's a trick we can do using the properties of "Auto Border Crop" and "Border Fill" to do this. First we will configure "Auto Border Crop" to crop the blue transcript and not the brown one. Looking at the brown transcript, that border is actually part of the document. As such, there is a sliver of white pixels around the document. We can use the "Maximum Border Weight" property to only drop out perfectly solid borders.

Navigate to the "Auto Border Crop" property and change the "Maximum Border Weight" from 90% to 100%.



However, since the blue transcript does have a solid black border around the image, it does crop the image. Or at least it mostly does. There is a slight border on the left and right sides still, but we'll fix that next.



Now, if we go back down to our "Border Fill" command and adjust the "Border Region Size" to "15pt" these left and right borders will be totally outside the border zone. We can set the "Method" back to "Exclusive" now and these borders will be dropped out.



However, for the brown transcript, the border intersects the border zone. So, using the "Exclusive" method keeps the border from being dropped out.



We are left with two originally bordered images, one of which was removed and one of which was not.


Granted, this really only worked because of how these documents came into Grooper. We got lucky in that there was a slight amount of white pixels on each edge of the brown transcript, making the "border" not perfectly solid. That being said, a lot of how you configure Grooper's properties to target certain documents and not others is based of analyzing certain aspects of the documents. We wouldn't have even known to try this approach if we hadn't noticed that border on the brown transcript wasn't a true border.

Sample Configuration of an IP Profile for Temporary Image Processing

For temporary image processing, we don't need to be concerned with how this image will look upon export. We only need to concern ourselves with cleaning up the image to improve OCR results.

The general plan is (as much as possible) get rid of anything on the page that is not text. This way non-text artifacts on the page will not interfere with the OCR Engine recognizing actual text characters. If these pixels simply are not present, the OCR engine won't have to figure out if they are part of a line, word, or character when segmenting the image. Similarly, if they aren't part of a character, once the image is segmented all the way down to an individual character, they won't confuse the OCR engine when it comes time to recognizing what text character that character segment should be.

Before anything, you need a sample document set of the kinds of documents targeted by your IP Profile. Once you have these documents ready in a Test Batch, it's a good idea to evaluate them, getting an idea about the kinds of issues the document set has. Below are the documents we will be looking at and some of the issues involved with them.

This document is full of interference for OCR. Table lines, check boxes, and partially shaded headers can all cause problems for accurate OCR results. Furthermore, we likely will want to use the line and box positions later during data extraction. A Temporary IP Profile can be configured to remove this elements but store their locations in memory for later use.

This too is filled with table lines, as well as having "negative regions" where portions of white text are on a black background. OCR must be able to read black pixels. So, we will switch the white text to black during image processing. It will also need to be turned into true black and white instead of grayscale.
This document may seem simple, but again, each command runs on each document in the document set. We will need to make sure the IP Profile works for each document in the set.
Last, this document will give us a few things to consider. It has a large border that is part of the document. It has a logo and other artifacts that could be removed. As well as these larger non-text artifacts, it will also end up having small specks that could interfere with OCR as well.

OCR absolutely must work with a black and white image. While OCR engines will turn image black and white on their own, they don't always do a great job at it. Furthermore, you have no control over how the OCR engine turns the image black and white. Grooper's image processing capabilities allow for greater configuration of how an image is turned into a black and white image before handing it to the OCR engine. The vast majority of temporary IP Profiles will contain a "Threshold" or "Binarize" step to convert color and grayscale images into true black and white.

Knowing this, let's use a "Threshold" command as our starting point.

  1. Press the "Add" button to add a new IP Step.
  2. Select the "Threshold" command from the "Format Conversions" category.

This document was a grayscale image previously, and now has been turned black and white using the Auto thresholding method.

  1. As you can see, the gray background behind some of the text (such as the portion highlighted here) has been turned white, leaving us a totally black and white image.
  2. Notice as well, the document's image format has changed as well, from "8-Bit Gray" to "Black & White"

We are going to keep the default settings for this step. For more information about thresholding methods, visit the Binarize article.


FYI

The only difference between the "Threshold" IP Command and the "Binarize" IP Command is what bit depth (or color depth) format the altered image takes.

  • "Threshold" will convert the image into a true 1-Bit black and white image. The pixels can be either black or white and nothing else (This bit depth allows for 2¹ colors, or 2 colors, white and black).
  • "Binarize" actually converts the image to an 8-Bit Grayscale image. This means the pixels can be black, white or multiple shades of gray in between (This bit depth allows for 2⁸ colors, or 256 colors, white, black and 254 shades of gray). However, only the white and black channels are used. Functionally, this gives you a black and white image in an 8-bit format.

For most operations, these two IP Commands are interchangeable. For example, if an OCR engine is handed an image processed by the "Binarize" command, it's still black and white even if it's in the grayscale format. The results will be no different than if it were handed an image processed by the "Threshold" command.

However, if a certain IP Command requires a bit depth larger than than single bit black and white (such as some of the "Filter" command's options), the "Binarize" IP Command allows the IP Profile to hand the next step an 8-Bit Grayscale image.

Moving onto the next document, we can see this document was indeed turned black and white, but there's another problem we have to deal with.

  1. This document has labels such as "Loan Terms" (here, highlighted) and "Projected Payments" that are white text on a black background. These are what we will call "inverted labels". OCR engines expect text to be black pixels, not white. So, they aren't going to recognize the text in these inverted labels.

This is a very common problem. Grooper's "Negative Region Removal" command is designed to address this.

  2. Press the "Add" button to add the next IP Step.
  3. Select "Negative Region Removal" from the "Feature Removal" category.

  2. Upon executing this step, the inverted label is now changed to black text on a white background, allowing for OCR to properly recognize the text.
    • Many OCR engines, such as Transym, have similar negative region inversion capabilities. However, these capabilities are "black boxed". At best, you can turn the operation off or on, but you will not be able configure it beyond that. The "Negative Region Removal" command, allows for greater configuration of the detection and removal of these regions. For example, you may notice the label is now outlined in a black border. You can actually remove that border by changing the Outline Thickness property from 1pt to 0pt.

Recall the three major reasons for image processing in Grooper: (1) Archival Adjustments (2) OCR Cleanup, and (3) Layout Data Collection. This next step will focus on getting some layout data (3), with the added benefit of helping out our OCR a little bit (2). This will also be the first step that illustrates the importance of configuring an IP Command's properties to narrow down what you do and don't want to remove from a document.

  1. We will use a "Box Removal" command to both remove checkboxes (to improve OCR accuracy) and determine if they are checked or blank through OMR (for data collection down the road).

OMR stands for Optical Mark Recognition. OMR has been around for even longer than OCR. Remember back in school when you took a test and filled in bubbles on an answer sheet with a No. 2 pencil? Well, those answer sheets were graded by OMR! The answer sheet was fed into a scanner (probably a Scantron) that would detect if a bubble was filled in or not.

Grooper is doing something similar here. The main difference, is first Grooper has to find the box! The "Box Removal" command first detects boxes and save their locations on the page to the LayoutData.json file attached to that page. Once it does, it will check to see if there are any marks inside the box, if any pixels are filled within the boundaries of the box. If they are, it will record that box as "checked" in the LayoutData.json file, or "unchecked" if blank. Last, it will remove the boxes from the page, clearing the way for better OCR results.

  2. Press the "Add" button to add the next IP Step.
  3. Select "Box Removal" from the "Feature Removal" category.

Based on the "Output Image" you can see the boxes on this document have been removed, such as the ones highlighted.

However, these default settings did not remove all the boxes on all our documents. Some of the boxes on this Closing Disclosure document are very small, smaller than our default size range for detecting boxes.

  2. In order to detect and remove these boxes, expand the Size Range property and change the Minimum value from 7pt to 6pt

However, when you make a configuration change that helps one kind of document, it's a good idea to do some unit testing to see if it hurts another.

Going back to our "Application for Cow Ownership" document, you can see we've made an issue for ourselves. We've dropped out the "o"s in "Houston". For this font, these characters are somewhat similar shaped as a box. Before we adjusted the Minimum Size Range, these characters were too small to be considered boxes. Now that we have, they are being detected as boxes and removed.

What are we to do? Ideally, we want to keep this character data but also remove the boxes on the other document. Is this a situation where we have to choose which is more important to us? For some cases, when it comes to image processesing, that may well be the case. However, for our case, we can have it both ways by further configuring the "Box Removal" step.

The Minimum Aspect Ratio and Maximum Aspect Ratio properties allow for variances from a perfectly square box. If they were set at 100%, each only square boxes would be detected and not slightly rectangular ones. The "o"s in Houston, are not perfectly square. They are more like a tall skinny rectangle. Knowing this, we can bump up the Minimum Aspect Ratio to only look for more square boxes.

  2. Change the Minimum Aspect Ratio property from 75% to 90%

But what about this "Employee Termination Form"? The boxes are still there. "Box Removal" failed to detect any boxes. Why?

The boxes here are not very square at all. They are taller skinnier rectangles. By adjusting our Minimum Aspect Ratio to 90%, we've eliminated them as candidates for boxes. If we lower that Minimum Aspect Ratio property back down, we'll run into the earlier problem of removing text data on another document. What are we to do?

If the checkbox states are important for data collection, bare minimum you need to know if they are there, where they are on the page, and whether or not they are checked. Removing them is nice, as it can help our OCR results, but if it comes down to removing them versus not detecting them at all, we may want to err on the side of keeping the boxes on them image but still collecting that layout data.

We can do that by adding in a "Box Detection" command that has looser restrictions on what counts as a box. It won't necessarily matter that the "o"s in Huston on the other document were detected as boxes, only if they are removed.

  1. Press the "Add" button to add the next IP Step.
  2. Select "Box Detection" from the "Feature Detection" category.
  2. Set the Minimum Size Range to 6pt.
  3. Set the Minimum Aspect Ratio to 70%.
  4. This will detect all the boxes within these detection settings, as seen in the "Boxes" diagnostic image.
    • In this diagnostic image, detected boxes are highlighted in green or red. Green boxes are checked. Red boxes are unchecked.

Notice as well, since we used "Box Detection" instead of "Box Removal", the "o"s in "Houston" from our "Application For Cow Ownership" document, remain intact.

FYI Technically, we actually could have used a second "Box Removal" command instead of "Box Detection" to remove the boxes on the "Termination Form" without removing the "o"s in "Houston" here. The boxes on the "Termination Form" are larger than 6pt high. If we kept the Minimum Size Range at 7pt, a "Box Removal" command would have removed the boxes on the "Termination Form" without removing the "o"s in "Houston".

However, this illustrates when you may want to use both a "Box Removal" and "Box Detection" command. If the boxes were smaller than 7pt high, you would have to use "Box Detection" instead of "Box Removal" like we did here.

One of the most common temporary image processing adjustments for OCR cleanup is the "Line Removal" command. Lines are present on most documents in one way or another. They are used to create and divide tables, sections or individual fields on a document. This is great for humans reading a document! They act as visual dividers of information. They are not so great for OCR. Simply removing lines, in most cases, will greatly improve your OCR results. We will add a "Line Removal" command, and look at some common configuration issues.

  1. Press the "Add" button to add the next IP Step.
  2. Select "Line Removal" from the "Feature Removal" category.

You may notice, "Line Removal" has a ton of configurable properties. That's partly because of how important removing lines is to get good OCR results. Rather than putting a black box on this operation, we want to give you a high degree of control when it comes to how lines are detected and how they are removed.

For the most part, these default settings work quite well. These defaults are configured to detect and remove most lines on most documents. The extra configurability is there should you need it. As you can see on our "Application for Cow Ownership" document, "Line Removal" removed all those lines without us lifting a finger.

However, check out Page 3 of this "Closing Disclosure" document. There's still lines there. They are faint, but we can see the default settings did not remove them.

Before we get to crazy about fine-tuning our "Line Removal" commands configurations, sit back for a second and think about what the problem is. These lines are indeed a little more light on the original image. Whenever we thresholded the image in our very first step, using the Auto method, part of these lines were translated as white pixels, and part as black, but not enough to make a solid line. What if we could use a different thresholding method just for "Line Removal" to make these lines come out better before detection?

We can! That's what the Binarization Settings are for on every IP Command in which they appear. In fact, every step we've added so far has this setting. This allows you to use different thresholding methods for different IP Commands. Perhaps for OCR'd text the Auto method works better to turn an image black and white, but, in this case, the Dynamic method is going to allow allow those faint lines to come through clearer as solid black lines, allowing us to detect and remove them.

First we need to move the "Threshold" step from the first step down the list after our "Line Removal" step. Order of operations is very important to IP Profiles. Each step alters the image and hands that altered image to the next step. As we have it setup so far, this IP Profile is already handing Line Removal a black and white image. So, changing the 'Threshold Method' will do nothing. But if we wait to threshold the image until after "Line Removal", we can utilize a different thresholding method before we turn the whole image black and white for good with the "Threshold" step.

  2. Select the "Threshold" step from the IP Profile.
  3. Press the "Move Down" button until the "Threshold" step is the last step in this IP Profile.
    • Note: You may also use the Ctrl + Up and Ctrl + Down keyboard shortcuts to move IP Steps up and down in an IP Profile.
  2. Next, select the "Line Removal" step in IP Profile
  3. Expand the Binarization Settings property to select the Thresholding Method property
  4. Using the dropdown menu, change this setting from Auto to Dynamic
  2. Select the "Binarized" diagnostic image to see how the image was processed using Dynamic thresholding.
    • As you can see, this is probably not the black and white image we want to use for OCR. However, all these lines are nice solid black lines that can be easily detected by "Line Removal".
  2. Select the final "Output Image" diagnostic image to see the end result.
    • We have a much cleaner image with all these lines detected and removed, without adjusting any of the myriad "Line Removal" settings.

Next, you may want to look for large artifacts you can remove, such as the black border on this image. The "Blob Removal" command is perfectly suited for this. We can tell the "Blob Removal" command to look for contiguous collections of pixels above a certain size.

  1. Press the "Add" button to add the next IP Step.
  2. Select "Blob Removal" from the "Feature Removal" category.

We will simply look for blobs that are wider than 5 inches. We can pretty much guarantee ourselves that a text character is going to be smaller than 5 inches wide on any document. Setting a Minimum Width of 5in will get rid of what we want, without getting rid of any text pixels for OCR.

  2. Expand the Width property and set the Minimum property to 5in

As well as large artifacts getting in the way of good OCR, we might want to get rid of small specks as well. The trick is, we will want to remove as many pixels that are not text data, without removing specks that could be text data. After all, a period or a comma can look a lot like a random speck elsewhere on the page. The "Speck Removal" command will help us do this.

  1. Press the "Add" button to add the next IP Step.
  2. Select "Blob Removal" from the "Feature Removal" category.
  2. Change the Max Speck Size to 3px
  3. Select the "Dropout Mask" diagnostic image.

You can see from the "Dropout Mask" all the tiny specks being removed from this image. Some of these specks are random noise, some of these specks are trivial text characters, but some of these are actual character data we want to preserve.

This is what the Quiet Zone Size property is for. It will create a buffer zone around larger pixel segments, such as text characters. If specks fall in this buffer zone, they will not be dropped out.

  2. Adjust the Quiet Zone Size property to 4pt, 2pt.
  • This will create a quiet zone of 4pt to the left and right of a character and 2pt above and below. You can also specify, left, right, top and bottom zones individually by expanding the Quiet Zone Size and adjusting them there.

These Quiet Zone settings are helpful for targeting small specks you do want to remove, while retaining text characters that would otherwise look like specks.

There are other adjustments we could probably make to clean up these documents, but at some point you have to stop tinkering and put your IP Profile into action.

Temporary IP Profiles are executed as part of an OCR Profile. During the Recognize activity, the IP Profile's steps will be applied to a temporary copy of the document's image. OCR will then be performed on that temporary image according the the OCR Profile's settings.

  1. To set the temporary IP Profile select the IP Profile property on an OCR Profile.
  2. Using the dropdown menu, select your IP Profile from the Node Tree.

When using the "OCR Testing" tab, you can see the temporary image your IP Profile hands the OCR engine.

  2. Select a page from the Test Batch, and press the "OCR Page" button.
  3. Select the "Diagnostics" tab.
  4. Select the "IP Image" diagnostics image from the Diagnostics Panel.

Conditional IP

What happens when you have documents in your document set that just don't fit your IP Profile? Perhaps one configuration of a Box Removal command works for most of the documents, but there's one type of document that needs an entirely different configuration. What happens if most of the documents in your set perform well using the standard "Auto" thresholding method, but one works better using "Adaptive"?

In these situations, you may be able to use conditional logic via the "Should Execute" and "Next Step" expressions on IP Steps and IP Groups in an IP Profile. These expressions allow us to use snippets of .NET code to access information about the image or steps in the profile, and use them to determine if and when a step should run in an IP Profile.

Example: Should Execute Based on the Success of the Previous Step

One thing you might want to do in an IP Profile is create a logical order of operations where a step should run only if the step before it fails. That way it can act as a secondary "fail safe" or use configurations outside the boundaries of the first step.

Consider the following example for "Box Removal".

The following example uses these documents. We want to make a simple IP Profile using "Box Removal" and "Line Removal" commands. However, as we will see, we will need conditional logic, using a "Should Execute" expression, to avoid certain issues.


Configure the First Box Removal

First, let's add a "Box Removal" and "Line Removal" command. Both are found in the "Feature Removal" category.



Selecting the first document, "Application for Cow Ownership", there are a few configurations we need to take into consideration. First, there are some very small boxes on this document. So, we need to adjust the "Size Range" property. Change the "Size Range" from "7pt - 16pt" to "6pt - 16pt" (Adjust the "Minimum" sub property to "6pt").

This will get all the tiny checkboxes on the document. However, we've got an unintended consequence. It's also removing the two "o"s in "Houston".



Not a huge deal. They are being removed because of some of our box detection properties, specifically the "Minimum Aspect Ratio". The aspect ration properties define the "squareness" of a box. At 100%, only perfectly square boxes will be removed. The default of 75% for the Minimum Aspect Ratio allows for variation in how "skinny" the rectangle is. It will allow boxes to be detected if they are 75% as wide as they are high. However, the boxes on this document are very square. If we increase this property to 90%, we are in the clear. The "o"s are no longer detected as boxes and therefore, not removed.


Configure the Second Box Removal

However, for the second document, the "Box Removal" command won't work. These boxes are indeed skinnier than normal squares. So our configurations will not work.

Go ahead and add a second "Box Removal" command and move it between the first and last steps.



To target these boxes we will need to decrease the Minimum Aspect Ratio. Adjusting this to 70% will detect and remove these boxes. Also, these boxes are on the smaller side. So we will need to adjust the Minimum Size to 6pt as well.



However, now this second "Box Removal" command is removing the "o"s in "Houston" on the first document.



No need to worry. We can use a "Should Execute" expression to run the first Box Removal on the first document, but the second Box Removal on the second.

Edit the Should Execute Expression

The conditional expressions are on the "Selected Step" tab of a selected step. Select the second "Box Removal" command, and switch to the "Selected Step" tab.



Select the "Should Execute Expression" property and press the ellipsis button at the end.



This will bring up the "Should Execute Expression" editor. Here you can write a snippet of code to either execute the selected property or not. If the expression evaluates to "true" the step will run. If it returns as "false", it will not.



From here, we need to make an expression based on the results of the previous Box Removal command. If the previous Box Removal did not find any boxes, we want this Box Removal to execute. Otherwise, we want to use the results of the first Box Detection. This will get around the second Box Detection command running on the first document and removing the "o"s in "Houston".

We can use the following expression: Results.Box_Removal.Boxes.Count = 0

This will return "True" if the Box Removal command before it found 0 boxes. As long as the Box Removal command found a single box, this second Box Removal command will not run.



Now, only the first Box Removal command runs on the first document, skipping over the second step entirely.



But, for the second document, since no boxes were detected using the first Box Removal command, the second step runs!



Example: Should Execute Based on Classify Image

Different IP Steps, IP Groups or even entire IP Profiles can be executed based on the results of the "Classify Image" command. The "Classify Image" command compares an image against a set of sample images and classifies the image based on which sample it is most similar to. It does this by analyzing the color space of an image. For example, the RBG color space is made up of a red channel, a green channel, and a blue channel. The similarity would then be based on how similar the information in these three channels is to another image. For example, if a sample image has a high value in the red channel but a low value in the blue channel, it would not match an image that has a high blue channel but a low red channel.

For this example, we will create a conditional expression for thresholding these two documents.



The blue transcript is a good candidate for using "Adaptive" thresholding over the "Auto" method.

Auto Adaptive


However, the brown transcript is handled better by the "Auto" thresholding method.


Auto Adaptive


We will use the "Classify Image" command to have one image use "Auto" and the other "Adaptive".

Add IP Steps

This IP Profile will have three IP Steps: One "Classify Image" and two "Threshold" commands, one of which uses the "Auto" method and the other which uses the "Adaptive" method. Here, the two "Threshold" commands have been renamed accordingly.


Give Classify Image Sample Images

Select the "Classify Image" step and navigate to the "Sample Images" property. Press the ellipsis button at the end of the property.



This will bring up the "Sample Images" window. Press the "Add" button to add a new sample image.



From here, select a sample from your test batch. For this example, we are selecting this blue transcript. Give this sample a name using the "Sample Name" box. Press "OK" when finished.



This will add the image to the list of Sample Images on the left. When finished adding sample images, press "Done"


Select the Color Space to Analyze

Next, select the color space you wish to use to classify the image. There are a variety of color space options, each of which measures different channels making up a document's color. For this example we are using the HSV color space, which measures hue, saturation, and value (pixel intensity).



Classify Image's Execution Log shows the measurements for the selected color channel and how similar they are to the sample image. All color spaces will have "Channel 1", "Channel 2", "Channel 3" and "Entropy" listed under "Source Image Features" These are the image based measurements of the selected image. The three channels correspond to the information in the three channels of the selected color space. For HSV, "Channel 1" is the hue, "Channel 2" is the saturation, and "Channel 3" is the value. "Entropy" is a measure of how "busy" the image is. The more black text on a document, the higher the entropy measure will be.

The similarity score to each trained image is seen under "Results for Trained Image Classes". We only trained one image, the "Blue Transcript", which is coming in as "99.77%" similar.

The image is assigned a classification based on how similar the three channels and the entropy are to the trained images (assuming it meets the "Minimum Similarity" score). This image is classified as "Blue Transcript". This can be verified by its "Class Name".



For the brown transcript, it was different enough from the trained image that it did not meet the minimum similarity of 85%. Hence it received no classification. This can be verified by its "Class Name", which is "None".



Now that we have a benchmark that can tell one image from the other, we can use it to conditionally threshold the image.

Set the Should Execute Expression

Next we need to figure out and apply our logic for thresholding these documents. If the image is classified as "Blue Transcript" we want to use the Adaptive method. Otherwise, we want to use the Auto method. The next step should be the Threshold (or Binarize) command using Adaptive thresholding.

Select the "Threshold - Adaptive" step. Navigate to the "Selected Step" tab. Select the "Should Execute Expression" property and press the ellipsis button at the end.



The expression we will write will reference the results of the "Image Command". The basic idea here is if the image was classified as "Blue Transcript" execute the command, otherwise do not. The code expression we can use is as follows: Results.Classify_Image.ClassName = "Blue Transcript"

Note, the class name you enter must match the sample image's name exactly. Press the "OK" button when finished.



Success! The blue transcript was turned black and white using the Adaptive method. However, the real test will be if the brown transcript skipped the "Threshold - Adaptive" step and went straight to the "Threshold - Auto" step.



! Although it may appear as if the "Threshold - Auto" step is skipped, all steps after the step using the "Should Execute Expression" are still applied. It only appears as if it doesn't run because it's being handed an image that is already black and white. The next step technically still runs. Should Execute Expressions only determine if a step is applied or not. They do not have anything to do with the order in which other steps are applied (The "Next Step Expression" can determine order).


Did the step execute on the brown transcript, which was not classified as the "Blue Transcript" image? It did not! The "Threshold - Adaptive" step was skipped and the next step in the sequence, "Threshold - Auto" ran as normal.


Example: IP Groups and the Next Step Expression

Imagine for the example above, we wanted one set of IP Commands to run on the brown transcripts and another to run on the blue transcripts. Since we already know we can classify these documents seperately and use that information to determine if a step should execute, we can use IP Groups to tell an IP Profile to execute a collection of IP Steps and conditionally based on the image classification and determine what happens next using the "Next Step Expression" property.

We will keep this example fairly simple. On the left we have a blue transcript and the right a brown one.

For the blue transcript we want to execute two IP Commands: Threshold using Adaptive method and a basic "Line Removal"

For the brown transcript we want to execute two IP Commands: Threshold using Auto method and a basic "Speck Removal"

We don't want the brown transcript to run "Line Removal" and we don't want the blue transcript to run "Speck Removal". We will use the same "Classify Image" command we used in the previous example to classify the blue transcript as "Blue Transcript".

Add an IP Group

IP Groups are collections of IP Steps in an IP Profile. This allows you to nest a series of steps within an IP Profile. You can think of IP Groups as mini-profiles or sub-profiles that can be used as a single step in the execution sequence of an IP Profile.

To add an IP Group, right click an IP Profile in the Node Tree, mouse over "Add" and select "IP Group"



Name the IP Group whatever you'd like. We will name this one "Blue Transcript IP"

Add the steps you wish the group to execute as if you were adding them to an IP Profile. This group will add a Threshold step using the Adaptive method and a basic Line Removal Command. The IP Group will be nested as a child of the IP Profile, with its own steps nested as children of the IP Group.



Navigate back up the Node Tree to the parent IP Profile. You will see the IP Group as a step in the profile. Without any conditional logic applied, the IP Steps in the IP Group simply run as if they were steps in the IP Profile.


Add the Remaining IP Steps

Next, we will add the steps we want the brown transcript to execute: A Threshold step using the Auto method, and a basic Speck Removal command.



Set the Should Execute Expression

This will be similar to the previous example. We will use the classification results of "Classify Image" to only execute the IP Group if the image was classified as "Blue Transcript".

Navigate to the IP Group (here named "Blue Transcript IP"). Using the "Selected Group" tab, select the "Should Execute Expression" property and press the ellipsis button at the end.



We can use the same code expression we used before: Results.Classify_Image.ClassName = "Blue Transcript"

Press "OK" when finished.



This will only execute the steps in this IP Group if the should submit expression evaluates to "True". In this case, if the image is classified as "Blue Transcript". However, the remainder of the IP Profile still runs for those images. Seen below, Speck Removal still executes. We want to tell the IP Profile to stop running at the end of the IP Group.




We can change the order of operations for this profile using the "Next Step Expression" property.

Navigate to the IP Group (here named "Blue Transcript IP"). Using the "Selected Group" tab, select the "Should Execute Expression" property and press the ellipsis button at the end.



The "Next Step Expression" dictates what happens next after the IP Step or IP Group executes. What we want to do is tell the IP Profile to stop running for the blue transcripts after the IP Group finishes, but continue onto the "Threshold - Auto" step for the brown transcript.

The code expression to execute this logic is as follows: If(Results.Classify_Image.ClassName = "Blue Transcript", Nothing, Steps.Threshold_Auto)

This follows the logic of If(condition, if condition is met, do this, otherwise do this)

The condition here is that the image is classified as a "Blue Transcript" by the Classify Image command. "Nothing" here means the IP Profile will stop processing and perform no more of the IP Steps in the profile. "Steps.Threshold_Auto" means the next step to execute will be the "Threshold - Auto" step in the profile.

Press the "OK" button when finished.



Now, the blue transcript quits running after the IP Group is finished. So, the rest of the IP Profile (the "Threshold - Auto" and "Speck Removal" commands) is skipped entirely.


Retrieved from "https://wiki.grooper.com/index.php?title=2.72:IP_Profile_(Object)&oldid=22356"