Key-Value Pair (Collation Provider) - Revision history

Randallkinard at 16:25, 28 August 2024

2024-08-28T16:25:59Z

← Older revision		Revision as of 11:25, 28 August 2024
Line 1:		Line 1:
	{{~~Migrated2023~~}}		{{Migrated}}
	{{2023:{{PAGENAME}}}}		{{2023:{{PAGENAME}}}}

Dgreenwood: Removed redirect to Key-Value Pair (Collation Provider) - 2023

2024-01-02T14:46:01Z

Removed redirect to Key-Value Pair (Collation Provider) - 2023

← Older revision		Revision as of 09:46, 2 January 2024
Line 1:		Line 1:
	~~#REDIRECT[[Key-Value Pair (Collation Provider) -~~ 2023]]		{{Migrated2023}}
			{{2023:{{PAGENAME}}}}

Rpatton: Redirected page to Key-Value Pair (Collation Provider) - 2023

2023-11-07T16:25:06Z

Redirected page to Key-Value Pair (Collation Provider) - 2023

Show changes

Dgreenwood at 22:19, 9 December 2022

2022-12-09T22:19:16Z

← Older revision		Revision as of 17:19, 9 December 2022
Line 253:		Line 253:
	</tab>		</tab>
	</tabs>		</tabs>

			[[Category:Articles]]
			[[Category:Version 2.90]]

Dgreenwood: /* Create A Key-Value Pair Extractor */

2020-08-27T17:15:59Z

Create A Key-Value Pair Extractor

← Older revision		Revision as of 12:15, 27 August 2020
Line 39:		Line 39:

	=== Create A Key-Value Pair Extractor ===		=== Create A Key-Value Pair Extractor ===

			If you would like to follow along with this tutorial, you can download the zip file below and import it into your Grooper repository (version 2.90).
			* [[media:Key-Value Pair (Collation Provider) (2.90).zip]]

	<tabs style="margin:20px">		<tabs style="margin:20px">

Dgreenwood: /* Create A Key-Value Pair Extractor */

2020-08-27T17:07:19Z

Create A Key-Value Pair Extractor

← Older revision		Revision as of 12:07, 27 August 2020
Line 153:		Line 153:
	# Change the property from ''Disabled'' to ''Enabled''.		# Change the property from ''Disabled'' to ''Enabled''.

	This will look for results of the "Value Extractor" and only return the closest ''vertically, below'' the "Key Extractor's" result. But... Something is not right here. What we want to extract is the phone number underneath the "home phone" label directly below it, as seen in the box labeled "Vertical Layout" on the document.		This will look for results of the "Value Extractor" and only return the closest ''vertically, below'' the "Key Extractor's" result. But... Something is not right here. What we want to extract is the phone number underneath the "home phone" label directly below it, as seen in the box labeled "Vertical Layout" on the document. But, we're also getting another result.

	: 4. The problem is, we're picking up the wrong key.		: 4. The problem is, we're picking up the wrong key. The key in the "Horizontal Layout" box uses a horizontal alignment. However, there just ''happens'' to be a number vertically bellow this key.

	The "Key Extractor" is set up to return "home phone", which appears three times on this document. The first time it appears is at the time of the page. If you draw a vertical line down from that label, sure enough, there is a phone number there, it's just using the wrong label.		The "Key Extractor" is set up to return "home phone", which appears three times on this document. The first time it appears is at the time of the page. If you draw a vertical line down from that label, sure enough, there is a phone number there, it's just using the wrong label.
Line 164:		Line 164:
	{\|cellpadding=10 cellspacing=5		{\|cellpadding=10 cellspacing=5
	\|style="width:40%" valign=top\|		\|style="width:40%" valign=top\|
	~~Coincidentally, this~~ could be the right value you're looking for, but more often than not, this will produce ''false positive'' results. Typically, keys are much closer to their values than what we see here. We need a way to restrict the space between the two extractor results to toss out these false positives.		This ''could'' coincidentally return the right value you're looking for, but more often than not, this will produce ''false positive'' results. For example, here the "Home Phone" key travels down to the middle of the page and finds a cell phone number in another section of the document. Typically, keys are much closer to their values than what we see here. We need a way to restrict the space between the two extractor results to toss out these false positives.

	There are two very common ways to do this for ''Key-Value Pair'' Collation, the '''''Maximum Distance''''' and the '''''Enforce Line Boundaries''''' properties.		There are two very common ways to do this for ''Key-Value Pair'' Collation, the '''''Maximum Distance''''' and the '''''Enforce Line Boundaries''''' properties.
Line 172:		Line 172:
	: 5a. Both '''''Vertical Layout''''' and '''''Horizontal Layout''''' have a '''''Maximum Distance''''' sub-property.		: 5a. Both '''''Vertical Layout''''' and '''''Horizontal Layout''''' have a '''''Maximum Distance''''' sub-property.
	: 6a. You can set a length here, in inches, points, millimetres or centimetres. If the distance between the key and the value are greater than the length set, that result will be tossed out.		: 6a. You can set a length here, in inches, points, millimetres or centimetres. If the distance between the key and the value are greater than the length set, that result will be tossed out.
	:* Here, we set the distance to ''0.25in''. This threw out the result we were getting because the vertical distance between the first "home phone" label is much longer than 0.25 inches. Then, the extractor looked for the next key value, which does have a vertically aligned value next to it, within 0.25 inches.		:* Here, we set the distance to ''0.25in''. This threw out the false positive result we were getting because the vertical distance between the first "home phone" label is much longer than 0.25 inches. Then, the extractor looked for the next key value, which does have a vertically aligned value next to it, and within 0.25 inches.
	:* FYI: With the '''''Maximum Distance''''' property left blank, there is no maximum distance. ''Key-Value Pair'' collation will go from the "Key Extractor's" result all the way down the page looking for the "Value Extractor's" result. However, this will ''not'' span pages. The '''''Horizontal''''' and '''''Vertical Layouts''''' will only look for values next to a key on the same page.		:* FYI: With the '''''Maximum Distance''''' property left blank, there is no maximum distance. ''Key-Value Pair'' collation will go from the "Key Extractor's" result all the way down the page looking for the "Value Extractor's" result. However, this will ''not'' span pages. The '''''Horizontal''''' and '''''Vertical Layouts''''' will only look for values next to a key on the same page.
	\|		\|
Line 193:		Line 193:
	[[File:Key-Value Pair - Grooper Screenshots 08.png]]		[[File:Key-Value Pair - Grooper Screenshots 08.png]]
	\|}		\|}

	~~{\|cellpadding=10 cellspacing=5~~
	~~\|style="width:40%" valign=top\|~~
	</tab>		</tab>
	<tab name="Flow Layout" style="margin:20px">		<tab name="Flow Layout" style="margin:20px">
	=== The Flow Layout ===		=== The Flow Layout ===

	The '''Flow Layout''' is a little different. Instead of using the linear horizontal or vertical relationship between the key and value data instances, it uses their relationship in the "flow" of the text data. This method travels from the key looking for the value as an English reader would, starting at the key, going character-by-character (typically) ~~left to right~~ and line-by-line top to bottom.		The '''Flow Layout''' is a little different. Instead of using the linear horizontal or vertical relationship between the key and value data instances, it uses their relationship in the "flow" of the text data. This method travels from the key looking for the value as an English reader would, starting at the key, going character-by-character left to right (typically) and line-by-line top to bottom.

			{\|cellpadding=10 cellspacing=5
			\|style="width:40%" valign=top\|
	# View the ''Key-Value Pair'' configuration properties by expanding the the '''''Collation''''' property.		# View the ''Key-Value Pair'' configuration properties by expanding the the '''''Collation''''' property.
	# Select '''''Flow Layout'''''.		# Select '''''Flow Layout'''''.
	# Change the property from ''Disabled'' to ''Enabled''.		# Change the property from ''Disabled'' to ''Enabled''.
			# This is not the result we're looking for. The result we're looking for is the one in the paragraph at the bottom of the document. The '''''Flow Layout''''' generally requires more configuration than just enabling the property.
			\|
			[[File:Key-Value Pair - Grooper Screenshots 10.png]]
			\|-
			\|valign=top\|
			The '''''Flow Layout''''' by default will look for a value separated from the key by a single space character (or if the first character of the value's result ''immediately'' follows the key in the ext flow).

			As you can see in the text data, the phone number value extracted is separated from the key "Home Phone" by a single space. That is why this value in particular returns with no configuration of the '''''Flow Layout''''' properties.

			In order to change the amount of allowable text between a key and a value, either the '''''Separator Expression''''' or '''''Maximum Character Distance''''' properties must be configured.
			\|
			[[File:Key-Value Pair - Grooper Screenshots 11.png]]
			\|}

			=== The Greediest Separator Expression: The .* Expression ===

			{\|cellpadding=10 cellspacing=5
			\|style="width:40%" valign=top\|
			If you know the approximate number of characters between a key and a value in the text flow, you can use the '''''Maximum Character Distance''''' property. Say you know the value is about 100 characters after the key, you can enter ''100'' for the '''''Maximum Character Distance'''''. If the value falls within 100 characters after the key, it will return. If it's 101 or more characters, it will not.

			But what if you don't know the distance between characters? One place people start is with a <code>.</code> separator expression. The <code>.</code> regular expression will match any character with a variable length of zero to unlimited characters. This expression is like setting a '''''Maximum Character Distance''''' of infinity.

			This expression will work fine in some cases. However, if we do that here, you'll see we end up matching a lot of data we do not want.

			For each key "Home Phone" found, the first phone number after it in the text flow is returned. Each value is satisfied by the '''''Separator Expression''''' </code>.*</code> used.
			\|
			[[File:Key-Value Pair - Grooper Screenshots 12.png]]
			\|}

			=== A More Restrictive Separator Expression ===

			{\|cellpadding=10 cellspacing=5
			\|style="width:40%" valign=top\|
			Often, a more restrictive separator expression is needed. The good thing about these separator expressions is you have the whole toolbox of regular expression at your disposal.

			For '''''Flow Layout''''' you also have a set of '''''Preprocessing Options''''' available as well, including [[Tab Marking]]. With tab marking enabled, tab characters replace space characters where whitespace gaps are longer than a normal space character. Since we are trying to match something in a written paragraph, we shouldn't ever see a tab character between the key and the value. So, we can match what we want by enabling '''''Tab Marking'''' and setting the '''''Separator Expression''''' to ''<code>[^\t]+</code>''

			# Expand the '''''Preprocessing Options''''' under the '''''Flow Layout''''' properties.
			# Change the '''''Tab Marking''''' property from ''Disabled'' to ''Enabled''
			# Select the '''''Separator Expression''''' property and enter ''[^\t]+''
			#* Now, the extractor will go character-by-character in the text flow looking for the value, traveling one to infinite characters, as long as that character is not a tab character. And, we end up getting the result we want.
			\|
			[[File:Key-Value Pair - Grooper Screenshots 13.png]]
			\|}
	</tab>		</tab>
	</tabs>		</tabs>

Dgreenwood: /* Create A Key-Value Pair Extractor */

2020-08-27T14:25:27Z

Create A Key-Value Pair Extractor

← Older revision		Revision as of 09:25, 27 August 2020
Line 160:		Line 160:
	\|		\|
	[[File:Key-Value Pair - Grooper Screenshots 06.png]]		[[File:Key-Value Pair - Grooper Screenshots 06.png]]
	\|-		\|}
	\|\|style="width:40%" valign=top\|		=== Resolving Common Problems ===
	~~'''Resolving Common Problems'''~~		{\|cellpadding=10 cellspacing=5
			\|style="width:40%" valign=top\|
	Coincidentally, this could be the right value you're looking for, but more often than not, this will produce ''false positive'' results. Typically, keys are much closer to their values than what we see here. We need a way to restrict the space between the two extractor results to toss out these false positives.		Coincidentally, this could be the right value you're looking for, but more often than not, this will produce ''false positive'' results. Typically, keys are much closer to their values than what we see here. We need a way to restrict the space between the two extractor results to toss out these false positives.

	There are two very common ways to do this for ''Key-Value Pair'' Collation, the '''''Maximum Distance''''' and the '''''Enforce Line Boundaries''''' properties.		There are two very common ways to do this for ''Key-Value Pair'' Collation, the '''''Maximum Distance''''' and the '''''Enforce Line Boundaries''''' properties.

			'''''Maximum Distance'''''

	: 5a. Both '''''Vertical Layout''''' and '''''Horizontal Layout''''' have a '''''Maximum Distance''''' sub-property.		: 5a. Both '''''Vertical Layout''''' and '''''Horizontal Layout''''' have a '''''Maximum Distance''''' sub-property.
Line 175:		Line 177:
	[[File:Key-Value Pair - Grooper Screenshots 07.png]]		[[File:Key-Value Pair - Grooper Screenshots 07.png]]
	\|-		\|-
	\|		\|valign=top\|
			'''''Enforce Line Boundaries'''''

	The other solution to this problem will only be possible if your document utilizes ''lines'' to break up sections or fields. For documents that do this, information is visually divided from other information by putting it in a box. This example also does that. The values horizontally aligned are put in one box. The values vertically aligned are put in another box. Because of this fact, we can take advantage of the '''''Enforce Line Boundaries''''' property.		The other solution to this problem will only be possible if your document utilizes ''lines'' to break up sections or fields. For documents that do this, information is visually divided from other information by putting it in a box. This example also does that. The values horizontally aligned are put in one box. The values vertically aligned are put in another box. Because of this fact, we can take advantage of the '''''Enforce Line Boundaries''''' property.

Dgreenwood: /* Create A Key-Value Pair Extractor */

2020-08-27T14:20:47Z

Create A Key-Value Pair Extractor

← Older revision		Revision as of 09:20, 27 August 2020
Line 107:		Line 107:
	They '''''Layout''''' method will define how the "Key Extractor" and "Value Extractor" results are spatially related to each other. Is the key above the value? Is it to the left of it? The right? Configuring these settings will dictate where you expect to find the value in relation to the key.		They '''''Layout''''' method will define how the "Key Extractor" and "Value Extractor" results are spatially related to each other. Is the key above the value? Is it to the left of it? The right? Configuring these settings will dictate where you expect to find the value in relation to the key.

	The '''''Layout''''' may be:		{\|cellpadding=10 cellspacing=5
	# Horizontal		\|style="width:40%" valign=top\|
	# Vertical		All Collation Providers have their own set of configurable properties, including ''Key-Value Pair''.
	# Flow		# To view the ''Key-Value'' pair properties, either double-click the '''''Collation''''' property or single click the arrow to the left of the property.
			# To select and configure a '''''Layout''''' enable one of the three Layout Providers.
			#* The '''''Layout''''' may be:
			#** Horizontal
			#** Vertical
			#** Flow
			\|
			[[File:Key-Value Pair - Grooper Screenshots 09.png]]
			\|}

			{\|cellpadding="10" cellspacing="5"
			\|-style="background-color:#36b0a7; color:white"
			\|style="font-size:14pt"\|'''FYI'''\|\|It is possible to enable multiple '''''Layouts''''' and use multiple layouts on a single '''Data Type''', but do so with caution.

			If there is only a ''single'' key found by the "Key Extractor" (This is often the case), only a ''single'' result will return, using a single '''''Layout'''''. Be careful when enabling multiple '''''Layouts''''' as there is a certain order of operations when it comes to which layout is used first. It is as follows: First, the '''''Horizontal Layout's''''' result is returned. If there is no result from that layout, then the '''''Flow Layout's''''' result is returned. Last, if no other layouts produce results, the '''''Vertical Layout's''''' result is returned.

			If there are ''multiple'' keys, this can get even more complicated with each key using its own '''''Layout''''', according to this order of operations.

			You may find it easier or more prudent to create a separate ''Key-Value Pair'' collated '''Data Type''' for each '''''Layout''''' as children of a parent '''Data Type'''. You then have more control of which result is returned via the '''''Order By''''' property.
			\|}
			</tab>
			<tab name="Horizontal Layout" style="margin:20px">
			=== The Horizontal Layout ===
	{\|cellpadding=10 cellspacing=5		{\|cellpadding=10 cellspacing=5
	\|style="width:40%" valign=top\|		\|style="width:40%" valign=top\|
	~~'''Horizontal Layout'''~~
	# View the ''Key-Value Pair'' configuration properties by expanding the the '''''Collation''''' property.		# View the ''Key-Value Pair'' configuration properties by expanding the the '''''Collation''''' property.
	# Select '''''Horizontal Layout'''''.		# Select '''''Horizontal Layout'''''.
Line 123:		Line 143:
	[[File:Key-Value Pair - Grooper Screenshots 05.png]]		[[File:Key-Value Pair - Grooper Screenshots 05.png]]
	\|}		\|}
			</tab>
			<tab name="Vertical Layout" style="margin:20px">
			=== The Vertical Layout ===

	{\|cellpadding=10 cellspacing=5		{\|cellpadding=10 cellspacing=5
	\|style="width:40%" valign=top\|		\|style="width:40%" valign=top\|
	~~'''Vertical Layout'''~~
	# View the ''Key-Value Pair'' configuration properties by expanding the the '''''Collation''''' property.		# View the ''Key-Value Pair'' configuration properties by expanding the the '''''Collation''''' property.
	# Select '''''Vertical Layout'''''.		# Select '''''Vertical Layout'''''.
Line 139:		Line 161:
	[[File:Key-Value Pair - Grooper Screenshots 06.png]]		[[File:Key-Value Pair - Grooper Screenshots 06.png]]
	\|-		\|-
	\|		\|\|style="width:40%" valign=top\|
			'''Resolving Common Problems'''

	Coincidentally, this could be the right value you're looking for, but more often than not, this will produce ''false positive'' results. Typically, keys are much closer to their values than what we see here. We need a way to restrict the space between the two extractor results to toss out these false positives.		Coincidentally, this could be the right value you're looking for, but more often than not, this will produce ''false positive'' results. Typically, keys are much closer to their values than what we see here. We need a way to restrict the space between the two extractor results to toss out these false positives.

Line 168:		Line 192:
	{\|cellpadding=10 cellspacing=5		{\|cellpadding=10 cellspacing=5
	\|style="width:40%" valign=top\|		\|style="width:40%" valign=top\|
	'''Flow Layout'''		</tab>
			<tab name="Flow Layout" style="margin:20px">
			=== The Flow Layout ===

			The '''Flow Layout''' is a little different. Instead of using the linear horizontal or vertical relationship between the key and value data instances, it uses their relationship in the "flow" of the text data. This method travels from the key looking for the value as an English reader would, starting at the key, going character-by-character (typically) left to right and line-by-line top to bottom.

	# View the ''Key-Value Pair'' configuration properties by expanding the the '''''Collation''''' property.		# View the ''Key-Value Pair'' configuration properties by expanding the the '''''Collation''''' property.
	# Select '''''Flow Layout'''''.		# Select '''''Flow Layout'''''.

Dgreenwood at 22:00, 26 August 2020

2020-08-26T22:00:28Z

← Older revision		Revision as of 17:00, 26 August 2020
Line 133:		Line 133:
	This will look for results of the "Value Extractor" and only return the closest ''vertically, below'' the "Key Extractor's" result. But... Something is not right here. What we want to extract is the phone number underneath the "home phone" label directly below it, as seen in the box labeled "Vertical Layout" on the document.		This will look for results of the "Value Extractor" and only return the closest ''vertically, below'' the "Key Extractor's" result. But... Something is not right here. What we want to extract is the phone number underneath the "home phone" label directly below it, as seen in the box labeled "Vertical Layout" on the document.

	~~#<li value=~~4> The problem is, we're picking up the wrong key.		: 4. The problem is, we're picking up the wrong key.

	The "Key Extractor" is set up to return "home phone", which appears three times on this document. The first time it appears is at the time of the page. If you draw a vertical line down from that label, sure enough, there is a phone number there, it's just using the wrong label.		The "Key Extractor" is set up to return "home phone", which appears three times on this document. The first time it appears is at the time of the page. If you draw a vertical line down from that label, sure enough, there is a phone number there, it's just using the wrong label.
Line 143:		Line 143:

	There are two very common ways to do this for ''Key-Value Pair'' Collation, the '''''Maximum Distance''''' and the '''''Enforce Line Boundaries''''' properties.		There are two very common ways to do this for ''Key-Value Pair'' Collation, the '''''Maximum Distance''''' and the '''''Enforce Line Boundaries''''' properties.

			: 5a. Both '''''Vertical Layout''''' and '''''Horizontal Layout''''' have a '''''Maximum Distance''''' sub-property.
			: 6a. You can set a length here, in inches, points, millimetres or centimetres. If the distance between the key and the value are greater than the length set, that result will be tossed out.
			:* Here, we set the distance to ''0.25in''. This threw out the result we were getting because the vertical distance between the first "home phone" label is much longer than 0.25 inches. Then, the extractor looked for the next key value, which does have a vertically aligned value next to it, within 0.25 inches.
			:* FYI: With the '''''Maximum Distance''''' property left blank, there is no maximum distance. ''Key-Value Pair'' collation will go from the "Key Extractor's" result all the way down the page looking for the "Value Extractor's" result. However, this will ''not'' span pages. The '''''Horizontal''''' and '''''Vertical Layouts''''' will only look for values next to a key on the same page.
	\|		\|
	[[File:Key-Value Pair - Grooper Screenshots 07.png]]		[[File:Key-Value Pair - Grooper Screenshots 07.png]]
			\|-
			\|
			The other solution to this problem will only be possible if your document utilizes ''lines'' to break up sections or fields. For documents that do this, information is visually divided from other information by putting it in a box. This example also does that. The values horizontally aligned are put in one box. The values vertically aligned are put in another box. Because of this fact, we can take advantage of the '''''Enforce Line Boundaries''''' property.

			: 5b. Select the '''''Enforce Line Boundaries''''' property
			: 6b. Set this to ''True''. If there is a line physically between the key and the value on the page, that result will be thrown out.
			:* Note: We set the '''''Maximum Distance''''' property back to blank. We're not cheating here.

			{\|cellpadding="10" cellspacing="5"
			\|-style="background-color:#f89420; color:white"
			\|style="font-size:14pt"\|'''⚠'''\|\|This will only work if the document has line location information in its folder or page's LayoutData.json file. This information must be obtained ''before'' the extractor runs via the [[Image Processing]] or [[Recognize]] activity.
			\|}
			\|
			[[File:Key-Value Pair - Grooper Screenshots 08.png]]
	\|}		\|}

Dgreenwood: /* Create A Key-Value Pair Extractor */

2020-08-26T20:48:29Z

Create A Key-Value Pair Extractor

@@ Line 100: / Line 100: @@
 Once you select ''Key-Value Pair'', you will ''not'' see the results list change.  It will still appear as if the two child extractors' results are being returned one by one (like the ''Individual'' Collation Provider).  Some Collation Providers, such as ''Key-Value Pair'', require some configuration before their results are collated.  Specifically, you ''must'' choose which '''''Layout''''' method is used.
+|
-[[File:Key-Value Pair - Grooper Screenshots 03.png]]
+[[File:Key-Value Pair - Grooper Screenshots 04.png]]
 |}
 </tab>
-<tab name="Set the Layout Setting" style="margin:20px">
+<tab name="Configure the Layout Settings" style="margin:20px">
 === Configure the Layout Settings ===
 They '''''Layout''''' method will define how the "Key Extractor" and "Value Extractor" results are spatially related to each other.  Is the key above the value?  Is it to the left of it?  The right?  Configuring these settings will dictate where you expect to find the value in relation to the key.
@@ Line 112: / Line 112: @@
 # Flow
-==== Horizontal Layout ====
+{|cellpadding=10 cellspacing=5
 # View the ''Key-Value Pair'' configuration properties by expanding the the '''''Collation''''' property.
 # Select '''''Horizontal Layout'''''.
 # Change the property from ''Disabled'' to ''Enabled''.
 </tab>
 </tabs>