2023:Lexical (Classify Method) - Revision history

Dgreenwood: Dgreenwood moved page 2023:Lexical (Classification Method) to 2023:Lexical (Classify Method)

2025-06-25T15:43:04Z

Dgreenwood moved page 2023:Lexical (Classification Method) to 2023:Lexical (Classify Method)

← Older revision	Revision as of 10:43, 25 June 2025
(No difference)

Dgreenwood at 21:20, 12 May 2025

2025-05-12T21:20:13Z

← Older revision		Revision as of 16:20, 12 May 2025
Line 111:		Line 111:
	=== Mixed Classification: Combining Training-Based and Rules-Based Approaches ===		=== Mixed Classification: Combining Training-Based and Rules-Based Approaches ===

	Furthermore, a [[~~Rules Based (Classification Method)\|~~Rules-Based]] approach can be taken in combination with the training based approach, when using the ''Lexical'' '''''Classification Method'''''. This can be done by setting a '''''Positive extractor''''' on the '''Document Type''' object of a '''Content Model'''. If the extractor yields a result, the document will be classified as that '''Document Type'''. Generally, this will "win out" over the training weightings, because the '''''Positive Extractor's''''' confidence result (as a percentage value) will be higher than the document's similarity to the trained examples (as a percentage value) for a '''Document Type'''.		Furthermore, a [[Rules-Based]] approach can be taken in combination with the training based approach, when using the ''Lexical'' '''''Classification Method'''''. This can be done by setting a '''''Positive extractor''''' on the '''Document Type''' object of a '''Content Model'''. If the extractor yields a result, the document will be classified as that '''Document Type'''. Generally, this will "win out" over the training weightings, because the '''''Positive Extractor's''''' confidence result (as a percentage value) will be higher than the document's similarity to the trained examples (as a percentage value) for a '''Document Type'''.

	This way, if you have a value that can be extracted that you know is going to be on a '''Document Type''', you can take advantage of setting a '''''Positive Extractor''''' on the '''Document Type''' to classify them. For example, document titles are often used as "rules". If you can extract text to match a title to a corresponding '''Document Type''', this is often a quick and easy way to classify a document. But, if that extractor fails for whatever reason (because of bad OCR or a new title not matching the extractor's regex), you have training data which can act as a backup classification method.		This way, if you have a value that can be extracted that you know is going to be on a '''Document Type''', you can take advantage of setting a '''''Positive Extractor''''' on the '''Document Type''' to classify them. For example, document titles are often used as "rules". If you can extract text to match a title to a corresponding '''Document Type''', this is often a quick and easy way to classify a document. But, if that extractor fails for whatever reason (because of bad OCR or a new title not matching the extractor's regex), you have training data which can act as a backup classification method.

	Many of the best classification strategies involve combining the training-based ''Lexical'' method with a rules-based approach.		Many of the best classification strategies involve combining the training-based ''Lexical'' method with a rules-based approach.

Randallkinard at 21:45, 21 November 2024

2024-11-21T21:45:40Z

← Older revision		Revision as of 16:45, 21 November 2024
Line 116:		Line 116:

	Many of the best classification strategies involve combining the training-based ''Lexical'' method with a rules-based approach.		Many of the best classification strategies involve combining the training-based ''Lexical'' method with a rules-based approach.

	~~== Glossary ==~~
	~~<u><big>'''Batch Folder'''</big></u>: {{#lst:Glossary\|Batch Folder}}~~

	~~<u><big>'''Batch Process Step'''</big></u>: {{#lst:Glossary\|Batch Process Step}}~~

	~~<u><big>'''Batch'''</big></u>: {{#lst:Glossary\|Batch}}~~

	~~<u><big>'''Classification Method'''</big></u>: {{#lst:Glossary\|Classification Method}}~~

	~~<u><big>'''Classification'''</big></u>: {{#lst:Glossary\|Classification}}~~

	~~<u><big>'''Classify'''</big></u>: {{#lst:Glossary\|Classify}}~~

	~~<u><big>'''Content Model'''</big></u>: {{#lst:Glossary\|Content Model}}~~

	~~<u><big>'''Content Type'''</big></u>: {{#lst:Glossary\|Content Type}}~~

	~~<u><big>'''Data Extraction'''</big></u>: {{#lst:Glossary\|Data Extraction}}~~

	~~<u><big>'''Data Type'''</big></u>: {{#lst:Glossary\|Data Type}}~~

	~~<u><big>'''Document Type'''</big></u>: {{#lst:Glossary\|Document Type}}~~

	~~<u><big>'''Extract'''</big></u>: {{#lst:Glossary\|Extract}}~~

	~~<u><big>'''Form Type'''</big></u>: {{#lst:Glossary\|Form Type}}~~

	~~<u><big>'''Lexical'''</big></u>: {{#lst:Glossary\|Lexical}}~~

	~~<u><big>'''OCR'''</big></u>: {{#lst:Glossary\|OCR}}~~

	~~<u><big>'''Reference'''</big></u>: {{#lst:Glossary\|Reference}}~~

	~~<u><big>'''Rules-Based'''</big></u>: {{#lst:Glossary\|Rules-Based}}~~

	~~<u><big>'''TF-IDF'''</big></u>: {{#lst:Glossary\|TF-IDF}}~~

Randallkinard at 14:55, 27 August 2024

2024-08-27T14:55:46Z

@@ Line 4: / Line 4: @@
 Classification is then performed using the '''[[Classify]]''' activity, using the trained examples and ''Lexical'' property configuration on a '''[[Content Model]]'''.
 == About ==
@@ Line 153: / Line 116: @@
 Many of the best classification strategies involve combining the training-based ''Lexical'' method with a rules-based approach.

Randallkinard at 15:16, 10 May 2024

2024-05-10T15:16:40Z

@@ Line 4: / Line 4: @@
 Classification is then performed using the '''[[Classify]]''' activity, using the trained examples and ''Lexical'' property configuration on a '''[[Content Model]]'''.
 == About ==
 ''Lexical'' classification can be enabled and configured on any '''Content Model''' object.  To do so, select the '''''Classification Method''''' property and select ''Lexical''.

Randallkinard at 14:34, 22 April 2024

2024-04-22T14:34:43Z

← Older revision		Revision as of 09:34, 22 April 2024
Line 1:		Line 1:
	{{AutoVersion}}		{{AutoVersion}}
	~~<section begin="glossary" />~~
	<blockquote>		<blockquote>{{#lst:Glossary\|Lexical}}</blockquote>
	~~The ''~~Lexical'' '''''[[Classification Method]]''''' is one of three methods of classifying documents available to Grooper. This method classifies documents according to their text content, obtained from [[OCR]] or extracted native PDF text (via the [[Recognize]] activity). It uses a [[Training-Based Approaches to Document Classification\|training-based approach]] to "teach" Grooper to classify a document from trained examples of the Document Type.
	</blockquote>
	~~<section end="glossary" />~~
	Classification is then performed using the '''[[Classify]]''' activity, using the trained examples and ''Lexical'' property configuration on a '''[[Content Model]]'''.		Classification is then performed using the '''[[Classify]]''' activity, using the trained examples and ''Lexical'' property configuration on a '''[[Content Model]]'''.

Dsmith: /* Feature Extractor */

2024-04-02T19:25:59Z

Feature Extractor

@@ Line 100: / Line 100: @@
+|
 Any '''Data Type''' can be a '''''Text Feature Extractor'''''.  You can customize this extractor however best suits your document classification needs.  However, there are a few pre-built feature extractors that ship with every Grooper install.  You can find them in the '''Data Extraction''' folder and the following folder path:  '''Data Types > Downloads > Features'''.
 |}

Dsmith: /* Document Frequency Mode */

2024-04-02T19:25:33Z

Document Frequency Mode

← Older revision		Revision as of 14:25, 2 April 2024
Line 131:		Line 131:

	For a more in depth explanation of TF-IDF, visit the [[TF-IDF]] article.		For a more in depth explanation of TF-IDF, visit the [[TF-IDF]] article.

	~~==== Document Frequency Mode ====~~
	Different document types can share terminology. Such is the nature of language. So, how can we help Grooper tell one Document Type from another? The frequency of the terms. That's where Document Frequency comes in. Document Frequency comes in two modes: Normal and Smooth. In Normal mode, words that show up multiple times throughout different Document Types, like stop words, are given a lower weighting, often of zero, so that they will not interfere with Classification. In Smooth mode, those commonly occurring words do have weight, and stand to affect classification.
	~~<br>~~
	~~<br>~~
	~~These weightings are calculated through a term frequency formula. For more information on the math behind said calculation, see here: [https://wiki.grooper.com/index.php?title=2023:TF-IDF]~~
	~~<br>~~
	~~<br>~~
	Suffice it to say, the term frequency is calculated using a logarithmic function. The main difference between Normal and Smooth mode is that Normal converges to zero at a quicker rate than Smooth. Which mode is better? Whichever is preferred. However, one might need to keep in mind what kind of documents they're dealing with: structured or unstructured? In some cases, omitting frequently occurring terms, like stop words by using Normal Frequency Mode might do more harm than good with Classification.

	=== Mixed Classification: Combining Training-Based and Rules-Based Approaches ===		=== Mixed Classification: Combining Training-Based and Rules-Based Approaches ===

Dsmith: /* Logarithmic Term Frequency */

2024-04-02T19:25:18Z

Logarithmic Term Frequency

← Older revision		Revision as of 14:25, 2 April 2024
Line 131:		Line 131:

	For a more in depth explanation of TF-IDF, visit the [[TF-IDF]] article.		For a more in depth explanation of TF-IDF, visit the [[TF-IDF]] article.

	~~==== Logarithmic Term Frequency ====~~
	Even amongst the same Document Type, not every document will have the same amount of pages. One Document may have ten pages, while another may have one hundred. Naturally, this could cause problems with term frequency. A longer document will have see more frequent uses of terms used for Classification. Higher frequency causes terms to have less weight. When terms have less weight, this results in documents potentially being classified incorrectly.
	~~<br>~~
	~~<br>~~
	So, what to do? Simple: Switch the Term Frequency Mode from Normal to Logarithmic. In Normal TF-IDF, Grooper calculates the term weightings independently of document page count. With Logarithmic, varying page count between documents is taken into consideration, and the term frequency is scaled logarithmically. After all, when it comes to Classification, it's sometimes more important that the term even appeared on the document, as opposed to how many times it appeared on the document.
	~~<br>~~
	~~<br>~~
	~~For more information on the math behind Logarithmic Term Frequency Mode, and the mode in general, see here: [https://wiki.grooper.com/index.php?title=2023:TF-IDF#The_Math]~~

	==== Document Frequency Mode ====		==== Document Frequency Mode ====

Dsmith: /* Training Advice */

2024-04-02T19:24:49Z

Training Advice

@@ Line 149: / Line 149: @@
 <br>
 Suffice it to say, the term frequency is calculated using a logarithmic function. The main difference between Normal and Smooth mode is that Normal converges to zero at a quicker rate than Smooth. Which mode is better? Whichever is preferred. However, one might need to keep in mind what kind of documents they're dealing with: structured or unstructured? In some cases, omitting frequently occurring terms, like stop words by using Normal Frequency Mode might do more harm than good with Classification.
 === Mixed Classification:  Combining Training-Based and Rules-Based Approaches ===

← Older revision		Revision as of 16:45, 21 November 2024
Line 116:		Line 116:

	Many of the best classification strategies involve combining the training-based ''Lexical'' method with a rules-based approach.		Many of the best classification strategies involve combining the training-based ''Lexical'' method with a rules-based approach.

	~~== Glossary ==~~
	~~<u><big>'''Batch Folder'''</big></u>: {{#lst:Glossary\|Batch Folder}}~~

	~~<u><big>'''Batch Process Step'''</big></u>: {{#lst:Glossary\|Batch Process Step}}~~

	~~<u><big>'''Batch'''</big></u>: {{#lst:Glossary\|Batch}}~~

	~~<u><big>'''Classification Method'''</big></u>: {{#lst:Glossary\|Classification Method}}~~

	~~<u><big>'''Classification'''</big></u>: {{#lst:Glossary\|Classification}}~~

	~~<u><big>'''Classify'''</big></u>: {{#lst:Glossary\|Classify}}~~

	~~<u><big>'''Content Model'''</big></u>: {{#lst:Glossary\|Content Model}}~~

	~~<u><big>'''Content Type'''</big></u>: {{#lst:Glossary\|Content Type}}~~

	~~<u><big>'''Data Extraction'''</big></u>: {{#lst:Glossary\|Data Extraction}}~~

	~~<u><big>'''Data Type'''</big></u>: {{#lst:Glossary\|Data Type}}~~

	~~<u><big>'''Document Type'''</big></u>: {{#lst:Glossary\|Document Type}}~~

	~~<u><big>'''Extract'''</big></u>: {{#lst:Glossary\|Extract}}~~

	~~<u><big>'''Form Type'''</big></u>: {{#lst:Glossary\|Form Type}}~~

	~~<u><big>'''Lexical'''</big></u>: {{#lst:Glossary\|Lexical}}~~

	~~<u><big>'''OCR'''</big></u>: {{#lst:Glossary\|OCR}}~~

	~~<u><big>'''Reference'''</big></u>: {{#lst:Glossary\|Reference}}~~

	~~<u><big>'''Rules-Based'''</big></u>: {{#lst:Glossary\|Rules-Based}}~~

	~~<u><big>'''TF-IDF'''</big></u>: {{#lst:Glossary\|TF-IDF}}~~