TF-IDF (Concept) - Revision history

Randallkinard at 17:06, 28 August 2024

2024-08-28T17:06:30Z

← Older revision		Revision as of 12:06, 28 August 2024
Line 1:		Line 1:
	{{~~Migrated2023~~}}		{{Migrated}}
	{{2023:{{PAGENAME}}}}		{{2023:{{PAGENAME}}}}

Randallkinard: Randallkinard moved page TF-IDF to TF-IDF (Concept): new naming convention

2024-04-18T16:38:51Z

Randallkinard moved page TF-IDF to TF-IDF (Concept): new naming convention

← Older revision	Revision as of 11:38, 18 April 2024
(No difference)

Dgreenwood: Removed redirect to TF-IDF - 2023

2024-01-03T16:22:04Z

Removed redirect to TF-IDF - 2023

← Older revision		Revision as of 11:22, 3 January 2024
Line 1:		Line 1:
	~~#REDIRECT[[TF-IDF -~~ 2023]]		{{Migrated2023}}
			{{2023:{{PAGENAME}}}}

Rpatton: Replaced content with "#REDIRECTTF-IDF - 2023"

2023-10-17T21:16:15Z

Replaced content with "#REDIRECTTF-IDF - 2023"

Show changes

Rpatton: Redirected page to TF-IDF - 2023

2023-10-17T14:43:07Z

Redirected page to TF-IDF - 2023

← Older revision		Revision as of 09:43, 17 October 2023
Line 1:		Line 1:
			#REDIRECT[[TF-IDF - 2023]]

	<onlyinclude>		<onlyinclude>
	<blockquote style="font-size:14pt">		<blockquote style="font-size:14pt">

Dgreenwood at 19:12, 12 December 2022

2022-12-12T19:12:40Z

← Older revision		Revision as of 14:12, 12 December 2022
Line 286:		Line 286:
	* [https://grooper.bisok.com/Documentation/2.90/Main/HTML5/index.htm#t=Object_Reference/Classification_Method/Lexical.htm Lexical (Classification Method)]		* [https://grooper.bisok.com/Documentation/2.90/Main/HTML5/index.htm#t=Object_Reference/Classification_Method/Lexical.htm Lexical (Classification Method)]
	** Contains information about the TF-IDF variants in the "Weighting Options" section.		** Contains information about the TF-IDF variants in the "Weighting Options" section.

			[[Category:Articles]]

Dgreenwood at 14:49, 24 August 2022

2022-08-24T14:49:00Z

← Older revision		Revision as of 09:49, 24 August 2022
Line 28:		Line 28:

	For TF-IDF, the fact that you see a term more frequently is an important feature of that document. The more time that term (also known as feature) repeats on a document, the higher that term's (or feature's) weighting value.		For TF-IDF, the fact that you see a term more frequently is an important feature of that document. The more time that term (also known as feature) repeats on a document, the higher that term's (or feature's) weighting value.

			==== <span style="color:#662d91">Defining Term Frequency</span> ====

			There are multiple ways of defining frequency, including:

			* The number of times the word appears in a document.
			** This is the "raw count" of a term/feature.
			* The term frequency adjusted for the length of the document.
			** This is the raw count of feature occurrences divided by the number of words in a document. Since a term may appear much more times in longer documents than shorter ones, this is a way of normalizing the term frequency value based on the size of the document.
			** We will refer to this as "normal" term frequency. This is the default way of calculating term frequency in Grooper.
			* Logarithmically scaled frequency.
			** Logarithmic scaling reduces the weighting value for terms appearing multiple times.
			** This variant is used in situations where it's more important ''that'' a feature appears in a document, and not as much ''how many times'' it appears. More frequent terms will still have a higher weighting value, but the term frequency is scaled in such a way that value will be ''less'' than it would under normal term frequency.
			** This is a supported term frequency variant in Grooper. See the [[#Logarithmic Term Frequency]] section of this article for more information.
			* Augmented frequency
			** Augmented frequency is calculated by dividing a terms raw frequency by the raw frequency of the most frequently occurring term in the document.
			** This is done to prevent a bias towards longer documents and is useful if you're attempting to classify a variety of documents who have wildly different lengths as the same class.
			** This is a supported term frequency variant in Grooper. See the [[#Augmented Term Frequency]] section of this article for more information.

	=== Inverse Document Frequency ===		=== Inverse Document Frequency ===

	But what about ~~Purchase Orders~~? ~~You’ll~~ see terms like ~~“invoice number” on those kinds~~ of ~~documents too~~. How does the algorithm distinguish between terms that are common to multiple document types in the set? That is where the second part of the equation, “Inverse Document Frequency”, comes into play.		But what about a sales contract document? You may very well see terms like "invoice date" or "invoice total" in a sales contract, or other terms relating to the purchase and sale of goods. How does the algorithm distinguish between terms that are common to multiple document types in the set? That is where the second part of the equation, “Inverse Document Frequency”, comes into play.

	Inverse ~~Document Frequency~~ is a statistical interpretation or measure of ''term-specificity'' in a set ~~of documents~~. Take an extreme example like the word “the”. That word is so common if we ''just'' used term frequency, we’d end up giving the word “the” much more importance than it should have. In fact, the word “the” should have little or no weight. It’s not specific to one document or another and is therefore not helpful when it comes to identifying one over the other. Inverse Document Frequency mitigates features that are common to multiple documents in the set.		Inverse document frequency is the measure of how common or uncommon a word is across a set of different types of documents (aka a "corpus"). It is a statistical interpretation or measure of ''term-specificity'' in a document set. Take an extreme example like the word “the”. That word is so common if we ''just'' used term frequency, we’d end up giving the word “the” much more importance than it should have. In fact, the word “the” should have little or no weight. It’s not specific to one document or another and is therefore not helpful when it comes to identifying one over the other. Inverse Document Frequency mitigates features that are common to multiple documents in the set.

	If you can find terms that appear frequently on one type of document that are ''also'' specific to that document type in the set, there’s a good chance if you see those terms on a different or unknown document, you’re looking at the same kind of document.		If you can find terms that appear frequently on one type of document that are ''also'' specific to that document type in the set, there’s a good chance if you see those terms on a different or unknown document, you’re looking at the same kind of document.

	This is essentially the logic behind using TF-IDF to classify documents. The algorithm will score frequent terms (or features) in a document with a higher weighting value, but it will then decrease the term’s weighting value if that term appears frequently across multiple documents in the set. So yes, ~~Purchase Orders~~ might have “invoice-y” terms on them, but the value of ~~“purchase order~~-y” terms specific to ~~Purchase Orders~~ are going to be much more important when it comes time to classify the document.		This is essentially the logic behind using TF-IDF to classify documents. The algorithm will score frequent terms (or features) in a document with a higher weighting value, but it will then decrease the term’s weighting value if that term appears frequently across multiple documents in the set. So yes, a sales contract might have “invoice-y” terms on them, but the value of “contract-y” terms specific to sales contracts are going to be much more important when it comes time to classify the document.

			By taking the inverse document frequency, you can minimize the weighting of frequent terms (common to all documents of all types) while making infrequent terms (more unique or specific to one document type) have a higher impact.

	=== How is TF-IDF Used in Machine Learning? ===		=== How is TF-IDF Used in Machine Learning? ===
Line 208:		Line 228:
	=== Logarithmic Term Frequency ===		=== Logarithmic Term Frequency ===

	Normal ~~(or Raw)~~ Term Frequency assigns a higher weight according to how many times the term appears on the document. If Document A contains twenty occurrences of the term, and Document B contains only one, it will carry 20 times the weight in Document A as it does in Document B. Sometimes, this may be an important feature to document classification. However, sometimes it can be a hinderance.		Normal Term Frequency assigns a higher weight according to how many times the term appears on the document. If Document A contains twenty occurrences of the term, and Document B contains only one, it will carry 20 times the weight in Document A as it does in Document B. Sometimes, this may be an important feature to document classification. However, sometimes it can be a hinderance.

	Often it is more important ''that'' you encountered the term on a document and not necessarily ''how many'' times it appears on an individual document. For example, a Memorandum of Oil and Gas Lease is a short form version of an Oil and Gas Lease. It serves as an official notice recorded with a county clerk of the Oil and Gas Lease. Memorandums of Lease will contain language similar to the Lease itself, including the word "lease". The term may appear frequently on the Memorandum as well as the Lease. However, the term "memorandum" may only appear once or twice. Even though it appears ''less'' often from a term like "lease", it will be a very good identifier you're looking at a Memorandum and not a Lease. However, with Normal Term Frequency, this term (being a less frequent term) may get a fairly low TF score, causing a Memorandum to share a high degree of similarity with a Lease.		Often it is more important ''that'' you encountered the term on a document and not necessarily ''how many'' times it appears on an individual document. For example, a Memorandum of Oil and Gas Lease is a short form version of an Oil and Gas Lease. It serves as an official notice recorded with a county clerk of the Oil and Gas Lease. Memorandums of Lease will contain language similar to the Lease itself, including the word "lease". The term may appear frequently on the Memorandum as well as the Lease. However, the term "memorandum" may only appear once or twice. Even though it appears ''less'' often from a term like "lease", it will be a very good identifier you're looking at a Memorandum and not a Lease. However, with Normal Term Frequency, this term (being a less frequent term) may get a fairly low TF score, causing a Memorandum to share a high degree of similarity with a Lease.

Dgreenwood: /* Understanding TF-IDF with an Example */

2022-06-17T19:13:22Z

Understanding TF-IDF with an Example

Dgreenwood: /* Introduction: What is TF-IDF? */

2022-06-17T19:11:31Z

Introduction: What is TF-IDF?

← Older revision		Revision as of 14:11, 17 June 2022
Line 13:		Line 13:
	TF-IDF statistics increase proportionally to the number of times that a word appears in a document but is offset by the number of documents in the set / corpus that contain that particular word. Put more simply, the TF-IDF stat for a word gets higher the more it’s used.		TF-IDF statistics increase proportionally to the number of times that a word appears in a document but is offset by the number of documents in the set / corpus that contain that particular word. Put more simply, the TF-IDF stat for a word gets higher the more it’s used.

	~~TF-IDF is how Grooper uses machine learning for training-based document classification (via the Lexical method) and data extraction (via the Field Class extractor). In a different example, search~~ engines use variations of TF-IDF as a central tool when scoring and ranking a document’s relevance for user queries.		Search engines use variations of TF-IDF as a central tool when scoring and ranking a document’s relevance for user queries. TF-IDF can also be used for stop-words filtering in various subject fields, such as text classification and summarization. TF-IDF is how Grooper uses machine learning for training-based document classification (via the Lexical method) and data extraction (via the Field Class extractor).

	TF-IDF can also be used for stop-words filtering in various subject fields, such as text classification and summarization.

	One of the simplest ranking functions is calculated by summing the TF-IDF for each query term. Much more sophisticated ranking functions are derived from this simple model.		One of the simplest ranking functions is calculated by summing the TF-IDF for each query term. Much more sophisticated ranking functions are derived from this simple model.

Dgreenwood: /* Introduction: What is TF-IDF? */

2022-06-17T19:09:54Z

Introduction: What is TF-IDF?

← Older revision		Revision as of 14:09, 17 June 2022
Line 20:		Line 20:

	Frequency can also put documents of varying word counts on the same playing field through normalization. This is achieved by dividing the frequency with the total number of words in a document		Frequency can also put documents of varying word counts on the same playing field through normalization. This is achieved by dividing the frequency with the total number of words in a document


	=== Term Frequency ===		=== Term Frequency ===
Line 41:		Line 40:

	This is essentially the logic behind using TF-IDF to classify documents. The algorithm will score frequent terms (or features) in a document with a higher weighting value, but it will then decrease the term’s weighting value if that term appears frequently across multiple documents in the set. So yes, Purchase Orders might have “invoice-y” terms on them, but the value of “purchase order-y” terms specific to Purchase Orders are going to be much more important when it comes time to classify the document.		This is essentially the logic behind using TF-IDF to classify documents. The algorithm will score frequent terms (or features) in a document with a higher weighting value, but it will then decrease the term’s weighting value if that term appears frequently across multiple documents in the set. So yes, Purchase Orders might have “invoice-y” terms on them, but the value of “purchase order-y” terms specific to Purchase Orders are going to be much more important when it comes time to classify the document.

			=== How is TF-IDF Used in Machine Learning? ===

			The algorithms in machine learning use numerical data, which presents a problem, as machine learning with natural language processing deals with text. How to overcome this problem? The text needs to be converted to numerical data through a process called vectorization.

			In TF-IDF vectorization, algorithms calculate a TF-IDF score for each word in the corpus as it relates to that document. That information is then put into a vector, which represents a document as a list of numbers, with one word for every word of the corpus. When you have vectors, you can compare them to other documents to check word similarity. Documents with similar vectors means they have similar relevant words.

			=== Using TF-IDF in Information Retrieval ===

			TF-IDF has several use cases in information retrieval and was created to help with document search. However, search engines also leverage TF-IDF in helping find the most relevant information, with results based on descending relevance. The higher the TF-IDF score means those web pages or files will be ranked higher.

	== Understanding TF-IDF with an Example ==		== Understanding TF-IDF with an Example ==

← Older revision		Revision as of 14:13, 17 June 2022
Line 52:		Line 52:

	When it comes down to the math, the algorithm looks something like this.		When it comes down to the math, the algorithm looks something like this.


	:: <span style="font-size:14pt">A Term’s Weighting Value = '''TF x IDF'''</span>		:: <span style="font-size:14pt">A Term’s Weighting Value = '''TF x IDF'''</span>


	where…		where…


	:: <span style="font-size:14pt">'''TF''' = (# times the term appears in a document) / (total # of terms in the document)</span>		:: <span style="font-size:14pt">'''TF''' = (# times the term appears in a document) / (total # of terms in the document)</span>


	and...		and...


	:: <span style="font-size:14pt">'''IDF''' = log(total # of document types / # of document types with the term in it)</span>		:: <span style="font-size:14pt">'''IDF''' = log(total # of document types / # of document types with the term in it)</span>
Line 77:		Line 82:

	For Document Type A, the Term Frequency of the word "this" is "0.2". There are five words, out of which the word "this" appears once. One divided by five is "0.2". Same for Document Type C. For Document Type B, the math works out to "0.1429".		For Document Type A, the Term Frequency of the word "this" is "0.2". There are five words, out of which the word "this" appears once. One divided by five is "0.2". Same for Document Type C. For Document Type B, the math works out to "0.1429".


	::<code>TF(“this”, Document Type A) = 1/5 = 0.2</code>		::<code>TF(“this”, Document Type A) = 1/5 = 0.2</code>
	::<code>TF(“this”, Document Type B) = 1/7 = 0.1429</code>		::<code>TF(“this”, Document Type B) = 1/7 = 0.1429</code>
	::<code>TF(“this”, Document Type C) = 1/5 = 0.2</code>		::<code>TF(“this”, Document Type C) = 1/5 = 0.2</code>


	Inverse document frequency is calculated against the whole document set. So, this value will be same for all three document types. The IDF value for the word "this" works out to "0" because all three documents contain the word "this". The logarithm of 1 is always 0.		Inverse document frequency is calculated against the whole document set. So, this value will be same for all three document types. The IDF value for the word "this" works out to "0" because all three documents contain the word "this". The logarithm of 1 is always 0.


	::<code>IDF(“this”, Doc Set) = log(3/3) = 0</code>		::<code>IDF(“this”, Doc Set) = log(3/3) = 0</code>


	Last, to find the TF-IDF weighting value for each document type, you just multiply those numbers together.		Last, to find the TF-IDF weighting value for each document type, you just multiply those numbers together.


	::<code>TF-IDF(“this”, Document A) = 0.2 x 0 = 0</code>		::<code>TF-IDF(“this”, Document A) = 0.2 x 0 = 0</code>
	::<code>TF-IDF(“this”, Document B) = 0.1429 x 0 = 0</code>		::<code>TF-IDF(“this”, Document B) = 0.1429 x 0 = 0</code>
	::<code>TF-IDF(“this”, Document C) = 0.2 x 0 = 0</code>		::<code>TF-IDF(“this”, Document C) = 0.2 x 0 = 0</code>


	Again, the term “this” is on all docs. For basic TF-IDF, its IDF value scales its weighting all the way to zero. The term is not specific to one document or the other. So, regardless of the term's frequency value, the TF-IDF weighting score equates to zero. It’s not unique enough to be a good identifier of one document type or another.		Again, the term “this” is on all docs. For basic TF-IDF, its IDF value scales its weighting all the way to zero. The term is not specific to one document or the other. So, regardless of the term's frequency value, the TF-IDF weighting score equates to zero. It’s not unique enough to be a good identifier of one document type or another.

	The term “example” is found a lot on Document Type B but not on Document Type A or on Document Type C. It should have a much higher weighting value for Document Type B than the other two.		The term “example” is found a lot on Document Type B but not on Document Type A or on Document Type C. It should have a much higher weighting value for Document Type B than the other two.


	::<code>TF(“example”, Document A) = 0/5 = 0</code>		::<code>TF(“example”, Document A) = 0/5 = 0</code>
Line 107:		Line 119:
	::<code>TF-IDF(“example”, Document B) = 0.4286 x 0.4771 = 0.2045</code>		::<code>TF-IDF(“example”, Document B) = 0.4286 x 0.4771 = 0.2045</code>
	::<code>TF-IDF(“example”, Document C) = 0 x 0.4771 = 0</code>		::<code>TF-IDF(“example”, Document C) = 0 x 0.4771 = 0</code>


	Sure enough, 0.205 is greater than 0.		Sure enough, 0.205 is greater than 0.

	The term “A” should be more interesting. It’s a weird one. It appears in two of the three documents, and it appears on Document Type A more than Document Type B. So, it’s not totally unique but might be more important to one document vs another. But still not a standout like the term “example” for Document Type C.		The term “A” should be more interesting. It’s a weird one. It appears in two of the three documents, and it appears on Document Type A more than Document Type B. So, it’s not totally unique but might be more important to one document vs another. But still not a standout like the term “example” for Document Type C.


	::<code>TF(“a”, Document A) = 2/5 = 0.4</code>		::<code>TF(“a”, Document A) = 2/5 = 0.4</code>
Line 123:		Line 137:
	::<code>TF-IDF(“a”, Document B) = 0 x 0.1761 = 0</code>		::<code>TF-IDF(“a”, Document B) = 0 x 0.1761 = 0</code>
	::<code>TF-IDF(“a”, Document C) = 0.2 x 0.1761 = 0.0352</code>		::<code>TF-IDF(“a”, Document C) = 0.2 x 0.1761 = 0.0352</code>


	The weightings bear out our ideas for this term. It shouldn’t be considered a feature of Document Type B at all, and its weighting is zero. It has a higher weighting value for Document Type A than Document Type C (0.0704 > 0.0352) because the term appears more frequently on Document Type C. But still, it isn’t as clear an identifier as the word “example” for Document Type B. So the weighting value is smaller (0.205 > 0.0704)		The weightings bear out our ideas for this term. It shouldn’t be considered a feature of Document Type B at all, and its weighting is zero. It has a higher weighting value for Document Type A than Document Type C (0.0704 > 0.0352) because the term appears more frequently on Document Type C. But still, it isn’t as clear an identifier as the word “example” for Document Type B. So the weighting value is smaller (0.205 > 0.0704)