HTTP Import (Import Provider) - Revision history

Dgreenwood at 19:53, 27 June 2025

2025-06-27T19:53:31Z

← Older revision		Revision as of 14:53, 27 June 2025
Line 7:		Line 7:
	\|		\|
	You may download the ZIP(s) below and upload it into your own Grooper environment (version 2025). It contains one or more '''Projects''' with resources used in examples throughout this article.		You may download the ZIP(s) below and upload it into your own Grooper environment (version 2025). It contains one or more '''Projects''' with resources used in examples throughout this article.
	* [[Media:2025 Wiki - HTTP Import ~~Projects~~.zip]]		* [[Media:2025 Wiki - HTTP Import Project.zip]]
	\|}		\|}

Dgreenwood at 19:52, 27 June 2025

2025-06-27T19:52:59Z

← Older revision		Revision as of 14:52, 27 June 2025
Line 1:		Line 1:
	{{AutoVersion}}		{{AutoVersion}}
	<blockquote>{{#lst:Glossary\|HTTP Import}}</blockquote>		<blockquote>{{#lst:Glossary\|HTTP Import}}</blockquote>

			{\|class="download-box"
			\|
			[[File:Asset 22@4x.png]]
			\|
			You may download the ZIP(s) below and upload it into your own Grooper environment (version 2025). It contains one or more '''Projects''' with resources used in examples throughout this article.
			* [[Media:2025 Wiki - HTTP Import Projects.zip]]
			\|}

	== About ==		== About ==

Dgreenwood: /* The Condition HTML command */

2025-06-17T19:13:16Z

The Condition HTML command

← Older revision		Revision as of 14:13, 17 June 2025
Line 239:		Line 239:
	<div class="flowlist">		<div class="flowlist">
	'''After "Condition HTML"'''		'''After "Condition HTML"'''
	* CSS styling is present.		* CSS styling is present. Fonts, font size, element sizes and other page styling affected by CSS style sheets are implemented.
	* Image links are present. Images show up on screen		* Image links are present. Images show up on screen
	</div>		</div>

Dgreenwood: /* The Condition HTML command */

2025-06-17T19:12:09Z

The Condition HTML command

← Older revision		Revision as of 14:12, 17 June 2025
Line 230:		Line 230:
	* Site URL = <nowiki>https://en.wikipedia.org</nowiki>		* Site URL = <nowiki>https://en.wikipedia.org</nowiki>
	* This prepends this URL to any relative links on the page.		* This prepends this URL to any relative links on the page.
	* For example, a relative link image path might be "/wiki/graphics/image.png".		* For example, a relative link image path might be <code>/wiki/graphics/image.png</code>.
	** Its full path would be "<nowiki>https://en.wikipedia.org/wiki/graphics/image/png</nowiki>".		** Its full path would be <code><nowiki>https://en.wikipedia.org/wiki/graphics/image.png</nowiki></code>.
	** The Site URL configuration converts "/wiki/graphics/image.png" to "<nowiki>https://en.wikipedia.org/wiki/graphics/image/png</nowiki>" in the HTML text.		** The Site URL configuration converts <code>/wiki/graphics/image.png</code> to <code><nowiki>https://en.wikipedia.org/wiki/graphics/image/png</nowiki></code> in the HTML text.
	</div>		</div>
	<br clear=all>		<br clear=all>

Dgreenwood: /* The Condition HTML command */

2025-06-17T19:09:40Z

The Condition HTML command

← Older revision		Revision as of 14:09, 17 June 2025
Line 258:		Line 258:
	** This can by used to remove unnecessary or repetitive content from the HTML.		** This can by used to remove unnecessary or repetitive content from the HTML.
	** Examples: Navigation elements, HTML elements containing ads, HTML elements containing repetitive text that does not help inform an AI Assistant's knowledge base.		** Examples: Navigation elements, HTML elements containing ads, HTML elements containing repetitive text that does not help inform an AI Assistant's knowledge base.


			'''Body Selector Example''': In an earlier example we imported several pages from ballotpedia.org.

			[[File:2025 HTTPImport 0X-BodySelector-01.png\|670px\|left\|class=simpleborder simpleshadow]]

			<div class="flowlist">
			The main text content we want is in a specific HTML element. The following selector will select what we need and get rid of most of what we don't want: <code>div#content</code>
			:*<li class="fyi-bullet"> Hint: Use your browser's developer tools to locate the element you want to select. How did we know <code>div#content</code> is the selector that works best for this webpage? We used the developer tools to locate the element as seen in the image on the left.
			</div>
			<br clear=all>

			[[File:2025 HTTPImport 0X-BodySelector-02.png\|670px\|left\|class=simpleborder simpleshadow]]

			<div class="flowlist">
			'''Before "Condition HTML"''':

			There are several unnecessary HTML elements cluttering the page.
			</div>
			<br clear=all>

			[[File:2025 HTTPImport 0X-BodySelector-03.png\|670px\|left\|class=simpleborder simpleshadow]]

			<div class="flowlist">
			'''"Condition HTML" configuration''':
			* Body Selector = div#content
			* This will replace the <code><main></code> element in the HTML with the selected element (the <code><nowiki><div></nowiki></code> element whose <code>id="content"</code>).
			</div>
			<br clear=all>

			[[File:2025 HTTPImport 0X-BodySelector-04.png\|670px\|left\|class=simpleborder simpleshadow]]

			<div class="flowlist">
			'''After "Condition HTML"''':

			Many unnecessary elements are gone (notably the header element up at the top of the page). This is now more streamlined for consumption by an AI Assistant.

			However, there are still more HTML elements that can be removed. Continue reading for an example of Condition HTML's "Removal Selector" feature.
			</div>
			<br clear=all>


			'''Removal Selector Example''': Continuing with the example above, In an earlier example we imported several pages from ballotpedia.org.

			[[File:2025 HTTPImport 0X-RemovalSelector-01.png\|670px\|left\|class=simpleborder simpleshadow]]

			<div class="flowlist">
			Condition HTML's "Removal Selector" can be used on its own or in combination with the Body Selector to further remove unwanted elements. The following selector will remove most of the remaining HTML elements that are not unique text on the web page: <code>table.infobox, table.navbox</code>
			:*<li class="fyi-bullet"> You can enter multiple selectors by separating them with a comma.
			:*<li class="fyi-bullet"> Hint: Use your browser's developer tools to locate the element you want to select. We knew we did not want the navigation box linking to all the articles in image on the left. How did we know <code>table.infobox</code> was the right selector? We used the developer tools to locate the element as seen in the image on the left.
			</div>
			<br clear=all>

			[[File:2025 HTTPImport 0X-RemovalSelector-02.png\|670px\|left\|class=simpleborder simpleshadow]]

			<div class="flowlist">
			'''Before "Condition HTML"''':

			There are several unnecessary HTML elements cluttering the page.

			Removal Selectors are often used to get rid of repetitive content. In this scenario the "Articles" navigation panel is located on every web page we imported. We don't need this panel. We are not using it to navigate a web site. Furthermore, we imported these pages with the end goal of creating an AI Assistant and adding them to its knowledge resources. The text in this panel is does not contain any worthwhile information to feed the AI Assistant.
			</div>
			<br clear=all>

			[[File:2025 HTTPImport 0X-RemovalSelector-03.png\|670px\|left\|class=simpleborder simpleshadow]]

			<div class="flowlist">
			'''"Condition HTML" configuration''':
			* Removal Selector = table.infobox, table.navbox
			* This will remove the HTML elments that match the <code>table.infobox</code> and <code>table.navbox</code> selectors.
			</div>
			<br clear=all>

			[[File:2025 HTTPImport 0X-RemovalSelector-04.png\|670px\|left\|class=simpleborder simpleshadow]]

			<div class="flowlist">
			'''After "Condition HTML"''':

			The HTML elements matched by the "Removal Selector" are removed. In the image on the left, the web page's Articles panel has been removed.
			</div>
			<br clear=all>

Dgreenwood at 16:20, 17 June 2025

2025-06-17T16:20:22Z

← Older revision		Revision as of 11:20, 17 June 2025
Line 1:		Line 1:
	{{AutoVersion}}		{{AutoVersion}}
	<blockquote>~~Under construction. Check back soon for more info!~~</blockquote>		<blockquote>{{#lst:Glossary\|HTTP Import}}</blockquote>

	== About ==		== About ==

Dgreenwood at 16:14, 17 June 2025

2025-06-17T16:14:09Z

Dgreenwood: /* Batch Process considerations */

2025-06-17T16:11:18Z

Batch Process considerations

← Older revision		Revision as of 11:11, 17 June 2025
Line 185:		Line 185:
	** Imports always run "single threaded" in Grooper. By executing this command in a Batch Process, the load operation (fully copying the document to the Grooper Repository) can be executing by multiple processing threads.		** Imports always run "single threaded" in Grooper. By executing this command in a Batch Process, the load operation (fully copying the document to the Grooper Repository) can be executing by multiple processing threads.
	** To do this, the first step in your Batch Process should be "Execute" with the "HTTP Link - Load Content" command added and configured.		** To do this, the first step in your Batch Process should be "Execute" with the "HTTP Link - Load Content" command added and configured.
	* '''HTML Document - Condition HTML''' - This command has two purposes: (1) It can make the HTML document more human-readable in the Document Viewer and (2) it can isolate and remove unwanted HTML elements. [[#The Condition HTML command\|See more in the section below.]]		* '''HTML Document - Condition HTML''' - This command has main two purposes: (1) It can make the HTML document more human-readable in the Document Viewer and (2) it can isolate and remove unwanted HTML elements. [[#The Condition HTML command\|See more in the section below.]]
			* '''HTML Document - Convert to PDF''' - This command converts the HTML page to a PDF document. Grooper can then process the PDF just like it processes any PDF.
			* '''HTML Document - Convert to Text''' - This command converts the HTML page to a TXT document. This is useful for webpages that present as text files ([https://www.govinfo.gov/content/pkg/CFR-2024-title2-vol1/html/CFR-2024-title2-vol1.htm For example this page from the US Code of Federal Regulations hosted on govinfo.gov]). It will get rid of unnecessary HTML elements and leave you with just plain text.

	=== The Condition HTML command ===		=== The Condition HTML command ===

			{\|class="fyi-box"
			\|
			'''FYI'''
			\|
			The Condition HTML command's "Attribute Rules" and "Wrap Rules" have a very niche purpose.

			The "Attribute Rules" and "Wrap Rules" assist in styling HTML elements in HTML documents. These properties were developed for a use case that involved converting XML documents to HTML documents using the XML Transform activity.
			* "Attribute Rules" add attributes to existing HTML elements.
			* "Wrap Rules" wrap text in an HTML element. Text is matched with regular expressions, then wrapped in an HTML element of your choosing.

			'''''Attribute Rules and Wrap Rules do not pertain to HTTP Import and will not be discussed further in this article.'''''
			\|}

	The Condition HTML command has main two purposes:		The Condition HTML command has main two purposes:

Dgreenwood: /* The Condition HTML command */

2025-06-17T16:05:04Z

The Condition HTML command

← Older revision		Revision as of 11:05, 17 June 2025
Line 189:		Line 189:
	=== The Condition HTML command ===		=== The Condition HTML command ===

	The Condition HTML command has two purposes:		The Condition HTML command has main two purposes:

	<big>1. It can make the HTML document more human-readable in the Document Viewer.</big>		<big>1. It can make the HTML document more human-readable in the Document Viewer.</big>

Dgreenwood: /* The Condition HTML command */

2025-06-17T16:04:22Z

The Condition HTML command

← Older revision		Revision as of 11:04, 17 June 2025
Line 234:		Line 234:

	Condition HTML can use CSS selectors to select the elements you want to keep and remove ones you want to discarded.		Condition HTML can use CSS selectors to select the elements you want to keep and remove ones you want to discarded.
			* This is accomplished by configuring the "Body Selector" and/or "Removal Selector" properties.
			* '''Body Selector''': Enter a CSS selector that matches the main text content you want to keep.
			** When set this replaces the <code><body></code> HTML element with the content of the selected element.
			** Example: This is an easy way to remove <code><header></code> and <code><footer></code> elements in scenarios where HTML <code><body></code> element contains a <code><header></code>, <code><main></code>, and <code><footer></code> element.
			*** For the "Body Selector" enter "main".
			*** This will replace the <code><body></code> element with the content of the <code><main></code> element. Effectively, this removes the <code><header></code> and <code><footer></code> elements.
			* '''Removal Selector''': Enter a CSS selector that matches one or more elements you wish to remove.
			** This can by used to remove unnecessary or repetitive content from the HTML.
			** Examples: Navigation elements, HTML elements containing ads, HTML elements containing repetitive text that does not help inform an AI Assistant's knowledge base.

← Older revision		Revision as of 11:14, 17 June 2025
Line 54:		Line 54:
	# Press the "Submit" button to submit the Import Job.		# Press the "Submit" button to submit the Import Job.
	# Assuming you have an Import Watcher service running, the webpage will be imported accordingly.		# Assuming you have an Import Watcher service running, the webpage will be imported accordingly.
	#:*<li class="fyi-bullet"> HTML documents are just HTML text. That text gets rendered on screen in a web browser. This includes text that links to images and CSS style sheets. If the webpage uses relative links to link to these resources (most do), it will look "weird" in the Grooper Document Viewer. The "Condition HTML" command can help here. See the [[#~~Conditioning~~ HTML ~~documents~~\|section on using Condition HTML below]] for more information.		#:*<li class="fyi-bullet"> HTML documents are just HTML text. That text gets rendered on screen in a web browser. This includes text that links to images and CSS style sheets. If the webpage uses relative links to link to these resources (most do), it will look "weird" in the Grooper Document Viewer. The "Condition HTML" command can help here. See the [[#The Condition HTML command\|section on using Condition HTML below]] for more information.

	<div style="position: relative; box-sizing: content-box; max-height: 80vh; max-height: 80svh; width: 100%; aspect-ratio: 1.7777777777777777; padding: 20px 0 40px 0;"><iframe src="https://app.supademo.com/embed/cm9rfeplu00yox00iyort2br6?embed_v=2" loading="lazy" title="HTTP Import - Import a single web page" allow="clipboard-write" frameborder="0" webkitallowfullscreen="true" mozallowfullscreen="true" allowfullscreen style="position: absolute; top: 0; left: 0; width: 100%; height: 100%;"></iframe></div>		<div style="position: relative; box-sizing: content-box; max-height: 80vh; max-height: 80svh; width: 100%; aspect-ratio: 1.7777777777777777; padding: 20px 0 40px 0;"><iframe src="https://app.supademo.com/embed/cm9rfeplu00yox00iyort2br6?embed_v=2" loading="lazy" title="HTTP Import - Import a single web page" allow="clipboard-write" frameborder="0" webkitallowfullscreen="true" mozallowfullscreen="true" allowfullscreen style="position: absolute; top: 0; left: 0; width: 100%; height: 100%;"></iframe></div>
Line 131:		Line 131:
	# Press the "Submit" button to submit the Import Job.		# Press the "Submit" button to submit the Import Job.
	# Assuming you have an Import Watcher service running, one webpage for each relative URL in the "Relative Page URLs" list be imported accordingly.		# Assuming you have an Import Watcher service running, one webpage for each relative URL in the "Relative Page URLs" list be imported accordingly.
	#:*<li class="fyi-bullet"> HTML documents are just HTML text. That text gets rendered on screen in a web browser. This includes text that links to images and CSS style sheets. If the webpage uses relative links to link to these resources (most do), it will look "weird" in the Grooper Document Viewer. The "Condition HTML" command can help here. See the [[#~~Conditioning~~ HTML ~~documents~~\|section on using Condition HTML below]] for more information.		#:*<li class="fyi-bullet"> HTML documents are just HTML text. That text gets rendered on screen in a web browser. This includes text that links to images and CSS style sheets. If the webpage uses relative links to link to these resources (most do), it will look "weird" in the Grooper Document Viewer. The "Condition HTML" command can help here. See the [[#The Condition HTML command\|section on using Condition HTML below]] for more information.

	==== Using Link Selectors ====		==== Using Link Selectors ====
Line 176:		Line 176:
	# Press the "Submit" button to submit the Import Job.		# Press the "Submit" button to submit the Import Job.
	# Assuming you have an Import Watcher service running, one webpage for each link opened will be imported accordingly.		# Assuming you have an Import Watcher service running, one webpage for each link opened will be imported accordingly.
	#:*<li class="fyi-bullet"> HTML documents are just HTML text. That text gets rendered on screen in a web browser. This includes text that links to images and CSS style sheets. If the webpage uses relative links to link to these resources (most do), it will look "weird" in the Grooper Document Viewer. The "Condition HTML" command can help here. See the [[#~~Conditioning~~ HTML ~~documents~~\|section on using Condition HTML below]] for more information.		#:*<li class="fyi-bullet"> HTML documents are just HTML text. That text gets rendered on screen in a web browser. This includes text that links to images and CSS style sheets. If the webpage uses relative links to link to these resources (most do), it will look "weird" in the Grooper Document Viewer. The "Condition HTML" command can help here. See the [[#The Condition HTML command\|section on using Condition HTML below]] for more information.

	== Batch Process considerations ==		== Batch Process considerations ==