Batch Archiving Guidance - Revision history

Dgreenwood: /* The Archive command and Dispose Batch activity */

2024-09-18T13:27:17Z

The Archive command and Dispose Batch activity

← Older revision		Revision as of 08:27, 18 September 2024
Line 100:		Line 100:
	* Larger volume environments		* Larger volume environments
	*: Because '''''Archive''''' and '''''Dispose Batch''''' can be made part of a '''Batch Process''', their archival operations can be automated. This makes it easier to archive large volumes of '''Batches''' than manually moving them to an archive folder.		*: Because '''''Archive''''' and '''''Dispose Batch''''' can be made part of a '''Batch Process''', their archival operations can be automated. This makes it easier to archive large volumes of '''Batches''' than manually moving them to an archive folder.
	:<li class="attn-bullet"> '''''BE AWARE:''''' To avoid thread-locking complications, you should run '''''Archive''''' or '''''Dispose Batch''''' ''single-threaded''. ~~Install~~ an '''Activity Processing''' service, set its '''''Number of Threads''''' to ''1'' and assign it a dedicated '''Processing Queue'''.		:<li class="attn-bullet"> '''''BE AWARE:''''' To avoid thread-locking complications, you should run '''''Archive''''' or '''''Dispose Batch''''' ''single-threaded''. As threads move '''Batches''' to different folders, Grooper locks the folder while the move operation completes. Problems can occur when multiple threads are attempting to write to the same folders. You should also consider if an '''Import Watcher''' is writing to the same folders you're archiving from.
			:<li class="fyi-bullet"> To run a step "single-threaded", first create a dedicated '''Processing Queue''' for the step. Assign that '''Processing Queue''' to the step's '''''Queue Name''''' property. Then, install an '''Activity Processing''' service, set its '''''Number of Threads''''' to ''1'' and assign it the dedicated '''Processing Queue'''.

Dgreenwood at 13:20, 18 September 2024

2024-09-18T13:20:37Z

← Older revision		Revision as of 08:20, 18 September 2024
Line 1:		Line 1:
			{{AutoVersion}}

	You've got your '''Grooper Repository''' all set up. You've architected your '''Content Model'''. You have a '''Batch Process''' up and running. Documents are coming in and being processed accordingly. But, now that you've reached the end of the '''Batch Process''', what do you do with the '''Batch''' itself?		You've got your '''Grooper Repository''' all set up. You've architected your '''Content Model'''. You have a '''Batch Process''' up and running. Documents are coming in and being processed accordingly. But, now that you've reached the end of the '''Batch Process''', what do you do with the '''Batch''' itself?

Dgreenwood: /* The Archive command and Dispose Batch activity */

2024-09-17T15:01:19Z

The Archive command and Dispose Batch activity

← Older revision		Revision as of 10:01, 17 September 2024
Line 94:		Line 94:

	Either the '''''Archive''''' command or the '''''Dispose Batch''''' activity can be applied manually to '''Batches''' or executed as part of a '''Batch Process'''. The '''''Archive''''' and '''''Dispose Batch''''' methods of archiving '''Batches''' are best suited for the following scenarios.		Either the '''''Archive''''' command or the '''''Dispose Batch''''' activity can be applied manually to '''Batches''' or executed as part of a '''Batch Process'''. The '''''Archive''''' and '''''Dispose Batch''''' methods of archiving '''Batches''' are best suited for the following scenarios.
			* Scanning documents into Grooper rather than importing them
			*: If you're not importing documents, you won't be able to use the '''''Organize By Date''''' feature to automatically organize new '''Batches''' into year/month/day folders. That property only exists on '''''Import Providers'''''.
			* Larger volume environments
			*: Because '''''Archive''''' and '''''Dispose Batch''''' can be made part of a '''Batch Process''', their archival operations can be automated. This makes it easier to archive large volumes of '''Batches''' than manually moving them to an archive folder.
			:<li class="attn-bullet"> '''''BE AWARE:''''' To avoid thread-locking complications, you should run '''''Archive''''' or '''''Dispose Batch''''' ''single-threaded''. Install an '''Activity Processing''' service, set its '''''Number of Threads''''' to ''1'' and assign it a dedicated '''Processing Queue'''.

Dgreenwood: /* The Archive command and Dispose Batch activity */

2024-09-17T14:56:09Z

The Archive command and Dispose Batch activity

← Older revision		Revision as of 09:56, 17 September 2024
Line 93:		Line 93:


	~~The~~ '''''Archive''''' ~~and~~ '''''Dispose Batch''''' ~~methods of archiving~~ '''Batches''' ~~are best suited~~		Either the '''''Archive''''' command or the '''''Dispose Batch''''' activity can be applied manually to '''Batches''' or executed as part of a '''Batch Process'''. The '''''Archive''''' and '''''Dispose Batch''''' methods of archiving '''Batches''' are best suited for the following scenarios.

	~~Either the~~ '''''~~Archive~~''''' ~~command or the~~ '''''~~Dispose Batch~~''''' ~~activity can be applied manually to~~ '''~~Batches~~''' ~~or executed as part~~ of a '''~~Batch Process~~'''.

Dgreenwood: /* The Archive command and Dispose Batch activity */

2024-09-17T14:55:40Z

The Archive command and Dispose Batch activity

← Older revision		Revision as of 09:55, 17 September 2024
Line 84:		Line 84:
	=== The Archive command and Dispose Batch activity ===		=== The Archive command and Dispose Batch activity ===

	The '''''Archive''''' command moves '''Batches''' into a folder and organizes them into three subfolder levels by the year, month and date the '''Batch''' was created. The '''''Dispose Batch''''' activity can dispose of '''Batches''' in several ways, including moving them to a folder in the Test branch. '''''Dispose Batch's''''' '''''Group By''''' property controls how the '''Batch''' is organized into subfolders. Options include by a single year folder only, by a single year folder quarter, by three folder levels reflecting the year/month/day, and more.		The '''''Archive''''' command moves '''Batches''' into a folder and organizes them into three subfolder levels by the year, month and date the '''Batch''' was created. The '''''Dispose Batch''''' activity can "dispose" of '''Batches''' in several ways, including moving them to a folder in the Test branch. Moving '''Batches''' to the Test branch effectively archives them. '''''Dispose Batch's''''' '''''Group By''''' property controls how the '''Batch''' is organized into subfolders. Options include by a single year folder only, by a single year folder quarter, by three folder levels reflecting the year/month/day, and more.

	'''''Archive''''' and '''''Dispose Batch''''' can both move '''Batches''' into a grouped folder structure. What's the difference?		'''''Archive''''' and '''''Dispose Batch''''' can both move '''Batches''' into a grouped folder structure. What's the difference?
Line 91:		Line 91:
	* '''''Archive''''' can only organize '''Batches''' into a year/month/day subfolder structure. '''''Dispose Batch''''' has more subfoldering options in its "'''''Group By'''''" configuration.		* '''''Archive''''' can only organize '''Batches''' into a year/month/day subfolder structure. '''''Dispose Batch''''' has more subfoldering options in its "'''''Group By'''''" configuration.
	* '''''Archive''''' will organize '''Batches''' into a year/month/day subfolder structure based on the date the '''Batch''' was ''created''. '''''Dispose Batch''''' will organize '''Batches''' into a year/month/day subfolder structure based on the date the '''Batch''' was ''archived'' (the day the '''''Dispose Batch''''' activity is executed).		* '''''Archive''''' will organize '''Batches''' into a year/month/day subfolder structure based on the date the '''Batch''' was ''created''. '''''Dispose Batch''''' will organize '''Batches''' into a year/month/day subfolder structure based on the date the '''Batch''' was ''archived'' (the day the '''''Dispose Batch''''' activity is executed).


			The '''''Archive''''' and '''''Dispose Batch''''' methods of archiving '''Batches''' are best suited

	Either the '''''Archive''''' command or the '''''Dispose Batch''''' activity can be applied manually to '''Batches''' or executed as part of a '''Batch Process'''.		Either the '''''Archive''''' command or the '''''Dispose Batch''''' activity can be applied manually to '''Batches''' or executed as part of a '''Batch Process'''.

Dgreenwood: /* Ways to archive Batches */

2024-09-17T13:36:27Z

Ways to archive Batches

← Older revision		Revision as of 08:36, 17 September 2024
Line 53:		Line 53:
	* Use the '''''Archive''''' command to organize '''Batches''' into a year/month/day folder structure. This can be automated in a '''Batch Process''' with the '''''Execute''''' activity.		* Use the '''''Archive''''' command to organize '''Batches''' into a year/month/day folder structure. This can be automated in a '''Batch Process''' with the '''''Execute''''' activity.
	* Use the '''''Dispose Batch''''' activity to move the '''Batches''' to the Test branch. This can be automated as a step in a '''Batch Process'''.		* Use the '''''Dispose Batch''''' activity to move the '''Batches''' to the Test branch. This can be automated as a step in a '''Batch Process'''.


	~~'''''Please note:''''' '''''Archive''''' and '''''Dispose Batch''''' can both move '''Batches''' into a grouped folder structure. What's the difference?~~
	* '''''Archive''''' can move '''Batches''' to ''either'' the Production branch or the Test branch. '''''Dispose Batch''''' can ''only'' move '''Batches''' to the Test branch.
	* '''''Archive''''' has an option to clear the '''Batch's''' job history, deleting the task history for the '''Batch'''. This ensures the tables logging task processing stats don't become overpopulated with historical data. '''''Dispose Batch''''' has no such option.
	* '''''Archive''''' can only organize '''Batches''' into a year/month/day subfolder structure. '''''Dispose Batch''''' has more subfoldering options in its "'''''Group By'''''" configuration.
	* '''''Archive''''' will organize '''Batches''' into a year/month/day subfolder structure based on the date the '''Batch''' was ''created''. '''''Dispose Batch''''' will organize '''Batches''' into a year/month/day subfolder structure based on the date the '''Batch''' was ''archived'' (the day the '''''Dispose Batch''''' activity is executed).

	=== Organize for archiving on import (Our preferred method) ===		=== Organize for archiving on import (Our preferred method) ===
Line 91:		Line 84:
	=== The Archive command and Dispose Batch activity ===		=== The Archive command and Dispose Batch activity ===

	~~===~~ The Archive command ~~===~~		The '''''Archive''''' command moves '''Batches''' into a folder and organizes them into three subfolder levels by the year, month and date the '''Batch''' was created. The '''''Dispose Batch''''' activity can dispose of '''Batches''' in several ways, including moving them to a folder in the Test branch. '''''Dispose Batch's''''' '''''Group By''''' property controls how the '''Batch''' is organized into subfolders. Options include by a single year folder only, by a single year folder quarter, by three folder levels reflecting the year/month/day, and more.

			'''''Archive''''' and '''''Dispose Batch''''' can both move '''Batches''' into a grouped folder structure. What's the difference?
			* '''''Archive''''' can move '''Batches''' to ''either'' the Production branch or the Test branch. '''''Dispose Batch''''' can ''only'' move '''Batches''' to the Test branch.
			* '''''Archive''''' has an option to clear the '''Batch's''' job history, deleting the task history for the '''Batch'''. This ensures the tables logging task processing stats don't become overpopulated with historical data. '''''Dispose Batch''''' has no such option.
			* '''''Archive''''' can only organize '''Batches''' into a year/month/day subfolder structure. '''''Dispose Batch''''' has more subfoldering options in its "'''''Group By'''''" configuration.
			* '''''Archive''''' will organize '''Batches''' into a year/month/day subfolder structure based on the date the '''Batch''' was ''created''. '''''Dispose Batch''''' will organize '''Batches''' into a year/month/day subfolder structure based on the date the '''Batch''' was ''archived'' (the day the '''''Dispose Batch''''' activity is executed).

	~~=== The~~ Dispose Batch ~~command ===~~		Either the '''''Archive''''' command or the '''''Dispose Batch''''' activity can be applied manually to '''Batches''' or executed as part of a '''Batch Process'''.

Dgreenwood: /* Manually archive Batches */

2024-09-17T13:25:26Z

Manually archive Batches

← Older revision		Revision as of 08:25, 17 September 2024
Line 88:		Line 88:
	* Small volume environments.		* Small volume environments.
	*: If you process a fairly small amount of '''Batches''' every day, it may be easy enough to have someone regularly move '''Batches''' to an archive folder as part of their daily/weekly duties. For larger environments, processing a large amount of content every day, this may be tedious or untenable.		*: If you process a fairly small amount of '''Batches''' every day, it may be easy enough to have someone regularly move '''Batches''' to an archive folder as part of their daily/weekly duties. For larger environments, processing a large amount of content every day, this may be tedious or untenable.

			=== The Archive command and Dispose Batch activity ===

	=== The Archive command ===		=== The Archive command ===

	=== The Dispose Batch command ===		=== The Dispose Batch command ===

Dgreenwood: /* Pros and cons: Deleting Batches */

2024-09-17T13:24:57Z

Pros and cons: Deleting Batches

← Older revision		Revision as of 08:24, 17 September 2024
Line 35:		Line 35:

	''Deleting Batches makes it more difficult to reprocess documents''		''Deleting Batches makes it more difficult to reprocess documents''
	: If you delete a '''Batch''', everything done to those documents in Grooper is gone too. If you need to reprocess previously exported content (say to extract new fields from them) ~~and~~ they have been deleted from Grooper, you will need to start all over again at the start of the '''Batch Process'''. ~~If the exported content is already~~ in Grooper, and ~~is still classified, still has OCR data, still has an extracted '''Data Model''' or whatever else the '''Batch Process''' did, you~~		: If you delete a '''Batch''', everything done to those documents in Grooper is gone too. Many steps in a '''Batch Process''' are present to condition a file or group of pages to the point where they are usable documents that can be classified and extracted. For example, '''''Recognize''''' collects machine-readable text that is used downstream by an '''''Extract''''' step. If you need to reprocess previously exported content (say to extract new fields from them) but they have been deleted from Grooper, you will need to start all over again at the start of the '''Batch Process'''. Archiving them in Grooper long term avoids this problem and makes it easier to iterate solution design.

	''You can't use AI Search''		''You can't use AI Search''

Dgreenwood: /* Pros and cons: Deleting Batches */

2024-09-17T13:15:57Z

Pros and cons: Deleting Batches

← Older revision		Revision as of 08:15, 17 September 2024
Line 22:		Line 22:
	''Can save on storage''		''Can save on storage''
	: If your files and data have been exported out of Grooper, keeping content in the Grooper Repository may be unncessicarily duplicating files by keeping them both in their export location and in Grooper.		: If your files and data have been exported out of Grooper, keeping content in the Grooper Repository may be unncessicarily duplicating files by keeping them both in their export location and in Grooper.
	:*<li class="fyi-bullet"> Sparsely importing documents into Grooper can help mitigate this. Sparse documents are not actually loaded into the Grooper file store. They are instead accessed on-demand by a link between Grooper are the external content management system.		:*<li class="fyi-bullet"> Sparsely importing documents into Grooper can help mitigate this in many scenarios. Sparse documents are not actually loaded into the Grooper file store. They are instead accessed on-demand by a link between Grooper are the external content management system.

	''Don't have to manage content in two systems''		''Don't have to manage content in two systems''

Dgreenwood: /* Pros and cons: Deleting Batches */

2024-09-17T13:15:35Z

Pros and cons: Deleting Batches

← Older revision		Revision as of 08:15, 17 September 2024
Line 21:		Line 21:

	''Can save on storage''		''Can save on storage''
	: If your files and data have been exported out of Grooper, keeping content in the Grooper Repository may be unncessicarily duplicating files in Grooper.		: If your files and data have been exported out of Grooper, keeping content in the Grooper Repository may be unncessicarily duplicating files by keeping them both in their export location and in Grooper.
	:*<li class="fyi-bullet"> Sparsely importing documents into Grooper can help mitigate this. Sparse documents are not actually loaded into the Grooper file store. They are instead accessed on-demand by a link between Grooper are the external content management system.		:*<li class="fyi-bullet"> Sparsely importing documents into Grooper can help mitigate this. Sparse documents are not actually loaded into the Grooper file store. They are instead accessed on-demand by a link between Grooper are the external content management system.