View source for 2023:CMIS Import (Import Provider)

{{AutoVersion}}

<blockquote>{{#lst:Glossary|CMIS Import}}</blockquote>

Documents are imported from '''CMIS Connections''' using either the '''''Import Descendants''''' or '''''Import Query Results''''' providers.  These can be used in two ways:

* To perform manual "ad-hoc" imports when creating a new '''[[Batch]]''' in Grooper Dashboard or Grooper Design Studio.
* To perform automated, scheduled imports using one or more '''Import Watcher''' Grooper services.

'''''Import Descendants''''' will import all documents within a designated folder location of a '''[[CMIS Repository]]'''.  '''''Import Query Results''''' allows you to use a query syntax similar to a SQL query (called a CMISQL query) to set conditions for import based on the item's available metadata, such as a documents name, file type, creation date, archive status, or other variables.

<u><big>'''SharePoint'''</big></u>: {{#lst:Glossary|SharePoint}}
{{#lst:CMIS+|cmisplus}}
== About CMIS Import ==
The '''''CMIS Import''''' provider is split into two different '''''Import Providers'''''

* '''''Import Descendants'''''
* '''''Import Query Results'''''

These providers are designed to import files from a folder structure of an on-premise or cloud-based document storage platform.  This is the primary method of '''Batch''' creation when importing digital documents into Grooper to process them with a '''Batch Process'''.  

In order to do this, a few requirements must be met first.

# A '''CMIS Connection''' object must made and configured.  This will connect Grooper to the document storage platform.
#* This may be a connection to a Windows folder, an email inbox, a true CMIS content management system, or other document storage platforms.  What the '''CMIS Connection''' connects to is determined by the '''''CMIS Binding''''' selected when configuring the '''''Connection Type''''' property of the '''CMIS Connection''' object.
# A '''CMIS Repository''' must be imported.  This will create an object Grooper can use to import documents from the folders in the document storage platform.
#* This acts as a "go-between" or a "hub" for Grooper to pull in documents from the content's source. Or, you may think of this as Grooper's representation of a folder location in the document storage platform.

For more information on adding a '''CMIS Connection''' and importing a '''CMIS Repository''', visit the '''[[CMIS Connection]]''' article.

As for the difference between the '''''Import Descendants''''' and '''''Import Query Results''''' providers, you can think of '''''Import Query Results''''' as a more specialized version of '''''Import Descendants'''''.

* '''''Import Descendants''''' is intended to import the full contents of a folder location.  It imports the "descendant" files of a parent folder.
* '''''Import Query Results''''' allows you to selectively import files using a SQL-like query (called a CMISQL query).  Only files returned by the query will be imported.  For example, using an '''''Exchange''''' or '''''IMAP''''' '''CMIS Connection''', you could query an inbox for emails from a specific sender and only import those emails.
** Note:  There are some import filtering capabilities available to '''''Import Descendants''''' as well using a SQL-like query.  However, the CMISQL querying capabilities of '''''Import Query Results''''' are much more robust.  
** That said, only certain '''''CMIS Bindings''''' can take advantage of this increased CMISQL query functionality.  The following '''''CMIS Bindings''''' are '''''not''''' currently suitable for the '''''Import Query Results''''' provider.
*** '''''FTP'''''
*** '''''SFTP'''''
*** '''''NTFS''''' (If the folder path is not indexed by the Windows Search service and/or Windows Search is not running on the storage server)

=== Import Descendants ===
<tabs style="margin:20px">
<tab name="Configuration Panel" style="margin:20px">
=== Configuration Panel ===

{|
[[File:2023_CMIS_Import_02_About_CMIS_Import_02_1_Import_Descendants_Configuration_Panel_01.png]]
|
|-
|valign=top|
[[File:2023_CMIS_Import_02_About_CMIS_Import_02_1_Import_Descendants_Configuration_Panel_02.png]]
|}
</tab>
<tab name="General Settings" style="margin:20px">
=== General Settings ===

{|
[[File:2023_CMIS_Import_02_About_CMIS_Import_02_1_Import_Descendants_General_Settings_00.png]]
|
|-
|valign=top|
[[File:2023_CMIS_Import_02_About_CMIS_Import_02_1_Import_Descendants_General_Settings_01.png]]
|-
|valign=top|
Back top the '''''Import Descendants''''' configuration screen, the '''CMIS Repository''' object is used to point Grooper to this folder location for import.

*The '''''Repository''''' property is configured to assign the '''CMIS Repository''' where the documents are located.
** Here the '''CMIS Repository''' named "Import and Export" connecting to the "Import and Export" folder of the local drive.

*The '''''Base Folder''''' property is configured to traverse the folder structure of the '''CMIS Repository'''.
** Here, we don't want to import ''all'' documents from ''every'' folder in the "Import and Export" folder.  We just want to import from the "Grooper Import Folder".

*The '''''Import Filter''''' property allows you to perform some basic import filtering to selectively choose which documents you want to import.
** <code>SELECT * FROM File</code> is the default filter.  It will import all files from the selected folder location.
** This is a SQL-like query to specify conditions for document import.  However, the '''''Import Query Results''''' provider was created to expand on this functionality and provides more filtering options as well as a simpler interface to perform the query (for the '''''CMIS Bindings''''' capable of utilizing this functionality).

*The '''''Content Type''''' property allows you to optionally assign the incoming documents with a '''Document Type'''.
** You can use this property to assign a default classification for all incoming documents.

[[File:2023_CMIS_Import_02_About_CMIS_Import_02_1_Import_Descendants_General_Settings_02(1).png]]
|}
</tab>
<tab name="Processing Options Settings" style="margin:20px">
=== Processing Options Settings ===
{|

The most important part of the '''''Processing Options''''' property section is the '''''Import Mode''''' property.

The '''''Import Mode''''' property allows control over the connections Grooper makes and/or retains to the imported documents.

For importing, documents contain two important sets of information:

* Content - Images and native text data
* Properties - Metadata associated with the file. Digital information, such as the document's filename, file type, creation date, and more.

Depending on the '''''Import Mode''''' selected all, some, or none of this information will be copied to your Grooper Repository's file store (in the case of the document's content) and database (in the case of the document's properties).  See below for more in depth explanation of each of the '''''Import Mode''''' options.
|
|valign=top|
[[File:2023_CMIS_Import_02_About_CMIS_Import_02_1_Import_Descendants_Processing_Options_Settings_01(2).png]]
|}

''Copy''

* Both properties and content will be loaded. This is a total duplication of the document from its source to your Grooper Repository's local file store.  This is the slowest import mode, because the full content of each document is copied during a ''single-threaded'' import process. As such, this mode is not well-suited for high-volume imports, but provides some useful advantages in low-volume import scenarios.

*For example, ''Copy'' mode allows items to be deleted immediately on import. Also, ''Full'' mode avoids the need for any follow-up content loading operations in the '''Batch Process'''.

*This mode was called ''Full'' in older versions of Grooper.

''Sparse''

*Properties will be loaded, but content will not. This mode is much faster than a ''Full'' import, because no content files are copied into your local Grooper file store. Instead, a link is saved on each Grooper document, and content is retrieved on demand directly from the '''CMIS Repository'''. This type of document is often referred to as a "sparse" document. Sparse documents can be used just like any other document, with the caveat that display and processing speeds may be reduced.  Grooper has to traverse the document link in order to display or process the document's image.

* However, after a ''Sparse'' import, document content can be loaded multi-threaded using the '''Execute''' activity in a '''Batch Process'''.  This can overall lead to importing a document's content faster than a ''Full'' import.  While the 
** Choose ''CMIS Document Link'' as the '''''Object Type''''' and ''Load Content'' as the '''''Command'''''

''Link Only''

* No content or properties will be loaded, making this the fastest import mode. It imports nothing more than a link to each document, and offloads all property and content loading to parallel operations in the Batch Process.

* However, this does not produce a usable document in Grooper.  After a ''LinkOnly'' import, document content ''must'' be loaded using the '''Execute''' activity in a '''Batch Process'''.
** Choose ''CMIS Document Link'' as the '''''Object Type''''' and ''Load Content'' as the '''''Command'''''

* You can think of the ''Link Only'' option as an even sparser sparse import.


See the table bellow for a summary of the '''''Import Mode''''' options.

{|cellpadding=10 cellspacing=5
|'''''Import Mode'''''||'''Speed'''||'''Comments'''
|-valign=top
|''Full''||Slow||Full import of content and their properties.
* Required if deleting content from the source on import.
|-valign=top
||''Sparse''||Fast||Imports a link to the document's source and its properties but not their content.
* This produces a usable document in Grooper without copying the full content into Grooper, saving time upon import.
* This mode is the same as enabling the old '''''Sparse Import''''' property in previous versions.
|-valign=top
||''Link Only''||Fastest||Only imports a link to the document's source.
*Does not produce a usable document. The document's properties must be loaded in a step in a Batch Process.
|}
</tab>
<tab name="Disposition Settings" style="margin:20px">
=== Disposition Settings ===
{|
The '''''Disposition''''' property settings allow you to do something with the source documents after importing them into Grooper, namely delete them, move them, or do nothing and just leave them alone where they came from.  This is often leveraged with the '''Import Watcher''' Grooper service to prevent repeatedly importing the same document.

In our example here, the '''''Move to Folder''''' property is configured to move the PDF documents to a folder named "Imported Documents".
* The folder location you're moving documents to ''must'' be accessible via the connected '''CMIS Repository'''.

If using the ''Full'' '''''Import Mode''''', you can enable the '''''Delete Item''''' property to delete each document after it is imported into the Grooper '''Batch'''.
* This property is ONLY available when choosing the ''Full'' '''''Import Mode.'''''  A sparsely imported document needs to call to the import storage location in order to load the document's image for display or processing.  If you deleted the document upon import, you wouldn't be able to view it or do anything with it.

The '''''Update Properties''''' property allows you to alter the document's property values upon import.  Property values are updated using a list of "key-value pairs" where the "key" is the name of the property and the "value" is what change you want to make to that property. You can type one entry per line in the format <code>key=value</code>.
* Examples:
* <code>Archive=true</code> Sets the archive attribute on a file
* <code>Status=PENDING</code> Sets the "Status" field on ApplicationXtender documents.
* <code>Imported=true</code> Sets the "Imported" field on SharePoint documents.
* <code>IsRead=true</code> Sets the "IsRead" flag on an Exchange message.
|
[[File:2023_CMIS_Import_02_About_CMIS_Import_02_1_Import_Disposition_Settings_01.png]]
|}
</tab>
<tab name="Batch Creation Settings" style="margin:20px">

=== Batch Creation Settings ===
{|
It's likely you're importing documents because you want to run them through a '''Batch Process'''.  The '''''Batch Creation''''' property settings allow you to define which '''Batch Process''' you wish to use to process the imported documents.

This is done using the '''''Starting Step''''' property, selecting a '''Batch Process Step''' in a '''Batch Process''' from the published '''Batch Processes''' in the Grooper Repository.  Upon import, a new '''Batch''' is created with each document as a '''Batch Folder''', and the selected '''Batch Process''' assigned to the '''Batch'''.

There are also further properties to control '''Batch''' creation.  You can limit the number of documents imported per '''Batch''' using the '''''Maximum Items per Batch''''' property.  By default, new '''Batches''' are named with a date/time stamp.  However, the '''''Batch Name Prefix''''' allows you to tack on a prefix to the '''Batch's''' name for easier identification.  The '''''Start Paused''''' property will automatically trigger the '''Batch Process''' if set to ''False''.
|
[[File:2023_CMIS_Import_02_About_CMIS_Import_02_1_Import_Descendants_Batch_Creation_Settings_01.png]]
|}
</tab>
</tabs>

=== Import Query Results ===
<tabs name="margin:20px">
<tab name="The Same But Different" style="margin:20px">
=== The Same, But Different ===

The '''''Import Query Results''''' provider's configuration panel is almost identical to the '''''Import Descendants''''' provider's configuration panel.  ''Both'' providers share the same '''''Processing Options''''', '''''Disposition''''', and '''''Batch Creation''''' property settings.  See the [[#Import Descendants|Import Descendants]] section for brief descriptions of these property sections.

{|
<br>
The big difference between the two providers is the highlighted '''''CMIS Query''''' property.  This allows users to enter a SQL-like query (called a CMISQL query) to selectively import documents from their source, based on certain metadata properties.  Only files returned by the query will be imported.  
* For example, you may want to only import documents of a certain file type(s).  You could include the file extension(s) as the query condition (or one of many conditions).  
* For another example, you can use CMISQL queries to easily filter email messages when importing from an inbox.  If you only wanted to import messages from a certain sender, from an certain folder, with a certain subject line and only ones that have not been read, you could filter out any emails that didn't meet those query conditions by comparing metadata properties (like "Sender" and "Subject") to your criteria.

{|class="attn-box"
|
'''&#9888;'''
|Only certain external storage platforms are currently queryable with the '''''CMIS Query''''' property.  The following '''''CMIS Binding''''' sources '''''cannot''''' be queried currently.  As such, they are '''''not''''' suitable for '''''Import Query Results'''''.  You should instead use '''''Import Descendants''''' for the following '''''CMIS Bindings'''''.

:&bull; '''''FTP'''''
:&bull; '''''SFTP'''''
:&bull; '''''NTFS''''' (If the folder path is not indexed by the Windows Search service and/or Windows Search is not running on the storage server)
|}
|
[[File:2023_CMIS_Import_02_About_CMIS_Import_02_3_Import_Query_Results_The_Same_But_Different_01.png]]
|
<br>
<br>
|valign=top|
Just like with '''''Import Descendants''''', there are some minimum requirements before configuring '''''Import Query Results'''''.  A '''CMIS Connection''' object must be created and a '''CMIS Repository''' must be imported.
<br>
|
[[File:2023_CMIS_Import_02_About_CMIS_Import_02_3_Import_Query_Results_The_Same_But_Different_02.png]]
|-
<br>
<br>
|valign=top|
|
[[File:2023_CMIS_Import_02_About_CMIS_Import_02_3_Import_Query_Results_The_Same_But_Different_03.png]]
|}
</tab>
<tab name="CMIS Query Configuration" style="margin:20px">
=== CMIS Query Configuration ===
Upon pressing the ellipsis button at the end of the '''''CMIS Query''''' property, the '''''CMIS Query Editor''''' window will appear.

This interface allows you to configure the CMISQL query based on available metadata from the '''''CMIS Binding'''''.  For example, the '''''Exchange''''' binding has a selection of queryable metadata for email messages, such as the email's subject, sender and date the message was received.
<br>
<br>
{|
[[File:2023_CMIS_Import_02_About_CMIS_Import_02_3_Import_Query_Results_The_Same_But_Different_04.png]]
|
For an in depth explanation of the '''''CMIS Query Editor''''' and how to use it to craft a CMISQL query, please visit the [[CMIS Query]] article.
|}
</tab>
</tabs>

== Version Differences ==
=== New CMIS Query Editor (2021) ===
'''Grooper 2021''' introduced a new and improved '''''CMIS Query Editor''''''.  This editor was designed to simplify construction of CMISQL queries using a property grid.  For more information, please visit the [[CMIS Query]] article.

=== Box Integration (2.90) ===
'''Grooper 2.9''' sees the addition of the [https://www.box.com/home Box.com] document storage platform into the CMIS fold via the '''''[[Box (CMIS Binding)]]'''''.

=== Legacy Providers (2.72) ===
Old import and export providers should be replaced with this new functionality.  While Grooper's older import and export providers are available as "Legacy Import" and "Legacy Export" providers, these components are depreciated.  They will still function but will no longer be upgraded in future versions of Grooper.

Grooper can import documents using '''CMIS Connections''' via '''''Import Descendants''''' and '''''Import Query Results'''''.  Grooper can export via the '''''CMIS Export''''' providers, '''''[[CMIS Export#Mapped Export|Mapped Export]]''''' and '''''[[CMIS Export#Unmapped Export|Unmapped Export]]'''''.

=== New Connection Types (2.72) ===
By creating the [[CMIS+]] architecture, we have been able to create new connections between Grooper and content management systems.  Grooper can now connect to Microsoft OneDrive, SharePoint, and Exchange via new '''''CMIS Bindings'''''.  Since these were created as '''''CMIS Bindings''''', they can be used by the '''''CMIS Import''''' and '''''CMIS Export''''' providers.  Instead of having to create three new import providers and three new export providers for a total of six brand new components, we can use the already established CMIS import and export providers in the CMIS+ framework.  A user can create a '''CMIS Connection''' using the '''''OneDrive''''', '''''SharePoint''''' or '''''Exchange''''' bindings, and use the same import and export providers for them as any of the other '''''CMIS Bindings'''''.

This will also allow Grooper to create '''''CMIS Bindings''''' to connect to currently unavailable content management systems in the future much quicker and easier.

=== Import Mode (2.72) ===
In version 2.72 the '''''Import Mode''''' property replaces previous versions' '''''Sparse Import''''' property.

=== Import Disposition (2.72) ===
2.72 adds the '''''Import Disposition''''' property to '''''CMIS Import'''''.  This allows you to change your documents disposition upon importing them into Grooper.  You can delete them, move them to a folder, or update one or more properties on the document itself.  This can be leveraged with '''Import Watcher''' to prevent repeatedly importing the same document.

== Glossary ==
<u><big>'''AppXtender'''</big></u>: {{#lst:Glossary|AppXtender}}

<u><big>'''Batch Folder'''</big></u>: {{#lst:Glossary|Batch Folder}}

<u><big>'''Batch Process Step'''</big></u>: {{#lst:Glossary|Batch Process Step}}

<u><big>'''Batch Process'''</big></u>: {{#lst:Glossary|Batch Process}}

<u><big>'''Batch'''</big></u>: {{#lst:Glossary|Batch}}

<u><big>'''Box'''</big></u>: {{#lst:Glossary|Box}}

<u><big>'''CMIS Connection Type'''</big></u>: {{#lst:Glossary|CMIS Connection Type}}

<u><big>'''CMIS Connection'''</big></u>: {{#lst:Glossary|CMIS Connection}}

<u><big>'''CMIS Export'''</big></u>: {{#lst:Glossary|CMIS Export}}

<u><big>'''CMIS Import'''</big></u>: {{#lst:Glossary|CMIS Import}}

<u><big>'''CMIS Query'''</big></u>: {{#lst:Glossary|CMIS Query}}

<u><big>'''CMIS Repository'''</big></u>: {{#lst:Glossary|CMIS Repository}}

<u><big>'''CMIS+'''</big></u>: {{#lst:Glossary|CMIS+}}

<u><big>'''CMIS'''</big></u>: {{#lst:Glossary|CMIS}}

<u><big>'''Content Type'''</big></u>: {{#lst:Glossary|Content Type}}

<u><big>'''Document Type'''</big></u>: {{#lst:Glossary|Document Type}}

<u><big>'''Exchange'''</big></u>: {{#lst:Glossary|Exchange}}

<u><big>'''Execute'''</big></u>: {{#lst:Glossary|Execute}}

<u><big>'''Export'''</big></u>: {{#lst:Glossary|Export}}

<u><big>'''FTP'''</big></u>: {{#lst:Glossary|FTP}}

<u><big>'''Grooper Repository'''</big></u>: {{#lst:Glossary|Grooper Repository}}

<u><big>'''IMAP'''</big></u>: {{#lst:Glossary|IMAP}}

<u><big>'''Import Descendants'''</big></u>: {{#lst:Glossary|Import Descendants}}

<u><big>'''Import Provider'''</big></u>: {{#lst:Glossary|Import Provider}}

<u><big>'''Import Query Results'''</big></u>: {{#lst:Glossary|Import Query Results}}

<u><big>'''NTFS'''</big></u>: {{#lst:Glossary|NTFS}}

<u><big>'''OneDrive'''</big></u>: {{#lst:Glossary|OneDrive}}

<u><big>'''Repository'''</big></u>: {{#lst:Glossary|Repository}}

<u><big>'''Service'''</big></u>: {{#lst:Glossary|Service}}

<u><big>'''SFTP'''</big></u>: {{#lst:Glossary|SFTP}}