CMIS Import (Import Provider)

This article is about the current version of Grooper. Note that some content may still need to be updated.
2025	2023	2021	2.90

CMIS Import refers to two Import Providers used to import content from settings_system_daydream CMIS Repositories: Import Descendants and Import Query Results. CMIS Imports allow users to import from various on-premise and cloud based storage platforms (including Windows folders, Outlook inboxes, Box accounts, AppEnhancer applications and more).

Documents are imported from CMIS Connections using either the Import Descendants or Import Query Results providers. These can be used in two ways:

• To perform manual "ad-hoc" imports when creating a new Batch on the "Imports" page.
• To perform automated, scheduled imports using one or more Import Watcher Grooper services.

Import Descendants will import all documents within a designated folder location of a CMIS Repository. Import Query Results allows you to use a query syntax similar to a SQL query (called a CMISQL query) to set conditions for import based on the item's available metadata, such as a documents name, file type, creation date, archive status, or other variables.

About CMIS+

"CMIS" stands for "Content Management Interoperability Services". It is an open standard that allows different content management systems to inter-operate over the Internet. Grooper expanded on this idea in version 2.72 to create our "CMIS+" architecture. CMIS+ unifies all content platforms under a single framework as if they were traditional CMIS endpoints.

Now, Grooper connects to all available external storage platforms by creating and configuring a CMIS Connection.

• Once a CMIS Connection is created, Grooper can "interoperate" with these platforms.
• "Interoperability " means Grooper has the same access to control the system as a human being does.
• Grooper has a "one-to-one" connection to the platform, allowing full and total control.
• Because we standardize connection to non-CMIS systems, this includes platforms like NTFS file systems (Windows) that are not CMIS servers.

Using this architecture, Grooper is able to create a simpler and more efficient import and export workflow, using a variety of storage platforms.

• You now use CMIS Import providers and CMIS Export for any storage platform you can connect to with a CMIS Connection.
• This also speeds up development for adding new connection types for import/export operations.

Anatomy of a CMIS Connection

When connecting Grooper to external storage platforms, you'll start by creating a CMIS Connection. There are three important parts to understanding a CMIS Connection:

1. The CMIS Connection itself
2. The platform it's connecting to. This is defined by the "CMIS Binding" (aka "connection type") selected for the CMIS Connection's "Connection Settings".
3. Its child CMIS Repositories
   • "Repository" is just a general term for a location where data lives. Different systems refer to "repositories" in different ways.
     • A folder in Windows could be a repository. An email inbox could be a repository. A document library in SharePoint could be a repository. An application in ApplicationEnhancer (formerly ApplicationXtender) could be a repository.
     • "Repository" is a normalized way of referring to various terms used by various storage platforms.

For newer users, the difference between a CMIS Connection and a CMIS Repository can be confusing. The key distinction is as follows:

• CMIS Connections connect to storage platforms.
  • It's the phone number you dial.
  • The specific platform you're connecting to is defined in its "Connection Settings".
• CMIS Repositories represent a location within the connected platform.
  • It's the person on the other end of that phone number you want to talk to.
  • CMIS Repositories represent storage locations (typically folders) in the storage platform. They are added as children to a parent CMIS Connection.
  • The CMIS Repository nodes are what Grooper actually uses when configuring import/export operations.
    • You don't talk to a phone number. You talk to a person.
    • You don't reference the parent CMIS Connection when configuring CMIS Import or CMIS Export. Instead you reference a CMIS Repository.

Basic creation steps

There are three basic steps involved to connect Grooper to external storage platforms:

1. Create a CMIS Connection
2. Configure the "Connection Settings".
   • Choose what platform you want to connect to (the CMIS Binding).
   • Enter the connection settings required to connect to the platform (This will differ from platform to platform)
3. Add child CMIS Repositories by importing the storage locations.

   • Importing a CMIS Repository is not the same as importing documents to a new Batch.
     • "Importing" here is more like importing a reference (or bringing the repository into a framework Grooper can use).
     • Upon importing the CMIS Repository, Grooper has full file access to that location in the storage platform.

CMIS Bindings (aka "connection types")

How you configure a CMIS Connection only differs based on what platform you're connecting to. Connection settings include folder paths, URL addresses or usernames or passwords.

• Example: Connecting to a Windows folder requires a networked folder's UNC path.
• Example: Connecting to a SharePoint site requires a URL address.
• Example: Connecting to a email inbox requires an server host name.
• Example: Connecting to Application Extender, Box, SharePoint, OneDrive, Exchange (Outlook) and more requires a username and password.

Each platform has its own connection requirements. These connection settings and the logic required to interoperate between Grooper and a specific platform are defined by the different "CMIS Binding"

Each CMIS Binding provides the settings and logic to connect Grooper to CMS platforms and file systems for import and export operations.

• Example: The "Exchange" binding contains all the information Grooper uses to connect to Microsoft Exchange email servers (i.e. Outlook inboxes).
• Example: The "AppXtender" binding contains all the information Grooper uses to connect to the ApplicationEnhancer (formerly AppXtender) content management system.
• Example: The "NTFS" binding contains all the information Grooper uses to connect to a Windows file system.
• And so on.

The first step in configuring a CMIS Connection is choosing what platform you want to connect to. You do this by selecting a "CMIS Binding".

• You will commonly hear "CMIS Binding" referred to as a "CMIS connection type" or "connection type".
• Or just "connection", as in an "Exchange connection".

Current CMIS Bindings (aka "connection types")

Grooper can connect to the following storage platforms using below using CMIS Bindings:

Most Commonly Used

• The Microsoft Windows NTFS file system.
• The following Microsoft content platforms
  • The Microsoft Exchange mail server platform.
  • The Microsoft OneDrive cloud storage platform.
  • Microsoft SharePoint sites.

Somewhat Commonly Used

• The ApplicationXtender document management platform.
• The Box cloud storage platform.

Less Commonly Used

• FTP (File Transfer Protocol) and SFTP (SSH File Transfer Protocol) servers.
• IMAP mail servers

Least Used

• Content management systems using CMIS 1.0 or CMIS 1.1 servers.
• The FileBound document management platform.
• The IBM FileNet platform.

About CMIS Import

There are two CMIS Import providers in Grooper.

• Import Descendants
• Import Query Results

Both these providers are used to import files from CMIS Repositories for Batch processing in Grooper. It will import files from a folder structure of an on-premise or cloud-based document storage platform.

• While less common, Import Descendants and Import Query Results can also import folders from CMIS Repositories. However, since importing files is most common, we focus on importing files in this article.

Just like any other Import Provider, Import Descendants and Import Query Results are used to submit "Import Jobs". Import Jobs are how Grooper brings in files from a storage location for processing. For example, it's how PDFs from a Windows folder get into Grooper or messages from an email inbox get into Grooper. When an Import Job runs, Grooper first creates a Batch and then creates a Batch Folder for each imported file. A copy of the file is attached to the Batch Folder. This becomes the Batch Folder's "attachment" and is used when applying activities like "Split Pages".

• When files are imported into Grooper, a link to that file is stored on the Batch Folder. This link maintains a connection between the file's source location and the document in Grooper. This link also makes "Sparse" imports possible. See below for more.

Import Jobs are submitted in one of two ways:

• By a user from the Imports page: Ad-hoc or "user directed" Import Jobs are submitted from the Imports Page, using the "Submit Import Job" button.
• From an Import Watcher service: Automated or "scheduled" Import Jobs are submitted by an Import Watcher service according to its Poling Loop or Specific Times specification.

In both cases, an "Import Descendants" or "Import Query Results" can be selected and configured using using the "Provider" property.

Similarities and differences between Import Query Results and Import Descendants

Overall, "Import Descendants" is a "simpler" version of "Import Query Results".

• We advise to use Import Query Results over Import Descendants, when possible.
  • Import Query Results can do everything Import Descendants can do and more.
  • Import Query Results has more robust file filtering capabilities. This allows for more targeted, selective imports.
  • Import Query Results is newer (and better maintained) than Import Descendants.
  • There are only a handful of scenarios where Import Descendants must be used over Import Query Results.

Similarities

• Both providers import files from a CMIS Repository.
• Both providers have the same Batch Creation settings.
• Both providers are capable of "Sparse" imports by changing the "Import Mode" to "Sparse".
• Both providers can dispose of files on import (using the "Delete Item", "Move Item", or "Update Properties")

Differences

The biggest difference is in how the providers determine which files are imported (import criteria).

• Import Descendants will import all files from a target location. This includes all files in all subfolders if present. You can, however, set a "Base Folder" within the CMIS Repository.
• Import Query Results will import files that match a CMIS Query. This is a specialized query language based on SQL syntax. This gives you many more options for import conditions, using a "WHERE" clause in the query. CMIS Queries also give you the capability to restrict imports to a folder location without importing files in subfolders (This is something Import Descendants cannot do).

• Import Descendants does have an "Import Filter" it can use to set import conditions. It also uses a SQL-like syntax. However, it is not as advanced as the CMIS Queries that Import Query Results uses.

CMIS Repositories that can only use Import Descendants

Certain CMIS Bindings are not queryable using CMIS Queries. Because of this, certain CMIS Repositories cannot utilize Import Query Results. The following CMIS Repositories must use Import Descendants to import file content:

• FTP
• SFTP
• NTFS (only if the directory has not been indexed by the Windows Search service or the Windows Search service is not running)

Prereqs: CMIS Repository

A CMIS Repository allows Grooper access to files and folders within a storage platform.

Because Import Descendants imports from a CMIS Repository, you can import from numerous storage platforms determined by the "CMIS Binding" used. These CMIS Bindings include:

• NTFS to connect to Windows folders
• FTP to connect to FTP directories
• SFTP to connect to SFTP directories
• Exchange to connect to Outlook inboxes
• SharePoint to connect to SharePoint sites (and document libraries)
• OneDrive to connect to OneDrive drives
• Box to connect to Box accounts
• AppXtender to connect to AppEnhancer applications

Before you can import files from these platforms using Import Descendants or Import Query Results, there's some setup required in the Grooper Design page. You must:

1. Create and configure a CMIS Connection.
2. Import a folder location as a CMIS Repository.

This will allow you to import files from folders accessed by the CMIS Repository. For information on CMIS Connections and CMIS Repositories, including how to create them in Grooper, visit the CMIS Connection page.

Click here for an interactive walkthrough.

Settings shared between Import Descendants and Import Query Results

The following properties/settings are part of configuring either Import Descendants or Import Query Results.

• Import Mode - Configuring this property allows you to perform "Sparse" imports to speed up the time it takes to ingest large numbers of files.
• Batch Creation settings - These settings control how Batches are created on import, including which Batch Process is used.
• File disposition options - Optionally, you can "dispose" of the source file after it has been imported into Grooper. These settings allow you to delete the file, move it or update one of its properties.

Import Mode (and "Sparse" imports)

What is an "Import Mode"?

The Import Provider's "Import Mode" controls how file content and data associated with that file (its properties and any metadata field values in a content management system) are imported into Grooper. Practically speaking, this has an effect on the overall speed of the import process.

Files can be imported using one of three "Import Modes":

• Copy - The files are fully copied to the Grooper Repository on import.
  • Certain file properties (file name, size, MIME type, attributes, the source import file's location, etc.) are stored in the CMIS Document Link attached to the Batch Folder.
  • A copy of the file is attached to the Batch Folder and stored in the Grooper File Store.
  • If using an Import Behavior, properties/metadata are mapped to the Batch Folder's Data Fields.
• Sparse - Only the file properties and mapped metadata are copied to the Grooper Repository.
  • Certain file properties (file name, size, MIME type, attributes, the source import file's location, etc.) are stored in the CMIS Document Link attached to the Batch Folder.
  • If using an Import Behavior, properties/metadata are mapped to the Batch Folder's Data Fields.
  • The file content is not copied over to the Grooper File Store. Instead, it is accessed by the link attached to the Batch Folder. The file can be copied to the Grooper File Store with the "CMIS Document Link > Load" command.
• Link Only (seldom used) - Nothing is copied to the Grooper Repository. Only a link to the source file is attached to the Batch Folder.
  • Certain file properties (file name, size, MIME type, attributes, the source import file's location, etc.) are stored in the CMIS Document Link attached to the Batch Folder.
  • File content and mapped properties/metadata must be brought into Grooper with the "CMIS Document Link > Load" command.

Sparse imports serve two functions:

• Primarily, they are used to speed up the overall import operation. This is actually a two step process.
  1. Enable the "Sparse" Import Mode when configuring the Import Provider.
  2. Have the first step in the Batch Process fully copy the files into the Grooper Repository (using the Execute activity and the "CMIS Document Link > Load" command).
• They can also be used to avoid file duplication between the import source location and the Grooper File Store.
  • A fully copied import creates a copy of the file in the Grooper File Store. A sparsely imported document is fully usable in Grooper, but no such copy exists in the File Store.
  • Instead, Grooper travels the link every time it needs to access the file (Example: When the file's image is pulled up in the Document Viewer).
  • While this does save on storage between two systems (Grooper and the import source), it does not save on processing time. Every time Grooper needs to access the document to view it, execute a command, or run an activity, it will take some time to travel the document link and fetch the document. Depending on latency, it may be preferable to load the file into Grooper even if it does duplicate the file (The file can always be removed from Grooper with a Dispose step at the end of a Batch Process too).

How does using "Sparse" speed up import?

It increases the parallelism of the overall import operation.

Import operations must run "single threaded" in Grooper. That means regardless of how much compute your server has, it's only ever going to use a single processing thread to import files.

When you're importing hundreds or thousands of documents by copying them from a source location to the Grooper File Store, it takes a long time for the Import Job to complete.

• By only importing a link to the file content, Sparse mode dramatically speeds up the time it takes to get a usable document into Grooper.
• To take full advantage of your system's resources, the first step in your Batch Process should be "Execute" using the "CMIS Document Link > Load" command. This will allow you to load the files into the Grooper File Store using multiple threads.
  • Be Aware: The "Load" command has three modes (1) Content (2) Properties and (3) Full. For "Properties" and "Full" to work appropriately, the Batch Folders must be classified on import and use an Import Behavior to map the properties.
• The end result is the overall import operation will be as if you had used the "Copy" mode. But it will be done in a way that runs multi-threaded.

Batch Creation settings

The Batch Creation settings allow you to define which Batch Process you wish to use to process the imported files.

You must configure the "Starting Step" property to assign the Batch Process.

• Use the property's dropdown editor to select a Batch Process Step from a list of Batch Processes.
• Only published Batch Processes will appear in this list.

Other notable Batch Creation properties:

• "Start Paused" - This determines if the Batch starts in a paused state or not. If "False", the first step's tasks will be automatically submitted to Activity Processing services. If "True", you will have to manually start the Batch.
• "Max Items Per Batch" - The default is 2500, meaning each Batch will have a maximum of 2500 Batch Folders before creating a new Batch on import. For users who want more Batches with fewer documents, lower this number.
• "Organize By Date" - This will organize Batches into subfolders in the Production branch in Grooper according to the year / month / day the Batch was created.
• "Priority" and "Increment Priority" - Controls the task processing priority for the Batch. "Increment Priority" is useful when submitting large user-directed imports from the Imports Page to ensure the first Batch created is the first that is fully processed by Activity Processing services.

File disposition settings

The "Disposition" settings allow you to do something with the source files after importing them into Grooper. This is important when using an Import Watcher service to schedule imports. If you do not configure a Disposition property, the imported file will remain in the same state after the Import Job completes. This can cause the Import Watcher to repeatedly attempt to import the same file over and over again.

There are three "Disposition" options:

• "Delete Item" - Turning this to "True" will simply delete the source file after the Import Job completes.
  • You may only configure this option if the "Import Mode" is set to "Copy".
• "Move To Folder" - This will move the files to a different folder in the CMIS Repository after the Import Job completes.
• "Update Properties" - This will update one or more file properties after the Import Job completes. Properties are updated by listing them as "key-value pairs" in the "Update Properties" list editor where key=value.
  • Examples:
  • Archive=false Sets the archive attribute on each imported file to "false".
  • IsRead=true Marks each imported email message as read.
  • Status=PENDING Sets a "Status" field on a document in an AppEnhancer application (assuming there is a "Status" field in the application).

Example Import Descendants configuration

Regardless of the platform you're accessing with a CMIS Repository, you configure Import Descendants largely the same. Just pick a CMIS Repository and configure the rest of Import Descendants as needed.

• Import Descendants will import all files in the base folder, including all descendant files in any subfolders if present.
• Unless you configure the "Base Folder" property, Import Descendants will start at the root of the CMIS Repository and continue down the folder structure.
• When using Import Descendants, setting the "Base Folder" to a terminal branch in the folder structure (a folder with no subfolders) is the only way to import files from a folder without importing descendant files (because there are none in this case).
  • A more technical way of saying this is Import Descendants does not support the IN_TREE CMIS search predicate. It only supports the IN_FOLDER predicate.

Example: Submitting Import Descendants from the Imports Page

Click here for a step by step walkthrough.

1. Go to the Imports Page.
2. Press the "New Import Job" button.
3. This brings up the "Submit Import Job" editor.
4. Enter a description in the Description property (This is required).
5. Open the "Provider" dropdown (Press the "☰" button).
6. Select "Import Descendants" from the dropdown list.
7. Expand the Provider settings to configure it.
8. Open the "Repository" node selector (Press the "☰" button).
9. Select the CMIS Repository you wish to import from.
10. Select a Base Folder, as needed.
11. Configure the Import Mode property, as needed.
12. Configure the Batch Creation settings, as needed.
13. Configure the file disposition options (Delete Item, Move To Folder, or Update Properties), as needed.
14. Configure any remaining Import Descendants properties, as needed.
15. Press the "Submit" button when finished.
16. Your Import Watcher service will pick up and execute the Import Job.

Example Import Query Results configuration

Import Query Results relies on a "CMIS Query" to import files from a CMIS Repository. The CMIS Query (aka CMISQL Query) uses a syntax structure similar to a SQL query. Instead of querying rows in a database based on column values, you're querying documents in a storage location based on file property and metadata values.

• The general CMIS Query format is: SELECT * FROM <a type of document in the CMIS Repository> WHERE <according to certain search conditions>
• What "type of document" in can search for in the FROM clause is determined by the storage platform and the CMIS Binding.
  • Example: For the NTFS binding you select "FROM File" for files in a Windows folder.
  • Example: For the Exchange binding you select "FROM Message" to search for email messages.
• In the WHERE clause, you can set search parameters based on file properties and metadata values called "CMIS properties". What CMIS properties are "queryable" will also depend on the the CMIS Repository and its CMIS Binding.
  • Example: The Exchange binding has a queryable "Subject" property.
  • Example: Fields in a Box metadata template are queryable for the Box binding.
  • Which CMIS properties are queryable can be determined by (1) navigating to the CMIS Repository in the Grooper node tree (2) going to the "Types" tab (3) selecting the CMIS document type whose properties you want to inspect and (4) reviewing the "Queryable" column for each CMIS property.
• The WHERE clause is also used to set the folder scope, using the IN_FOLDER and IN_TREE predicates (where supported).
• More information on CMIS Queries (including unsupported query configurations for various CMIS Bindings) can be found in the CMIS Query article.

Example: Submitting Import Query Results from the Imports Page

Click here for a step by step walkthrough.

1. Go to the Imports Page.
2. Press the "New Import Job" button.
3. This brings up the "Submit Import Job" editor.
4. Enter a description in the Description property (This is required).
5. Open the "Provider" dropdown (Press the "☰" button).
6. Select "Import Query Results" from the dropdown list.
7. Expand the Provider settings to configure it.
8. Open the "Repository" node selector (Press the "☰" button).
9. Select the CMIS Repository you wish to import from.
10. Open the "CMIS Query" editor (Press the "..." button).
11. Enter the CMIS Query by either:
    • Typing it into the Query Editor.
    • Or, using the Query Editor's property grid to construct the query (The text will populate the Query Editor as you configure these properties).
      • A more in depth explanation of the Query Editor and CMIS Queries can be found in the CMIS Query article.
12. Configure the Import Mode property, as needed.
13. Configure the Batch Creation settings, as needed.
14. Configure the file disposition options (Delete Item, Move To Folder, or Update Properties), as needed.
15. Configure any remaining Import Descendants properties, as needed.
16. Press the "Submit" button when finished.
17. Your Import Watcher service will pick up and execute the Import Job.