CMIS Import - 2.90
Documents are imported from CMIS Connections using either the Import Descendants or Import Query Results providers. These can be used in two ways:
- To perform manual "ad-hoc" imports when creating a new Batch in Grooper Dashboard or Grooper Design Studio.
- To perform automated, scheduled imports using one or more Import Watcher Grooper services.
Import Descendants will import all documents within a designated folder location of a CMIS Repository. Import Query Results allows you to use a query syntax similar to a SQL query (called a CMISQL query) to set conditions for import based on the item's available metadata, such as a documents name, file type, creation date, archive status, or other variables.
- 1 About CMIS and CMIS Connections
- 2 CMIS+ Architecture
- 3 About CMIS Import
- 4 Version Differences
About CMIS and CMIS Connections
CMIS stands for "Content Management Interoperability Services". It is an open standard that allows different content management systems to inter-operate over the Internet. This standard protocol allows Grooper to use many different platforms for importing and exporting documents and their contents. Once a CMIS Connection is created, Grooper can exchange documents with these platforms. "Interoperability " means Grooper has the same access to control the system as a human being does. It is a "one-to-one" connection to the platform, allowing full and total control.
Upon connecting to an external digital storage platform, Grooper will be able to see the "repositories" associated with it. A repository, in computer science, is a general term for a location where data lives.
Different systems refer to "repositories" in different ways.
- An email inbox could be a repository. A folder in Windows could be a repository. A cabinet in ApplicationXtender could be a repository.
- We standardize the various terms used by various storage platforms to simply "repository". Put simply, it's a place to put things.
These repositories are "imported" into Grooper as a CMIS Repository object, created as a child of the CMIS Connection object. This doesn't import data into Grooper in the traditional sense of importing documents into a new Batch. "Importing" here is more like bringing the repository into a framework Grooper can use. Upon importing the repository, Grooper has full read and write file access to that location in the storage platform. This allows Grooper to import files from the digital source upon Batch creation and export processed content during the Export step of a Batch Process.
For our purposes, repositories are like filing cabinets full of documents. Once a connection is established, it's like giving Grooper a key to that cabinet. You can open the various drawers of that cabinet. You can pull out files and put files into. The storage platform or content management system is like the cabinet.
- The CMIS Connection object is like the key.
- The CMIS Repository object is like a drawer in the cabinet.
- You "connect" to the cabinet by turning the key. You "import" the repository by opening the drawer. Now you can see there are documents in there! You can take them out. You can read them and put them back in. You can put new ones in. You can use this "open" connection to the "drawer" however you need.
Grooper expanded on this idea in version 2.72 to create our CMIS+ architecture. CMIS+ unifies all content platforms under a single framework as if they were traditional CMIS endpoints. Prior to version 2.72, there was only one type of CMIS Connection, a true CMIS connection using CMIS 1.0 or CMIS 1.1 servers. Now, connections to additional non-CMIS document storage platforms can be made via "CMIS Bindings". This provides standardized access to document content and metadata across a variety of external storage platforms.
Using this architecture, Grooper is able to create a simpler and more efficient import and export workflow, using a variety of storage platforms. You now use the CMIS Import Import Provider and the CMIS Export Export Type, regardless of the storage platform. They connect to a CMIS Repository imported from a CMIS Connection and use that as Grooper's import or export path.
How you create a CMIS Connection only differs from CMIS Binding to CMIS Binding, as each binding has a different way of connecting to it. You don't connect to an Outlook inbox the same way you connect to a Windows file folder, for example. Thus, the property configuration for the Exchange binding is different from the NTFS binding.
A CMIS Binding provides connectivity logic for external storage platforms, allowing CMIS Connection objects to import and export content.
Grooper's CMIS+ architecture expands connectivity from traditional CMIS servers to a variety of on-premise and cloud-based storage platforms by exposing connections to these platforms as CMIS Bindings. Each distinct CMIS Binding contains the settings and logic required to exchange documents between Grooper and each distinct platform.
- For example, the AppXtender Binding contains all the information Grooper uses to connect to the ApplicationXtender content management system. The Exchange Binding contains necessary logic to connect to Microsoft Exchange email servers (like an Outlook inbox). The NTFS Binding allows connectivity to a Windows file system.
CMIS Bindings are referenced when creating a CMIS Connection object and configuring its Connection Type property. Which binding you use (and therefore which platform you connect to) is set by the Connection Type.
- First, the user selects which CMIS Binding they want to use, selecting which storage platform they want to connect to.
- Second, the user enters connection settings for that binding, such as login information for many of the CMIS Bindings.
Current CMIS Bindings
Below is a list of all current platforms with their own CMIS Bindings:
- The ApplicationXtender document management platform.
- The Box Content Management System.
- The FileBound document management platform.
- Content management systems using CMIS 1.0 or CMIS 1.1 servers.
- The following Microsoft content platforms
- FTP (File Transfer Protocol) and SFTP (SSH File Transfer Protocol) servers.
- The Box cloud document management system.
- IMAP mail servers
- The Microsoft Windows NTFS file system.
About CMIS Import
The CMIS Import provider is split into two different Import Providers
- Import Descendants
- Import Query Results
These providers are designed to import files from a folder structure of an on-premise or cloud-based document storage platform. This is the primary method of Batch creation when importing digital documents into Grooper to process them with a Batch Process.
In order to do this, a few requirements must be met first.
- A CMIS Connection object must made and configured. This will connect Grooper to the document storage platform.
- This may be a connection to a Windows folder, an email inbox, a true CMIS content management system, or other document storage platforms. What the CMIS Connection connects to is determined by the CMIS Binding selected when configuring the Connection Type property of the CMIS Connection object.
- A CMIS Repository must be imported. This will create an object Grooper can use to import documents from the folders in the document storage platform.
- This acts as a "go-between" or a "hub" for Grooper to pull in documents from the content's source. Or, you may think of this as Grooper's representation of a folder location in the document storage platform.
For more information on adding a CMIS Connection and importing a CMIS Repository, visit the CMIS Connection article.
As for the difference between the Import Descendants and Import Query Results providers, you can think of Import Query Results as a more specialized version of Import Descendants.
- Import Descendants is intended to import the full contents of a folder location. It imports the "descendant" files of a parent folder.
- Import Query Results allows you to selectively import files using a SQL-like query (called a CMISQL query). Only files returned by the query will be imported. For example, using an Exchange or IMAP CMIS Connection, you could query an inbox for emails from a specific sender and only import those emails.
- Note: There are some import filtering capabilities available to Import Descendants as well using a SQL-like query. However, the CMISQL query Import Query Results uses is much more robust. That said, only certain CMIS Bindings can take advantage of this increased CMISQL query functionality.
- The following CMIS Bindings are not currently suitable for the Import Query Results provider.
- NTFS (If the folder path is not indexed by the Windows Search service and/or Windows Search is not running on the storage server)
This is the configuration screen for the Import Descendants provider. This example uses a simple configuration to import a few PDFs from a local Windows folder. Configuration is divided into four sections:
At bare minimum, you will need to tell Grooper where to look for the imported files. That is, mostly, what the General property settings are for.
In this case, we want to import the PDF files in this folder on the local hard drive.
As discussed earlier, there are some minimum requirements before configuring Import Descendants.
Back top the Import Descendants configuration screen, the CMIS Repository object is used to point Grooper to this folder location for import.
Processing Options Settings
The most important part of the Processing Options property section is the Import Mode property.
The Import Mode property allows control over the connections Grooper makes and/or retains to the imported documents.
For importing, documents contain two important sets of information:
Depending on the Import Mode selected all, some, or none of this information will be copied to your Grooper Repository's file store (in the case of the document's content) and database (in the case of the document's properties). See below for more in depth explanation of each of the Import Mode options.
- Both properties and content will be loaded. This is a total duplication of the document from its source to your Grooper Repository's local file store. This is the slowest import mode, because the full content of each document is copied during a single-threaded import process. As such, this mode is not well-suited for high-volume imports, but provides some useful advantages in low-volume import scenarios.
- For example, Full mode allows items to be deleted immediately on import. Also, Full mode avoids the need for any follow-up content loading operations in the Batch Process.
- Properties will be loaded, but content will not. This mode is much faster than a Full import, because no content files are copied into your local Grooper file store. Instead, a link is saved on each Grooper document, and content is retrieved on demand directly from the CMIS Repository. This type of document is often referred to as a "sparse" document. Sparse documents can be used just like any other document, with the caveat that display and processing speeds may be reduced. Grooper has to traverse the document link in order to display or process the document's image.
- However, after a Sparse import, document content can be loaded multi-threaded using the Execute activity in a Batch Process. This can overall lead to importing a document's content faster than a Full import. While the
- Choose CMIS Document Link as the Object Type and Load Content as the Command
- No content or properties will be loaded, making this the fastest import mode. It imports nothing more than a link to each document, and offloads all property and content loading to parallel operations in the Batch Process.
- However, this does not produce a usable document in Grooper. After a LinkOnly import, document content must be loaded using the Execute activity in a Batch Process.
- Choose CMIS Document Link as the Object Type and Load Content as the Command
- You can think of the Link Only option as an even sparser sparse import.
See the table bellow for a summary of the Import Mode options.
|Full||Slow||Full import of content and their properties.
|Sparse||Fast||Imports a link to the document's source and its properties but not their content.
|Link Only||Fastest||Only imports a link to the document's source.
The Disposition property settings allow you to do something with the source documents after importing them into Grooper, namely delete them, move them, or do nothing and just leave them alone where they came from. This is often leveraged with the Import Watcher Grooper service to prevent repeatedly importing the same document.
In our example here, the Move to Folder property is configured to move the PDF documents to a folder named "Imported Documents".
If using the Full Import Mode, you can enable the Delete Item property to delete each document after it is imported into the Grooper Batch.
The Update Properties property allows you to alter the document's property values upon import. Property values are updated using a list of "key-value pairs" where the "key" is the name of the property and the "value" is what change you want to make to that property. You can type one entry per line in the format
Batch Creation Settings
It's likely you're importing documents because you want to run them through a Batch Process. The Batch Creation property settings allow you to define which Batch Process you wish to use to process the imported documents.
This is done using the Starting Step property, selecting a Batch Process Step in a Batch Process from the published Batch Processes in the Grooper Repository. Upon import, a new Batch is created with each document as a Batch Folder, and the selected Batch Process assigned to the Batch.
There are also further properties to control Batch creation. You can limit the number of documents imported per Batch using the Maximum Items per Batch property. By default, new Batches are named with a date/time stamp. However, the Batch Name Prefix allows you to tack on a prefix to the Batch's name for easier identification. The Start Paused property will automatically trigger the Batch Process if set to False.
Import Query Results
The Import Query Results provider's configuration panel is almost identical to the Import Descendants provider's configuration panel. Both providers share the same Processing Options, Disposition, and Batch Creation property settings. See the Import Descendants section for brief descriptions of these property sections.
The big difference between the two providers is the highlighted CMIS Query property. This allows users to enter a SQL-like query (called a CMISQL query) to selectively import documents from their source, based on certain metadata properties. Only files returned by the query will be imported. For example, you may want to only import documents of a certain file type. You could include the file extension as the query condition. You can use CMISQL queries to easily filter email messages when importing from an inbox. If you only wanted to import messages from a certain sender, you could include the sender's email address as the query condition.
Just like with Import Descendants, there are some minimum requirements before configuring Import Query Results. A CMIS Connection object must be created and a CMIS Repository must be imported.
CMIS Query Configuration
Upon pressing the ellipsis button at the end of the CMIS Query property, the following window will appear.
This interface allows you to configure the CMISQL query based on available metadata from the CMIS Binding. For example, the Exchange binding has a selection of queryable metadata for email messages, such as the email's subject, sender and date the message was received.
The query example here selectively filters an email inbox based on the following conditions:
This configuration editor writes the CMISQL query for you. You can verify the query using the "CMISQL" tab.
Whether in the "Basic Search" tab or the "CMISQL" tab you can verify the results of the query, using the "Execute Query" button. This will display a list of items that will be imported by the Import Query Results provider.
These are samples of a query string. They take the following general form.
SELECT * FROM <ContentType> WHERE <Criteria> ORDER BY <Sort>
Let's break each component, or "clause", to get a better idea of how this works
This specifies which properties are to be returned with query results.
If you are querying all properties the asterisk or
* will indicate all properties should be returned.
Otherwise, you will list them out separated by commas.
For example: Let's say you are querying an Exchange repository and you only want to search the sender and recipients properties. You'd type
SELECT Sender, ToRecipients, CcRecipients, BccRecipients to limit your query to only those four properties.
This clause indicates the type of object to search for. This will be a content type defined in the CMIS Repository. If the content type is document based, the query result will be a CMIS Document. If it is folder based, it will be a CMIS Folder.
The content type specified in the
FROM clause has two jobs. One, it defines what properties are available to the other clauses. Two, it limits the scope of the search to only objects of the type specified in the clause.
For example: Let's say you are querying an Exchange repository and want to search email messages and not contacts or tasks or appointments. You'd type
FROM Message to limit your query to just the Message content type.
This is how you define what search conditions must be met to be included in your set of returns. Multiple conditions can be joined with the AND or OR or NOT operators. You can change the order of operations by using nested parenthesis. Each condition is followed by a predicate. The following is a list of predicates. Note not every property type may be able to utilize every predicate. For example, the Subject property on the Exchange binding cannot use the "=" operator.
|Comparison Predicate||Specifies a condition for an individual property using comparisons, such as "equals to" or "less than". The
||Specifies a list of allowed values for a property. This list is separated by commas.|
||Specifies a full-text query. You can use AND, OR and NOT operators.|
|Scope Predicate||Restricts the search scope to children or descendants of a folder|
Note: The NOT operator cannot be used with the
For example: Let's say you are querying an Exchange repository and want to find an email which contains the words "cake" and "free" but not "birthday" in it, that was not received last Christmas Day found in the inbox folder and has attachments. That would look something like
WHERE IN_FOLDER(/Inbox) AND CONTAINS('cake AND free AND NOT birthday') AND (DateTimeReceived<>'12/25/2018') AND (HasAttachments=False)
This is an optional clause which allows you to specify the order in which results are returned. You can sort by multiple properties using a comma separated list. Optionally, each property name may be followed by
DESC to indicate ascending or descending sort direction. The default sort direction is ascending.
For example: If you wanted to sort a query of an Exchange repository by both whether they have attachments and by size in descending order you would type
ORDER BY HasAttachments, Size DESC
Putting It All Together
Let's mash all our examples together and search for email messages in the Inbox that have the words "cake" and "free" but not "birthday" in the body, received any day besides Christmas Day. I'm going to go ahead and search all the properties available to me, and I want to sort the results by whether the message has attachments and by size in descending order. This would be the resulting query
SELECT * FROM Message
WHERE IN_FOLDER(/Inbox) AND CONTAINS('cake AND free AND NOT birthday') AND (DateTimeReceived<>'12/25/2018')
ORDER BY HasAttachments, Size DESC
||Import all descendant files. This will import all files in the repository without any foldering.|
||Import files which are immediate children. This will only import files at that level, not from subsequent levels.|
||Import folders which are immediate children. This will import both files and their foldering.|
||Import files with a specific naming pattern, using regular expression.|
||Import files with a name starting with ca.|
||Import files larger than 10,000 bytes.|
Box Integration (2.90)
Legacy Providers (2.72)
Old import and export providers should be replaced with this new functionality. While Grooper's older import and export providers are available as "Legacy Import" and "Legacy Export" providers, these components are depreciated. They will still function but will no longer be upgraded in future versions of Grooper.
New Connection Types (2.72)
By creating the CMIS+ architecture, we have been able to create new connections between Grooper and content management systems. Grooper can now connect to Microsoft OneDrive, SharePoint, and Exchange via new CMIS Bindings. Since these were created as CMIS Bindings, they can be used by the CMIS Import and CMIS Export providers. Instead of having to create three new import providers and three new export providers for a total of six brand new components, we can use the already established CMIS import and export providers in the CMIS+ framework. A user can create a CMIS Connection using the OneDrive, SharePoint or Exchange bindings, and use the same import and export providers for them as any of the other CMIS Bindings.
This will also allow Grooper to create CMIS Bindings to connect to currently unavailable content management systems in the future much quicker and easier.
Import Mode (2.72)
In version 2.72 the Import Mode property replaces previous versions' Sparse Import property.
Import Disposition (2.72)
2.72 adds the Import Disposition property to CMIS Import. This allows you to change your documents disposition upon importing them into Grooper. You can delete them, move them to a folder, or update one or more properties on the document itself. This can be leveraged with Import Watcher to prevent repeatedly importing the same document.