2023:CMISQL Query

From Grooper Wiki
Revision as of 17:26, 7 November 2023 by Randallkinard (talk | contribs) (Created page with "<blockquote style="font-size:125%"> '''''CMIS Queries''''' are utilized to search documents in '''CMIS Repositories''' and to filter documents upon import when using the '''''Import Query Results''''' provider. </blockquote> {| class="wikitable" style="margin:left" ! Previous Versions |- | CMIS Query - 2021 <br> |} Also called a "CMISQL query", the querying language '''''CMIS Queries''''' use is based on a subset of the SQL-92 grammar. Where SQL is a querying lang...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

CMIS Queries are utilized to search documents in CMIS Repositories and to filter documents upon import when using the Import Query Results provider.

Previous Versions

CMIS Query - 2021

Also called a "CMISQL query", the querying language CMIS Queries use is based on a subset of the SQL-92 grammar. Where SQL is a querying language to search and select data in a database, "CMISQL" is a querying language to search and select documents (and their metadata properties) in a storage location, represented by a CMIS Repository in Grooper.

About

CMIS Queries use clauses to retrieve content from a CMIS Repositories based on certain metadata property values, much like a SQL query uses clauses to retrieve data from a database based on column values. In both cases, a SELECT clause is used to select content based on certain conditions determined by a WHERE clause

In general the CMISQL statement takes the following form: 

SELECT * FROM <Content Type>
WHERE <Conditions>
ORDER BY <Sort Criteria>


For example, if you wanted to retrieve all email messages in your Inbox folder over an Exchange CMIS Connection from a particular sender and order results by the date you received the email (oldest first), you could use the following query: 

SELECT * FROM Message
WHERE (Sender LIKE '%user@example.com' AND  IN_FOLDER('/Inbox'))
ORDER BY DateTimeReceived ASC


Next, let's break down this query further and look at what each clause is doing.

Clause Information

SELECT

SELECT * FROM Message
WHERE (Sender LIKE '%user@example.com' AND IN_FOLDER('/Inbox'))
ORDER BY DateTimeReceived ASC

The SELECT clause defines a set of property values to be returned with each query result. "Property values" refers to the available metadata properties for the object type selected for the From statement. For example, when querying email messages using in an Exchange CMIS Repository, there are metadata properties for the sender, the subject line, the date the email was sent, and more. All these metadata properties are part and parcel to the email message file itself.

From a practical standpoint, your CMIS Query is ultimately returning documents. Files in a digital storage location. You aren't retrieving individual metadata values, rather the whole file with all its various metadata included. To that end you will use the * character to select all property values, when crafting a CMISQL query.

You will always start your CMIS Query with SELECT * when using a CMIS Query to import documents.

FYI

When would you use something besides SELECT *? When would you construct a CMISQL query, such as SELECT Property_A, Property_B, Property_C and so on?

The answer is CMIS Lookups. A CMIS Lookup operates much like a database lookup. Instead of querying a database table's columns for data, a CMIS Query queries the metadata properties of documents in a storage location.

  • Technically, a CMIS Query is used in that case as well. However, CMIS Queries are much more commonly used to retrieve documents from a storage location upon import, which is the focus of this article.
  • For more information on CMIS Lookups please visit the CMIS Lookup article.

FROM

SELECT * FROM Message
WHERE (Sender LIKE '%user@example.com' AND IN_FOLDER('/Inbox'))
ORDER BY DateTimeReceived ASC

The FROM clause determines what type of content you're searching for, such as a digital file or a folder and their metadata properties.

  • The type of content you select determines what metadata properties you have access to when building the conditional logic in your query's WHERE clause.

Depending on the CMIS Connection type, you will have different options to select from. Some have simple "Document" and "Folder" content types, representing documents and folders respectively. Some have a multitude of content types, each with metadata properties specific to the CMIS Binding.

  • For example, the Exchange binding's Message content type has property values relating to email messages, such as Subject, Sender and DateTimeSent.
  • For example, The SharePoint binding's content types can be Document Libraries, giving you access to custom metadata properties you've created on the SharePoint site by adding custom columns to the Document Library.

In our example, we wanted to query email messages in an Exchange repository. So, we selected the Message type by adding FROM Message to our query.

WHERE

SELECT * FROM Message
WHERE (Sender LIKE '%user@example.com' AND IN_FOLDER('/Inbox'))
ORDER BY DateTimeReceived ASC

The WHERE clause defines what search conditions must be met for a document to be retrieved. Each condition is defined by a "predicate". You can use any metadata available to the content type (as determined by the FROM clause) in your query conditions.

  • Multiple conditions/predicates can be joined with the AND or OR operators.
  • You can manipulate the logical order of operations by using nested parenthesis.
  • You can logically negate predicates using the NOT operator.

The following is a list of predicates:

Predicate Description Example
Comparison Predicate

Specifies a condition for an individual property using comparisons, such as "equals to" = or "less than" < or "greater than" >.

  • The LIKE and IS operators are also a comparison predicates.
 DateTimeCreated < '12/31/2007'
In Predicate Specifies a list of allowed values for a property. This list is separated by commas. FileExtension IN ('.pdf', '.docx', '.xlsx')
Contains Predicate Specifies a full-text query.
  • You can use AND, OR and NOT operators when crafting search parameters.
  • Please note only external platforms that support full text searching are compatible with with the Contains Predicate.
 CONTAINS('mortgage AND payment AND NOT vehicle')
Scope Predicate Restricts the search scope to a specific folder (IN_FOLDER) or a specific folder including its subfolders(IN_TREE) IN_FOLDER('/Documents/Grooper')
  • Note: Not every property type may be able to utilize every predicate/operators within a predicate. For example, the Subject property on the Exchange binding cannot use the = operator when crafting a Comparison Predicate.
  • Note: The NOT operator cannot be used with the IN_FOLDER or IN_TREE Scope Predicate.

In our example, we used a combination of the Comparison Predicate and Scope Predicate to only retrieve email messages from certain sender (Sender LIKE '%user@example.com') and in our inbox folder ((IN_FOLDER('/Inbox/')) Since we wanted to filter content by multiple criteria, the two conditions were joined with the AND operator.

ORDER BY

SELECT * FROM Message
WHERE (Sender LIKE '%user@example.com' AND IN_FOLDER('/Inbox'))
ORDER BY DateTimeReceived ASC

The ORDER BY clause is an optional clause which allows you to specify the order in which results are returned.  You can sort by multiple properties using a comma separated list.  Each property name may be followed by ASC or DESC to indicate ascending or descending sort direction.

  • The default sort direction is ascending. However, it is still considered best practice to include the ASC tag to properties you wish to sort in ascending order.

In our example, we wanted to sort our incoming messages by oldest received first.

CMIS Query Editor

If you are not familiar with the CMISQL syntax (or even SQL), you may find it easier to use the CMIS Query Editor, instead of writing the full query yourself. This allows you to configure the CMISQL query using a series of property grids and menu selections instead of writing out the query from scratch.

The CMIS Query Editor is accessible in two locations:

  1. When configuring the Import Query Results Import Provider's CMIS Query property.
  2. Using a CMIS Repository's "Search Repository" tab.
⚠

Only certain external storage platforms are currently queryable with a CMIS Query. The following CMIS Binding sources cannot be queried currently.

  • FTP
  • SFTP
  • NTFS (If the folder path is not indexed by the Windows Search service and/or Windows Search is not running on the storage server)

As such, these bindings are not suitable for the Import Query Results provider (You should instead use Import Descendants). Furthermore, you will not have access to the "Search Repository" tab for these bindings when selecting a CMIS Repository.
Again, the exception to this is NTFS. If the folder path IS indexed in Windows and the Windows Search service IS running, then you CAN use the "Search Repository" tab and CMIS Queries.

How to Access the CMIS Query Editor

Option 1: Import Query Results > CMIS Query

Most commonly, you will use a CMISQL query to import documents using the Import Query Results provider.

FYI

The Import Query Results provider is one of two Import Providers that use a CMIS Connection to bring document content into Grooper. If you need more information on Import Query Results, please visit the CMIS Import article.


  1. When configuring the import settings, you will first select which CMIS Repository you want to query by selecting one using the Repository property.
  2. Then, select the CMIS Query property.
    • Note: Some CMIS Bindings are not CMISQL queryable (For example, the FTP and SFTP CMIS Bindings). If the CMIS Repository is not queryable, you will not see this CMIS Query property.
  3. Press the ellipsis button at the end of the property to bring up the CMIS Query Editor.


This is the CMIS Query Editor. Using this interface you can craft a CMISQL query in one of two ways:

  1. You can use this property grid to configure the various clauses of the query statement.
  2. You can free-type the query in this text editor.


We will demonstrate how to use this query editor in the next section

Option 2: The Search Repository Tab

You can also use the CMIS Query Editor when searching for documents in a CMIS Repository using the "Search Repository" tab.


  1. Select the CMIS Repository you want to query in the Node Tree.
  2. Navigate to the "Search Repository" tab.
    • Note: Some CMIS Bindings are not CMISQL queryable (For example, the FTP and SFTP CMIS Bindings). If the CMIS Repository is not queryable, you will not see this "Search Repository" tab.
  3. You can use this property grid to configure the various clauses of the query statement.
  4. You can free-type the query in this text editor.


We will demonstrate how to use this query editor in the next section

How to Use the CMIS Query Editor

Next, we're going to illustrate how to use the CMIS Query Editor in order to filter email messages based on certain criteria. The CMIS Query Editor's interface allows you to configure the CMISQL query based on available metadata to the CMIS Binding.

We will use the Exchange binding, which has a selection of queryable metadata for email messages, such as the email's subject and date the message was received. Our query will import email messages based on the following conditions:

  1. Only messages are to be imported (as opposed to other Exchange content, such as appointments or tasks).
  2. Only messages in a certain folder will be imported.
  3. Only messages containing specific text search terms will be imported.
  4. Only messages from a certain sender will be imported.
  5. Only messages that have not been read yet will be imported.

This will get us through the basics of building a CMISQL query using the CMIS Query Editor.

The SELECT and FROM Clauses

The property grid is divided into the various clauses that form the CMISQL query. Adjusting these properties will adjust the CMISQL query.

  1. The Select property will adjust the SELECT clause.
    • By default this property is set to *.
    • This forms the start of the query SELECT *
  2. The From property will adjust the FROM clause.
    • Each CMIS Binding will default to a different content type. Generally, the default content type corresponds to a "document" in the storage location.
    • The Exchange binding's default is Item. However, we actually want to change this. An "item" is too generic. It refers to message files, appointment files, task files, and note files in an Exchange inbox.
      • We want to import email messages, and define our query conditions to filter out emails using metadata specifically accessible to those types of files. We need to use the Message content type to do this properly.

Changing Content Types in the FROM Clause

We will use the CMIS Query Editor's property panel to edit our query going forward.

  1. Since we want to adjust the content type in our FROM clause, we will select the From property.
  2. Using the dropdown menu, we've selected Message.
    • This will query specifically message files over our Exchange connection, and give use access to their metadata for our querying conditions.
  3. This automatically updates the full text of our CMISQL query.
    • SELECT * FROM Item changed to SELECT * FROM Message


As we use the property grid to further configure our query, the query's text will update automatically.

  • This gives you a method of editing the CMISQL query without free-typing the query.
  • However, you certainly can type out the full query text if you prefer not to use the property grid to enter the CMISQL query.
    • You can also go back and forth between the property grid and the text editor.

The WHERE Clause

Next, we will use the Where property to configure the various filter conditions we've set for ourselves.

  1. Select the Where property.
  2. Using the dropdown list, select the search predicate you wish to use.
    • If you are filtering based of multiple criteria. you will ALWAYS choose Predicate Collection.
    • We need to filter based on the values of multiple metadata properties, and we need to use both a Comparison Predicate and a Scope Predicate. So, we have selected Predicate Collection.

Using Predicate Collection for Multiple Search Conditions

  1. Expand the Where sub-properties to add your query criteria.
  2. The default Logical Operator is AND.
    • This is appropriate for our situation, as we want to use the AND operator. ALL conditions must be met in our case.
    • But be aware you also have access to the OR operator.
  3. Next, we will add our various query conditions using the Expressions property.
  4. Press the ellipsis button at the end to bring up the Where Predicate Expressions collection editor.

Adding Search Predicates

Now that we're in the Where Predicate Expressions collection editor, we can start adding and configuring our search conditions.

  1. Press the Add button to add a predicate to the list.
  2. We will start by adding a Scope Predicate to restrict the search scope to a specific folder.

The Scope Predicate

The Scope Predicate allows you to choose a folder from which you want to retrieve documents.

  1. Select the Search Scope property and press the ellipsis button at the end.
  2. This will bring up a folder browser, representing your connected CMIS Repository's hierarchical folder structure.
  3. We're going to restrict our scope to this subfolder named "Wiki" in the connected email account.
  4. Press OK to finalize your selection.
FYI

If Include Subfolders is False, the IN_FOLDER() predicate will be used.

• This will restrict the query's search to the single selected folder in the folder hierarchy.

If Include Subfolders is True, the IN_TREE() predicate will be used.

• This will restrict the query's search to the selected folder and search any of its subfolders in the folder hierarchy.

The Contains Predicate

If the storage location your CMIS Repository is connected to supports full text searching, you can use the Contains Predicate to query documents based on specific text search terms.

For example, we want to retrieve email messages pertaining to "Wiki Vitals", a specific kind of report generated nightly about the Grooper wiki and sent to my email.

To add a Contains Predicate:

  1. Press the Add button and select Contains Predicate
  2. This adds a new Contains Predicate to our list of predicates
  3. Using the Search String property, enter the text you would like to search for.
    • In our case, only messages with the text Wiki Vitals somewhere in the subject line or email body or otherwise in the message's metadata will be retrieved.
    • Notice Grooper automatically generates the property syntax for the query, CONTAINS('Wiki AND Vitals').
⚠

Please note only external platforms that support full text searching are compatible with with the Contains Predicate.

The NOT Operator

All predicates can be logically negated using the NOT operator. For example, most of these "Wiki Vitals" reports are daily reports. These are the emails I want to process. However, once a month, there is a monthly report. I DO NOT want to process those documents.

The only way I know it's a monthly report is the term "monthly report" is listed in the body of the email. I can use the Contains Predicate and the NOT operator to exempt these monthly reports from the query.

To logically negate a predicate:

  1. Add the predicate using the Add button.
  2. We've added another Contains Predicate.
  3. Configure the predicate's search parameters.
    • What do we not want to retrieve in this case? Messages with the text Monthly Report somewhere in their body. So, we've configured the Search String property to search for that term.
  4. Change the Not property to True.
    • This will negate the predicate's logic. If the email contains the search term, it will not be retrieved.

The Comparison Predicate

The Comparison Predicate specifies search criteria based on a document's metadata property values, using comparison operators such as >, < or =.

For example, we will use a Comparison Predicate to only import emails that have not been read yet, whose "IsRead" property value equals "False".

To add a Comparison Predicate':

  1. Press the Add button and select Comparison Predicate
  2. This adds a new Comparison Predicate to our list of predicates
  3. Using the Property Name property, select a property from a list of the documents' available metadata properties.
    • What metadata properties are available will be determined by the CMIS Repository's binding/connection type and the content type selected in the FROM clause.
    • We've selected the IsRead property.
  4. Using the Comparison Operator select which operator you want to use for comparison.
    • Depending on the value's type (string, decimal, Boolean, etc.) different comparison operators may be available.
    • Here, we are checking for equivalency using the = operator.
  5. Using the Search Value property, enter the control variable you want to use for comparision.
    • In our case, we want to check if these emails have not been read, indicated by the email message's "IsRead" property being False.

The LIKE Operator

The LIKE operator is used to match substrings of string values. In many cases, it must be used instead of the = when you're comparing string variables.

For example, if we want to use a Comparison Predicate to filter messages based on the "Sender" (a string value), we will need to use the LIKE variable to do so. Furthermore, we will also need to encase the search term in the % wildcard symbol.

  1. Here, we've added a new Comparison Predicate to only retrieve messages sent by a specific email address.
  2. We've slected Sender for the Property Name
  3. This property must use the LIKE Comparison Operator.
  4. For the Search Value property we've entered %cdearner@bisok.com%.
    • Effectively, we just enter the email address we want to search for (cdearner@bisok.com) with percent sign wildcards at the front and end.

FYI

The LIKE operator specifies that a property value must contain a substring. It takes the general form PropertyName LIKE 'Expression', where PropertyName is the query name of a string property, and Expression is a string literal using the % and _ characters as wildcards. The LIKE operator is not case sensitive.

• The % wildcard substitutes for zero or more characters.
• The _ wildcard substitutes for exactly one character.
• A LIKE expression with no wildcards is equivalent to using the = operator.
• For example, the expression 'grooper%' will match cases where the property value begins with "grooper", while '%grooper' will match cases where the property value ends with "grooper". The expression 'grooper%review' will match cases where the property value begins with "grooper" and ends with "review".

Some CMIS Bindings have limited support for LIKE expressions, and will raise errors when unsupported features are used. Examples of such issues are as follows:

• Some bindings only support the % wildcard at the beginning and end of the expression (i.e. '%grooper%').
• Some bindings do not support the "ends with" form of LIKE (i.e. '%grooper').
• Some bindings do not support the _ wildcard at all.

Finish Adding Predicates

  1. When you have finished adding predicates to the collection list, press the OK button.


  2. When finished, the search predicates will form the conditional parameters of the query's WHERE clause.

We could be done at this point. We have the three basic parts of a CMIS Query.

  1. The query starts with SELECT *
  2. The FROM clause determines which content (documents/folders and their metadata properties) we're querying from the CMIS Repository/storage location.
  3. The WHERE clause defines the query's search conditions.

Optionally, you may want to order the documents Grooper retrieves. We will do that next, with the ORDER BY clause.

The ORDER BY Clause

The ORDER BY clause is an optional CMISQL querying clause used to change the sort order of the documents Grooper retrieves. You can sort by any metadata property available.

For example, we may want to sort by the date the email message was sent. And specifically we want the most recent email first in the list and the last email last in the list. We're going to need an ORDER BY clause to do this.


To add an ORDER BY clause:

  1. Select the Order By property, and press the ellipsis button at the end.

  2. This brings up the Order By collection editor.
  3. Press Add to add a new sort element.
  4. Using the Property Name' property, select which document metadata property you want to sort by.
    • In our case, we've chosen the email messages DateTimeSent property, storing what date and time the email was sent.
  5. Using the Sort Direction property, select which direction you want to sort by.
    • Choose ASC to sort in ascending order.
    • Choose DESC to sort in descending order.

Testing the Query

  1. With this ORDER BY clause added, our query is now complete.
  2. This message indicates the query is valid. There is nothing wrong with our syntax.
  3. You can test the query by pressing the Execute Query button.
  4. The query's results will appear in the Query Results pannel
  5. If you double-click on an item in the list, it will appear in the Document Preview panel.

Unsupported Query Configurations

The NTFS Binding will not support full text search (the CONTAINS() predicate) unless the Windows file system has been indexed by the Windows Indexing Service.

The SharePoint Binding does not support the IN_FOLDER predicate when selecting a folder location to query. You must use the IN_TREE predicate instead.

  • Be aware the IN_TREE search is recursive where IN_FOLDER is not. This means any subfolders will be queried as well as the targeted folder.

The SharePoint Binding has a hard limitation when it comes to querying larger repositories, with many thousands of folders and files. There is currently a threshold limit of 5000 rows. This means if a query has to scan more than 5000 items to return an item, SharePoint will return the following error:

  • "Internal Server Error (500): The attempted operation is prohibited because it exceeds the list view threshold."
Retrieved from "https://wiki.grooper.com/index.php?title=2023:CMISQL_Query&oldid=15246"