Skip to end of metadata
Go to start of metadata

In this topic:

Note

The Coveo Cloud V2 On-Premises Crawling Module is still a beta product. Several manual installation and configuration steps will be automated in subsequent beta releases.

Moreover, the Crawling Module is not available in trial organizations yet.

Once you started your workers, you can create a source to crawl (see Managing the On-Premises Crawling Module Using the REST API). For now, a configuration UI in the Coveo Cloud V2 administration console is available for File and Database sources only (see Creating a File or Database Source). If you want to index a different source type, you must use the Coveo Cloud V2 Platform API (see Creating a Source Using the Source API).

Note

For now, these procedures do not take into account the option you have to index the permissions corresponding to your secured content. If you want to index secured content and take access permissions into account, contact the Coveo Support team.

Creating a Crawling Module Source Using the Source API

Since there is no administration console UI yet to create On-Premises Crawling Module sources in Coveo Cloud V2, you must create your source via the Coveo Cloud V2 Platform API (see Source API). To do so, in the Create a source from simple configuration call, under Parameters, you must provide:

  • The organization ID of the organization you linked with Maestro (see Review the Organization ID and Linking Maestro to the Coveo Cloud V2 Platform). This replaces {organizationId} in the request path.

    Note

    The Crawling Module is not available in trial organizations yet.

  • The JSON source configuration with placeholders replaced by your source values. Each source has a different JSON configuration to provide in the request body. The JSON configurations to provide, along with your organizationId, in the Create a source from simple configuration request are shown in the source-specific sections below.

    Note

    If you need a more advanced configuration, contact the Coveo Support team for assistance.

    The following basic properties are common to all or most of the Coveo On-Premises Crawling Module source configurations:

    • The name property value is the source name as it should appear in the Coveo Cloud V2 administration console. Replace <SourceDisplayName> with the desired source name.

      Warning

      You cannot change the name of a source once it has been created, so make sure the name you choose fits the content you intend to index with that source.

    • The sourceVisibility property determines who can see the source items in their search results (see Source Permission Types). Choose SHAREDSECURED, or PRIVATE.
    • The sourceType value determines the type of source you create. Provide the value corresponding to the source to index (see Possible sourceType Values).
    • Boolean properties pushEnabled and onPremisesEnabled must always be true.
    • The username and password values allow the crawler to access the secured content to index. Replace the value placeholders with the corresponding credentials.

Also in the Create a source from simple configuration call, leave the updateSecurityProviders and rebuild parameter values to true unless otherwise instructed by the Coveo Support team.

For an example of a request and of a response body, see Creating a Basic Shared Web Source in the Source API documentation.

Confluence

Note

Depending on the content you want to index, you might need to download the Coveo Plugin for Atlassian Confluence before creating a Confluence source (see Installing the Coveo Plugin for Atlassian Confluence).

In the request body, beside providing an adequate value for the the basic properties listed above, make sure to  replace  <["URL", "URL"]>  with the addresses to crawl (see Basic Properties).

JIRA

Note

Depending on the content you want to index, you might need to download the Coveo Plugin for Atlassian JIRA before creating a JIRA source (see Installing the Coveo Plugin for Atlassian JIRA).

In the request body, beside providing an adequate value for the the basic properties listed above, make sure to (see Basic Properties):

  • Replace <SERVER_URL> with the address of your JIRA server.

    Example

    http://MyJiraServer:8080/

  • Indicate true or false for:
    • indexAttachments, which determines whether binary files attached to an issue should be indexed. Attachments are indexed with the same level and sets of their parent issue.
    • indexComments, which determines whether comments on an issue should be indexed. Comments are indexed with the same level and sets of their parent issue. When permissions on the comments are supported, if a comment is restricted to a group or a project role, an additional set with the group or the role is added.
    • indexWorkLogs, which determines whether time entry on a issue should be indexed. Work logs are indexed with the same level and sets of their parent issue. If a work log is restricted to a group or a project role, an additional set with the group or the role is added.
    • supportCommentPermissions, which determines whether only users allowed to see a comment in JIRA can also see it in their search results. If this property value is true, an issue and its comments are indexed as separate items, leading to lower search relevance. If the value is false, the issue and its comments are indexed as one item, allowing to find an item via either an issue or its comments. However, there are no restrictions on users seeing comments on an issue.

Jive

Note

Depending on the content you want to index, you might need to download the Coveo Plugin for Jive before creating a Jive source (see Installing the Coveo Plugin for Jive).

In the request body, beside providing an adequate value for the the basic properties listed above, make sure to (see Basic Properties):

  • Replace <INSTANCE_URL> with the address of your Jive server.

    Example

    https://myjiveserver.mycompany.com

  • Enter a value for the following properties  if you wish to leverage the corresponding feature or customization. If not, indicate null.
    • For startingSpaceURL, enter the URL of the space at which the crawling should start. Only content within this space and its subspaces will be available in the index. This does not affect social groups and people.
    • If you use phrase substitutions in your Jive community, you must provide a value for substitutionLocale and substitutionTheme  (see Jive Substituting Phrases in the UI documentation). For substitutionLocale, enter  the phrase substitution locale that the connector should use. The default locale is  default , meaning that the actual machine locale is used. Enter the locale in any of the following three formats:
      • Language only (e.g. en)
      • Language and country (e.g. en_US)
      • Language, country, and variant (e.g. en_US_NY)
    • For substitutionTheme, enter the phrase substitution theme that the connector should use. The default theme is  custom .
    • For placesTitles, enter the metadata to use as a title for Jive places. The metadata must be specified in the following order: space_meta;group_meta;project_meta;person_meta;blog_meta.

    • For ignoreItemsOfTypes, enter a list of all Jive item types to ignore, using semi-colons to separate entries. The crawler indexes all items, except those of the specified types. Possible values are: Announcement, Attachment, Checkpoint, Comment, Discussion, Dm, Document, File, Group, Idea, Message, Poll, Project, Space, SystemBlog, Task, Update, and Video.

  • Indicate true or false for:
    • indexSpaces, which determines whether Jive spaces and any item they contain should be indexed.
    • indexProjects, which determines whether Jive projects and any item they contain should be indexed.
    • indexGroups, which determines whether Jive groups and any item they contain should be indexed.
    • indexSystemBlogs, which determines whether system blogs and any item they contain should be indexed.
    • indexPeoplewhich determines whether  Jive people and any item they contain should be indexed.
    • indexPublishedOnly, which determines whether only items with a Published status are indexed. Content that is incomplete, pending approval, or rejected is not indexed.
    • allowAnonymousAccess, which determines whether the source maps the Everyone groupe in Jive to the Everyone group of the Email identity security provider (see Coveo Cloud V2 Management of Security Identities and Item Permissions) . Therefore, when this property value is true, all users (authenticated and anonymous) can see the Jive content allowed to Jive Everyone in search results.

Open Database Connectivity (Database)

Note

You can configure a Database source though the Coveo Cloud V2 administration console if you use a 64-bit driver to connect to your database (see Creating a File or Database Source). If you use a 32-bit driver, you must use the Coveo Cloud V2 Platform API and the JSON configuration below.

In the request body, beside providing an adequate value for the the basic properties listed above, make sure to (see Basic Properties):

  • For connectionString, replace < Driver=DRIVER;Server=SERVERNAME;Database=DBNAME;Uid=@uid;Pwd=@pwd;> with the connection string used to connect to the database.  

    Notes

    Icon
    • The connection string syntax differs from one database type to another. Refer to the appropriate documentation for the format of the connection string specific to your database (see Connection Strings). 

    • The same connection string can be used for different sources. However, there can only be one connection string per source. 

    • Specify the exact name of the desired driver. Refer to the list of available drivers, depending on the driver type you select using the ForceX86 property below (see Viewing the Available Drivers for an ODBC Source).

    • You can hide the password and the user ID in the connection string (see CES 7.0  Replacing the Identity in Database Connection Strings  documentation).

  •  Indicatetrue for a 32-bit driver and false for a 64-bit driver for the ForceX86 property, as it determines which drive type should be used.

    Note

    Icon

    The driver type you choose must match the type of the driver you specified in your connection string.

  • Provide an XML configuration file with the desired content, mappings, allowed users, etc. to enable Coveo to retrieve and copy the data from record fields to Coveo default and standard source fields (see Example of a Configuration File). This configuration file must however be base64-encoded for your JSON source configuration to be valid. Use the Base64 Encode and Decode online tool, and then replace <Base64EncodedContent> with the encoded output.
  • For DriverType, specify which type of driver provides access to the database. 

  • For ItemType, replace <COMMA,SEPARATED,VALUES> with the Mapping type values from your XML configuration file above. These values are the table or object names to retrieve and must be separated by commas. 

SharePoint

Note

Although your SharePoint instance is of the on-premises type, you must enter SHAREPOINT_ONLINE as the sourceType value. This value refers to the version 2 of the SharePoint connector developed by Coveo, and this connector is used both for cloud and on-premises instances.


In the request body, beside providing an adequate value for the the basic properties listed above, make sure to (see Basic Properties):

  • Replace  <["URL", "URL"]>  with the addresses to crawl

  • Select the appropriate AuthenticationType value from the following:

    ValueDescription
    WindowsClassicDefault Microsoft NTLM authentication mode.
    WindowsUnderClaimsWindows authentication mode under claims.
    SpOnlineNativeAuthentication for Sharepoint Online Native.
    SpOnlineFederatedAuthentication for Sharepoint Single Sign-On using trusted identity providers (ADFS).
    AdfsUnderClaimsAuthentication for Trusted Security Providers.
  • Select the appropriate CrawlScope value for the content type that you want to include in relation with the source URL your specified:

    ValueContent to Index
    WebApplicationAll site collections of the specified web application.
    SiteCollectionAll websites of the specified site collection.
    WebAndSubWebsThe specified website and its sub webs only.
    ListThe specified list or document library only.

Sitemap

You can use a Sitemap source to make searchable the content of listed web pages from a Sitemap (Sitemap file or a Sitemaps index file).

In the request body, beside providing an adequate value for the the basic properties listed above, make sure to (see Basic Properties):

  • Replace  <["URL", "URL"]>  with the addresses to crawl.

  • Provide the value of the user-agent HTTP header to use as userAgent. This is the identifier used when downloading web pages. Default is  Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.110 Safari/537.36.

  • Indicate true or false for enableJavaScript, which determines whether JavaScript should be evaluated and rendered before indexation. This option is useful when you want to index the dynamically rendered content of crawled pages. However, activating this option has a significant impact on the crawling performance.

  • Provide a JSON scraping configuration to use as scrapingConfiguration or leave the quotations marks empty (see Web Scraping Configuration).

Web

You can use the Coveo On-Premises Crawling Module to crawl an internal website, i.e. pages available on a certain network only. 

If you want to crawl a public website, i.e. a website that is globally available on the Internet, you can use the Coveo Cloud V2 administration console (see Add a Web Source).

In the request body, beside providing an adequate value for the the basic properties listed above, make sure to replace  <http://www.example.com>  with the addresses to crawl. (see Basic Properties).

File

You can also create a File source using the Coveo Cloud V2 administration console. However, if you must create a File source via an API call, you can use the following configuration.

In the request body, beside providing an adequate value for the the basic properties listed above, make sure to (see Basic Properties):

  • Replace <file://Path/To/Shared/Folder> with the address to crawl.

  • Indicate true or false for:

    • expandMailArchives, which determines whether the content of mail archives (.pst) should be indexed. Default value is false.
    • indexSharePermissions, which determines whether share and NTFS permissions should be taken into account and applied in Coveo Cloud V2 (see Share and NTFS Permissions on a File Server).

      Note

      If you want to take permssions into account, contact the Coveo Support team.

What's Next?

Review the default refresh schedule and optionally change it to better fit your needs (see Edit a Source Schedule).

  • No labels