Documentum REST Services, DQL Keyset Paging & Zero-Trust Enterprise RAG: Announcing the OpenText Documentum Repository Connector

1. Why Documentum Ingestion Demands Deep Domain Rigor

Building Retrieval-Augmented Generation (RAG) and semantic search architectures over OpenText Documentum Content Server presents specialized protocol and security challenges that generic crawlers completely overlook:

• Subtle Multi-Entry Permission Logic: Documentum determines user access by evaluating multiple entries in a permission set (dm_acl): direct accessors, group memberships, dm_owner, dm_group, and dm_world. Access permits (0) grant rights, but access restrictions (3) strictly take rights away. Required groups (6) and required group sets (7) enforce an AND-conjunction across groups. Any RAG mapping that ignores restrictions or mishandles required groups exposes restricted corporate data to unauthorized LLM queries.
• ACL Changes Do Not Bump Document Timestamps: In Documentum, modifying a permission set changes neither r_modify_date nor i_vstamp of the sysobjects that reference that ACL. Incremental crawlers that rely solely on document modification watermarks miss permission updates indefinitely, leaving vector stores with stale, leaking security rules.
• Offset Paging Drift in Live Repositories: Running paged DQL queries with traditional page numbers (page=1, 2, 3...) re-executes the query on the server. If documents are saved or checked in during a crawl, page offsets shift dynamically, causing documents to be skipped or repeated.
• Time Zone Mismatches in DQL Date Literals: While Documentum REST Services return dates in ISO 8601 with timezone offsets, Content Server interprets DQL DATE('...') literals strictly in its local server time zone. A discrepancy between client and server time zones silently shifts watermarks by hours, causing entire batches of newly modified documents to be missed without any error.
• Credential Containment: Content metadata can point to external Accelerated Content Server (ACS) or Branch Office Caching Services (BOCS) URLs. Forwarding Documentum REST credentials or OTDS bearer tokens to third-party hosts violates security boundaries.

The new oc-documentum-repository-connector is built from the ground up on Java 25, Spring Boot, and the Open Ingestion Standard (OIS) to solve each of these domain nuances natively.

2. Zero Proprietary Dependencies & Standards-Based REST Protocol

Unlike legacy tools that require bundling proprietary Documentum Foundation Classes (DFC) jars, native libraries, or licensed installer packages, oc-documentum-repository-connector connects purely over HTTP using standard Java 25 HttpClient and Jackson.

Every request carries Accept: application/vnd.emc.documentum+json, executing queries through the repository's native dql link template (GET /repositories/{repo}?dql=...&page=...&items-per-page=...) and downloading content streams from LOCAL enclosures on the REST server itself.

Authentication is fully configurable:

• auth-type: basic: Authenticates with username and password, or passes a Documentum session ticket (DM_TICKET=...) directly as the password.
• auth-type: oauth2: Client-credentials grant against OpenText Directory Services (OTDS) or another OAuth2 identity provider (oauth2.token-url, oauth2.client-id, oauth2.client-secret). Tokens are cached with expiry margins and proactively renewed upon HTTP 401 Unauthorized.
• auth-type: bearer: Direct pre-issued bearer access token (oauth2.access-token).

To protect enterprise credentials, DocumentumClient verifies download URLs: content links naming a proxy's host are redirected safely to the configured URL, while any link targeting an external ACS or BOCS host is rejected with FOREIGN_LINK, ensuring credentials never leave the REST gateway.

3. Keyset Cursor Paging & Server Time Zone Probing

To eliminate offset paging drift, every DQL listing query is ordered by r_modify_date, r_object_id and paged using a keyset cursor:

SELECT r_object_id, i_chronicle_id, r_object_type, r_modify_date, i_vstamp,
       acl_domain, acl_name, owner_name, group_name
FROM dm_document
WHERE (r_modify_date > t OR (r_modify_date = t AND r_object_id > id))
ORDER BY r_modify_date, r_object_id

Because Documentum's r_modify_date has one-second precision, the combined cursor condition guarantees that even if hundreds of documents share the exact same timestamp, every document is processed without gaps or duplicates.

Server Date Literal Probing

On startup, connectionInfo() queries dm_cabinet to inspect a known cabinet's modification date, formats that date into a DQL literal, and executes a probe query to confirm that Content Server interprets the literal in the exact same time zone. If a mismatch is detected (e.g. Content Server runs in CET but client assumed UTC), the connector halts with a clear error rather than crawling with a skewed watermark.

4. Fail-Closed Zero-Trust Security Mapping

Documentum permission sets (dm_acl) are resolved and mapped by DocumentumSecurityMapper according to Documentum's native security evaluation:

• Basic Permits: Access permits (r_permit_type = 0) with level $\ge 3$ (READ, RELATE, VERSION, WRITE, DELETE) map to OIS read grants for that accessor. Access permits below READ (NONE, BROWSE) grant nothing.
• Access Restrictions (Deny Precedence): Access restrictions (r_permit_type = 3) with level $\le 3$ map to OIS deny rules. In OpenCrawling, deny unconditionally wins over grants, mirroring Documentum's restriction behavior.
• Dynamic Owner & Group: dm_owner dynamically resolves to the document's owner_name (user or group), and dm_group resolves to group_name.
• World Identity: dm_world maps to the configured group identity (default dm_world) or to public when world-public: true. A restriction on dm_world leaves only fallback principals.
• Required Groups & Sets (AND Conjunction): Required groups (permit type 6) and required group sets (permit type 7) enforce an AND-conjunction that standard OIS security cannot express without leakage. Documents with required groups are excluded by default (required-group-documents: exclude) or mapped to fallback principals.
• Login Domain Collision Prevention: With principal-format: login, user identities resolve to user_login_name. If multiple users across different user_login_domain instances share the same login name, the connector detects the ambiguity and refuses to grant permissions, preventing cross-domain authorization leakage.

5. Incremental Watermarks, ACL Version Tracking & Deletions

The connector persists incremental progress atomically in DocumentumCrawlState (data/documentum-state/):

• Watermark & Skip-Unchanged: Incremental runs scan r_modify_date >= <watermark>. Documents whose version, i_vstamp, and permission fingerprint match the saved state are skipped immediately.
• ACL i_vstamp Tracking: The connector tracks the version stamp (i_vstamp) of every permission set in use. If an administrator edits an ACL, the next incremental crawl detects the change and re-checks all documents bound to that ACL.
• Moved Folder Detection: In folder mode, folders modified since the watermark are re-listed in full, ensuring moved documents receive updated documentum.folderPath metadata.
• Deletion Tracking:
  • reconcile (default): Conducts an ID sweep of the crawl scope, compares against previously known documents, and emits OIS DELETE tombstones for vanished items.
  • audit: Queries dm_audittrail events (dm_destroy, dm_prune) with a 26-hour safety lookback against timezone and daylight saving drift.

6. Configuration Reference

Sample application.yml

spring:
  opencrawling:
    connector:
      type: documentum
      documentum:
        url: "https://documentum.company.com/dctm-rest"
        repository-name: "corp_repo"
        auth-type: "basic"
        username: "dctm_crawler"
        password: "${DOCUMENTUM_PASSWORD}"
        crawl-mode: "folder"
        root-cabinet-path: "/Regulatory/Submissions"
        object-types: "my_submission"
        where: "a_status = 'Approved'"
        version-mode: "current_only"
        prefer-rendition: "pdf"
        deletion-tracking: "reconcile"
        batch-size: 100
        parallelism: 4

7. Tested Against the OpenAPI-Conforming DCTM Mock

Because no public container images exist for Documentum, the connector was verified using the **DCTM REST mock** (oc-documentum-repository-connector/documentum-mock)—an emulator implementing REST 16.7 responses, DQL query execution, and Documentum permission evaluation:

• OpenAPI Spec Conformance: check_spec.py validates every mock endpoint against OpenText's published Documentum REST OpenAPI document.
• OpenText Python Client Validation: Unmodified documentum-rest-client-python passes navigation, link traversal, object retrieval, and downloads against the mock.
• Standalone & Decoupled Integration Suites: Tested with scripts/test-documentum-repository-connector.sh across both eastern (+02:00) and western (-05:00 with 5-row page caps) time zones, and verified end-to-end via scripts/test-documentum-decoupled.sh with Kafka, Claim-Check store, pgvector, and the Secure MCP Server.