Intelligent Content Automation Meets Enterprise RAG: Announcing the Doxis Repository Connector

1. The Enterprise ECM Challenge: Liberating Doxis Knowledge for AI

Organizations managing tens of millions of documents in Doxis face a significant hurdle when adopting enterprise Generative AI and Retrieval-Augmented Generation (RAG):

• Beyond Plain Files: Doxis does not merely store binary blobs. Documents are deeply contextualized inside Akten (electronic dossiers), governed by rich typed metadata descriptors (contract dates, object numbers, cost centers, document classes), and protected by strict multi-tier permissions.
• Multi-Tenant Isolation: In shared or hosted environments, a Doxis Content Service Bus (CSB) instance hosts multiple customer tenants (e.g. ACME). Naively indexing permissions into vector stores without qualifying identities creates severe cross-tenant privilege bleed vulnerabilities.
• Fail-Secure Permission Boundaries: In Doxis CSB, class-level permissions ("All instances") are not queryable over the REST contract. A crawler that defaults missing permissions to "public" risks catastrophic data exposure. Security must be deny-by-default.
• Session & Server Stability: Long-running crawlers that open unbounded sessions can exhaust CSB connection pools, trigger HTTP 429 throttling, or leave orphaned search result sets on the server.

The oc-doxis-repository-connector was engineered from the ground up to solve every one of these enterprise challenges natively, adhering strictly to the Open Ingestion Standard (OIS).

2. Native Transport: The oc-doxis-client Subsystem

Rather than relying on third-party dependencies or legacy SOAP interfaces, OpenCrawling communicates with Doxis via the modern Doxis Content Service Bus (CSB) REST API (contract 14.4.1+) using a dedicated, zero-dependency Java 25 client module: oc-doxis-client.

Key architectural highlights of the client layer include:

• Triple Authentication Support: Basic Authentication (basic), CSB Session Tickets (ticket), and OAuth2/OIDC token exchange (oauth2) with automated background token expiration detection and renewal.
• Session Lifecycle Guarantee: Authenticated session tokens (passed via X-Doxis-SessionTicket or Bearer headers) are maintained throughout the crawl and cleanly invalidated upon completion.
• Paged Search Cursor Cleanup: Server-side search result sets created during paged queries are systematically released using DELETE /documents/searchResults/{searchId}, preventing resource leaks on the CSB application servers.
• Resilience & Exponential Backoff: Built-in retry handling for rate limits (HTTP 429 honoring Retry-After headers) and transient gateway errors (HTTP 502, 503, 504).

3. Dual Crawling Strategies: CQL Search & E-File (*Akten*) Traversal

Different enterprise workloads require different crawl patterns. The connector supports two distinct execution modes:

Mode A: High-Speed CQL Search Crawling (crawl-mode: search)

Leverages Doxis Context Query Language (CQL) to query repositories directly. Crawl jobs can target specific document classes, date ranges, or descriptor attributes:

spring:
  opencrawling:
    connector:
      doxis:
        crawl-mode: search
        search-query: "OBJECTNAME LIKE 'Contract*' AND ObjectDate >= '2026-01-01'"
        document-classes:
          - "ContractDocument"
          - "ProcurementRecord"
        batch-size: 100

Hits are retrieved in configurable page batches (default: 100) via GET /documents/searchResults/{searchId}, streaming items directly into Java 25 Virtual Threads for downstream parsing.

Mode B: Hierarchical Dossier Navigation (crawl-mode: folder)

In document-centric business processes (e.g. HR employee dossiers, legal matter files, loan origination *Akten*), relationships between documents and their enclosing folders are vital. In folder mode, the connector starts at a designated root-folder-id and traverses the hierarchy recursively, filing documents under their respective dossier paths while capturing child dossiers.

Flexible Version Handling

Control version indexing using version-mode:

• latest_only (default): Ingests the current active version of each document, stamping doxis.isLatestVersion: "true".
• all_versions: Ingests the complete historical revision chain. Historical versions receive immutable version-qualified canonical URIs: doxis://ACME/DB1/documents/{docUuid}/versions/{versionNum}.

4. Typed Descriptor Normalization & Schema Integrity

Doxis metadata descriptors contain the true business context of your files. The connector's normalization engine introspects descriptor types and converts them to standardized formats:

5. Zero-Trust Security, E-File Inheritance & Multi-Tenant Isolation

Enterprise AI search requires that a user querying the vector database or an MCP agent can never see results for documents they lack permission to view in Doxis:

1. Granular ACE Translation

The DoxisSecurityMapper inspects Access Control Entries (ACEs) on both documents and e-files:

• VIEW_DOCUMENT_CONTENTS & VIEW_FOLDER_CONTENTS map to OIS read access.
• Negative permission variants map to explicit OIS deny rules.
• UPDATE_* permissions map to OIS write access.
• Principals are categorized as user, group, or role via CSB principal lookups.

2. Hierarchical Dossier Inheritance

Documents filed inside an electronic file (*Akte*) inherit their parent dossier's ACEs. The connector merges e-file permissions with document-level instance ACEs, emitting inheritanceEnabled: true to ensure seamless downstream policy enforcement.

3. Fail-Secure Fallbacks (Deny by Default)

Because class-level permissions are not exposed over the REST API, documents lacking explicit instance ACEs receive configured fallback-principals (e.g. group:Legal) or an empty ACL. They are never marked public, preventing accidental data leaks.

4. Multi-Tenant Isolation

When tenant-isolation: true is configured, all user and group identities are automatically qualified by the customer tenant name:

"security": {
  "inheritanceEnabled": true,
  "permissions": [
    { "identity": "ACME/Legal", "identityType": "group", "access": "read" },
    { "identity": "ACME/elena.weber", "identityType": "user", "access": "write" },
    { "identity": "ACME/restricted.user", "identityType": "user", "access": "deny" }
  ]
}

This allows multiple business units or SaaS tenants to share a single PostgreSQL pgvector, Qdrant, or Solr index while maintaining ironclad tenant boundaries during retrieval filtering.

6. Claim-Check Streaming & Disk Leak Prevention with TempFileInputStream

In enterprise content repositories, files can range from kilobytes to gigabytes. Holding document payloads in JVM heap memory causes Out-Of-Memory (OOM) errors, while naive temporary disk caching can fill up worker volumes if files are not deleted after indexing.

OpenCrawling solves this with TempFileInputStream:

• Binary payloads are spooled to temporary files under java.io.tmpdir.
• The custom TempFileInputStream hooks into stream closure: the moment Apache Tika or the Claim-Check uploader finishes reading the stream, the underlying temporary file is immediately unlinked and deleted from disk.
• Massive blobs exceeding max-content-size-bytes (default: 50 MB) skip binary transfer while preserving full descriptor and ACL indexing.
• External content links (doxis.contentLink) are captured directly without binary transfer.

7. Incremental Synchronization & Deletion Tombstones

Crawling millions of documents repeatedly is unacceptable in production. OpenCrawling maintains state atomically in data/doxis-state/{customer}-{repository}.json:

{
  "lastCrawlTime": "2026-10-17T12:00:00.000Z",
  "documentStates": {
    "3f2a9c1e-5b7d-4e8a-9c0f-1a2b3c4d5e6f": {
      "version": "1",
      "modifiedDate": "2026-09-28T14:15:30.000Z",
      "hash": "a1b2c3d4..."
    }
  }
}

8. Sample OIS Document Payload

Here is an example of an ingested Doxis contract document mapped to canonical OIS JSON:

{
  "id": "doxis://ACME/DB1/documents/3f2a9c1e-5b7d-4e8a-9c0f-1a2b3c4d5e6f",
  "action": "UPSERT",
  "source": {
    "type": "doxis",
    "instance": "https://csb.enterprise.local/restws/publicws/rest/api/v1",
    "customer": "ACME",
    "repository": "DB1"
  },
  "content": {
    "mimeType": "application/pdf"
  },
  "metadata": {
    "name": "MSA_Vendor_Enterprise_2026.pdf",
    "title": "Master Services Agreement 2026",
    "mimeType": "application/pdf",
    "doxis.documentId": "3f2a9c1e-5b7d-4e8a-9c0f-1a2b3c4d5e6f",
    "doxis.customer": "ACME",
    "doxis.repository": "DB1",
    "doxis.documentClass": "ContractDocument",
    "doxis.version": "1",
    "doxis.isLatestVersion": "true",
    "doxis.parentFolderName": "Acme Corp Master File",
    "doxis_desc_ObjectName": "Master Services Agreement 2026",
    "doxis_desc_ObjectNumberExternal": "CTR-2026-0042",
    "doxis_desc_ObjectDate": "2026-03-15"
  },
  "security": {
    "inheritanceEnabled": true,
    "permissions": [
      { "identity": "ACME/Legal", "identityType": "group", "access": "read" },
      { "identity": "ACME/elena.weber", "identityType": "user", "access": "write" },
      { "identity": "ACME/restricted.user", "identityType": "user", "access": "deny" }
    ]
  }
}

9. Configuration & Deployment

Configure the Doxis connector via Spring Boot properties or environment variables:

Testing Connectivity with OpenCrawling CLI

# Check connector health and session auth
$ oc connector check --name Doxis_Repository --type repository

# Run automated integration test script
$ ./scripts/test-doxis-repository-connector.sh