Bidirectional Cloud-Native Content & Parquet Vector Bridging: Announcing the Hyland OpenArch Integration

01 / Bridging Cloud-Native Content Services & Enterprise AI

Hyland OpenArch represents the next generation of open-source, cloud-native content services. Built on Spring Boot, MongoDB, and OpenSearch, OpenArch combines a rich hierarchical document model (SysFile, SysFolder), a declarative query language (HXQL), fine-grained access control lists (sys_acl), and first-class vector embedding storage via the SysHasEmbeddings mixin and Apache Parquet ingestion pipelines.

Until now, enterprises adopting OpenArch faced two disconnected challenges:

• Harvesting OpenArch as a Knowledge Source: Extracting millions of documents, custom schemas, binary streams, and hierarchical ACLs from OpenArch into external vector stores (pgvector, Vespa, Milvus, Qdrant, OpenSearch, Solr, Luxir) or Apache Ozone lakes.
• Enriching & Populating OpenArch as an AI Target: Migrating content from legacy ECM systems (Alfresco, Doxis 4, OASIS CMIS, FileSystem, BPMN engines) into OpenArch while simultaneously generating and attaching OpenArch-native Apache Parquet vector embeddings (embedding.avsc).

With the release of two new first-class modules—oc-openarch-repository-connector and oc-openarch-output-connector—alongside a dedicated OpenArchMcpSearchBackend in oc-mcp-server, OpenCrawling turns Hyland OpenArch into a fully bidirectional participant in the Open Ingestion Standard (OIS) ecosystem.

02 / Source Harvesting: Paged HXQL & Java 25 Structured Concurrency

The oc-openarch-repository-connector module (OpenArchRepositoryConnector, registered as OpenArch_Repository) connects directly to Hyland OpenArch's REST API using the multi-tenant X-HXCS-Repository header and HTTP Basic Authentication. It supports two complementary crawl modes:

• QUERY Mode (Default): Executes arbitrary HXQL statements (defaulting to SELECT * FROM SysFile) via POST /api/query.
• FOLDER Mode: Traverses a specific folder hierarchy starting at rootFolderId (00000000-0000-0000-0000-000000000000 by default), automatically constructing scoped HXQL predicates for sys_parentId (single level) or sys_AncestorIds (recursive subtree traversal when includeSubfolders=true).

Cursor-Based Pagination & Incremental Watermarks

Deep offset pagination degrades rapidly on multi-million-document repositories. OpenArchRepositoryClient natively tracks OpenArch's nextSearchAfter cursor arrays across page batches, falling back to numeric offset pagination only when needed. When incremental=true and lastCrawlTimestamp is configured, the connector automatically injects a sys_modified >= TIMESTAMP '...' filter into the HXQL WHERE clause (preserving any ORDER BY clauses) so only modified documents are harvested.

Java 25 Virtual Threads & Claim-Check Binary Streaming

Each page of OpenArch documents is processed concurrently inside a Java 25 StructuredTaskScope.open() block running on a dedicated named virtual thread factory (openarch-crawler-lane). Every virtual thread task is wrapped in ObservabilityTask.observed(...) so OpenTelemetry trace IDs and job MDC contexts propagate seamlessly across threads:

try (var scope = StructuredTaskScope.open()) {
    for (Map<String, Object> doc : documents) {
        scope.fork(ObservabilityTask.observed("openarch-process-doc", () -> {
            processDocument(doc, properties, isDryRun);
            return null;
        }));
    }
    scope.join();
}

When includeContentStream=true and a document contains a file_content / file:content blob descriptor, the virtual thread streams the binary directly from GET /api/download/{id} into OpenCrawling's ClaimCheckStore (Apache Ozone or Local FS) without buffering large files in heap memory, emitting lightweight openarch://<repositoryId>/<docId> envelopes onto the opencrawling-ingestion Kafka topic.

03 / Output Connector & The Native Parquet Vector Bridge

What makes the oc-openarch-output-connector module truly unique is its dual capability: it acts both as an Enterprise Content Services target (for repository-to-repository migration and ACL synchronization) and as a Parquet Vector Bridge that feeds Hyland OpenArch's native OpenSearch vector engine.

1. Document, Binary & Mixin Upsert

When OpenArchOutputConnector indexes an OIS document (or receives raw streams), it:

1. Streams the binary payload to POST /api/upload (`multipart/form-data`), receiving an OpenArch blob key, digest, and length.
2. Constructs a SysFile document payload enriched with the SysHasEmbeddings mixin (sys_mixinTypes: ["SysHasEmbeddings"]), attaches the uploaded blob reference to file:content, translates the OIS security permissions into OpenArch sys_acl entries, and upserts the document at POST /api/documents/{rootFolderId}?upsert=true.

2. Conforming to OpenArch's embedding.avsc Parquet Specification

Hyland OpenArch ingests vector embeddings asynchronously via SNAPPY-compressed Apache Parquet files that strictly adhere to its internal Avro schema (/schemas/embedding.avsc, namespace com.hyland.openarch.embeddings.avro.Embedding).

OpenCrawling's OpenArchParquetEmbeddingsWriter serializes embedded chunks directly into this exact schema—including nested provenance location unions for text offsets, bounding boxes, media timestamps, and spreadsheet coordinates:

{
  "type": "record",
  "name": "Embedding",
  "namespace": "com.hyland.openarch.embeddings.avro",
  "fields": [
    { "name": "id", "type": "string" },
    { "name": "type", "type": "string" },
    { "name": "vector", "type": { "type": "array", "items": "double" } },
    { "name": "text", "type": ["null", "string"], "default": null },
    { "name": "blobref", "type": ["null", "string"], "default": null },
    {
      "name": "location",
      "type": ["null", {
        "type": "record",
        "name": "Location",
        "fields": [
          { "name": "text", "type": ["null", "TextLocation"], "default": null },
          { "name": "position", "type": ["null", "Position"], "default": null },
          { "name": "timestamp", "type": ["null", "Timestamp"], "default": null },
          { "name": "spreadsheet", "type": ["null", "SpreadsheetLocation"], "default": null }
        ]
      }],
      "default": null
    }
  ]
}

04 / Bidirectional Zero-Trust ACL Mapping

Enterprise RAG cannot compromise on access controls. Both OpenArch connectors share a deterministic, lossless security translator (OpenArchSecurityMapper) that maps Hyland OpenArch's sys_acl Access Control Entries (ACEs) to and from OpenCrawling's Open Ingestion Standard (OIS) SecurityConfig:

OpenArch sys_acl Permission	granted Flag	OIS SecurityConfig.Permission	Reverse Mapping (OIS → OpenArch)
Read, ReadProperties, ReadContent, ReadChildren, Browse	true	access: "read"	permission: "Read", granted: true
Write, ReadWrite, WriteProperties, WriteContent, AddChildren	true	access: "write"	permission: "ReadWrite", granted: true
Delete, DeleteChild	true	access: "delete"	permission: "Delete", granted: true
ManageSecurity, ManageLock, ManageRetention, Everything, All	true	access: "admin"	permission: "Everything", granted: true
Any permission	false	access: "deny" (Explicit Deny Precedence)	permission: "Read", granted: false

Principal types (user vs. group) are preserved across both directions, ensuring that downstream vector databases and OpenCrawling's authoritative McpAclEvaluator enforce explicit deny precedence and group inheritance identically to OpenArch.

05 / Universal MCP Server: 9th Backend & Dedicated OpenArch Tools

With Hyland OpenArch added as the 9th pluggable backend (spring.opencrawling.output.type=openarch) in oc-mcp-server, AI agents and Copilots (Claude Desktop, Cursor, Windsurf, Spring AI, Antigravity) can query OpenArch repositories over SSE and Streamable HTTP using two patterns:

• Canonical SPI Tools: secureVectorSearch, getDocumentContent, and listAccessibleSources automatically delegate to OpenArchMcpSearchBackend, embedding the user's natural-language query into a dense vector, calling OpenArch's POST /api/query/embedding endpoint, hydrating parent documents, and running McpAclEvaluator.isAuthorized(...) before returning any chunk.
• Dedicated OpenArch MCP Tools: Registered directly on McpVectorServer for direct repository interaction:
  • openarch_search(query, userPrincipal, userRoles, maxResults, minScore): Executes hybrid semantic + HXQL search against Hyland OpenArch with Zero-Trust caller ACL enforcement.
  • openarch_fetch_document(documentId, userPrincipal, userRoles): Fetches full document metadata and content by OpenArch UUID or openarch:// URI after verifying caller read permissions.

06 / Configuration, Admin UI & CLI

Configuring both the source and output connectors in application.yml takes only a few lines:

spring:
  opencrawling:
    # 1. Hyland OpenArch Source Repository Connector
    connector:
      openarch:
        enabled: true
        endpoint-url: http://localhost:9080
        repository-id: default
        username: Administrator
        password: ${OPENARCH_PASSWORD:Administrator}
        crawl-mode: QUERY
        hxql-query: "SELECT * FROM SysFile"
        batch-size: 100
        include-acls: true
        include-content-stream: true

    # 2. Hyland OpenArch Output Connector & Parquet Vector Bridge
    output:
      type: openarch
      openarch:
        enabled: true
        endpoint-url: http://localhost:9080
        repository-id: default
        username: Administrator
        password: ${OPENARCH_PASSWORD:Administrator}
        root-folder-id: "00000000-0000-0000-0000-000000000000"
        parquet-batch-threshold: 50
        embeddings-type: mxbai-embed-large
        dimensions: 1024
        upsert: true
        sync-acls: true
        upload-binaries: true

We have also updated the React Admin UI (`ConnectorForm.tsx`) with dedicated Hyland OpenArch configuration forms and a live "Test OpenArch Connectivity" button, and enhanced the OpenCrawling CLI (`oc-cli`) with a new test subcommand alias:

# Verify Hyland OpenArch connectivity from the CLI
oc connector test OpenArch_Repository

# Trigger an on-demand crawl job
oc crawl start OpenArch_Repository