Zero-Trust Data Provider Agent & Active Ingestion: Powering Enterprise Multi-Agent Systems with OpenCrawling

01.The Multi-Agent System (MAS) Challenge

In modern enterprise architectures, Large Language Models (LLMs) and autonomous agent frameworks—such as Spring AI, LangGraph, AutoGen, CrewAI, and Antigravity—are deployed not as monolithic bots, but as collaborative swarms of specialized agents. An agent swarm tasked with preparing an audit package, resolving a regulatory compliance inquiry, or generating an investment briefing must access diverse enterprise content silos: Microsoft SharePoint, Alfresco, Hyland Content Innovation Cloud, IBM FileNet, OpenText Documentum, and Apache Ozone / AWS S3 data lakes.

However, traditional data ingestion architectures fail multi-agent swarms across three critical dimensions:

1. The Context Security Leak: Standard agent tools expose vector databases via raw similarity search without passing caller identity. An agent querying the vector index risks surfacing restricted executive salary data or unredacted PII to unauthorized users.
2. The Batch Latency Gap: Enterprise search traditionally relies on nightly or hourly batch indexing. When a business process concludes in Flowable or Alfresco Process Services (e.g., an urgent vendor contract approved 5 minutes ago), agents cannot access it until hours later.
3. Passive vs. Active Ingestion: Traditional crawlers require human operators to statically pre-configure job boundaries. Autonomous agents facing dynamic goals need to lightweightly explore repository trees (dry-run discovery), formulate targeted crawl scopes on the fly, and initiate scoped ingestion pipelines with idempotency guarantees.

02.The 4-Phase Active Ingestion Architecture

Under the architecture delivered in Issue #203, OpenCrawling provides an end-to-end active ingestion lifecycle:

Phase 1: Lightweight Discovery & Repository Exploration

Agents can inspect repository structures and metadata without triggering expensive full-scale downloads or vector embeddings. Using the new MCP discovery tools (DiscoveryMcpTools), agents retrieve folder trees, MIME types, and preview crawl scopes via dry-run queries across registered connectors (Alfresco, SharePoint, CMIS, Documentum, FileNet, M-Files, Hyland CIC, etc.).

Phase 2: In-Flight Pre-Embedding Governance Gate

Before any extracted text is converted into high-dimensional vectors or committed to persistent data lakes, it passes through the in-flight Governance Gate (GovernanceGate). The gate evaluates PII masking rules—masking email addresses, valid payment card numbers, and validated IBANs with [REDACTED:<TYPE>] placeholders in both text and metadata—and enforces fail-closed policies before chunking and embedding.

Phase 3: Targeted On-Demand Ingestion

When an agent determines that specific documents or sub-folders must be indexed, it submits a targeted ingestion request via MCP or REST (POST /api/jobs/dynamic or POST /api/jobs/dynamic/documents). The job is dynamically orchestrated with custom token-chunking rules, deduplication idempotency keys, and source-level Access Control List (ACL) extraction.

Phase 4: Zero-Trust Identity-Aware Retrieval

When querying the vectorized knowledge base or inspecting persistent cloud migration sidecars (<key>.ois.json), OpenCrawling enforces Fail-Closed Zero-Trust evaluation. Deny rules authoritatively override grants, and callers are bound by OAuth 2.0 RFC 8693 delegated multi-agent identity.

03.Federated Model Context Protocol (MCP) Tools

Issue #203 expands OpenCrawling's native Spring AI Model Context Protocol server (oc-mcp-server) with a comprehensive suite of enterprise tools:

Tool Name	SPI / Component	Purpose & Description
listRepositoryConnectors	DiscoveryMcpTools	Lists all registered repository connectors (SharePoint, Alfresco, CMIS, FileNet, etc.) permitted for agent interaction.
exploreRepositoryTree	DiscoveryMcpTools	Explores folder hierarchies, document IDs, file sizes, and MIME types in a repository without downloading binary content.
queryRepositoryDryRun	DiscoveryMcpTools	Previews what a crawl of a registered repository would ingest without reading content or embedding: returns sample documents and lower-bound counts within the caller's read permissions.
triggerDynamicJob	DynamicJobMcpTools	Dynamically launches a crawl job on a specific folder with custom chunk sizes, idempotency keys, and correlation IDs. Supports validateOnly dry-run.
ingestDocument	DynamicJobMcpTools	Ingests or refreshes a single document by its source repository ID (as discovery returns it), extracting text, evaluating governance, and updating embeddings.
getDynamicJobStatus	DynamicJobMcpTools	Polls live execution status, document count, and stage progression of an active dynamic job.
listSidecarStores	SidecarMcpTools	Enumerates cloud/lakehouse object stores (Apache Ozone, AWS S3) containing migrated OIS sidecars.
readOisSidecars	SidecarMcpTools	Reads *.ois.json metadata sidecars, binary checksums, and business attributes directly from object stores under caller ACL filtering.
secureVectorSearch	McpVectorServer	Federated semantic similarity search across 9 target vector and search backends (pgvector, Milvus, Qdrant, Vespa, Solr, OpenSearch, Elasticsearch, Luxir, and Hyland OpenArch) with Zero-Trust ACL filtering.
fetchJobTraces & getErrorLogs	ObservabilityMcpTools	AIOps telemetry tools allowing AI agents to diagnose cluster health and OpenTelemetry span latencies.

04.In-Flight Pre-Embedding Governance Gate (GovernanceGate)

A cornerstone of Luis Cabaceira's implementation is the in-flight Pre-Embedding Governance Gate (GovernanceGate) in oc-runtime. Unlike traditional search engines that index raw text and attempt post-hoc filtering, OpenCrawling intercepts documents before tokenization and vectorization.

The gate evaluates configured hooks (PreEmbeddingGovernanceHook) sequentially and returns one of four definitive statuses:

• APPROVED: The document passes all hooks unaltered and proceeds to chunking and embedding.
• SANITIZED: One or more hooks masked sensitive entities in the text or metadata; the sanitized payload proceeds to embedding.
• REJECTED: A hook explicitly rejected the document (or sanitization left blank content); nothing is embedded. Whether the document's previously indexed version is purged from the vector database depends on the onRejectedUpdate configuration (PURGE vs. RETAIN).
• UNAVAILABLE: The normative fail-closed safeguard: if a hook throws an unhandled exception or gives no answer, evaluation halts immediately, nothing is embedded, and the earlier indexed version is left untouched.

// Document processing with in-flight GovernanceGate evaluation
GovernanceGate.Result result = governanceGate.evaluate(
    documentId, uri, text, metadata, security
);

switch (result.status()) {
    case APPROVED, SANITIZED -> {
        // Embed the approved or sanitized text and metadata
        proceedToEmbedding(result.text(), result.metadata());
    }
    case REJECTED -> {
        log.warn("Document {} rejected by {}: {}", documentId, result.decidedBy(), result.reason());
        if (governanceGate.purgeOnReject()) {
            purgePreviousVersion(documentId);
        }
    }
    case UNAVAILABLE -> {
        // Fail-closed: hook error or missing answer leaves earlier version intact
        log.error("Governance unavailable for {}: {}", documentId, result.reason());
    }
}

When writing the lakehouse migration sidecar (<key>.ois.json), the gate's outcome is recorded in the normative ois:governance metadata block:

• decision: Normative OIS policy outcome written to sidecars (ALLOW, REDACT, DROP).
• PII Masking (PiiMaskingGovernanceHook): The built-in hook detects three well-formed types: email addresses, payment card numbers (validated with Luhn check digits), and IBANs (validated with MOD 97 check digits). Rather than tokenizing values, it masks them with [REDACTED:<TYPE>] placeholders in text and metadata, recording masked counts (e.g. masked.email, masked.card, masked.iban) in the audit map without storing sensitive data. (SSN and phone number detection can be plugged in via external HTTP governance hooks).
• Tamper-Proof Protected Keys: GovernanceGate.PROTECTED_KEYS stops hooks from changing, adding, or removing protected metadata keys. It involves no signatures or hashes: the protected set encompasses ois:source, ois:governance, ois:provenance, and source-level security/ACL keys (such as acl, security, sys_acl, documentId, uri, and lastModified), guaranteeing that downstream hooks cannot forge or alter access permissions and provenance.

05.Real-Time BPM & Workflow Event Integration

To close the Batch Latency Gap, OpenCrawling now features real-time BPM Document-Transition Webhooks:

POST /api/events/bpm/document-transition
Content-Type: application/json

{
  "source": "flowable-enterprise",
  "eventType": "TASK_COMPLETED",
  "processInstanceId": "proc-99482",
  "documentId": "workspace://SpacesStore/9924-1184-contract.pdf",
  "connectorName": "Alfresco_Corporate",
  "timestamp": "2026-11-12T19:30:00Z"
}

Upon receiving a workflow event from supported engines—Flowable or Alfresco Process Services—OpenCrawling processes targeted document ingestions that usually complete in seconds, not milliseconds (in accordance with our Configuration Guide).

Rather than relying on automatic deletions, an administrator-configured route (matching engine, process, and transition) authoritatively determines whether an event triggers an UPSERT or a DELETE.

In keeping with Zero-Trust principles, the BPM event path enforces robust safeguards:

• Multi-Scheme Authentication: Senders authenticate via HMAC signature, a Basic static token (with a secret of at least 32 characters), or an OAuth 2.0 bearer token carrying the oc:events:write scope.
• Sender-Scoped Deduplication: Inbound event IDs are tracked and deduplicated per sender to avoid duplicate processing on retried webhooks.
• Disabled by Default: The feature is turned off by default (opencrawling.events.bpm.enabled=false), ensuring no unintended exposure.

06.Delegated Multi-Agent Identity (OAuth 2.0 RFC 8693)

When an autonomous agent acts on behalf of a human user, security cannot rely on service-account permissions. OpenCrawling enforces OAuth 2.0 Token Exchange (RFC 8693) delegation:

• onBehalfOf (sub): The verified human user principal whose source permissions bound the entire ingestion and retrieval boundary.
• actorAgent (act.sub): The SPIFFE ID or URI of the autonomous agent orchestrating the operation.
• Normative Fail-Closed Enforcement: If an agent attempts to explore, ingest, or retrieve a document outside the human principal's permission tree, the request is authoritatively blocked with FAIL_CLOSED access denial.

07.Verified by Automated Conformance Testing

To ensure that the reference implementation and the Open Ingestion Standard (OIS) remain in lockstep, Issue #203 introduces the ois-conformance CI verification job.

On every pull request and commit to main, OpenCrawling downloads the live document.schema.json from the OIS specification repository and validates all emitted migration sidecars, tombstones, and governed envelopes using strict format assertions. If the spec and the implementation drift by even a single property, the build fails.

08.Getting Started

You can spin up the full OpenCrawling Zero-Trust Data Provider Agent and MCP Server locally using Docker Compose:

# Clone repository
git clone https://github.com/opencrawling/opencrawling.git
cd opencrawling

# Launch the decoupled stack (PostgreSQL, Kafka, Redis, Ollama, oc-mcp-server)
docker compose -f docker-compose-decoupled.yml up -d

# Verify MCP Server handshake and run end-to-end integration tests
./scripts/test-mcp-decoupled.sh