Announcing the Open Ingestion Standard (OIS) & OpenCrawling

In the race to build production-grade Retrieval-Augmented Generation (RAG) and enterprise AI applications, one fundamental piece of infrastructure is often treated as an afterthought: data ingestion.

Moving files from enterprise repositories—like SharePoint, Amazon S3, databases, and local file shares—into vector databases is a complex, high-risk process. In most corporate environments, data pipelines are:

1. Security-Blind: They extract text but ignore source permissions (Access Control Lists), leading to critical security leaks where AI models serve sensitive documents to unauthorized users.
2. Monolithic & Coupled: Large file uploads clog message brokers and search APIs, causing timeouts and system-wide bottlenecks.
3. Proprietary & Locked-in: Ingestion configurations are tied to specific search vendors, making it incredibly difficult to migrate or scale.

To solve these challenges, we are proud to introduce the Open Ingestion Standard (OIS) alongside its modern JVM reference implementation, OpenCrawling.

1. Introducing the Open Ingestion Standard (OIS)

The Open Ingestion Standard (OIS) is an open specification, community manifesto, and schema standard for secure, decoupled, and vendor-neutral data ingestion pipelines.

At its core, OIS establishes four foundational pillars:

The Four Pillars of OIS

• Zero-Trust Ingestion (Security First): Document security permissions (User/Group SIDs) are treated as first-class metadata. AI search queries must respect these permissions at query time to ensure users only see content they are authorized to access.
• Vendor-Neutral Interoperability: Payload schemas (JSON) and job configurations (YAML/JSON) are platform-agnostic, preventing vendor lock-in. OIS now natively standardizes properties from CMIS (Content Management Interoperability Services) compliant repositories and ingests BPMN (Business Process Model and Notation) XML workflow definitions directly.
• Event-Driven Decoupling: Pipelines are split into independent, horizontally scalable stages connected by asynchronous queues.
• Claim Check Pattern: To protect the message broker, bulky binary payloads are kept off primary queues. Instead, queue messages carry lightweight reference claims, pulling binary content only when needed.

2. Meet OpenCrawling: The Reference Implementation

To prove the power of OIS, we built OpenCrawling. OpenCrawling is an enterprise data integration platform modeled after Apache ManifoldCF, but rebuilt for the modern era.

The Modern Tech Stack

• Java 25 (Preview Features): Employs Virtual Threads and Structured Concurrency to manage millions of concurrent crawl tasks efficiently with minimal overhead.
• Spring Boot & Spring AI: Drives the backend framework and handles AI integration.
• Apache Kafka: KRaft-mode message broker orchestrating decoupled event stages.
• pgvector: For storing high-dimensional embeddings natively inside PostgreSQL.
• Ollama: For local, secure embedding generation (using models like mxbai-embed-large).
• React, Vite, & Tailwind CSS: A sleek, real-time admin panel for scheduling and telemetry.

3. Architecture in Action: How it Works

OpenCrawling implements the OIS manifesto through a highly decoupled, multi-stage pipeline:

The Claim Check Code

Instead of passing megabytes of binary PDFs or Word documents directly over Kafka, OpenCrawling publishes a lightweight payload containing a URI reference (the claim check).

Here is the Java representation of the Kafka payload:

public record IngestionMessage(
    String documentId,
    String uri,
    Map<String, List<String>> metadata,
    String acl,
    String lastModified
) {}

When the IngestionConsumer picks up this message, it resolves the storage pointer asynchronously, parses the content using Apache Tika, and passes it to a token text-splitter:

@KafkaListener(topics = KafkaConfig.TOPIC_NAME)
public void consume(IngestionMessage message) {
    URI fileUri = new URI(message.uri());
    
    // Resolve the binary payload via Claim Check
    try (InputStream contentStream = Files.newInputStream(Paths.get(fileUri))) {
        String text = tika.parseToString(contentStream);
        
        // Map ACLs & Metadata to the Spring AI Document
        Map<String, Object> metadata = new HashMap<>(message.metadata());
        metadata.put("acl", message.acl());
        
        Document aiDoc = new Document(message.documentId(), text, metadata);
        
        // Chunk and publish to the next topic stage
        List<Document> chunks = textSplitter.apply(List.of(aiDoc));
        for (Document chunk : chunks) {
            kafkaTemplate.send(KafkaConfig.CHUNKS_TOPIC_NAME, chunk.getId(), chunkMsg);
        }
    }
}

This decoupled flow ensures that heavy text parsing, embedding API calls, and database writes never block each other, allowing you to scale each step independently.

4. Preserving Security at Scale

Security is where standard ingestion pipelines fail. If a user queries your RAG application, how do you prevent the LLM from synthesizing an answer based on a file they don't have access to?

By following the OIS Zero-Trust Ingestion pillar, OpenCrawling:

1. Connects to the repository and extracts the Access Control List (ACL) security metadata (like Active Directory Windows SIDs or OAuth Groups).
2. Carries these permissions as structured metadata fields inside the document payload throughout the entire Kafka ingestion pipeline.
3. Indexes them into pgvector along with the embeddings.
4. Allows application queries to perform pre-filtering against these SIDs, guaranteeing that search queries only retrieve documents matching the user's active permissions.

Join the Movement

We believe secure data ingestion should be open, standardized, and built on modern technology.

Here is how you can get involved today:

• Star & Explore the Code: Check out OpenCrawling on GitHub and run the local stack with Docker.
• Read & Contribute to the Standard: Check out the Open Ingestion Standard Specification and review our JSON schemas.
• Join the Discussion: Help shape our roadmap! What should we build next—SharePoint, S3, Qdrant, or Confluence?

Let's build a secure, vendor-neutral future for AI data routing together! 🌐