A few weeks ago, we published the core manifesto for the **Open Ingestion Standard (OIS)**. Our target was clear: provide a secure, decoupled, and vendor-neutral blueprint to ingest enterprise data into RAG and LLM systems.

Today, we are taking a major step forward by publishing our official Specification Roadmap. The goal is to evolve OIS from a set of design schemas used by our reference implementation, OpenCrawling, into a fully standardized, community-governed specification.

timeline title OIS Specification Roadmap Phase 1 : Schema Stabilization (OIS 1.0) : RFC Process Initialization : Registry Publishing Phase 2 : Multi-language SDKs (Python, TS, Go) : CLI Validation Tools : Conformance Test Suite Phase 3 : LangChain & LlamaIndex Loaders : Apache ManifoldCF Compatibility : Open Governance Model

Here is an in-depth breakdown of the three phases outlined in the newly released OIS Roadmap:

Phase 1: Schema Stabilization (OIS 1.0)

Before standardizing developer tooling, we must guarantee that our core JSON validation schemas are stable, production-ready, and future-proof. During this phase, the steering team will focus on:

  • OIS 1.0.0-RC1 Release: Finalizing the validation constraints of document.schema.json and job.schema.json, including the newly introduced CMIS properties schemas and BPMN workflow definition mappings.
  • Setting up the RFC (Request for Comments) Process: Opening GitHub issue boards for structural amendments. Our first planned proposal is RFC-001 (Security Identity Mapping), which tackles how to standardize Active Directory Windows SIDs, OAuth scopes, and Okta group claims inside JSON arrays.
  • Registry Publishing: Serving version-controlled JSON schema files at fixed, public URLs so that IDEs and CI/CD testing pipelines can automatically validate payloads against them.

Phase 2: Multi-Language SDKs & Validation Tooling

An open standard is only as good as its developer experience. To lower the barrier to entry, Phase 2 centers on building CLI utilities and helper libraries in major programming ecosystems:

  • ois-validator CLI: A lightweight, cross-platform CLI tool that runs in any CI pipeline or local shell. Developers can validate document streams and job configs with a single command (e.g. ois validate my-document.json).
  • Lightweight Client SDKs:
    • Python: For AI developers building custom ingest scripts.
    • TypeScript: For web developers and Node.js ingestion microservices.
    • Go: For high-performance, low-footprint connector agents.
  • Conformance Test Suite: A standard testing framework containing mock payloads, complex edge-case ACL trees, and job configurations to certify that external crawlers are OIS-compliant.

Phase 3: Broad Framework Integration & Open Governance

The final phase drives mainstream adoption, aligns OIS with popular developer orchestrators, and shifts the specification into a vendor-neutral governance model:

  • AI Orchestrator Loaders: Contributing loaders to LangChain, LlamaIndex, and Haystack. This will allow those platforms to parse OIS document files out-of-the-box, retaining security permissions implicitly.
  • Connector Bridging: Partnering with established open-source projects (like Apache ManifoldCF) to write translators, letting legacy systems export crawled data directly into OIS-compliant JSON formats.
  • Transition to Open Governance: Transitioning steering control of the specification to an established neutral body (such as the CNCF or the Linux Foundation) to guarantee that no single company dominates the standard's direction.

Get Involved!

The OIS Roadmap is fully public and open-source. We are looking for security architects, software engineers, and database maintainers to review our schemas and join the discussion.

Read the official roadmap details in the open-ingestion-standard repository, open an RFC issue, or share your thoughts on GitHub! 💬