OpenCrawling 1.0.4 Released: The First Official General Availability Milestone for Enterprise AI & Search Ingestion

01 The Enterprise AI Ingestion Crisis & The OpenCrawling Mission

Across enterprises worldwide, AI initiatives face an urgent bottleneck. While large language models (LLMs) and vector retrieval technologies have matured rapidly, the data pipelines feeding them remain fragile, fragmented, and insecure. Enterprise knowledge is trapped across heterogeneous repositories—ranging from modern Microsoft SharePoint Online libraries and M-Files vaults, to legacy ECM platforms like Alfresco, Doxis, and CMIS archives, relational databases, and Apache Iceberg analytical data lakes.

Existing ingestion tools force enterprises into an unacceptable compromise: either write brittle, proprietary point-to-point scripts that drop security permissions, or run monolithic crawlers that collapse under multi-gigabyte files and leak restricted intellectual property into vector indexes where any employee or LLM agent can retrieve it.

OpenCrawling was born to solve this crisis. Guided by the vendor-neutral Open Ingestion Standard (OIS), OpenCrawling establishes a modern, distributed, memory-safe ingestion engine. It treats Access Control Lists (ACLs) as immutable, first-class citizens, decouples heavy binary parsing from vector generation, and delivers native LLM integration through the Model Context Protocol (MCP).

02 Core Architectural Pillars: Java 25, Kafka & Isolated Tika 4

At the heart of OpenCrawling 1.0.4 is a distributed, event-driven streaming architecture designed for extreme throughput, fault isolation, and horizontal scale:

1. Java 25 & Virtual Threads (Project Loom) Everywhere

OpenCrawling leverages Java 25 Virtual Threads throughout the entire runtime. Instead of managing complex reactive callback chains or risking operating system thread exhaustion under heavy I/O, every document fetch, HTTP request, database call, and vector transmission runs on a dedicated virtual thread. Using structured concurrency scopes (StructuredTaskScope), crawlers achieve tens of thousands of concurrent operations with negligible memory overhead and native stack traces.

2. Isolated Apache Tika 4 Text Extraction via PipesForkParser

Extracting text from arbitrary enterprise binaries (corrupted PDFs, complex DOCX documents, macro-enabled spreadsheets, massive compressed archives) has historically been the primary cause of crawler crashes due to memory leaks, native library segfaults, or CPU infinite loops.

In OpenCrawling 1.0.4, document parsing is completely isolated using Apache Tika 4's PipesForkParser. Each parsing task executes in a sandboxed, memory-bounded child JVM process governed by strict timeouts. If a malicious or poisoned document triggers an out-of-memory error or process crash, only the ephemeral child process terminates—the parent ingestion service logs the error, emits an audit event, and continues processing without downtime.

3. Claim-Check Pattern with Redis Stack & Apache Ozone

Message brokers are optimized for high-volume, low-latency streaming—not multi-megabyte binary payloads. OpenCrawling implements the Claim-Check Pattern:

• When a repository connector discovers a binary payload, the raw bytes are streamed directly into an enterprise object store (Apache Ozone S3 / Native RPC or Redis Stack).
• A lightweight OIS Claim-Check Token containing payload reference URIs, SHA-256 integrity digests, and metadata is published to Apache Kafka.
• Downstream consumer workers retrieve the binary payload on-demand only when extraction is executed, keeping Kafka topic throughput blazing fast and broker memory lean.

03 Comprehensive Connector Ecosystem at Launch

OpenCrawling 1.0.4 ships with an expansive matrix of enterprise-grade connectors, each featuring native Zero-Trust security mapping, deterministic chunking, and incremental synchronization:

Target Output & Vector Store Connectors

On the target indexing side, OpenCrawling 1.0.4 provides robust, production-tested output connectors:

• PostgreSQL / PGVector: Relational metadata storage paired with pgvector similarity search and SQL-level ACL filtering.
• Apache Solr 10: Native Solr 10 Output Connector with Dense Vector indexing (HNSW) and hybrid lexical + semantic retrieval.
• OpenSearch 2 & OpenSearch 3: Output connectors featuring k-NN vector search, filtered queries, and cluster security integration.
• Milvus & Zilliz Cloud: High-scale vector database connector with AUTOINDEX automation and masked credential protection in the Admin UI.
• Qdrant: Production vector search engine output connector with payload filtering.
• Vespa: High-throughput search and recommendation engine output connector.
• Apache SeaTunnel: Distributed data integration connector to route documents to dozens of big data ecosystems.
• Apache Ozone Output Connector: High-throughput Migration Mode streaming raw binaries and sidecar .ois.json manifests bit-for-bit to Apache Ozone.
• Hyland OpenArch Parquet Vector Bridge: SNAPPY Apache Parquet vector bridge conforming strictly to OpenArch's embedding.avsc schema.

04 Zero-Trust Security & The Model Context Protocol (MCP)

The most perilous vulnerability in enterprise AI deployments is knowledge leakage. When internal documents are converted to vector embeddings and stored in a shared database without authorization controls, any employee or AI agent querying the index can access confidential financial reports, executive memos, or HR records.

OpenCrawling enforces a strict Fail-Closed Zero-Trust Architecture:

1. Lossless ACL Ingestion: Every connector captures fine-grained access control lists—explicit users, Active Directory / Entra ID security groups, and inheritance flags—and normalizes them into the OIS security schema.
2. Deterministic Chunk Lineage: Document chunks inherit parent document permissions with deterministic, hash-based chunk IDs, ensuring that updates or deletions at the source automatically invalidate downstream vector chunks.
3. Universal Secure MCP Server (oc-mcp-server): OpenCrawling embeds a native Model Context Protocol server exposing tools such as secure_vector_search, openarch_search, and openarch_fetch_document directly to LLM environments like Claude Desktop, Cursor, ChatGPT, and enterprise autonomous agents.

// Example: OpenCrawling Zero-Trust OIS Document Manifest
{
  "oisVersion": "1.0.0",
  "documentId": "sp-sec-2026-audit-report",
  "sourceUri": "https://company.sharepoint.com/sites/legal/Audits/2026.pdf",
  "title": "Confidential Security Audit Report 2026",
  "mimeType": "application/pdf",
  "security": {
    "inheritanceEnabled": false,
    "allowEntities": [
      "user:auditor@company.com",
      "group:SECURITY_COMPLIANCE_DIRECTORS"
    ],
    "denyEntities": [
      "group:EXTERNAL_CONTRACTORS"
    ]
  },
  "metadata": {
    "classification": "TOP_SECRET",
    "retentionYears": 7
  }
}

When an AI agent invokes secure_vector_search, the MCP server evaluates the caller's verified identity tokens against the stored ACLs before returning candidate chunks. If the caller lacks authorization, the chunk is filtered out at the vector database level—it is never passed into the LLM context.

05 Developer & Operator Experience

OpenCrawling 1.0.4 delivers a developer experience designed for rapid adoption, straightforward automation, and effortless day-2 operations:

1. OpenAPI 3.1 & Embedded Swagger UI

Full OpenAPI 3.1 specification compliance with rich interactive documentation available out-of-the-box at http://localhost:8080/swagger-ui.html. Explore endpoints, test crawler job dispatches, introspect connector configurations, and generate SDK clients in any programming language.

2. OpenCrawling CLI (oc-cli) & Homebrew Tap

Operators can control OpenCrawling from the terminal, bastion hosts, or CI/CD pipelines using our Picocli-powered CLI. Install it instantly on macOS and Linux via our official Homebrew tap:

# Install OpenCrawling CLI via Homebrew
brew tap opencrawling/tap
brew install oc-cli

# Test connectivity to a repository connector
oc connector test SharePoint_Online

# Start an on-demand crawl job
oc crawl start SharePoint_Online

# Inspect cluster health and active workers
oc status

3. Official Java Client SDK & Maven Connector Archetypes

Developers building custom integrations can take advantage of the official Java Client SDK (oc-java-client-sdk), which provides type-safe clients for managing crawler jobs and submitting OIS documents.

To create custom plugins, developers can generate complete, production-ready Maven projects in seconds using our official archetypes:

mvn archetype:generate \
  -DarchetypeGroupId=org.opencrawling \
  -DarchetypeArtifactId=opencrawling-archetype-repository-connector \
  -DarchetypeVersion=1.0.4 \
  -DgroupId=com.mycompany.ingestion \
  -DartifactId=custom-repository-connector \
  -Dversion=1.0.0-SNAPSHOT

4. Modern React 19 Admin UI & Lakehouse Audit Ledger

Version 1.0.4 includes a responsive, dark/light theme React 19 + Vite web console featuring live connectivity testing buttons, visual crawl pipeline monitoring, and credential masking.

For governance and compliance, an embedded DuckDB Lakehouse Audit Reporting engine allows operators to run SQL analytical queries directly over historical ingestion events, error rates, and security classifications without installing external data warehouses.

06 Flexible Deployment: Containers, Bare-Metal & VMs

Recognizing that many regulated enterprise environments, defense organizations, and air-gapped data centers prohibit container runtimes (Docker, Podman, Kubernetes), OpenCrawling 1.0.4 provides first-class support for both containerized and container-free environments:

Container Deployments (Docker Hub & GHCR)

Official multi-arch container images are published on Docker Hub (opencrawling/*) and GitHub Packages (ghcr.io/opencrawling/*). Spin up the complete decoupled production stack with a single command:

# Launch full decoupled stack (Kafka, Redis, Postgres, Ollama, Runtime, Embedding, UI)
docker compose -f docker-compose-decoupled-dockerhub.yml up -d

Self-Contained Distribution ZIP Packages (Bare-Metal & VMs)

For native deployments across Linux (RHEL, Ubuntu, Debian, SLES), Windows Server, and macOS, OpenCrawling publishes 5 self-contained Distribution ZIP archives with every release:

Every distribution archive includes native startup wrappers (.sh and .bat), production configuration templates, pre-configured systemd service units, and SHA-256 cryptographic checksums for compliance verification.

07 Quick Start: Up and Running in 3 Minutes

Getting started with OpenCrawling 1.0.4 takes less than three minutes:

# 1. Download the decoupled production Docker Compose definition
curl -O https://opencrawling.org/docker-compose-decoupled-dockerhub.yml

# 2. Start the full decoupled infrastructure and application stack
docker compose -f docker-compose-decoupled-dockerhub.yml up -d

# 3. Access the services:
# • React Admin UI:       http://localhost:8080/
# • OpenAPI / Swagger UI:   http://localhost:8080/swagger-ui.html
# • Secure MCP Server:      http://localhost:8080/mcp
# • gRPC Gateway:           localhost:9095 & localhost:9096

08 Community, Acknowledgments & What's Next

The 1.0.4 GA milestone is the collective achievement of a passionate community of open-source engineers, enterprise architects, and search and AI specialists. We extend our deepest gratitude to everyone who filed bug reports, participated in roadmap discussions, and contributed code.

Special thanks to our core contributors and new community members who contributed to this release:

• @OpenPj (Piergiorgio Lucidi) for lead architecture, the Open Ingestion Standard, Java 25 virtual threading, and core connector design.
• @simonhearne for the Milvus Output Connector, Zilliz Cloud AUTOINDEX integration, and admin security fixes.
• @michaelcizmar for the Qdrant Output Connector and enterprise vector search architecture.
• @textercabaceira (Luís Cabaceira) for the SharePoint Online, M-Files, and Doxis 4 repository connectors, the Vespa output connector, and the ManifoldCF migrator.
• @dpol1 (Davide Polato) for StormCrawler Bolt text extraction and deletion wiring, stable chunk ID hashing, and HTTP OIS job endpoints.
• @mkjsix for developer experience, build workflows, and documentation improvements.

Looking ahead on our 2026/2027 roadmap, we are working on expanded multi-modal embedding pipelines (audio, video, OCR image chunking), native Kubernetes Operator integration, and Apache Software Foundation incubation preparations.