Enterprise Object Storage: Announcing Apache Ozone Integration in OpenCrawling

1. The Binary Payload Bottleneck in AI Ingestion

In large-scale enterprise deployments, crawling millions of files (PDFs, office documents, scanned records, media assets) can produce hundreds of gigabytes of raw binary content. Streaming these raw payloads directly through message brokers like Apache Kafka introduces severe operational bottlenecks:

• Topic Partition Congestion: Multi-megabyte message payloads hit Kafka's max.message.bytes thresholds and inflate JVM garbage collection pauses.
• Memory Pressure: Downstream consumers experience high memory spikes during message buffering.
• Consumer Degradation: Slow payload transfers disrupt consumer group rebalancing timeouts.

OpenCrawling solves this challenge using the Claim Check Pattern. Instead of sending raw document bytes across message queues, the crawler uploads the raw binary payload to a centralized object store, generating a lightweight URI claim check reference (e.g. s3://claims/doc-uuid.pdf or ofs://s3v/claims/doc-uuid.pdf). This URI circulates through Kafka while consumers retrieve the binary payload only when ready for text extraction or vectorization.

2. Why Apache Ozone? Big Data Object Storage for AI

Apache Ozone is a highly scalable, redundant, and distributed object store optimized for Big Data and cloud-native analytics workloads. Designed to scale to billions of objects, Ozone provides seamless integration with Hadoop ecosystems and AWS S3 SDKs via its high-performance S3 Gateway (s3g).

By bringing Apache Ozone into OpenCrawling as a first-class Claim Check Store provider alongside local filesystem and standard S3 storage, enterprise teams can manage petabyte-scale data lakes without relying on proprietary cloud lock-in.

3. Supported URIs & Protocol Handling

The OzoneClaimCheckStore implementation natively resolves and parses all standard Apache Ozone storage URI schemes:

• S3 Protocol URIs: s3://claims/doc-123.pdf
• Ozone File System (OFS) URIs: ofs://s3v/claims/doc-123.pdf
• HTTP/S Gateway URIs: http://s3g:9878/s3v/claims/doc-123.pdf

4. Configuration & Spring Boot Properties

Activating Apache Ozone in your OpenCrawling runtime instance is completely seamless via Spring Boot configuration properties (application.yml) or environment variables:

spring:
  opencrawling:
    claim-check:
      store: ozone                       # Options: "local", "ozone", or "s3"
      cleanup-on-consume: true           # Delete object from Ozone after ingestion
      ozone:
        s3-endpoint: http://localhost:9878
        volume: s3v                      # Target Ozone Volume name
        bucket: claims                   # Target Ozone Bucket name
        access-key: ozone
        secret-key: ozone-secret
        path-style-access: true          # Required for Ozone S3 Gateway
        region: us-east-1
        auto-create-bucket: true         # Automatically create volume/bucket if missing

Environment Variables Matrix

5. Verifying Stored Content in Apache Ozone

Developers and operations teams can verify raw payloads stored in Apache Ozone using standard S3 toolchains or native Ozone CLIs:

A. Verification via HTTP curl

# Fetch raw payload directly from S3 Gateway
curl -s http://localhost:9878/claims/data_ozone-claim-check-test.txt

B. Verification via AWS CLI

# List objects in claims bucket
AWS_ACCESS_KEY_ID=ozone AWS_SECRET_ACCESS_KEY=ozone-secret \
aws --endpoint-url=http://localhost:9878 s3 ls s3://claims/

# Read object contents
AWS_ACCESS_KEY_ID=ozone AWS_SECRET_ACCESS_KEY=ozone-secret \
aws --endpoint-url=http://localhost:9878 s3 cp s3://claims/data_ozone-claim-check-test.txt -

C. Verification via Native Apache Ozone CLI

# List keys in volume s3v / bucket claims
docker exec -it ozone-om ozone sh key list /s3v/claims

# Inspect key metadata inside Ozone Manager container
docker exec -it ozone-om ozone sh key info /s3v/claims/data_ozone-claim-check-test.txt