New Connector: Apache Iceberg Repository Connector for OpenCrawling

1. Bridging Enterprise Data Lakehouses to AI

Apache Iceberg has emerged as the open table format standard for modern data lakehouses, enabling ACID transactions, time travel, and schema evolution over massive petabyte-scale Parquet datasets stored in S3 or HDFS.

However, bridging table data into Retrieval-Augmented Generation (RAG) and vector search pipelines has historically required custom Spark or Flink jobs. With the oc-iceberg-repository-connector, OpenCrawling connects directly to your Iceberg catalogs, scans Parquet data files concurrently, and converts records into standard RepositoryDocument streams with zero intermediate data copies.

2. Architecture & Concurrent Scanning

The connector implements the standard RepositoryConnector interface. During a scan, it plans FileScanTasks over the target table schema and forks task execution across Java 25 Virtual Threads managed by StructuredTaskScope:

3. Configuration & Constructor Parameters

The IcebergRepositoryConnector can be configured via Java properties or Spring environment settings:

// Example Java initialization connecting to a REST Catalog
IcebergRepositoryConnector connector = new IcebergRepositoryConnector(
    "rest",                             // catalogType: rest, hive, hadoop, glue, in-memory
    "http://iceberg-rest:8181",         // catalogUri
    "s3://warehouse-bucket/iceberg",    // warehouse location
    "id"                                // idColumn (optional primary key column)
);

Catalog Support Matrix

4. Document Identifiers & URI Scheme

Each table record is serialized into a RepositoryDocument carrying a uniform URI identifier format:

iceberg://<table-identifier>/<document-id>

Document IDs are deterministically resolved in the following priority order:

1. Explicit primary key column value passed via idColumn.
2. Composite keys constructed from Iceberg schema identifierFieldNames().
3. SHA-256 fingerprint hash generated automatically from record JSON payload bytes.