Auto-Narrativization Copilot: Bridging Structured Schemas to Vector Embeddings

01. The Structured Data Vector Dilemma

Vector embedding models (such as mxbai-embed-large or OpenAI text-embedding-3) are trained primarily on natural, continuous prose. When ingesting structured content sources like Apache Iceberg tables, database rows, or enterprise document metadata, simply concatenating field keys and values (e.g. id=123, amount=450.0, region=EU) results in fragmented semantic spaces.

To achieve peak retrieval performance in RAG (Retrieval-Augmented Generation) applications, structured records should be narrativized into coherent human sentences (e.g. "Transaction 123 processed an amount of €450.0 within the EU region."). However, hand-writing templates for hundreds of tables and connectors is tedious and error-prone.

02. Introducing the Auto-Narrativization Copilot

OpenCrawling's Auto-Narrativization Copilot solves this challenge by dynamically converting connector field schemas into expressive Mustache templates and typed mock datasets.

POST /api/transformation/copilot/generate
Content-Type: application/json

{
  "connectorType": "iceberg",
  "fields": [
    { "name": "id",        "type": "STRING",    "description": "Primary transaction ID" },
    { "name": "amount",    "type": "DOUBLE",    "description": "Monetary value in EUR" },
    { "name": "region",    "type": "STRING",    "description": "Geographical region code" },
    { "name": "timestamp", "type": "TIMESTAMP", "description": "Transaction date and time" }
  ]
}

Response:

{
  "template": "Transaction {{id}} processed an amount of €{{amount}} in region {{region}} at {{timestamp}}.",
  "mockData": {
    "id": "TX-98421",
    "amount": 1250.50,
    "region": "EU-WEST",
    "timestamp": "2026-07-23T14:30:00Z"
  }
}

03. Per-Job Granularity in Admin UI & Runtime

Because enterprise teams frequently crawl the same repository source for different analytical purposes, narrativization is configured at the Per-Job level.

Inside the OpenCrawling Admin UI, each ingestion pipeline form includes a dedicated Auto-Narrativization Copilot panel:

• Interactive Schema Introspection: Automatically populates table fields from connectors like IcebergRepositoryConnector.
• One-Click Generation: Click "Generate with Copilot" to fetch an AI-crafted Mustache template.
• Live Template Preview: Instant local preview showing rendered narrative text filled with sample mock values.
• Job Persistence: Stores the template directly within the job payload (`JobDTO.narrativization`).

04. Stream Execution via MustacheTransformationConnector

During job execution, `JobOrchestrator` inspects the job's `NarrativizationConfig`. When enabled, it passes document streams through the high-performance `MustacheTransformationConnector` in `oc-core`:

// Applied during scan in virtual thread
if (mustacheConnector != null) {
    doc = mustacheConnector.transform(initialDoc).blockFirst();
}

The rendered narrative replaces the document stream before metadata references are published to Apache Kafka, ensuring downstream embedding models (in oc-embedding-service) process rich, contextualized prose.