Auto-Narrativization Copilot: Bridging Structured Schemas to Vector Embeddings

01. The Structured Data Vector Dilemma

Vector embeddings work best with rich, contextual text. When indexing unstructured documents like PDFs, Word files, or HTML pages, embedding models accurately capture semantic meaning. However, modern enterprise data pipelines increasingly ingest semi-structured and structured dataset records—such as database tables, Apache Iceberg catalogs, or CSV exports.

Directly concatenating raw JSON strings (e.g., {"user_id": 402, "status": 3, "rev": 1205.50}) produces sparse, noisy vector representations. Important semantic context is lost because embedding models cannot infer field relationships without natural language structure.

02. How the Auto-Narrativization Copilot Works

OpenCrawling's Auto-Narrativization Copilot bridges this gap by transforming raw JSON schemas into human-readable narrative prose before documents enter the embedding pipeline.

+-----------------------+     +-------------------------------+     +------------------------------+
|  Structured Dataset   | --> |  Auto-Narrativization Copilot | --> | Generated Narrative Template |
| Schema (e.g. Iceberg) |     |  (Ollama llama3.2 / Fallback) |     | (Mustache Template & Mock)   |
+-----------------------+     +-------------------------------+     +------------------------------+

The copilot workflow consists of three automated steps:

1. Schema Introspection: Extracts field names, data types, and sample key-value maps from the ingestion source.
2. AI Template Synthesis: Invokes TemplateGenerationCopilot using Spring AI connected to a local Ollama instance running llama3.2 (or a deterministic offline fallback template generator).
3. Mustache Rendering: Generates a reusable Mustache template (e.g. "User {{user_id}} created transaction {{transaction_id}} for amount ${{amount}}") alongside mock dataset instances for UI validation.

// Example Copilot Request Payload
{
  "fields": ["customer_name", "account_status", "monthly_spend", "region"],
  "sampleData": {
    "customer_name": "Acme Corp",
    "account_status": "Active",
    "monthly_spend": 4500.00,
    "region": "US-East"
  }
}

03. Per-Job Granularity in Admin UI & Runtime

Because enterprise teams frequently crawl the same repository source for different analytical purposes, narrativization is configured at the Per-Job level.

Inside the OpenCrawling Admin UI, each ingestion pipeline form includes a dedicated Auto-Narrativization Copilot panel:

• Interactive Schema Introspection: Automatically populates table fields from connectors like IcebergRepositoryConnector.
• One-Click Generation: Click "Generate with Copilot" to fetch an AI-crafted Mustache template.
• Live Template Preview: Instant local preview showing rendered narrative text filled with sample mock values.
• Job Persistence: Stores the template directly within the job payload (`JobDTO.narrativization`).

04. Stream Execution via MustacheTransformationConnector

During job execution, `JobOrchestrator` inspects the job's `NarrativizationConfig`. When enabled, it passes document streams through the high-performance `MustacheTransformationConnector` in `oc-core`:

// Applied during scan in virtual thread
if (mustacheConnector != null) {
    doc = mustacheConnector.transform(initialDoc).blockFirst();
}

The rendered narrative replaces the document stream before metadata references are published to Apache Kafka, ensuring downstream embedding models (in oc-embedding-service) process rich, contextualized prose.