Hybrid Vector & BM25 Lexical Ranking at Enterprise Scale: Announcing the Vespa Output Connector

01. Why Vespa for Production-Grade Enterprise RAG?

In enterprise Retrieval-Augmented Generation (RAG), pure vector similarity is often not enough. Searching for exact part numbers, legal clause identifiers, or domain-specific acronyms requires traditional BM25 keyword matching alongside dense vector embeddings. Vespa (developed and open-sourced by Yahoo) stands out in the engine landscape by evaluating lexical, vector, and structured query expressions within a single unified rank profile in real-time at massive scale.

The oc-vespa-output-connector bridges enterprise repository ingestion (from Filesystem, Alfresco, Flowable BPMN, Iceberg, and more) directly into Vespa schemas, enabling hybrid retrieval workflows that out-perform single-mode vector search engines while respecting strict document-level security.

02. Architecture & Decoupled Pipeline Integration

The Vespa Output Connector operates seamlessly in both direct (single-JVM) and decoupled Kafka-driven microservice deployments:

Dynamic Multi-Dimension Vector Routing

Different embedding models produce different vector dimensions (e.g. all-minilm at 384, nomic-embed-text at 768, and mxbai-embed-large at 1024). Rather than forcing a single fixed vector dimension per instance, VespaDocumentMapper.resolveDocumentType() inspects the vector array of each chunk in real-time and routes it to the corresponding Vespa document type:

• 384 dimensions → opencrawling_chunk_384
• 768 dimensions → opencrawling_chunk_768
• 1024 dimensions → opencrawling_chunk_1024
• Custom/fallback dimensions → Default document type (opencrawling_chunk)

This allows multiple ingestion jobs using different AI embedding models to run side-by-side without restarting containers or redeploying schemas.

03. ACL-Aware Security & Strongly Typed Schema Mapping

Vespa enforces strict schema validation: feeding an undeclared field causes immediate ingestion rejection. OpenCrawling solves this by mapping standard metadata keys (`chunk_id`, `text`, `uri`, `acl`, `lastModified`) directly to Vespa schema fields while bundling arbitrary metadata into a unified `metadata_json` blob.

Security rules from the Open Ingestion Standard (OIS) are indexed on every fed document:

// Example Vespa Document Structure generated by oc-vespa-output-connector
{
  "fields": {
    "chunk_id": "doc-9042_chunk_01",
    "text": "Quarterly Financial Analysis & Risk Disclosures...",
    "uri": "file:///data/finance/q3_report.pdf",
    "acl": "finance-admin",
    "lastModified": "2026-08-01T12:00:00Z",
    "security_inheritance": true,
    "security_allowed_read": ["alice@company.com", "finance-team"],
    "security_denied_read": ["external-contractor"],
    "metadata_json": "{\"title\":[\"Q3 Financial Report\"],\"author\":[\"Corporate Finance\"]}",
    "embedding": { "values": [0.0123, -0.0456, 0.0891, ...] }
  }
}

04. Admin UI Model Insights & Bundled Schema Deploy

The oc-admin-ui Output Connector tab includes a dedicated Vespa Model Insights control panel (`/api/vespa/*` REST endpoints) providing:

• Live Health Checks (GET /api/vespa/health): Proxies Vespa container `/state/v1/health`.
• Per-Document-Type Chunk Counts (GET /api/vespa/document-counts): Displays live indexed counts for `opencrawling_chunk`, `_384`, `_768`, and `_1024`.
• Interactive Hybrid Query Tester (POST /api/vespa/query): Runs BM25, vector (`semantic`), or fused (`hybrid`) queries directly from the UI.
• One-Click Schema Deployment (POST /api/vespa/deploy/bundled): Deploys the classpath-packaged `vespa-app/` package directly to the Vespa Config Server.

05. Vespa-Native Secure MCP Server

The release includes VespaMcpVectorServer, activating automatically when spring.opencrawling.output.type=vespa. It exposes three enterprise tools over the Model Context Protocol (MCP):

06. Configuration Parameters & Quickstart

Spinning Up via Docker Compose & Running Integration Tests

# Boot the full decoupled pipeline with Vespa
docker compose -f oc-vespa-output-connector/docker/docker-compose-decoupled-with-vespa.yml up -d --build

# Run the standalone Vespa connector smoke test
./scripts/test-vespa-connector.sh

# Run the full decoupled pipeline test with Model Insights & MCP assertion
./scripts/test-vespa-decoupled.sh

07. Community Acknowledgments

A very special thanks to Luis Cabaceira for architecting and contributing the Vespa Output Connector to OpenCrawling! His contribution bringing Yahoo's vespa-feed-client integration, dynamic multi-dimension vector routing, ACL pre-filtering, and native secure MCP search into the project is a major milestone for enterprise-grade hybrid retrieval in OpenCrawling.