Dense Vector Search, Quantized HNSW & Zero-Trust kNN Pre-Filtering: Announcing the Elasticsearch 8.x & 9.x Output Connector

1. Bridging Enterprise Repositories to Elasticsearch 8.x & 9.x

Elasticsearch powers search and analytics across thousands of enterprises worldwide—from self-managed on-premises clusters to Elastic Cloud and Elastic Cloud Serverless deployments. With modern dense_vector HNSW indexing, scalar quantization, and Better Binary Quantization (bbq_hnsw), Elasticsearch has evolved into a high-performance vector database for Retrieval-Augmented Generation (RAG) and agentic AI workflows.

Until now, OpenCrawling offered dedicated output connectors for OpenSearch 2.x and 3.x alongside Apache Solr 10, Vespa, pgvector, Milvus, Qdrant, and Luxir. With the release of oc-elasticsearch-output-connector, teams running Elasticsearch 8.x or 9.x can stream documents, precomputed embeddings, and normalized Zero-Trust Access Control Lists (ACLs) directly into Elasticsearch—and expose them immediately to AI agents through the OpenCrawling Secure MCP Server.

🔎 Direct & Decoupled Ingestion

Monolithic & Kafka Ready: Supports both direct in-process extraction/chunking (ElasticsearchOutputConnector) and horizontally scaled Kafka streaming (ElasticsearchStoreWriterConsumer) with automatic shrink-chunk cleanup.

🛡️ Zero-Trust kNN Search

Defense-in-Depth MCP Backend: Pushes caller read keys (security_read_keys) and explicit deny exclusions directly into Elasticsearch's top-level knn.filter, followed by authoritative server-side verification in DocumentAccessGate.

2. Unified Elastic 8.x & 9.x Support via Rest5Client

The connector is built on the official Elastic Java API Client 9.4.x (co.elastic.clients:elasticsearch-java) backed by the modern Rest5Client (Apache HttpClient 5 transport), aligning cleanly with Spring Boot 4 and Java 25 without legacy REST client dependencies.

To allow a single runtime binary to talk to both Elasticsearch 8.x and Elasticsearch 9.x clusters out of the box, ElasticsearchClientFactory negotiates the REST API version header via spring.opencrawling.output.elasticsearch.compatible-with (default 8):

compatible-with: 8 (Default): Attaches Accept: application/vnd.elasticsearch+json; compatible-with=8 and matching Content-Type headers, verified against both Elasticsearch 8.19+ and Elasticsearch 9.4+ clusters.
compatible-with: 9: Uses native 9.x media headers for 9.x-only clusters.
compatible-with: 0: Leaves headers at the underlying client transport default.

3. Automatic `dense_vector` Index Provisioning & Quantization

On startup, ElasticsearchIndexInitializer verifies whether the target index (default enterprise_kb) exists, retrying up to 30 times (with a 3-second interval) so multi-container Docker Compose and Kubernetes deployments can boot concurrently without race conditions. If another worker creates the index simultaneously, resource_already_exists_exception is treated as a clean success.

When creating the index, OpenCrawling configures a strict, RAG-optimized mapping with date_detection: false—ensuring that arbitrary string metadata fields that happen to look like dates in one document never cause mapping rejections in subsequent documents:

{
  "mappings": {
    "date_detection": false,
    "properties": {
      "id":                     { "type": "keyword" },
      "document_id":            { "type": "keyword" },
      "text":                   { "type": "text" },
      "uri":                    { "type": "keyword" },
      "acl":                    { "type": "keyword" },
      "last_modified":          { "type": "date", "ignore_malformed": true },
      "security_inheritance":   { "type": "boolean" },
      "security_allowed_read":  { "type": "keyword" },
      "security_denied_read":   { "type": "keyword" },
      "security_read_keys":     { "type": "keyword" },
      "embeddings": {
        "type": "dense_vector",
        "dims": 1024,
        "index": true,
        "similarity": "cosine",
        "index_options": {
          "type": "hnsw"
        }
      }
    }
  }
}

⚡

First-Class HNSW Vector Quantization: You can cut memory consumption by ~75% (or more) on large vector corpora by setting spring.opencrawling.output.elasticsearch.index-type to INT8_HNSW, INT4_HNSW, or Elasticsearch's Better Binary Quantization (BBQ_HNSW), or use exact brute-force scoring with FLAT / INT8_FLAT.

4. Lifecycle Integrity: Deterministic Chunks, Shrink Purge & Tombstones

Enterprise documents are living artifacts: policies are revised, pages are shortened, permissions are tightened, and obsolete files are deleted. If an ingestion pipeline merely appends or overwrites chunks by index without cleaning up trailing chunks, a shortened document leaves "ghost chunks" in the vector store carrying outdated text and stale ACLs.

oc-elasticsearch-output-connector prevents ghost chunks across both execution paths:

Deterministic Chunk IDs: Every chunk is indexed under {documentId}#chunk-{index}, making re-crawls idempotent.
Shrink Purge in Decoupled Kafka Mode: When ElasticsearchStoreWriterConsumer processes chunk 0 of an updated document, it inspects total_chunks and executes a targeted delete_by_query removing any existing chunk for that document_id where chunk_index >= total_chunks.
Atomic Replace in Direct Mode: ElasticsearchOutputConnector purges prior chunks for the document before streaming new batches (chunked by bulk-batch-size, default 500), inspecting per-item bulk statuses so partial failures are never ignored.
OIS Delete Tombstones: When a repository emits a DocumentAction.DELETE tombstone, all chunks matching document_id or id are immediately purged with conflicts=proceed.

5. Zero-Trust kNN Pre-Filtering & MCP Server Integration

When `spring.opencrawling.output.type=elasticsearch` is active, ElasticsearchMcpSearchBackend powers the canonical Model Context Protocol (MCP) tools (secureVectorSearch, getDocumentContent, and listAccessibleSources).

Every indexed chunk stores normalized read keys (security_read_keys) computed by ReadKeys.forAllowed(effectiveAcl):

user:Alice → u:alice
group:Finance → g:finance
Bare finance → both u:finance and g:finance
Public documents → public

In metadata and shadow authorization modes, ElasticsearchAclQueries.readableBy(userPrincipal, userRoles) pushes the caller's read keys (`ReadKeys.forCaller`) and explicit deny exclusions directly into Elasticsearch's top-level knn.filter clause:

{
  "size": 15,
  "_source": { "excludes": ["embeddings"] },
  "knn": {
    "field": "embeddings",
    "query_vector": [0.0214, -0.0418, ...],
    "k": 15,
    "num_candidates": 100,
    "filter": {
      "bool": {
        "filter": [
          { "terms": { "security_read_keys": ["public", "u:alice", "g:finance"] } }
        ],
        "must_not": [
          { "terms": { "security_denied_read": ["alice", "user:alice", "finance", "group:finance"] } }
        ]
      }
    }
  }
}

🔒

Two-Layer Defense-in-Depth & OpenFGA ReBAC Support: Even after Elasticsearch pre-filters the kNN candidate window, every returned hit passes through OpenCrawling's unified DocumentAccessGate. In intersection and openfga modes, store pre-filtering is automatically bypassed and the candidate window is widened so OpenFGA can evaluate real-time relationship tuples without re-indexing.

6. Elastic Cloud API Keys, Basic Auth & CA Fingerprint TLS

Whether you connect to a local development container, a self-managed TLS cluster with auto-generated certificates, or Elastic Cloud Serverless, ElasticsearchClientFactory supports flexible authentication and transport security:

API Key Authentication: Set spring.opencrawling.output.elasticsearch.api-key to your Base64-encoded Elastic API key (ApiKey <base64> header). Takes precedence over basic auth.
HTTP Basic Authentication: Set username (default elastic) and password.
CA Certificate Fingerprint Pinning: Elasticsearch 8+ and 9+ clusters generate a self-signed root CA on first boot. Instead of copying JKS/PKCS12 files across containers, simply pass the 64-character SHA-256 hex fingerprint via ssl.ca-fingerprint—the client verifies the issuer certificate in the chain against the pinned digest automatically.
PKCS12 / JKS Truststore: Alternatively, configure ssl.trust-store-path, ssl.trust-store-password, and ssl.trust-store-type.

7. Configuration Reference

All settings are bound under spring.opencrawling.output.elasticsearch.* (and configurable through the React Admin UI when spring.opencrawling.output.type=elasticsearch):

Property	Default	Description
`spring.opencrawling.output.elasticsearch.uris`	`http://localhost:9200`	Comma-separated URLs of Elasticsearch nodes
`spring.opencrawling.output.elasticsearch.username`	`elastic`	Basic authentication username (used when `password` is non-empty and `api-key` is blank)
`spring.opencrawling.output.elasticsearch.password`	(empty)	Basic authentication password
`spring.opencrawling.output.elasticsearch.api-key`	(empty)	Base64 API key for Elastic Cloud / Serverless (supersedes basic auth)
`spring.opencrawling.output.elasticsearch.index-name`	`enterprise_kb`	Target index where document chunks and vectors are stored
`spring.opencrawling.output.elasticsearch.dimensions`	`1024`	`dense_vector` dimension count (must match embedding model)
`spring.opencrawling.output.elasticsearch.similarity`	`COSINE`	Vector similarity metric: `COSINE`, `DOT_PRODUCT`, `L2_NORM`, or `MAX_INNER_PRODUCT`
`spring.opencrawling.output.elasticsearch.index-type`	`HNSW`	ANN index type: `HNSW`, `INT8_HNSW`, `INT4_HNSW`, `BBQ_HNSW`, `FLAT`, or `INT8_FLAT`
`spring.opencrawling.output.elasticsearch.bulk-batch-size`	`500`	Maximum chunks sent in a single bulk indexing request
`spring.opencrawling.output.elasticsearch.create-index`	`true`	Automatically provisions the index and mapping on startup if missing
`spring.opencrawling.output.elasticsearch.compatible-with`	`8`	REST API compatibility version (`8` works with 8.x & 9.x, `9` for 9.x-only, `0` for client default)
`spring.opencrawling.output.elasticsearch.ssl.ca-fingerprint`	(empty)	64-char SHA-256 hex fingerprint of the cluster CA certificate
`spring.opencrawling.output.elasticsearch.ssl.trust-store-path`	(empty)	Filesystem path to PKCS12 or JKS truststore for HTTPS nodes

8. Testing & Decoupled Docker Compose Stack

The Elasticsearch Output Connector ships with comprehensive automated verification:

Unit & Testcontainers Integration Suite: 34 unit tests plus 21 Testcontainers integration tests running against both Elasticsearch 8.19.20 (Elasticsearch8OutputConnectorIT) and Elasticsearch 9.4.3 (Elasticsearch9OutputConnectorIT), verifying index creation, kNN read-key pre-filtering, explicit deny enforcement, re-crawl shrink purge, and tombstone deletions.
18-Step End-to-End Decoupled Pipeline Test: A complete multi-container stack (oc-elasticsearch-output-connector/docker/docker-compose-decoupled-with-elasticsearch.yml) with Elasticsearch 8.19.20, Apache Kafka, Apache Ozone Claim-Check Store, Redis, Ollama (mxbai-embed-large), and all OpenCrawling microservices, verified end-to-end including MCP secureVectorSearch and OIS DELETE tombstone propagation:

# Run unit & Testcontainers integration tests against ES 8.19 & ES 9.4
./scripts/test-elasticsearch-connector.sh --with-it

# Run the full 18-step decoupled pipeline + MCP verification suite
./scripts/test-elasticsearch-decoupled.sh

Stream Enterprise Knowledge into Elasticsearch 8.x & 9.x

Try the new Elasticsearch Output Connector in our interactive browser simulator, or explore the source code on GitHub.

Try Live Ingestion Simulator → Read Connectors Wiki Guide 📖