Lightweight Hybrid Search & Dense Vector kNN: Announcing the Luxir Output Connector

01. What is Luxir and Why Luxir for Enterprise AI?

Luxir is an open-source, lightweight search engine built from the ground up to provide high-speed full-text indexing, dense vector kNN similarity search, and hybrid ranking using Reciprocal Rank Fusion (RRF). Unlike legacy search clusters that require complex JVM orchestrations or heavy multi-container topologies, Luxir runs as a sleek, low-footprint single process with native JSON REST APIs over HTTP on port 9400.

In enterprise Retrieval-Augmented Generation (RAG) pipelines, simplicity, fast start times and zero client footprint are critical. Luxir brings a refreshingly modern approach with dynamic field typing conventions and native vector support, making it an ideal choice for both local AI development and high-throughput production workloads.

02. End-to-End Pipeline Architecture

The Luxir Output Connector integrates seamlessly into OpenCrawling's event-driven, decoupled microservice pipeline:

Once document chunks are converted to high-dimensional embeddings by oc-embedding-consumer (powered by Spring AI and Ollama's mxbai-embed-large), they are published to the opencrawling-embedded Kafka topic. The LuxirStoreWriterConsumer processes these messages in batches and indexes them directly into Luxir via its batch /_update REST endpoint.

03. Dynamic Schema Conventions & Auto-Provisioning

Luxir features dynamic schema type inference based on field name suffix conventions:

• _t: Text field analyzed for full-text search (e.g. text_t, title_t)
• _s: Keyword string field (e.g. doc_id_s, uri_s, security_inheritance_s)
• _ss: Multi-valued string array for tags and security tokens (e.g. acl_ss)
• _v: Dense floating-point vector for kNN search (e.g. embedding_v)
• _vs: Multi-vector array for passage retrieval
• _i: Integer numeric field (e.g. chunk_index_i)

To ensure optimal vector similarity search, OpenCrawling includes LuxirCollectionInitializer. Upon startup, it automatically provisions the collection and declares explicit vector field properties using Luxir's _schema endpoint:

POST /collections/opencrawling/_schema
{
  "fields": {
    "embedding_v": {
      "type": "vector",
      "dims": 1024,
      "metric": "cosine"
    },
    "passages_vs": {
      "type": "vector",
      "dims": 1024,
      "metric": "cosine",
      "multi": true
    }
  }
}

04. Zero-Trust Security ACLs & OIS v1.1 Tombstone Lifecycle

In accordance with the Open Ingestion Standard (OIS), every document chunk indexed into Luxir carries granular enterprise security tokens. Access Control Lists are mapped into dynamic string array fields:

• acl_ss: Consolidated security identifiers and role tokens.
• security_allowed_read_ss: Principals granted explicit read access.
• security_denied_read_ss: Principals explicitly denied read access (enabling Zero-Trust precedence).
• security_inheritance_s: Inheritance flag mapped as a string ("true" or "false") for dynamic compatibility.

Furthermore, when an upstream repository document is deleted, OpenCrawling emits an OIS tombstone message (action: DELETE). The LuxirStoreWriterConsumer processes this tombstone by invoking Luxir's native delete_ids batch command, ensuring that deleted knowledge is immediately and permanently purged from downstream RAG retrieval indexes:

POST /collections/opencrawling/_update
{
  "delete_ids": ["doc-001_chunk_0"],
  "commit": {}
}

05. Configuration Properties

The Luxir Output Connector is configured via standard Spring Boot properties prefixed with spring.opencrawling.output.luxir.*:

06. Executing Full-Text & Dense Vector Queries

Querying indexed content in Luxir is fast and intuitive using JSON payloads over HTTP POST:

1. Document Count

curl -s -X POST "http://localhost:9400/collections/opencrawling/_search" \
     -H "Content-Type: application/json" \
     -d '{"query": "*:*", "get_number": true}' | jq .

2. Full-Text BM25 Search

curl -s -X POST "http://localhost:9400/collections/opencrawling/_search" \
     -H "Content-Type: application/json" \
     -d '{"query": "text_t:OpenCrawling"}' | jq .

3. 1024-Dimension Dense Vector kNN Search

curl -s -X POST "http://localhost:9400/collections/opencrawling/_search" \
     -H "Content-Type: application/json" \
     -d '{
       "query": {
         "knn": {
           "field": "embedding_v",
           "query": [0.0123, -0.0456, 0.0891, ...],
           "k": 5
         }
       }
     }' | jq .

07. Admin UI & Real-Time Diagnostics

The OpenCrawling Admin UI (oc-admin-ui) fully supports the Luxir connector out-of-the-box. Administrators can select Luxir Hybrid Search Store from the output connector dropdown, configure collection endpoints, and click Check Connection to trigger dynamic connectivity testing through ConnectorCheckerService.

08. Automated Decoupled Integration Testing

OpenCrawling ships with a complete Docker Compose decoupled definition (docker-compose-decoupled-with-luxir.yml) and an automated test suite. The script sets up the full stack—Luxir, Kafka, Zookeeper, Redis, Ollama, and OpenCrawling microservices—and verifies the entire pipeline:

# Run end-to-end integration test
./scripts/test-luxir-decoupled.sh