Sub-Millisecond Vector Search & Payload ACL Pre-Filtering: Announcing the Qdrant Output Connector

01. Why Qdrant for Enterprise Vector Search?

As AI applications expand across enterprise ecosystems, vector storage demands extreme query throughput, low memory footprint, and strict security guarantees. While vector databases excel at similarity search, enterprise content repositories (like SharePoint, Alfresco, and Iceberg) require document-level Access Control Lists (ACLs) to ensure users only retrieve information they have explicit rights to see.

Qdrant is engineered in Rust with SIMD hardware acceleration, making it one of the fastest vector engines in existence. But what truly sets Qdrant apart for enterprise RAG is its Payload Index Engine. By supporting KEYWORD payload indexes directly alongside vector points, Qdrant allows filtering by payload attributes (such as user security SIDs) before distance calculations occur, preventing security leaks without sacrificing search performance.

02. Architecture & Decoupled Data Flow

The oc-qdrant-output-connector integrates directly into OpenCrawling's event-driven decoupled microservice architecture:

1. Repository Ingestion: Repository connectors (Filesystem, Alfresco, Flowable, Camunda, S3) discover documents and publish lightweight claim-check messages to Kafka.
2. Text Extraction & Chunking: IngestionConsumer extracts clean text via Apache Tika and generates chunked payloads with associated document ACL security SIDs.
3. Scalable Embedding: oc-embedding-service dynamically computes vector embeddings (via Ollama or OpenAI) and pushes embedded chunks to Kafka.
4. High-Speed Binary Upsert: QdrantStoreWriterConsumer receives the embedded chunks and uses the gRPC Java SDK (io.qdrant:client) to upsert PointStruct batches into Qdrant over port 6334.

03. Automated Collection Provisioning & Payload ACL Indexing

Upon startup, QdrantCollectionInitializer verifies whether the configured target collection (e.g. enterprise_kb) exists. If not, it automatically provisions the collection with the desired vector dimensions and distance metric (COSINE, DOT, or EUCLID).

Simultaneously, the connector creates KEYWORD payload indexes on the security payload fields:

• security_allowed_read: List of user/group SIDs permitted to access the document.
• security_denied_read: Explicit deny SIDs taking precedence over allow rules.

// Example Qdrant Point Payload Structure generated by oc-qdrant-output-connector
{
  "id": "c8a912e4-712d-4b92-b2fa-1092a9f1a091",
  "vector": [0.0142, -0.0521, 0.0892, ...],
  "payload": {
    "text": "Enterprise quarterly financial report and forecasts...",
    "uri": "file:///data/finance/q3_report.pdf",
    "security_allowed_read": ["ROLE_FINANCE", "USER_S-1-5-21-3623811015"],
    "security_denied_read": ["ROLE_EXTERN"],
    "lastModified": "2026-07-29T18:00:00Z"
  }
}

04. Configuration & Deployment

The Qdrant Output Connector is fully configurable via standard Spring Boot YAML properties or environment variables:

05. Get Started Today

The Qdrant Output Connector is available starting in OpenCrawling 1.0.0-SNAPSHOT. Check out the documentation or run the Docker Compose environment to start building secure, low-latency RAG pipelines with Qdrant!