Zero-Trust AI Retrieval Across Any Vector Store: Announcing the Universal MCP Server & Defense-in-Depth Security Guard

1. The Context Leak Dilemma in Enterprise RAG

In modern Retrieval-Augmented Generation (RAG), Large Language Models (LLMs) and autonomous AI agents rely on vector search tools to discover relevant domain knowledge. However, standard vector databases are fundamentally identity-blind: nearest-neighbor algorithms index high-dimensional embeddings and return the top-K closest chunks regardless of whether the querying user possesses read clearance.

In enterprise settings, this creates a catastrophic vulnerability: an employee asking an AI assistant a general operational question might unwittingly trigger the retrieval of unredacted payroll spreadsheets, executive M&A memos, or restricted HR disciplinary files. Furthermore, enterprise infrastructures are rarely homogenous. While one team indexes documents into PostgreSQL (pgvector), another leverages Vespa for BM25 hybrid ranking, Qdrant or Milvus for multi-million vector scale, OpenSearch for compliance audit integration, Apache Solr for enterprise search, or Luxir for ultra-low latency C++ search.

To solve this at enterprise scale, OpenCrawling has unified its Model Context Protocol implementation into a pluggable, storage-agnostic Zero-Trust gateway that authoritatively validates document Access Control Lists (ACLs) before any text enters an LLM's prompt context.

2. The McpVectorSearchBackend SPI Architecture

At the architectural core of this release is a clean Service Provider Interface (SPI): McpVectorSearchBackend. Decoupled from Spring AI's WebMVC and SSE transport controllers, this interface standardizes security-filtered vector search, document content retrieval, and source enumeration across all storage engines:

package org.opencrawling.runtime.mcp;

public interface McpVectorSearchBackend {
    String backendType();

    List<DocumentSearchResult> secureVectorSearch(
        String query, String userPrincipal, List<String> userRoles,
        int maxResults, double minScore, Integer dimensions);

    DocumentDetailsResult getDocumentContent(
        String documentUri, String userPrincipal, List<String> userRoles,
        Integer dimensions);

    List<Map<String, Object>> listAccessibleSources(
        String userPrincipal, List<String> userRoles, Integer dimensions);
}

The runtime dynamically selects the active backend based on spring.opencrawling.output.type, giving developers and platform architects identical MCP tool signatures whether querying a local PostgreSQL development container or a production Vespa or Qdrant cluster.

3. Full Native Support for 8 Vector Engines

OpenCrawling ships native, optimized implementations of McpVectorSearchBackend for all 8 supported vector database stores, complete with on-the-fly query embedding generation via Ollama or OpenAI:

4. Defense-in-Depth & Explicit Deny Precedence (McpAclEvaluator)

Why isn't database-level pre-filtering alone sufficient? In real-world enterprise deployments, subtle differences in SQL JSON path operators, case sensitivity, or tokenization rules across disparate vector databases can lead to false-positive context leaks.

OpenCrawling enforces a strict two-tier defense-in-depth architecture:

1. Database Pre-Filtering: Prunes the vast majority of non-candidate vectors at the storage layer.
2. Authoritative In-Memory Post-Filtering (McpAclEvaluator): Evaluates every single candidate hit against the caller's credentials in memory before returning data to the MCP client.

The evaluator transparently normalizes heterogeneous security representations:

• Structured OIS Security: Inspects metadata["security"] containing lists of PermissionRule objects (identity, identityType, and access: "read"|"write"|"deny").
• Flattened Security Arrays: Inspects security_allowed_read and security_denied_read string arrays.
• Legacy ACL Strings: Evaluates legacy flat acl tokens (e.g. finance, public, or individual SIDs).

5. Canonical Tools & Backward-Compatible Extensions

The unified MCP server exposes three standard tools regardless of the backend:

• secureVectorSearch(query, userPrincipal, userRoles, maxResults, minScore, dimensions): Performs natural language semantic search with identity verification.
• getDocumentContent(documentUri, userPrincipal, userRoles, dimensions): Retrieves complete document text and metadata, verifying access rights and resolving Claim-Check URIs.
• listAccessibleSources(userPrincipal, userRoles, dimensions): Discovers indexed repository documents accessible to the caller.

Vespa Backward Compatibility

For organizations running existing Vespa integrations, OpenCrawling provides VespaMcpVectorServer, a dedicated delegate that activates when output.type=vespa. It retains legacy tool names (vespaSecureVectorSearch, vespaGetDocumentContent, vespaListAccessibleSources) and supports Vespa rank profile selection (hybrid, semantic, default) while delegating internally to VespaMcpSearchBackend.

AIOps Observability MCP Tools

In addition to knowledge search, the server exposes system-level telemetry tools from ObservabilityMcpTools: fetchJobTraces, getErrorLogs, and queryThroughputMetrics. Connected AI Copilots can inspect live Virtual Thread stack traces and ingestion throughput without accessing sensitive document text.

6. Streamable HTTP & SSE Transports

OpenCrawling adopts Spring AI 2.0's Streamable HTTP transport as its primary MCP protocol interface (POST /mcp), while maintaining full backward compatibility with Server-Sent Events (/mcp/sse).

During protocol negotiation, the client issues a JSON-RPC 2.0 initialize request. The server establishes an active session and returns a unique Mcp-Session-Id response header (e.g. Mcp-Session-Id: 550e8400-e29b-41d4-a716-446655440000). All subsequent requests (notifications/initialized, tools/list, tools/call) pass this session header.

7. Automated Decoupled Integration Test Suite (test-mcp-decoupled.sh)

To ensure enterprise reliability, OpenCrawling includes a dedicated decoupled integration test suite: scripts/test-mcp-decoupled.sh. It can be run directly or through the master test runner:

# Run only MCP tests via keyword filter
./run-integration-tests.sh mcp

The suite executes 16 automated verification steps across a decoupled cluster:

1. Cluster Boot: Boots PostgreSQL/pgvector, Redis, Ollama, Kafka, consumers, and oc-mcp-server.
2. Health Probes: Asserts ollama-model-puller pulled mxbai-embed-large and MCP endpoint is ready.
3. Protocol Handshake: Negotiates JSON-RPC 2.0 initialize and captures Mcp-Session-Id.
4. Tool Introspection: Verifies tools/list schemas for search, retrieval, and AIOps tools.
5. Live AIOps Tool Invocations: Executes fetchJobTraces, getErrorLogs, and queryThroughputMetrics.
6. OIS Multi-ACL Ingestion: Ingests public, role-restricted, and explicit-deny documents through Kafka.
7. Zero-Trust Verification: Proves public users see only public docs, and finance users see finance docs.
8. Explicit Deny Assertion: Proves a user in engineering who also has the contractors role is completely blocked from restricted documents.
9. Multi-Dimension Routing: Queries across dimension-specific tables (vector_store_1024).
10. OIS Tombstone DELETE: Emits a DELETE tombstone and asserts vector eviction from MCP search results.
11. In-Process Parity: Executes JUnit assertions covering all 8 backends and McpAclEvaluator (61/61 passing).

8. Connecting Claude Desktop, Cursor & Agent Frameworks

Connecting your AI agent to OpenCrawling is instantaneous. For Claude Desktop, add the following to claude_desktop_config.json:

{
  "mcpServers": {
    "opencrawling-secure-knowledge": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/client-cli",
        "sse",
        "http://localhost:8080/mcp/sse"
      ]
    }
  }
}

For enterprise Python AI applications (LangChain, LlamaIndex, or AutoGen), connect over the Python mcp SDK:

import asyncio
from mcp import ClientSession
from mcp.client.sse import sse_client

async def main():
    async with sse_client("http://localhost:8080/mcp/sse") as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            
            # Secure Zero-Trust vector retrieval
            response = await session.call_tool(
                "secureVectorSearch",
                arguments={
                    "query": "Fiscal strategy and quarterly projections",
                    "userPrincipal": "alice@company.com",
                    "userRoles": "finance,executive",
                    "maxResults": 5
                }
            )
            print("Search Results:", response.content)

asyncio.run(main())