Embedding Scale-Out: Decoupling the AI Vectorization Layer in OpenCrawling

1. The Bottleneck: Why Embedding Was the Chokepoint

In the original monolithic OpenCrawling runtime (oc-runtime), the entire pipeline — crawling, text extraction, chunking, embedding, and vector writing — ran inside a single Spring Boot application. While this simplified deployment and local development, it introduced a critical scalability constraint: embedding generation.

Generating high-dimensional embedding vectors (e.g., 1024-dim with mxbai-embed-large or 768-dim with nomic-embed-text) is computationally expensive. Each document chunk requires a forward pass through a deep learning model. In a monolithic deployment, a single slow embedding step blocks the entire Kafka consumer thread, causing backpressure that saturates the chunk topic partition lag and degrades end-to-end throughput.

2. The Solution: oc-embedding-service

The new oc-embedding-service module is a standalone Spring Boot application that contains only the embedding logic, decoupled from the ingestion runtime. It participates in the pipeline purely through Kafka topics, following the Claim Check Pattern:

The EmbeddingConsumer listens to the opencrawling-chunks topic, reads each DocumentChunkMessage, delegates to the EmbeddingModelFactory to route the text to the appropriate model engine (Ollama, OpenAI, or HuggingFace), and publishes the resulting DocumentEmbeddedMessage containing the pre-computed float vector to the opencrawling-embedded topic.

3. Inside the Code: EmbeddingModelFactory

The heart of the new service is the EmbeddingModelFactory. Instead of hardcoding a single embedding model, this factory reads the Transformation Connector engine configuration from the message metadata and dynamically selects the appropriate embedding provider:

@Component
public class EmbeddingModelFactory {

    private final OllamaEmbeddingModel ollamaModel;
    private final OpenAiEmbeddingModel openAiModel;

    public EmbeddingModel resolve(String engineType) {
        return switch (engineType.toLowerCase()) {
            case "openai"  -> openAiModel;
            case "ollama"  -> ollamaModel;
            default        -> ollamaModel; // safe fallback
        };
    }
}

4. Horizontal Scaling with Docker Compose

Because oc-embedding-service is fully stateless, it can be scaled out to any number of replicas using a single Docker Compose command. Kafka automatically rebalances partitions across the consumer group:

# Scale the embedding service to 3 replicas
docker compose -f docker-compose-apps.yml scale oc-embedding-service=3

# Verify the consumer group rebalance
docker exec -it opencrawling-kafka kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 \
  --describe --group opencrawling-embedding-group

5. Running Locally in Development Mode

For local development, run the embedding service in a separate terminal alongside the main runtime:

# Terminal 1: Start the main ingestion runtime
mvn spring-boot:run -pl oc-runtime -Dspring-boot.run.profiles=dev

# Terminal 2: Start the embedding microservice
mvn spring-boot:run -pl oc-embedding-service

# Terminal 3: Start the Admin React UI
cd oc-admin-ui && npm run dev

6. Fully Decoupled Docker Deployment

For production-grade deployments, the docker-compose-decoupled.yml file spins up five separate containerized services, each communicating exclusively through Kafka topics:

1. oc-crawler — Repository Connector & Claim Check Publisher
2. oc-ingestion-consumer — Apache Tika Extractor & Token Chunker
3. oc-embedding-service — Embedding Generator (horizontally scalable)
4. oc-writer-consumer — Vector Store Writer (pgvector/Elasticsearch/Qdrant)
5. oc-mcp-server — Secure MCP Server for AI Agents

# Build all decoupled service images
docker compose -f docker-compose-decoupled.yml build

# Start the entire pipeline
docker compose -f docker-compose-decoupled.yml up -d

# Run the automated end-to-end integration test
./scripts/test-docker-decoupled.sh

7. Kafka Topic Partition Configuration

Configure your Kafka topics with a partition count matching your maximum desired replica count. To support up to 8 embedding replicas:

# opencrawling-chunks and opencrawling-embedded topics
partitions: 8
replication-factor: 1

Kafka guarantees that each partition is consumed by at most one consumer within the same consumer group (opencrawling-embedding-group). With 8 partitions and 3 replicas, Kafka will assign approximately 2–3 partitions per replica automatically.

8. Ollama Multi-Model Support

The release adds automatic configuration of multiple Ollama embedding models routed to separate pgvector tables by dimension:

• mxbai-embed-large (1024-dim) → vector_store_1024
• nomic-embed-text (768-dim) → vector_store_768
• all-minilm (384-dim) → vector_store_384

docker exec -it ollama ollama pull mxbai-embed-large
docker exec -it ollama ollama pull nomic-embed-text
docker exec -it ollama ollama pull all-minilm

9. What’s Next

• HuggingFace Inference Endpoint Support — Route chunks to cloud-hosted HuggingFace models without local GPU infrastructure.
• Adaptive Batching — Group multiple chunks into embedding batch requests, reducing API call overhead.
• Prometheus Metrics & Kubernetes HPA — Expose consumer lag metrics enabling Kubernetes to auto-scale embedding replicas based on Kafka partition lag.
• Dynamic Model Routing — Per-connector model configuration, allowing different document types to be vectorized with different models simultaneously.