As OpenCrawling matures into a production-grade, Zero-Trust platform for enterprise AI data ingestion, every subtle invariant under load matters: per-document message ordering across partitioned Kafka topics, deterministic test execution in CI, and zero-drop failure recovery across downstream vector stores.
Today, we are proud to welcome Rafa Haro (@rafaharo) to the OpenCrawling core team as Lead Architect!
Search Pedigree, Apache Open Source & Pipeline Resilience
Rafa brings 20+ years of experience across distributed systems, search & information retrieval, and applied AI—combining his leadership as Principal Software Engineer at Elastic and his Apache Software Foundation stewardship as a PMC Member for Apache ManifoldCF and Apache Stanbol.
Search Engineering & Apache Open-Source Heritage
Few engineers bring as direct a lineage to the exact problems OpenCrawling solves as Rafa Haro:
- Principal Software Engineer at Elastic: Building large-scale distributed search, indexing, and retrieval infrastructure around Elasticsearch and Apache Lucene, alongside production-grade cloud platforms across AWS and GCP using Kubernetes, Terraform, and CDK.
- PMC Member for Apache ManifoldCF: Apache ManifoldCF provides an open-source framework for connecting source content repositories—such as Microsoft SharePoint and EMC Documentum—to target repositories or search indexes such as Apache Solr, OpenSearchServer, and Elasticsearch, while defining a security model that allows target repositories to enforce source-repository security policies. Having a ManifoldCF PMC veteran alongside fellow ManifoldCF PMC member Piergiorgio Lucidi strengthens OpenCrawling's Zero-Trust connector DNA—and naturally complements our dedicated Apache ManifoldCF Migrator.
- PMC Member & Committer for Apache Stanbol: Deep hands-on experience in semantic content enhancement, NLP pipelines, knowledge extraction, and ASF community-led architecture.
- 20+ Years Across Data Pipelines & Applied AI: Designing high-throughput data and event pipelines with Apache Spark, Apache Kafka, and Python alongside modern ML/NLP workflows spanning Transformers, Sentence Embeddings, and AI-driven automation.
First Contributions Already Landed in OpenCrawling
Rather than waiting to settle in, Rafa immediately dove into the core runtime and streaming pipeline—identifying subtle distributed systems edge cases and delivering clean, thoroughly tested engineering fixes:
| Pull Request | Status | Architectural Impact |
|---|---|---|
| PR #307 | Merged | Keyed opencrawling-chunks and opencrawling-embedded Kafka messages by documentId so chunks and DELETE tombstones never reorder across partitions. |
| PR #310 | Merged | Fixed Spring component scan leakage and gRPC push port collisions so the full oc-runtime unit and JUnit IT suite runs cleanly in one JVM. |
| PR #312 | In Review | Replaced silent exception swallowing with exponential back-off retries and <topic>.DLT dead-letter topics across ingestion, embedding, and store writers. |
1. Eliminating Out-of-Order Tombstone Races Across Kafka Partitions (PR #307)
In decoupled deployments, OpenCrawling routes extracted text and embeddings across partitioned Kafka topics (opencrawling-chunks and opencrawling-embedded). Previously, chunk UPSERT messages were keyed by chunkId while deletion tombstones (DocumentAction.DELETE) were keyed by documentId. Across 3 partitions, chunks and deletes for the same document could land on different partitions with no ordering guarantee between them—meaning a slow consumer or rapid recrawl/delete sequence could apply the DELETE first and re-index an earlier chunk afterward, resurrecting a deleted document with its old permissions.
Rafa introduced a canonical partitionKey() method on DocumentChunkMessage and DocumentEmbeddedMessage in oc-core (returning documentId with a safe chunkId fallback without altering JSON wire payloads) and updated IngestionConsumer, EmbeddingConsumer, and SeaTunnelOutputConnector to publish every chunk and DELETE under the document's key. Every operation for a document now stays strictly ordered on a single partition, verified by new unit tests (MessagePartitionKeyTest, IngestionConsumerTest, SeaTunnelOutputConnectorTest) and the decoupled end-to-end suites.
2. Restoring Single-JVM JUnit Integration Tests in oc-runtime (PR #310)
When running the full mvn verify -pl oc-runtime -DskipITs=false suite, Spring integration tests were failing to start due to two subtle test-infrastructure collisions: IdentityTestApplication and its nested Endpoints stand-in leaked into OpenCrawlingApplication's component scan (creating duplicate McpVectorSearchBackend beans and duplicate OIS ingest endpoint mappings), while GrpcPushServerManager kept fixed port 9096 bound across cached test contexts.
By annotating IdentityTestApplication and Endpoints with @TestComponent and configuring free ephemeral gRPC ports (opencrawling.connector.grpc-push.port=0 and spring.grpc.server.port=0) across OpenCrawlingIT, RecrawlAndDeleteIT, OisIngestIT, and IcebergConnectorIT, Rafa restored clean single-JVM execution across 707 unit tests, 10 HTTP identity tests, and all 7 runtime integration tests.
3. Zero-Drop Resilience: Back-Off Retries & Dead-Letter Topics (PR #312)
Building directly on per-document ordering, Rafa also opened PR #312 (currently in review) to ensure failed Kafka records are never silently dropped during transient model-server restarts or vector-store outages. The change pairs a unified DefaultErrorHandler with exponential back-off (configured via opencrawling.kafka.retry.* across oc-runtime and oc-embedding-service) and a DeadLetterPublishingRecoverer writing exhausted failures—as well as undeserializable payloads wrapped by ErrorHandlingDeserializer—to opencrawling-documents.DLT, opencrawling-chunks.DLT, and opencrawling-embedded.DLT.
Across the pipeline, IngestionConsumer, EmbeddingConsumer, and 8 vector and search output consumers (pgvector, Qdrant, Milvus, Vespa, Apache Solr, OpenSearch 2.x/3.x, and Luxir) now rethrow failures cleanly instead of logging and committing offsets, verified by KafkaConfigTest, per-consumer failure tests, and a real Kafka + pgvector DeadLetterIT integration test.
Looking Ahead Together
From distributed search architecture and Apache ManifoldCF security models to Applied AI pipelines and production cloud operations, Rafa's combination of deep engineering rigor and pragmatic open-source mentorship strengthens every layer of OpenCrawling and the Open Ingestion Standard (OIS). Having fellow Apache ManifoldCF PMC members collaborating alongside our enterprise search, ECM, and stream-processing architects sets an exciting foundation for what comes next.
Connect with Rafa Haro
Explore Rafa's profile on our Team Page, follow his contributions on GitHub (@rafaharo), connect with him on LinkedIn, or join the conversation in the OpenCrawling Slack Community.