Scale-Out Ingestion: Welcoming Michael Cizmar as Lead Architect

The OpenCrawling initiative has grown significantly since its launch. What began as a zero-trust security mapping framework bridging legacy document repositories to vector search architectures has evolved into an active ecosystem. Today, we are taking a massive leap forward.

I am thrilled to announce that enterprise search veteran and big data systems architect Michael Cizmar has officially joined the OpenCrawling core steering team as Lead Architect.

Bridging Big Data & AI Vector Ingestion

Modern Enterprise AI and Retrieval-Augmented Generation (RAG) require processing millions of files while respecting strict Row-Level Security (RLS) and document Access Control Lists (ACLs). Doing this in a single monolithic service is a recipe for system bottlenecks.

Michael’s background is uniquely suited to address this scaling crisis. Over the past two decades, he has designed and optimized massive information retrieval and indexing clusters for Fortune 500 companies. His expertise in building robust, decoupled microservice systems will guide OpenCrawling’s transition from a single reference engine to a fully distributed, enterprise-grade ingestion standard.

"Preserving permissions at the document level during vectorization is the single most pressing challenge in Enterprise RAG. If you bypass ACL checks, you leak data. OpenCrawling is the first open-source project to solve this at the architecture layer. I'm excited to collaborate with Piergiorgio to scale this engine out globally."

— Michael Cizmar, Lead Architect

What Michael is Architecting Next

Michael has already begun contributions to our technical roadmap. In the coming weeks, his focus will center on three core areas:

• Distributed Embedding Clustering: Standardizing the new oc-embedding-service Kafka consumer group mappings, enabling users to spin up multiple parallel worker pods to process high-volume text embeddings.
• Multi-Tenant Vector Stores: Hardening security mappers for downstream database outputs (pgvector, Elasticsearch, Qdrant) so that metadata mappings are isolated securely.
• CMIS & Enterprise Connectors: Steering standard definitions inside the Open Ingestion Standard (OIS) to streamline connector integration with enterprise platforms like Alfresco and SharePoint.

A Note from Piergiorgio Lucidi

"Having worked on enterprise search systems for most of my career, I know that scaling ingestion pipelines requires a deep understanding of thread pools, event queues, and indexing limits. Michael's architectural insights will accelerate our core engine features, allowing users to deploy OpenCrawling at scale without compromising security or throughput. I am honored to welcome him to the steering team."

Join Us in Welcoming Michael!

We are building a collaborative, open ecosystem. You can check out Michael's updates on our new Team Page, connect with him directly on GitHub, or say hello in our newly launched Slack Community Server!