A data ingestion and security mapping framework is only as powerful as the ecosystem of repositories it can reach. For OpenCrawling to successfully bridge siloed enterprise content to modern RAG architectures, we need a robust set of pluggable connectors.

Today, we have established our core bootstrap architecture and launched our first reference connectors: the local Filesystem Repository Connector and the pgvector Output Connector. Now, we are looking to the community to prioritize what comes next.

We've opened a dedicated GitHub Roadmap Discussion to gather your votes and detailed requirements. Here is a breakdown of the connectors currently under review:

📥 1. Repository Connectors (Sources)

Enterprise data is scattered across legacy systems, cloud buckets, and relational stores. We are considering the following source connectors for our next development cycle:

  • Microsoft SharePoint (Online & On-Premises) Under Discussion

    Crucial for enterprise search, but presents complex Access Control List (ACL) inheritance structures. The connector must map Active Directory SIDs dynamically to maintain document permissions.

  • Amazon S3 & MinIO Planned

    Object storage crawlers that poll buckets for changes, handle metadata extraction from S3 user headers, and route objects via Kafka claim checks.

  • Atlassian Confluence (Cloud & Server) Under Discussion

    Crawls space hierarchies, pulls page contents, and maps space/page restrictions to ensure zero-trust search query boundaries.

  • Google Drive / Workspace Planned

    Crawls shared drives, pulls Google Docs/Sheets texts, and translates Google OAuth permissions to local search permission matrices.

📤 2. Vector Store Output Connectors

Once data is crawled, text-extracted, and embedded, it must be indexed. While pgvector is our launch database, we want to integrate native outputs for specialized high-scale vector stores:

  • Qdrant Output Connector Planned

    High-performance vector engine supporting hardware-accelerated search and payload pre-filtering. Perfect for enterprise RAG requiring sub-millisecond lookups.

  • Elasticsearch / OpenSearch Output Under Discussion

    Bridges hybrid search (combining sparse BM25 text queries with dense vector search) while leveraging native document-level security query filters.

  • Milvus Output Connector Planned

    Distributed vector database built for massive billion-scale vector workloads.

🛡️ How Security Dictates the Roadmap

At OpenCrawling, a connector is not just a text extractor. It must extract security permissions (ACLs) as first-class metadata. For example, when building the SharePoint connector, we cannot simply extract PDF texts; we must also fetch Active Directory User SIDs, Group SIDs, and inheritance states.

This is why we need your feedback. If you use a repository like Confluence or Google Drive, how do you manage security mappings inside your organization? Sharing your architecture in our discussion helps us design robust schemas that solve real-world problems.

💬 Join the Roadmap Discussion!

Your input directly influences what our core team works on next. We want to know:

  1. Which repository contains the bulk of your enterprise knowledge?
  2. What vector database is powering your search applications?
  3. Are you willing to test early release connector builds?

👉 Jump over to our GitHub Discussions Roadmap Thread, cast your votes, and let us know your requirements!