Connector Release 9 min read

Zero-SDK Google Workspace RAG, Multi-Replica Changes API & Fail-Closed ACLs: Announcing the Google Drive Repository Connector

Google Drive and Shared Drives are where modern engineering specs, runbooks, and executive memos live—yet connecting them safely to enterprise AI pipelines is notoriously fragile. Today we are introducing the native Google Drive Repository Connector (oc-googledrive-repository-connector, issue #339 / PR #436), built directly on Java 25 HttpClient and pure java.security RS256 JWT signing with zero external Google SDK bloat, structure-preserving HTML exports, multi-replica ControlPlaneStore state, explicit TOO_LARGE reporting, and fail-closed Zero-Trust ACLs.


1. Why Naive Google Drive Loaders Break in Production

Almost every RAG tutorial starts with a twenty-line Google Drive loader. Put that loader into a regulated Google Workspace organization and five structural cracks open on day one:

Least-Privilege First — No Domain-Wide Delegation Required: Share a folder or Shared Drive directly with your Google Cloud Service Account email (Viewer / drive.readonly) and start crawling immediately. Domain-Wide Delegation (delegated-user / JWT sub claim) remains strictly optional, and serviceAccountJson is automatically masked as ******** across the Platform API and Admin UI.

2. Zero-SDK Java 25 Architecture & Decoupled Flow

Rather than pulling dozens of transitive Google Java SDK and Guava/Protobuf conflicts into the runtime classpath, GoogleDriveClient talks directly to the official Google Drive REST API v3 over Java 25's native java.net.http.HttpClient. Service Account JWT assertions are signed in pure JDK java.security (PKCS#8 RSA SHA256withRSA / RS256), cached in memory with proactive renewal, retried once on HTTP 401, and protected by jittered exponential backoff + Retry-After handling on HTTP 429, Drive 403 rate-limit responses, and 5xx errors.

Google Drive API v3 → Zero-Trust Multi-Replica Decoupled Pipeline
Google Drive & Shared Drives
RS256 JWT (drive.readonly)
GoogleDriveRepositoryConnector
Changes API, Tree & Discovery
ControlPlaneStore (JDBC)
Shared startPageToken
Kafka + Tika 4 + Embed
HTML Export & TOO_LARGE Ledger
Vector Store + MCP
Zero-Trust ACL Retrieval

Zero External Google SDKs

Native Java 25 HttpClient + pure JDK PKCS#8 RS256 JWT signing, OAuth 2.0 refresh tokens, and static Bearer tokens for local emulators.

Structure-Preserving Exports

Google Docs export as text/html (keeping headings intact for structure-aware chunking), Slides as text/plain, and Sheets & Forms are excluded by default.

Explicit TOO_LARGE Auditing

Files over max-file-size-bytes (50 MB) or Drive's 10 MB /export cap are logged as SKIPPED (errorCode="TOO_LARGE") in the job audit ledger.

3. Rich Metadata, Constant-Heap Streaming & TOO_LARGE Visibility

Every indexed file receives a canonical internal URI (googledrive://drives/<driveId>/files/<fileId>) while preserving human-facing context for search results and AI citations:

4. Multi-Replica Changes API, Cascading Tombstones & Empty-Listing Guard

In crawl-mode: changes (the default), the connector anchors incremental sync to Drive's /changes/startPageToken and /changes endpoints. Instead of relying on local container files, ControlPlaneGoogleDriveStateStore persists the page token, lastFullScanAt, folder hierarchy map, and file versionKey + SHA-256 permissionsFingerprint snapshots inside OpenCrawling's shared ControlPlaneStore (PostgreSQL oc_setting in JDBC mode). Any runtime replica can pick up the next scheduled run without losing cursor position.

5. Fail-Closed Zero-Trust ACL Mapping & Pre-Crawl Discovery

GoogleDriveSecurityMapper converts GET /drive/v3/files/{fileId}/permissions into OIS SecurityConfig rules enforced end-to-end by OpenCrawling's Secure MCP Server (DocumentAccessGate / McpAclEvaluator). Deleted permissions and expired grants (expirationTime <= now) are stripped before mapping:

Mapping between Google Drive v3 permission types and Open Ingestion Standard (OIS) security principals
Drive Permission Type OIS Security Principal Zero-Trust Guarantee
type: "user" identityType: "user", lowercased emailAddress owner, organizer, fileOrganizer, and writer map to write; commenter and reader map to read.
type: "group" identityType: "group", lowercased group email Directly matches Google Group claims supplied in MCP userRoles.
type: "domain" identityType: "group", domain:<domain> Prefix customizable via domain-identity-prefix (default domain:).
type: "anyone" Ignored by default (public only if allow-anyone-public=true) Strict opt-in prevents accidental exposure of "anyone with the link" drafts.
Missing / Unreadable ACL SecurityConfig.none() (permissions: []) Always fails closed—never defaults to public.

The connector also implements DiscoverableRepositoryConnector (explore, dryRun, crawlStart) and DocumentFetchingRepositoryConnector (fetchDocument), enforcing live Drive permissions page-by-page through VisibilityFilter so AI agents can inspect Shared Drives and trigger scoped dynamic crawls without probing hidden folders.

6. Quick Configuration Reference

Sample application.yml configuration for oc-googledrive-repository-connector
spring:
  opencrawling:
    repository-connector-type: googledrive
    connector:
      type: googledrive
      googledrive:
        auth-type: "service_account"             # service_account | oauth2 | bearer
        service-account-json: "${GOOGLEDRIVE_SERVICE_ACCOUNT_JSON}"
        delegated-user: ""                       # optional; leave blank unless using DWD
        drive-id: "0AO_SUPPORT"                  # Shared Drive ID (empty for My Drive)
        root-folder-ids: "folder-procedures"
        include-subfolders: true
        include-shared-drives: true
        crawl-mode: "changes"                    # changes | tree
        full-scan-interval: "P7D"
        full-scan-window: "01:00-05:00"
        full-scan-timezone: "UTC"
        export-doc-mime-type: "text/html"
        export-slides-mime-type: "text/plain"
        excluded-mime-types: "application/vnd.google-apps.spreadsheet,application/vnd.google-apps.form"
        max-file-size-bytes: 52428800            # 50 MB -> larger files reported as TOO_LARGE
        include-custom-properties: true
        include-acls: true
        allow-anyone-public: false
        parallelism: 4

7. Dual-Tier Automated Verification (Drive v3 Emulator & Decoupled Stack)

Because Google Workspace SaaS cannot run as a local container, PR #436 ships a zero-dependency Python 3 Google Drive REST API v3 & OAuth2 emulator (oc-googledrive-repository-connector/googledrive-mock/googledrive_mock.py) generating valid Tika-parseable PDFs, Docs HTML exports, Slides plain-text exports, excluded Sheets/Forms, a 100 MB oversized TOO_LARGE PDF, and live mutation endpoints powering two automated test suites:

Run the standalone and full decoupled Google Drive test suites locally
# 1. Fast standalone Drive API v3 contract, unit & multi-replica runtime wiring suite
./scripts/test-googledrive-connector.sh

# 2. Full 9-step decoupled pipeline test (Mock -> Kafka -> Tika -> Ollama -> pgvector -> MCP)
./scripts/test-googledrive-decoupled.sh --local

Ready to Connect Google Drive & Shared Drives to Zero-Trust Enterprise AI?

Explore the full Wiki specification, test Google Drive interactively in our browser pipeline simulator, or inspect PR #436 on GitHub.