1. Why Naive Google Drive Loaders Break in Production
Almost every RAG tutorial starts with a twenty-line Google Drive loader. Put that loader into a regulated Google Workspace organization and five structural cracks open on day one:
- Over-privileged Domain-Wide Delegation: Many connectors force Google Workspace admins to grant tenant-wide Domain-Wide Delegation before a single folder can be crawled—creating an immediate InfoSec blocker when teams only want to index one Shared Drive or two procedure folders.
- Flat Text Exports Destroy Structure: Exporting Google Docs as
text/plainthrows away<h1>–<h6>heading boundaries, breaking structure-aware chunking and degrading citation relevance, while blindly crawling Google Sheets and Forms floods vector stores with noisy cell grids. - Silent Oversized File Drops: Google Drive caps native
/exportresponses at 10 MB (exportSizeLimitExceeded), and enterprises regularly store 100 MB+ PDFs. Silently skipping those files leaves operators guessing why critical manuals never showed up in search. - Replica State Drift: Storing the Changes API
startPageTokenon local container disk breaks the moment OpenCrawling runs behind a cluster-safe scheduler across multiple replicas. - Permission Leakage: Public-link files (
type=anyone), expired sharing grants, or unreadable permissions silently turning intopublicchunks undermine Zero-Trust retrieval.
Least-Privilege First — No Domain-Wide Delegation Required: Share a folder or Shared Drive directly with your Google Cloud Service Account email (Viewer / drive.readonly) and start crawling immediately. Domain-Wide Delegation (delegated-user / JWT sub claim) remains strictly optional, and serviceAccountJson is automatically masked as ******** across the Platform API and Admin UI.
2. Zero-SDK Java 25 Architecture & Decoupled Flow
Rather than pulling dozens of transitive Google Java SDK and Guava/Protobuf conflicts into the runtime classpath, GoogleDriveClient talks directly to the official Google Drive REST API v3 over Java 25's native java.net.http.HttpClient. Service Account JWT assertions are signed in pure JDK java.security (PKCS#8 RSA SHA256withRSA / RS256), cached in memory with proactive renewal, retried once on HTTP 401, and protected by jittered exponential backoff + Retry-After handling on HTTP 429, Drive 403 rate-limit responses, and 5xx errors.
RS256 JWT (drive.readonly)
Changes API, Tree & Discovery
Shared startPageToken
HTML Export & TOO_LARGE Ledger
Zero-Trust ACL Retrieval
Zero External Google SDKs
Native Java 25 HttpClient + pure JDK PKCS#8 RS256 JWT signing, OAuth 2.0 refresh tokens, and static Bearer tokens for local emulators.
Structure-Preserving Exports
Google Docs export as text/html (keeping headings intact for structure-aware chunking), Slides as text/plain, and Sheets & Forms are excluded by default.
Explicit TOO_LARGE Auditing
Files over max-file-size-bytes (50 MB) or Drive's 10 MB /export cap are logged as SKIPPED (errorCode="TOO_LARGE") in the job audit ledger.
3. Rich Metadata, Constant-Heap Streaming & TOO_LARGE Visibility
Every indexed file receives a canonical internal URI (googledrive://drives/<driveId>/files/<fileId>) while preserving human-facing context for search results and AI citations:
- Clickable Browser Citations: Maps Drive's
webViewLinktourl,webViewLink, andgoogledrive.webViewLink, and preserves the original Drive file name astitlewhile appending export extensions (.html,.txt) onname/filenamefor downstream Apache Tika detection. - Folder Breadcrumbs & Custom Properties: Resolves ancestor chains (including mapping the My Drive
"root"alias viaGET /files/root) intobreadcrumbsandpath, and emits DrivepropertiesandappPropertiesasgoogledrive_prop_<key>. - Composite
versionKeySkipping: Combines Drive'sversion,headRevisionId, andmd5Checksumintogoogledrive.versionKeyso unchanged files are skipped before opening a content download stream. - Constant-Heap Spooling &
TOO_LARGELedger Entries: Both?alt=mediabinaries and/exportstreams write straight to auto-deleting temporary files (TempFileInputStream). Whenever a file breachesmax-file-size-bytesor triggers Drive'sexportSizeLimitExceedederror,GoogleDriveDocumentBuilder.buildTooLarge(...)flagsopencrawling.document.status=TOO_LARGE, andJobOrchestratorrecords an explicitDocumentLedgerEntry.skipped(..., "TOO_LARGE", reason)plus aWARNtrace event.
4. Multi-Replica Changes API, Cascading Tombstones & Empty-Listing Guard
In crawl-mode: changes (the default), the connector anchors incremental sync to Drive's /changes/startPageToken and /changes endpoints. Instead of relying on local container files, ControlPlaneGoogleDriveStateStore persists the page token, lastFullScanAt, folder hierarchy map, and file versionKey + SHA-256 permissionsFingerprint snapshots inside OpenCrawling's shared ControlPlaneStore (PostgreSQL oc_setting in JDBC mode). Any runtime replica can pick up the next scheduled run without losing cursor position.
- Cascading Folder Tombstones: Trashing, deleting, or moving a folder outside the configured
root-folder-idsscope automatically emitsDocumentAction.DELETEtombstones for every known descendant file. - Automatic Recovery on Expired Tokens: HTTP
400or410invalid page tokens cleanly trigger a full reconciliation pass and acquire a freshstartPageToken. - Periodic Full Reconciliation + Empty-Listing Guard: Configuring
full-scan-interval(e.g.P7D) with optionalfull-scan-windowandfull-scan-timezoneperiodically reconciles silent permission-only updates—and if an outage or revoked grant causes a full enumeration to return0files while items are indexed, the empty-listing guard refuses to emit mass tombstones.
5. Fail-Closed Zero-Trust ACL Mapping & Pre-Crawl Discovery
GoogleDriveSecurityMapper converts GET /drive/v3/files/{fileId}/permissions into OIS SecurityConfig rules enforced end-to-end by OpenCrawling's Secure MCP Server (DocumentAccessGate / McpAclEvaluator). Deleted permissions and expired grants (expirationTime <= now) are stripped before mapping:
| Drive Permission Type | OIS Security Principal | Zero-Trust Guarantee |
|---|---|---|
type: "user" |
identityType: "user", lowercased emailAddress |
owner, organizer, fileOrganizer, and writer map to write; commenter and reader map to read. |
type: "group" |
identityType: "group", lowercased group email |
Directly matches Google Group claims supplied in MCP userRoles. |
type: "domain" |
identityType: "group", domain:<domain> |
Prefix customizable via domain-identity-prefix (default domain:). |
type: "anyone" |
Ignored by default (public only if allow-anyone-public=true) |
Strict opt-in prevents accidental exposure of "anyone with the link" drafts. |
| Missing / Unreadable ACL | SecurityConfig.none() (permissions: []) |
Always fails closed—never defaults to public. |
The connector also implements DiscoverableRepositoryConnector (explore, dryRun, crawlStart) and DocumentFetchingRepositoryConnector (fetchDocument), enforcing live Drive permissions page-by-page through VisibilityFilter so AI agents can inspect Shared Drives and trigger scoped dynamic crawls without probing hidden folders.
6. Quick Configuration Reference
application.yml configuration for oc-googledrive-repository-connectorspring:
opencrawling:
repository-connector-type: googledrive
connector:
type: googledrive
googledrive:
auth-type: "service_account" # service_account | oauth2 | bearer
service-account-json: "${GOOGLEDRIVE_SERVICE_ACCOUNT_JSON}"
delegated-user: "" # optional; leave blank unless using DWD
drive-id: "0AO_SUPPORT" # Shared Drive ID (empty for My Drive)
root-folder-ids: "folder-procedures"
include-subfolders: true
include-shared-drives: true
crawl-mode: "changes" # changes | tree
full-scan-interval: "P7D"
full-scan-window: "01:00-05:00"
full-scan-timezone: "UTC"
export-doc-mime-type: "text/html"
export-slides-mime-type: "text/plain"
excluded-mime-types: "application/vnd.google-apps.spreadsheet,application/vnd.google-apps.form"
max-file-size-bytes: 52428800 # 50 MB -> larger files reported as TOO_LARGE
include-custom-properties: true
include-acls: true
allow-anyone-public: false
parallelism: 4
7. Dual-Tier Automated Verification (Drive v3 Emulator & Decoupled Stack)
Because Google Workspace SaaS cannot run as a local container, PR #436 ships a zero-dependency Python 3 Google Drive REST API v3 & OAuth2 emulator (oc-googledrive-repository-connector/googledrive-mock/googledrive_mock.py) generating valid Tika-parseable PDFs, Docs HTML exports, Slides plain-text exports, excluded Sheets/Forms, a 100 MB oversized TOO_LARGE PDF, and live mutation endpoints powering two automated test suites:
- Fast Standalone Suite (
./scripts/test-googledrive-connector.sh): Wired into CI (.github/workflows/integration-tests.yml), runs contract smoke checks against the mock, executes all 21 connector tests, and runs 128 runtime wiring tests—includingControlPlaneGoogleDriveStateStoreTestproving two independent JDBC replicas share Changes API state andJobOrchestratorrecordsTOO_LARGEasSKIPPED, plusPlatformApiV1UnitTestconfirmingserviceAccountJsonmasking. - 9-Step Decoupled E2E Suite (
./scripts/test-googledrive-decoupled.sh): Boots the full Docker Compose stack (googledrive-mock, PostgreSQL 17 +pgvector+ JDBC Control Plane, Redis, Ollamamxbai-embed-large, Kafka, Tika 4 ForkParser, and Secure MCP Server), validating indexed content, default Sheets/Forms exclusion,1 too_largereporting, 5-persona MCP Zero-Trust enforcement, live Changes API mutations + cascading folder trash tombstones, persistedstartPageTokenadvancement inoc_setting, and permission-only reconciliation viacrawl-mode: tree.
# 1. Fast standalone Drive API v3 contract, unit & multi-replica runtime wiring suite
./scripts/test-googledrive-connector.sh
# 2. Full 9-step decoupled pipeline test (Mock -> Kafka -> Tika -> Ollama -> pgvector -> MCP)
./scripts/test-googledrive-decoupled.sh --local
Ready to Connect Google Drive & Shared Drives to Zero-Trust Enterprise AI?
Explore the full Wiki specification, test Google Drive interactively in our browser pipeline simulator, or inspect PR #436 on GitHub.