Microsoft Graph v1.0 & Zero-Trust Enterprise RAG: Announcing the SharePoint Online Repository Connector

1. Why SharePoint Online Needs a Fail-Closed, Graph-Native RAG Connector

Connecting Microsoft SharePoint Online to an enterprise Retrieval-Augmented Generation (RAG) pipeline requires far more than listing files in a drive. Production SharePoint tenants expose subtle security, throttling, and synchronization challenges that naive scripts fail to handle safely:

• Complex Multi-Facet Permissions: A single SharePoint file can be shared simultaneously with individual Microsoft Entra users (exposed via user.email or claims like i:0#.f|membership|alice@contoso.com), Entra security and M365 groups (c:0t.c|tenant|<guid>), site-local SharePoint groups (siteGroup, e.g., Engineering Members), specific-people sharing links, organization-wide links, and anonymous links.
• Display Name Spoofing & Broad-Grant Over-Exposure: Treating human display names or unmapped organization/anonymous sharing links as principals can leak confidential files across the entire tenant. Every principal must be anchored to an immutable identity (UPN/email, Entra object GUID, or site-qualified group ID) and fail closed by default.
• Permission-Only Changes vs. Least-Privilege Scopes: Microsoft Graph's driveItem/delta feed only reports permission-only changes when the caller holds Sites.FullControl.All. Enterprises running with least-privilege Sites.Selected or Sites.Read.All need a way to detect and reconcile ACL changes without re-downloading unchanged file content.
• Tenant-Wide Graph Throttling & Token Leakage: Flooding Microsoft Graph from concurrent worker threads without a shared Retry-After gate triggers cascading 429 Too Many Requests throttling, while forwarding Graph bearer tokens to pre-authenticated CDN download URLs violates security best practices.

The new oc-sharepoint-repository-connector was engineered on Java 25 and the OpenCrawling SPI to address every one of these requirements—backed by a 6-tier verification suite including official Microsoft Graph docs conformance, OpenAPI schema validation, Microsoft Dev Proxy chaos injection, and an independent 16-user RBAC oracle.

2. Entra ID App-Only Authentication & Site / Library Discovery

GraphClient authenticates against Microsoft Entra ID (https://login.microsoftonline.com/{tenantId}/oauth2/v2.0/token) using the OAuth 2.0 client_credentials grant for https://graph.microsoft.com/.default. It supports both tenant-wide Sites.Read.All and least-privilege Sites.Selected application permissions, caches the bearer token in memory, refreshes it proactively before expiry, and re-acquires it automatically once on HTTP 401.

Target sites can be specified either by their human-friendly SharePoint URL (site-url, e.g., https://contoso.sharepoint.com/sites/Engineering, resolved via GET /sites/{hostname}:/{server-relative-path}) or by their composite Graph Site ID (site-id). By default, the connector discovers and crawls every documentLibrary drive on the site, or filters to specific libraries via drive-ids or drive-names (e.g., Documents, Policies).

3. Two Complementary Crawl Modes: delta and tree

SharePointRepositoryConnector provides two crawl modes via crawl-mode, backed by persistent per-drive state in SharePointCrawlState (data/sharepoint-state/):

1. Incremental Delta Crawl (delta, default)

Walks GET /drives/{driveId}/root/delta across @odata.nextLink pages and persists the final @odata.deltaLink together with each file's eTag, path, and SHA-256 permission fingerprint:

• New or Modified Files: Emitted as OIS action: "UPSERT" documents (and skipped when both eTag and the permission fingerprint match the stored state).
• Cascading Folder & File Deletions: Items carrying a deleted facet or @removed emit OIS action: "DELETE" tombstones. When a folder is deleted, the connector cascades DELETE tombstones to every file previously recorded under that folder path.
• Moves & Renames Without Redundant Downloads: When a file or parent folder is moved or renamed without content or ACL changes, tracked paths are updated without re-downloading binaries.
• 410 Gone Expired Delta Token Recovery: If a delta token expires, the connector follows the Location header (or restarts from /root/delta), reconciles all live items against stored state, emits UPSERT only for changed items, and emits DELETE tombstones for items that disappeared while the token was expired.

2. Folder Tree Traversal & Permission-Only Reconciliation (tree)

Traverses GET /drives/{driveId}/root/children (or /root:/{path}:/children when root-folder-path is set), recursing into subfolders when recursive: true. When combined with incremental-enabled: true, tree mode compares each file's current eTag and SHA-256 permission fingerprint against stored state—re-emitting files whose permissions changed even when their eTag did not, and emitting DELETE tombstones for vanished files.

4. SharePoint List Item Columns & Secure Pre-Signed Downloads

When include-custom-fields: true (default) is enabled, SharePointDocumentBuilder fetches GET /drives/{driveId}/items/{itemId}/listItem?$expand=fields and maps custom SharePoint columns into OIS metadata under sharepoint_field_<InternalName> (configurable via field-prefix). Complex person, lookup, and managed-metadata fields are automatically reduced to their clean display labels, while internal SharePoint system properties (@odata.*, _UIVersionString, DocIcon, Edit) are stripped.

For binary content (include-content: true), GraphClient streams bytes from the pre-authenticated @microsoft.graph.downloadUrl without forwarding the Microsoft Graph Authorization: Bearer header (falling back to GET /drives/{driveId}/items/{itemId}/content 302 redirect if the pre-signed URL has expired). Content is spooled to a size-capped temporary file (TempFileInputStream, bounded by max-file-size-bytes, default 50 MB) that deletes itself on stream close, and handed to the OpenCrawling Claim-Check store (Apache Ozone / S3) and Apache Tika 4.

5. Fail-Closed Zero-Trust Security Mapping (SharePointSecurityMapper)

For every crawled file (include-acls: true), SharePointSecurityMapper evaluates GET /drives/{driveId}/items/{itemId}/permissions on a strict fail-closed basis. Only roles that grant content read access (read, write, owner, sp.full control, sp.contribute, sp.read) are accepted; metadata-only or custom roles such as sp.restricted view are ignored:

6. Shared Throttling Gate & Java 25 Structured Concurrency

Drive items are enriched concurrently using Java 25 StructuredTaskScope.open() and ObservabilityTask.observed(...), bounded by a configurable Semaphore (parallelism, default 4). Every request carries Microsoft's recommended ISV User-Agent header (ISV|OpenCrawling|OpenCrawling-SharePoint-Connector/1.0) and a per-request client-request-id UUID.

Crucially, all concurrent Virtual Thread workers share a single coordinated throttle gate inside GraphClient: when any request receives HTTP 429 Too Many Requests, 503 Service Unavailable, or 504 Gateway Timeout, all workers pause until the server's Retry-After window (with a 1-second floor) has elapsed before retrying with exponential backoff and jitter. Server errors (500, 502, 507) fail the item so the @odata.deltaLink never advances past an unindexed file.

7. Sample OIS Document Payload

{
  "id": "sharepoint://contoso.sharepoint.com,11111111-2222-3333-4444-555555555555,66666666-7777-8888-9999-000000000000/drives/b!drive123/items/01ABCDEF23456789",
  "action": "UPSERT",
  "acl": "alice@contoso.com,aaaa1111-bbbb-2222-cccc-333333333333,dddd4444-eeee-5555-ffff-666666666666,sharepoint-group:contoso.sharepoint.com,11111111-2222-3333-4444-555555555555,66666666-7777-8888-9999-000000000000:7",
  "metadata": {
    "name": ["Architecture-Decision-Record-012.pdf"],
    "title": ["ADR 012 - Event Driven Ingestion"],
    "mimeType": ["application/pdf"],
    "sharepoint.siteId": ["contoso.sharepoint.com,11111111-2222-3333-4444-555555555555,66666666-7777-8888-9999-000000000000"],
    "sharepoint.siteName": ["Engineering"],
    "sharepoint.driveId": ["b!drive123"],
    "sharepoint.driveName": ["Documents"],
    "sharepoint.itemId": ["01ABCDEF23456789"],
    "sharepoint.path": ["/Architecture/Architecture-Decision-Record-012.pdf"],
    "sharepoint.eTag": ["\"{D9B3F1E0-1234-4BCD-8EF0-112233445566},3\""],
    "sharepoint.quickXorHash": ["aGVsbG8gd29ybGQ="],
    "sharepoint_identity_users": ["alice@contoso.com", "aaaa1111-bbbb-2222-cccc-333333333333"],
    "sharepoint_identity_groups": [
      "dddd4444-eeee-5555-ffff-666666666666",
      "sharepoint-group:contoso.sharepoint.com,11111111-2222-3333-4444-555555555555,66666666-7777-8888-9999-000000000000:7"
    ],
    "sharepoint_field_Title": ["ADR 012 - Event Driven Ingestion"],
    "sharepoint_field_Department": ["Platform Engineering"],
    "sharepoint_field_Confidentiality": ["Internal"],
    "sharepoint_field_ReviewStatus": ["Approved"]
  },
  "security": {
    "inheritanceEnabled": true,
    "permissions": [
      { "identity": "alice@contoso.com", "identityType": "user", "access": "read" },
      { "identity": "aaaa1111-bbbb-2222-cccc-333333333333", "identityType": "user", "access": "read" },
      { "identity": "dddd4444-eeee-5555-ffff-666666666666", "identityType": "group", "access": "read" },
      { "identity": "sharepoint-group:contoso.sharepoint.com,11111111-2222-3333-4444-555555555555,66666666-7777-8888-9999-000000000000:7", "identityType": "group", "access": "read" }
    ]
  }
}

8. 6-Tier Verification: OpenAPI Conformance, Dev Proxy Chaos & 16-User RBAC Oracle

To verify production fidelity without requiring a live Microsoft 365 tenant in CI, the connector ships with a 6-tier verification suite:

# 1. Unit, MockWebServer & verbatim Microsoft Graph documentation examples
$ mvn test -pl oc-sharepoint-repository-connector -am

# 2. Stateful Graph v1.0 Mock + Microsoft OpenAPI schema & docs conformance
$ ./scripts/test-sharepoint-repository-connector.sh

# 3. Microsoft Dev Proxy chaos & Retry-After throttling test (50% error rate)
$ ./scripts/test-sharepoint-devproxy.sh

# 4. Independent mock-graph-server + 16-user RBAC oracle comparison (144 access checks)
$ ./scripts/test-sharepoint-independent-mock.sh

# 5. Full decoupled Docker Compose pipeline (Kafka + Tika + Ollama + pgvector + Secure MCP)
$ ./scripts/test-sharepoint-decoupled.sh