Connector Release 10 min read

Zero-SDK Confluence Cloud & Data Center RAG, Structure-Aware Macros & Fail-Closed ACLs: Announcing the Confluence Repository Connector

Atlassian Confluence is the living institutional memory of modern engineering, security, and operations teams—spanning architecture decision records, incident playbooks, product specs, and attached PDFs. Today we are announcing OpenCrawling's native Atlassian Confluence Repository Connector (oc-confluence-repository-connector, issue #341 / PR #440), delivering unified Confluence Cloud REST API v2+v1 and Data Center REST API v1 ingestion with zero external Atlassian SDK dependencies, structure-aware Storage XHTML macro hygiene, coupled binary attachments with cross-origin auth stripping, multi-replica CQL delta sync, explicit TOO_LARGE reporting, and fail-closed Space + inherited page read restriction ACLs.


1. Why Naive Confluence Loaders Break in Enterprise RAG

Confluence looks deceptively simple from a REST quickstart—until you connect a production multi-space wiki to an enterprise AI assistant. Five failure modes hit almost every off-the-shelf loader:

Zero-Trust Security Built Into Every Layer: ConfluenceClient follows attachment redirects manually and strips the Authorization header whenever a redirect crosses origins (such as Confluence Cloud's signed CDN download URLs). Anonymous Space view access is ignored by default (allow-anonymous-public=false), and unreadable Space permissions or page restrictions always fail closed (SecurityConfig.none()).

2. Zero-SDK Dual-Mode Java 25 Architecture (Cloud v2+v1 & Data Center v1)

Built directly on Java 25's native java.net.http.HttpClient, Jackson 2, and jsoup with zero external Atlassian SDK dependencies, ConfluenceClient speaks both Confluence Cloud and Confluence Data Center / Server from a single connector module. Set deployment-type: auto to detect Cloud vs. Data Center automatically from cloud-id, .atlassian.net, or api.atlassian.com, or pin cloud / data_center explicitly.

Atlassian Confluence (Cloud v2+v1 & Data Center v1) → Zero-Trust Decoupled Pipeline
Confluence Cloud & DC
API Token / OAuth 2.0 / PAT
ConfluenceRepositoryConnector
Storage XHTML, CQL & Discovery
ControlPlaneStore (JDBC)
Shared CQL Watermark
Kafka + Tika 4 + Embed
HTML5, PDFs & TOO_LARGE Ledger
Vector Store + MCP
Space ∩ Ancestor ACLs

Four Enterprise Auth Modes

Cloud api_token (Basic), Cloud oauth2 3LO (cloud-id + proactive token refresh + 401 retry + clickable web-base-url), Data Center Bearer pat, and basic auth—all secrets masked in the API & Admin UI.

Structure-Aware Macro Hygiene

ConfluenceStorageConverter transforms body.storage XHTML into clean semantic HTML5—retaining callouts, expands, tables, task lists, and code blocks while stripping toc, pagetree, and jira noise.

Explicit TOO_LARGE Auditing

Binary attachments exceeding max-attachment-size-bytes (50 MB default) emit opencrawling.document.status=TOO_LARGE, recorded as SKIPPED in the execution ledger instead of failing the crawl.

Dual-mode REST API endpoint routing across Confluence Cloud (v2 + v1) and Data Center (v1)
Operation Confluence Cloud (v2 + v1) Confluence Data Center (v1)
Spaces & Permissions GET /api/v2/spaces & /spaces/{id}/permissions (+ cached /rest/api/group/by-id) GET /rest/api/space?expand=permissions (inline Space permissions)
Pages & Blog Posts GET /api/v2/spaces/{id}/pages & /blogposts (body-format=storage) GET /rest/api/space/{key}/content/page & /rest/api/content?type=blogpost
Labels & Properties GET /api/v2/pages/{id}/labels & /properties Inline metadata.labels.results & /rest/api/content/{id}/property
read Restrictions & CQL GET /rest/api/content/{id}/restriction/byOperation/read & /rest/api/content/search Same v1 restriction & CQL endpoints

3. Structure-Aware Storage Format Conversion & Label Scoping

Every page or blog post receives a deterministic canonical URI (confluence://<host>/spaces/<spaceKey>/pages/<contentId> or /blogposts/<contentId>) alongside clickable browser url / confluence.webUrl citations, ancestor breadcrumbs, custom properties (confluence_prop_<key>), and multi-value tags mapped from Confluence labels.

4. Coupled Binary Attachments, Cross-Origin Redirect Safety & TOO_LARGE Auditing

Confluence pages rarely live alone—engineering architecture specs, compliance PDFs, and Word runbooks sit right underneath as page attachments. When include-attachments: true is enabled, the connector models every eligible attachment as a coupled OIS document (.../pages/<parentPageId>/attachments/<attachmentId>):

5. Multi-Replica CQL Delta Sync, Periodic Reconciliation & Empty-Listing Guard

In crawl-mode: cql_delta (the default), ControlPlaneConfluenceStateStore persists a server-Date-anchored lastModifiedUtc watermark, lastFullScanAt, a render-config renderSignature (invalidating cached items if macro/body settings change), and per-item (version, labelFingerprint, aclFingerprint, parentId, attachmentIds) snapshots in OpenCrawling's shared JDBC ControlPlaneStore (PostgreSQL oc_setting). Any horizontally scaled runtime replica can pick up the next scheduled run seamlessly.

6. Fail-Closed Zero-Trust ACL Intersection & Pre-Crawl Discovery

ConfluenceSecurityMapper evaluates effective access by intersecting Space View Permissions with every restricted ancestor and direct page read restriction (deliberately ignoring update restrictions, which do not restrict viewing in Confluence). Every output rule is enforced end-to-end by OpenCrawling's Secure MCP Server (DocumentAccessGate / McpAclEvaluator):

Mapping between Atlassian Confluence Space/Page permissions and Open Ingestion Standard (OIS) security principals
Confluence Permission Source OIS Security Principal Zero-Trust Guarantee
Space View + Unrestricted Page Space read users & groups (user / group) Confluence Cloud group UUIDs are resolved & cached via /rest/api/group/by-id to match OIDC/MCP group names.
Ancestor & Page read Restrictions Intersection across every restricted ancestor, page, and Space View Supports optional expand-group-members: true for exact Space-group ∩ page-user evaluation; ignores update restrictions.
GDPR Cloud User Identity email_and_id (default), email, or id Emits Atlassian accountId / DC userKey cleanly even when Cloud GDPR privacy rules hide user email addresses.
Anonymous Space View Ignored by default (public only if allow-anonymous-public=true and page unrestricted) Strict opt-in prevents accidental leakage of publicly accessible Space portals.
Empty / Unreadable ACL SecurityConfig.none() (permissions: []) Always fails closed—never defaults to public.

The connector also implements DiscoverableRepositoryConnector (explore, dryRun, crawlStart) and DocumentFetchingRepositoryConnector (fetchDocument) for pages, blog posts, and attachments—applying live Space and page-restriction checks through VisibilityFilter so AI agents can browse Confluence Spaces and trigger scoped dynamic crawls without probing restricted trees.

7. Quick Configuration Reference

Sample application.yml configuration for oc-confluence-repository-connector
spring:
  opencrawling:
    repository-connector-type: confluence
    connector:
      type: confluence
      confluence:
        base-url: "https://acme.atlassian.net/wiki"
        deployment-type: "cloud"                 # cloud | data_center | auto
        auth-type: "api_token"                   # api_token | pat | oauth2 | basic
        username: "crawler-bot@acme.com"
        api-token: "${CONFLUENCE_API_TOKEN}"
        space-keys: "ENG,KB,SEC"
        include-personal-spaces: false
        included-labels: "kb"
        excluded-labels: "draft,archived"
        include-blog-posts: true
        include-attachments: true
        body-format: "storage"                   # storage | export_view | view
        max-attachment-size-bytes: 52428800      # 50 MB -> larger files reported as TOO_LARGE
        crawl-mode: "cql_delta"                  # cql_delta | tree
        cql-overlap-minutes: 5
        full-scan-interval: "P7D"
        full-scan-window: "02:00-05:00"
        full-scan-timezone: "UTC"
        include-acls: true
        user-identity: "email_and_id"
        allow-anonymous-public: false
        page-size: 50
        parallelism: 4

8. Dual-Tier Automated Verification (Cloud + DC Emulator & 9-Step Decoupled Stack)

To verify both Cloud REST API v2+v1 and Data Center REST API v1 deterministically in CI without external tenant dependencies, PR #440 ships a zero-dependency Python 3 Confluence Cloud & Data Center emulator (oc-confluence-repository-connector/confluence-mock/confluence_mock.py) serving Cloud /wiki/api/v2, Data Center /confluence/rest/api, OAuth 2.0 /oauth/token, cross-origin /cdn/ attachment redirects, Tika-parseable PDFs, an excluded PNG image, a 100 MB oversized TOO_LARGE PDF, a draft-labeled page, an anonymous PUB Space, and live mutation endpoints powering two automated test suites:

Run the standalone and full decoupled Confluence test suites locally
# 1. Fast standalone Cloud + Data Center contract, unit & multi-replica runtime wiring suite
./scripts/test-confluence-connector.sh

# 2. Full 9-step decoupled pipeline test (Mock -> Kafka -> Tika -> Ollama -> pgvector -> MCP)
./scripts/test-confluence-decoupled.sh --local

Ready to Connect Confluence Cloud & Data Center to Zero-Trust Enterprise AI?

Read the complete Wiki guide, simulate Confluence Storage XHTML and cascading tombstones live in your browser, or explore PR #440 on GitHub.