1. Why Naive Confluence Loaders Break in Enterprise RAG
Confluence looks deceptively simple from a REST quickstart—until you connect a production multi-space wiki to an enterprise AI assistant. Five failure modes hit almost every off-the-shelf loader:
- Storage Format Macro Pollution: Feeding raw Confluence
body.storageXML or HTML dumps directly into a chunker floods embeddings with navigation macros (toc,pagetree,children,jira,recently-updated) while mangling content-bearing macros (expand,info,warning,panel,code) and task lists. - Silent Label & Restriction Drift: Confluence does not bump
version.number—and often does not update CQLlastModified—when an author adds anarchivedlabel or tightens a page's view restrictions. Pure delta crawlers silently leave restricted or archived pages exposed in the vector index. - Orphaned Attachments & Credential Leaks: Trashing or archiving a parent page leaves its PDF attachments searchable forever unless tombstones cascade. Worse, blindly enabling HTTP redirect following leaks your Atlassian
Authorizationheader to third-party signed CDN hosts during Cloud attachment downloads. - The Cloud v2 vs. Data Center v1 Split: Organizations routinely operate Confluence Data Center on-premises alongside Confluence Cloud tenants—yet Cloud deprecated v1 page/space listing in favor of
/api/v2while keeping CQL search and page restrictions on/rest/api. - Broken Permission Inheritance: Confluence's effective read access is the intersection of Space View permissions and every restricted ancestor in the page tree (while ignoring
updaterestrictions). Flattening or skipping ancestor restrictions leaks confidential pages across teams.
Zero-Trust Security Built Into Every Layer: ConfluenceClient follows attachment redirects manually and strips the Authorization header whenever a redirect crosses origins (such as Confluence Cloud's signed CDN download URLs). Anonymous Space view access is ignored by default (allow-anonymous-public=false), and unreadable Space permissions or page restrictions always fail closed (SecurityConfig.none()).
2. Zero-SDK Dual-Mode Java 25 Architecture (Cloud v2+v1 & Data Center v1)
Built directly on Java 25's native java.net.http.HttpClient, Jackson 2, and jsoup with zero external Atlassian SDK dependencies, ConfluenceClient speaks both Confluence Cloud and Confluence Data Center / Server from a single connector module. Set deployment-type: auto to detect Cloud vs. Data Center automatically from cloud-id, .atlassian.net, or api.atlassian.com, or pin cloud / data_center explicitly.
API Token / OAuth 2.0 / PAT
Storage XHTML, CQL & Discovery
Shared CQL Watermark
HTML5, PDFs & TOO_LARGE Ledger
Space ∩ Ancestor ACLs
Four Enterprise Auth Modes
Cloud api_token (Basic), Cloud oauth2 3LO (cloud-id + proactive token refresh + 401 retry + clickable web-base-url), Data Center Bearer pat, and basic auth—all secrets masked in the API & Admin UI.
Structure-Aware Macro Hygiene
ConfluenceStorageConverter transforms body.storage XHTML into clean semantic HTML5—retaining callouts, expands, tables, task lists, and code blocks while stripping toc, pagetree, and jira noise.
Explicit TOO_LARGE Auditing
Binary attachments exceeding max-attachment-size-bytes (50 MB default) emit opencrawling.document.status=TOO_LARGE, recorded as SKIPPED in the execution ledger instead of failing the crawl.
| Operation | Confluence Cloud (v2 + v1) | Confluence Data Center (v1) |
|---|---|---|
| Spaces & Permissions | GET /api/v2/spaces & /spaces/{id}/permissions (+ cached /rest/api/group/by-id) |
GET /rest/api/space?expand=permissions (inline Space permissions) |
| Pages & Blog Posts | GET /api/v2/spaces/{id}/pages & /blogposts (body-format=storage) |
GET /rest/api/space/{key}/content/page & /rest/api/content?type=blogpost |
| Labels & Properties | GET /api/v2/pages/{id}/labels & /properties |
Inline metadata.labels.results & /rest/api/content/{id}/property |
read Restrictions & CQL |
GET /rest/api/content/{id}/restriction/byOperation/read & /rest/api/content/search |
Same v1 restriction & CQL endpoints |
3. Structure-Aware Storage Format Conversion & Label Scoping
Every page or blog post receives a deterministic canonical URI (confluence://<host>/spaces/<spaceKey>/pages/<contentId> or /blogposts/<contentId>) alongside clickable browser url / confluence.webUrl citations, ancestor breadcrumbs, custom properties (confluence_prop_<key>), and multi-value tags mapped from Confluence labels.
- XML-Exact
jsoupConversion (body-format: storage):ConfluenceStorageConverterparses Atlassian's namespaced XHTML viaParser.xmlParser()into clean HTML5, preserving<h1>–<h6>headings, lists, tables (includingcolspan/rowspan), blockquotes,ac:task-listcheckboxes rendered as[x]/[ ], andac:linktitles. - Customizable Keep vs. Drop Macro Lists: Content-bearing macros (
expand,panel,info,note,warning,tip,status,excerpt,section,column,layout,code,noformat) are unwrapped into clean semantic blocks, while boilerplate macros (toc,toc-zone,children,pagetree,jira,recently-updated,contributors,contentbylabel,livesearch) are dropped completely—both tunable viakeep-macrosanddrop-macros. - Space, Subtree & Label Filtering: Scope crawls by
space-keys(personal spaces~userexcluded by default), restrict to a page hierarchy withroot-page-id, toggle blog posts viainclude-blog-posts, and filter byincluded-labels/excluded-labels(where excludes likedraft,archivedalways win).
4. Coupled Binary Attachments, Cross-Origin Redirect Safety & TOO_LARGE Auditing
Confluence pages rarely live alone—engineering architecture specs, compliance PDFs, and Word runbooks sit right underneath as page attachments. When include-attachments: true is enabled, the connector models every eligible attachment as a coupled OIS document (.../pages/<parentPageId>/attachments/<attachmentId>):
- Parent Context & ACL Inheritance: Coupled attachments automatically inherit the owning page's
tags(labels), ancestorbreadcrumbs, and effective Space + inherited pagereadACLs. - Smart MIME Filtering & Constant-Heap Streaming: Noise binaries (
image/*,video/*,audio/*, ZIP/GZIP archives, andapplication/octet-stream) are skipped by default viaexcluded-attachment-mime-types. Eligible PDFs and Office documents stream straight to auto-deletingTempFileInputStreamfiles—and any redirect to an external Cloud CDN host automatically strips theAuthorizationheader. - Explicit
TOO_LARGELedger Entries: Attachments exceedingmax-attachment-size-bytes(50 MB default) produceopencrawling.document.status=TOO_LARGE, soJobOrchestratorrecords an auditableDocumentLedgerEntry.skipped(..., "TOO_LARGE", reason)and aWARNtrace event. - Cascading Attachment Tombstones: Trashing, deleting, moving out of scope, or excluding a page by label (for example adding
archived) immediately emitsDocumentAction.DELETEtombstones for both the page and every tracked attachment ID in its state record.
5. Multi-Replica CQL Delta Sync, Periodic Reconciliation & Empty-Listing Guard
In crawl-mode: cql_delta (the default), ControlPlaneConfluenceStateStore persists a server-Date-anchored lastModifiedUtc watermark, lastFullScanAt, a render-config renderSignature (invalidating cached items if macro/body settings change), and per-item (version, labelFingerprint, aclFingerprint, parentId, attachmentIds) snapshots in OpenCrawling's shared JDBC ControlPlaneStore (PostgreSQL oc_setting). Any horizontally scaled runtime replica can pick up the next scheduled run seamlessly.
- Three-Track CQL Delta Query: Each incremental run queries modified pages/blog posts, modified attachments (reconciling their parent page even when the page body wasn't touched), and
status = trashedcontent within a configurablecql-overlap-minuteslookback window. - Periodic Full Reconciliation for Label & ACL Drift: Setting
full-scan-interval(such asP7D) with optionalfull-scan-window(e.g.02:00-05:00) andfull-scan-timezoneperiodically performs a full Space enumeration that catches Confluence's version-less label and restriction updates. - Empty-Listing Guard: If an upstream outage, expired token, or transient permission revocation causes a full enumeration to return
0items while documents are already indexed, the empty-listing guard aborts tombstone emission rather than wiping the vector index.
6. Fail-Closed Zero-Trust ACL Intersection & Pre-Crawl Discovery
ConfluenceSecurityMapper evaluates effective access by intersecting Space View Permissions with every restricted ancestor and direct page read restriction (deliberately ignoring update restrictions, which do not restrict viewing in Confluence). Every output rule is enforced end-to-end by OpenCrawling's Secure MCP Server (DocumentAccessGate / McpAclEvaluator):
| Confluence Permission Source | OIS Security Principal | Zero-Trust Guarantee |
|---|---|---|
| Space View + Unrestricted Page | Space read users & groups (user / group) |
Confluence Cloud group UUIDs are resolved & cached via /rest/api/group/by-id to match OIDC/MCP group names. |
Ancestor & Page read Restrictions |
Intersection across every restricted ancestor, page, and Space View | Supports optional expand-group-members: true for exact Space-group ∩ page-user evaluation; ignores update restrictions. |
| GDPR Cloud User Identity | email_and_id (default), email, or id |
Emits Atlassian accountId / DC userKey cleanly even when Cloud GDPR privacy rules hide user email addresses. |
| Anonymous Space View | Ignored by default (public only if allow-anonymous-public=true and page unrestricted) |
Strict opt-in prevents accidental leakage of publicly accessible Space portals. |
| Empty / Unreadable ACL | SecurityConfig.none() (permissions: []) |
Always fails closed—never defaults to public. |
The connector also implements DiscoverableRepositoryConnector (explore, dryRun, crawlStart) and DocumentFetchingRepositoryConnector (fetchDocument) for pages, blog posts, and attachments—applying live Space and page-restriction checks through VisibilityFilter so AI agents can browse Confluence Spaces and trigger scoped dynamic crawls without probing restricted trees.
7. Quick Configuration Reference
application.yml configuration for oc-confluence-repository-connectorspring:
opencrawling:
repository-connector-type: confluence
connector:
type: confluence
confluence:
base-url: "https://acme.atlassian.net/wiki"
deployment-type: "cloud" # cloud | data_center | auto
auth-type: "api_token" # api_token | pat | oauth2 | basic
username: "crawler-bot@acme.com"
api-token: "${CONFLUENCE_API_TOKEN}"
space-keys: "ENG,KB,SEC"
include-personal-spaces: false
included-labels: "kb"
excluded-labels: "draft,archived"
include-blog-posts: true
include-attachments: true
body-format: "storage" # storage | export_view | view
max-attachment-size-bytes: 52428800 # 50 MB -> larger files reported as TOO_LARGE
crawl-mode: "cql_delta" # cql_delta | tree
cql-overlap-minutes: 5
full-scan-interval: "P7D"
full-scan-window: "02:00-05:00"
full-scan-timezone: "UTC"
include-acls: true
user-identity: "email_and_id"
allow-anonymous-public: false
page-size: 50
parallelism: 4
8. Dual-Tier Automated Verification (Cloud + DC Emulator & 9-Step Decoupled Stack)
To verify both Cloud REST API v2+v1 and Data Center REST API v1 deterministically in CI without external tenant dependencies, PR #440 ships a zero-dependency Python 3 Confluence Cloud & Data Center emulator (oc-confluence-repository-connector/confluence-mock/confluence_mock.py) serving Cloud /wiki/api/v2, Data Center /confluence/rest/api, OAuth 2.0 /oauth/token, cross-origin /cdn/ attachment redirects, Tika-parseable PDFs, an excluded PNG image, a 100 MB oversized TOO_LARGE PDF, a draft-labeled page, an anonymous PUB Space, and live mutation endpoints powering two automated test suites:
- Fast Standalone Suite (
./scripts/test-confluence-connector.sh): Wired into CI (.github/workflows/integration-tests.yml), probes HTTP 401 rejection, Cloud API Token Basic auth, Data Center PAT Bearer auth, and OAuth 2.0 token refresh against the mock, then executes all 19 connector tests and 158 runtime wiring tests—includingControlPlaneConfluenceStateStoreTestproving two independent JDBC replicas share CQL state andJobOrchestratorrecordsTOO_LARGEattachments asSKIPPED, plusPlatformApiV1UnitTestconfirmingapiToken/personalAccessTokenmasking. - 9-Step Decoupled E2E Suite (
./scripts/test-confluence-decoupled.sh): Boots the full Docker Compose stack (confluence-mock, PostgreSQL 17 +pgvector+ JDBC Control Plane, Redis, Ollamamxbai-embed-large, Kafka, Tika 4 ForkParser, and Secure MCP Server), verifying keptinfo/expandmacros and droppedtoc/jiramacros, coupled Tika PDF attachment ingestion with parent label inheritance,draft&image/pngexclusion,1 too_largeledger reporting, fail-closed anonymousPUBSpace, 3-persona MCP Zero-Trust enforcement, live CQL delta mutations + cascading attachment tombstones on page trash, persistedlastModifiedUtcwatermark inoc_setting, and restriction-only reconciliation viacrawl-mode: tree.
# 1. Fast standalone Cloud + Data Center contract, unit & multi-replica runtime wiring suite
./scripts/test-confluence-connector.sh
# 2. Full 9-step decoupled pipeline test (Mock -> Kafka -> Tika -> Ollama -> pgvector -> MCP)
./scripts/test-confluence-decoupled.sh --local
Ready to Connect Confluence Cloud & Data Center to Zero-Trust Enterprise AI?
Read the complete Wiki guide, simulate Confluence Storage XHTML and cascading tombstones live in your browser, or explore PR #440 on GitHub.