1. Why Helpdesk RAG Breaks Without Article-Level Isolation
In an ITSM and helpdesk platform like Zammad, a single support ticket mixes two completely different trust zones: customer-facing replies (internal: false) and confidential internal agent triage notes, escalation runbooks, and diagnostic dumps (internal: true).
Flattening an entire ticket thread into one vector document—or indexing tickets without preserving Zammad's group, customer, and shared-organization boundaries—immediately leaks internal engineering notes to customers and exposes one customer's incidents to another organization. Even worse, Zammad Knowledge Base articles carry granular category role permissions and future-scheduled publication and archiving timestamps that naive scrapers routinely ignore.
Verified Against Real Zammad 7.2.3 + Elasticsearch 9.5.5: Every API call, incremental Elasticsearch delta query, Knowledge Base permission lookup, binary attachment stream, tombstone deletion, and 5-persona MCP Zero-Trust scenario is automatically verified in CI against official ghcr.io/zammad/zammad:7.2.3-0008 and elasticsearch:9.5.5 containers.
2. Architecture & Decoupled Ingestion Flow
ZammadRepositoryConnector connects to self-hosted or Zammad Cloud instances over REST API v1 (/api/v1/...) using personal HTTP access tokens (Authorization: Token token=…), OAuth2 Bearer tokens, or Basic authentication, with optional X-On-Behalf-Of user delegation. Tickets are processed concurrently on Java 25 Virtual Threads via StructuredTaskScope and bounded semaphores, streaming binary attachments through size-capped temporary files (TempFileInputStream) into OpenCrawling's Claim-Check Store and Kafka pipeline.
Tickets, Notes, Files & KB
Virtual Threads & State
Public / Internal Split ACL
Claim-Check & Chunking
Zero-Trust Retrieval
3. Strict Public vs. Internal Ticket & Attachment Partitioning
Instead of merging public replies and private agent notes into a single document, the connector partitions every ticket according to article-granularity:
SPLIT_PUBLIC_AND_INTERNAL(default): Emits one OIS HTML document for customer-visible articles (zammad://<host>/tickets/<id>/public) and a separate OIS document for internal agent notes (zammad://<host>/tickets/<id>/internal). If a ticket contains only internal notes, no empty/publicdocument is emitted—preventing ticket titles, tags, and owner metadata from leaking to customers.PER_ARTICLE: Emits one OIS document per article (zammad://<host>/tickets/<id>/articles/<articleId>) for fine-grained citation and attribution.PUBLIC_ONLY: Ingests only customer-visible articles and their attachments.- Binary Attachment Streaming & Deduplication: Streams real attachments via
GET /api/v1/ticket_attachment/{ticket}/{article}/{att}(default cap 50 MiB), skips Zammad's syntheticcontent-alternative: trueemail HTML duplicates, and strictly inherits the parent article's public or internal ACL.
4. Fail-Closed Zero-Trust ACL Mapping
ZammadSecurityMapper translates Zammad's native authorization model into OIS SecurityConfig rules enforced authoritatively by the Secure MCP Server (DocumentAccessGate / McpAclEvaluator):
| Zammad Entity | OIS Security Principals | Guarantee |
|---|---|---|
| Public Ticket Thread & Public Attachments | group:zammad-group-<group>, user:<customerEmail>, and group:zammad-org-<org> (only if organization.shared == true) |
Non-shared organizations (shared: false) never grant cross-colleague read access. |
| Internal Agent Notes & Internal Attachments | group:zammad-group-<group> only |
Customer and organization principals—plus zammad.customer and zammad.organization metadata—are stripped. |
| Knowledge Base Answers | group:zammad-role-<role> from Zammad's computed permissions_effective (or public only when published & allow-public-kb=true) |
Categories whose explicit granular permissions deny all roles (access: "none") fail closed instead of falling back to all agent groups. |
| Unresolvable / Orphan Group | Empty SecurityConfig (permissions: []) |
Strict fail-closed behavior—never defaults to public. |
5. Hardened Against Real Zammad 7.2 API Behavior
Running our connector against a live Zammad 7.2.3-0008 + Elasticsearch 9.5.5 deployment surfaced several real-world API details that we hardened directly into the v1 release:
- Knowledge Base REST Contract: Loads KB structure via
POST /api/v1/knowledge_bases/init(with automaticGETfallback on404/405), fetches translation bodies with explicit comma-joined?include_contents=<content_ids>, resolvesKnowledgeBaseLocale.system_locale_idagainstLocaleassets, and reads authoritativepermissions_effective. - Scheduled Publishing (
CanBePublished): Evaluatespublished_at,internal_at, andarchived_atagainst Zammad's server clock—so drafts scheduled for next week are never prematurely indexed, and active answers scheduled for future archival remain searchable until their archive timestamp arrives. - Server-Clock Watermarks & Elasticsearch Range Syntax: Incremental
SEARCH_DELTAqueries use Lucene range syntaxupdated_at:[<watermark> TO *]anchored to Zammad's HTTPDateresponse header (minus a 60-second overlap) to resist host-vs-server clock skew. - Zammad Pagination Caps & Lifecycle Hygiene: Respects Zammad's hard endpoint limits (
max 100on/api/v1/tickets,max 200on/api/v1/tickets/search), paginates metadata lookups (/groups,/roles,/ticket_states,/ticket_priorities), stops ticket paging only on empty or duplicate pages (preventing silent truncation or falseFULLtombstones), closes ownedHttpClientpools ondisconnect(), refreshes user/organization caches on reconnect, and honors cooperativeFluxcancellation without advancing state. - Pre-Crawl Discovery & On-Demand Fetch: Implements
DiscoverableRepositoryConnector(explore,dryRun,crawlStart) andDocumentFetchingRepositoryConnector(fetchDocument) with page-by-pageVisibilityFilterenforcement for AI agents.
6. Quick Configuration Reference
application.yml configuration for oc-zammad-repository-connectorspring:
opencrawling:
connector:
type: zammad
zammad:
base-url: "https://support.acme.com"
auth-type: "token" # token | oauth2 | basic
access-token: "${ZAMMAD_ACCESS_TOKEN}"
group-names: "Tier-2-Identity-Security,Billing-Operations"
article-granularity: "SPLIT_PUBLIC_AND_INTERNAL"
include-internal-articles: true
include-attachments: true
include-knowledge-base: true
include-tags: true
crawl-mode: "search_delta" # search_delta | full
state-file: "/data/zammad-state"
max-attachment-size-bytes: 52428800
include-acls: true
include-customer-in-public-acl: true
include-shared-organization-in-public-acl: true
allow-public-kb: false
page-size: 50
parallelism: 4
7. Dual-Tier Automated Verification (Mock & Real Zammad 7.2)
PR #372 ships two dedicated test suites wired directly into GitHub Actions (.github/workflows/integration-tests.yml):
- Fast Standalone Suite (
./scripts/test-zammad-connector.sh): Launches a zero-dependency Python 3 Zammad REST API v1 emulator (oc-zammad-repository-connector/zammad-mock/zammad_mock.py), validates REST contracts, executes 21 connector unit, regression, and over-HTTP tests, and verifies 101oc-runtimewiring tests in under 30 seconds. - Real Zammad 7.2 Decoupled E2E Suite (
./scripts/test-zammad-decoupled.sh): Boots official Zammad7.2.3-0008+ Elasticsearch9.5.5alongside Kafka, Tika 4 ForkParser, Ollama (mxbai-embed-large), PostgreSQLpgvector, and the Secure MCP Server—seeding real tickets, attachments, and KB answers, validating Zero-Trust MCP queries across 5 personas, applying live REST mutations to verify incrementalsearch_deltaupserts and tombstones, and confirmingfullhard-delete reconciliation.
# 1. Fast standalone REST contract, emulator, unit & runtime wiring suite
./scripts/test-zammad-connector.sh
# 2. Full 9-step decoupled E2E suite against real Zammad 7.2.3 + Elasticsearch 9.5.5
./scripts/test-zammad-decoupled.sh --local
Ready to Connect Zammad to Your Zero-Trust Enterprise AI Pipeline?
Read the full Zammad Wiki documentation, try Zammad in our interactive pipeline simulator, or run the real Zammad 7.2 Compose stack locally.