1. Closing the Loop on SER Group Doxis 4 Integration
Enterprises consolidating legacy file shares, SharePoint sites, BPMN process attachments, or third-party repositories into a long-term regulated archive face a recurring problem: vector stores optimize retrieval, but regulated records still belong inside an enterprise Document Management System (DMS) with strict schema typing, dossier (Akte) hierarchy, immutable version trails, and auditable access control.
With oc-doxis-output-connector, OpenCrawling becomes an end-to-end ingestion and archival bridge into Doxis 4 through the Content Service Bus (CSB) REST API (/restws/publicws/rest/api/v1, verified against CSB 14.4.1). Built on the shared oc-doxis-client foundation, the connector preserves every bit of OIS governance while adding automatic read-back verification and atomic rollback so partial or unprotected records never linger in production DMS repositories.
Two-Plane Content Planning
ContentPlanner decides between multipart UPLOAD, zero-copy Blueline CONTENT_LINK, or REFERENCE_ONLY from metadata alone—wrapping Claim-Check streams in LazyInputStream so multi-terabyte files never load needlessly into memory.
Read-Back Verify & Rollback
With content.verify=true, every write checks stored byte length and SHA-256 via /versions?initializeRepresentations=true. If verification, e-file filing, or strict ACL assignment fails on a new document, OpenCrawling executes an immediate DELETE rollback.
Licensing & Session Capacity Notice: Archiving into Doxis requires a valid SER Group Doxis 4 licence on the target CSB instance that permits API / technical-user sessions. OpenCrawling does not bundle Doxis or proprietary SER client JARs. Plan concurrent job capacity with your Doxis administrator.
2. Dual Execution Paths & Multi-Gigabyte Extraction Bypass
Unlike vector output connectors that wait for Tika parsing and embeddings on opencrawling-embedded, the Doxis Output Connector operates directly on crawled document records across two deployment models:
- Per-Job / Direct Path (
DoxisOutputConnector): Registered asDoxis_Output(OutputKind.DOXIS) in the Admin UI, CLI, and REST API viaDoxisConnectorSettings.fromConfiguration(...).ConnectorFactorycreates an isolated connector instance per job run on a Java 25 Virtual Thread and callsconnectorFactory.release(...)in afinallyblock to executePOST /logoutimmediately when the job finishes—leaving no dangling technical-user sessions. - Decoupled Kafka Path (
DoxisStoreWriterConsumer): Activated withspring.opencrawling.output.type=doxisandopencrawling.consumer.writer.enabled=true. It consumesIngestionMessageevents directly fromopencrawling-documentsin an isolated consumer group (opencrawling-doxis-writer), whileDoxisConnectionInitializervalidates login, repository, document class, and external-ID attributes asynchronously at startup. - Large Binary Extraction Guard (
max-extract-bytes): When running decoupled RAG extraction alongside Doxis archival,IngestionConsumerenforcesopencrawling.ingestion.max-extract-bytes(default536870912bytes / 512 MiB). Oversized archives skip Tika text extraction cleanly whileDoxisStoreWriterConsumerstill receives the fullopencrawling-documentsevent for metadata, ACLs, e-file filing, and content linking.
3. Metadata-Driven Content Strategies (ContentPlanner)
Rather than eagerly opening binary streams from Apache Ozone or S3, ContentPlanner evaluates sizeInBytes, contentLength, sha256, local file: stat metadata, and configured URI prefixes upfront:
| Strategy | Runtime Behavior | Primary Scenario |
|---|---|---|
AUTO (Default) |
Selects per document: optional experimental PREDEFINED_LOCATOR if configured; UPLOAD when known size ≤ content.upload-max-bytes (2 GiB default); CONTENT_LINK if resolver matches and Blueline writer is loaded; otherwise content.fallback (default REFERENCE_ONLY). |
Mixed enterprise workloads combining normal office files with multi-terabyte media or NAS assets. |
UPLOAD |
Streams multipart binary via LazyInputStream to POST …/documents or POST …/versions. Fails fast if known size exceeds content.upload-max-bytes. |
Standard compliance archiving where binaries must reside inside Doxis storage volumes. |
CONTENT_LINK |
Registers an external UNC path or URL via oc-doxis-blueline-content-link (IDocument.addContentLinkPartDocument) without copying file bytes. |
Multi-terabyte engineering, video, or NAS datasets stored in place (verified against a 5 TB UNC test on CSB 14.4.1). |
REFERENCE_ONLY |
Creates the Doxis document metadata record without a binary content object and optionally stamps the source URI into reference-attribute. |
Metadata cataloguing and external pointer indexes. |
PREDEFINED_LOCATOR |
Experimental. Sends predefinedLocator, byte length, and optional SHA-256 over REST without an input stream. Disabled by default in standard SER storage adapters and unsupported in CSB 14.4.1. |
Bespoke Doxis deployments with custom pre-staged storage adapters. |
4. Zero-Copy Blueline Content Links with Isolated ClassLoading
In CSB 14.4.1, SER's REST API does not expose an endpoint for attaching external content links—attempts with predefinedLocator either fail or produce empty NO_CONTENT documents. To deliver true zero-copy archiving without forcing OpenCrawling to ship proprietary vendor libraries, PR #328 introduces the optional Maven module oc-doxis-blueline-content-link (enabled with -Pdoxis-blueline).
Because SER's Doxis Java (Blueline) client JARs bundle their own legacy CXF, Jackson, and Spring transitive stack compiled at Java 17 bytecode (14.4.1-1), ContentLinkWriterLoader dynamically loads oc-doxis-blueline-content-link alongside the administrator-staged JARs from content-link.client-lib-dir inside a child-first isolated class loader. Only the writer SPI, SLF4J, and JDK classes cross the boundary—communicating cleanly with CSB's transfer SOAP service (http://<csb-host>:<port>/sedna-transfer-service-xf/services/) with zero classpath conflicts in Java 25.
# Install licensed SER client JARs into local Maven repository (referenced artifacts:
# de.ser.doxis4.java-api:blueline, sednaclient-blueline, de.ser.doxis4.csb.client:sednaclient-api-nodep)
mvn install:install-file \
-Dfile=/path/to/WEB-INF/lib/blueline-14.4.1-1.jar \
-DgroupId=de.ser.doxis4.java-api -DartifactId=blueline \
-Dversion=14.4.1-1 -Dpackaging=jar -DgeneratePom=true
# Build the optional zero-copy adapter and stage alongside SER client libs
mvn -Pdoxis-blueline install -pl oc-doxis-blueline-content-link
cp /path/to/WEB-INF/lib/*.jar /opt/doxis-client-lib/
cp oc-doxis-blueline-content-link/target/oc-doxis-blueline-content-link-*.jar /opt/doxis-client-lib/
CSB Content-Link Semantics: Note that CSB does not stream external CONTENT_LINK binaries via REST GET …/contentObjects/{id} (returning SEDNA0104); Doxis clients resolve the UNC share directly. Re-crawling an unchanged or updated CONTENT_LINK document refreshes its typed descriptors and ACLs while preserving the existing link.
5. Upserts, Typed Descriptors & Electronic File (Akte) Filing
Before writing any record, DoxisOutputConnector queries CSB via CQL using external-id-attribute (default ObjectNumber). IDs longer than the target descriptor field limit are hashed deterministically with a h:<sha256Prefix> key so lookups never fail on column truncation.
- Conflict Resolution & No-Op Detection: Existing matches follow
conflict-resolution:NEW_VERSION(POST …/versions),UPDATE_METADATA(in-place descriptor & ACL patch), orSKIP. Settingcontent.change-marker-attribute(for instanceSystemId1) records<lastModified>|<length>so unmodified re-crawls skip redundant version proliferation. - Schema-Aware Descriptor Mapping (
DoxisDocumentMapper): Maps pseudo-keys (id,uri,lastModified) and custom metadata fields against live CSBDoxisSchemadefinitions—handlingSTRINGNUL sanitization/truncation,INTEGER,LONGINTEGER,FLOATINGPOINT,BOOL, and ISO-8601/epochDATE/DATETIMEconversions across single and multi-value attributes. - Electronic File (Akte) Filing (
DoxisRecordFiler): Supports five filing modes—NONE,FIXED,METADATA(readingdoxisRecordId),SOURCE_FOLDER(derived from parent URI segments), andKEY_METADATA—withfiling.auto-create=trueprovisioning missing e-files byrecord-class. Attach documents viaPRIMARY_PARENT(PUT …/primaryParent) orRELATIONSHIPinto an auto-created document-capableSTATICnode (defaultDocuments) under the e-file'sNODES_ONLYroot, with automatic stale cache eviction on HTTP404or DoxisINSTANCE*responses.
6. Zero-Trust OIS ACLs & Lifecycle Deletion Tombstones
Security enforcement in DoxisAclMapper resolves Open Ingestion Standard (OIS) principals against CSB /users and /groups (mapping OIS public to Doxis group everybody) and translates permissions consistently across both documents and enclosing e-files:
| OIS Permission | Document ACE (DocumentAceParams) |
E-File (Akte) ACE |
|---|---|---|
read |
GRANT VIEW_DOCUMENT_CONTENTS |
GRANT VIEW_FOLDER_CONTENTS |
write |
GRANT VIEW_DOCUMENT_CONTENTS, UPDATE_DOCUMENT, VERSION_DOCUMENT |
GRANT VIEW_FOLDER_CONTENTS, UPDATE_FOLDER, EDIT_FOLDER_DESCRIPTORS |
deny |
DENY VIEW_DOCUMENT_CONTENTS |
DENY VIEW_FOLDER_CONTENTS |
- Flexible Security Scoping (
security.mode): Apply ACEs atDOCUMENTlevel (default),RECORDlevel (inherited from the enclosing e-file when configured in Doxis Designer),DOCUMENT_AND_RECORD, orNONE. - Anti-Lockout Connector Guard (
grant-connector-user=true): Automatically grants the technical user management rights on e-files it creates (includingCREATE_FOLDERandSET_PRIMARY_PARENT) so restrictive instance ACLs never lock OpenCrawling out of subsequent filings. - Sync, Pruning & Strictness: Choose
CREATE_ONLYorADDITIVEe-file ACL updates (record-acl-sync), enableremove-stale=trueto revoke connector-managed ACEs removed at source without disturbing native Doxis admin grants, and togglesecurity.strict=trueto force rollback on unresolved principals orSECU0050rejections. - Reversible or Physical Deletion Tombstones: OIS
DocumentAction.DELETEevents resolve records viaexternal-id-attributeand execute eitherLOGICALremoval (POST …/remove, recommended default protecting linked storage) orPHYSICALpurge (DELETE …/documents/{uuid}, verified on CSB 14.4.1 to delete only the Doxis metadata record forCONTENT_LINKassets without touching the external file).
7. Configuration Reference & Verification
Every property binds under spring.opencrawling.output.doxis.* with corresponding doxis* UI keys in DoxisConnectorSettings. Below is a production-style configuration combining direct uploads with zero-copy UNC fallback for large binaries and automatic folder-based e-file creation:
spring:
opencrawling:
output:
type: doxis
doxis:
base-url: "http://csb.example.com:8080/restws/publicws/rest/api/v1"
customer-name: faststarter
username: crawler
password: "${DOXIS_PASSWORD}"
role: admins
repository: D_TEXTER
document-type: TX_MigratedDocument
external-id-attribute: ObjectNumberExternal
title-attribute: ObjectName
reference-attribute: URL
attribute-mapping:
author: ObjectAuthors
lastModified: ObjectDate
content:
strategy: AUTO
upload-max-bytes: 2147483648
verify: true
change-marker-attribute: SystemId1
content-link:
client-lib-dir: /opt/doxis-client-lib
uri-prefix: "file:///mnt/archive/"
link-prefix: "\\\\fileserver\\archive\\"
link-type: UNC
document-type: TX_LargeExternalDocument
filing:
mode: SOURCE_FOLDER
method: PRIMARY_PARENT
record-class: TX_SourceFolder
auto-create: true
security:
mode: RECORD
record-acl-sync: ADDITIVE
grant-connector-user: true
opencrawling:
consumer:
writer:
enabled: true
ingestion:
max-extract-bytes: 536870912
Automated Test Suites & Live CSB Smoke Check
PR #328 ships 77 unit and OkHttp MockWebServer contract tests modeled on recorded Doxis CSB 14.4.1 payloads (alongside 25 tests in oc-doxis-client) plus a live CSB smoke script that skips cleanly when CSB coordinates are absent:
# Run unit & CSB 14.4.1 fixture tests locally
mvn test -pl oc-doxis-client,oc-doxis-output-connector -am
# Validate connectivity via OpenCrawling CLI (logs in and calls POST /logout)
oc connector check --name Doxis_Output --type output
# Run live contract & optional write/verify/cleanup test against CSB 14.4.1+
./scripts/test-doxis-connector.sh
Ready to Archive & Liberate Content with Doxis 4?
Test the Doxis 4 Output Connector directly in the live browser pipeline simulator, explore the full Wiki property reference, or inspect PR #328 on GitHub.