Metadata-Driven Knowledge Work Automation Meets Enterprise RAG: Announcing the M-Files Repository Connector

1. Why Metadata-Driven ECM Needs Purpose-Built RAG Ingestion

Unlike traditional folder-centric content repositories, M-Files organizes enterprise knowledge around rich object graphs—linking Documents, Customers, Projects, Matters, and Workflows through typed property definitions (PropertyDef) and dynamic views. Bridging an M-Files vault to an enterprise Retrieval-Augmented Generation (RAG) pipeline presents unique architectural challenges:

• Truncation Without Offset Pagination: The M-Files Web Service (MFWS) REST API caps search results at limit (e.g., 500 objects) and provides no offset parameter. Naive crawlers silently lose every document beyond the first page when MoreResults: true is returned.
• Multi-File Document Bundles (MFD): In M-Files, a single logical object—such as a Master Services Agreement—can hold multiple constituent binary files (the main contract, signed schedules, and technical exhibits). Flattening or dropping child files breaks retrieval fidelity.
• Multi-Component Access Control Lists: Effective object permissions in M-Files combine a CustomComponent with zero or more AutomaticComponents driven by workflow states or Named ACLs, alongside metadata-driven pseudo-users (e.g., "Project Manager"). A security mapper must intersect active components restrictively and fail closed.
• ManifoldCF Migration Continuity: Enterprises migrating from Apache ManifoldCF need drop-in compatibility with existing mfiles:user:<id> and mfiles:group:<id> permission tokens as well as custom Vault Application Framework (VAF) indexer endpoints.

The new oc-mfiles-repository-connector was engineered from the ground up on the OpenCrawling SPI and Java 25 to solve each of these requirements natively while emitting standardized Open Ingestion Standard (OIS) documents and deletion tombstones.

2. Native MFWS Client, Triple Authentication & Host Pinning

At the transport layer, MFilesClient communicates directly with <server>/REST over native Java 25 HttpClient with zero heavyweight SDK dependencies. It supports three authentication modes configured via auth-type:

• Vault Authentication Token (token, default): Exchanges credentials, optional Windows Domain, VaultGuid, and a per-scan SessionID at POST /server/authenticationtokens for an X-Authentication token. Empty credentials are rejected upfront because M-Files otherwise issues an anonymous token that fails every subsequent vault call.
• Header Basic Auth (basic): Sends X-Username (formatted as DOMAIN\username when a domain is configured), X-Password, and X-Vault headers.
• OAuth 2.0 / Microsoft Entra ID (oauth2): Sends Authorization: Bearer <token> and X-Vault using either a pre-issued bearer token or an automated client-credentials grant (MFilesOAuth2TokenSource) that refreshes tokens proactively before expiry.

If a session token expires mid-crawl (HTTP 401 or 403), MFilesClient automatically re-authenticates once and retries the request. Transient rate limits (HTTP 429) and gateway errors (502–504) are retried with exponential backoff. Furthermore, when following file download URLs supplied by VAF payloads, the client enforces strict host pinning so authentication headers are never forwarded to external hosts.

3. Four Crawl Modes & Recursive Binary Time-Window Splitting

Enterprises organize M-Files vaults in different ways, so MFilesRepositoryConnector provides four distinct crawl modes via crawl-mode:

1. Class & Condition Search (search, default) & All Objects (all_objects)

Queries GET /objects per configured class (class-ids, passed as p100=<id>), restricted to object-type (0 = Document, or -1 to crawl all object types) and optional MFWS search conditions (search-query, such as p1020=5&q=contract, or plain text for quick search). Setting crawl-mode: all_objects crawls the entire vault without class or query restrictions.

2. Recursive View Traversal (view)

Traverses a configured M-Files view (view-id, e.g., 101 or v101/L5) via GET /views/<path>/items, recursively descending into sub-views, lookup/text property folders, and traditional folders up to depth 32. Because an object can appear under multiple property folders in a view, the connector tracks visited object keys and emits each object exactly once.

3. Vault Application Framework Change Feed (vaf)

Consumes the change feed (<vaf-base-path>/changes?since=<cursor>) and normalized object endpoint (<vaf-base-path>/object/{objType}/{objId}/{version}/normalized) of an M-Files VAF application—matching the protocol used by the Apache ManifoldCF M-Files connector—where effective permissions (including metadata-driven pseudo-users) are resolved server-side inside the vault.

Overcoming the MFWS Truncation Limit with Recursive Time-Window Splitting

Because MFWS truncates GET /objects responses at limit (batch-size, default 500) and offers no offset pagination, how do you guarantee 100% recall across millions of vault objects?

Whenever a search returns MoreResults: true, MFilesRepositoryConnector automatically switches to recursive last-modified time-window splitting using MFWS property 21 (LastModified, p21>= and p21<=). Starting from the lower bound (modified-after, the incremental watermark, or 1970-01-01T00:00:00Z) up to the scan start timestamp, the connector recursively bisects time intervals in half until every sub-window returns MoreResults: false, deduplicating objects that fall on window boundaries.

4. Single-File, Multi-File (MFD), Non-File Objects & Version Modes

MFilesDocumentBuilder maps every M-Files object type and attachment topology into clean, canonical OIS URIs:

• Single-File Documents (SFD): Emitted as a single OIS document under mfiles://<vaultGuid>/objects/<type>/<id> with its binary stream attached.
• Multi-File Documents (MFD): Every constituent file is emitted as its own first-class OIS document under mfiles://<vaultGuid>/objects/<type>/<id>/files/<fileId>, stamping mfiles.parentId with the parent object URI (mfiles://<vaultGuid>/objects/<type>/<id>). Master agreements and every attached exhibit become independently searchable while retaining parent-child lineage.
• Non-File Objects: Metadata-only objects (such as Customers, Vendors, or Projects when crawling non-document object types) are indexed with their full set of normalized properties.
• Version Control (version-mode): In latest_only mode (default), objects use versionless stable URIs so new revisions cleanly replace older ones in downstream vector stores. In all_versions mode, the connector traverses GET /objects/{type}/{id}/history and emits prior versions under .../versions/<n> (and .../versions/<n>/files/<fileId>).

5. Typed Property Normalization (MFilesValues)

At connection time, the connector loads the vault's property definitions from GET /structure/properties. For each object, MFilesValues normalizes every property value according to its MFDataType under <property-prefix><camelCaseName> (default prefix: mfiles_prop_):

6. Fail-Closed Zero-Trust Security & Restrictive Component Intersection

In an enterprise RAG system, security cannot be an afterthought. MFilesSecurityMapper evaluates object permissions from GET /objects/{type}/{id}/{version}/permissions (or VAF aclTokens / denyTokens) on a strict fail-closed basis:

• Restrictive Component Intersection: An M-Files object's effective AccessControlList contains a CustomComponent and zero or more active AutomaticComponents (contributed by workflow states or Named ACLs). Because M-Files combines components restrictively, MFilesSecurityMapper grants read access to an identity only if every active component allows it.
• Explicit Deny Precedence & Verified Numeric Codes: On live M-Files servers, ReadPermission: 1 (or "Allow", or bit 1 of AccessFlags) grants read access. Any other numeric ReadPermission code or "Deny" is mapped to an explicit OIS access: "deny" rule—ensuring unverified codes can only restrict access, never widen it.
• Safe Handling of Metadata Pseudo-Users: Metadata-driven entries (flagged IsPsudoUser: true or HasConcreteUserOrGruopID: false, such as "Project manager") cannot be resolved to concrete user IDs over plain MFWS. Rather than guessing, the connector grants nothing for pseudo-users in MFWS mode and resolves them 1-to-1 when run in vaf mode.
• Token or Name Principals & Multi-Vault Qualification: Emit principals as ManifoldCF-compatible tokens (mfiles:user:<id>, mfiles:group:<id>) or set principal-format: name to resolve account and group names via built-in value lists 6 (Users) and 16 (User Groups). Enable qualify-principals: true to prefix identities with <vaultGuid>/ so per-vault IDs never collide in shared multi-vault vector indexes.
• Deny-by-Default Fallbacks: If permissions are unreadable (e.g., HTTP 404), the connector applies configured fallback-principals or an empty ACL—documents are never marked public by accident.

7. Java 25 Virtual Threads, TempFileInputStream & Incremental Tombstones

High-throughput vault ingestion combines concurrent I/O with strict resource hygiene:

• Java 25 Structured Concurrency: Object batches are enriched concurrently using StructuredTaskScope.open() and ObservabilityTask.observed(...), bounded by a configurable Semaphore (parallelism, default 4) so W3C OpenTelemetry traces and MDC context propagate cleanly across virtual threads without overwhelming the M-Files server.
• Zero Disk Leaks with TempFileInputStream: Binary file streams downloaded from GET /objects/{type}/{id}/{version}/files/{fileId}/content are spooled to size-capped temporary files (max-file-size-bytes, default 50 MB). The moment Apache Tika 4 or the Claim-Check store (Apache Ozone / S3) closes the stream, TempFileInputStream immediately deletes the temporary file.
• Incremental Watermarks & Multi-File Lineage Pruning (MFilesCrawlState): When incremental-enabled: true is set, the connector persists per-vault, per-scope watermarks (with a configurable clock-skew-seconds overlap, default 300s) and tracks every emitted document/file URI per object in data/mfiles-state/.

8. Sample OIS Document & Tombstone Payloads

Below is a real example of an ingested M-Files NDA document mapped to an OIS UPSERT payload, complete with normalized properties and Zero-Trust permissions:

{
  "id": "mfiles://C0B3B52E-6512-4BD8-8F2B-2A8E9A349201/objects/0/48192",
  "action": "UPSERT",
  "acl": "mfiles:user:12,mfiles:group:5",
  "metadata": {
    "name": ["Global Vendor NDA - Final Signed.pdf"],
    "title": ["Global Vendor NDA - Final Signed"],
    "mimeType": ["application/pdf"],
    "mfiles.vaultGuid": ["C0B3B52E-6512-4BD8-8F2B-2A8E9A349201"],
    "mfiles.vaultName": ["Contracts"],
    "mfiles.objType": ["0"],
    "mfiles.objId": ["48192"],
    "mfiles.version": ["3"],
    "mfiles.isLatestVersion": ["true"],
    "mfiles.title": ["Global Vendor NDA - Final Signed"],
    "mfiles.classId": ["14"],
    "mfiles.className": ["Non-Disclosure Agreement"],
    "mfiles.singleFile": ["true"],
    "mfiles.fileCount": ["1"],
    "mfiles.fileId": ["7"],
    "mfiles.fileName": ["Global Vendor NDA - Final Signed.pdf"],
    "mfiles.fileSize": ["1482910"],
    "mfiles.created": ["2026-04-10T10:30:00Z"],
    "mfiles.createdBy": ["sarah.connor"],
    "mfiles.lastModified": ["2026-09-27T15:20:00Z"],
    "mfiles.lastModifiedBy": ["alex.vance"],
    "mfiles.state": ["Approved"],
    "mfiles_identity_users": ["mfiles:user:12"],
    "mfiles_identity_groups": ["mfiles:group:5"],
    "mfiles_prop_customer": ["Cyberdyne Systems"],
    "mfiles_prop_customer_id": ["5"],
    "mfiles_prop_contractValue": ["150000"],
    "mfiles_prop_effectiveDate": ["2026-05-01"],
    "mfiles_prop_tags": ["Legal", "Vendor"]
  },
  "security": {
    "inheritanceEnabled": false,
    "permissions": [
      { "identity": "mfiles:user:12", "identityType": "user", "access": "read" },
      { "identity": "mfiles:group:5", "identityType": "group", "access": "read" },
      { "identity": "mfiles:user:9", "identityType": "user", "access": "deny" }
    ]
  }
}

And when an object or attached file is deleted in the vault, the connector dispatches a lightweight OIS DELETE tombstone:

{
  "id": "mfiles://C0B3B52E-6512-4BD8-8F2B-2A8E9A349201/objects/0/48192",
  "action": "DELETE",
  "metadata": {
    "mfiles.vaultGuid": ["C0B3B52E-6512-4BD8-8F2B-2A8E9A349201"],
    "mfiles.objType": ["0"],
    "mfiles.objId": ["48192"],
    "mfiles.status": ["DELETED"]
  }
}

9. Configuration, CLI Diagnostics & 9-Step Decoupled Verification

Configure the M-Files Repository Connector via Spring Boot properties (spring.opencrawling.connector.mfiles.*), Admin UI (oc-admin-ui), or connectors.json:

End-to-End Verification with the Stateful MFWS Mock & Secure MCP Server

We ship a comprehensive test suite including a stateful Python MFWS emulator (oc-mfiles-repository-connector/mfiles-mock/mfws_mock.py) that serves a deterministic generated vault with real PDF, DOCX, and TXT files, multi-component ACLs, and live mutation endpoints:

# 1. Verify vault connectivity via OpenCrawling CLI
$ oc connector check --name mfiles-vault --type repository

# 2. Run standalone unit + MockWebServer + stateful MFWS mock tests
$ ./scripts/test-mfiles-repository-connector.sh

# 3. Run the 9-step full decoupled Docker Compose pipeline (Kafka, pgvector, Ollama, Secure MCP Server)
$ ./scripts/test-mfiles-decoupled.sh

The 9-step decoupled test script (test-mfiles-decoupled.sh) boots the complete microservice topology, reconciles indexed chunks against the mock vault's ground truth (/mock/stats), verifies Zero-Trust ACL component intersection and explicit deny rules through the Secure MCP Server (secureVectorSearch, getDocumentContent, and listAccessibleSources over Streamable HTTP /mcp), triggers live vault mutations (removing an exhibit from a multi-file document, deleting a document, and adding a new NDA), and confirms that an incremental crawl updates pgvector and purges deleted chunks via tombstones.