Introducing the Alfresco Repository Connector for OpenCrawling

1. Why Alfresco? The Enterprise ECM Challenge

Alfresco Content Services is the backbone of document management for thousands of enterprises globally — legal firms, healthcare organizations, financial institutions, and government agencies store millions of documents in Alfresco repositories. These documents hold critical institutional knowledge: contracts, case files, compliance records, technical specifications, and policy documents.

However, making this knowledge searchable and accessible to AI agents has historically required expensive, proprietary middleware or ad-hoc scripting. The oc-alfresco-repository-connector changes that: it provides a production-grade, reactive, zero-config bridge from Alfresco to your AI pipeline using only the Alfresco REST API — no AMP packages, no Alfresco-side customization required.

2. Architecture: How the Connector Works

The AlfrescoRepositoryConnector implements the standard RepositoryConnector interface from oc-core. Its scan(String basePath) method returns a reactive Flux<RepositoryDocument> stream, integrating seamlessly into both the direct output mode (single JVM) and the decoupled microservices pipeline (Kafka-based).

Recursive Node Tree Traversal with Structured Task Scope

When scan() is called, the connector resolves the starting node and recursively enumerates all children via paginated API calls to the /nodes/{nodeId}/children endpoint. Folder traversal is parallelised using Java 25’s Structured Task Scope — each discovered subfolder is forked into a new virtual thread, while the parent scope waits for all children to complete before propagating upstream:

// Concurrent subfolder scanning — Java 25 Structured Task Scope
try (var scope = StructuredTaskScope.open()) {
    for (JsonNode entryWrapper : entriesNode) {
        JsonNode entry = entryWrapper.path("entry");
        boolean isFolder = entry.path("isFolder").asBoolean(false);
        boolean isFile   = entry.path("isFile").asBoolean(false);

        if (isFolder) {
            scope.fork(() -> {
                scanFolder(childId, null, sink); // recursive, parallel
                return null;
            });
        } else if (isFile) {
            RepositoryDocument doc = createDocument(entry);
            sink.next(doc);  // emit to reactive Flux stream
        }
    }
    scope.join(); // wait for all virtual threads to complete
}

Each virtual thread independently paginates through the children of its assigned folder, fetching up to batch-size entries per API call. This means even repositories with thousands of deeply-nested folders are scanned concurrently without blocking the main crawl thread.

The Claim Check Pattern for Remote Content

Unlike the FileSystem connector (which references local files directly), Alfresco documents must be fetched over HTTP. The connector streams each document’s binary content via the /nodes/{nodeId}/content endpoint and saves it to a shared local directory (/data/claims/) before publishing the URI to Kafka. This decouples the slow HTTP download from the fast Kafka publish step and makes content available to downstream consumers without re-fetching from Alfresco.

3. Flexible SCAN_PATH: Root, Path, or Node UUID

The connector resolves the SCAN_PATH argument (or environment variable) using three strategies, giving you precise control over the crawl scope:

This means you can run targeted crawl jobs on specific Alfresco sites, document libraries, or individual folder trees — without scanning the entire repository every time.

4. Rich Metadata Extraction

The connector appends ?include=properties to all Alfresco API calls, retrieving the full Alfresco content model properties alongside each node entry. Every cm:* property returned by the API — titles, descriptions, authors, custom site properties — is mapped directly into the RepositoryDocument metadata map and stored in the vector index:

// All Alfresco properties mapped to document metadata
JsonNode propertiesNode = entry.path("properties");
propertiesNode.properties().iterator().forEachRemaining(prop -> {
    metadata.put(prop.getKey(), List.of(prop.getValue().asText()));
});
// Result in PGVector: {"cm:title": ["Q4 Strategy"], "cm:author": ["J. Smith"], ...}

This means your RAG queries can filter or boost results based on Alfresco metadata — for example, retrieving only documents authored by a specific team member, or filtering by nodeType to exclude cm:thumbnail entries.

Metadata extracted per document

5. Configuration Reference

The connector is configured via Spring Boot properties, which map directly to environment variables in Docker deployments:

6. Docker Compose: Running the Alfresco Pipeline

Activate the Alfresco connector by setting CONNECTOR_TYPE=alfresco on the oc-crawler service. Here is a minimal configuration pointing the crawler at an Alfresco instance on the same Docker network:

# In docker-compose-decoupled.yml — oc-crawler service configuration
oc-crawler:
  environment:
    - CONNECTOR_TYPE=alfresco
    - SCAN_PATH=/Company Home/Sites/engineering-docs
    - ALFRESCO_URL=http://alfresco:8080/alfresco/api/-default-/public/alfresco/versions/1
    - ALFRESCO_USERNAME=admin
    - ALFRESCO_PASSWORD=admin
    - ALFRESCO_BATCH_SIZE=100
    - SPRING_KAFKA_BOOTSTRAP_SERVERS=kafka:9094
    - SPRING_OPENCRAWLING_CRAWL_ON_STARTUP=true
  volumes:
    - ./oc-runtime/data:/data   # shared volume for Claim Check files
  depends_on:
    - kafka
    - postgres

Start the full decoupled pipeline and watch the crawler scan your Alfresco repository in real time:

# Build JARs (source builds only)
mvn clean install -DskipTests

# Start the full decoupled pipeline
docker compose -f docker-compose-decoupled.yml up -d --build

# Follow crawler logs to watch the Alfresco node tree scan
docker compose -f docker-compose-decoupled.yml logs -f oc-crawler-service

7. Verifying Ingestion with PGVector

Once the crawl job finishes, verify that Alfresco document chunks are correctly stored in the vector index using these standard SQL queries:

-- Connect to the vector database
docker exec -i postgres-vector-decoupled psql -U opencrawling -d opencrawling

-- Count total Alfresco chunks stored
SELECT count(*) FROM vector_store_1024;

-- Inspect document content and Alfresco metadata
SELECT id,
       LEFT(content, 200)      AS preview,
       metadata->>'name'       AS filename,
       metadata->>'mimeType'   AS mime_type,
       metadata->>'nodeType'   AS node_type,
       metadata->>'cm:author'  AS author
FROM vector_store_1024
ORDER BY id
LIMIT 10;

-- Semantic similarity search over Alfresco content
SELECT id,
       LEFT(content, 300)      AS preview,
       metadata->>'name'       AS filename,
       1 - (embedding <=> (
           SELECT embedding FROM vector_store_1024 LIMIT 1
       ))                      AS similarity
FROM vector_store_1024
ORDER BY similarity DESC
LIMIT 5;

8. Headless Mode for Scheduled Nightly Crawls

For automated, scheduled re-indexing (e.g., a nightly job that crawls all content modified since the last run), use the Standalone/Headless mode. The container boots, runs the full crawl, publishes all content to Kafka, and exits cleanly:

services:
  oc-crawler-alfresco-nightly:
    build:
      context: .
      dockerfile: docker/Dockerfile.crawler
    container_name: oc-crawler-alfresco-nightly
    environment:
      - SPRING_KAFKA_BOOTSTRAP_SERVERS=kafka:9094
      - SPRING_OPENCRAWLING_CRAWL_ON_STARTUP=true
      - SPRING_OPENCRAWLING_REPOSITORY_CONNECTOR_TYPE=alfresco
      - SPRING_OPENCRAWLING_SCAN_PATH=/Company Home/Sites
      - SPRING_OPENCRAWLING_TRANSFORMATION_CONNECTOR=Ollama local
      - ALFRESCO_URL=http://alfresco:8080/alfresco/api/-default-/public/alfresco/versions/1
      - ALFRESCO_USERNAME=crawl-service-account
      - ALFRESCO_PASSWORD=${ALFRESCO_SERVICE_PASSWORD}
    volumes:
      - ./oc-runtime/data:/data
    networks:
      - opencrawling-network

This pattern integrates natively with Kubernetes CronJob resources, Apache Airflow DAGs, and any CI/CD scheduler that supports Docker-based task execution.

9. What’s Next for the Alfresco Connector

• ACL Synchronization — Mapping Alfresco node permission entries to ACL tokens in the vector index, enabling the Secure MCP Server to enforce Alfresco document-level access control during AI similarity queries.
• Incremental Crawling — Leveraging the Alfresco Audit Service and modifiedAt timestamps to crawl only nodes changed or added since the previous run, dramatically reducing re-index times.
• CMIS Protocol Support — A companion CMIS-based connector for cross-vendor ECM compatibility with other content repositories implementing the CMIS 1.1 standard.
• Site & Library Filtering — Allow-list and deny-list configuration for Alfresco sites, document libraries, and content types to narrow crawl scope without changing SCAN_PATH.
• OAuth 2.0 Authentication — Support for OAuth 2.0 / OpenID Connect in addition to Basic Auth, enabling integration with Alfresco Identity Service (Keycloak-based) deployments.