Migrating Enterprise Crawler Workloads: Announcing the Apache ManifoldCF Migrator in OpenCrawling

01. The Motivation: Modernizing Enterprise Ingestion for the AI Era

For over a decade, Apache ManifoldCF (MCF) has served as the open-source cornerstone of enterprise search ingestion. Organizations have invested years crafting intricate crawler configurations, filtering rules, scheduling intervals, and repository connections across countless enterprise systems.

However, the demands of the modern enterprise AI stack have fundamentally shifted. Traditional monolithic crawlers designed solely for keyword search engines struggle to meet the requirements of today's Retrieval-Augmented Generation (RAG) ecosystems. Modern architectures demand decoupled, event-driven microservices, high-throughput asynchronous embedding generation, vector database integrations (such as Qdrant, Vespa, Milvus, Solr 10, and Luxir), and standardized payload contracts defined by the Open Ingestion Standard (OIS).

The oc-mcf-migrator bridges this divide. It provides enterprise teams with a reliable, audited path to modernize their ingestion infrastructure without losing years of accumulated operational rules, connection parameters, and scheduling logic.

02. Core Philosophy: Transparency and Fidelity Over Guesswork

Migration scripts often fail because they attempt to approximate or silently discard unsupported configurations, leading to subtle runtime failures weeks later. oc-mcf-migrator is built upon strict engineering invariants:

• Zero Guesswork: Connection classes must match registered mappers exactly. If an MCF connector has no direct OpenCrawling counterpart, it is skipped and clearly reported in the audit log—never guessed or silently dropped.
• Field-Level Fidelity Notes: Every configuration parameter is audited with precise taxonomy tags: EXACT, DROPPED, DEFAULTED, CONVERTED, or RUNTIME_RISK. Every warning includes an actionable recommendedAction suggestion.
• Automatic Secret Redaction: Passwords, private keys, API tokens, and credentials are automatically masked across all report formats, keeping migration plans safe to share across engineering teams.
• Job-Level Atomicity: An MCF job is migratable only if its repository connection and all referenced pipeline stages (transformation and output) are individually supported or explicitly overridden.

03. Architecture & Migration Pipeline

The migration engine operates across a structured, multi-stage pipeline:

04. Three Operational Modes

Whether evaluating an environment, generating declarative GitOps manifests, or performing a live cutover, oc-mcf-migrator provides three purpose-built execution modes:

05. Flexible Ingestion Sources

The migrator can read your existing Apache ManifoldCF topology through three distinct mechanisms:

• Live REST API: Connects directly to an active ManifoldCF JSON API endpoint using --mcf-url http://host:8345/mcf-api-service/json (with optional basic authentication credentials).
• Offline JSON Snapshots: Reads a directory containing exported JSON payloads (repositoryconnections.json, outputconnections.json, transformationconnections.json, authorityconnections.json, and jobs.json) via --mcf-input-dir.
• Native ManifoldCF XML Exports: Ingests the single-file XML archive produced by ManifoldCF's built-in ExportConfiguration utility via --mcf-export-file, enabling air-gapped migrations without requiring active network connectivity.

06. Extensible Connector Mapping & Manual Overrides

The migrator uses a standard Java ServiceLoader registry for connector mappers. The initial release includes native mappers for key enterprise workloads:

• FileSystem: Translates org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector into OpenCrawling's FileSystem repository connector, preserving root directory specifications and scan rules.
• Vespa: Maps org.apache.manifoldcf.agents.output.vespa.VespaOutputConnector directly to oc-vespa-output-connector, preserving document schemas and endpoint routing.
• Elasticsearch / OpenSearch: Translates legacy org.apache.manifoldcf.agents.output.elasticsearch.ElasticSearchOutputConnector configurations into oc-opensearch3-output-connector (targeting OpenSearch 2/3 clusters).

Additionally, CronTranslator automatically translates ManifoldCF's granular schedule records (day-of-week, hour, and minute matrices) into standard 5-part UNIX cron expressions, ensuring your automated ingestion schedules continue running seamlessly in OpenCrawling.

07. Visual Migration Wizard in Admin UI

For teams that prefer a graphical interface, the same migration engine powers an interactive, step-by-step wizard inside the OpenCrawling Admin Dashboard (oc-admin-ui → ManifoldCF Migration):

1. Configure: Enter your ManifoldCF API endpoint or select an offline snapshot directory.
2. Plan & Review: Inspect the parsed connections, jobs, compatibility scores, and field-level fidelity notes. Deselect individual jobs or connections with a single click.
3. Apply: Trigger the idempotent import into the OpenCrawling cluster.
4. Results: View execution outcomes, download the comprehensive migration report in JSON or HTML, and optionally start migrated crawling jobs immediately.

The wizard is backed by two dedicated endpoints on oc-runtime (POST /api/mcf-migration/plan and POST /api/mcf-migration/apply), ensuring that the web interface and the CLI share the exact same underlying logic, response DTOs, and compatibility checks.

08. CLI Quick Start

You can invoke oc-mcf-migrator using either the standalone JAR or the OpenCrawling CLI (oc mcf):

1. Discover Available Mappers

# Using the standalone JAR
java -jar oc-mcf-migrator-cli.jar list-mappers

# Using the OpenCrawling CLI
oc mcf list-mappers

2. Perform a Dry-Run Audit

java -jar oc-mcf-migrator-cli.jar audit \
  --mcf-url http://localhost:8345/mcf-api-service/json \
  --oc-url http://localhost:8080 \
  --report-format html \
  --report-file ./mcf-migration-report.html

3. Convert Jobs into OIS Declarative YAML Manifests

java -jar oc-mcf-migrator-cli.jar convert \
  --mcf-url http://localhost:8345/mcf-api-service/json \
  --output-dir ./ois-jobs

4. Import Live Workloads Directly into OpenCrawling

java -jar oc-mcf-migrator-cli.jar import \
  --mcf-url http://localhost:8345/mcf-api-service/json \
  --oc-url http://localhost:8080