Introducing Maven Connector Archetypes: Scaffold Custom Connectors in Seconds

Standardized Extension for OpenCrawling

OpenCrawling's architecture is built around extensible connector interfaces: RepositoryConnector for ingestion sources, OutputConnector for vector/search stores, and TransformationConnector for data processing pipelines.

To facilitate developers building custom enterprise connectors, OpenCrawling now provides an official suite of Maven Archetypes published under org.opencrawling.archetypes.

The Three Connector Archetypes

• opencrawling-archetype-repository-connector: For building ingestion sources (CMS, SaaS REST APIs, Databases, Proprietary Repositories).
• opencrawling-archetype-output-connector: For building vector search, search engine, or database output destinations.
• opencrawling-archetype-transformation-connector: For custom text chunking, anonymization, and metadata enrichment pipelines.

Quickstart Scaffolding Command

Generate a complete, production-ready connector project with a single command:

mvn archetype:generate \
  -DarchetypeGroupId=org.opencrawling.archetypes \
  -DarchetypeArtifactId=opencrawling-archetype-repository-connector \
  -DarchetypeVersion=1.0.0-SNAPSHOT \
  -DgroupId=com.mycompany.connectors \
  -DartifactId=my-repository-connector \
  -Dversion=1.0.0-SNAPSHOT \
  -DconnectorName=SampleRepositoryConnector

Built-in Docker Compose Overlay & Admin UI Testing

Every generated project includes a complete integration environment:

• Surefire & Failsafe Integration Tests: Pre-wired unit tests (*Test.java) and integration tests (*IT.java) executed during mvn verify.
• Docker Compose Overlay: Automatically mounts the compiled connector JAR into OpenCrawling's plugin directory (/app/plugins/).
• Admin UI Integration: Includes a pre-seeded connectors.json overlay file, instantly rendering the new custom connector inside the Admin UI (http://localhost:3000) dropdown menus for visual job configuration.