Accelerating Developer Velocity: Announcing run.sh and run.bat Root CLI Helpers for OpenCrawling

1. The Modern Microservices Developer Friction

OpenCrawling represents a state-of-the-art, enterprise-grade ingestion and RAG platform. Its decoupled architecture spans an array of specialized microservices—including the crawler engine, ingestion consumer, embedding vectorizer, writer consumer, and secure Model Context Protocol (MCP) server—alongside backing infrastructure such as Apache Kafka, PostgreSQL with pgvector, Redis, Ollama, and OpenTelemetry.

While this architecture provides unmatched enterprise resilience and horizontal scalability, local development iterated under significant friction:

• Verbose Command Syntax: Orchestrating local stacks required typing lengthy, error-prone Docker Compose commands with multiple flags:

  docker compose -f docker-compose-decoupled.yml -f docker-compose-decoupled.local.yml up -d --build
  docker compose -f docker-compose-maven-builder.yml run --rm oc-maven-builder mvn clean package -DskipTests

• Sluggish Cold Rebuilds: Without BuildKit cache integration, rebuilding container images downloaded hundreds of megabytes of Maven dependencies over the network on every source iteration (4 to 10+ minutes per build).
• Platform Discrepancies on Windows: While Linux and macOS developers leveraged helper scripts in scripts/, Windows developers working outside WSL or Git Bash lacked native CMD/batch tooling to easily spin up local stacks.

With Issue #217, we solved this by designing unified root CLI helpers that eliminate friction while preserving complete build hermeticity for continuous integration.

2. Zero CI Impact: Two-Tier Build Architecture

A core architectural principle of OpenCrawling is that CI production Dockerfiles (docker/Dockerfile.* and oc-runtime/Dockerfile) remain completely untouched. CI workflows on GitHub Actions must always execute clean, isolated, reproducible builds from scratch.

To accelerate local development without compromising production Dockerfiles, run.sh and run.bat leverage a dual-mode strategy:

1. Local Cache Dockerfiles (docker/local/Dockerfile.*): Local Dockerfiles include BuildKit cache directives:

   RUN --mount=type=cache,target=/root/.m2,id=opencrawling-m2-cache,sharing=shared \
       mvn clean package -DskipTests

   Redundant dependency downloads (such as mvn dependency:go-offline) are eliminated, and all decoupled services share the same persistent cache volume on the local Docker engine.
2. Containerized Maven Builder (oc-maven-builder): Mounts the developer's host ~/.m2 repository directly into an official Eclipse Temurin JDK 25 builder container:

   docker compose -f docker-compose-maven-builder.yml run --rm oc-maven-builder

   Developers can compile, package, and execute Maven goals without needing JDK 25 or Maven installed on their host workstations!

3. Cross-Platform Parity: macOS, Linux, and Windows CMD

Whether you develop on macOS, Ubuntu, or Windows Command Prompt, the experience is identical:

• macOS & Linux (./run.sh): Implemented in POSIX Bash with strict error handling (set -euo pipefail), ANSI color styling, and automated fallback to unstyled text when NO_COLOR=1 is set or output is piped.
• Windows CMD (run.bat): Native batch script with clean status banners, full argument forwarding (%*), and automatic translation of %USERPROFILE% to HOME:

  if not defined HOME set "HOME=%USERPROFILE%"
  if not exist "%HOME%\.m2" mkdir "%HOME%\.m2"

  This ensures that Docker Compose volume mounts (${HOME}/.m2:/root/.m2) function seamlessly on Windows without requiring manual environment configuration.

4. Complete Developer Command Matrix

run.sh and run.bat group everyday engineering workflows into three intuitive categories:

5. Pre-Flight Diagnostics & Error Handling

To prevent cryptic container crashes, run.sh and run.bat validate prerequisites before dispatching commands:

• Docker Daemon Verification: Verifies that the Docker CLI exists and that the Docker daemon is responding (docker info). If Docker Desktop is stopped, the script exits immediately with a helpful troubleshooting message.
• Tooling Detection for UI & Host Builds: Checks for mvn on host builds, and checks for node and npm before attempting to start the React Admin UI Vite dev server (ui:dev).
• Self-Healing Cache Folders: Automatically ensures ${HOME}/.m2 exists before launching Docker Compose, preventing Docker from creating root-owned empty directories on Linux workstations.

6. Automated Verification: 7-Step Test Suite & CI Matrix

Developer convenience scripts must be as rigorously tested as production code. We implemented a dedicated test suite under scripts/test-run-scripts.sh:

This test suite is automatically executed as part of the GitHub Actions CI matrix in .github/workflows/integration-tests.yml under the job name Root CLI Runner Scripts Test Suite.

7. Quick Start: Try It Today!

Clone the repository and test the new CLI runners immediately:

# macOS or Linux
git clone https://github.com/opencrawling/opencrawling.git
cd opencrawling

./run.sh help             # Explore available commands
./run.sh build            # Compile via containerized oc-maven-builder
./run.sh decoupled:up     # Launch fast local decoupled microservices stack
./run.sh status           # Inspect container health and port bindings
./run.sh down             # Stop local containers

# Windows Command Prompt
run.bat build
run.bat decoupled:up
run.bat down