Contract-First Ingestion & Living Documentation: Announcing OpenAPI 3.1 & Interactive Swagger UI in OpenCrawling

1. The Modern Enterprise Ingestion Dilemma: Beyond Documentation Drift

Building high-performance document ingestion engines on modern runtimes (Java 25 Virtual Threads, Structured Concurrency, Spring Boot 4) provides immense computing power. However, as OpenCrawling's footprint expanded across 10 functional domains—spanning job lifecycle orchestration, dynamic connector registration, OIS direct streaming, columnar Parquet lakehouse auditing, and AIOps root-cause telemetry—we recognized an architectural imperative:

• Eliminating Documentation Drift: Handwritten wiki tables and static markdown guides inevitably lag behind rapid controller code enhancements. Developers need living documentation where every query parameter, payload DTO schema, HTTP status code, and validation rule is derived directly from the active codebase.
• Interactive In-Browser Exploration: DevOps and integration engineers testing new pipelines should not need to manually craft complex curl commands or configure Postman collections. An interactive explorer with a "Try it out" sandbox accelerates day-0 onboarding.
• Preventing Frontend Runtime Mismatches: The OpenCrawling Admin UI (oc-admin-ui), built on React 19 and Vite, requires strict type safety. Generating TypeScript interfaces and API client functions directly from backend contracts guarantees zero payload mismatches.
• Autonomous AI Agent Operability: LLMs (GPT-4o, Claude 3.5, Gemini 2.0) and agentic frameworks (LangChain, LlamaIndex, Semantic Kernel) natively translate OpenAPI 3.1 definitions into structured function-calling schemas. A standard OpenAPI foundation turns OpenCrawling into an out-of-the-box toolkit for autonomous agents.

2. Platform Architecture: Standardizing on OpenAPI 3.1

Rather than adopting legacy Swagger 2.0 or OpenAPI 3.0, OpenCrawling standardizes strictly on OpenAPI 3.1. This version represents a milestone alignment with JSON Schema 2020-12, supporting full schema polymorphism, standard type arrays, and modern web interoperability.

Under the hood, OpenCrawling leverages springdoc-openapi-starter-webmvc-ui (v3.1+), seamlessly integrated with Spring Boot 4. The architecture utilizes a code-first approach via standard Swagger annotations, preserving developer velocity in Java while generating contract-first specifications for downstream consumers:

// oc-runtime OpenApiConfig.java
@Configuration
public class OpenApiConfig {

    public static final String OIS_TOKEN_AUTH = "OisTokenAuth";

    @Bean
    public OpenAPI openCrawlingOpenAPI() {
        return new OpenAPI()
                .info(new Info()
                        .title("OpenCrawling Runtime REST API")
                        .description("Enterprise-grade document crawling, RAG ingestion, and Lakehouse analytics pipeline.")
                        .version("1.0.0")
                        .license(new License().name("Apache License 2.0").url("https://www.apache.org/licenses/LICENSE-2.0")))
                .components(new Components()
                        .addSecuritySchemes(OIS_TOKEN_AUTH, new SecurityScheme()
                                .name(OIS_TOKEN_AUTH)
                                .type(SecurityScheme.Type.HTTP)
                                .scheme("bearer")
                                .bearerFormat("Token")
                                .description("Bearer token authentication for OIS document direct ingestion")));
    }
}

3. Interactive Swagger UI Explorer: Zero-Setup Sandbox

Every OpenCrawling runtime instance now serves the interactive Swagger UI out-of-the-box:

The explorer allows developers to expand any endpoint, inspect required query filters (e.g. page, size, status), review comprehensive JSON request bodies, and execute live queries directly against running clusters.

4. Comprehensive 10-Domain API Coverage

Every single REST controller and DTO record in oc-runtime has been rigorously annotated with @Tag, @Operation, @ApiResponse, and @Schema annotations across 10 functional domains:

5. Securing Endpoints with Bearer Token Authentication

The OpenAPI specification explicitly defines the OisTokenAuth HTTP Bearer security scheme. In the Swagger UI portal, developers can click the Authorize 🔓 button, input their ingestion token, and test token-protected endpoints like POST /api/v1/ingest/ois/{jobId} seamlessly without having to configure HTTP header flags manually.

// Sample OisIngestController Swagger documentation
@Operation(
    summary = "Direct Ingestion of OIS Document",
    description = "Publishes an Open Ingestion Standard (OIS) document payload directly into the active job pipeline.",
    security = @SecurityRequirement(name = OpenApiConfig.OIS_TOKEN_AUTH)
)
@ApiResponses(value = {
    @ApiResponse(responseCode = "202", description = "Document accepted and published to Kafka"),
    @ApiResponse(responseCode = "400", description = "Malformed JSON or invalid OIS document structure"),
    @ApiResponse(responseCode = "401", description = "Missing or invalid bearer token"),
    @ApiResponse(responseCode = "404", description = "Endpoint disabled or target job ID not found"),
    @ApiResponse(responseCode = "409", description = "Job runs in migration mode (OIS supports RAG mode only)"),
    @ApiResponse(responseCode = "503", description = "Kafka ingestion event bus unavailable")
})
@PostMapping("/{jobId}")
public ResponseEntity<String> ingest(...) { ... }

6. End-to-End Type Safety for React 19 & Client SDKs

With machine-readable OpenAPI schemas exported at /v3/api-docs, downstream clients benefit from automated contract generation:

• React 19 Frontend (oc-admin-ui): Using openapi-typescript, frontend developers can run npm run generate:api to synthesize fully typed interfaces directly into src/lib/api-generated.ts. If backend engineers alter a DTO record or add query options, TypeScript build checks immediately flag type breakages before runtime deployment.
• Java Client SDK (oc-java-client-sdk): The official Java SDK models align with the server-side OpenAPI schema. Automated contract verification tests validate that Java SDK records match backend response DTOs.
• Multi-Language Client Generation: Python, Go, and Node.js teams can generate custom clients instantly using standard tooling:

  npx @openapitools/openapi-generator-cli generate \
    -i http://localhost:8080/v3/api-docs \
    -g python \
    -o ./opencrawling-python-sdk

7. Empowering Autonomous AI Agents & MCP Tool Calling

One of the most exciting advantages of standardizing on OpenAPI 3.1 is its native compatibility with LLM Function Calling and the Model Context Protocol (MCP):

• Automated Tool Definitions: Autonomous agents running on LangChain, LlamaIndex, or OpenAI Custom GPTs can ingest the OpenAPI 3.1 schema directly, converting each REST endpoint into an executable tool with full parameter descriptions and JSON Schema validations.
• Natural Language Crawling Operations: An agent can receive an instruction like "Inspect job #4, run a DuckDB diagnostic query on failed PDFs, and fetch the AIOps root-cause trace", map it to the corresponding OpenAPI operations, and execute the multi-step investigation autonomously.

8. Configuration & Nginx Docker Reverse Proxy

OpenAPI docs and Swagger UI paths are fully configurable via Spring Boot properties in application.yml:

springdoc:
  api-docs:
    path: ${SPRINGDOC_API_DOCS_PATH:/v3/api-docs}
    enabled: ${SPRINGDOC_API_DOCS_ENABLED:true}
    version: OPENAPI_3_1
  swagger-ui:
    path: ${SPRINGDOC_SWAGGER_UI_PATH:/swagger-ui.html}
    enabled: ${SPRINGDOC_SWAGGER_UI_ENABLED:true}
    display-request-duration: true
    operations-sorter: method
    tags-sorter: alpha
    doc-expansion: none
  show-actuator: false

In containerized deployments, oc-admin-ui's Nginx reverse proxy routes all /v3 and /swagger-ui traffic transparently to the backend container, while Spring WebMVC's single-page application router (WebConfig.java) preserves these documentation paths from client-side fallback forwards.

9. Automated CI/CD Contract Testing

To ensure API governance remains airtight across community contributions, OpenCrawling enforces documentation assertions via reflection tests in OpenApiDocumentationTest.java: