Introducing OpenCrawling Java Client SDK: Programmatic Ingestion Control for Java 25 & Spring Boot

01. Why a Dedicated Java SDK?

As OpenCrawling adoption expands across enterprise systems, integrating existing Spring Boot applications, Quarkus services, or custom backend jobs with OpenCrawling required manually constructing raw HTTP REST requests, serializing JSON payloads, and managing error handling manually.

The new oc-java-client-sdk eliminates this friction by delivering a strongly typed, fluent builder-pattern API built on modern Java 25. It leverages native java.net.http.HttpClient for lightweight, non-blocking I/O with Virtual Threads, requiring zero heavy third-party HTTP transport libraries.

02. Core Capabilities Overview

• Fluent Client API: Intuitive client builder supporting API key authentication, custom base URLs, timeouts, and headers.
• Job Lifecycle Control: Programmatically create, start, pause, stop, and delete ingestion jobs matching Open Ingestion Standard (OIS) models.
• Connector Administration: Dynamically register and configure repository scanning, vector store output, and AI transformation connectors.
• Auto-Narrativization Copilot Integration: Trigger AI narrative generation from schema field definitions using Spring AI, Ollama, or OpenAI.
• AIOps Observability & RCA: Query automated Root Cause Analysis (RCA) diagnostic reports and correlated OpenTelemetry spans programmatically.
• Spring Boot Autoconfiguration: Includes embedded Spring Boot starter support (`opencrawling.client.*`) for automatic OpenCrawlingClient bean injection.

03. Quick Start Guide

Add the dependency to your pom.xml:

<dependency>
    <groupId>org.opencrawling</groupId>
    <artifactId>oc-java-client-sdk</artifactId>
    <version>1.0.0</version>
</dependency>

Initializing the Client

import org.opencrawling.sdk.OpenCrawlingClient;
import java.time.Duration;

OpenCrawlingClient client = OpenCrawlingClient.builder()
        .baseUrl("http://localhost:8080")
        .apiKey("your-api-key")
        .connectTimeout(Duration.ofSeconds(10))
        .readTimeout(Duration.ofSeconds(30))
        .build();

Creating and Triggering an Ingestion Job

import org.opencrawling.sdk.models.JobRequest;
import org.opencrawling.sdk.models.JobResponse;
import org.opencrawling.sdk.models.NarrativizationConfig;

JobResponse job = client.jobs().create(
    JobRequest.builder()
        .name("Enterprise Documentation Crawler")
        .targetUrl("https://docs.example.com")
        .repositoryConnector("FileSystem_Local")
        .outputConnector("PGVector_Output")
        .transformationConnector("Ollama_Embedding_Default")
        .narrativization(NarrativizationConfig.builder()
            .enabled(true)
            .template("Document titled {{title}} with content: {{content}}")
            .build())
        .build()
);

// Trigger job execution
client.jobs().start(job.id());

04. Spring Boot Autoconfiguration

When running inside a Spring Boot application, simply configure properties in application.yml:

opencrawling:
  client:
    base-url: http://localhost:8080
    api-key: your-api-key
    connect-timeout: 10s
    read-timeout: 30s

The OpenCrawlingClient bean will be autoconfigured and ready for injection into your Spring services:

@Service
public class IngestionService {

    private final OpenCrawlingClient client;

    public IngestionService(OpenCrawlingClient client) {
        this.client = client;
    }

    public void triggerJob(String jobId) {
        client.jobs().start(jobId);
    }
}