High-Speed Binary Payload Streaming & Zero-Downtime Resilience: Announcing gRPC Internal Transport in OpenCrawling

01. The Challenge: HTTP/1.1 JSON Overhead at Enterprise Scale

In enterprise RAG and vector ingestion pipelines, processing millions of document chunks per hour demands extreme network throughput and low CPU overhead. While standard HTTP/1.1 REST/JSON APIs are simple for single-node development, transporting heavy document payloads (containing raw extracted text, rich metadata maps, and high-dimensional floating-point vector arrays) over JSON introduces significant bottlenecks:

Verbose JSON Serialization: Repeated string keys and ASCII float formatting inflate payload sizes by 2x–3x compared to binary streams.
CPU & Garbage Collection Pressure: Constant JSON parsing and object allocations stress JVM heaps and trigger frequent GC pauses.
Connection Pooling Bottlenecks: HTTP/1.1 requires managing multiple TCP sockets, incurring setup handshakes and head-of-line blocking.

With the new oc-grpc-api module, OpenCrawling transitions internal microservice transport to binary Protobuf streaming over HTTP/2, reducing network bandwidth by up to 50% while cutting transport CPU overhead in half.

⚡

Up to 50% Bandwidth Compression: Protocol Buffers encode floating-point vectors and document metadata into dense binary wire formats, eliminating JSON key repetition and string parsing overhead across internal nodes.

02. Architecture & Operational Resilience Modes

To ensure zero-downtime upgrades and guarantee that ingestion jobs never fail due to unexpected network configuration issues, gRPC in OpenCrawling operates alongside an automatic HTTP/REST Fallback Engine:

OpenCrawling Internal Transport Architecture

Repository Connector

→

Internal Transport Manager

→

gRPC Channel (Port 9095)

→

Fallback: HTTP/REST

→

Worker / Vector Store

Supported Protocol Modes

Mode	Protocol Behaviour	Recommended Use Case
`AUTO` (Recommended)	Attempts high-speed gRPC binary streaming over port `9095`. If the gRPC channel fails or times out, it automatically falls back to HTTP/REST transport without losing data.	Production enterprise clusters, hybrid cloud setups, zero-downtime upgrades
`GRPC` (Strict)	Enforces strict Protobuf gRPC streaming across all internal processing nodes. Payload delivery fails if gRPC is unreachable.	High-throughput dedicated clusters with verified gRPC networking
`REST` (Standard)	Disables internal gRPC listener and routes all internal payload traffic through standard HTTP/REST JSON endpoints.	Single-node local development, restricted network environments

03. Strongly Typed Protobuf Contract (`oc-grpc-api`)

The internal communication contract is compiled from Google Protocol Buffers definitions inside the oc-grpc-api module:

syntax = "proto3";

package opencrawling.internal.v1;

option java_multiple_files = true;
option java_package = "org.opencrawling.internal.v1";

service InternalPayloadService {
  // Bi-directional document streaming
  rpc StreamDocumentPayloads (stream DocumentPayloadRequest) returns (stream PayloadIngestionResponse);

  // Single document payload ingestion
  rpc SendDocumentPayload (DocumentPayloadRequest) returns (PayloadIngestionResponse);

  // Live diagnostic health check probe for Admin UI & CLI
  rpc PingTransport (PingRequest) returns (PingResponse);
}

message DocumentPayloadRequest {
  string task_id = 1;
  string repository_id = 2;
  string document_id = 3;
  bytes raw_content = 4;
  map<string, string> metadata = 5;
  repeated string security_acls = 6;
  int64 timestamp = 7;
}

message PayloadIngestionResponse {
  string document_id = 1;
  enum Status { SUCCESS = 0; PROCESSING = 1; FAILED = 2; }
  Status status = 2;
  string error_message = 3;
}

04. Enterprise TLS / mTLS Security

For secure enterprise deployments, internal gRPC communication can be encrypted using TLS / mTLS directly in application.yml or via the Admin UI:

Server Certificate Chain (certChainPath): Absolute path to .crt or .pem file.
Private Key (privateKeyPath): Absolute path to .key file.
Graceful Fallback: If certificates are unreadable or missing, InternalTransportManager logs a security warning and safely falls back to standard HTTP/REST or plaintext mode according to your resilience settings.

05. Admin UI Hot-Reload & Live Diagnostic Probes

Administrators can manage gRPC transport from **Section 4: Internal Communication & Transport (gRPC / REST)** in the Admin UI:

Mode Selector Cards: Interactively toggle active transport between AUTO, GRPC, and REST.
Hot Reloading: Updating transport settings dynamically restarts the underlying gRPC Netty server lifecycle without restarting the Spring Boot application or container.
Diagnostic Connection Probe: Click "Test gRPC Connectivity" to fire a live PingTransport RPC against host 127.0.0.1:9095, measuring latency in milliseconds before applying changes.

🧪

Live Test Endpoint: Test gRPC channel health programmatically at any time via cURL:
curl -X POST "http://localhost:8080/api/v1/admin/settings/transport/test-grpc" -H "Content-Type: application/json" -d '{"host":"127.0.0.1","port":9095}'

06. Configuration Parameters & Quickstart

Spring Property Key	Environment Variable	Default	Description
`opencrawling.transport.mode`	`OPENCRAWLING_TRANSPORT_MODE`	`AUTO`	Active protocol mode (`AUTO`, `GRPC`, `REST`)
`opencrawling.transport.grpc.port`	`OPENCRAWLING_TRANSPORT_GRPC_PORT`	`9095`	gRPC Netty server listener port
`opencrawling.transport.grpc.enabled`	`OPENCRAWLING_TRANSPORT_GRPC_ENABLED`	`true`	Enables or disables gRPC server lifecycle
`opencrawling.transport.grpc.fallback-to-rest`	`OPENCRAWLING_TRANSPORT_GRPC_FALLBACK_TO_REST`	`true`	Enables auto-fallback to REST on channel error
`opencrawling.transport.grpc.max-message-size-mb`	`OPENCRAWLING_TRANSPORT_GRPC_MAX_MESSAGE_SIZE_MB`	`32`	Maximum inbound gRPC payload size in MB

Updated Docker & Docker Compose Support

All 6 multi-stage Maven build Dockerfiles (`oc-runtime/Dockerfile`, `docker/Dockerfile.*`) and all 11 Docker Compose deployment overlays (`docker-compose-apps.yml`, `docker-compose-decoupled.yml`, `oc-*-connector/docker/docker-compose-decoupled-with-*.yml`) have been updated to include `oc-grpc-api` and publish port 9095:9095.

07. Summary & Next Steps

The introduction of gRPC Internal Transport reinforces OpenCrawling's commitment to delivering enterprise-grade performance, resilience, and vendor-neutral open standards for modern data ingestion.