Embedded Web Crawling, Politeness & Zero-Trust SSRF Defense: Announcing the Web Repository Connector

1. The Right Tool for Web Ingestion in Enterprise AI

Retrieval-Augmented Generation (RAG) and semantic search architectures frequently need to ingest web-accessible content: technical documentation portals (MkDocs, Docusaurus, Sphinx), internal developer hubs, Confluence pages, regulatory filings, and corporate intranets.

While OpenCrawling already provides a distributed web crawler through our oc-stormcrawler-repository-connector (running on Apache Storm clusters for web-scale crawling across millions of URLs), many teams simply need to crawl a few thousand pages per job without provisioning, supervising, and maintaining an external distributed compute cluster.

The brand-new Web Repository Connector (oc-web-repository-connector) is designed specifically for this sweet spot: it runs completely embedded within the OpenCrawling runtime, requires zero external infrastructure, and leverages Java 25 Virtual Threads to achieve high concurrency while strictly adhering to web etiquette and enterprise security policies.

2. RFC 9309 Robots.txt & Automatic Sitemap Discovery

Web etiquette is non-negotiable for enterprise crawlers. The connector integrates crawler-commons 1.5 to enforce full compliance with RFC 9309 (the standard for robots exclusion):

• Robots.txt Evaluation: Fetches and parses robots.txt per host, accurately evaluating User-agent:, Disallow:, and Allow: rules before requesting any page.
• Crawl-delay Respect: Honors Crawl-delay directives issued by webmasters, dynamically raising the request interval up to a configurable ceiling (maxCrawlDelayMs).
• Sitemap Ingestion: Automatically extracts Sitemap: lines declared inside robots.txt, and allows operators to supply explicit sitemap URLs (sitemapUrls).
• Page-Level Directives: Honors <meta name="robots" content="noindex, nofollow"> and HTTP X-Robots-Tag headers, preventing indexing of unindexed pages while respecting link traversal limits.

3. Runtime-Wide Politeness: The Shared HostRegistry

In enterprise deployments where multiple crawl jobs might run concurrently (or alongside other automated tasks), multiple workers targeting the same domain could accidentally trigger rate limiters or cause denial of service.

The Web Repository Connector solves this through a runtime-wide singleton HostRegistry. Every active crawl job in the JVM shares this registry. When any job requests a URL on docs.example.com, the host gate ensures that at least delayMs (default 1000ms, or the host's Crawl-delay) has elapsed since the last request to that same host, across all jobs.

4. Spider Trap Prevention & Linear-Time Filtering

Web crawlers inevitably encounter "spider traps": dynamically generated calendar paths, infinite redirect chains, session-ID loops, or deeply nested directories. The connector incorporates defensive guards:

• Depth & Hop Limits: Restricts link exploration to maxDepth link hops from the seed (default 3, configurable from 0 to 50).
• Trap Caps: Rejects URLs exceeding maxUrlLength (2048 characters), query parameter counts exceeding maxQueryParams (32), or repeated directory segments (maxRepeatedSegments: 3).
• ReDoS-Free Regex Filtering: URL inclusion (includePatterns) and exclusion (excludePatterns) are compiled with Google's RE2/J engine, guaranteeing linear-time execution and safeguarding the runtime against catastrophic backtracking attacks.
• Query Parameter Normalization: Automatically strips tracking tokens (utm_*, fbclid, gclid, msclkid, session IDs like jsessionid, phpsessid, sid) and sorts remaining query parameters for canonical consistency.
• Content Deduplication: Follows rel="canonical" link elements and calculates cryptographic SHA-256 body hashes to skip duplicate pages.

5. Enterprise SSRF Guard with DNS Rebinding Protection

Allowing a web crawler to fetch arbitrary URLs in an enterprise environment presents serious security risks: malicious users or manipulated links could trick the crawler into hitting internal network services (Server-Side Request Forgery - SSRF), such as Kubernetes control planes, internal databases, or cloud instance metadata endpoints (e.g., AWS/GCP 169.254.169.254).

The Web Repository Connector implements a deep defense layer via SsrfGuard:

Crucially, SsrfGuard validates target IP addresses at DNS resolution time using a custom Apache HttpClient 5 DnsResolver. It inspects every resolved IP before establishing a TCP socket connection, and repeats this check on every redirect hop, effectively neutralizing DNS rebinding attacks.

For crawling internal corporate intranets, operators can set allowPrivateNetworks: true accompanied by an explicit privateAllowlist specifying approved hostnames (e.g. intranet.corp.internal, .corp.local) or CIDR blocks (e.g. 10.50.0.0/16).

6. Zero-Trust Security & Reactive Backpressure

Like all OpenCrawling connectors, the Web Repository Connector natively participates in the Zero-Trust security model:

• ACL Modes: Supports PUBLIC (default for public documentation) and STATIC mode (attaching user or group read keys such as group:engineering-staff to all documents).
• Accidental Exposure Protection: When allowPrivateNetworks: true is enabled, setting aclMode: PUBLIC fails validation unless the administrator explicitly provides confirmPublicExposure: true, preventing accidental public exposure of internal intranet documents.
• Reactive Backpressure: Emits a reactive Flux<RepositoryDocument> backed by a thread-safe Semaphore demand controller. Virtual crawling threads automatically pause when downstream Tika extraction, Ollama embeddings, or vector database writes fall behind.

7. Configuration & Job Examples

Settings can be specified at the job level (in crawlSpec JSON), at the connector level, or as Spring properties:

Example Job: Public Documentation Crawl

{
  "name": "Product Documentation Crawl",
  "repositoryConnector": "WebCrawler",
  "outputConnector": "PGVector_Output",
  "path": "https://docs.opencrawling.org/",
  "crawlSpec": {
    "maxDepth": 4,
    "maxPages": 5000,
    "delayMs": 1000,
    "scope": "PREFIX",
    "includePatterns": ["^/docs/"],
    "excludePatterns": ["\\.(png|jpe?g|gif|svg|webp|css|js)$"],
    "discoverSitemaps": true
  }
}

Example Job: Authenticated Intranet Portal Crawl

{
  "name": "Corporate Engineering Wiki",
  "repositoryConnector": "WebCrawler",
  "outputConnector": "PGVector_Output",
  "path": "https://wiki.corp.internal/",
  "crawlSpec": {
    "maxDepth": 5,
    "maxPages": 10000,
    "allowPrivateNetworks": true,
    "privateAllowlist": ["wiki.corp.internal", "10.50.0.0/16"],
    "aclMode": "STATIC",
    "aclIdentity": "group:engineering-staff",
    "maxThreads": 8
  }
}