Standardized Document Lifecycles & Deletion Tombstones in the Open Ingestion Standard (OIS)

01. The Enterprise Removal Problem

In modern enterprise RAG and search pipelines, data ingestion flows continuously from repositories like SharePoint, S3, CMIS servers, and web crawlers (such as Apache StormCrawler) into vector stores (Qdrant, Vespa, OpenSearch 3, Apache Solr 10, Milvus, pgvector).

While document ingestion (creation and modification) is straightforward, document deletion handling has historically suffered from fragmentations:

• Ghost Vectors & Hallucinations: When a source document is deleted, its vector embeddings and chunk payloads remain in destination indices unless explicitly removed.
• Ad-Hoc Metadata Hacks: Connectors were forced to invent proprietary flags (e.g., metadata["isDeleted"] = true), breaking vendor neutrality and requiring customized downstream parsing logic.
• Unnecessary Compute Overhead: Passing full deletion messages through chunking and LLM embedding stages wastes substantial CPU and GPU resources.

02. The OIS Lifecycle Specification: UPSERT vs DELETE

To solve this, the **Open Ingestion Standard (OIS)** document schema (schemas/document.schema.json) officially incorporates a top-level action property:

• "UPSERT" (default for backwards compatibility): Signals document creation or content modification. Triggers text extraction, token chunking, vector embedding generation, and index insertion/update.
• "DELETE": Represents a standardized deletion tombstone. Signals that the item identified by id has been removed from the source system and MUST be purged from target vector and search stores.

Conditional Payload Schema Validation

OIS 1.0 enforces conditional JSON schema validation (oneOf / if-then) for deletion tombstones:

• For action: "UPSERT": The fields id, source, content, metadata, and security are expected.
• For action: "DELETE": Only id, source, and action are mandatory. Text content (content) and Access Control Lists (security) become optional.

03. Payload Code Examples

Here is how standard document ingestion and deletion tombstone payloads are structured under the OIS standard:

A. Standard Ingestion Payload (UPSERT)

{
  "oisVersion": "1.0.0",
  "id": "https://example.com/docs/security-policy",
  "action": "UPSERT",
  "source": {
    "type": "stormcrawler",
    "instance": "https://example.com"
  },
  "content": {
    "mimeType": "text/html",
    "text": "Enterprise Security Policy plain text content..."
  },
  "metadata": {
    "title": "Enterprise Security Policy"
  },
  "security": {
    "inheritanceEnabled": false,
    "permissions": []
  }
}

B. Standardized Deletion Tombstone Payload (DELETE)

{
  "oisVersion": "1.0.0",
  "id": "https://example.com/docs/deprecated-policy",
  "action": "DELETE",
  "source": {
    "type": "stormcrawler",
    "instance": "https://example.com"
  },
  "metadata": {
    "stormcrawler.status": "DELETED",
    "http.status": "404",
    "deletedAt": "2026-09-03T12:00:00Z"
  }
}

04. End-to-End Pipeline Routing & Downstream Execution

In the reference implementation (OpenCrawling), receiving an OIS deletion tombstone activates an optimized fast-path pipeline execution:

1. Connector Event Capture: Web crawlers (e.g. StormCrawler status stream catching HTTP 404/410), CMIS event listeners, database CDC connectors, and RESTHeart change streams emit lightweight action: "DELETE" tombstones to Kafka.
2. OpenCrawling Core Routing (oc-core): The ingestion runtime recognizes the DELETE action and routes the tombstone directly to target output writers—bypassing text chunking and vector embedding workers completely.
3. Native Vector & Search Store Purge: Downstream output connectors execute native deletion calls:
   • oc-qdrant-output-connector → Executes qdrantClient.deletePoints(id).
   • oc-vespa-output-connector → Executes vespaFeedClient.remove(id).
   • oc-solr-output-connector → Executes solrClient.deleteById(id).
   • oc-opensearch3-output-connector → Executes openSearchClient.delete(id).
   • oc-milvus-output-connector → Executes milvusClient.delete(id).

05. Standard Roadmap & Community Participation

The OIS Document Lifecycle Deletion Tombstone specification is fully integrated into the schemas/document.schema.json formal specification and verified across all OpenCrawling reference output connectors.

We invite developers, enterprise search maintainers, and standard contributors to explore the schemas and contribute to standard discussions!