Wanlin Fire / AI Sources / Delivery

Machine Delivery and Crawl Budget Center

Efficient clients check the release and delta indexes before downloading large catalogs. Static datasets support gzip, ETag and Last-Modified validators so unchanged content can return 304 without retransmitting the body.

Datasets: 67 · uncompressed total: 21.2 MiB · large datasets: 11

Start here

release.json → machine-index.json → task-specific data.

Incremental sync

delta-index.json explains change-first synchronization.

Fetch tiers

fetch-plan.json separates discovery, incremental, task-specific and full-corpus requests.

Delivery health

delivery-health.json identifies large representations without treating size as authority.

Recommended client behavior

  1. Send Accept-Encoding: gzip.
  2. Save the ETag or Last-Modified value.
  3. Use conditional GET on the next scheduled check.
  4. Fetch full corpora only for initial indexing or when their SHA-256 changes.
  5. Respect 429 and Retry-After; keep parallelism low.