pagedigest.json

One file that tells crawlers what changed.

A site publishes a JSON manifest at /.well-known/pagedigest.json. Each URL gets a monotonic integer that moves only when the content moves. A crawler fetches the manifest, compares integers, and fetches only the pages that changed — one request instead of thousands.

“If you would stop hammering my server for pages that haven’t changed, I wouldn’t have to rate-limit or ban you.”
The publisher
“If you would stop hiding behind bot detection, I wouldn’t have to hammer your server — I just need to know what’s new.”
The crawler

They are describing the same waste. pagedigest is the coordination.

Two layers of cooperative automation

pagedigest prevents redundant fetching. Its sibling project dotrepo prevents redundant repository interpretation. Together, they are publisher-declared state for automated clients: fetch less, infer less, and preserve provenance.

PageDigest

One manifest tells automated clients what changed.

dotrepo

One trust-aware record tells automated clients what a repository is and how to use it.

Together

Fetch less. Re-interpret less. Preserve provenance.

The manifest

For a 10,000-page site that changes 20 pages a week, a consumer makes one manifest request plus twenty page fetches per cycle — instead of ten thousand per-URL checks. site_rev is the one-request fast path: if it hasn’t moved, nothing has.

{
  "version": 1,
  "generated": "YYYY-MM-DDThh:mm:ssZ",
  "site_rev": 18294,
  "entries": {
    "/": { "rev": 47 },
    "/about": { "rev": 12 },
    "/blog/hello-world": {
      "rev": 4,   // was 3 — this is the only page to re-fetch
      "digest": "sha256:2cf24dba5fb0a30e26e83b2ac5b9e29e…"
    }
  }
}

The optional digest lets consumers spot-audit publisher claims: publishers who lie get caught, publishers who cooperate accumulate trust.

How a consumer uses it

  1. Fetch the manifest

    One GET to /.well-known/pagedigest.json. If it’s missing or malformed, fall back to whatever you did before — the protocol never makes things worse.

  2. Compare site_rev

    Same integer as last visit? Nothing on the site changed. The crawl cycle is over after a single request.

  3. Fetch only what moved

    For each URL whose rev incremented, re-fetch. Everything else is provably unchanged, on the publisher’s own word.

  4. Audit occasionally

    Hash a sampled page and compare it to the manifest’s digest. A mismatch means the manifest can’t be trusted — downgrade and fall back.

Why this is not another sitemap or cache

Sitemaps tell you what exists. ETags tell you, one request at a time, what changed. pagedigest tells you — in one request — what didn’t.

MechanismIts jobThe remaining gap
sitemap + lastmodDiscovery and advisory timestampsNo monotonic site-wide fast path or audit model
ETag / 304Per-resource validationStill one request per URL
RSS / AtomRecent-entry feedCannot establish that older or omitted pages stayed unchanged
IndexNowPublisher push to participating search enginesNot a pullable manifest for arbitrary consumers
WebSubHub-mediated pushRequires subscription and callback state
CDN / cacheMake repeated serving cheaperStill serves or validates the read

After checking the manifest, a consumer can make cooperation visible in logs:

PageDigest-State: site_rev=18294

The header is corroborating evidence, not authentication. Publishers compare it with manifest access and unchanged-page overfetch before changing treatment.

Live in production

First production publisher

dotrepo.org — a public index of repository metadata for AI agents — publishes a pagedigest manifest covering its full corpus: 613 repositories and 3,067 JSON payloads in its current checked-in snapshot. Mirrors and agent caches sync the whole index with one manifest request plus only the files whose revision moved.

curl -s https://dotrepo.org/.well-known/pagedigest.json | jq .site_rev
This site, too

pagedigest.org publishes its own manifest at /.well-known/pagedigest.json, generated by the reference generator at build time.

Observable cooperation

The deployed Cloudflare Pages Worker logs incoming PageDigest-State values and exposes a tiny same-origin counter at /__pagedigest/cooperation.json.

Checking observer status…

The bargain

pagedigest is a coordination protocol, not a defense mechanism. Each side takes on an obligation that is already in its own interest.

Publishers commit

  • The integers move when content moves — and only then.
  • Digests are accurate hashes of what the server actually sends.
  • A publisher with an honest manifest has earned the right to rate-limit consumers that ignore it.

Consumers commit

  • Fetch the manifest first; skip what hasn’t moved.
  • Audit digests occasionally to keep publishers honest.
  • A consumer that respects the manifest is not the problem — that’s the intended use.

Status

v1.0

PageDigest version 1 is final. The RC path is archived in the release-candidate announcement.

Version 1 wire format

Field names, semantics, and the /.well-known/pagedigest.json location are stable for v1.0.

Discovery

Publishers advertise the manifest with Link: </.well-known/pagedigest.json>; rel="https://pagedigest.org/rel" on ordinary responses.

Reference implementations

A Rust generator and a Python consumer library exist today, with a conformance vector suite. Source is on GitHub.

Get involved

Crawler operators and publishers can report adoption notes in the adoption feedback issue or email hello [at] pagedigest.org.