v1.0
The wire format is final. Generator and launcher 0.3.0, plus Python and Astro 0.2.0, ship the consumer safety release. See the migration notes and verification. The RC path is archived in the release-candidate announcement.
PageDigest is a JSON change manifest for public websites. Publishers list resource revisions; recurring crawlers, indexers, and caches compare them with local state and avoid refetching resources they already have.
Designed for mostly static content and consumers that retain results between runs. Optional SHA-256 digests support spot checks of published representations.
For an established cache covering 10,000 pages with 20 changes per cycle, the ideal
count is one manifest plus twenty page fetches. A 1% audit adds roughly 100 requests.
Initial synchronization and missing local results require additional fetches.
site_rev is the one-request
fast path: an unchanged value permits reuse of complete local results for covered resources.
{
"version": 1,
"generated": "YYYY-MM-DDThh:mm:ssZ",
"site_rev": 18294,
"entries": {
"/": { "rev": 47 },
"/about": { "rev": 12 },
"/blog/hello-world": {
"rev": 4,
"digest": "sha256:2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824"
}
}
}
Illustrative timestamp placeholder; use an actual UTC timestamp when publishing. The optional digest lets consumers spot-check selected representations. A matching sample does not prove correct revision history or every skipped resource.
One GET to /.well-known/pagedigest.json. If it’s missing or
malformed, fall back to ordinary crawling. Missing local results still require work.
site_rev
The same integer describes the covered set, not local cache completeness. Reuse only results that remain available at their recorded revisions.
Fetch new, changed, missing, or invalid local results. An unchanged revision is a publisher assertion; unlisted resources retain ordinary discovery and validation.
Hash a sampled page and compare it to the manifest’s digest.
Compare with stored representation evidence too. Retry inconclusive checks and
deployment races; refresh or fall back when evidence conflicts. Audit even on no-change cycles.
Batching accurate change information avoids per-resource checks. PageDigest chooses a small JSON format with ordered revisions; maintaining those counters requires durable publisher state.
| Mechanism | Its job | The remaining gap |
|---|---|---|
| sitemap + lastmod | Discovery and incremental refresh timestamps | No monotonic site-wide fast path or audit model |
| ResourceSync | Lists, changes, hashes, and synchronization discovery | PageDigest offers a smaller JSON surface with fewer capabilities |
| Fingerprint manifest | Bulk byte comparison and manifest ETags | Counters add ordering and optional semantic separation, with durable-history cost |
| ETag / 304 | Per-resource validation | Still one request per URL |
| RSS / Atom | Recent-entry feed | Cannot establish that older or omitted pages stayed unchanged |
| IndexNow | Publisher push to participating search engines | Not a pullable manifest for arbitrary consumers |
| WebSub | Hub-mediated push | Requires subscription and callback state |
| CDN / cache | Make repeated serving cheaper | Still serves or validates the read |
After checking the manifest, a consumer can make cooperation visible in logs:
PageDigest-State: site_rev=18294
The header is corroborating evidence, not authentication. Publishers compare it with manifest access and unchanged-page overfetch before changing treatment.
Seven consumer strategies across six scenarios reproduced the same current resource set and bytes before savings were counted. For 100 unchanged pages, an accurate sitemap or fingerprint manifest needed one request. With the same 1% audit policy, both the fingerprint and PageDigest strategies needed two.
The persistent consumer also read and hashed all 100 cached bodies. The comparison counts that local work separately from network traffic. These are controlled fixture results; independent recurring-consumer evidence remains open.
The dotrepo case study covers 2,453 repository JSON records. It estimates 136 requests including the manifest, versus 2,453 full record fetches, before audits. Estimated payload-byte reduction is 70.8% including the manifest. These are export-derived figures, not an independently measured production consumer or model-token savings. The live manifest may now describe a different snapshot.
curl -s https://dotrepo.org/.well-known/pagedigest.json | jq .site_rev
pagedigest.org publishes its own manifest at
/.well-known/pagedigest.json,
generated by the reference generator at build time.
The deployed Cloudflare Pages Worker logs incoming
PageDigest-State values and exposes a tiny same-origin
counter at /__pagedigest/cooperation.json.
Checking observer status…
Publishers preserve revision history; consumers preserve usable results. A skipped download must not hide missing content, interrupted processing, or links to changed pages.
The reference generators compare full build bytes. Semantic revisions require a publisher integration supplying that signal. Digest sampling checks representations, not every historical claim.
Controlled benchmark and limitations: equivalent results are checked before counting requests. Independent recurring-consumer evidence remains the next adoption milestone.
The wire format is final. Generator and launcher 0.3.0, plus Python and Astro 0.2.0, ship the consumer safety release. See the migration notes and verification. The RC path is archived in the release-candidate announcement.
Field names, semantics, and the
/.well-known/pagedigest.json location are stable for v1.0.
Publishers advertise the manifest with
Link: </.well-known/pagedigest.json>; rel="https://pagedigest.org/rel"
on ordinary responses.
A Rust generator and a Python consumer library exist today, with a conformance vector suite. Source is on GitHub.
Crawler operators and publishers can report adoption notes in the adoption feedback issue or email hello [at] pagedigest.org.