The CT logs that serve tiles but publish no checkpoint
Three logs, zero errors, 279,000 entries behind. The fast way to read them was sitting right there, behind a missing file.
A Certificate Transparency log is an append-only record of every TLS certificate a public CA issues, and anyone can read one. My server reads all 45 that Chrome and Apple trust. Three of them had been falling behind for weeks: not failing, not rate limited, not erroring, just slowly and steadily losing ground.
TrustAsia 'log2026b' 278,890 entries behind errors: 0
TrustAsia 'log2026a' 79,392 entries behind errors: 0
TrustAsia 'HETU2027' 37,898 entries behind errors: 0Every request was succeeding. The health check was green. The watcher was simply not able to read entries as fast as the log was writing them.
Nineteen of forty five
Before getting to those three, the wider number is worth sitting with. Over a two hour run against every Chrome- and Apple-trusted log:
| Across 45 logs | Count |
|---|---|
| Reporting healthy | 45 |
| Healthy and more than 5,000 entries behind | 19 |
| Furthest behind, in entries | 1,297,126 |
Every one of those 19 was green. Requests were succeeding, responses were valid, error counters were at zero. Nearly half the logs I was monitoring were quietly falling behind, and the log furthest adrift was more than a million entries back, which at its fetch rate meant it would never return.
That is because the status field answers "are requests to this log succeeding" and has no opinion on whether you are keeping up. The two questions feel like the same question right until they are not. A watcher that fetches successfully at half the rate the log is being written stays green forever while the gap grows every minute.
I fixed the monitoring gap first: lag is now a metric rather than a subtraction you perform on an API response by hand. But knowing which logs are behind does not tell you why, and the three worst offenders had something in common.
Two ways to read a CT log
The classic interface, from RFC 6962, is get-entries: you ask for a range of indices and the log returns JSON. It is an application endpoint. The log has to serve it from something that knows how to answer arbitrary ranges.
The newer interface, static-ct-api, is a set of static files. Entries live in data tiles of 256 at fixed paths, and a signed checkpoint file names the current tree size. Nothing needs to be computed per request, so the whole thing sits behind a CDN.
These three logs speak RFC 6962. They are listed as RFC 6962 in the log lists. But they also serve tile data, which I only knew because Effy Elden filed an issue pointing it out, having traced it through a ct-policy discussion where the operator confirmed the tiles are deliberate.
So I measured both paths against the same 256 entries, near the head, on the same connection:
| Path | Transferred | Time |
|---|---|---|
/tile/data/x010/x076/918 | 189 KB | 5.96s cold, 0.36s warm |
get-entries?start=…&end=… | 696 KB | 30.4s / 30.7s / 25.2s |
Same data. One is roughly seventy times faster once the CDN has it, and transfers a quarter as much because the tiles are served compressed. Three separate get-entries requests took 25 to 31 seconds each, so this is not a cold-start artifact.
The same operator runs a fourth log that does publish a checkpoint, and my server was already reading that one over tiles. It sat 1,310 entries behind while its three siblings were 38,000 to 279,000 behind. Same operator, same network path, same client. The only difference was which interface I was reading.
Why the tile path was not simply available
Because these logs return 404 for /checkpoint, and every tile client starts by reading the checkpoint to learn the tree size. My static-CT watcher hard-required it: fetch the checkpoint, verify its signature, take the tree size from it, then fetch tiles up to that point. No checkpoint meant no tree size, and the watcher exited after its retries.
The operator has said they will not serve one. They publish the tree head through get-sth, signed, the RFC 6962 way.
So the shape of the fix is obvious once you see it: take the tree size from get-sth, take the entries from the tiles. Nothing else about the tile path cares where the number came from.
static_logs:
- name: "TrustAsia log2026a"
url: "https://ct2026-a.trustasia.com/log2026a"
expected_log_id: "dNudWPfUfp39eHoWKpkcGM9pjafHKZGMmhiwRQ26RLw="
tree_size_source: get_sthThree things the real logs taught me that the tests did not
Every unit test passed before I pointed this at production logs. Then it met the internet.
Load balanced heads disagree
Within a minute of starting, one watcher stopped advancing entirely. The tree size it read was lower than the tree size it had read a moment earlier.
A static-CT watcher treats that as an integrity signal, and rightly: a signed checkpoint whose tree size regresses means an operator bug, an incomplete replica, or something worse. My code penalised the log's health and backed off, which is correct behaviour for a checkpoint.
It is the wrong behaviour for get-sth. That endpoint is served by a fleet, replicas are a few entries apart at the head, and reading a slightly older head is completely routine. The guard was firing on normal operation and pushing a perfectly healthy watcher into exponential backoff.
The fix counts the regression as a metric and re-polls, instead of penalising health. Progress is still bounded by the check that never reads tiles past the head the log reported.
High water marks latch onto the fastest replica
The same guard keeps a high water mark of the largest tree size ever seen. With a fleet behind the endpoint, that mark converges on whichever replica happens to be furthest ahead, and every subsequent read from any other replica looks like a regression. A watcher can sit still waiting for the rest of the fleet to catch up to a number one machine reported once.
Partial tiles are not owed to you
The spec requires partial tiles (.p/<W>, the incomplete tile at the head) only for tree sizes that a checkpoint was published for. A log that publishes no checkpoint owes none, and these do not serve them. Ask for one and you get a 404.
So the head has to be floored to the last complete tile, which means the newest 0 to 255 entries wait for their tile to fill. For a firehose that is a second or two of extra latency. For a monitor watching one specific domain it is worth knowing about.
The part that is not a bug, just a number you have to pick
With all that working, one of the three logs still could not keep up. The tiles were arriving in groups of four with roughly twenty seconds between groups, which is exactly what four concurrent fetches look like when each one takes twenty seconds.
Tiles near the head are new. A CDN has not been asked for them yet, so they miss, and a miss goes to the origin. The 0.36 second number in the table is a cache hit, and during catch-up most requests are hits because you are reading tiles the rest of the world already asked for. At the head they are all misses.
Raising in-flight fetches from 4 to 16 took the three logs from about 50 entries per second in aggregate to 276.
| Log | Lag before | Lag after |
|---|---|---|
| TrustAsia 'HETU2027' | 37,898 | 0 |
| TrustAsia 'log2026a' | 79,392 | 0 |
| TrustAsia 'log2026b' | 278,890 | catching up |
What you give up
Verifiability, and it should be said plainly. There is no checkpoint, so there is no signed head to check the tiles against. Worse, you could not check them against the get-sth signature either: these tiles carry the static-CT leaf_index extension in their TimestampedEntry, so re-encoding a tile leaf does not reproduce the RFC 6962 Merkle leaf hash. The author of certspotter looked at the same logs and declined to support them for exactly this reason, which is the right call for a tool whose job is cryptographic monitoring.
My server broadcasts certificates to downstream consumers. It does not verify inclusion proofs and never claimed to. For that job, trading verifiability for data you can actually keep up with is a reasonable trade, as long as it is opt-in, logged loudly at startup, and documented as unverified. It is not a trade a monitor should make.
The general lesson
The failure modes in this work did not come from my code being wrong about its own logic. They came from operator infrastructure: a load balancer behind an endpoint that the protocol treats as authoritative, a CDN whose cache behaviour differs completely between catch-up and steady state, a rate limiter keyed on something I had not thought about.
None of them are expressible as a unit test until after you know they exist. If you are writing a client for someone else's distributed system, budget time for running it against the real thing and watching it for a while, because that is where those live. And when you build a guard against a protocol violation, ask which endpoint it is guarding: the same invariant that is sacred for a signed artifact can be routine noise one hop away.