GitHub
Architecture

How certstream works

The path a certificate takes, from a Certificate Transparency log to a connected client.

Overview

End-to-end data flow

From CT logs to WebSocket and SSE clients.

flowchart LR CT["RFC 6962\nCT Logs"] --> W["RFC 6962\nWatchers"] SCT["Static CT Logs\n(Willow, Sycamore)"] --> SW["Static CT\nWatchers"] W --> P["Parser"] SW --> P P --> D["Dedup Filter"] D --> S["Pre-Serialize\n(simd-json)"] S --> B["Arc<PreSerializedMessage>\nBroadcast"] B --> WS["WebSocket"] B --> SSE["SSE"]

Serialization and shared state

Every certificate is serialized once using simd-json (SIMD-accelerated, enabled by default) into three Bytes payloads (full, lite, domains_only), which are then wrapped in an Arc<PreSerializedMessage>. Each subscriber receives a clone of the Arc and a Utf8Bytes text frame that borrows the same buffer, so the payload is not re-serialized or re-encoded per client. When no clients are connected, the serialization step is skipped by a receiver_count() == 0 guard. Shared state uses DashMap, so there are no global read/write mutexes in the hot path. CPU-heavy parsing runs on the blocking pool. The allocator is jemalloc.

Ingest · RFC 6962

CT log polling

How certificates are fetched from classic Certificate Transparency logs.

  1. Fetch the log lists

    On startup the server fetches the Chrome-trusted log list from Google and the Apple log list in parallel, filters out rejected and retired logs, dedupes by log ID, and merges in any custom logs from config.

  2. Spawn watchers

    Each CT log gets its own async task (tokio::spawn). Tasks run independently. If a state file exists, each watcher resumes from its saved position.

  3. Poll tree size

    The watcher calls /ct/v1/get-sth to read the current tree size and compares it with the tracked position to find new entries.

  4. Fetch entries

    Entries are fetched via /ct/v1/get-entries?start=X&end=Y, with up to fetch_concurrency windows pipelined per watcher during catch-up. The requested window (default: 1024) adapts to whatever page size the server actually serves, and the index only advances by the number of entries actually returned. Each request pays a token to the per-operator rate-limit bucket, so pipelining does not raise the sustained request rate.

  5. Track health

    Consecutive failures move a log through Healthy → Degraded → Unhealthy. Unhealthy logs pause behind a circuit breaker with exponential backoff.

  6. Save state

    After each batch the position is recorded so the server can resume after a restart.

Ingest · static-CT-API

Static CT protocol

Checkpoint and tile-based fetching for newer CT logs.

Why static CT?

Let's Encrypt has retired its RFC 6962 logs in favor of static, tile-based logs, where the tree is served as immutable tiles instead of dynamic get-entries calls. Chrome has accepted static-ct-api logs since April 2025, and the format is used by many newer logs.

  1. Fetch checkpoint

    The watcher polls /checkpoint for the current tree size. Checkpoints are signed text files containing the origin, tree size, and root hash. The log's ECDSA P-256 signature is verified against the key from the signed catalog. In warn mode, the default, signature failures are counted; in enforce mode, checkpoints with an invalid signature are rejected.

  2. Calculate tile range

    Each tile holds 256 entries. The watcher computes which tiles to fetch from the current index and the tree size, including validation of the partial tile width at the end of the tree.

  3. Fetch tile data

    Tiles are downloaded from /tile/data/<path> using hierarchical path encoding, for example x001/234 for tile 1234. Tiles may be gzip-compressed. A hard limit is applied to the decompressed size.

  4. Parse binary entries

    The binary parser extracts the timestamp, entry type (x509 or precert), DER certificate, and chain fingerprints from each entry in the tile.

  5. Fetch issuer certificates

    Chain certificates are referenced by SHA-256 fingerprint and fetched from /issuer/<hex>. They are stored pre-parsed in a single issuer cache shared by all watchers, so every leaf that chains to the same intermediate reuses one parsed Arc<ChainCert> instead of parsing the DER again. Issuer blobs that cannot be parsed are negative-cached so they are fetched at most once.

  6. Dedup and broadcast

    Certificates pass through the cross-log deduplication filter before being serialized and broadcast to clients.

Pipeline

Cross-log deduplication

The same certificate can appear in more than one CT log.

flowchart LR A["Certificate"] --> B["SHA-256"] B --> C{"Dedup Filter"} C -->|New| D["Broadcast"] C -->|Duplicate| E["Discard"]

How it works

The deduplication filter uses a DashMap<[u8; 32], Instant> keyed by the raw 32-byte SHA-256 digest stored in LeafCert::sha256_raw. Using the fixed-size digest as the key avoids the heap allocation a String key would need on every lookup. The map is hashed with ahash, since the digest is already uniformly distributed. The first occurrence of a certificate passes through and duplicates received within the TTL window are discarded. The default TTL is 900 seconds (15 minutes) and the default capacity is 200,000 entries; both are configurable. A background task removes expired entries periodically rather than clearing the cache when it reaches capacity.

Pipeline

Certificate parsing

X.509 decoding, from a CT entry to the fields the server exposes.

flowchart LR A["CT Entry"] --> B["Base64"] B --> C["MerkleTreeLeaf"] C --> D["x509_parser"] D --> E["Extract"]

MerkleTreeLeaf structure (RFC 6962)

Byte 0 Version
Byte 1 LeafType
Bytes 2-9 Timestamp
Bytes 10-11 EntryType (0 = X509, 1 = Precert)
Bytes 12-14 Certificate length
Byte 15+ DER certificate

Precert extra_data (RFC 6962)

3 bytes pre-certificate length
variable pre-certificate (X509 with CT poison extension)
3 bytes chain length
variable certificate chain

Extracted fields

Subject / Issuer: CN, O, C, L, ST, OU, Email
Hashes: SHA1, SHA256, fingerprint
Validity: not_before, not_after
Extensions: SubjectAltName, KeyUsage, BasicConstraints
Domains: collected from CN and SAN DNS entries

Pipeline

Pre-serialization

Messages are serialized once before being broadcast.

flowchart LR A["Message"] --> B["serialize()"] B --> C["Arc"] C --> D["broadcast"] D --> E["Clients"]

Serializing once per certificate

Rather than serializing a message separately for each connected client, certstream serializes each certificate into three byte payloads (full, lite, domains_only), wraps them in an Arc<PreSerializedMessage>, and gives each subscriber a clone of the Arc. The number of serializations per certificate does not change with the number of connected clients. A slow client does not block the rest of the stream: a client that falls behind misses messages, and one that falls too far behind is disconnected.

Output

Stream formats

Three payload formats are available.

full

/full-stream

Complete certificate data, including the chain and the DER-encoded certificate.

~15-50 KB / message

lite

/ (default)

Certificate metadata without the chain or DER certificate. This is the default format.

~2-5 KB / message

domains_only

/domains-only

The array of domain names only.

~100-500 B / message
Durability

State persistence

Resume from the last position after a restart.

flowchart LR A["StateManager"] --> B["DashMap"] B --> C["state.json"] C --> D["Resume"]

State structure

Each CT log's state tracks current_index (last processed entry), tree_size (last known tree size), and last_success (timestamp used for health tracking). State is written to a JSON file every 30 seconds and during shutdown. Positions are tracked for both RFC 6962 and static CT watchers.

Dirty flag

The dirty flag uses an AtomicBool rather than a lock. State is flushed on graceful shutdown (SIGINT / SIGTERM) and when the periodic save task is cancelled. Persistence is enabled by default with state_file: "certstream_state.json".