Does HTTP/2 multiplexing hurt throughput against rate-limited servers?

It can, when the server limits per connection rather than per client, because HTTP/2 puts every concurrent request on one connection while HTTP/1.1 uses one per in-flight request. Whether it actually hurts depends on the server. In the case measured here the connection count dropped from three to one and throughput was unchanged at about 50 entries a second, because the constraint was the server's per-request work.

How do I check how many connections my HTTP client is opening?

Read /proc/net/tcp and /proc/net/tcp6 inside the process's network namespace and group by remote address. This works in a container without installing ss or netstat. Comparing the count before and after a transport change tells you whether the change took effect, though not whether it helped.

Why does a CT log return fewer entries than requested?

RFC 6962 permits returning fewer entries than the requested range, and operators use it to bound response size. The logs measured here returned between 29 and 125 entries for a 1024-entry request, varying between consecutive requests, which suggests a byte budget rather than a fixed entry cap.

How can I force HTTP/1.1 in reqwest?

Build a separate client with .http1_only(true) for the hosts that need it rather than disabling HTTP/2 process-wide, so you keep multiplexing everywhere else. Measure the outcome before assuming it helped.

HTTP/2 multiplexing vs per-connection rate limits: measured

Certificate Transparency logs are append-only records of every TLS certificate a public CA issues; monitors read them to spot certificates as they appear. Six logs from one operator had been falling behind for weeks in a server of mine that reads all of them. No errors, no rate limit responses, no failed requests. The largest of them sat 33,000 entries behind the head and pulled 3.7 entries a second, which meant it was never going to catch up.

There was a good explanation available, and I had already implemented it. It turned out to be wrong, and the way it was wrong is more interesting than if it had been right.

The explanation that fit

The client is reqwest, which negotiates HTTP/2 when the server offers it. Under HTTP/2 every request to a host is multiplexed onto a single TCP connection. That is the point of the protocol: one connection, many concurrent streams, no head-of-line blocking at the HTTP layer, no handshake per request.

Now suppose the server on the other end throttles per connection rather than per client. Every one of your concurrent requests is sharing one bucket, and your fetch concurrency setting buys you nothing at all. Worse, moving from HTTP/1.1 to HTTP/2 would have made you slower, because HTTP/1.1 opens a connection per in-flight request and each one carries its own quota.

This is not a hypothetical. The author of certspotter hit something like it with the same operator and dealt with it by dropping keep-alives outright. So the server grew an option to pin chosen operators to HTTP/1.1, and that option shipped.

What it did not have was a measurement.

Counting the connections

The first thing to check is whether the mechanism is even present. It is a socket count, and you can read it from inside the container:

$ docker exec certstream sh -c 'cat /proc/net/tcp /proc/net/tcp6' \
    | awk 'NR>1 {print $3}' | cut -d: -f1 | sort | uniq -c | sort -rn

Against the operator's addresses, with the default transport:

# HTTP/2
1 connection -> 54.71.133.191
1 connection -> 3.137.57.126

And with those operators pinned to HTTP/1.1:

# HTTP/1.1
3 connections -> 44.231.171.103
3 connections -> 3.130.169.13

Exactly as described. One connection carrying everything, versus one per in-flight fetch. If the server throttles per connection, this change should have tripled throughput.

What it actually did

Two runs, 25 minutes each, same machine, same 23 logs, same everything except the transport. Rate is entries actually fetched per second, with index jumps excluded (a 400 response makes the watcher skip to the tree head, and that is not fetched data).

Log	HTTP/2	HTTP/1.1
Sphinx2026h2	3.7	3.1
Wyvern2027h1	23.5	19.6
sphinx2027h1	21.0	20.2
Wyvern2026h2	0.8	6.1
Wyvern2027h2	0.0	0.6
sphinx2027h2	0.0	0.3
Total	49.0	50.0

The last two rows sit near zero in both runs because those logs were caught up and idle at the head; they are in the table because leaving them out would flatter whichever column you please.

Two percent, in favour of the change. For that to mean anything it has to sit outside the run-to-run variance, and it does not: sampling every 30 seconds, the per-interval rate for the busiest log moves by 15 to 20% either side of its mean as the log's own write rate fluctuates. A 2% difference in a 25 minute total is well inside that. Three times the connections, the same data.

The one row that looks like a win is not one. Wyvern2026h2 started that run 45,000 entries behind instead of 2,400, so it spent the whole window in catch-up while in the other run it was mostly idle at the head. Comparing a log against itself across runs only works when both runs start from a similar position, and that one did not.

Where the throughput was actually going

If the transport is not the constraint, ask what the server does with a single request. The watcher asks for 1024 entries at a time:

$ curl -s "https://.../ct/v1/get-entries?start=N&end=N+1023"

  attempt 1: 1.75s, 763 KB, 125 entries
  attempt 2: 1.68s, 457 KB,  77 entries
  attempt 3: 1.30s, 167 KB,  29 entries

RFC 6962 lets a log return fewer entries than asked for, and this one returns far fewer: between 29 and 125 against a request for 1024. It is also inconsistent from one request to the next, which rules out a fixed max_get_entries and points at something dynamic, probably a response size budget.

So each request costs roughly a second and a half and yields something like 80 entries. The ceiling that produces has nothing to do with how many TCP connections you opened; it is set by how much work the server is willing to do per request, multiplied by how many requests per second your own rate limiter permits. Opening three connections to ask three slow questions instead of one gets you three slow answers.

Why the hypothesis was attractive anyway

Every part of it was true except the part that mattered. reqwest really does multiplex onto one connection. The socket count really does drop to one. Servers really do sometimes throttle per connection. Another maintainer really did hit a related problem with this operator.

What was missing was any evidence that this operator was throttling this client per connection right now. The reasoning was sound and the conclusion did not follow, and that combination survives review: everyone checking it checks the mechanism rather than the outcome.

I have not removed the option. It is opt-in, it costs nothing when unused, and an operator behind a different edge, at a different time, from a different address may well see what the reasoning predicts. What has changed is the documentation, which no longer implies it will help.

What I would do differently

Measure the outcome, not the mechanism. Confirming that the connection count fell to one felt like confirming the theory, and it confirmed nothing except that HTTP/2 works as documented. The number that mattered was entries per second, and it took twenty five minutes per run to get it.

When a client is slow against someone else's server, count what the server gives you per request before you touch the transport. One curl would have shown me a 1024-entry request coming back with 29 entries, and I would have started somewhere else entirely.

And compare runs from the same starting position. The one row in my table that looked like a seven-fold improvement was a log that happened to be further behind, doing catch-up. If I had reported that row on its own it would have been a lie told with real data.

I blamed HTTP/2 multiplexing. Then I counted the connections.