Handing a request to a handler costs 39 % of a Ktor CIO service's CPU — and 1 % on Netty

A previous post measured what share of a Ktor service's profile user code owns and found 1–4 %. The biggest number in that profile belonged to nobody in the question: between a fifth and a third of /business CPU sat in the coroutine dispatcher's queue, and the verdict — "a property of CIO under this load" — was drawn from the only engine the stand had. This post is that comparison: the same service, the same jars, the same process on CIO, Netty and Jetty, one -Dbench.engine apart. The stand, the harness and every run summary are in zavarnik under bench/, the full write-up in docs/research/research-engines.md.

The stand

• Service: Ktor 3.5.2, kotlinx.serialization 1.11.0, kotlinx.coroutines 1.11.0, slf4j-simple. Three endpoints: /echo, /items (JSON CRUD over an in-memory store), /business (collection chains, a regex, three disabled logger.debug templates per request).
• Engines as they ship: CIO, Netty 4.2.16, Jetty 12.0.35 through ktor-server-jetty-jakarta. No engine configuration is touched, so connectionGroupSize = workerGroupSize = parallelism / 2 + 1 and callGroupSize = parallelism, the Ktor defaults.
• Load: oha, 64 keep-alive connections; JVM pinned to cores 0–7, generator to 8–15; -Xms1g -Xmx1g -XX:+UseG1GC.
• Profiler: async-profiler 4.5, 120 s of cpu then 120 s of alloc, after 60 s of warm-up and a 120 s profiler-free window that the throughput and cost numbers come from.
• Machine: Ubuntu 24.04 in WSL2, Core Ultra 7 255HX, OpenJDK 25.0.4.

One thing in that protocol used to be a lie. taskset -pc <pid> sets the affinity of a single thread: after it, 49 of the JVM's 50 threads still read Cpus_allowed_list: 0-19, and the process burned 12.6 cores while nominally capped at 8. Every run of the earlier post — including the configuration its text calls pinned — had the service and the load generator fighting over the same 20 cores. Pinning on the exec is inherited by every thread, and the JVM then sizes its pools to the cores it can see. The old protocol is kept as a control below, and it reproduces the published numbers.

What a request costs

/business, 64 connections, from the clean window without profilers. The absolute number is utime+stime of the process over that window divided by the requests served in it.

On /echo the same order holds at 125 / 56 / 88 µs, on /items at 142 / 88 / 120. All three respond with identical bytes — same status, same Content-Length, no chunked framing — so the difference is machinery, not payload. Netty's rps understates the gap: at 100 023 rps it used 5.6 of its 8 cores while the generator used 3.0 of its own, so its throughput there is bounded by the offered concurrency (64 connections at 0.58 ms is ~110 k rps by Little's law), not by CPU. CIO, at 7.2 cores of 8, is bounded by CPU.

Where the CPU goes

By owner — the first frame from the leaf that is neither JDK nor JVM — on /business:

The leaf frames say it more plainly. CIO: LockFreeTaskQueueCore.removeFirstOrNull 17.3 %, LimitedDispatcher.obtainTaskOrDeallocateWorker 6.9 %, LimitedDispatcher.dispatch 4.3 %. Netty: writev 6.9 %, __write 5.8 %, epoll_wait 5.6 %, and the first application frame is Pricing.quote at 1.6 %. Jetty: pthread_cond_signal 20.4 % plus libc 20.9 %, LockSupport.park and unpark at 2 % each.

Summed into one bucket — everything that moves a request from an I/O thread to a handler and back — that is 33–48 % of self CPU on CIO across the three endpoints, 25–30 % on Jetty and 1–2.5 % on Netty.

The reason is in the engines' source, not in the profile. CIOApplicationEngine runs both its engine and its user dispatcher on Dispatchers.IO, and Dispatchers.IO on the JVM is UnlimitedIoScheduler.limitedParallelism(max(64, availableProcessors)) — a LimitedDispatcher holding one LockFreeTaskQueue that up to 64 worker loops poll. Netty's NettyDispatcher returns false from isDispatchNeeded when the code already runs on the channel's event loop, so most calls are never handed anywhere; what remains goes to that channel's own executor. Jetty's JettyKtorHandler dispatches into its own ThreadPoolExecutor over a SynchronousQueue — a rendezvous, paid for in thread wake-ups rather than in polling.

So the queue is not the price of coroutines. It is the price of one shared MPMC queue with 64 workers contending for its head.

The lever

If that is the mechanism, cutting the worker count to the core count must cut the cost — and must do nothing to Netty, which does not route calls through Dispatchers.IO. Both halves hold. Throughput measured without profilers, three variants alternating, medians of three runs:

java -Dkotlinx.coroutines.io.parallelism=8 -jar service.jar

That is +70 % throughput, −44 % CPU per request and a p99 six times lower, from one system property; the three runs of each variant do not overlap with the others on any of the three figures. The dispatch bucket falls with it: 38.9 % of self CPU at 64 workers, 23.2 % at 16, 15.4 % at 8. On Netty the same flag moves nothing — 82 889 rps against 79 337, 79 µs against 82.

The same thing without an engine

The explanation above — workers contending for the head of one queue — is read out of the sources and supported by everything the service does. It is also testable without Ktor: 64 coroutines that do nothing but resume on a dispatcher, four resumptions per operation, no sockets and no HTTP. On a two-core box, all variants saturating the same 1.77 cores:

Same work, same cores; only the number of workers over the shared queue changes, and one dispatch costs eight times more at 64 workers than at eight. The fourth and fifth rows are the ones that matter for a library: a limitedParallelism view sized at the call site buys exactly what the process-wide property buys — 0.99 µs against 1.00 — while leaving Dispatchers.IO alone for everything else in the process. Ktor cannot currently be asked for that: CIOApplicationEngine holds its dispatcher in a private field, and a dispatcher supplied through the parent coroutine context is overridden, which is KTOR-6797, open since February 2024. KTOR-6462, "clients and servers should use Dispatchers.IO.limitedParallelism(...) wherever possible", is closed as fixed; in 3.5.2 and on main the server CIO engine still dispatches into the shared Dispatchers.IO.

The last row is a separate observation: Dispatchers.Default, which has no LimitedDispatcher in the chain at all, is another five times cheaper per dispatch than the cheapest view of Dispatchers.IO.

What the lever costs

io.parallelism is also how many threads CIO has to run handlers on. /blocking is a handler that sleeps 5 ms — a JDBC call without the database — and it inverts the table:

The flag that buys CIO 70 % on non-blocking handlers takes away 7.7× on a blocking one, and Netty — the cheapest engine above — is the worst here at its defaults, because callGroupSize defaults to the processor count. Both defaults are coherent: CIO's 64 workers are the slack that blocking code lives on, and every non-blocking request pays for that slack. Neither engine is "faster"; they place their defaults differently between two regimes, and choosing an engine is choosing a regime.

Where this stops being true

Under 8 connections instead of 64, CIO's queue polling nearly disappears — the queue is empty, the workers park — and the cost reappears as 10.8 context switches per request and 231 µs. Under 256, it is 1.19 switches and 110 µs with a p99 of 37 ms. The handoff is paid in every regime; only the currency changes.

The share, though, is a statement about load, not about CIO. Re-attributing the profiles of a real service — konekt, Ktor CIO with Exposed and Postgres on one core — gives the dispatcher queue 3.4 % of self CPU at 200 rps and 1.1 % at 50 rps, against 36 % on the stand at 43 k rps. A service that waits for a database pays for the handoff too, and does not notice.

How it was measured, and why not in rps

Run-to-run spread of throughput on this machine reaches ±13 % for one unchanged variant, while CPU microseconds per request repeat within 2–9 %. That, plus the concurrency ceiling above, is why the comparison is stated in CPU per request and rps is quoted beside it. Profiles and throughput come from separate windows: sampling at 1 ms costs between 0.4 % of throughput (CIO on /echo) and 8.3 % (Netty on the same endpoint, where there are twice as many requests to sample).

CPU per request is utime+stime from /proc/<pid>/stat, which is tick-sampled, and the engines differ by an order of magnitude in context switches per request — so the accounting itself was checked against the scheduler's own counter (sum_exec_runtime from /proc/<pid>/task/*/schedstat) on a machine whose kernel exposes it. Over the six runs of the table above, from 2 876 to 123 088 context switches per second, the two agree within 0.2 %.

The stand's own machine has no such counter, so there the per-process number was put against the busy time of the cores it is pinned to — those cores run nothing else, and their busy time cannot exceed eight seconds per second. It does: the process reads 8.54 cores where the cores themselves read exactly 8.00. The overstatement is 6.4–7.2 %, and it is the same at 25 context switches per second as at 298 000, so it is a property of that kernel rather than of an engine's threading. Every variant carries the same factor: the ratios in this post stand, and the absolute microseconds are about seven per cent high — 166 per request is nearer 155. The old, unpinned protocol is kept as a control and reproduces the earlier post's numbers — CIO at 34 495 rps against the published 33 935, coroutines owning 72.4 % of CPU against the published 73.9 % for all of kotlinx — which is also how you can tell the new numbers differ by protocol and not by the three engines now sharing a classpath.

Limitations

• One machine, a mobile CPU in WSL2, with no way to fix the clock. Ratios inside a process repeat; absolute throughput does not.
• HTTP/1.1 without TLS, keep-alive, responses of 14–1521 bytes, handlers in fractions of a millisecond. HTTP/2, TLS, WebSocket, streaming bodies and Tomcat were not measured.
• Engines measured as they ship. Raising Netty's callGroupSize would very likely remove its loss on /blocking; that was not run. io.parallelism is process-wide — it changes Dispatchers.IO for every coroutine in the service, not only for the engine.
• /blocking models blocking with Thread.sleep, which occupies a thread without burning CPU. A real JDBC call does both; those rows are about thread capacity, not about the cost of a query.
• The dispatcher table was measured on a different machine from the rest — two cores, another kernel, a background of 0.22 cores — so its microseconds are not comparable with the service numbers above. The ratios between its rows are the point.
• One profile run per configuration, with the repeated A/B series as the check that the per-request CPU numbers hold. Individual frame shares — Jetty's 20.4 % in pthread_cond_signal, say — were not repeated three times.
• I have not filed anything with Ktor about this. The question it raises — why the CIO engine dispatches into the shared Dispatchers.IO rather than its own limitedParallelism sized to the cores — is a question, not a defect report, and the tracker search I ran found only client-engine issues.

Source: github.com/youndie/zavarnik — bench/ for the service and the harness, bench/profile/results/ for the summary of every run cited here, docs/research/research-engines.md for the fact tables with their addresses. The same stand's earlier measurements: User code is 1–4 % of a Ktor service's CPU.