# Handing a request to a handler costs 39 % of a Ktor CIO service's CPU — and 1 % on Netty

> The same service on CIO, Netty and Jetty, one -D apart: 166, 82 and 146 microseconds of CPU per request. A third of CIO's CPU is the queue inside Dispatchers.IO; one system property removes most of it and costs 7.7x on a handler that blocks. The stand, the numbers and the limits.

Page: https://kotlin.website/blog/three-ktor-engines-under-one-protocol · Published: 2026-09-11

A [previous post](https://kotlin.website/blog/user-code-share-of-a-ktor-service) measured what share of a Ktor service's
profile user code owns and found 1–4 %. The biggest number in that profile belonged to nobody in
the question: between a fifth and a third of `/business` CPU sat in the coroutine dispatcher's
queue, and the verdict — "a property of CIO under this load" — was drawn from the only engine the
stand had. This post is
that comparison: the same service, the same jars, the same process on CIO, Netty and Jetty, one
`-Dbench.engine` apart. The stand, the harness and every run summary are in
[zavarnik](https://github.com/youndie/zavarnik) under `bench/`, the full write-up in
`docs/research/research-engines.md`.

## The stand

- Service: Ktor 3.5.2, kotlinx.serialization 1.11.0, kotlinx.coroutines 1.11.0, slf4j-simple.
  Three endpoints: `/echo`, `/items` (JSON CRUD over an in-memory store), `/business` (collection
  chains, a regex, three disabled `logger.debug` templates per request).
- Engines as they ship: CIO, Netty 4.2.16, Jetty 12.0.35 through `ktor-server-jetty-jakarta`. No
  engine configuration is touched, so `connectionGroupSize` = `workerGroupSize` =
  `parallelism / 2 + 1` and `callGroupSize` = `parallelism`, the Ktor defaults.
- Load: `oha`, 64 keep-alive connections; JVM pinned to cores 0–7, generator to 8–15;
  `-Xms1g -Xmx1g -XX:+UseG1GC`.
- Profiler: async-profiler 4.5, 120 s of `cpu` then 120 s of `alloc`, after 60 s of warm-up and a
  120 s profiler-free window that the throughput and cost numbers come from.
- Machine: Ubuntu 24.04 in WSL2, Core Ultra 7 255HX, OpenJDK 25.0.4.

One thing in that protocol used to be a lie. `taskset -pc <pid>` sets the affinity of a single
*thread*: after it, 49 of the JVM's 50 threads still read `Cpus_allowed_list: 0-19`, and the
process burned 12.6 cores while nominally capped at 8. Every run of the earlier post — including
the configuration its text calls pinned — had the service and the load generator fighting over the
same 20 cores. Pinning on the `exec` is inherited by every thread, and the JVM then sizes its pools
to the cores it can see. The old protocol is kept as a control below, and it reproduces the
published numbers.

## What a request costs

`/business`, 64 connections, from the clean window without profilers. The absolute number is
`utime+stime` of the process over that window divided by the requests served in it.

| Engine | rps | p50 | p99 | CPU µs/request | ctx switches/request | threads |
|---|---|---|---|---|---|---|
| CIO | 43 472 | 0.83 ms | 13.50 ms | **166** | 2.53 | 103 |
| Netty | 79 337 | 0.73 ms | 2.05 ms | **82** | 0.63 | 43 |
| Jetty | 52 383 | 1.13 ms | 2.94 ms | **146** | 9.60 | 206 |

On `/echo` the same order holds at 125 / 56 / 88 µs, on `/items` at 142 / 88 / 120. All three
respond with identical bytes — same status, same `Content-Length`, no chunked framing — so the
difference is machinery, not payload. Netty's rps understates the gap: at 100 023 rps it used 5.6
of its 8 cores while the generator used 3.0 of its own, so its throughput there is bounded by the
offered concurrency (64 connections at 0.58 ms is ~110 k rps by Little's law), not by CPU. CIO, at
7.2 cores of 8, is bounded by CPU.

## Where the CPU goes

By owner — the first frame from the leaf that is neither JDK nor JVM — on `/business`:

| Owner | CIO | Netty | Jetty |
|---|---|---|---|
| kotlinx.coroutines | **56.9 %** | **8.9 %** | **29.6 %** |
| the engine itself | — | 43.1 % | 15.5 % |
| Ktor | 21.0 % | 16.0 % | 7.1 % |
| user code | 3.1 % | 6.1 % | 3.3 % |

The leaf frames say it more plainly. CIO: `LockFreeTaskQueueCore.removeFirstOrNull` 17.3 %,
`LimitedDispatcher.obtainTaskOrDeallocateWorker` 6.9 %, `LimitedDispatcher.dispatch` 4.3 %. Netty:
`writev` 6.9 %, `__write` 5.8 %, `epoll_wait` 5.6 %, and the first application frame is
`Pricing.quote` at 1.6 %. Jetty: `pthread_cond_signal` 20.4 % plus libc 20.9 %, `LockSupport.park`
and `unpark` at 2 % each.

Summed into one bucket — everything that moves a request from an I/O thread to a handler and back —
that is 33–48 % of self CPU on CIO across the three endpoints, 25–30 % on Jetty and 1–2.5 % on
Netty.

The reason is in the engines' source, not in the profile. `CIOApplicationEngine` runs both its
engine and its user dispatcher on `Dispatchers.IO`, and `Dispatchers.IO` on the JVM is
`UnlimitedIoScheduler.limitedParallelism(max(64, availableProcessors))` — a `LimitedDispatcher`
holding one `LockFreeTaskQueue` that up to 64 worker loops poll. Netty's `NettyDispatcher` returns
`false` from `isDispatchNeeded` when the code already runs on the channel's event loop, so most
calls are never handed anywhere; what remains goes to that channel's own executor. Jetty's
`JettyKtorHandler` dispatches into its own `ThreadPoolExecutor` over a `SynchronousQueue` — a
rendezvous, paid for in thread wake-ups rather than in polling.

So the queue is not the price of coroutines. It is the price of one shared MPMC queue with 64
workers contending for its head.

## The lever

If that is the mechanism, cutting the worker count to the core count must cut the cost — and must
do nothing to Netty, which does not route calls through `Dispatchers.IO`. Both halves hold.
Throughput measured without profilers, three variants alternating, medians of three runs:

```bash
java -Dkotlinx.coroutines.io.parallelism=8 -jar service.jar
```

| Variant | rps median (runs) | p99 | CPU µs/request |
|---|---|---|---|
| CIO, as it ships (`io.parallelism` = 64) | 40 673 (40 536 · 40 673 · 42 909) | 14.45 ms | 176 |
| CIO, `io.parallelism=16` | 68 382 (66 003 · 68 382 · 70 383) | 5.90 ms | 105 |
| CIO, `io.parallelism=8` | **69 130** (67 175 · 69 130 · 72 691) | **2.45 ms** | **98** |

That is +70 % throughput, −44 % CPU per request and a p99 six times lower, from one system
property; the three runs of each variant do not overlap with the others on any of the three
figures. The dispatch bucket falls with it: 38.9 % of self CPU at 64 workers, 23.2 % at 16, 15.4 %
at 8. On Netty the same flag moves nothing — 82 889 rps against 79 337, 79 µs against 82.

## The same thing without an engine

The explanation above — workers contending for the head of one queue — is read out of the sources
and supported by everything the service does. It is also testable without Ktor: 64 coroutines that
do nothing but resume on a dispatcher, four resumptions per operation, no sockets and no HTTP. On a
two-core box, all variants saturating the same 1.77 cores:

| Dispatcher | ops/s | CPU µs per op | CPU µs per dispatch | threads |
|---|---|---|---|---|
| `Dispatchers.IO`, parallelism 64 (the default) | 219 816 | **8.04** | 2.01 | 114 |
| `Dispatchers.IO`, `io.parallelism=8` | 1 802 299 | **1.00** | 0.25 | 25 |
| `Dispatchers.IO`, `io.parallelism=2` | 2 581 194 | **0.68** | 0.17 | 15 |
| `Dispatchers.IO.limitedParallelism(8)`, default left at 64 | 1 797 108 | **0.99** | 0.25 | 24 |
| `Dispatchers.IO.limitedParallelism(2)`, default left at 64 | 3 071 939 | **0.57** | 0.14 | 15 |
| `Dispatchers.Default` | 8 773 056 | **0.20** | 0.05 | 8 |

Same work, same cores; only the number of workers over the shared queue changes, and one dispatch
costs eight times more at 64 workers than at eight. The fourth and fifth rows are the ones that
matter for a library: a `limitedParallelism` view sized at the call site buys exactly what the
process-wide property buys — 0.99 µs against 1.00 — while leaving `Dispatchers.IO` alone for
everything else in the process. Ktor cannot currently be asked for that: `CIOApplicationEngine`
holds its dispatcher in a private field, and a dispatcher supplied through the parent coroutine
context is overridden, which is [KTOR-6797](https://youtrack.jetbrains.com/issue/KTOR-6797), open
since February 2024. [KTOR-6462](https://youtrack.jetbrains.com/issue/KTOR-6462), "clients and
servers should use `Dispatchers.IO.limitedParallelism(...)` wherever possible", is closed as fixed;
in 3.5.2 and on `main` the server CIO engine still dispatches into the shared `Dispatchers.IO`.

The last row is a separate observation: `Dispatchers.Default`, which has no `LimitedDispatcher` in
the chain at all, is another five times cheaper per dispatch than the cheapest view of
`Dispatchers.IO`.

## What the lever costs

`io.parallelism` is also how many threads CIO has to run handlers on. `/blocking` is a handler that
sleeps 5 ms — a JDBC call without the database — and it inverts the table:

| Variant | rps median | p50 | expected ceiling from thread count |
|---|---|---|---|
| CIO, as it ships | **11 818** | 5.38 ms | 64 / 5 ms = 12 800 |
| CIO, `io.parallelism=8` | **1 538** | 37.89 ms | 8 / 5 ms = 1 600 |
| Netty, as it ships | **1 552** | 41.24 ms | `callGroupSize` = 8 → 1 600 |
| Jetty, as it ships | **11 945** | 5.32 ms | pool grows to `callGroupSize × 8` = 64 |

The flag that buys CIO 70 % on non-blocking handlers takes away 7.7× on a blocking one, and Netty
— the cheapest engine above — is the worst here at its defaults, because `callGroupSize` defaults
to the processor count. Both defaults are coherent: CIO's 64 workers are the slack that blocking
code lives on, and every non-blocking request pays for that slack. Neither engine is "faster";
they place their defaults differently between two regimes, and choosing an engine is choosing a
regime.

## Where this stops being true

Under 8 connections instead of 64, CIO's queue polling nearly disappears — the queue is empty, the
workers park — and the cost reappears as 10.8 context switches per request and 231 µs. Under 256,
it is 1.19 switches and 110 µs with a p99 of 37 ms. The handoff is paid in every regime; only the
currency changes.

The share, though, is a statement about load, not about CIO. Re-attributing the profiles of a real
service — [konekt](https://github.com/youndie/konekt), Ktor CIO with Exposed and Postgres on one
core — gives the dispatcher queue 3.4 % of self CPU at 200 rps and 1.1 % at 50 rps, against 36 % on
the stand at 43 k rps. A service that waits for a database pays for the handoff too, and does not
notice.

## How it was measured, and why not in rps

Run-to-run spread of throughput on this machine reaches ±13 % for one unchanged variant, while CPU
microseconds per request repeat within 2–9 %. That, plus the concurrency ceiling above, is why the
comparison is stated in CPU per request and rps is quoted beside it. Profiles and throughput come
from separate windows: sampling at 1 ms costs between 0.4 % of throughput (CIO on `/echo`) and
8.3 % (Netty on the same endpoint, where there are twice as many requests to sample).

CPU per request is `utime+stime` from `/proc/<pid>/stat`, which is tick-sampled, and the engines
differ by an order of magnitude in context switches per request — so the accounting itself was
checked against the scheduler's own counter (`sum_exec_runtime` from `/proc/<pid>/task/*/schedstat`)
on a machine whose kernel exposes it. Over the six runs of the table above, from 2 876 to 123 088
context switches per second, the two agree within 0.2 %.

The stand's own machine has no such counter, so there the per-process number was put against the
busy time of the cores it is pinned to — those cores run nothing else, and their busy time cannot
exceed eight seconds per second. It does: the process reads 8.54 cores where the cores themselves
read exactly 8.00. The overstatement is 6.4–7.2 %, and it is the same at 25 context switches per
second as at 298 000, so it is a property of that kernel rather than of an engine's threading. Every
variant carries the same factor: the ratios in this post stand, and the absolute microseconds are
about seven per cent high — 166 per request is nearer 155. The old,
unpinned protocol is kept as a control and reproduces the earlier post's numbers — CIO at 34 495
rps against the published 33 935, coroutines owning 72.4 % of CPU against the published 73.9 % for
all of kotlinx — which is also how you can tell the new numbers differ by protocol and not by the
three engines now sharing a classpath.

## Limitations

- One machine, a mobile CPU in WSL2, with no way to fix the clock. Ratios inside a process repeat;
  absolute throughput does not.
- HTTP/1.1 without TLS, keep-alive, responses of 14–1521 bytes, handlers in fractions of a
  millisecond. HTTP/2, TLS, WebSocket, streaming bodies and Tomcat were not measured.
- Engines measured as they ship. Raising Netty's `callGroupSize` would very likely remove its loss
  on `/blocking`; that was not run. `io.parallelism` is process-wide — it changes `Dispatchers.IO`
  for every coroutine in the service, not only for the engine.
- `/blocking` models blocking with `Thread.sleep`, which occupies a thread without burning CPU. A
  real JDBC call does both; those rows are about thread capacity, not about the cost of a query.
- The dispatcher table was measured on a different machine from the rest — two cores, another
  kernel, a background of 0.22 cores — so its microseconds are not comparable with the service
  numbers above. The ratios between its rows are the point.
- One profile run per configuration, with the repeated A/B series as the check that the per-request
  CPU numbers hold. Individual frame shares — Jetty's 20.4 % in `pthread_cond_signal`, say — were
  not repeated three times.
- I have not filed anything with Ktor about this. The question it raises — why the CIO engine
  dispatches into the shared `Dispatchers.IO` rather than its own `limitedParallelism` sized to the
  cores — is a question, not a defect report, and the tracker search I ran found only client-engine
  issues.

---

Source: [github.com/youndie/zavarnik](https://github.com/youndie/zavarnik) — `bench/` for the
service and the harness, `bench/profile/results/` for the summary of every run cited here,
`docs/research/research-engines.md` for the fact tables with their addresses. The same stand's
earlier measurements: [User code is 1–4 % of a Ktor service's
CPU](https://kotlin.website/blog/user-code-share-of-a-ktor-service).