# User code is 1–4 % of a Ktor service's CPU: a negative result for a bytecode optimiser

> I set out to build a Kotlin/JVM optimiser plugin and measured first. User code owns 1–4 % of CPU and 3–10 % of allocations on a Ktor service, R8 cannot even run as the baseline on this stack, and the plugin was not built. The numbers and the stand.

Page: https://kotlin.website/blog/user-code-share-of-a-ktor-service · Published: 2026-09-07

The plan was a Gradle plugin with two engines: a K2 IR pass over user code — boxing at `suspend`
boundaries, fused collection chains, lazy logging, hoisted constants — and an ASM transform over
the whole runtime classpath. The brief put a gate in front of it: measure what share of a real
Ktor CIO service's profile user code actually owns, and what R8 with a sensible server
configuration already delivers, because those two numbers bound what such a plugin can ever
move. This is the result of that gate. The plugin was not built; the stand, the harness and the
research document are in [zavarnik](https://github.com/youndie/zavarnik) under `bench/` and
`docs/research/research-optimizer.md`.

## The stand

- Service: Ktor 3.5.2 on CIO, kotlinx.serialization 1.11.0, slf4j-simple. Three endpoints:
  `/echo`, `/items` (JSON CRUD over an in-memory store), `/business` (collection chains, a regex
  validation, three `logger.debug` templates per request with debug disabled).
- Load: `oha`, 64 connections, keep-alive; JVM pinned to cores 0–7, generator to 8–15,
  `-Xms1g -Xmx1g -XX:+UseG1GC`. **Correction, 11 September 2026:** the pinning did not take.
  `taskset -pc <pid>` sets the affinity of a single *thread*, so 49 of the JVM's 50 threads kept
  the full 20-core mask and the service shared every core with its own load generator. Every
  number in this post was measured that way. Shares by owner are ratios inside one process and do
  not depend on it; the throughput figures do. Measured again with the pinning fixed in [three
  Ktor engines under one protocol](https://kotlin.website/blog/three-ktor-engines-under-one-protocol), where the old
  protocol is kept as a control and reproduces these numbers.
- Profiler: async-profiler 4.5, `cpu` for 120 s then `alloc` for 120 s after 60 s of warm-up.
- Machine: Ubuntu 24.04 in WSL2, Core Ultra 7 255HX, OpenJDK 25.0.4.
- Attribution: every sample is classified twice. *Self* is the leaf frame; *owner* is the first
  frame from the leaf that is neither JDK nor JVM — whose code asked for this. Owner is the
  number that matters: it is the ceiling for anything that rewrites that code.

## The share

| Endpoint | rps | p99 | user CPU, self / owner | user alloc, self / owner |
|---|---|---|---|---|
| `/echo` | 35 740 | 11.4 ms | 0.0 % / 0.0 % | 0.2 % / 0.2 % |
| `/items?limit=20` | 38 008 | 8.0 ms | 0.8 % / 0.8 % | 0.3 % / 0.3 % |
| `/business` | 33 935 | 10.2 ms | 1.1 % / 2.1 % | 3.6 % / 9.9 % |

On the endpoint written to give user code the most to do, user code owns 2.1 % of CPU and 9.9 %
of allocated bytes. Three further CPU profiles of the same endpoint, taken while checking a
hypothesis about core pinning, gave 1.7–2.0 %. The rest of `/business` by owner: CPU — kotlinx
73.9 %, Ktor 15.5 %, stdlib 8.4 %; allocations — Ktor 36.8 %, kotlinx 26.4 %, stdlib 26.8 %.

A stand can be accused of being written for the answer, so the same harness was run inside the
container of a service that was not written for it: [konekt](https://github.com/youndie/konekt),
my own reference project — an eSIM operator's account backend on Ktor CIO with Exposed and
Postgres, sixteen of its own modules on the server's classpath as direct dependencies — under
its own k6 scenario at a constant arrival rate, one core and 1 GiB. A reference project, not
production traffic; what it adds is a database, a serialiser and a framework stack that the
stand does not have.

| Rate | p95 | user CPU, self / owner | user alloc, self / owner |
|---|---|---|---|
| 50 rps | 3.97 ms | 2.5 % / 4.0 % | 1.0 % / 3.4 % |
| 200 rps | 2.93 ms | 0.5 % / 0.9 % | 1.3 % / 5.4 % |

konekt is a database driver, a serialiser and a framework with a thin layer of user code on
top; the share is lower than on the stand, not higher. Its biggest allocation owners
were stdlib (26.6 %), Ktor (24.8 %), kotlinx (13.6 %) and Exposed (13.5 %) — none of them the
territory of a pass over user code.

## Where the profile actually is

Between a fifth and a third of `/business` CPU is the coroutine dispatcher's queue, under
`LimitedDispatcher`. In the run the tables above come from it was 36 %:
`LockFreeTaskQueue.removeFirstOrNull` at 30.1 % self plus `LockFreeTaskQueueCore.removeFirstOrNull`
at 5.9 %. A repeat two days later, same machine and same harness, put the same family of frames
at 21.6 %, with `LockFreeTaskQueueCore.removeFirstOrNull` the only one of the two present, at
13.8 %. The share moves with how fast the handlers are — the repeat ran at half the throughput
with a p50 of 2.6 ms against 1.2 — and which of the two is cause and which effect is not
established here, so the honest form of the number is the range. I took it for an artefact of
pinning the JVM to 8 cores while `availableProcessors` saw 20, and measured it three more ways:
pinned, pinned with `-XX:ActiveProcessorCount=8`, and unpinned. The queue is the largest
Kotlin-side cost in all of them, with the frames shifting between `removeFirstOrNull` and
`obtainTaskOrDeallocateWorker`. It is a property of CIO under this load — 64 keep-alive
connections — not of the harness, and neither an IR pass nor a bytecode peephole can touch it at
either end of the range. Netty against CIO under the same protocol is the obvious next
measurement; it is not in this post.

**Follow-up, 11 September 2026.** It has been measured: [three Ktor engines under one
protocol](https://kotlin.website/blog/three-ktor-engines-under-one-protocol). On the same service under Netty,
kotlinx.coroutines owns 8.9 % of CPU against CIO's 56.9 %, so this is the price of the route
through `Dispatchers.IO` — a `LimitedDispatcher` with one queue and up to 64 workers polling it —
rather than the price of coroutines; `-Dkotlinx.coroutines.io.parallelism` set to the core count
removes most of it, at the cost of the threads a blocking handler needs. The three configurations
named just above were, because of the `taskset` mistake, two.

The rest, by size:

- The top of the allocation profile is `byte[]` at 18.4 %, `String` 6.8 %, `Object[]` 5.9 %:
  Ktor's I/O buffers and JSON parsing.
- `kotlin.reflect.jvm.internal.KClassImpl.toString` runs on every `call.receive<T>()`, from the
  default body transformations, and costs 1.98 % of all bytes on `/business`.
- Boxing — `Integer`, `Long`, `Double` — is 1.26 % of allocated bytes. The brief's red line for a
  boxing pass was 3 %.

What user code does spend, all on `/business`: a `Regex` constructed inside the handler, 6.45 %
of bytes including the `Matcher` and its arrays; intermediate collections of `filter`/`map`
chains, about 4.6 %; the three debug templates, 1.57 %. The first two are fixed by moving one line
each. Nothing else is above 2 %.

## R8 as the baseline

The brief asked for R8 with a server configuration — no obfuscation, keeps on serialisation,
Ktor, coroutines and slf4j, `-assumenosideeffects` on `Intrinsics.check*` — as the line a plugin
must beat. R8 9.4.17 processes the 26 jars in 26 s, 6503 classes to 5416. The result does not
start:

```text
java.lang.VerifyError: Bad invokespecial instruction: interface method to invoke is not in a direct superinterface
    at kotlinx.coroutines.CompletableDeferred.access$cancel$jd
```

The mechanism is visible in `javap`. Kotlin compiles the synthetic accessor as
`invokespecial InterfaceMethod CompletableDeferred.cancel:()V`, a call to a default method through
the current interface. R8 rebinds the reference to the declaring interface,
`invokespecial InterfaceMethod Job.cancel:()V`, and the verifier requires the interface in an
`invokespecial` to be the current class or a *direct* superinterface; `Job` is reached through
`Deferred`. `-dontoptimize -dontshrink` gives the same error, so does `--no-desugaring`, so does
9.5.10-dev; keeping coroutines out of the input moves the error into Ktor's `Routing`. kotlinc
had the same bug on its side until 1.5.0 (KT-42753). What synthesises the accessor is Kotlin's
compatibility mode, not a flag a project opts into: under `-jvm-default=enable` — the default on
Kotlin 2.4.10, and what the old `-Xjvm-default=all-compatibility` selects — an interface gets
both the default method and the legacy `DefaultImpls` class, and a separate class cannot reach a
default method with `invokespecial`, so the compiler bridges them with a static `access$…$jd` on
the interface whose body is exactly the instruction above. Under `-jvm-default=no-compatibility`,
which is what the old `-Xjvm-default=all` maps to, there is no `DefaultImpls`, no accessor and
nothing to rebind. The affected set is the default setting, which is how it covers the whole
coroutines and Ktor stack. The rule is a classfile rule; DEX has no equivalent, which is
presumably why the classfile backend never met it.

What can be measured is R8 over the user jar alone, with the 25 library jars on `--classpath`.
That removes all 56 `Intrinsics.check*` calls from user code and cuts its bytecode by 35 %. The
effect on the service, same protocol:

| Endpoint | alloc/req, baseline | alloc/req, R8 | rps A/B median, baseline | rps A/B median, R8 |
|---|---|---|---|---|
| `/echo` | 14.6 KiB | 14.6 KiB | 36 990 | 41 434 |
| `/items` | 23.7 KiB | 23.2 KiB | 37 439 | 37 143 |
| `/business` | 33.8 KiB | 34.0 KiB | 27 531 | 28 241 |

Allocations per request did not move (within 2 %): null checks do not allocate, and inlining
inside 39 classes changed nothing. Throughput was measured separately, without profilers, both
variants alternating, three runs each, medians. `/echo` — where there is no user code and R8 had
nothing to change — came out 12 % faster under B. That number is the noise ruler of this stand,
and the `/business` delta of +2.6 % sits inside it.

## What the stand taught about measuring

- A CPU profiler at 1 ms costs throughput: `/echo` served 35 740 rps under it and 44 906 rps in
  the allocation window of the same run. Profile and throughput are two protocols, not one.
- Run-to-run spread on this machine is ±15 % even on the endpoint nothing changed on. WSL2
  exposes neither `cpufreq` nor `intel_pstate`; frequency belongs to the Windows host. One run
  per variant is not a measurement.
- The first inlining log had zero mentions of user code and looked like a clean negative. It was
  taken without `-XX:+PrintCompilation`, which is what names the compilation roots. With it, five
  refusals name user methods, all of them "hot method too big" — `Pricing.quote` at 1827 bytes
  against C2's `FreqInlineSize` of 325.

## Verdict

The gate had two conditions: green if user code owns at least 25 % of allocations, *or* if R8
improves the service by less than 3 % — the second on the assumption that R8 sets the bar and
a plugin has room above it. By that letter the result is green: R8 moved nothing. But the
reason it moved nothing is not that R8 is weak; it is that there was nothing to move. The
ceiling of a pass over user code is the share of the profile user code owns, and that share is
2.1 % of CPU and 9.9 % of allocations on the stand, 1–4 % and 3–5 % on konekt. Removing
*every* user allocation would shift alloc/req by a tenth; the brief's own kill criterion —
5 % of throughput or 10 % of allocations with every pass on — is unreachable by construction. The ASM
layer reaches the whole process, but what is large in the profile — the dispatcher queue, the
I/O buffers, `KClassImpl.toString` — is not a peephole. The plugin was not built.

What survived: a method-size diagnostic (10 of 296 user methods over 325 bytes, three of them
refused as hot) now lives as a lint task in the portfolio's build conventions, and the stand
stays in zavarnik as the harness for its warm-up measurements.

## Re-run 2026-09-08

The profile was taken again from scratch two days after the numbers above, same machine, same
harness. The shares held to within half a point — user code owns 2.5 % of CPU and 10.2 % of
allocations on `/business` against 2.1 % and 9.9 % here — and so did everything about R8: 6503
classes to 5416, the same `VerifyError` at the same method, all 56 `Intrinsics.check*` calls
gone from the user jar, the same ten methods over 325 bytes with `Pricing.quote` at 1827.
Throughput did not reproduce at all: `/business` served 16 902 rps against 33 935, which is this
post's own point about what the stand can and cannot measure. Corrected in the text: the
dispatcher-queue share now carries the range it actually has, and two counts — konekt's modules
and the method denominator — were wrong.

## Correction 2026-09-11

The paragraph above used to say that any interface compiled with `-Xjvm-default=all` is
affected. That is precisely the mode that is not. The issue came back with a reproduction and an
explanation pointing at Kotlin's compatibility mode instead, and compiling the same
three-interface hierarchy four ways on Kotlin 2.4.10 settles it: the `access$…$jd` accessor is
emitted with no flag at all and under `-jvm-default=enable`, and is absent under
`-jvm-default=no-compatibility` and `-jvm-default=disable`. The shipped
`kotlinx-coroutines-core-jvm-1.11.0.jar` carries 24 `DefaultImpls` classes, which is the same
mode seen from the other side. The correction widens the blast radius rather than narrowing it:
it is what you get by default, not what you asked for.

## Limitations

- One machine, a mobile CPU in WSL2, with no way to fix the clock. Shares by owner are ratios
  inside one process and were stable across repeats; throughput figures are not.
- Not all of the CPU profile resolves to a name: in the repeat, 15.2 % of `/business` self time
  is `libc.so.6` with no further attribution. Nothing here rests on it, and it is neither user
  code nor anything a plugin over user code could reach, but it is a sixth of the profile that
  this post does not account for.
- Allocation bytes are sampled by async-profiler; alloc/req is comparable between variants,
  not an absolute.
- One real service, one scenario. A service whose handlers do their own arithmetic would score
  higher; none of mine do.
- R8 was tried at 9.4.17 and 9.5.10-dev on a Kotlin 2.4.10 / coroutines 1.11.0 classpath. It is
  filed upstream as [issue 558351430](https://issuetracker.google.com/issues/558351430), with
  R8's own input dump attached. Whether it was already known I could not establish, and that
  link may show you nothing: the tracker is readable only after a sign-in. The `javap`
  before/after and the reproducer are in `bench/profile/` for anyone who wants to.

---

Source: [github.com/youndie/zavarnik](https://github.com/youndie/zavarnik) — `bench/` for the
service and the harness, `bench/profile/results/` for every profile cited,
`docs/research/research-optimizer.md` for the fact tables with their addresses. The first phase
of the same repository, the AOT cache, has its own post:
[OpenJDK 25.0.0–25.0.3 uses a stale AOT cache without saying so](https://kotlin.website/blog/stale-aot-cache-on-jdk25).