Ktor on Kotlin/Native under a container limit

The previous post left a Kotlin/Native Ktor service that starts in 28 ms and 20 MiB and falls apart under load, with GC.collect() on a timer as the only lever I had. This is the follow-up with a proper stand, and the collector turned out to be a bystander in both halves of the problem.

With one change in Ktor — released as 3.6.0 — and one build option, the same service holds 2 000 requests a second inside a 128 MiB container, ten runs out of ten, at a p99 of 5.5 ms and a peak of 61 MB. Before the two changes that cell was six OOM kills out of six. For scale, measured in the same cell: the JVM answers in 29.6 ms holding 105 MB, and Go in 2.3 ms holding 6 MB.

None of this is the runtime's fault in general, and the ladder says so: a bare epoll loop with no Ktor at all holds the same 2 000 rps in 5 MB at a p99 of 1.52 ms, and the rung below HTTP parsing — raw sockets through ktor-network — goes slow in 0 of 12 runs. Below a thousand requests a second on these four cores, unpatched Kotlin/Native was already the fastest of the three: 2.10 ms against the JVM's 3.60. Everything below is about what changes between there and 2 000.

Both changes were reported upstream, and one of them is now a version number. The Ktor one — KTOR-9891 — is fixed and released: ktorio/ktor#5874 shipped in 3.6.0 on 16 September 2026, so on that release the throughput half of this post is an upgrade and nothing else. It was re-measured there rather than taken on trust — On 3.6.0 as released, below. The runtime one, KT-89365, was closed on 16 September as a duplicate of KT-74834, which asks for a way to bound the empty allocator pages a thread keeps and has been open since January 2025. There is a runnable reproducer for the first: youndie/ktor-9891-repro — read its README before running it, because on a machine with two cores online none of this happens.

The stand

Two dedicated hosts, four cores and 7.6 GiB each, Ubuntu 26.04, subject and generator on separate machines over a private link. Kotlin 2.4.10 linuxX64 release binaries, Ktor 3.5.2 with ktor-server-cio, one route returning one JSON object. k6, 30 s per run, memory bounded by a cgroup and read from it. The release check in On 3.6.0 as released is that same pair and that same reproducer on 17 September 2026, with the version in the build file as the only difference between its arms.

One methodological note, because every number below depends on it. At 2 000 rps this service has two speeds: the same binary on the same machine either answers in single-digit milliseconds or in hundreds, decided in the first five seconds of a run. Across 156 runs there is nothing in between — every p99 is under 70 ms or over 110. A p99 from one run is therefore not a number about the program. Everything here is scored as the share of runs that land in the slow state, twelve runs per arm, arms rotated, with Fisher's exact test.

Where the throughput goes

Not into the collector. A profile of a slow run puts 32–73 % of all CPU inside one function — io.ktor.utils.io.locks.SynchronizedObject#lock — and the collector's own functions at one or two per cent. Attaching gdb under load and walking every thread, all 211 frames inside that function have the same two callers: DefaultPool#borrow from HttpHeadersMap's constructor, and DefaultPool#recycle from its release.

HttpHeadersMap — the map CIO parses every request's headers into — keeps its storage in two process-wide pools, and on Kotlin/Native DefaultPool is a mutableListOf behind one lock taken on every borrow and every recycle. So every HTTP request takes two process-wide locks at least twice each, while the process's hundred-odd threads do the same on the same two atomics.

That lock is not a monitor. On Kotlin/Native it is a CAS loop over an AtomicReference<LockState> that allocates a new LockState on every attempt, so a lost CAS is an allocation rather than a cheap retry and the cost of an acquisition rises with the number of threads trying. On the JVM the class is erased and synchronized becomes a monitor, which is why the same Ktor source has no such behaviour there.

It shows up as sensitivity to concurrency rather than to rate. Same binary, same 2 000 rps offered, ten runs per arm:

The fix is a deletion, and it is not mine. kotlinx.atomicfu rewrote that lock in 0.28.0 — the owner lives in an AtomicLong, waiters park on a monitor, nothing is allocated. ktor-io already declared atomicfu as an api dependency and already resolved 0.32.1, so the fixed implementation was on the classpath of every Kotlin/Native Ktor build, behind a copy of the version it replaced. Twelve runs per arm, four builds from the same tree differing in one dependency:

Zero of seventy-two patched runs slow against nineteen of twenty-four stock, p = 1.7 × 10⁻¹³. Replacing the lock alone is as good as rewriting the pool and better on p99, which is why the pull request changes the lock. At 2 000 rps and 512 MiB the patched service answers in 6.5 ms where the JVM answers in 15.7.

On 3.6.0 as released

Three of those four builds are 3.5.2 with ktor-io swapped out, which is not an artifact anyone will depend on, so the release was measured as an arm of its own. In 3.6.0 the class stays and its three methods delegate to kotlinx.atomicfu.locks.SynchronizedObject; the copy of the lock, its LockState and the native mutex pool behind it are gone from ktor-io/posix, and the atomicfu resolved with it moves from 0.32.1 to 0.33.0. DefaultPool is untouched — the header pools are still process-wide, the lock under them is simply no longer allocating.

Same reproducer, same stand, the two released versions interleaved run for run, twelve rounds each at 200 connections and the same 2 000 rps offered:

p = 7.4 × 10⁻⁷, and the version in the build file is the only thing that differs between the arms.

Those p99s do not line up with the table above, and a third arm in the same sweep says why: 3.5.2 with the pull request's ktor-io — the 8.4 ms row — rebuilt and run alongside the release on the same day lands at 23.7 ms against the release's 24.1, 0 of 10 slow in both arms, with the ranges overlapping everywhere. Between sweeps on this stand a fast arm's p99 moves by nearly a factor of three; its slow-run share does not move at all. Compare arms inside one sweep, and across sweeps compare the share.

If you cannot move to 3.6.0

On 3.5.2 and earlier two things work. Cap the connections an instance accepts — the table above is the whole argument: twelve connections go slow in 0 of 10 runs at the same offered rate, two hundred in 10 of 10. A reverse proxy in front with a bounded keep-alive pool to the service turns the variable you cannot control into one you can, and costs nothing else. Or build ktor-io yourself with the three-method change and substitute it; that is what every patched number here was measured on.

Where the memory goes

Also not into the heap. Bounding maxHeapBytes at half the container limit — the application reading memory.max itself, since the runtime does not — moved the peak by thirteen megabytes and survival not at all: seven survivals of twelve at 256 MiB with the bound, with it at half, and without it.

Pooled across 413 surviving runs of every sweep taken, peak RSS against peak thread count gives r = 0.994 at 2.96 MB per thread over a 10 MB intercept. The runtime source says why: CustomAllocator is per-thread and holds one FixedBlockPage per block-size class plus a 256 KiB next-fit page, and a thread that has touched a size class keeps that page for as long as it lives, occupied or not. No GC setting counts them — they are pages, not objects.

-Xbinary=fixedBlockPageSize resizes them; the value is in KiB. Eight runs per arm at 50 connections:

Fitted per build, the slope is 2.96, 0.66, 0.40 and 0.06 MB per thread. The 16 KiB page is free: four times less memory at the same p99 and the same CPU — 5.65 ms against 6.00, 179 % against 181 %, which is noise in both columns. There is no argument for the default in a service with a thread pool.

That slope is also why this table says 48.7 MB at 128 MiB and the lead says 61. Same cell, two sweeps: this one held a median of 62 threads, the ten-run sweep in What to set held 79. Seventeen more threads at 0.66 MB each is 11.6 MB, and the gap is 12.7. Quote the larger number when sizing a container — RSS here is a function of how many threads the load happens to spin up, not a constant.

And the thread count is itself a setting, one layer up: capping the dispatcher the CIO engine runs on takes the same service from 67 MB to 36 MB at 200 connections, at a lower p99 and the same CPU. That needs a fork rather than a build option, and it is measured in the next post.

What to set

Ten runs per arm, every arm serving essentially the full offered rate:

kotlin {
    linuxX64 {
        binaries.executable {
            binaryOption("fixedBlockPageSize", "16")
        }
    }
}

-Xallocator=std is the one real trade here: 2.3 times the p99 and 34 % more CPU at 512 MiB and 200 connections, in exchange for the lowest floor. At that floor the other two columns are Go, unmoved at 6.5 MB and 2.67 ms, and the JVM, which does not run there at all — at 64 MiB it starts, sizes its heap to half the limit and is OOM-killed before serving a request; at 32 MiB it never reaches its ready line. Six runs each.

All of it is on top of the lock fix, which now means Ktor 3.6.0 or newer. Without it — so on 3.5.2 and earlier — a build at 200 connections serves a ninth of the offered rate whatever its allocator is set to.

Limitations

linuxX64 is the only target measured. The lock lives in ktor-io/posix, the source set every native target shares, so the behaviour should be the same on Apple and MinGW targets — should be, not is.

Four cores is the only machine. The core count turned out to matter in a way I cannot explain: with cores taken offline, so the runtime itself counts fewer, the service goes slow in 0 of 12 runs at two cores and 0 of 12 at three against 15 of 20 at four. Restricting the same four-core machine to two CPUs with a cpuset — which leaves the online count at four — does not help, and the thread count is not the difference either. Something that follows the number of online CPUs decides it, and naming it needs the runtime's own instrumentation rather than a load generator.

One route returning one small JSON object is the only workload. A service whose handlers allocate more will move the numbers; the two mechanisms are about per-request framework overhead and per-thread allocator pages, and neither depends on what the handler does.

The fixedBlockPageSize numbers are peaks over 30-second runs at a steady rate. Nothing here says what a day-long process does, and page-cache behaviour is exactly the kind of thing that could differ over hours.