Ktor on Kotlin/Native under a container limit

The previous post left a Kotlin/Native Ktor service that starts in 28 ms and 20 MiB and falls apart under load, with GC.collect() on a timer as the only lever I had. This is the follow-up with a proper stand, and the collector turned out to be a bystander in both halves of the problem.

With one change in Ktor and one build option, the same service holds 2 000 requests a second inside a 128 MiB container, ten runs out of ten, at a p99 of 5.5 ms and a peak of 61 MB. Before the two changes that cell was six OOM kills out of six. For scale, measured in the same cell: the JVM answers in 29.6 ms holding 105 MB, and Go in 2.3 ms holding 6 MB.

None of this is the runtime's fault in general, and the ladder says so: a bare epoll loop with no Ktor at all holds the same 2 000 rps in 5 MB at a p99 of 1.52 ms, and the rung below HTTP parsing — raw sockets through ktor-network — goes slow in 0 of 12 runs. Below a thousand requests a second on these four cores, unpatched Kotlin/Native was already the fastest of the three: 2.10 ms against the JVM's 3.60. Everything below is about what changes between there and 2 000.

Both changes are reported upstream: KTOR-9891 with a pull request at ktorio/ktor#5874, and KT-89365.

The stand

Two dedicated hosts, four cores and 7.7 GiB each, Ubuntu 26.04, subject and generator on separate machines over a private link. Kotlin 2.4.10 linuxX64 release binaries, Ktor 3.5.2 with ktor-server-cio, one route returning one JSON object. k6, 30 s per run, memory bounded by a cgroup and read from it.

One methodological note, because every number below depends on it. At 2 000 rps this service has two speeds: the same binary on the same machine either answers in single-digit milliseconds or in hundreds, decided in the first five seconds of a run. Across 156 runs there is nothing in between — every p99 is under 70 ms or over 110. A p99 from one run is therefore not a number about the program. Everything here is scored as the share of runs that land in the slow state, twelve runs per arm, arms rotated, with Fisher's exact test.

Where the throughput goes

Not into the collector. A profile of a slow run puts 32–73 % of all CPU inside one function — io.ktor.utils.io.locks.SynchronizedObject#lock — and the collector's own functions at one or two per cent. Attaching gdb under load and walking every thread, all 211 frames inside that function have the same two callers: DefaultPool#borrow from HttpHeadersMap's constructor, and DefaultPool#recycle from its release.

HttpHeadersMap — the map CIO parses every request's headers into — keeps its storage in two process-wide pools, and on Kotlin/Native DefaultPool is a mutableListOf behind one lock taken on every borrow and every recycle. So every HTTP request takes two process-wide locks at least twice each, while the process's hundred-odd threads do the same on the same two atomics.

That lock is not a monitor. On Kotlin/Native it is a CAS loop over an AtomicReference<LockState> that allocates a new LockState on every attempt, so a lost CAS is an allocation rather than a cheap retry and the cost of an acquisition rises with the number of threads trying. On the JVM the class is erased and synchronized becomes a monitor, which is why the same Ktor source has no such behaviour there.

It shows up as sensitivity to concurrency rather than to rate. Same binary, same 2 000 rps offered, ten runs per arm:

The fix is a deletion, and it is not mine. kotlinx.atomicfu rewrote that lock in 0.28.0 — the owner lives in an AtomicLong, waiters park on a monitor, nothing is allocated. ktor-io already declares atomicfu as an api dependency and already resolves 0.32.1, so the fixed implementation is on the classpath of every Kotlin/Native Ktor build today, behind a copy of the version it replaced. Twelve runs per arm, four builds from the same tree differing in one dependency:

Zero of seventy-two patched runs slow against nineteen of twenty-four stock, p = 1.7 × 10⁻¹³. Replacing the lock alone is as good as rewriting the pool and better on p99, which is why the pull request changes the lock. At 2 000 rps and 512 MiB the patched service answers in 6.5 ms where the JVM answers in 15.7.

Until that merges

Two things work today. Cap the connections an instance accepts — the table above is the whole argument: twelve connections go slow in 0 of 10 runs at the same offered rate, two hundred in 10 of 10. A reverse proxy in front with a bounded keep-alive pool to the service turns the variable you cannot control into one you can, and costs nothing else. Or build ktor-io yourself with the three-method patch from the pull request and substitute it; that is what every patched number here was measured on.

Where the memory goes

Also not into the heap. Bounding maxHeapBytes at half the container limit — the application reading memory.max itself, since the runtime does not — moved the peak by thirteen megabytes and survival not at all: seven survivals of twelve at 256 MiB with the bound, with it at half, and without it.

Pooled across 413 surviving runs of every sweep taken, peak RSS against peak thread count gives r = 0.994 at 2.96 MB per thread over a 10 MB intercept. The runtime source says why: CustomAllocator is per-thread and holds one FixedBlockPage per block-size class plus a 256 KiB next-fit page, and a thread that has touched a size class keeps that page for as long as it lives, occupied or not. No GC setting counts them — they are pages, not objects.

-Xbinary=fixedBlockPageSize resizes them; the value is in KiB. Eight runs per arm at 50 connections:

Fitted per build, the slope is 2.96, 0.66, 0.40 and 0.06 MB per thread. The 16 KiB page is free: four times less memory at the same p99 and the same CPU — 5.65 ms against 6.00, 179 % against 181 %, which is noise in both columns. There is no argument for the default in a service with a thread pool.

That slope is also why this table says 48.7 MB at 128 MiB and the lead says 61. Same cell, two sweeps: this one held a median of 62 threads, the ten-run sweep in What to set held 79. Seventeen more threads at 0.66 MB each is 11.6 MB, and the gap is 12.7. Quote the larger number when sizing a container — RSS here is a function of how many threads the load happens to spin up, not a constant.

What to set

Ten runs per arm, every arm serving essentially the full offered rate:

kotlin {
    linuxX64 {
        binaries.executable {
            binaryOption("fixedBlockPageSize", "16")
        }
    }
}

-Xallocator=std is the one real trade here: 2.3 times the p99 and 34 % more CPU at 512 MiB and 200 connections, in exchange for the lowest floor. At that floor the other two columns are Go, unmoved at 6.5 MB and 2.67 ms, and the JVM, which does not run there at all — at 64 MiB it starts, sizes its heap to half the limit and is OOM-killed before serving a request; at 32 MiB it never reaches its ready line. Six runs each.

All of it is on top of the lock fix. Until that lands, a stock 3.5.2 build at 200 connections serves a ninth of the offered rate whatever its allocator is set to.

Limitations

linuxX64 is the only target measured. The lock lives in ktor-io/posix, the source set every native target shares, so the behaviour should be the same on Apple and MinGW targets — should be, not is.

Four cores is the only machine. The core count turned out to matter in a way I cannot explain: with cores taken offline, so the runtime itself counts fewer, the service goes slow in 0 of 12 runs at two cores and 0 of 12 at three against 15 of 20 at four. Restricting the same four-core machine to two CPUs with a cpuset — which leaves the online count at four — does not help, and the thread count is not the difference either. Something that follows the number of online CPUs decides it, and naming it needs the runtime's own instrumentation rather than a load generator.

One route returning one small JSON object is the only workload. A service whose handlers allocate more will move the numbers; the two mechanisms are about per-request framework overhead and per-thread allocator pages, and neither depends on what the handler does.

The fixedBlockPageSize numbers are peaks over 30-second runs at a steady rate. Nothing here says what a day-long process does, and page-cache behaviour is exactly the kind of thing that could differ over hours.