# Ktor on Kotlin/Native under a container limit

> A Ktor service that was OOM-killed in every run at 128 MiB now serves 2 000 rps there in 61 MB at a p99 of 5.5 ms. Two changes, neither of them a GC setting.

Page: https://kotlin.website/blog/under-a-container-limit · Published: 2026-09-14

[The previous post](https://kotlin.website/blog/native-under-load) left a Kotlin/Native Ktor service that starts in 28 ms
and 20 MiB and falls apart under load, with `GC.collect()` on a timer as the only lever I had. This
is the follow-up with a proper stand, and the collector turned out to be a bystander in both halves
of the problem.

With one change in Ktor — released as 3.6.0 — and one build option, the same service holds 2 000
requests a second inside a **128 MiB** container, ten runs out of ten, at a p99 of **5.5 ms** and a
peak of **61 MB**. Before the two changes that cell was six OOM kills out of six. For scale,
measured in the same cell: the JVM answers in 29.6 ms holding 105 MB, and Go in 2.3 ms holding 6
MB.

None of this is the runtime's fault in general, and the ladder says so: a bare `epoll` loop with no
Ktor at all holds the same 2 000 rps in **5 MB** at a p99 of 1.52 ms, and the rung below HTTP parsing
— raw sockets through `ktor-network` — goes slow in **0 of 12** runs. Below a thousand requests a
second on these four cores, unpatched Kotlin/Native was already the fastest of the three: 2.10 ms
against the JVM's 3.60. Everything below is about what changes between there and 2 000.

Both changes were reported upstream, and one of them is now a version number. The Ktor one —
[KTOR-9891](https://youtrack.jetbrains.com/issue/KTOR-9891) — is **fixed and released**:
[ktorio/ktor#5874](https://github.com/ktorio/ktor/pull/5874) shipped in **3.6.0** on 16 September
2026, so on that release the throughput half of this post is an upgrade and nothing else. It was
re-measured there rather than taken on trust — *On 3.6.0 as released*, below. The runtime one,
[KT-89365](https://youtrack.jetbrains.com/issue/KT-89365), was closed on 16 September as a duplicate
of [KT-74834](https://youtrack.jetbrains.com/issue/KT-74834), which asks for a way to bound the
empty allocator pages a thread keeps and has been open since January 2025. There is a runnable
reproducer for the first: [youndie/ktor-9891-repro](https://github.com/youndie/ktor-9891-repro) —
read its README before running it, because on a machine with two cores online none of this happens.

## The stand

Two dedicated hosts, four cores and 7.6 GiB each, Ubuntu 26.04, subject and generator on separate
machines over a private link. Kotlin 2.4.10 `linuxX64` release binaries, Ktor 3.5.2 with
`ktor-server-cio`, one route returning one JSON object. k6, 30 s per run, memory bounded by a cgroup
and read from it. The release check in *On 3.6.0 as released* is that same pair and that same
reproducer on 17 September 2026, with the version in the build file as the only difference between
its arms.

One methodological note, because every number below depends on it. At 2 000 rps this service has
**two speeds**: the same binary on the same machine either answers in single-digit milliseconds or in
hundreds, decided in the first five seconds of a run. Across 156 runs there is nothing in between —
every p99 is under 70 ms or over 110. A p99 from one run is therefore not a number about the program.
Everything here is scored as the share of runs that land in the slow state, twelve runs per arm,
arms rotated, with Fisher's exact test.

## Where the throughput goes

Not into the collector. A profile of a slow run puts 32–73 % of all CPU inside one function —
`io.ktor.utils.io.locks.SynchronizedObject#lock` — and the collector's own functions at one or two
per cent. Attaching gdb under load and walking every thread, all 211 frames inside that function have
the same two callers: `DefaultPool#borrow` from `HttpHeadersMap`'s constructor, and
`DefaultPool#recycle` from its `release`.

`HttpHeadersMap` — the map CIO parses every request's headers into — keeps its storage in two
**process-wide** pools, and on Kotlin/Native `DefaultPool` is a `mutableListOf` behind one lock taken
on every borrow and every recycle. So every HTTP request takes two process-wide locks at least twice
each, while the process's hundred-odd threads do the same on the same two atomics.

That lock is not a monitor. On Kotlin/Native it is a CAS loop over an `AtomicReference<LockState>`
that **allocates a new `LockState` on every attempt**, so a lost CAS is an allocation rather than a
cheap retry and the cost of an acquisition rises with the number of threads trying. On the JVM the
class is erased and `synchronized` becomes a monitor, which is why the same Ktor source has no such
behaviour there.

It shows up as sensitivity to concurrency rather than to rate. Same binary, same 2 000 rps offered,
ten runs per arm:

| Connections | Slow runs | p99 | Requests served in 30 s |
|---|---|---|---|
| 12 | 0/10 | 5.7 ms | 59 362 |
| 50 | 7/10 | 352 ms | 48 884 |
| 200 | 10/10 | 1 620 ms | 6 361 |

**The fix is a deletion, and it is not mine.** `kotlinx.atomicfu` rewrote that lock in **0.28.0** —
the owner lives in an `AtomicLong`, waiters park on a monitor, nothing is allocated. `ktor-io` already
declared atomicfu as an `api` dependency and already resolved 0.32.1, so the fixed implementation was
on the classpath of every Kotlin/Native Ktor build, behind a copy of the version it replaced.
Twelve runs per arm, four builds from the same tree differing in one dependency:

| Build | 200 conn | p99 | Requests | 50 conn | p99 |
|---|---|---|---|---|---|
| 3.5.2 as released | 12/12 slow | 1 500 ms | 7 082 | 7/12 slow | 310 ms |
| pool made lock-free | 0/12 | 11.3 ms | 59 924 | 0/12 | 7.8 ms |
| lock delegated to atomicfu | 0/12 | 8.4 ms | 59 928 | 0/12 | 6.5 ms |
| both | 0/12 | 8.8 ms | 60 053 | 0/12 | 6.4 ms |

Zero of seventy-two patched runs slow against nineteen of twenty-four stock, p = 1.7 × 10⁻¹³.
Replacing the lock alone is as good as rewriting the pool and better on p99, which is why the pull
request changes the lock. At 2 000 rps and 512 MiB the patched service answers in 6.5 ms where the
JVM answers in 15.7.

### On 3.6.0 as released

Three of those four builds are 3.5.2 with `ktor-io` swapped out, which is not an artifact anyone
will depend on, so the release was measured as an arm of its own. In 3.6.0 the class stays and its
three methods delegate to `kotlinx.atomicfu.locks.SynchronizedObject`; the copy of the lock, its
`LockState` and the native mutex pool behind it are gone from `ktor-io/posix`, and the atomicfu
resolved with it moves from 0.32.1 to 0.33.0. `DefaultPool` is untouched — the header pools are
still process-wide, the lock under them is simply no longer allocating.

Same reproducer, same stand, the two released versions interleaved run for run, twelve rounds each
at 200 connections and the same 2 000 rps offered:

| Build | Slow runs | p99 (median) | Requests served in 30 s |
|---|---|---|---|
| 3.5.2 as released | 12/12 slow | 1.50 s | 6 850 |
| 3.6.0 as released | **0/12** | 23.7 ms | 59 700 |

p = 7.4 × 10⁻⁷, and the version in the build file is the only thing that differs between the arms.

Those p99s do not line up with the table above, and a third arm in the same sweep says why: 3.5.2
with the pull request's `ktor-io` — the 8.4 ms row — rebuilt and run alongside the release on the
same day lands at 23.7 ms against the release's 24.1, 0 of 10 slow in both arms, with the ranges
overlapping everywhere. Between sweeps on this stand a fast arm's p99 moves by nearly a factor of
three; its slow-run share does not move at all. Compare arms inside one sweep, and across sweeps
compare the share.

### If you cannot move to 3.6.0

On 3.5.2 and earlier two things work. **Cap the connections an instance accepts** — the table above
is the whole argument: twelve connections go slow in 0 of 10 runs at the same offered rate, two
hundred in 10 of 10. A reverse proxy in front with a bounded keep-alive pool to the service turns
the variable you cannot control into one you can, and costs nothing else. **Or build `ktor-io`
yourself** with the three-method change and substitute it; that is what every patched number here
was measured on.

## Where the memory goes

Also not into the heap. Bounding `maxHeapBytes` at half the container limit — the application reading
`memory.max` itself, since the runtime does not — moved the peak by thirteen megabytes and survival
not at all: seven survivals of twelve at 256 MiB with the bound, with it at half, and without it.

Pooled across 413 surviving runs of every sweep taken, peak RSS against peak **thread count** gives
r = 0.994 at 2.96 MB per thread over a 10 MB intercept. The runtime source says why:
`CustomAllocator` is per-thread and holds one `FixedBlockPage` per block-size class plus a 256 KiB
next-fit page, and a thread that has touched a size class keeps that page for as long as it lives,
occupied or not. No `GC` setting counts them — they are pages, not objects.

`-Xbinary=fixedBlockPageSize` resizes them; the value is in KiB. Eight runs per arm at 50
connections:

| Allocator setting | Peak at 512 MiB | p99 | CPU | 128 MiB | Peak | p99 |
|---|---|---|---|---|---|---|
| default | 268.4 MB | 6.00 ms | 181 % | 0/8 alive | — | — |
| `fixedBlockPageSize=16` | **52.8 MB** | 5.65 ms | 179 % | **8/8** | 48.7 MB | 5.46 ms |
| `fixedBlockPageSize=4` | 39.5 MB | 7.13 ms | 183 % | 8/8 | 44.1 MB | 7.07 ms |
| `-Xallocator=std` | 24.5 MB | 9.04 ms | 219 % | 8/8 | 24.2 MB | 9.73 ms |

Fitted per build, the slope is 2.96, 0.66, 0.40 and 0.06 MB per thread. **The 16 KiB page is free**:
four times less memory at the same p99 and the same CPU — 5.65 ms against 6.00, 179 % against 181 %,
which is noise in both columns. There is no argument for the default in a service with a thread pool.

That slope is also why this table says 48.7 MB at 128 MiB and the lead says 61. Same cell, two sweeps:
this one held a median of **62 threads**, the ten-run sweep in *What to set* held **79**. Seventeen
more threads at 0.66 MB each is 11.6 MB, and the gap is 12.7. Quote the larger number when sizing a
container — RSS here is a function of how many threads the load happens to spin up, not a constant.

And the thread count is itself a setting, one layer up: capping the dispatcher the CIO engine runs
on takes the same service from 67 MB to 36 MB at 200 connections, at a lower p99 and the same CPU.
That needs a fork rather than a build option, and it is measured in
[the next post](https://kotlin.website/blog/dispatcher-parallelism-on-kotlin-native).

## What to set

Ten runs per arm, every arm serving essentially the full offered rate:

| Your limit | Connections | Set | Expect |
|---|---|---|---|
| 512 MiB and up | 200 | `fixedBlockPageSize=16` | 61 MB, 7.2 ms |
| 128 MiB | 50 | `fixedBlockPageSize=16` | 61 MB, 5.5 ms |
| 64 MiB | 50 | `fixedBlockPageSize=4` | 39 MB, 6.3 ms |
| 32 MiB | 50 | `-Xallocator=std` | 24 MB, 8.4 ms, and 34 % more CPU |

```kotlin
kotlin {
    linuxX64 {
        binaries.executable {
            binaryOption("fixedBlockPageSize", "16")
        }
    }
}
```

`-Xallocator=std` is the one real trade here: 2.3 times the p99 and 34 % more CPU at 512 MiB and 200
connections, in exchange for the lowest floor. At that floor the other two columns are Go, unmoved at
6.5 MB and 2.67 ms, and the JVM, which does not run there at all — at 64 MiB it starts, sizes its heap
to half the limit and is OOM-killed before serving a request; at 32 MiB it never reaches its ready
line. Six runs each.

All of it is on top of the lock fix, which now means Ktor 3.6.0 or newer. Without it — so on 3.5.2
and earlier — a build at 200 connections serves a ninth of the offered rate whatever its allocator is
set to.

## Limitations

`linuxX64` is the only target measured. The lock lives in `ktor-io/posix`, the source set every native
target shares, so the behaviour should be the same on Apple and MinGW targets — should be, not is.

Four cores is the only machine. The core count turned out to matter in a way I cannot explain: with
cores taken **offline**, so the runtime itself counts fewer, the service goes slow in 0 of 12 runs at
two cores and 0 of 12 at three against 15 of 20 at four. Restricting the same four-core machine to
two CPUs with a cpuset — which leaves the online count at four — does not help, and the thread count
is not the difference either. Something that follows the number of online CPUs decides it, and naming
it needs the runtime's own instrumentation rather than a load generator.

One route returning one small JSON object is the only workload. A service whose handlers allocate
more will move the numbers; the two mechanisms are about per-request framework overhead and
per-thread allocator pages, and neither depends on what the handler does.

The `fixedBlockPageSize` numbers are peaks over 30-second runs at a steady rate. Nothing here says
what a day-long process does, and page-cache behaviour is exactly the kind of thing that could differ
over hours.