Instrumentation PGO on Kotlin/Native

A profile can be taken off a Kotlin/Native binary and applied back to its IR on stock tools, with no fork and no patched toolchain. Building an optimised binary out of the result means going around the compiler's own LTO — which on a dispatch benchmark costs nothing measurable, while the profile returns 10–13 %. What that detour does lose is internalisation: 3 335 defines against an ordinary build's 495, and a service binary 3.8× the size. Whether that costs a service anything in CPU is not measured here, and it is the gap between this benchmark and a recommendation. The rest is how, why, and the numbers.

The recipe is below and it is seven commands; it also exists as a script that checks every step, because a PGO pipeline that runs cleanly and applies nothing looks exactly like one that worked.

The wall is not the linker and it is not specific to large projects. A module carrying profile metadata cannot pass Kotlin/Native's own LTO or its codegen step, on a single file as much as on a service — the benchmark in this post is built around it too. The link itself replays byte for byte.

The subject is xyk, a webhook gateway with SQLite on the request path — Kotlin/Native, Ktor, about 8 ms of CPU per request on four cores. The study, its raw logs, the recipe as a script and the backlog with the hypotheses that were dropped are in pgo-native-spike; the thresholds were written down before the first measurement and the amendments carry the date and the reason each was made.

Everything here is Kotlin 2.4.20 and LLVM 21 on linuxX64.

What you need

llvm-profdata already ships with Kotlin/Native. opt does not, and opt is what runs both PGO passes. A normal install pulls the essentials LLVM bundle; the dev bundle is a separate download named in konan.properties:

grep -E 'llvm\.linux_x64\.dev|dependenciesUrl' \
  ~/.konan/kotlin-native-prebuilt-linux-x86_64-2.4.20/konan/konan.properties
# llvm.linux_x64.dev=llvm-21-x86_64-linux-dev-116
# dependenciesUrl = https://download.jetbrains.com/kotlin/native

curl -fsSL -O https://download.jetbrains.com/kotlin/native/resources/llvm/21-x86_64-linux/llvm-21-x86_64-linux-dev-116.tar.gz
tar xzf llvm-21-x86_64-linux-dev-116.tar.gz -C ~/.konan/dependencies/

It carries opt, clang, llvm-ar and libclang_rt.profile.a, and opt --print-passes lists pgo-instr-gen, pgo-instr-use, instrprof and pgo-icall-prom.

The recipe

KN=~/.konan/kotlin-native-prebuilt-linux-x86_64-2.4.20
L=~/.konan/dependencies/llvm-21-x86_64-linux-dev-116

# 1. Compile normally and keep the IR at the point where Kotlin code and the runtime are one
#    module. The directory must already exist: kotlinc warns and exits 0 if it does not.
mkdir -p ir
$KN/bin/kotlinc-native -opt -Xsave-llvm-ir-directory=$PWD/ir \
    -Xsave-llvm-ir-after=LinkBitcodeDependencies -o base app.kt

# 2. Instrument.
$L/bin/opt -passes="pgo-instr-gen,instrprof" ir/out.LinkBitcodeDependencies.ll -o instr.bc

# 3. Rebuild the profile runtime with an IR-level version variable (see below).
printf 'long long __llvm_profile_raw_version = (1LL << 56) | 10;\n' > version.c
$L/bin/clang -c -o version.o version.c
mkdir rtx && cd rtx && $L/bin/llvm-ar x $L/lib/clang/21/lib/x86_64-unknown-linux-gnu/libclang_rt.profile.a
rm -f InstrProfilingVersionVar.c.o && cp ../version.o . \
  && $L/bin/llvm-ar rcs ../libprofile-ir.a *.o && cd ..

# 4. Link the instrumented binary. -u__llvm_profile_runtime is load-bearing (see below).
$KN/bin/kotlinc-native -opt -Xcompile-from-bitcode=$PWD/instr.bc \
    -linker-option -u__llvm_profile_runtime -linker-option $PWD/libprofile-ir.a -o train

# 5. Train, and merge.
LLVM_PROFILE_FILE=$PWD/p-%p.profraw ./train.kexe
$L/bin/llvm-profdata merge -output=p.profdata p-*.profraw
$L/bin/llvm-profdata show p.profdata | grep -i 'instrumentation level'   # must say IR

# 6. Apply.
$L/bin/opt -passes="pgo-instr-use" -pgo-test-profile-file=$PWD/p.profdata \
    ir/out.LinkBitcodeDependencies.ll -o applied.bc

Step 6 leaves annotated IR, not a binary. Getting a binary out of it needs a seventh command, and the CG Profile collision described below fires here too — so the LTO pipelines have to be neutered and the optimisation done in opt instead:

# 7. Optimise externally, because kotlinc's own pipelines cannot run over a module that carries
#    profile metadata, and hand it back for codegen and linking.
$L/bin/opt -passes="default<O3>" applied.bc -o optimised.bc
$KN/bin/kotlinc-native -opt -Xcompile-from-bitcode=$PWD/optimised.bc     -Xllvm-module-passes=verify -Xllvm-lto-passes=verify -o a2

Arms still have to be compared along one route, which is what the numbers below do — but the route itself turns out to be nearly free on this benchmark: 1.075 ns/op against the ordinary build's 1.063 on the interface call. An earlier version of this post put that cost at 57–68 %, which was a mistake: those two figures came from different machines, and the study they came from did not record the host.

Safepoints are the countable difference between the routes, and they are not the cost. Kotlin/Native's own pipeline strips the polls; default<O3> in opt does not, because the passes that do it are not in it:

That looks like an explanation and it is not one. Removing all 4 179 remaining poll sites from the module and rebuilding changed the no-dispatch control by nothing measurable — 0.470 ns/op against 0.447. Most of those polls are in stdlib the benchmark never executes, and the hot loop does not pay for them. The difference is real, countable and not the mechanism, which is the sort of thing a toggle establishes and a symbol count does not.

The two steps that are load-bearing

Both were checked by removing them, because a PGO pipeline that runs cleanly and applies nothing looks exactly like one that worked.

-u__llvm_profile_runtime. Without it the binary links, runs, exits successfully and writes no .profraw at all. Nothing warns. The flag forces in the registration object that the profile runtime's constructors hang off.

The replaced version variable. libclang_rt.profile.a ships InstrProfilingVersionVar.c.o declaring the raw profile version as front-end. Leave it and the profile merges as Instrumentation level: Front-end, and pgo-instr-use then refuses it outright:

error: p.profdata: Not an IR level instrumentation profile

Replacing that one object with a one-line C file is the whole fix.

Where it stops on a real service

The recipe above ends at a binary. Where a service and a single-file program part company is step 7 — the workaround that makes that binary possible is the thing a service cannot afford.

-Xcompile-from-bitcode has no klib graph. DependenciesTracker builds the native dependency list from the klib graph during a normal compilation, and a resume from bitcode has none — so every native symbol a klib would have contributed goes undefined. Supplying archives by hand is unbounded: adding librdkafka's five moved the failure on to sqlx4k's symbols.

The fix is not to resume from bitcode at all. -Xverbose-phases=Linker prints the whole ld.lld invocation and -Xtemporary-files-dir keeps the object it refers to, so the link can be replayed with a different object substituted:

$KN/bin/kotlinc-native -opt -Xtemporary-files-dir=$PWD/tmp -Xverbose-phases=Linker \
    -o base app.kt 2>&1 | grep -oE '/[^ ]*ld\.lld .*' > link.cmd

Replaying that line with the compiler's own object reproduces the ordinary build byte for byte — on a 20 MB statically linked service binary, not only on a toy. With the instrumented object substituted it produces a working training binary, because instrumented bitcode carries no profile metadata and kotlinc runs its own full pipeline over it.

The use side is where it stops. A module that does carry profile metadata makes kotlinc emit the CG Profile module flag twice — once from its LTO pipeline, once from the clang++ codegen step — and reject its own module:

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"

Neither copy comes from the input: opt -passes=pgo-instr-use emits no such flag. The only -Xllvm-lto-passes value that avoids the collision is one that does no LTO — which is exactly the workaround step 7 uses on the benchmark.

So the service arm can be built; it just cannot be built in a form worth measuring against the shipped binary. Kotlin/Native's own LTO internalises and dead-strips 3 027 defines down to 411, and the externally optimised binary came out 3.8× larger. On the benchmark that trade is acceptable because both arms take the identical route and the question is whether the mechanism fires. On the service the question is what PGO adds to the binary that ships, and a 3.8× binary is not that binary.

The same sentence covers both, and an earlier version of this post did not say it. Comparing two arms along one route isolates the mechanism and says nothing about what PGO would add to the production pipeline. That is true of the benchmark numbers above as much as of a service arm — the difference is only that on the benchmark isolating the mechanism was the whole question.

What the profile is worth

Two independent measurements, because "PGO helps" is not a number.

Indirect call promotion, on a microbenchmark. Eight implementors of an interface so the compiler's closed-world devirtualisation cannot resolve the site by itself, receivers drawn from a controlled distribution, nine interleaved rounds, 99 % intervals:

Three builds, nine interleaved rounds, 99 % intervals, one machine and one session — which also answers the question a two-arm table cannot: is the PGO binary faster than one built the ordinary way, with no detour at all?

The detour is free within resolution — every one of its three intervals overlaps the ordinary build's. And the profile is worth −10.0 % on the interface call and −13.1 % on the virtual call against an ordinary build, both separating, while the no-dispatch control separates from nothing at all. That last clause is what makes the other two readable.

So on this benchmark PGO is a win, and the detour is not a tax you pay for it.

An earlier version of this post reported the opposite — that an ordinary build was 34 % faster — from three non-interleaved runs whose numbers came off two different machines. The correction is why the table above says how it was measured.

The control of known outcome sits in an earlier run and is quoted as a ratio only. With each arm trained on its own distribution, eight receivers in rotation gained nothing — +5.2 % and +3.5 %, intervals overlapping — which is what the measured icp-remaining-percent-threshold = 30 predicts at 12.5 % per receiver. Those runs are from a machine the log does not name, so only the ratio is used; the levels above supersede theirs.

The IR carries the expected shape at those sites — a guard against the dominant target with the callee's body inlined behind it — six such sites against zero without the profile. The uniform row is the control: at 12.5 % per receiver nothing clears the measured icp-remaining-percent-threshold = 30, and nothing is promoted.

That is 0.11 ns per promoted call — 0.113 on the interface call and 0.108 on the virtual one — and it is the bound to carry into a service for promotion alone, with a callee whose body is a single addition. It says nothing about code layout, hot/cold splitting or hotness-scaled inlining thresholds, which are separate PGO effects:

(An earlier version of this post said 0.16–0.18 ns and put those counts at 60 000 and 460 000. Both were derived from the superseded levels.)

And the share of CPU PGO can touch, on the service. Sampled with perf at 50–70 % of each endpoint's own saturation, attributed by symbol with a grammar that has its own control:

The gradient runs the wrong way for this: the endpoint that parses nothing and touches no database has twice the Kotlin share of the one that renders a page. At a 20–40 % bucket, a 5–15 % return on the code PGO touches is 1–6 % of a request, and the difference two interleaved arms can resolve on this stand is 5 %.

That 5–15 % is a prior, not a measurement — it is the range this study wrote down for C and C++ servers before starting, and it is the one number here that was not measured. Nothing in the arithmetic above improves if the real figure is higher, because the bucket is what bounds it.

For scale, on the same service and the same stand, one build property — Kotlin/Native's paged allocator, which is the platform default — is worth 19.01 % ± 2.33 % of CPU per request.

That is not 19 % lying on the floor, and the pinned build turns the allocator off on purpose. The paged allocator keeps a page per size class per thread for the life of the thread, so resident memory follows thread count rather than live heap: under a 64 MiB limit this service survived 1 run in 10 with it and 10 in 10 without, which is what Ktor on Kotlin/Native under a container limit is about. The honest form is that the memory criterion costs this service a fifth of its request CPU. What nobody has measured is whether the paged allocator fits that limit with fewer threads — if it does, the trade dissolves.

Either way, machine-code quality is not where this service spends its time.

How long a profile lives

A profile is keyed by function name and CFG hash, so the question is what a commit costs. Trained on one revision, applied to earlier ones, counting both functions and the counters they carry:

Counting functions overstates the loss, because what stops matching is cold. That last row is a single version transition and is not a rate for framework bumps in general.

What moves is not names. Of the functions that disappear by name, most are generated lambdas inside the module whose code actually changed, and together they carry 19 counters out of 1.46 billion. The expensive losses are CFG hash mismatches in shared code: one stdlib function, kotlinx.cinterop.DeferScope#executeAllDeferred, accounted for 442 333 of 468 374 lost counters with its own source unchanged.

The reason is worth knowing on its own. Kotlin/Native devirtualises on a closed world, and invoke on the deferred lambdas becomes a chain of kclass guards whose arm count tracks the set of implementations reachable in the whole program:

; two reachable defer-lambdas — a guard, then two direct calls
%8 = icmp eq ptr %7, @"kclass:…getAddressInfo$$inlined$memScoped$1#internal"
br i1 %8, label %when_case1, label %when_next2
  call void @"kfun:…getAddressInfo$$inlined$memScoped$1.invoke#internal"
  call void @"kfun:kotlinx.cinterop.refTo$$inlined$usingPinned$1…invoke#internal"

; one reachable defer-lambda — one direct call, no comparison at all
  call void @"kfun:…getAddressInfo$$inlined$memScoped$1.invoke#internal"

Isolated on two builds of one revision differing only by an added defer { } in application code: the binary grows 984 bytes and two untouched stdlib symbols move — kotlinx.cinterop.ArenaBase#clearImpl from 283 to 331 bytes, and the usingPinned getter lambda from 223 to 297.

The difference from C and C++ is pass order, not the absence of the transformation. C++ has the same class of whole-program rewriting through LTO and -fwhole-program-vtables. What differs is where the profile takes its key: LLVM hashes early, before the inliner and before whole-program devirtualisation, so a function's own source fixes its hash. Kotlin/Native devirtualises in the frontend, before LLVM IR exists at all, so every instrumentation point reachable through opt is downstream of it. A profile hash here is a property of the whole program's composition rather than of the function's source, and no choice of instrumentation point changes that.

The stand

Two Hetzner machines, 4 vCPU each, Ubuntu with glibc 2.43: the subject on one, k6 on the other, never co-located. The unit is microseconds of CPU per request, read from the subject's own process accounting over a window in which the offered rate is shown to be met and nothing is dropped.

Arms are interleaved round by round and the first round is discarded. Repeating the whole protocol with the same binary on both arms puts the paired standard deviation at 2.92 % per round, which is what any two-arm difference is measured against: at eight counted rounds the resolvable difference is 5 %.

Two cautions from running it. That 2.92 % was characterised on a statically linked binary; a dynamically linked one ran at roughly twice the spread, and disabling ASLR did not clearly account for the gap. And the microbenchmark numbers above are nanoseconds per operation in a loop — they are about dispatch, not about a service, which is exactly why the per-call bound is quoted next to them.

Limitations

No service arm was measured, and the reason is narrower than "it cannot be built". It can be built, by the same step-7 workaround the benchmark uses. What cannot be had is an arm comparable to the shipped binary, because that workaround discards Kotlin/Native's own LTO. The ceiling and the per-call bound predict an effect below this stand's resolution anyway; that is a prediction and not a measurement, and it is why the weakened arm was not run.

What promotion is worth under Kotlin/Native's own pipeline is unknown. Every figure here is from the external-LTO route. Without internalisation LLVM's inliner sees a different program, and the native LTO may already capture part of what promotion captures here — so 11-13 % is the mechanism's effect on that route, not an increment to a production build. The CG Profile wall is precisely what stops anyone measuring the increment.

A second way round the wall, also untried: rebuild the internalisation list. What the detour loses is not speed on a benchmark — it is internalisation, and with it the 3 335 defines against an ordinary build's 495, and a service binary 3.8× the size. That list is recoverable: dump the IR of an ordinary build after its own LTO, take the defines with external linkage, and hand them to opt -passes='internalize,globaldce,default<O3>' -internalize-public-api-list=…. If the resulting binary matches an ordinary one in size, the detour stops being a detour and a service arm becomes measurable. One thing to check first: Kotlin/Native may run lto<O3> rather than default<O3>, which its own pipeline string would show.

linuxX64 only. Nothing here was run on macOS or on an Apple target, and the profile runtime is rebuilt from a path with x86_64-unknown-linux-gnu in it.

The CG Profile collision looks avoidable, and I have not tried it. clang enables the pass only with the integrated assembler — BackendUtil.cpp, PTO.CallGraphProfile = !CodeGenOpts.DisableIntegratedAS, with a comment saying non-integrated assemblers do not recognise the .cgprofile section. So -fno-integrated-as on the codegen step should remove the second copy of the flag. The price is a system as from binutils and building both arms the same way, and whether it works through kotlinc on Kotlin/Native is untested — what is established is only that the mechanism in clang is this one. The blunter lever is -Xdisable-phases, which exists, but -Xlist-phases printed nothing on 2.4.20, so the phase name and its cost are both unknown.

The static link and the profile runtime are incompatible as shipped. -static plus the profile runtime fails with undefined hidden symbol: _DYNAMIC. Only the training binary needs the runtime, and a profile is keyed by name and CFG hash rather than by linkage, so training dynamically and applying to the pinned module is sound — but it has to be done deliberately.

One subject, one workload. Every share above comes from one service with SQLite on the request path. The recipe transfers; the shares do not.