# Instrumentation PGO on Kotlin/Native

> A profile can be taken and applied on stock tools, with no fork. Turning it into a binary means going around the compiler's own LTO, which costs nothing measurable on a benchmark — and the profile returns 10–13 % there. What that leaves unanswered for a service, and the recipe.

Page: https://kotlin.website/blog/instrumentation-pgo-on-kotlin-native · Published: 2026-09-21

**A profile can be taken off a Kotlin/Native binary and applied back to its IR on stock tools,
with no fork and no patched toolchain. Building an optimised binary out of the result means
going around the compiler's own LTO — which on a dispatch benchmark costs nothing measurable,
while the profile returns 10–13 %.** What that detour does lose is internalisation: 3 335
defines against an ordinary build's 495, and a service binary 3.8× the size. Whether that costs
a *service* anything in CPU is not measured here, and it is the gap between this benchmark and a
recommendation. The rest is how, why, and the numbers.

The recipe is below and it is seven commands; it also exists as [a script that checks every
step](https://github.com/youndie/pgo-native-spike/blob/main/recipe/pgo.sh), because a PGO
pipeline that runs cleanly and applies nothing looks exactly like one that worked.

**The wall is not the linker and it is not specific to large projects.** A module carrying
profile metadata cannot pass Kotlin/Native's own LTO or its codegen step, on a single file as
much as on a service — the benchmark in this post is built around it too. The link itself
replays byte for byte.

The subject is [xyk](https://github.com/youndie/xyk), a webhook gateway with SQLite on the
request path — Kotlin/Native, Ktor, about 8 ms of CPU per request on four cores. The study,
its raw logs, the recipe as a script and the backlog with the hypotheses that were dropped are
in [pgo-native-spike](https://github.com/youndie/pgo-native-spike); the thresholds were written
down before the first measurement and the amendments carry the date and the reason each was made.

Everything here is Kotlin 2.4.20 and LLVM 21 on `linuxX64`.

## What you need

`llvm-profdata` already ships with Kotlin/Native. **`opt` does not**, and `opt` is what runs both
PGO passes. A normal install pulls the *essentials* LLVM bundle; the *dev* bundle is a separate
download named in `konan.properties`:

```bash
grep -E 'llvm\.linux_x64\.dev|dependenciesUrl' \
  ~/.konan/kotlin-native-prebuilt-linux-x86_64-2.4.20/konan/konan.properties
# llvm.linux_x64.dev=llvm-21-x86_64-linux-dev-116
# dependenciesUrl = https://download.jetbrains.com/kotlin/native

curl -fsSL -O https://download.jetbrains.com/kotlin/native/resources/llvm/21-x86_64-linux/llvm-21-x86_64-linux-dev-116.tar.gz
tar xzf llvm-21-x86_64-linux-dev-116.tar.gz -C ~/.konan/dependencies/
```

It carries `opt`, `clang`, `llvm-ar` and `libclang_rt.profile.a`, and `opt --print-passes` lists
`pgo-instr-gen`, `pgo-instr-use`, `instrprof` and `pgo-icall-prom`.

## The recipe

```bash
KN=~/.konan/kotlin-native-prebuilt-linux-x86_64-2.4.20
L=~/.konan/dependencies/llvm-21-x86_64-linux-dev-116

# 1. Compile normally and keep the IR at the point where Kotlin code and the runtime are one
#    module. The directory must already exist: kotlinc warns and exits 0 if it does not.
mkdir -p ir
$KN/bin/kotlinc-native -opt -Xsave-llvm-ir-directory=$PWD/ir \
    -Xsave-llvm-ir-after=LinkBitcodeDependencies -o base app.kt

# 2. Instrument.
$L/bin/opt -passes="pgo-instr-gen,instrprof" ir/out.LinkBitcodeDependencies.ll -o instr.bc

# 3. Rebuild the profile runtime with an IR-level version variable (see below).
printf 'long long __llvm_profile_raw_version = (1LL << 56) | 10;\n' > version.c
$L/bin/clang -c -o version.o version.c
mkdir rtx && cd rtx && $L/bin/llvm-ar x $L/lib/clang/21/lib/x86_64-unknown-linux-gnu/libclang_rt.profile.a
rm -f InstrProfilingVersionVar.c.o && cp ../version.o . \
  && $L/bin/llvm-ar rcs ../libprofile-ir.a *.o && cd ..

# 4. Link the instrumented binary. -u__llvm_profile_runtime is load-bearing (see below).
$KN/bin/kotlinc-native -opt -Xcompile-from-bitcode=$PWD/instr.bc \
    -linker-option -u__llvm_profile_runtime -linker-option $PWD/libprofile-ir.a -o train

# 5. Train, and merge.
LLVM_PROFILE_FILE=$PWD/p-%p.profraw ./train.kexe
$L/bin/llvm-profdata merge -output=p.profdata p-*.profraw
$L/bin/llvm-profdata show p.profdata | grep -i 'instrumentation level'   # must say IR

# 6. Apply.
$L/bin/opt -passes="pgo-instr-use" -pgo-test-profile-file=$PWD/p.profdata \
    ir/out.LinkBitcodeDependencies.ll -o applied.bc
```

Step 6 leaves annotated IR, not a binary. Getting a binary out of it needs a seventh command,
and **the `CG Profile` collision described below fires here too** — so the LTO pipelines have to
be neutered and the optimisation done in `opt` instead:

```bash
# 7. Optimise externally, because kotlinc's own pipelines cannot run over a module that carries
#    profile metadata, and hand it back for codegen and linking.
$L/bin/opt -passes="default<O3>" applied.bc -o optimised.bc
$KN/bin/kotlinc-native -opt -Xcompile-from-bitcode=$PWD/optimised.bc     -Xllvm-module-passes=verify -Xllvm-lto-passes=verify -o a2
```

**Arms still have to be compared along one route**, which is what the numbers below do — but
the route itself turns out to be nearly free on this benchmark: 1.075 ns/op against the ordinary
build's 1.063 on the interface call. An earlier version of this post put that cost at 57–68 %,
which was a mistake: those two figures came from different machines, and the study they came
from did not record the host.

**Safepoints are the countable difference between the routes, and they are not the cost.**
Kotlin/Native's own pipeline strips the polls; `default<O3>` in `opt` does not, because the
passes that do it are not in it:

| same source, final IR | defines | `call … safePoint` |
|---|---:|---:|
| ordinary build | 495 | **25** |
| step-7 route | 3 335 | **4 189** |

That looks like an explanation and it is not one. Removing all 4 179 remaining poll sites from
the module and rebuilding changed the no-dispatch control by nothing measurable — 0.470 ns/op
against 0.447. Most of those polls are in stdlib the benchmark never executes, and the hot loop
does not pay for them. **The difference is real, countable and not the mechanism**, which is the
sort of thing a toggle establishes and a symbol count does not.

## The two steps that are load-bearing

Both were checked by removing them, because a PGO pipeline that runs cleanly and applies nothing
looks exactly like one that worked.

**`-u__llvm_profile_runtime`.** Without it the binary links, runs, exits successfully and writes
**no `.profraw` at all**. Nothing warns. The flag forces in the registration object that the
profile runtime's constructors hang off.

**The replaced version variable.** `libclang_rt.profile.a` ships `InstrProfilingVersionVar.c.o`
declaring the raw profile version as front-end. Leave it and the profile merges as
`Instrumentation level: Front-end`, and `pgo-instr-use` then refuses it outright:

```text
error: p.profdata: Not an IR level instrumentation profile
```

Replacing that one object with a one-line C file is the whole fix.

## Where it stops on a real service

The recipe above ends at a binary. Where a service and a single-file program part company is
step 7 — the workaround that makes that binary possible is the thing a service cannot afford.

**`-Xcompile-from-bitcode` has no klib graph.** `DependenciesTracker` builds the native
dependency list from the klib graph during a normal compilation, and a resume from bitcode has
none — so every native symbol a klib would have contributed goes undefined. Supplying archives by
hand is unbounded: adding librdkafka's five moved the failure on to sqlx4k's symbols.

The fix is not to resume from bitcode at all. `-Xverbose-phases=Linker` prints the whole `ld.lld`
invocation and `-Xtemporary-files-dir` keeps the object it refers to, so the link can be replayed
with a different object substituted:

```bash
$KN/bin/kotlinc-native -opt -Xtemporary-files-dir=$PWD/tmp -Xverbose-phases=Linker \
    -o base app.kt 2>&1 | grep -oE '/[^ ]*ld\.lld .*' > link.cmd
```

Replaying that line with the compiler's own object reproduces the ordinary build **byte for
byte** — on a 20 MB statically linked service binary, not only on a toy. With the instrumented
object substituted it produces a working training binary, because instrumented bitcode carries no
profile metadata and kotlinc runs its own full pipeline over it.

**The use side is where it stops.** A module that *does* carry profile metadata makes kotlinc
emit the `CG Profile` module flag twice — once from its LTO pipeline, once from the `clang++`
codegen step — and reject its own module:

```text
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
```

Neither copy comes from the input: `opt -passes=pgo-instr-use` emits no such flag. The only
`-Xllvm-lto-passes` value that avoids the collision is one that does no LTO — which is exactly
the workaround step 7 uses on the benchmark.

**So the service arm can be built; it just cannot be built in a form worth measuring against the
shipped binary.** Kotlin/Native's own LTO internalises and dead-strips 3 027 defines down to 411,
and the externally optimised binary came out 3.8× larger. On the benchmark that trade is
acceptable because both arms take the identical route and the question is whether the mechanism
fires. On the service the question is what PGO adds to *the binary that ships*, and a 3.8×
binary is not that binary.

**The same sentence covers both, and an earlier version of this post did not say it.** Comparing
two arms along one route isolates the mechanism and says nothing about what PGO would add to the
production pipeline. That is true of the benchmark numbers above as much as of a service arm —
the difference is only that on the benchmark isolating the mechanism was the whole question.

## What the profile is worth

Two independent measurements, because "PGO helps" is not a number.

**Indirect call promotion, on a microbenchmark.** Eight implementors of an interface so the
compiler's closed-world devirtualisation cannot resolve the site by itself, receivers drawn from
a controlled distribution, nine interleaved rounds, 99 % intervals:

Three builds, nine interleaved rounds, 99 % intervals, one machine and one session — which also
answers the question a two-arm table cannot: **is the PGO binary faster than one built the
ordinary way, with no detour at all?**

| build | interface call | virtual call | no dispatch |
|---|---:|---:|---:|
| ordinary, no PGO | 1.066 ± 0.025 | 0.804 ± 0.031 | 0.486 ± 0.029 |
| step-7 route, no profile | 1.071 ± 0.016 | 0.806 ± 0.028 | 0.494 ± 0.050 |
| **step-7 route, with profile** | **0.959 ± 0.026** | **0.699 ± 0.036** | 0.498 ± 0.056 |

**The detour is free within resolution** — every one of its three intervals overlaps the
ordinary build's. **And the profile is worth −10.0 % on the interface call and −13.1 % on the
virtual call against an ordinary build**, both separating, while the no-dispatch control
separates from nothing at all. That last clause is what makes the other two readable.

So on this benchmark PGO is a win, and the detour is not a tax you pay for it.

An earlier version of this post reported the opposite — that an ordinary build was 34 % faster —
from three non-interleaved runs whose numbers came off two different machines. The correction is
why the table above says how it was measured.

**The control of known outcome sits in an earlier run and is quoted as a ratio only.** With each
arm trained on its own distribution, eight receivers in rotation gained nothing — +5.2 % and
+3.5 %, intervals overlapping — which is what the measured `icp-remaining-percent-threshold = 30`
predicts at 12.5 % per receiver. Those runs are from a machine the log does not name, so only the
ratio is used; the levels above supersede theirs.

The IR carries the expected shape at those sites — a guard against the dominant target with the
callee's body inlined behind it — six such sites against zero without the profile. The uniform
row is the control: at 12.5 % per receiver nothing clears the measured
`icp-remaining-percent-threshold = 30`, and nothing is promoted.

**That is 0.11 ns per promoted call** — 0.113 on the interface call and 0.108 on the virtual
one — and it is the bound to carry into a service **for promotion alone**, with a callee whose
body is a single addition. It says nothing about code layout, hot/cold splitting or
hotness-scaled inlining thresholds, which are separate PGO effects:

| to move | needs |
|---|---|
| 1 % of a 1 ms request | ~90 000 promoted calls |
| 1 % of a 7.8 ms request | ~710 000 promoted calls |

(An earlier version of this post said 0.16–0.18 ns and put those counts at 60 000 and 460 000.
Both were derived from the superseded levels.)

**And the share of CPU PGO can touch, on the service.** Sampled with `perf` at 50–70 % of each
endpoint's own saturation, attributed by symbol with
[a grammar that has its own control](https://github.com/youndie/pgo-native-spike/blob/main/scripts/attribution.py):

| endpoint | Kotlin self | Kotlin + runtime |
|---|---:|---:|
| `GET /health/live` | 32.55 % | 50.38 % |
| `POST /hooks/{id}` | 21.22 % | 32.33 % |
| `GET /api/events` | 16.40 % | 37.13 % |
| `GET /journal` | 13.84 % | 41.24 % |

The gradient runs the wrong way for this: the endpoint that parses nothing and touches no
database has twice the Kotlin share of the one that renders a page. At a 20–40 % bucket, a
5–15 % return on the code PGO touches is 1–6 % of a request, and the difference two interleaved
arms can resolve on this stand is 5 %.

**That 5–15 % is a prior, not a measurement** — it is the range this study wrote down for C and
C++ servers before starting, and it is the one number here that was not measured. Nothing in the arithmetic
above improves if the real figure is higher, because the bucket is what bounds it.

For scale, on the same service and the same stand, one build property — Kotlin/Native's paged
allocator, which is the platform default — is worth **19.01 % ± 2.33 %** of CPU per request.

**That is not 19 % lying on the floor, and the pinned build turns the allocator off on purpose.**
The paged allocator keeps a page per size class *per thread* for the life of the thread, so
resident memory follows thread count rather than live heap: under a 64 MiB limit this service
survived 1 run in 10 with it and 10 in 10 without, which is what
[Ktor on Kotlin/Native under a container limit](https://kotlin.website/blog/under-a-container-limit) is about. The
honest form is that the memory criterion costs this service a fifth of its request CPU. What
nobody has measured is whether the paged allocator fits that limit with fewer threads — if it
does, the trade dissolves.

Either way, machine-code quality is not where this service spends its time.

## How long a profile lives

A profile is keyed by function name and CFG hash, so the question is what a commit costs. Trained
on one revision, applied to earlier ones, counting both functions and the counters they carry:

| distance from the trained revision | retained, functions | retained, counter weight |
|---|---:|---:|
| build-only commit, no source change | 99.965 % | **100.000 %** |
| two commits | 99.896 % | **99.999 %** |
| three commits | 99.584 % | **99.968 %** |
| four commits, across a framework minor version | 96.460 % | **99.683 %** |

Counting functions overstates the loss, because what stops matching is cold. That last row is a
single version transition and is not a rate for framework bumps in general.

**What moves is not names.** Of the functions that disappear by name, most are generated lambdas
inside the module whose code actually changed, and together they carry 19 counters out of 1.46
billion. The expensive losses are CFG hash mismatches in *shared* code: one stdlib function,
`kotlinx.cinterop.DeferScope#executeAllDeferred`, accounted for 442 333 of 468 374 lost counters
with its own source unchanged.

The reason is worth knowing on its own. Kotlin/Native devirtualises on a closed world, and
`invoke` on the deferred lambdas becomes a chain of `kclass` guards whose arm count tracks the
set of implementations reachable **in the whole program**:

```llvm
; two reachable defer-lambdas — a guard, then two direct calls
%8 = icmp eq ptr %7, @"kclass:…getAddressInfo$$inlined$memScoped$1#internal"
br i1 %8, label %when_case1, label %when_next2
  call void @"kfun:…getAddressInfo$$inlined$memScoped$1.invoke#internal"
  call void @"kfun:kotlinx.cinterop.refTo$$inlined$usingPinned$1…invoke#internal"

; one reachable defer-lambda — one direct call, no comparison at all
  call void @"kfun:…getAddressInfo$$inlined$memScoped$1.invoke#internal"
```

Isolated on two builds of one revision differing only by an added `defer { }` in application
code: the binary grows 984 bytes and two untouched stdlib symbols move —
`kotlinx.cinterop.ArenaBase#clearImpl` from 283 to 331 bytes, and the `usingPinned` getter lambda
from 223 to 297.

**The difference from C and C++ is pass order, not the absence of the transformation.** C++ has
the same class of whole-program rewriting through LTO and `-fwhole-program-vtables`. What differs
is where the profile takes its key: LLVM hashes early, before the inliner and before
whole-program devirtualisation, so a function's own source fixes its hash. Kotlin/Native
devirtualises in the frontend, before LLVM IR exists at all, so **every instrumentation point
reachable through `opt` is downstream of it**. A profile hash here is a property of the whole
program's composition rather than of the function's source, and no choice of instrumentation
point changes that.

## The stand

Two Hetzner machines, 4 vCPU each, Ubuntu with glibc 2.43: the subject on one, `k6` on the other,
never co-located. The unit is **microseconds of CPU per request**, read from the subject's own
process accounting over a window in which the offered rate is shown to be met and nothing is
dropped.

Arms are interleaved round by round and the first round is discarded. Repeating the whole
protocol with the *same binary* on both arms puts the paired standard deviation at **2.92 % per
round**, which is what any two-arm difference is measured against: at eight counted rounds the
resolvable difference is 5 %.

Two cautions from running it. That 2.92 % was characterised on a **statically linked** binary; a
dynamically linked one ran at roughly twice the spread, and disabling ASLR did not clearly
account for the gap. And the microbenchmark numbers above are nanoseconds per operation in a
loop — they are about dispatch, not about a service, which is exactly why the per-call bound is
quoted next to them.

## Limitations

**No service arm was measured, and the reason is narrower than "it cannot be built".** It can be
built, by the same step-7 workaround the benchmark uses. What cannot be had is an arm comparable
to the shipped binary, because that workaround discards Kotlin/Native's own LTO. The ceiling and
the per-call bound predict an effect below this stand's resolution anyway; that is a prediction
and not a measurement, and it is why the weakened arm was not run.

**What promotion is worth under Kotlin/Native's own pipeline is unknown.** Every figure here is
from the external-LTO route. Without internalisation LLVM's inliner sees a different program, and
the native LTO may already capture part of what promotion captures here — so 11-13 % is the
mechanism's effect *on that route*, not an increment to a production build. The `CG Profile` wall
is precisely what stops anyone measuring the increment.

**A second way round the wall, also untried: rebuild the internalisation list.** What the detour
loses is not speed on a benchmark — it is internalisation, and with it the 3 335 defines against
an ordinary build's 495, and a service binary 3.8× the size. That list is recoverable: dump the
IR of an ordinary build after its own LTO, take the defines with external linkage, and hand them
to `opt -passes='internalize,globaldce,default<O3>' -internalize-public-api-list=…`. If the
resulting binary matches an ordinary one in size, the detour stops being a detour and a service
arm becomes measurable. One thing to check first: Kotlin/Native may run `lto<O3>` rather than
`default<O3>`, which its own pipeline string would show.

**`linuxX64` only.** Nothing here was run on macOS or on an Apple target, and the profile runtime
is rebuilt from a path with `x86_64-unknown-linux-gnu` in it.

**The `CG Profile` collision looks avoidable, and I have not tried it.** `clang` enables the
pass only with the integrated assembler —
[`BackendUtil.cpp`](https://github.com/llvm/llvm-project/blob/242996efee4aaa30dd0ddb4359675b8bc103294a/clang/lib/CodeGen/BackendUtil.cpp#L908),
`PTO.CallGraphProfile = !CodeGenOpts.DisableIntegratedAS`, with a comment saying non-integrated
assemblers do not recognise the `.cgprofile` section. So `-fno-integrated-as` on the codegen step
should remove the second copy of the flag. The price is a system `as` from binutils and building
both arms the same way, and **whether it works through kotlinc on Kotlin/Native is untested** —
what is established is only that the mechanism in clang is this one. The blunter lever is
`-Xdisable-phases`, which exists, but `-Xlist-phases` printed nothing on 2.4.20, so the phase
name and its cost are both unknown.

**The static link and the profile runtime are incompatible as shipped.** `-static` plus the
profile runtime fails with `undefined hidden symbol: _DYNAMIC`. Only the *training* binary needs
the runtime, and a profile is keyed by name and CFG hash rather than by linkage, so training
dynamically and applying to the pinned module is sound — but it has to be done deliberately.

**One subject, one workload.** Every share above comes from one service with SQLite on the
request path. The recipe transfers; the shares do not.