Kotlin/Native in `FROM scratch`: a 593 KB image, two linker flags, one patch

A Kotlin/Native hello-world, linked statically against the host's glibc: 1 393 592 bytes, no PT_INTERP, no NEEDED entry at all. In a FROM scratch image that is 606 825 bytes to pull, and inside it getaddrinfo still resolves example.com.

That first paragraph is a hello-world, and the number in the title is its image. The same recipe applied to a real service — Ktor, sqlx4k, kotlinx.serialization, KSP, SQLite migrations — also works: it starts in scratch, runs its migrations, and answers 401 to an unauthenticated request, from an image of 5 383 000 bytes to pull against 15 542 026 for the same service on distroless/cc. Section 4 is about that service, because a hello-world cannot tell you what static linking costs.

One word about the sizes before anything else, because it is easy to get wrong and I did. docker image inspect --format '{{.Size}}' reports the compressed size — what a pull downloads, not the bytes on disk. Calibrated here on a scratch image holding 10 MB of /dev/urandom (reports 10 005 026) against one holding 10 MB of zeros (reports 11 602).

That is true with the containerd image store, which is the default for fresh installations from Docker Engine 29.0; this box runs 29.1.3, and docker info says Storage Driver: overlayfs with driver-type: io.containerd.snapshotter.v1. On the classic graphdriver the same field is the unpacked size on disk instead, so if your numbers come out several times larger than the ones here, check which store you are on before assuming anything about the binary. Every image figure below is a pull size and says so.

Everything here is one script: sborka/docs/research/static-probe. Kotlin 2.4.10, linuxX64, Ubuntu 24.04 with glibc 2.39.

1. Why this was not possible

Three separate obstructions, each with its own ticket.

One — platform.posix names libc's libraries in a klib manifest. Every Linux binary the compiler produces links against -lresolv -lm -lpthread -lutil -lcrypt -lrt, because that list is linkerOpts in the platform.posix klib's manifest. No -Xoverride-konan-properties value reaches it: that flag names konan.properties keys, and no key names that list. On a modern distribution five of the six supply nothing — against a host glibc 2.39, only libm exports anything the binary imports (ceil, floor), and libcrypt exports nothing on any version. libcrypt is simply the one that breaks first, because distributions dropped libcrypt.so.1 soonest, which is why the issue is filed under its name: KT-55643.

Two — -dynamic-linker is emitted unconditionally. -linker-option -static does not produce a static binary, because the link command undoes it twice, in flags emitted after the user's own. GccBasedLinker.finalLinkCommands adds -dynamic-linker and its value with no condition, twenty-nine lines above the +linkerArgs that brings in your flags; and linkerKonanFlags.linux_x64 carries a hardcoded -Bdynamic in the middle of its list, which switches the linker back to preferring shared libraries so that -lc resolves to a GNU ld script naming the shared libc.so.6. The first you cannot reach from a build file at all; the second you can, once you know it is there. Filed as KT-89362, with a three-line patch and its test in JetBrains/kotlin#8127.

Three — musl deadlocks in the runtime. The other way to a small image is musl, and it does not work. Building a musl sysroot from Alpine packages — including Alpine's own musl-built libstdc++.a, which is what the runtime needs — produces a binary that links cleanly and then hangs before its first println: forty-one lines of strace, no write(2) at all, three threads all asleep in FUTEX_WAIT, two of them named Main GC thread and GC Timer thread. That is KT-85658, a lost FUTEX_WAKE in GC thread coordination. With -Xbinary=gc=noop the hang goes away and is replaced by a segfault at address 0 ten syscalls in, before the first clone — so there is a second problem behind the first. The same switch on the glibc build of the same program runs clean, which is how you know the switch is not the cause.

The prediction that was wrong

The research brief behind this work predicted that even if the link succeeded, DNS would fail. The reasoning was correct and out of date: getaddrinfo under a static glibc used to dlopen the NSS modules, and an image with nothing in it has none. Since glibc 2.34, nss_files and nss_dns are compiled into libc, so a static binary resolves names with nothing on disk to help it.

That is also the second reason the sysroot the compiler ships is the problem rather than the answer. It is glibc 2.19. Even with both link-line obstructions fixed, that route would lose name resolution — it is old on the far side of the line that makes any of this work.

The claim is controlled, because "it printed ok" is not a measurement. The same image run with --network none answers dns-lookup=FAIL(rc=-3 Temporary failure in name resolution), and a name that does not exist answers FAIL(rc=-2 Name or service not known), while the /etc/hosts lookup stays ok in both. Which is worth knowing for another reason: a scratch container is not config-free. Docker and Kubernetes mount /etc/hosts and /etc/resolv.conf into every container whatever the image holds, and that is what the lookups use.

2. The recipe, on a stock 2.4.10

No patched compiler needed. Two obstructions have workarounds; the third does not apply once you are linking statically.

linuxX64 {
    binaries.executable {
        linkerOpts("-static", "--no-dynamic-linker", "-L/usr/lib/x86_64-linux-gnu")
        freeCompilerArgs.addAll(
            "-Xoverride-konan-properties=" +
                "targetSysRoot.linux_x64=/;" +
                "crtFilesLocation.linux_x64=usr/lib/x86_64-linux-gnu;" +
                "libGcc.linux_x64=usr/lib/gcc/x86_64-linux-gnu/13;" +
                "linkerGccFlags=-lgcc -lgcc_eh -lc;" +
                "linkerKonanFlags.linux_x64=-Bstatic -lstdc++ -ldl -lm -lpthread " +
                "--defsym __cxa_demangle=Konan_cxa_demangle --gc-sections",
        )
    }
}

Five properties rather than one, and every one of them earns its place:

• targetSysRoot points at the host instead of the bundled glibc 2.19, which is the whole trick;
• crtFilesLocation and libGcc still aim into the toolchain otherwise — libGcc is documented as sysroot-relative, so it is its own key rather than a subdirectory of the first;
• linkerGccFlags loses -lgcc_s, which has no static archive anywhere;
• linkerKonanFlags is the stock value with -Bdynamic removed and nothing else changed. The --defsym __cxa_demangle=Konan_cxa_demangle --gc-sections at the end is its stock tail, copied as is — the --defsym wires the runtime's own demangler into the linker, and --gc-sections drops unreferenced sections. The first draft of this recipe rewrote the key from scratch and so quietly lost --gc-sections too; that cost 224 432 bytes of binary for nothing. Print the key before you override it: grep linkerKonanFlags.linux_x64 ~/.konan/kotlin-native-prebuilt-*/konan/konan.properties, and note that its value continues onto a second line.

And two flags that are not properties. --no-dynamic-linker is what removes the PT_INTERP that -static alone leaves behind — omit it and the kernel hands your statically linked program to glibc's dynamic loader, which relocates it as though it were dynamic and segfaults with no output. -L/usr/lib/x86_64-linux-gnu is needed because with the sysroot at / the compiler emits -L/lib, -L/usr/lib, -L/lib64 and -L/usr/lib64, which is the layout of its sysroot; on a multiarch distribution every archive is somewhere else, and without this the link fails with unable to find library -lc rather than with anything about static linking.

You need libc6-dev and libstdc++-dev on the build machine. The 13 in libGcc is the gcc version — check ls /usr/lib/gcc/x86_64-linux-gnu/.

With the patch of KT-89362, --no-dynamic-linker and the linkerKonanFlags override both go away and -static means static. That is measured, not assumed: a distribution was built from the same tree twice, with and without the three lines, and the probe compiled with -linker-option -static and nothing else gives PT_INTERP=1 and exit 139 without them, PT_INTERP=0 and a clean run with them.

A caveat that belongs next to the recipe rather than at the end. -Xoverride-konan-properties pins five konan.properties keys, and they are not a stable interface. Sergey Bogolepov of JetBrains, on KT-38876 in March 2021:

Contents of konan.properties (located in $HOME/.konan/kotlin-native-.../konan) may change in every release (even in a patch release), so be sure to check it before using -Xoverride-konan-properties.

So this is a recipe to use knowingly in something you build and ship, checking the keys on every Kotlin bump — not to put in a shared convention plugin where it will break in someone else's service without anyone touching it.

3. Which base image each variant starts in

The probe, built four ways, run in four images. Sizes are pull bytes.

Two things in that table are worth reading twice.

The workaround that circulates in both tickets is --as-needed together with -Xoverride-konan-properties=linkerGccFlags=-lgcc -lgcc_eh -lc, passed around as one cure. They reach different lists. The property removes exactly one library — libgcc_s, the one linkerGccFlags contributes — and leaves libcrypt in place; only --as-needed removes the six from the manifest. So the half that does nothing for the bug this is usually filed under is the half that gets you one image smaller, from cc to base.

And exec: no such file or directory in the bottom two rows is the loader, not the binary. The binary is there; there is no /lib64/ld-linux-x86-64.so.2 to run it. That message is obstruction two as the kernel reports it.

4. What it costs, on a service rather than a hello-world

katcher is a crash-report service: Ktor CIO, sqlx4k over SQLite, kotlinx.serialization, KSP, schema migrations at start. Nothing in its sources changed — the build gained the block from section 2 and that is all.

The risk worth naming first was sqlx4k, a Rust staticlib: a static link can demand static versions of whatever it pulls in, and OpenSSL is the usual casualty. It did not. Zero missing archives, zero undefined symbols, no -lssl or -lcrypto anywhere.

Size

Static linking makes the binary bigger, not smaller — by 893 312 bytes stripped, which is the line worth putting in your notes. Where it goes: +794 480 into .text as glibc, libstdc++ and libgcc come inside, +134 880 into .rodata, +34 264 into .eh_frame, minus 93 641 of .dynsym, .dynstr and .gnu.hash that a static binary no longer needs. What collapses is not the binary, it is everything else: the 10 643 700 bytes of distroless/cc are simply gone.

strip would take another quarter off either build. It also takes every kfun: name out of the symbol table, and a Kotlin/Native stack trace is nothing but those names, so that is a trade against your crash reports rather than free.

RSS and latency

Two sessions of three alternating rounds each, 1500 requests per round, same route, same harness. The second session ran while the machine was busy, which is why its numbers are both larger and wider — and why it is quoted rather than dropped:

p95 is lower for the static build in all six rounds, by 5–7 % on the quiet box and more on the busy one. RSS after start is lower in five of the six — the one exception is a busy-box round where it came out 0.3 MB higher — which is the dynamic loader and the shared-library mappings not being there, not a large effect. The medians overlap, so the honest reading is "no penalty, with a small edge at the tail", not a speed-up.

The harness is a Python client on the same host and its absolute numbers say more about itself than about the service; and the route is a static resource, because every path that touches the database is behind authentication, so this measures the HTTP path and not sqlx4k.

Cold start on a small node

A 2-core, 5.9 GiB VPS with k0s running beside the test, image cache emptied before every round:

Both answer 401. The transfer term is derived rather than measured: the node is behind NAT from the build host, and staging a real registry pull would have meant publishing the images, so it is bytes over the node's own measured bandwidth — which was noisy enough (2.37–6.36 MB/s) to be a range, 2.3–6.3 s against 0.8–2.2 s. Added up, 4.1–8.1 s against 1.5–2.8 s; call it 2.8×, and note that the byte ratio behind it does not depend on the bandwidth at all.

One thing that was not being looked for: the binary was linked against glibc 2.39 on Ubuntu 24.04 and runs on a different distribution entirely. The rule that a build image and a runtime image have to be paired by glibc version stops applying when there is no runtime image to pair with.

5. Where this does not go

• linuxX64 only. linuxArm64 was never attempted. Nothing here is architecture-specific in principle, but "in principle" is not a measurement.
• No dlopen. nss_files and nss_dns are inside libc since 2.34, which is why name resolution survives — anything that actually loads a shared object at run time does not. If your service calls dlopen, directly or through a library, this is not for you.
• ASLR was already not there. A static non-PIE executable loses address-space randomisation of the main image — except that Kotlin/Native's Linux executables are ET_EXEC to begin with. readelf -h says Type: EXEC (Executable file) for both builds of this service, static and dynamic alike, so static linking takes nothing away here; the randomisation was never there to lose. That is worth knowing in its own right if you are hardening a service. -static-pie was not attempted, and whether the Kotlin/Native runtime survives it is an open question rather than a known answer.
• ktor-client-curl is out of scope. It bundles its own statically linked OpenSSL, which is a different linking question and was not examined. katcher uses Ktor's server with CIO.
• musl is a no, and section 1 says why. It is not a matter of finding the right flags: the runtime deadlocks in GC thread coordination before the first println, and there is a second failure behind that one. The route into a small image is static glibc, not musl.
• Windows and macOS are a different question. MingwBasedLinker and MacOSBasedLinker are separate code paths and none of this was measured against them.
• The konan.properties keys are not a stable interface, as section 2 says. Today's recipe is five pinned keys; if KT-89362 lands it becomes one flag, and this article's section 2 becomes shorter.

What would make this ordinary

Three lines in GccBasedLinker.finalLinkCommands, skipping the interpreter when the user asked for a static link — that is the patch, with a test that fails without it. After that, -linker-option -static means what it says, --no-dynamic-linker stops being folklore, and the remaining overrides are about which glibc you link against rather than about undoing the toolchain.

The platform.posix manifest is the older and more structural of the two: six libraries added to every Linux binary, five of them dead on any modern glibc, and no setting a user can reach. That one needs a decision rather than a patch, and it is KT-55643.

Reproduction, transcripts and the exact commands: sborka/docs/research/static-probe. ./experiments.sh on a Linux host with a JDK, docker and binutils produces every number above in one run; it fetches the ~1 GB Kotlin/Native toolchain on a first run and says so.