Leyden AOT cache for a plain Kotlin/JVM service: what it gives, what it costs, and the three things Spring does for you that you'll do yourself

This is the how-to I could not find when I needed it: a Ktor service on the plain application plugin, JDK 25, and the question of whether Project Leyden's AOT cache is worth an afternoon. Spring Boot and Quarkus answer it with a flag. Everything else gets two JVM options and a list of ways the result stops working without telling anyone. The numbers below are from a reference service and from a stand, each with its methodology next to it; the commands are the ones the plugin at the end runs for you, written out so you know what it hides.

What the cache does, and what it does not

A training run of your application under -XX:AOTCacheOutput=app.aot (JEP 514, JDK 25) records every class it loaded and linked, a set of heap objects — the module graph, class mirrors, the lambda proxies it resolved — and the method profiles the JIT gathered (JEP 515). At the next start under -XX:AOTCache=app.aot the JVM maps that file and skips the loading, parsing, verification and linking of those classes, and starts with the archived objects in place. That is the whole effect: the part of start-up that is class loading disappears, and the first request pays less because its classes are already linked.

What the cache does not hold on JDK 25 and 26 is compiled code. AOT code compilation is still a draft JEP, so C1 and C2 compile the same methods after the start either way — on zavarnik's Ktor sample C2 compiled 1 171 methods cold and 1 172 with the cache, ten runs each. The profiles help the JIT choose what to compile first, not skip it. A service is not warm from the first request, and throughput after warm-up does not change. And the cache is tied to what it was trained on: the same jars, byte for byte and timestamp for timestamp, in the same order, on the same JDK build, with the same modules and agents. Change any of that and the JVM starts without the cache, prints three lines to stderr and goes on, exit code 0.

What it gave a Ktor service

konekt is my reference project: an eSIM operator's account backend, Ktor CIO with Exposed and Postgres, 128 jars on the classpath, eclipse-temurin:25-jre, deployed with a limit of one core and 1 GiB. Two images that differ by one layer — the cache, 65 MiB, trained inside a container of the first image against the stand's database and verified under -XX:AOTMode=on, 5 329 of 5 332 application classes from the cache. Ten restarts of each, in alternation, on the build box under the chart's limits; docker start to the first 200 on /health measured from outside the container.

Readiness halves, the first request halves, the hundred after it do not move. In the cluster the pod is Ready 3 s after its container starts, with the cache and a startup probe that asks every second; the 11 s it replaced included a readiness probe that waited five seconds before its first ask, so only part of that gap is the cache, and the gain reached a rollout at all only because the probe was retuned with it. The image totals docker image ls printed for the two images, 602 and 685 MB, differ by more than the layer docker history attributes to the cache; the layer is what the cache adds. The methodology and the CSVs are in the repository under docs/research/measurements-2026-09-07/aot/.

The share of classes matters more than the milliseconds, because it is what the milliseconds follow from. On the Ktor stand in zavarnik — Ktor 3.5.2 on CIO, kotlinx.serialization, three routes, 26 jars — 2 322 of 2 322 application, Ktor, coroutines and stdlib classes came from the cache (-Xlog:class+load, source: shared objects file), and time to the first 200 on /health fell by 59–69 % over four series of twenty runs on two collectors. A service whose start is not class loading gains less: a broker of mine on the same JDK build went from 1 377 to 1 225 ms at the median of seven runs, eleven per cent, because most of its start is opening its log.

Memory is the question I get asked next, so I measured it: RSS of the sample from /proc/<pid>/status, five starts per variant in alternation, read at readiness and after 200 requests.

The sample's 31 MiB cache is not visible in resident memory; the difference is smaller than the spread of either row. What it costs is disk — the file and its layer in every image — and the training run: after the training JVM exits a second one assembles the cache, and JEP 514 says that sub-invocation "uses its own Java heap with the same size as the heap used for the training run" — a 4 GB training heap needs 8 GB.

Why Spring and Quarkus have it and you do not

Spring Boot's recipe is one property: -Dspring.context.exit=onRefresh makes the application exit as soon as the context is up, so the training run ends by itself. Its documentation lists the conditions in one paragraph: the classpath "must be the same as the one used to create it, in the same order", extra jars only "at the end (but will not be cached)", "the timestamps of the JARs must be preserved", and "avoid the usage of directories and * wildcard characters". Quarkus runs its @QuarkusIntegrationTest tests as the training under ./mvnw verify -Dquarkus.package.jar.aot.enabled=true -DskipITs=false and packs the result. Both frameworks own three things a plain application distribution does not:

1. Training. Something has to start the application, drive it through the hot path — the cache holds what the run loaded, so an idle start yields a cache that covers half the classes — and stop it in a way that writes the cache. SIGTERM and System.exit write it; SIGKILL does not. Then wait: the process runs a second JVM to assemble the file, and the application process being gone does not mean the cache is there.
2. Verification. A rejected cache is silent by default. The only way to know that production will use the cache is to start with it made mandatory (-XX:AOTMode=on) and count classes. And on JDK 25.0.0–25.0.3 and 26.0.0–26.0.1 even that is not enough: those builds never compare the cache with the jars, so a jar rebuilt after training runs its old classes with exit code 0 — the previous post has the logs. Something other than the JVM has to compare the jars.
3. Packaging. The JVM checks each jar's size and modification time, the classpath string, and the JDK down to the size of lib/modules. A zip loses the timestamps. A -jre image refuses a cache trained on the -jdk image of the same build. A lib/* wildcard expands in the filesystem's own order, which is not the same order on every container runtime.

The same by hand, in five steps

For a service on the application plugin with a /health route on 8080. Every step is what the plugin does; nothing here needs it.

1. Install, and pin the jar timestamps. The JVM compares mtimes in whole seconds against the cache, and distTar and Docker COPY preserve them; set them to a constant now so the cache matches wherever the tar is unpacked. The constant is Gradle's own for reproducible tars — TarCopyAction.CONSTANT_TIME_FOR_TAR_ENTRIES, 2 January 1970: one day rather than zero, because some tools read a zero mtime as no timestamp at all.

./gradlew installDist && cd build/install/app
find lib -name '*.jar' -exec touch -d @86400 {} +

2. Train. Start with -XX:AOTCacheOutput, wait for readiness, hit the hot path, stop with SIGTERM, and wait for the process — after the application exits it runs a second JVM that assembles the cache and prints AOTCache creation is complete when it is done. -XX:-AOTAdapterCaching keeps machine code for the training CPU's instruction set out of the file; it is a diagnostic flag, which is what -XX:+UnlockDiagnosticVMOptions is for, and a later JDK may rename or drop it — the mainline loader already checks CPU features itself. A cache trained on a CI runner with AVX-512 has been reported to SIGILL on a narrower machine, and a cache trained without that code is accepted by a JVM started without the flags.

JAVA_OPTS="-XX:AOTCacheOutput=$PWD/lib/app.aot -XX:+UnlockDiagnosticVMOptions -XX:-AOTAdapterCaching" bin/app &
until curl -sf localhost:8080/health >/dev/null; do sleep 0.1; done
for i in $(seq 20); do curl -s localhost:8080/api/items >/dev/null; done
kill -TERM $! && wait $!

3. Write down what the cache was trained against. On a fixed JDK the JVM will refuse a changed jar; on the six builds that skip the check this file is the only thing that will.

sha256sum lib/*.jar > lib/app.aot.jars

4. Verify. Start with the cache mandatory and class loading logged, then count. The application classes must all come from shared objects file; a source: file: line for one of them names what the training did not reach.

sha256sum -c --quiet lib/app.aot.jars
JAVA_OPTS="-XX:AOTCache=$PWD/lib/app.aot -XX:AOTMode=on -Xlog:class+load=info" bin/app > verify.log &
until curl -sf localhost:8080/health >/dev/null; do sleep 0.1; done; kill -TERM $!
grep -c 'source: shared objects file' verify.log; grep 'source: file:' verify.log | head

5. Ship. The start script has to add -XX:AOTCache only when the file is there, because the JVM refuses that flag next to -XX:AOTCacheOutput and the same script is the training launcher. Three lines before the exec in bin/app:

if [ -f "$APP_HOME/lib/app.aot" ]; then
    DEFAULT_JVM_OPTS="$DEFAULT_JVM_OPTS \"-XX:AOTCache=$APP_HOME/lib/app.aot\""
fi

Then distTar, not distZip, and in a Dockerfile steps 2–4 run in the runtime stage, on the -jre image that will start the service — not in the -jdk build stage, whose cache the runtime JVM will refuse. Steps 1–4 again after every change to any jar, on every build.

The same through the plugin

zavarnik is those five steps as Gradle tasks, with the verification made a red build. It is on the Plugin Portal, so this is the whole installation:

plugins {
    application
    id("io.github.youndie.zavarnik") version "0.1.0"
}

zavarnik {
    training {
        readyWhen.url("http://127.0.0.1:8080/health")
        workload {
            post("http://127.0.0.1:8080/auth/login", "application/json", """{"user":"demo"}""") {
                capture("token", "accessToken")
            }
            get("http://127.0.0.1:8080/api/home") { header("Authorization", "Bearer {{token}}") }
        }
    }
}

./gradlew check runs aotTrain — the installed distribution through its own start script, timestamps pinned, SIGTERM, the manifest — and aotVerify, which fails the build with the name of the thing that failed: a jar that changed since training, a JVM that refused the cache (with the JVM's own reason), or a class share below 90 %. distTar carries the cache; installDist refuses a wildcard on the classpath. For the image, every distribution's lib/ carries zavarnik-runner.jar, the same training and verification on a bare JRE with no Gradle and no curl, so the runtime stage of the Dockerfile trains the cache on the JVM that will use it:

FROM eclipse-temurin:25.0.4_7-jre
COPY --from=build /src/build/install/app /opt/app
RUN java -cp /opt/app/lib/zavarnik-runner.jar io.github.youndie.zavarnik.runner.Main train /opt/app \
 && java -cp /opt/app/lib/zavarnik-runner.jar io.github.youndie.zavarnik.runner.Main verify /opt/app
CMD ["/opt/app/bin/app"]

A service that cannot start on the build machine sets training { onAssemble = false } and trains where it can — konekt does it in its release pipeline, inside a container of the release image on the stand's network, and adds the cache as a layer. On ktor { docker { } } or Jib directly, jib { containerizingMode = "packaged" } plus jibAotTrain and jibAotVerify do the same through a container of the Jib image; that path needs a Docker daemon on the build machine. aotReport prints readiness with and without the cache and, as a second table, what the JIT still compiles afterwards — so nobody reads the first table as a warm service.

What breaks

• A stale cache accepted on JDK 25.0.0–25.0.3 and 26.0.0–26.0.1. The JVM never looks at the jars, -XX:AOTMode=on exits 0, and the old classes run. Move to 25.0.4 or 26.0.2; until then the manifest from step 3 is the check (the post).
• -jdk at training, -jre in production. The same Temurin build, two sizes of lib/modules, cache refused: This file is not the one used while building the AOT cache: '/opt/java/openjdk/lib/modules', size has changed. Train in the image that runs.
• A wildcard on the classpath. lib/* expands in readdir order; Docker's overlay2 on the CI runner and containerd on a k0s node answered differently, and the first cache to reach a cluster was refused there with The name of app classpath [1] does not match. No check on the build machine can see this, because it verifies where it trained. List the jars.
• Anything that changes the JVM's view of the classpath or modules on one side only: an entry added at the front (at the end is fine), --add-modules, a -javaagent (an agent on both sides is fine), ZGC on one side and another collector on the other, on JDK 25 and 26 alike — the boundary is compressed oops, not the collector; ZGC on both sides works.
• A killed training run. SIGKILL, or a container stopped without a grace period, writes no cache and no error.

What the plugin's first version does not do: train on Windows (no SIGTERM there; production on Windows works), ship a cache in distZip, make a missing cache fatal in production (the start script runs without it, by design), or reproduce the cross-CPU SIGILL on its own hardware — the adapter code is left out by default on the strength of two published reports, and a positive control on a pair of machines is still open.

Source: github.com/youndie/zavarnik — experiments/ktor-readiness/, experiments/jit-warmup/ and experiments/memory/ for the stand figures with their logs, docs/research/research-architecture.md §1 for what the JVM checks, with a path for every row; github.com/youndie/konekt, docs/research/measurements-2026-09-07/aot/, for the service. The bug this post keeps pointing at has its own write-up: OpenJDK 25.0.0–25.0.3 uses a stale AOT cache without saying so.