CRaC on a Ktor service with HikariCP and Exposed: every socket that refuses the checkpoint

Project Leyden's AOT cache saves the class loading and nothing else — after the start the JIT compiles exactly what it would have compiled anyway, which is measured and is the honest limit of it. CRaC — Coordinated Restore at Checkpoint — saves the process: a snapshot of a warmed-up JVM, restored in a fraction of the time with its JIT already hot. On my reference service the restore is ready in 131 ms against 2 317, and it serves its first signed-in screen in 32 ms against 118.

The price is that a JVM refuses to checkpoint while anything holds an open socket, and a service holds several kinds. What follows is every one of them, in the order they refused, on a stack nobody has written this up for: Ktor CIO, HikariCP, Exposed, Postgres and a message broker. The scripts and the logs are in zavarnik under experiments/crac-*/ and in konekt under docs/research/measurements-2026-09-11/crac/.

What it is worth

Two services, both on Zulu 25.0.4.1 with CRaC, Docker on one Linux box, the two variants restarted in alternation — five times each on the sample, ten on konekt. First a Ktor CIO sample with no database at all, timed from docker run to the first 200 on /health:

The last three rows come from a second series of five with -XX:+PrintCompilation on, whose own readiness was 639 ms against 82 — the flag costs a little at start. The third row is the one the AOT cache cannot produce: the restored process compiles about a fifth as much, because the compiled code came back with it. Then konekt, an eSIM operator's account backend — Ktor CIO, HikariCP with ten connections, Exposed, Postgres, a broker, 128 jars — under its own compose stand:

Those two columns compare with each other and with nothing else: they ran without a CPU limit, while the same service's AOT-cache numbers were taken under the one core its chart gives it. And the restore is not a stunt — the probe signs in, tops up, buys a plan, confirms it, watches the order reach completed and receives twelve realtime updates over the event stream, the same twelve a plain start receives.

What refuses the checkpoint, in order

The listening socket, and the selector holding it. With nothing configured, jcmd <pid> JDK.checkpoint on a running Ktor CIO server fails and the process carries on as if nothing happened:

CheckpointOpenSocketException: sun.nio.ch.ServerSocketChannelImpl[]
CheckpointOpenSocketException: EPoll FD 30 left open in sun.nio.ch.EPollSelectorImpl@... with registered keys

One rule in a file-descriptor policy fixes both — the selector needs no rule of its own:

type: SOCKET
listening: true
action: reopen

CRaC closes the socket before the snapshot and binds the same local address again on restore. No code in the application, no Resource implementation, nothing. Worth knowing because the project's own documentation on GitHub still says reopen for sockets is not implemented and will throw after restore; on Azul's 25.0.4.1 build it is implemented and works.

The connection pool — and it names itself. On a service with a database the same rule leaves ten sockets standing. Turn on -Djdk.crac.collect-fd-stacktraces=true and the JVM says who owns each one:

CheckpointOpenSocketException: Socket[addr=postgres/172.18.0.2,port=5432,localport=47200]
  Caused by: java.lang.Exception: This file descriptor was created by HikariPool-1:connection-adder

That flag is the single most useful thing in this whole exercise, and it is off by default.

Closing the pool's sockets does not work. The obvious rule — action: close on port 5432 — is the one that fails, and it fails in a way worth understanding. The connections do close; HikariCP notices immediately (Failed to validate connection … Unable to set network timeout); and its connection-adder thread opens replacements while the checkpoint is still being taken. The snapshot then dies on a socket that did not exist when it started:

CheckpointOpenSocketException: FD fd=133 type=socket path=socket:[117055],port=0

You cannot win a race against a pool whose whole job is to keep itself full.

Ignoring them does work — for a reason that is not general. action: ignore leaves the sockets in the snapshot. On restore the engine replaces each with /dev/null, and then two things happen that are properties of these particular libraries: HikariCP validates every connection before handing it out, discards all ten and opens new ones; and the broker client notices EOF and reconnects itself, saying so in the log. The service works, with no change to the application.

A library that does neither would hand out a dead connection, silently. That is why the verification has to be a request that reaches the far side, not a process that started — and why this result is konekt's rather than everyone's.

Your own tooling's connections. This one cost me three iterations. The thing taking the checkpoint warms the application up over HTTP, and HTTP keep-alive is doing its job: the server's accepted end of the warm-up connection is an open socket like any other.

BusySelectorException: Selector has registered keys from channels:
  [SocketChannel[connected local=/127.0.0.1:18090 remote=/127.0.0.1:43872]]

Closing the workload's client was not enough; the readiness probe held one too. Nothing in CRaC's documentation warns about it, because that documentation is about the application's sockets, not about the sockets of whatever is driving it.

What a restore freezes, besides the state

The path of every file the process will read again. A snapshot itself travels fine: taken into a mounted directory at /out/crac, it restores from /opt/app/zavarnik/crac inside an image built as "that image plus one layer" — 46 ms to ready, route answers 200. What does not travel is the policy file: the JVM records jdk.crac.resource-policies as the string it was given and reads the file again on restore.

warp: Restore successful!
OpenResourcePolicies$ConfigurationException:
  File /out/crac/zavarnik-crac-policies.yaml used in property jdk.crac.resource-policies does not exist

Read those two lines together. The engine reports success, the JVM then throws, and on a stand with the same directory mounted it never happens at all — it happens to whoever first runs the image somewhere else. Anything the restored process re-reads has to live inside the image.

The configuration read at startup. The restored container's environment does not reach it. konekt takes its brand from an environment variable at boot; a restored container started with BRAND=brand-b answers with brand-a's name, while a plain start of the same image answers with brand-b's. So a snapshot belongs to one combination of environment, and anything read once at boot — a brand, a database URL, a secret — is frozen into it. Spring's documentation puts the security half plainly: assume any sensitive data the JVM has seen ends up in the files.

The random number generators, in a way that is worse than it looks. Restore one snapshot twice and compare:

The third row is the surprise, and the fourth is the same surprise for SplittableRandom: making a fresh thread or a fresh instance does not help, because the seeder that initialises new ones is in the snapshot too. Math.random() is one shared Random behind a static, and in a running service something has already touched it. And that third row is the one Kotlin code lands on: kotlin.random.Random.Default resolves, on a JDK 8 or newer, to kotlin.random.jdk8.PlatformThreadLocalRandom backed by java.util.concurrent.ThreadLocalRandom — asked of the running JVM through reflection rather than read off the source, because the platform implementation is chosen at runtime.

And yet konekt, whose eSIM mock draws its activation codes from exactly that, produced five different codes across five restores of one snapshot. The stream is shared; the consumption is not, because sign-in, token issuing, screens, Exposed and HikariCP all draw from it first, on whichever pool thread served the request. A collision is possible and not reproducible on demand — which also means a check that restores twice and compares the product's own output would be green for the wrong reason. Probe the generators, not the answers.

The image, down to its bytes. The same snapshot under Zulu 25.0.4 — one build older, same major — fails with Cannot find build-id … validation failed: the engine verifies every file the process had mapped, by the path it had at checkpoint.

The CPU, and the advice that makes it worse

Across machines the hazard starts as a loud refusal, which is better than the AOT cache's SIGILL — and stays better only until you follow the JVM's own advice about it. Measured on two machines, an AMD EPYC-Genoa with AVX-512 and an Intel Core Ultra 7 without it, one pinned image digest so the JDK is byte-identical on both:

Two things in that table are worth more than the rest. The first: without the flag the constraint is an exact match, not a subset — narrow → wide refuses too, though every instruction the snapshot could want is present. The engine prints both bitmaps and calls it Image constraint 'cpu.features' (bitmap subset) does not match.

The second is the trap. On the refusal the JVM prints try using -XX:CPUFeatures=0x… on checkpoint. Do that, and on the machine that refused, the restore no longer refuses — it segfaults, with no JVM message and no hs_err file. generic behaves the same way, and so does the older CRIU engine: a generic snapshot taken with -XX:CRaCEngine=criuengine restores on the machine that took it and crashes the same silent way on the other — so this is not a property of warp. The documented workaround converts a loud failure into a silent one, in the direction you are most likely to need it: from the larger build machine to the smaller production one.

The rule that does work: take the snapshot on the narrowest CPU it will ever be restored on, with -XX:CPUFeatures=generic. What that costs, six restores per variant in alternation with oha at 32 connections for ten seconds behind each: readiness unchanged, 42 ms against 46; throughput lower in all six pairs, 5 526 against 4 089 requests a second at the median. The magnitude is not trustworthy — the ranges overlap and the box was not isolated — the direction is.

This is also what rules the snapshot out of the pipeline that motivated it. konekt's release image is built on a hosted CI runner, so the machine that would take the checkpoint is whatever the provider hands out that day, and it changes between runs. Before "which CPU does production have" comes "where can a checkpoint be taken on a CPU that is the same next week".

What it costs to have at all

A JDK with CRaC, and for JDK 25 that means Azul Zulu and nobody else — BellSoft's own page for Liberica with CRaC lists 17 and 21, and its container images carry the same two. Linux. A layer the size of the snapshot on every image — 143 MB on konekt, 82 on the sample. A snapshot per environment combination. Secrets inside that snapshot.

What it no longer costs is privileges. The CRIU engine wanted root, or CHECKPOINT_RESTORE and SYS_PTRACE in a container; since October 2024 Azul ships warp, which needs neither, and on 25.0.4.1 it is the default. On my box --cap-add CHECKPOINT_RESTORE --cap-add SYS_PTRACE and --privileged changed nothing, because nothing was needed.

Where the frameworks are, and where they are not

Spring Framework 6.1 and Boot 3.2 stop beans before the checkpoint and start them after, with a dedicated lifecycle for HikariCP. Micronaut suspends the Hikari pool, waits for its connections to close, and resumes it afterwards. Quarkus and Helidon 4.2 have their own support.

Underneath, the picture is thinner. HikariCP's own issue for this — "Allow to stop/restart HikariPool" — has been open since 2023, and its comments describe exactly why close loses the race: suspend does not guarantee the connections are closed, and after softEvictConnections the sockets linger for about half a second. Ktor's request, KTOR-6485, has been Submitted since November 2023 with three votes. Exposed and pgjdbc have no issue mentioning CRaC at all. A Netty pull request to close its epoll descriptors at checkpoint was closed without merging.

So the frameworks did the half that is theirs — a lifecycle around the checkpoint — and on a Ktor plus Exposed stack there is neither a framework doing it nor a pool that can be told to. What fills the gap here is three rules in a policy file and a verification that actually talks to the database.

Doing it

By hand, in an image whose JDK has CRaC:

java -XX:CRaCCheckpointTo=/snapshot -Djdk.crac.resource-policies=/app/policies.yaml -jar app.jar &
# wait until it is ready, drive the hot path, then:
jcmd <pid> JDK.checkpoint          # this ends the process; the snapshot is complete when it exits
java -XX:CRaCRestoreFrom=/snapshot # every other flag comes back from the snapshot

The policy file for a service with a database and a broker:

type: SOCKET
listening: true
action: reopen
---
type: SOCKET
remotePort: 5432
action: ignore
---
type: SOCKET
remotePort: 9092
action: ignore

zavarnik does this as build tasks, because the snapshot has to be taken by the JVM that will restore it, inside the image that will carry it. crac { ignoreRemotePort(5432, 9092) } is the only thing it cannot work out for itself. On Jib — ktor { docker { } } included — jibCracCheckpoint takes the snapshot in a container of the image just built, and the next build lays it over that image and replaces the entrypoint with java -XX:CRaCRestoreFrom=…, because a restore carries neither classpath nor main class. An image therefore holds either an AOT cache or a snapshot, never both. That image answers /health 58 ms after docker run on a GitHub runner, which is where its check runs.

Limitations

• One stack, measured once: Ktor CIO, HikariCP 7.1, Exposed 1.5, pgjdbc 42.7, on Zulu 25.0.4.1. The ignore recipe rests on HikariCP validating on borrow; a pool that does not would need the suspend-and-wait dance Micronaut implements.
• The memory row is not a saving. A restored process maps the snapshot lazily and its RSS stays low until pages are touched; a hundred requests touch some of them, not all. Read 36 / 62 MiB against 70 / 100 as "different accounting", not as "CRaC uses less memory".
• Numbers from one box, five or ten restarts per variant in alternation, no CPU limit. The AOT-cache figures they invite comparison with were taken under one core; do not pair them.
• The restore times are docker run to the first 200, from outside the container. What a Kubernetes rollout does with them depends on probes, and that is not measured here.
• Netty is not covered: Ktor's CIO engine is what was tested, and the pull request that would have taught Netty to close its epoll descriptors was closed unmerged.
• The CPU pair is one pair: an EPYC Genoa guest against a Core Ultra laptop part, both x86_64. The silent crash reproduces with both of Zulu's engines, so it is not the engine; whether it is this build or that specific feature gap is not established here — what is established is that it happens when you do what the JVM's message says.
• Nothing here says a restore is safe for a given application. It says what refuses to checkpoint and what comes back working; whether a frozen configuration and a shared random stream are acceptable is a property of the service, not of CRaC.

Source: github.com/youndie/zavarnik — experiments/crac-smoke/ and experiments/crac-ktor/ for the scripts and every log quoted, docs/research/research-crac.md for the fact tables with their addresses; github.com/youndie/konekt, docs/research/measurements-2026-09-11/crac/, for the service and the ten hypotheses it was put through. The AOT cache, which is the other half of this story, has its own how-to: Leyden AOT cache for a plain Kotlin/JVM service.