# CRaC on a Ktor service with HikariCP and Exposed: every socket that refuses the checkpoint

> A CRaC checkpoint of a real Ktor service restores in 131 ms against 2 317 and its JIT is warm. Getting there means every open socket, and HikariCP names its own. What breaks, in the order it breaks, with the logs.

Page: https://kotlin.website/blog/crac-on-a-ktor-service-with-hikari-and-exposed · Published: 2026-09-11

Project Leyden's AOT cache saves the class loading and nothing else — after the start the JIT
compiles exactly what it would have compiled anyway, which is
[measured](https://kotlin.website/blog/leyden-aot-cache-for-a-plain-kotlin-jvm-service) and is the honest limit of it.
[CRaC](https://openjdk.org/projects/crac/) — Coordinated Restore at Checkpoint — saves the
*process*: a snapshot of a warmed-up JVM, restored in a fraction of the time with its JIT already
hot. On my reference service the restore is ready in 131 ms against 2 317, and it serves its first
signed-in screen in 32 ms against 118.

The price is that a JVM refuses to checkpoint while anything holds an open socket, and a service
holds several kinds. What follows is every one of them, in the order they refused, on a stack
nobody has written this up for: Ktor CIO, HikariCP, Exposed, Postgres and a message broker. The
scripts and the logs are in [zavarnik](https://github.com/youndie/zavarnik) under
`experiments/crac-*/` and in [konekt](https://github.com/youndie/konekt) under
`docs/research/measurements-2026-09-11/crac/`.

## What it is worth

Two services, both on Zulu 25.0.4.1 with CRaC, Docker on one Linux box, the two variants
restarted in alternation — five times each on the sample, ten on konekt. First a Ktor CIO sample
with no database at all, timed from `docker run` to the first `200` on `/health`:

| | plain start | restore |
|---|---|---|
| readiness, median of 5 | 642 ms (543–953) | **50 ms** (45–52) |
| first `POST` after readiness | 31 ms (25–44) | **12 ms** (11–14) |
| JIT compilations during the first 100 requests | 1 555–1 591 | **321–366** |
| container memory, ready / after 100 requests | 70 / 100 MiB | 36 / 62 MiB |

The last three rows come from a second series of five with `-XX:+PrintCompilation` on, whose own
readiness was 639 ms against 82 — the flag costs a little at start. The third row is the one the
AOT cache cannot produce: the restored process compiles about a fifth as much, because the
compiled code came back with it. Then [konekt](https://github.com/youndie/konekt),
an eSIM operator's account backend — Ktor CIO, HikariCP with ten connections, Exposed, Postgres,
a broker, 128 jars — under its own compose stand:

| | plain start | restore |
|---|---|---|
| `docker run` → `/health`, median of 10 | 2 317 ms (2 141–2 513) | **131 ms** (115–151) |
| first signed-in home screen | 118 ms (102–128) | **32 ms** (27–50) |
| the snapshot as an image layer | — | 143 MB on this run (143–165 across the day's runs) |

Those two columns compare with each other and with nothing else: they ran without a CPU limit,
while the same service's AOT-cache numbers were taken under the one core its chart gives it. And
the restore is not a stunt — the probe signs in, tops up, buys a plan, confirms it, watches the
order reach `completed` and receives twelve realtime updates over the event stream, the same
twelve a plain start receives.

## What refuses the checkpoint, in order

**The listening socket, and the selector holding it.** With nothing configured, `jcmd <pid>
JDK.checkpoint` on a running Ktor CIO server fails and the process carries on as if nothing
happened:

```text
CheckpointOpenSocketException: sun.nio.ch.ServerSocketChannelImpl[]
CheckpointOpenSocketException: EPoll FD 30 left open in sun.nio.ch.EPollSelectorImpl@... with registered keys
```

One rule in a file-descriptor policy fixes both — the selector needs no rule of its own:

```yaml
type: SOCKET
listening: true
action: reopen
```

CRaC closes the socket before the snapshot and binds the same local address again on restore. No
code in the application, no `Resource` implementation, nothing. Worth knowing because the
project's own documentation on GitHub still says `reopen` for sockets is not implemented and will
throw after restore; on Azul's 25.0.4.1 build it is implemented and works.

**The connection pool — and it names itself.** On a service with a database the same rule leaves
ten sockets standing. Turn on `-Djdk.crac.collect-fd-stacktraces=true` and the JVM says who owns
each one:

```text
CheckpointOpenSocketException: Socket[addr=postgres/172.18.0.2,port=5432,localport=47200]
  Caused by: java.lang.Exception: This file descriptor was created by HikariPool-1:connection-adder
```

That flag is the single most useful thing in this whole exercise, and it is off by default.

**Closing the pool's sockets does not work.** The obvious rule — `action: close` on port 5432 —
is the one that fails, and it fails in a way worth understanding. The connections do close;
HikariCP notices immediately (`Failed to validate connection … Unable to set network timeout`);
and its `connection-adder` thread opens replacements while the checkpoint is still being taken.
The snapshot then dies on a socket that did not exist when it started:

```text
CheckpointOpenSocketException: FD fd=133 type=socket path=socket:[117055],port=0
```

You cannot win a race against a pool whose whole job is to keep itself full.

**Ignoring them does work — for a reason that is not general.** `action: ignore` leaves the
sockets in the snapshot. On restore the engine replaces each with `/dev/null`, and then two
things happen that are properties of these particular libraries: HikariCP validates every
connection before handing it out, discards all ten and opens new ones; and the broker client
notices EOF and reconnects itself, saying so in the log. The service works, with **no change to
the application**.

A library that does neither would hand out a dead connection, silently. That is why the
verification has to be a request that reaches the far side, not a process that started — and why
this result is konekt's rather than everyone's.

**Your own tooling's connections.** This one cost me three iterations. The thing taking the
checkpoint warms the application up over HTTP, and HTTP keep-alive is doing its job: the server's
*accepted* end of the warm-up connection is an open socket like any other.

```text
BusySelectorException: Selector has registered keys from channels:
  [SocketChannel[connected local=/127.0.0.1:18090 remote=/127.0.0.1:43872]]
```

Closing the workload's client was not enough; the readiness probe held one too. Nothing in CRaC's
documentation warns about it, because that documentation is about the application's sockets, not
about the sockets of whatever is driving it.

## What a restore freezes, besides the state

**The path of every file the process will read again.** A snapshot itself travels fine: taken
into a mounted directory at `/out/crac`, it restores from `/opt/app/zavarnik/crac` inside an
image built as "that image plus one layer" — 46 ms to ready, route answers 200. What does not
travel is the *policy file*: the JVM records `jdk.crac.resource-policies` as the string it was
given and reads the file again on restore.

```text
warp: Restore successful!
OpenResourcePolicies$ConfigurationException:
  File /out/crac/zavarnik-crac-policies.yaml used in property jdk.crac.resource-policies does not exist
```

Read those two lines together. The engine reports success, the JVM then throws, and on a stand
with the same directory mounted it never happens at all — it happens to whoever first runs the
image somewhere else. Anything the restored process re-reads has to live inside the image.

**The configuration read at startup.** The restored container's environment does not reach it.
konekt takes its brand from an environment variable at boot; a restored container started with
`BRAND=brand-b` answers with brand-a's name, while a plain start of the same image answers with
brand-b's. So a snapshot belongs to one combination of environment, and anything read once at
boot — a brand, a database URL, a secret — is frozen into it. Spring's documentation puts the
security half plainly: assume any sensitive data the JVM has seen ends up in the files.

**The random number generators, in a way that is worse than it looks.** Restore one snapshot
twice and compare:

| Generator | Same in both replicas? |
|---|---|
| `Random` constructed before the checkpoint | **yes** |
| `ThreadLocalRandom` on a thread that predates the checkpoint | **yes** |
| `ThreadLocalRandom` on a thread created *after* the restore | **yes** |
| `SplittableRandom`, before the checkpoint or after the restore | **yes** |
| `Math.random()`, once anything called it before the checkpoint | **yes** |
| `new Random()` constructed after the restore | no |
| `RandomGenerator.getDefault()`, `UUID.randomUUID()` | no |
| `SecureRandom()` | no — the JDK reseeds it |
| `SecureRandom(byte[] seed)` | no — documented as *not* reseeded, but the Linux provider mixes the seed with system entropy |

The third row is the surprise, and the fourth is the same surprise for `SplittableRandom`: making
a fresh thread or a fresh instance does not help, because the seeder that initialises new ones is
in the snapshot too. `Math.random()` is one shared `Random` behind a static, and in a running
service something has already touched it. And that third row is the one Kotlin code lands
on: `kotlin.random.Random.Default` resolves, on a JDK 8 or newer, to
`kotlin.random.jdk8.PlatformThreadLocalRandom` backed by `java.util.concurrent.ThreadLocalRandom`
— asked of the running JVM through reflection rather than read off the source, because the
platform implementation is chosen at runtime.

And yet konekt, whose eSIM mock draws its activation codes from exactly that, produced five
different codes across five restores of one snapshot. The stream is shared; the *consumption* is
not, because sign-in, token issuing, screens, Exposed and HikariCP all draw from it first, on
whichever pool thread served the request. A collision is possible and not reproducible on demand
— which also means a check that restores twice and compares the product's own output would be
green for the wrong reason. Probe the generators, not the answers.

**The image, down to its bytes.** The same snapshot under Zulu 25.0.4 — one build older, same
major — fails with `Cannot find build-id … validation failed`: the engine verifies every file the
process had mapped, by the path it had at checkpoint.

## The CPU, and the advice that makes it worse

Across machines the hazard starts as a loud refusal, which is better than the AOT cache's
`SIGILL` — and stays better only until you follow the JVM's own advice about it. Measured on two
machines, an AMD EPYC-Genoa with AVX-512 and an Intel Core Ultra 7 without it, one pinned image
digest so the JDK is byte-identical on both:

| checkpoint on | flag | restore on | result |
|---|---|---|---|
| wide | none | narrow | refused, exit 1, message names a mask |
| wide | `-XX:CPUFeatures=generic` | narrow | **SIGSEGV**, exit 139, no message at all |
| narrow | `-XX:CPUFeatures=generic` | wide | restored |
| narrow | none | wide | refused, exit 1 |
| wide | `generic` | wide, itself | restored — the control that says the flag broke nothing |

Two things in that table are worth more than the rest. The first: **without the flag the
constraint is an exact match, not a subset** — narrow → wide refuses too, though every instruction
the snapshot could want is present. The engine prints both bitmaps and calls it
`Image constraint 'cpu.features' (bitmap subset) does not match`.

The second is the trap. On the refusal the JVM prints `try using -XX:CPUFeatures=0x… on
checkpoint`. Do that, and on the machine that refused, the restore no longer refuses — it
segfaults, with no JVM message and no `hs_err` file. `generic` behaves the same way, and so does
the older CRIU engine: a `generic` snapshot taken with `-XX:CRaCEngine=criuengine` restores on the
machine that took it and crashes the same silent way on the other — so this is not a property of
`warp`. The documented workaround converts a loud failure into a silent one, in the direction you
are most likely to need it: from the larger build machine to the smaller production one.

**The rule that does work:** take the snapshot on the narrowest CPU it will ever be restored on,
with `-XX:CPUFeatures=generic`. What that costs, six restores per variant in alternation with
`oha` at 32 connections for ten seconds behind each: readiness unchanged, 42 ms against 46; throughput lower in all
six pairs, 5 526 against 4 089 requests a second at the median. The magnitude is not trustworthy —
the ranges overlap and the box was not isolated — the direction is.

This is also what rules the snapshot out of the pipeline that motivated it. konekt's release image
is built on a hosted CI runner, so the machine that would take the checkpoint is whatever the
provider hands out that day, and it changes between runs. Before "which CPU does production have"
comes "where can a checkpoint be taken on a CPU that is the same next week".

## What it costs to have at all

A JDK with CRaC, and for JDK 25 that means Azul Zulu and nobody else — BellSoft's own page for
Liberica with CRaC lists 17 and 21, and its container images carry the same two. Linux. A layer the size of the snapshot on every image — 143 MB on konekt, 82 on
the sample. A snapshot per environment combination. Secrets inside that snapshot.

What it no longer costs is privileges. The CRIU engine wanted root, or `CHECKPOINT_RESTORE` and
`SYS_PTRACE` in a container; since October 2024 Azul ships `warp`, which needs neither, and on
25.0.4.1 it is the default. On my box `--cap-add CHECKPOINT_RESTORE --cap-add SYS_PTRACE` and
`--privileged` changed nothing, because nothing was needed.

## Where the frameworks are, and where they are not

Spring Framework 6.1 and Boot 3.2 stop beans before the checkpoint and start them after, with a
dedicated lifecycle for HikariCP. Micronaut suspends the Hikari pool, waits for its connections
to close, and resumes it afterwards. Quarkus and Helidon 4.2 have their own support.

Underneath, the picture is thinner. HikariCP's own issue for this — "Allow to stop/restart
HikariPool" — has been open since 2023, and its comments describe exactly why `close` loses the race:
`suspend` does not guarantee the connections are closed, and after `softEvictConnections` the
sockets linger for about half a second. Ktor's request, KTOR-6485, has been *Submitted* since
November 2023 with three votes. Exposed and pgjdbc have no issue mentioning CRaC at all. A Netty
pull request to close its epoll descriptors at checkpoint was closed without merging.

So the frameworks did the half that is theirs — a lifecycle around the checkpoint — and on a
Ktor plus Exposed stack there is neither a framework doing it nor a pool that can be told to.
What fills the gap here is three rules in a policy file and a verification that actually talks to
the database.

## Doing it

By hand, in an image whose JDK has CRaC:

```bash
java -XX:CRaCCheckpointTo=/snapshot -Djdk.crac.resource-policies=/app/policies.yaml -jar app.jar &
# wait until it is ready, drive the hot path, then:
jcmd <pid> JDK.checkpoint          # this ends the process; the snapshot is complete when it exits
java -XX:CRaCRestoreFrom=/snapshot # every other flag comes back from the snapshot
```

The policy file for a service with a database and a broker:

```yaml
type: SOCKET
listening: true
action: reopen
---
type: SOCKET
remotePort: 5432
action: ignore
---
type: SOCKET
remotePort: 9092
action: ignore
```

[zavarnik](https://github.com/youndie/zavarnik) does this as build tasks, because the snapshot
has to be taken by the JVM that will restore it, inside the image that will carry it. `crac {
ignoreRemotePort(5432, 9092) }` is the only thing it cannot work out for itself. On Jib —
`ktor { docker { } }` included — `jibCracCheckpoint` takes the snapshot in a container of the
image just built, and the next build lays it over that image and **replaces** the entrypoint with
`java -XX:CRaCRestoreFrom=…`, because a restore carries neither classpath nor main class. An
image therefore holds either an AOT cache or a snapshot, never both. That image answers `/health`
58 ms after `docker run` on a GitHub runner, which is where its check runs.

## Limitations

- One stack, measured once: Ktor CIO, HikariCP 7.1, Exposed 1.5, pgjdbc 42.7, on Zulu 25.0.4.1.
  The `ignore` recipe rests on HikariCP validating on borrow; a pool that does not would need the
  suspend-and-wait dance Micronaut implements.
- The memory row is not a saving. A restored process maps the snapshot lazily and its RSS stays
  low until pages are touched; a hundred requests touch some of them, not all. Read
  36 / 62 MiB against 70 / 100 as "different accounting", not as "CRaC uses less memory".
- Numbers from one box, five or ten restarts per variant in alternation, no CPU limit. The AOT-cache
  figures they invite comparison with were taken under one core; do not pair them.
- The restore times are `docker run` to the first `200`, from outside the container. What a
  Kubernetes rollout does with them depends on probes, and that is not measured here.
- Netty is not covered: Ktor's CIO engine is what was tested, and the pull request that would
  have taught Netty to close its epoll descriptors was closed unmerged.
- The CPU pair is one pair: an EPYC Genoa guest against a Core Ultra laptop part, both x86_64.
  The silent crash reproduces with both of Zulu's engines, so it is not the engine; whether it is
  this build or that specific feature gap is not established here — what is established is that
  it happens when you do what the JVM's message says.
- Nothing here says a restore is *safe* for a given application. It says what refuses to
  checkpoint and what comes back working; whether a frozen configuration and a shared random
  stream are acceptable is a property of the service, not of CRaC.

---

Source: [github.com/youndie/zavarnik](https://github.com/youndie/zavarnik) —
`experiments/crac-smoke/` and `experiments/crac-ktor/` for the scripts and every log quoted,
`docs/research/research-crac.md` for the fact tables with their addresses;
[github.com/youndie/konekt](https://github.com/youndie/konekt),
`docs/research/measurements-2026-09-11/crac/`, for the service and the ten hypotheses it was put
through. The AOT cache, which is the other half of this story, has its own how-to:
[Leyden AOT cache for a plain Kotlin/JVM service](https://kotlin.website/blog/leyden-aot-cache-for-a-plain-kotlin-jvm-service).