# Silence is not a clean run

> Durability that 779 tests could not see, a fuzzer that needed two files to get through the door, and a harness that reported targets it never ran.

Page: https://kotlin.website/blog/silence-is-not-a-clean-run · Published: 2026-08-27

[bochka](https://github.com/youndie/bochka) is an S3-compatible object store on the JVM: one
process, one node, one disk. It passes 518 of the 744 cases in `ceph/s3-tests` with a classified
reason on every remaining failure, and the gate is 933 tests.

Four milestones went into asking whether those tests check anything — mutation testing, then
fuzzing. The instruments found real gaps. Each of them also reported, at least once, that all was
well when it had done nothing at all; those two outputs are byte-identical.

## You cannot test `fsync` from inside a JVM

The write order is the rule the store is built around: file, flush, then the index record. The
wrong order leaves a dangling reference after a crash — a `500` on a key the server itself named.
There are crash tests for it: a child JVM writes, the parent `SIGKILL`s it mid-write, then reopens
the store and requires everything the log admitted to to read back and match its checksum.

Removing **both** `fsync` calls leaves all **779 tests green** — the whole gate at the time, crash
tests included.

`SIGKILL` kills a process; the page cache belongs to the machine. The parent reads back everything
the child wrote whether or not a byte was ever flushed, so the crash test checks the *order* of
writes — a real property, and not this one. Durability is unobservable from inside the JVM under
test, which means no test there can be written to see it, and the absence of one looks exactly like
a passing one. Mutation testing could not name it either: pitest swaps classes through a loader in
its own JVM while the writer is a different process, so a mutation there survives always — a
property of the harness, not of the tests.

A bytecode gate holds it now: seven functions that must contain a call to `force`, each with its
reason beside it. The gate also states what it does **not** promise — that the call is present,
yes; that it is in the right place, no. A guard overstating its reach is the next thing to be
trusted wrongly.

## Two seed files were worth twenty-seven million inputs

The HTTP head parser is the first code an unauthenticated stranger reaches — the specification
requires a refusal to be possible from the headers alone, so parsing happens before any signature is
checked. Pointing a coverage-guided fuzzer at it, with an empty corpus:

```text
Done 27820013 runs in 61 second(s)
```

Nothing. Then two seed files — an ordinary request, and one carrying both `Content-Length` and
`Transfer-Encoding` — and it found something on input **24**.

Same target, same machine, same minute of wall clock. Random bytes essentially never spell
`<method> <target> <version>`, so everything behind the request line is unreachable: a corpus-less
run measures the corridor in front of the first door, and reports it clean.

What it reached, once let in, was `S3Router.route` — a function with no `throw` anywhere in its
source, which is what made "it never throws" worth asserting. Its query parser reaches the
percent-decoder, where a truncated `%` came back as a bare `IllegalArgumentException` and the
generic failure handler answered `500 InternalError`. The connection does not drop; that half was
fixed a milestone earlier, when a malformed header closed the socket with no bytes in it. The status
rode along behind the fix untouched — and `500` tells a client to retry a request that can never
succeed, which `aws-cli` and `boto3` both do, five times.

## My own harness reported targets it never ran, twice

The run script prints how many targets it fuzzed. It said two; it had fuzzed one. libFuzzer takes
the process over, so one JVM fuzzes one target and the rest never start — while the count came from
counting source files. That is exactly the guard the script exists to be, and it failed by counting
the wrong noun. Fixed to one process per class, three commits later it claimed four targets having
run three: the process is owned per **method**, and one class had two.

The count comes from the `Done N runs` line now, which only libFuzzer can print, and a run where a
target did not fuzz is red rather than quiet. Confirmed with a decoy that looks like a target and
fuzzes nothing:

```text
FAIL    DecoyFuzzTest did not fuzz at all — no libFuzzer run in its output
only 2 of 3 targets fuzzed, which is a failure rather than a clean run
```

## The rest of it

- A mutation percentage measures its denominator. Of 4618 mutations, **499 were written by the
  compiler** — null-check intrinsics, coroutine resumption, `data class` equality. That ratio moves
  when you add a `data class`, so the report is read as four named buckets and never as a score.
- Totals from a mutation run repeat; a verdict on one mutant does not. Two runs over identical code
  disagreed about the `fsync` mutants, and naming the killer by hand showed the report was wrong. A
  rule explaining a survivor is written after a manual substitution, not after a line in a report.
- Twelve survivors in the routing cascade were reported at lines 747–794 of a 742-line file: Kotlin
  encodes inlined code through SMAP with numbers past the end of the source. Both obvious guesses
  were checked by hand and both are caught by the suite, so those twelve stay filed as unexplained
  rather than explained wrongly.
- libFuzzer will not generate an input over 4096 bytes and the shipped limits are 8 KiB a line, so a
  target built on the real constants cannot reach the branches enforcing them — and reports full
  coverage of everything except the bounds it exists for.
- Three times the assertion in a fuzz target was wrong rather than the code: `505` for an unknown
  HTTP version is correct, a `content-length-range` need not be ordered, and `route` is not total.
  The third found the defect **because** it was too strong, then was weakened. A target's assertion
  is the weakest true one, and a stronger one is kept exactly until it fires.
- The seed corpus is `-text` in `.gitattributes`, because in an HTTP seed the CRLF is the framing. A
  `git stash` round trip taken before that rule existed stripped the CR out of all three seeds, and
  each went on passing — as a "line without CRLF" refusal rather than the request it was written as.
  Nothing went red, because a corpus file has no expected result.

## Ask what the tool would say if it had done nothing

Every failure above has the same shape. A gate with the durability removed, a fuzzer that never got
past the request line, a harness that ran one target of two — each produced the output of a clean
run, because *a tool that finds nothing and a tool that does nothing are indistinguishable from
outside*.

So the question to ask of a green instrument is not whether it passed, but: **what would this
report if it had done no work at all, and how would I tell that apart from what it just told me?**
For the harness that meant counting the line only libFuzzer can print; for durability, admitting no
test in that JVM can see it and moving the check to the bytecode; for the corpus, noticing that
twenty-seven million of anything ought to have found *something*.

An instrument is a stand, and a stand is checked before its output is allowed to become a claim.

---

Source: [github.com/youndie/bochka](https://github.com/youndie/bochka)