Silence is not a clean run

bochka is an S3-compatible object store on the JVM: one process, one node, one disk. It passes 518 of the 744 cases in ceph/s3-tests with a classified reason on every remaining failure, and the gate is 933 tests.

Four milestones went into asking whether those tests check anything — mutation testing, then fuzzing. The instruments found real gaps. Each of them also reported, at least once, that all was well when it had done nothing at all; those two outputs are byte-identical.

You cannot test fsync from inside a JVM

The write order is the rule the store is built around: file, flush, then the index record. The wrong order leaves a dangling reference after a crash — a 500 on a key the server itself named. There are crash tests for it: a child JVM writes, the parent SIGKILLs it mid-write, then reopens the store and requires everything the log admitted to to read back and match its checksum.

Removing both fsync calls leaves all 779 tests green — the whole gate at the time, crash tests included.

SIGKILL kills a process; the page cache belongs to the machine. The parent reads back everything the child wrote whether or not a byte was ever flushed, so the crash test checks the order of writes — a real property, and not this one. Durability is unobservable from inside the JVM under test, which means no test there can be written to see it, and the absence of one looks exactly like a passing one. Mutation testing could not name it either: pitest swaps classes through a loader in its own JVM while the writer is a different process, so a mutation there survives always — a property of the harness, not of the tests.

A bytecode gate holds it now: seven functions that must contain a call to force, each with its reason beside it. The gate also states what it does not promise — that the call is present, yes; that it is in the right place, no. A guard overstating its reach is the next thing to be trusted wrongly.

Two seed files were worth twenty-seven million inputs

The HTTP head parser is the first code an unauthenticated stranger reaches — the specification requires a refusal to be possible from the headers alone, so parsing happens before any signature is checked. Pointing a coverage-guided fuzzer at it, with an empty corpus:

Done 27820013 runs in 61 second(s)

Nothing. Then two seed files — an ordinary request, and one carrying both Content-Length and Transfer-Encoding — and it found something on input 24.

Same target, same machine, same minute of wall clock. Random bytes essentially never spell <method> <target> <version>, so everything behind the request line is unreachable: a corpus-less run measures the corridor in front of the first door, and reports it clean.

What it reached, once let in, was S3Router.route — a function with no throw anywhere in its source, which is what made "it never throws" worth asserting. Its query parser reaches the percent-decoder, where a truncated % came back as a bare IllegalArgumentException and the generic failure handler answered 500 InternalError. The connection does not drop; that half was fixed a milestone earlier, when a malformed header closed the socket with no bytes in it. The status rode along behind the fix untouched — and 500 tells a client to retry a request that can never succeed, which aws-cli and boto3 both do, five times.

My own harness reported targets it never ran, twice

The run script prints how many targets it fuzzed. It said two; it had fuzzed one. libFuzzer takes the process over, so one JVM fuzzes one target and the rest never start — while the count came from counting source files. That is exactly the guard the script exists to be, and it failed by counting the wrong noun. Fixed to one process per class, three commits later it claimed four targets having run three: the process is owned per method, and one class had two.

The count comes from the Done N runs line now, which only libFuzzer can print, and a run where a target did not fuzz is red rather than quiet. Confirmed with a decoy that looks like a target and fuzzes nothing:

FAIL    DecoyFuzzTest did not fuzz at all — no libFuzzer run in its output
only 2 of 3 targets fuzzed, which is a failure rather than a clean run

The rest of it

• A mutation percentage measures its denominator. Of 4618 mutations, 499 were written by the compiler — null-check intrinsics, coroutine resumption, data class equality. That ratio moves when you add a data class, so the report is read as four named buckets and never as a score.
• Totals from a mutation run repeat; a verdict on one mutant does not. Two runs over identical code disagreed about the fsync mutants, and naming the killer by hand showed the report was wrong. A rule explaining a survivor is written after a manual substitution, not after a line in a report.
• Twelve survivors in the routing cascade were reported at lines 747–794 of a 742-line file: Kotlin encodes inlined code through SMAP with numbers past the end of the source. Both obvious guesses were checked by hand and both are caught by the suite, so those twelve stay filed as unexplained rather than explained wrongly.
• libFuzzer will not generate an input over 4096 bytes and the shipped limits are 8 KiB a line, so a target built on the real constants cannot reach the branches enforcing them — and reports full coverage of everything except the bounds it exists for.
• Three times the assertion in a fuzz target was wrong rather than the code: 505 for an unknown HTTP version is correct, a content-length-range need not be ordered, and route is not total. The third found the defect because it was too strong, then was weakened. A target's assertion is the weakest true one, and a stronger one is kept exactly until it fires.
• The seed corpus is -text in .gitattributes, because in an HTTP seed the CRLF is the framing. A git stash round trip taken before that rule existed stripped the CR out of all three seeds, and each went on passing — as a "line without CRLF" refusal rather than the request it was written as. Nothing went red, because a corpus file has no expected result.

Ask what the tool would say if it had done nothing

Every failure above has the same shape. A gate with the durability removed, a fuzzer that never got past the request line, a harness that ran one target of two — each produced the output of a clean run, because a tool that finds nothing and a tool that does nothing are indistinguishable from outside.

So the question to ask of a green instrument is not whether it passed, but: what would this report if it had done no work at all, and how would I tell that apart from what it just told me? For the harness that meant counting the line only libFuzzer can print; for durability, admitting no test in that JVM can see it and moving the check to the bytecode; for the corpus, noticing that twenty-seven million of anything ought to have found something.

An instrument is a stand, and a stand is checked before its output is allowed to become a claim.

Source: github.com/youndie/bochka