# Green is not the same as checked

> A lock that only ever answered yes, an out-of-scope label with eight defects behind it, and a benchmark the optimiser deleted.

Page: https://kotlin.website/blog/green-is-not-checked · Published: 2026-08-19

[bochka](https://github.com/youndie/bochka) is an S3-compatible object store on the JVM: one
process, one node, one disk. It passes 426 of the 744 cases in `ceph/s3-tests`, answers `aws-cli`,
`boto3`, `mc` and `rclone`, and runs inside a test in the same process as the code under test.

Adding lifecycle rules to it took one milestone. Finding out what the existing checks had not been
checking took two more, and that was the useful part. Three of them had been green the whole time.

## A lock that only ever answered "yes"

Object lock had shipped a milestone earlier with fifteen tests. Every one of them asked whether the
lock holds: put a retention on a version, try to delete it, expect a refusal. They all passed, and
they all pass today.

They could not see that the lock opened. `PutObjectRetention` refuses a change that **weakens** the
retention, and the check compared dates — later is fine, earlier is not. Changing `COMPLIANCE` to
`GOVERNANCE` keeps the date exactly as it was. Nothing was weakened by the only measure being
applied, so the request went through, and a promise nobody can break became one anybody can.

The second half was worse and quieter. `CreateMultipartUpload` takes the same `x-amz-object-lock-*`
headers as a `PUT`, and the object they protect appears minutes later at the completion. The upload
carried the metadata, the checksum algorithm and the checksum type across those minutes; it did not
carry the lock. A multipart upload created under legal hold finished as an object anyone could
delete, and the client was told the upload had succeeded — which it had.

Both are now the same shape of test, written the other way round: ask for something forbidden and
require the refusal. The version is also born locked in one index record rather than locked a moment
after it exists, because the gap between "the object is there" and "the object is protected" is
exactly what a lock is bought to close.

## An "out of scope" label is a defect's best hiding place

Most of `ceph/s3-tests` exercises things this store does not have, so every failure carries a
classified reason in a file in the repository — `out-of-scope`, `deferred` or `defect` — and a
failure matching no rule is reported by name as `unclassified`. That count has been zero for a long
time, and the zero was doing real work: it is the number that gets watched.

The reasons underneath it were rotting. `cors`, `tagging` and `object_lock` all said "out of the v1
scope" — about features built six and ten milestones earlier. Twenty-nine cases sat behind
sentences that had stopped being true, and nothing in the tooling can notice that, because a rule
that matches keeps matching whatever it says.

Re-reading the reasons rather than the counts turned up **eight real defects**, including all three
locks above and an `x-amz-tagging` header that closed the connection instead of answering. None of
them was found by running anything.

An "out of scope" label over a defect is worse than no label at all. Unclassified is visible and
asks for attention; a wrong reason looks like a closed question, and closed questions are what
everyone scrolls past.

## The wire cannot be asked about nanoseconds

Lifecycle rules put work on the read path: every `GET` and `HEAD` now looks up the bucket's rules
and, if one matches, formats an `x-amz-expiration` header. A milestone that touches the read path
here does not close without a number, so: two machines, 1 KiB objects, presigned SigV4, seven runs
of ten seconds per variant.

The result was three variants inside their own noise — and in an order that contradicts the
mechanism. A bucket with no rules ought to be the cheapest; it came out between the other two. That
ordering is the signature of drift between runs minutes apart, not of cost. A request that costs
~280 µs across a public link cannot be asked about a map lookup.

Measured in-process instead, on one thread: **8 ns** for a bucket with no rules, 62 ns when rules
exist and do not match, 555 ns when the header is built — of which the rule lookup is 62 and the
rest is `DateTimeFormatter`.

The first version of that measurement said **2 ns** for the no-rules case. Six cycles for two
`ConcurrentHashMap` lookups does not happen. Every branch returned a constant zero into a sum that
went nowhere, so the JIT deleted the calls, and the number was a measurement of the optimiser
wearing the shape of an excellent result.

## The rest of it

- A suite case named `test_lifecycle_set_invalid_date` sends `'20200101'` and expects a refusal.
  botocore reads that as **epoch seconds** and puts `1970-08-22T19:08:21Z` on the wire — a perfectly
  well-formed date. The only thing separating it from a real one is S3's rule that the time is
  always midnight UTC. A test that looks like a parser check is a semantics check.
- The XML reader refused self-closing elements on the grounds that accepting syntax nobody sends is
  surface with no purpose. The rule was right and the fact was wrong: botocore writes an empty
  structure and an empty string **only** that way, so `Filter {}` arrived as a malformed document
  from the standard client.
- `handle` had been wrapped in try/catch since it was written. `screen`, which reads the request
  head, never was — so an exception while parsing a header left the client with a closed socket and
  no bytes at all. The compatibility suite reported that as `ConnectionClosedError` for a whole
  milestone, which reads as a network fault rather than as a server that refused to answer.
- One sweep of lifecycle rules over a million versions takes **4.5 s** against a period of an hour,
  so passes cannot overlap. Depth is cheaper than width: 4.47 µs per version across a million keys,
  1.46 µs across a thousand keys a hundred versions deep.

## A check earns its green by being able to go red

Not by passing. Each of these three could pass forever without touching what it claimed to cover: a
test that only asks whether a lock holds cannot see it open; a classification rule keeps matching
long after its reason stops being true; a benchmark whose result the optimiser is free to delete
measures the optimiser.

The practical form is a question to ask of anything green. **What would have to break for this to go
red, and is that the thing I say it protects?** For the lock, that meant writing the tests that ask
for something forbidden. For the classification, re-reading reasons instead of counts. For the
benchmark, making every branch return something different and putting the sum in the output.

---

Source: [github.com/youndie/bochka](https://github.com/youndie/bochka)