bochka is an S3-compatible object store on the JVM: one process, one node, one disk. It passes 426 of the 744 cases in ceph/s3-tests, answers aws-cli, boto3, mc and rclone, and runs inside a test in the same process as the code under test.
Adding lifecycle rules to it took one milestone. Finding out what the existing checks had not been checking took two more, and that was the useful part. Three of them had been green the whole time.
A lock that only ever answered "yes"
Object lock had shipped a milestone earlier with fifteen tests. Every one of them asked whether the lock holds: put a retention on a version, try to delete it, expect a refusal. They all passed, and they all pass today.
They could not see that the lock opened. PutObjectRetention refuses a change that weakens the retention, and the check compared dates — later is fine, earlier is not. Changing COMPLIANCE to GOVERNANCE keeps the date exactly as it was. Nothing was weakened by the only measure being applied, so the request went through, and a promise nobody can break became one anybody can.
The second half was worse and quieter. CreateMultipartUpload takes the same x-amz-object-lock-* headers as a PUT, and the object they protect appears minutes later at the completion. The upload carried the metadata, the checksum algorithm and the checksum type across those minutes; it did not carry the lock. A multipart upload created under legal hold finished as an object anyone could delete, and the client was told the upload had succeeded — which it had.
Both are now the same shape of test, written the other way round: ask for something forbidden and require the refusal. The version is also born locked in one index record rather than locked a moment after it exists, because the gap between "the object is there" and "the object is protected" is exactly what a lock is bought to close.
An "out of scope" label is a defect's best hiding place
Most of ceph/s3-tests exercises things this store does not have, so every failure carries a classified reason in a file in the repository — out-of-scope, deferred or defect — and a failure matching no rule is reported by name as unclassified. That count has been zero for a long time, and the zero was doing real work: it is the number that gets watched.
The reasons underneath it were rotting. cors, tagging and object_lock all said "out of the v1 scope" — about features built six and ten milestones earlier. Twenty-nine cases sat behind sentences that had stopped being true, and nothing in the tooling can notice that, because a rule that matches keeps matching whatever it says.
Re-reading the reasons rather than the counts turned up eight real defects, including all three locks above and an x-amz-tagging header that closed the connection instead of answering. None of them was found by running anything.
An "out of scope" label over a defect is worse than no label at all. Unclassified is visible and asks for attention; a wrong reason looks like a closed question, and closed questions are what everyone scrolls past.
The wire cannot be asked about nanoseconds
Lifecycle rules put work on the read path: every GET and HEAD now looks up the bucket's rules and, if one matches, formats an x-amz-expiration header. A milestone that touches the read path here does not close without a number, so: two machines, 1 KiB objects, presigned SigV4, seven runs of ten seconds per variant.
The result was three variants inside their own noise — and in an order that contradicts the mechanism. A bucket with no rules ought to be the cheapest; it came out between the other two. That ordering is the signature of drift between runs minutes apart, not of cost. A request that costs ~280 µs across a public link cannot be asked about a map lookup.
Measured in-process instead, on one thread: 8 ns for a bucket with no rules, 62 ns when rules exist and do not match, 555 ns when the header is built — of which the rule lookup is 62 and the rest is DateTimeFormatter.
The first version of that measurement said 2 ns for the no-rules case. Six cycles for two ConcurrentHashMap lookups does not happen. Every branch returned a constant zero into a sum that went nowhere, so the JIT deleted the calls, and the number was a measurement of the optimiser wearing the shape of an excellent result.
The rest of it
- A suite case named
test_lifecycle_set_invalid_datesends'20200101'and expects a refusal. botocore reads that as epoch seconds and puts1970-08-22T19:08:21Zon the wire — a perfectly well-formed date. The only thing separating it from a real one is S3's rule that the time is always midnight UTC. A test that looks like a parser check is a semantics check. - The XML reader refused self-closing elements on the grounds that accepting syntax nobody sends is surface with no purpose. The rule was right and the fact was wrong: botocore writes an empty structure and an empty string only that way, so
Filter {}arrived as a malformed document from the standard client. handlehad been wrapped in try/catch since it was written.screen, which reads the request head, never was — so an exception while parsing a header left the client with a closed socket and no bytes at all. The compatibility suite reported that asConnectionClosedErrorfor a whole milestone, which reads as a network fault rather than as a server that refused to answer.- One sweep of lifecycle rules over a million versions takes 4.5 s against a period of an hour, so passes cannot overlap. Depth is cheaper than width: 4.47 µs per version across a million keys, 1.46 µs across a thousand keys a hundred versions deep.
A check earns its green by being able to go red
Not by passing. Each of these three could pass forever without touching what it claimed to cover: a test that only asks whether a lock holds cannot see it open; a classification rule keeps matching long after its reason stops being true; a benchmark whose result the optimiser is free to delete measures the optimiser.
The practical form is a question to ask of anything green. What would have to break for this to go red, and is that the thing I say it protects? For the lock, that meant writing the tests that ask for something forbidden. For the classification, re-reading reasons instead of counts. For the benchmark, making every branch return something different and putting the sum in the output.
Source: github.com/youndie/bochka