A ledger that cannot be read must mean stop, not “nothing recorded yet

The failure: a power cut turned a permission ledger into a green light

We run an automation that writes comments on other people’s posts. To keep it from repeating itself, it consults a permission ledger before acting. The ledger records what has already been written, and the system treats it as the authority on what it may do next.

A power cut left that ledger unreadable. The system treated unreadable as “nothing recorded yet”. An empty record says that nothing has happened, so every comment ever written counted as not yet written, and the system re-authorised all of them.

Nobody wrote a rule saying that when the ledger is broken, everything is allowed. What happened is that “I cannot tell” was given an answer by default, and the default was “go ahead”.

The mechanism: three states squeezed into two

A check against a ledger has three possible answers:

1. Recorded. This action has already happened, so refuse.
2. Not recorded. The ledger was read, this action is not in it, so proceed.
3. Unknown. The ledger could not be read, so nothing can be said about the action.

Our system had two outcomes for three states. Unknown was folded into not recorded, and from the permission check’s side, a ledger that had been read and found empty looked identical to a ledger that could not be read at all. Both said “go”.

Merging those two states is a decision about cost, whether or not anyone intended it. Every fallback encodes an assumption about which error is cheaper. Treating unreadable as empty assumes that wrongly allowing an action costs less than wrongly blocking one.

For this ledger, that assumption is backwards. The ledger exists to prevent repeats. Wrongly blocking an action costs a delay: a comment waits until someone repairs the file. Wrongly allowing costs the very thing the ledger was built to stop, which is every comment ever written becoming eligible to go out a second time. The cheaper error was blocking, and the system picked the dearer one.

Why a fail-open default stays quiet

Compare the two directions a failure can take.

• A fail-closed ledger that cannot be read stops the job. The stop produces an error, someone sees it, and the file gets repaired. The failure is loud because it is inconvenient, and the noise is what gets it fixed.
• A fail-open ledger that cannot be read lets the job carry on. The job runs, the run looks like any other, and nothing in it says the safeguard was missing.

The inconvenient failure announces itself, and the dangerous one passes without a sound. That asymmetry is why this kind of default survives. Nobody is prompted to revisit it, because from the outside it never misbehaves until the day it matters.

We have seen the same shape in our own stack, in a different place. One lane died for six days (30 August to 5 September 2026) and every log read clean. It crashed while assembling its own context, after printing a normal start line and listing every profile it was about to visit. The log looked busy right up to the traceback, and only the exit code at the very end told the truth. We found it by auditing exit codes across fourteen lanes, not by reading a single log body.

A clean log tells you nothing about whether the work happened. An unreadable ledger that raises no alarm tells you nothing about whether the safeguard ran. In both cases the absence of an error was being read as evidence, and it is not evidence.

Why it bites harder when the system acts alone

A person handed a damaged spreadsheet has a chance to look at it and say that it seems wrong. An automated system has no such moment. It reads the ledger, gets whatever answer its code produces, and acts on that answer with nobody watching the individual step.

That changes what a safeguard has to do. In a manual process, a missing record gets questioned by whoever notices. In an automated one, the missing record is just an input, and the code decides what it means before any person sees it. Whichever way the code guessed is the way every action goes until someone looks.

The ledger failure needed no attacker and no clever input. It needed a power cut. Reviews of automated systems tend to start with who might attack from outside, but this fault came from inside: the system lost the thing it consults before acting, and it had no stated position on what that should mean.

What a ledger reader has to do

The remedy is a design rule rather than a feature.

• Keep “empty” and “unreadable” as different answers. The reader should be able to report “I read it and nothing is recorded” and “I could not read it” in ways the caller cannot confuse. An empty result must only ever mean that the ledger was read.
• Decide the direction for each safeguard, in writing. For a ledger that prevents repeats, unreadable means stop. For a monitor that only reports, unreadable might mean carry on and flag it. What you cannot do is leave the direction to whatever happens by default.
• Make the stop visible. A job that halts because its ledger is unreadable should say so somewhere a person looks, and should exit non-zero, so the halt cannot pass for a successful run. The six-day lane is the reason for that last part.

None of this is hard to write. The difficulty is that nothing forces you to write it until the first time the fallback fires, and by then it has already chosen for you.

A test you can run this week

Pick any check in your system that asks “has this already happened?” It might be a deduplication table, a sent-messages log, an approvals record, or a rate limiter. Then break its source on purpose in a test environment: truncate the file, revoke read access, or point it at a path that does not exist.

Watch what the system does next. There are three outcomes:

1. It stops and says why. That is the behaviour you want.
2. It proceeds as if nothing was ever recorded. That is the failure described above.
3. It proceeds, and you cannot tell from the output which of the first two happened. That is worse, because you have no way to find out later.

If you get the second or third outcome, you have found a safeguard that works only while its storage is healthy.

Where we stand

Our position is that “I cannot tell” is a state that needs its own answer, and that for any safeguard whose job is to prevent harm, the answer is to refuse. A fail-open default feels convenient because the system keeps running. That convenience is the cost: the system is running without its safeguard and showing no sign of it.

Aethon Automation builds systems that act on their own. Much of that work is deciding, dependency by dependency, what the system does when that dependency goes dark. If you want a second set of eyes on those decisions in your own automation, you can book a consultation with us.

When one of your systems cannot tell whether something already happened, which way does it guess?