An unreadable permission ledger must halt the run, not re-authorise everything

The failure: unreadable was read as empty

We run a ledger that records which people have already had a comment written on their post. Before the system writes anything, it checks the ledger. If the person is on it, the system stays quiet. If not, it may go ahead.

A power cut left the ledger unreadable. The system treated “unreadable” as “nothing recorded yet.” An empty ledger means everyone is new, so every comment ever written was authorised again.

The ledger held the records it was meant to hold. The failure was in the answer to one question: when the system cannot tell whether something already happened, what does it do? That answer was “go ahead,” and nobody had chosen it on purpose.

The mechanism: every fallback is a bet on which error is cheaper

Any code that reads state has three outcomes: the state says yes, the state says no, or the state could not be read. The first two get attention. The third gets whatever answer happened to be written in, and that answer is a position on which mistake costs less:

• Fail open says that wrongly acting is cheaper than wrongly waiting.
• Fail closed says that wrongly waiting is cheaper than wrongly acting.

Neither is right everywhere. A lock on a fire exit should fail open, because a trapped person costs more than an unlocked door. A ledger that gates an outward action sits on the other side. A comment, once published, is public and attached to someone else’s post, so a wrong “go ahead” cannot be taken back. A wrong “wait” costs a delayed run, and a later run can make it up.

For this ledger, the cheaper error was never in doubt. The trouble is that the system chose the more expensive error without anyone deciding to. “Unreadable means nothing recorded” is what the system does when a read failure is folded into the same path as an empty result.

Why folding the two together is tempting

Treating a failed read as an empty record looks like good manners. A new system has no ledger on its first run, so the code that loads one has to cope with a missing file. The shortest route is to treat “nothing there” and “can’t read it” as the same case and carry on.

That covers the first run, and it covers corruption in the same stroke. The two situations need opposite answers:

• The record does not exist because the system has never run. Empty is true. Nothing has happened.
• The record exists and cannot be read. Empty is false. Something happened, and the record of it is out of reach.

Merging those two branches turns “we don’t know” into “nothing happened,” and the system then acts on a fact nobody established. The remedy I would defend is that the loading step tells “absent” apart from “present but unreadable,” and the second one stops the run. The caller should never receive a value it could mistake for a clean slate.

The same shape in three other places

The ledger is one instance of a pattern. Three other entries in our records have the same structure: a signal that cannot tell two different situations apart, with the system leaning toward proceeding.

• One exit code carried two meanings. A lane returned the number of messages it had sent, and the caller compared that to 2, which was also the code for a dead session. A run that delivered exactly two messages reported itself as dead. In reverse, a genuinely dead session became indistinguishable from a good run, and both were suppressed. A success and a failure sharing one signal is worse than no signal.
• A hold maintained by memory. A weekly job was disabled by commenting out its schedule line, while the code it ran already refused correctly on its own. The comment was a second, human-maintained gate that would sit there until somebody remembered it.
• A clean log. A lane died for six days, 30 Aug to 5 Sep 2026, and every log read clean. It crashed while assembling its own context, after printing a normal start line and listing every profile it was about to visit. The log looks busy right up to the traceback, and only the exit code at the very end told the truth. It was found by auditing exit codes across fourteen lanes, not by reading a single log body.

In each case a state nobody had classified was read as the comfortable one. The absence of an error was taken as evidence, and it carries no information about whether the work happened.

Where this meets data governance

The ledger is a small version of something larger organisations are asked to maintain: a record of what was done, to whom, and with what permission. Consent records, erasure requests and records of which data went into which system are all ledgers. A regulator can ask whether the record exists. The sharper question is what your systems do when the record cannot be consulted.

A governance record that fails open is a gap in the control, however good the policy document looks. If a consent store goes down and the pipeline reads that as “no objections on file,” the control inverts at the moment it is needed. The same applies to AI-assisted tooling. A model or an agent that acts on a permission store needs a written answer for the case where the store is silent, and that answer should be “do not act.”

None of this needs a new product. It needs someone to find every place a state is read and ask what happens on the third outcome.

How to audit your own fallbacks

You can run this review in an afternoon. For every place your automation reads a record of past action, check four things:

1. What does the code do when the read fails? Find the exception handler, the default argument, the or {}. Read what it returns.
2. Can the caller tell “empty” from “unreadable”? If both arrive as the same value, they will be confused eventually.
3. Is the action reversible? A re-sent internal report can fail open. A public comment, a payment, a deletion or a message to a customer cannot.
4. Does a stop reach a person? Failing closed only helps if someone learns the run was halted. A silent stop is a second failure, and it needs its own alert.

The fourth check matters because curing a fail-open path can leave a silent stall in its place. A halted system that says nothing looks the same as a healthy idle one, which is the clean-log problem again. The halt has to be loud even when the failure was handled correctly.

The position

The default on a failed read should be set by the cost of the action it gates, not by what is convenient for the code that loads the record. If the action cannot be undone, “I can’t tell” means stop, and a person finds out.

The ledger incident is a reminder that a fallback encodes an assumption about which error is cheaper, whether or not anyone wrote that assumption down. One power cut was enough to show which way ours leaned.

So the question for your own stack: when one of your systems cannot tell whether something already happened, which way does it guess?

Aethon Automation Solutions engineers the systems that power your business. We work with organisations in regulated industries where a wrong default carries compliance consequences. If you want a second pair of eyes on how your automation behaves when its records can’t be read, book a consultation with our team.