The phone that couldn't be cleaned: a dead end nobody could see

One of our six phones spent over six hours this week in a state that, on paper, should not exist: it was available — rentable, listed in the catalog — and it was flagged wipe_required — meaning the rent gate would refuse anyone who tried to rent it. Available and un-rentable. Also, as it turned out, un-wipeable. It was stuck, and nothing in the system was going to unstick it on its own.

This is the story of how it got there, why nothing noticed, and the anticlimactic way it fixed itself the moment we stopped looking at the wrong door.

A flag designed to outlive its neighbor

wipe_required exists on purpose as something that can be true even after status has moved on. The idea: a phone finishes a rental, gets parked back into available while the wipe queue picks it up in the background, and only the rent gate needs to know it's still dirty. Customers shouldn't see "sanitizing" for a wipe that takes four seconds; they should just never be handed a phone that hasn't been cleaned.

That's a reasonable design. The trap was that four different functions across the codebase had each independently decided the honest way to answer "does this phone still owe a wipe?" was to read status, not the flag built for exactly that question. A new source of truth is only true in the places that were actually updated to read it. Everywhere else, it's still lying, just less obviously.

How it got in

The entrance was almost polite. An admin flipped a phone from maintenance to available — a completely reasonable action, done in good faith — and 284 milliseconds later the Pi sent its next wipe-failure report for that same device. That report no-op'd, silently, because the handler that processes it only fires when status = 'sanitizing'. The phone was already available by then. Nothing told the admin the flag hadn't cleared. As far as anyone looking at a screen could tell, the wipe had finished normally.

There was a second way in that needed no admin at all: report_device_presence — the function that fires when a phone reconnects — promoted offline straight to available without ever checking wipe_required. Unplug the USB cable and plug it back in, and you'd land in the identical trap. No human required, no audit trail to follow.

Why it never healed

Here's the part that actually kept it stuck. The wipe queue's own query was:

sanitizing OR (maintenance AND wipe_required)

An available phone with wipe_required = true matches neither branch. It wasn't being actively sanitized, and it wasn't in maintenance. So it simply fell out of the queue. The Pi never asked about it again — not because it was broken, but because nothing was offering it the phone anymore. A wipe that never gets attempted can't fail, and it can't succeed either. It just sits.

The twist

Once we understood the shape of the trap, the obvious fix was to make the flag itself the thing the queue keys on, not a combination of columns that could silently stop matching. We half-expected the phone would also need someone physically at the rack — an earlier read of this incident assumed the screen lock left over from the last renter would need a human to clear it.

It didn't. The instant the queue fix went live, the Pi was offered that phone for the first time in over six hours, and the wipe just worked — seven renter files removed, done in under three seconds. The lock had already cleared itself ages ago. The phone hadn't been waiting for a fix. It had been waiting for someone to check on it again, and nothing ever had.

The lesson underneath that is worth sitting with: a "this needs hardware intervention" verdict has a shelf life. Re-test before you accept it, because the ground it was standing on can move while you're not looking.

The retry storm we didn't know we'd been running

Fixing the dead end surfaced something less charming. It turned out the trap had been the only thing capping retries. A phone that's maintenance and flagged is deliberately still in the queue — that's correct — but with no dead end to eventually fall into, the agent had been hammering that state at one attempt per eight-second poll. Before this phone dropped into the trap, it logged 387 failed wipe attempts in 53 minutes. That's 387 of the 469 total rows in the entire wipe-attempt history table, from one phone, in less than an hour.

So the real fix isn't just "join the queue correctly" — it's also a cooldown: 30 seconds, doubling up to 15 minutes, scaled by how many failures have piled up since the last success. A phone that wipes cleanly every time is never delayed. A phone that's actually stuck stops drowning the log instead of quietly becoming the majority of it.

What actually shipped

The dead end closes with a queue predicate that keys on the flag, not a status combination that can drift out of sync with it. The retry storm closes with a real backoff, applied on the database side so it holds regardless of which version of the wipe agent happens to be running on the Pi that week. And there's now a step-up-gated admin action to clear the flag by hand, for the day this happens again and nobody wants to wait for a queue fix to deploy.

Six phones, zero flagged, and one very small database query that now means what it says.