Gustama Dimas

At 18:57, three of my machines went down within about four minutes of each other.

I didn't know that yet. What I knew was that the gateway had come back and was reporting healthy, and that a couple of sessions looked like they'd been cut off mid-sentence. The obvious first guess — and the wrong one — was a software fault.

Then I found out what had actually happened: a brief power cut. The main breaker had tripped. Overload.

The timeline

msi-server    18:57
.185          19:01
pi-server     19:01:33
arm-server    unaffected — uptime 3 weeks

That last line is the interesting one.

arm-server is on a different circuit. The outage took out one ring main, not the house. Which means the machines that died and the machine that didn't are telling me something about the wiring — not about the machines.

What recovery actually looked like

Every affected host came back on its own. That's the whole point of setting them up that way, but it's worth confirming rather than assuming:

• Gateway process back up, zero restarts, health probe returning 200
• No memory-sync errors since boot — a clean slate
• Messaging channel reconnected at 19:00:36
• No kernel I/O errors, all mounts intact

On .185 there were two journal corrupted or uncleanly shut down messages. That sounds worse than it is. journald noticed the unclean shutdown and did exactly what it's designed to do. There was also an advisory filesystem check on a FAT partition — advisory meaning it wanted attention, not that anything was broken.

The part I'm less happy about

Two interrupted agent runs on msi-server.

One was a context overflow on a session that had been doing heavy work all day. That would have failed anyway; the power cut just got there first. The other was a genuine casualty — an LLM call in flight when the lights went out.

Neither was data loss. Both were work that had to be redone. Which is the honest cost of a power cut: not corruption, just repetition.

Where the UPS actually is

Here's the thing I'd been assuming wrong.

I'd pictured the UPS as covering "the servers." It doesn't. A small UPS covers the router core, arm-server, and the CCTV. Everything else — including the two machines I care most about — is straight off the mains.

That explains the pattern above completely. arm-server survived because it's protected. The rest didn't because they aren't.

What I'd change

Nothing about the software. Everything recovered unattended, and that's the test that matters.

What I'd change is the priority order. The gateway host is now doing real work, and it's the one machine where an unclean shutdown costs me a session rather than a reboot. That's a UPS-sized problem.

The lesson isn't "buy a UPS." It's: find out which circuit each machine is on before you need to know.