Gustama Dimas

This site runs on Chyrp Lite — a small, self-hosted blogging engine that does roughly what I need and nothing else. It's been up for a while on the default theme, which is perfectly good and completely anonymous. I finally rewrote it.

What was wrong

Not much, honestly. The theme worked. But three things bothered me.

First, the navigation listed ten pages in one flat list, and seven of them existed only to embed a service in an iframe. Several of those embeds pointed at endpoints that no longer answer, or that refuse to be framed at all. The sidebar was offering links to broken panels.

Second, the layout was a single narrow column with a sidebar and no sense of what the site was for. A visitor landed on it and learned my name. That was about it.

Third, the content was a list of facts with no way in.

What I changed

A real homepage. Instead of dropping visitors straight into the blog index, there's now a front page: who I am, where I am, what I do, and four ways to go deeper. It answers the question the old site left hanging.

Navigation that reflects the actual structure. The per-service pages are gone from the menu. They still exist — every old link still resolves — but the Home Lab page now links out to each service directly, with a one-line description of what it is. Seven iframe panels became one readable index.

A theme with an opinion. IBM Plex Sans and Plex Mono, self-hosted rather than pulled from a CDN. Monospace is reserved for labels, metadata, and anything that reads as data. The palette is warm off-white with a single copper accent, used for links and almost nothing else.

The service list is styled as a rack: index number, name, what it does, and whether it's open to the world or only reachable on the LAN. That last column is the honest one — it tells you before you click whether a link is about to ask for a password.

Accessibility wasn't an afterthought

• A skip link for keyboard users, since the sidebar comes first in the markup
• Visible focus outlines on every interactive element, in the accent colour
• prefers-reduced-motion respected — no transitions, no smooth scrolling, for anyone who's asked for neither
• prefers-contrast: more pushes borders and text to full black
• A print stylesheet, because these pages are worth keeping on paper

What I'd do differently

Write the content before the theme.

I spent a while designing components — a hero, a status strip, a rack table — and then had to go back and rewrite the page copy to fit them. The design turned out fine. The order was backwards.

Next time: decide what the page needs to say, then decide what it needs to look like.

At 00:32 a log line started repeating. By 16:45 it had appeared 785 times.

[memory] sync failed … Plugin "memory-core" runtime is no longer active

The memory plugin on my assistant gateway had stopped working. Not crashed — something worse. It was still installed, still reporting as loaded, and completely unable to do anything.

The wrong fix

I reloaded the plugin. The errors stopped.

For fourteen minutes.

Then traffic picked up, and they came back exactly as before. That fourteen-minute silence was the most misleading thing in the whole incident. It looked like a fix. It wasn't. It was an idle window with nothing left to fail.

The actual cause

I stopped guessing and read the runtime source.

When a plugin is hot-reloaded, the gateway builds a new registry generation. Anything still holding a reference to the old generation is now holding a dead handle. The memory-sync bridge — the part that embeds and indexes content — had captured its reference at startup and never re-pointed it. Every call after the reload threw.

This is why reloading the plugin forever would never have worked. The plugin was fine. The thing calling it was holding a stale pointer, and no reload of the plugin itself can fix that.

The fix

A full gateway restart. That rebuilds every generation from scratch, so the bridge captures a live handle again.

The numbers before and after: 815 occurrences of the error across the day, the last at 16:52:03. After the restart, zero. Not "zero so far" — zero across every sync since.

Two things worth keeping

The CLI was healthy the entire time.

Through all 785 failures, memory status --deep reported embeddings ready, vector store ready, semantic vectors ready. The data layer was never broken. Only the in-gateway path was.

If I'd trusted that status command and stopped there, I'd have concluded everything was fine — and been wrong in a way that mattered. A health check that tests the wrong path is worse than no health check, because it hands you confidence you haven't earned.

A quiet window is not evidence.

I checked after the reload, saw nothing for fourteen minutes, and nearly called it fixed. The correct check was under load, or with a deliberate trigger. Silence during idleness proves nothing about a code path that only runs when there's work to do.

That second one I've now written down properly, because I've made the same mistake before in a different costume.

At 18:57, three of my machines went down within about four minutes of each other.

I didn't know that yet. What I knew was that the gateway had come back and was reporting healthy, and that a couple of sessions looked like they'd been cut off mid-sentence. The obvious first guess — and the wrong one — was a software fault.

Then I found out what had actually happened: a brief power cut. The main breaker had tripped. Overload.

The timeline

msi-server    18:57
.185          19:01
pi-server     19:01:33
arm-server    unaffected — uptime 3 weeks

That last line is the interesting one.

arm-server is on a different circuit. The outage took out one ring main, not the house. Which means the machines that died and the machine that didn't are telling me something about the wiring — not about the machines.

What recovery actually looked like

Every affected host came back on its own. That's the whole point of setting them up that way, but it's worth confirming rather than assuming:

• Gateway process back up, zero restarts, health probe returning 200
• No memory-sync errors since boot — a clean slate
• Messaging channel reconnected at 19:00:36
• No kernel I/O errors, all mounts intact

On .185 there were two journal corrupted or uncleanly shut down messages. That sounds worse than it is. journald noticed the unclean shutdown and did exactly what it's designed to do. There was also an advisory filesystem check on a FAT partition — advisory meaning it wanted attention, not that anything was broken.

The part I'm less happy about

Two interrupted agent runs on msi-server.

One was a context overflow on a session that had been doing heavy work all day. That would have failed anyway; the power cut just got there first. The other was a genuine casualty — an LLM call in flight when the lights went out.

Neither was data loss. Both were work that had to be redone. Which is the honest cost of a power cut: not corruption, just repetition.

Where the UPS actually is

Here's the thing I'd been assuming wrong.

I'd pictured the UPS as covering "the servers." It doesn't. A small UPS covers the router core, arm-server, and the CCTV. Everything else — including the two machines I care most about — is straight off the mains.

That explains the pattern above completely. arm-server survived because it's protected. The rest didn't because they aren't.

What I'd change

Nothing about the software. Everything recovered unattended, and that's the test that matters.

What I'd change is the priority order. The gateway host is now doing real work, and it's the one machine where an unclean shutdown costs me a session rather than a reboot. That's a UPS-sized problem.

The lesson isn't "buy a UPS." It's: find out which circuit each machine is on before you need to know.

I have two machines that matter.

msi-server is the newer one — a small MSI Cubi 5 running Ubuntu Server, and the box my assistant gateway lives on. .185 is the older desktop, still doing useful work, still holding a few things I haven't got round to migrating.

For a long time the only channel between them was me. I'd notice something on one machine, walk over to the other, and retype it. That works fine right up until it doesn't.

So I built a relay. Not a clever one — two one-way pipes.

Pipe one: .185 → me

A small script on .185 posts a message to an HTTPS endpoint on the gateway. It's token-authenticated, restricted to a single agent, and delivered to my phone labelled 📡 .185 so I always know which machine is speaking.

The important part is what it doesn't do. Nothing on .185 listens for inbound connections for this. The machine only ever makes outbound requests. No new port, no new attack surface.

Pipe two: me → .185

This is the interesting half, because the obvious design is wrong.

The mirror-image approach would be an HTTP endpoint on .185 that I post to. I rejected it. That machine has a history: a gateway process that occasionally hangs, and an agent database that has grown large enough to overflow its context window under real work. Adding a new always-listening inbound surface to a box with those properties is how you end up with a service that dies at 3am and leaves no trace of why.

Instead I reused something already trusted and already tested: SSH.

The flow is one command:

1. I decide to send something to .185
2. A script on msi-server opens an SSH connection
3. It runs the agent CLI there against a fixed session key
4. The agent's reply comes back on stdout
5. I relay that into the chat thread

One round trip. No new port open anywhere.

What it looks like when it works

Four tests, all end to end:

transport smoke     → PONG-185-FASE2
simple message      → FASE2-V2-OK
tool use on .185    → returned hostname, uptime, version
final               → FASE2-FINAL-OK

The third one is the one that mattered. It didn't just prove the pipe carried text — it proved that a request originating on one machine could reach the other, cause it to run a shell command there, and bring the real output back. That's the difference between a notification and a remote control.

The limitation, stated honestly

The relay is synchronous. A request waits for a reply, and the timeout is 300 seconds by default. For chat and short queries that's generous. For anything heavy — a ten-minute build, a long coding task — it will time out, and the work gets orphaned.

The fix is an async variant: fire the request, get an acknowledgement, deliver the result when it's ready. I haven't built it, because nothing yet needs it. Writing it now would be writing code for a problem I don't have.

What I'd take from this

The best interface between two machines is usually the one that's already there.

SSH was already trusted, already keyed, already understood. An HTTP endpoint would have been more modern and less reliable. The question isn't "what's the correct way to do this" — it's "what's the smallest new thing I can add to make this work."

Here the answer turned out to be: nothing, on the receiving end. Just a script and a key that already existed.