The Demise and Resurrection of STEENSBURROW

Published by steen on 2026-07-19 Tags: #nix #homeserver

A few nights ago, my partner and I decided it was the perfect time to install a specific Linux ISO. We'd heard glowing praise for it from the crew at Red Letter Systems, it had a stellar reputation on boundingboxd, and the installation trailer was particularly appealing to my partner.

I pulled up Tremotesf on my smartphone to see if the ISO was already sitting on my Raspberry Pi, but the torrent list failed to load. Rather than killing the evening's vibe with a tedious troubleshooting session, I assumed it was just a glitch caused by some recent tweaks to my NixOS configuration. I decided to deal with it the following day.

The Autopsy

The next morning, I retrieved my dedicated debugging monitor from the closet, cracked my knuckles, and declared:

"It's debugging time!" (or some variation thereof).

I rebooted the machine. The current NixOS generation attempted to boot, the kernel loaded, and then... silence.

The Troubleshooting Process

My first instinct was to run fsck. A building-wide power outage had occurred a few days prior; I suspected the power might have cut out exactly while the system was writing a critical boot block.

1. First attempt: fsck reported the filesystem was clean.
2. Second attempt: I ejected the SD card and re-inserted it.
3. Result: More errors surfaced.

The card was glitching. In reality, the microSD card was dead. The blackout had simply been the coup de grâce.

Taking Inventory

This Raspberry Pi 4B had been running nearly 24/7 for years, so the failure wasn't entirely shocking. It is widely understood that SD cards have a finite number of write cycles. If we represent the total write endurance as $W_{total}$ and the daily writes as $w_{day}$, the lifespan $L$ is roughly:

$L \approx \frac{W_{total}}{w_{day}}$

Given how poorly I had treated the card—using it as the root FS with almost no write-reduction measures—I'm actually surprised it lasted this long.

Data Loss Assessment

I've never had a primary drive fail like this before (only a few unimportant USB sticks). This triggered a deep-seated anxiety stemming from my teenage years, when I accidentally wiped my MacBook's main OS X partition while attempting my first Linux install. To prevent a recurrence, I decided to implement rigorous precautions.

Architecting the Rebirth

I approached the rebuild with a specific set of goals:

• Minimize write operations to the microSD card.
• Establish redundancy for data on external drives.

Handling Volatile Data

Since I have 8GB of RAM, swap is still useful, but writing swap to an SD card is a death sentence for the hardware. I had two choices: slow external HDDs or zram.

zram (formerly compcache) creates a compressed block device within the RAM. I enabled zram for swap but opted for a standard RAM disk for /tmp. I figured the kernel could manage swapping "cold" pages from /tmp into the compressed swap as needed.

# Example configuration snippet
{
  # Use tmpfs for /tmp to avoid disk writes
  useTmpfs = true; 
}

I also configured journald to store logs in memory. I'm still researching if I can periodically flush these to an HDD subvolume, but for now, they stay volatile:

{ 
  # Store journal logs in memory
  services.journald.storage = "volatile"; 
}

Finally, I disabled atime on the root partition to prevent a write every time a file is read:

" / " = { 
  device = " /dev/disk/by-label/takodachi "; 
  fsType = " ext4 "; 
  options = [ " noatime " ]; 
};

The External Storage Strategy

In the year 2026, buying new hardware is a nightmare. Between the "AI psychosis" and the corporate hunger for datacenter components, prices for RAM, GPUs, and drives are absurd. The only logical move is to be scrappy.

I have a "digital junk drawer" of old drives. I decided to put them to a higher purpose.

1. Drive A: A 500GB Hitachi drive with an Apple logo (likely from an old 2006 white MacBook or a Mac Mini).
2. Drive B: A drive from my partner's old 3KG "portable" business laptop (a behemoth of a machine).

Side note: I eventually replaced that laptop's HDD with an SSD and moved my partner to a Surface Pro 5 running NixOS. While setting up someone else's machine with NixOS might sound like domestic violence a risky move, they are actually very happy with it.

The Btrfs Pool

I connected these two relics to a docking station and initialized a btrfs pool with RAID1 replication for redundancy:

sudo mkfs.btrfs --data raid1 --metadata raid1 --label ponkotsu /dev/sdX /dev/sdY

I chose btrfs because of my familiarity with it (I use it on my desktop and laptop), my appreciation for subvolumes, and its overall flexibility.