Essay 05 / Resilience

Resilience Is a Practice, Not a Product

Redundancy sits on a shelf. Resilience is what happens when someone knows how to use it.

A rechargeable work lamp is tested beside a paper checklist, timer and unplugged charging cable.
Readiness is something to rehearse. An original illustrative scene.

There is a pleasing certainty to buying a spare. The object arrives, a gap appears to close, and a future problem feels smaller. Sometimes it really is smaller. A replacement part on the shelf can save days of interruption.

But the part is only one condition of recovery. Someone must notice the failure, distinguish it from other faults, find the spare, know whether it is still suitable, and fit it. The object carries a promise. Practice is how we discover what that promise requires.

Preparedness has a maintenance cost

Every reserve ages in some way. Batteries lose charge. Stored instructions drift away from the equipment they describe. Software changes. Contact details become obsolete. People forget procedures they have never used.

This is not an argument against preparation. It is an argument for including maintenance in its price. An arrangement that needs more attention than we can give will gradually become a collection of untested assumptions.

Simple preparations often perform well because they are easy to inspect. A labelled document is easier to keep current than an elaborate dashboard. A familiar tool is easier to use under stress than a specialised device still in its packaging. A short recovery procedure is more likely to be rehearsed than a binder written for every imaginable event.

Recovery is a sequence

It helps to walk through the sequence in ordinary language. How would we know something was wrong? Who decides what to do? What is the first safe action? Where are the instructions? What would tell us that recovery has succeeded?

These questions reveal gaps that inventory lists miss. A backup may exist, but no one has agreed which version to restore. A substitute communication channel may work, but no one knows when to check it. A set of keys may be accessible, but the person who needs them does not know where they are kept.

A recovery plan is a hypothesis until someone has followed it.

Rehearsal does not have to be theatrical. Restore a noncritical file to a temporary folder. Ask another person to find a document using your instructions. Check that a stored contact number still reaches the intended person. Small exercises create useful evidence without disrupting daily life.

Intermediate · Restore one snapshot, do not admire the job log

A backup that has never been restored is a rumour. With restic (Borg is the same idea):

export RESTIC_REPOSITORY=sftp:backup-host:/restic/laptop
restic snapshots
restic restore latest --target /tmp/restore-test --include /home/you/Documents/sample.md
sha256sum /tmp/restore-test/home/you/Documents/sample.md

Check three things the dashboard will not: the snapshot is recent, the file opens in a second editor, and the restore did not need the original laptop. Then delete /tmp/restore-test. Write the date.

A second copy on the same disk is not an off-site. Power, theft, and filesystem bugs travel together.

Test the plan, not the person

A good exercise makes it safe to discover confusion. If someone cannot follow the instructions, that is information about the instructions and the system. It is not a reason to make them feel foolish.

The person who built an arrangement carries a great deal of invisible context. “Open the backup” may conceal knowledge about a drive, a password, a program, and a particular folder. Watching a second person work through the process is one of the quickest ways to reveal those assumptions.

Write down the actual steps after the rehearsal, while the details are fresh. Record what worked, what did not, and what changed. A brief dated note is often more useful than a polished plan with no evidence of use.

Do not simulate dangerous failures to make an exercise realistic. A tabletop conversation can explore a power interruption; a copy of a file can test restoration. Keep experiments away from live safety systems and seek qualified help for work that requires it.

Leave room to improvise

No plan can describe every future condition. Resilience includes the ability to recognise that the expected recovery path does not fit the situation and to choose another.

This is where understanding beats rote procedure. A person who knows what each part is doing can often construct a smaller working arrangement. A person who has only memorised a sequence may be stranded when one screen changes.

Documentation should therefore explain purpose as well as action. “Copy this folder because it contains the original source files” gives the reader more room to reason than “Copy folder A.” A diagram that shows where information moves can remain useful after particular software has been replaced.

Expert · The host that cannot boot is not a backup target

If the machine that holds restic also holds the only copy of RESTIC_PASSWORD, you have one object, not two. Split:

  • repository (off-site, append-only if the backend allows it),
  • password / key file (elsewhere; a printed envelope is more honest than “in Bitwarden, which is on this disk”),
  • a known-good boot image that can run restic without the original OS.

For a home server, NUT talking to a UPS is part of the restore path: an unclean ext4 journal after a brown-out is how “I had backups” becomes “the last snapshot is corrupt.” Test restic check --read-data-subset=10% on a schedule, not after the fire.

Make it ordinary

The most sustainable practice is attached to something that already happens. Review household records when another regular administrative task comes around. Rehearse a restore after changing the backup arrangement. Revisit a recovery guide when you replace the device it describes.

There is no universal interval. The appropriate rhythm depends on how quickly the system changes and what would happen if your assumptions were wrong. The important thing is that someone owns the next check and knows what successful recovery looks like.

Resilience does not arrive in a delivery box. It develops in the loop between preparation, use, observation, and adjustment. The spare still matters. So does the quiet afternoon when you discover whether it fits.

Next essay Back to the beginning ↑