The restore drill: testing backups you have never restored

2026-10-12 · backups, restic

For three years I had nightly restic backups of every machine I look after, a green checkmark in my monitoring, and no evidence at all that any of it could be restored. A backup you have never restored is a hypothesis. This spring I finally started testing it.

The rule

Once a quarter, pick one machine at random and rebuild its data on a throwaway VM using only what is written down. No logging into the original box to "check something". If I need to, the documentation is wrong, and fixing it is part of the drill.

The routine

# 1. list snapshots and pick the most recent one
restic -r sftp:backup@vault:/srv/restic/web1 snapshots --latest 3

# 2. restore into a scratch directory
restic -r sftp:backup@vault:/srv/restic/web1 restore latest \
    --target /scratch/web1 --include /etc --include /var/lib/postgresql

# 3. verify the repository itself (reads 5% of pack data)
restic -r sftp:backup@vault:/srv/restic/web1 check --read-data-subset=5%

Then the important part: actually start the service from the restored files and see if it answers a request. Files existing is not the same as a database that starts.

What the first drill found

• The PostgreSQL backup was a file-level copy taken while the server was running. It restored, and then refused to start. I now dump with pg_dump -Fc before the restic run and back up the dump.
• The repository password lived only in a file on the machine being backed up. Restoring a dead machine would have required the dead machine.
• Restoring 40 GB over the home uplink took most of an afternoon. Fine for a drill, worth knowing before a real outage.

Timing it

The mail server took longest to bring up relative to its size, almost entirely because of DNS and certificate steps that were not in the notes. They are now.

The goal is not a perfect restore. The goal is to find out what is missing while nothing is on fire.