A green tick from your backup software means a job finished and some data landed somewhere. It says nothing about whether you could rebuild the business from that data, or how long it would take. Until you have restored something, you have an assumption, and it is one you may only get to test on the worst day of your year.

Start with two numbers

These two terms get confused often, so it helps to be precise.

  • Recovery point objective (RPO): the maximum amount of data loss you can accept, measured in time. An RPO of one hour means losing up to an hour of work is tolerable. Nightly backups can't meet it, since a failure late in the day could lose nearly 24 hours.
  • Recovery time objective (RTO): the maximum acceptable time to restore a service after a disruption. An RTO of four hours means it must be working again within four hours.

Both are business decisions, because both trade cost against harm. Ask the people who run each function what an hour of lost orders or a day of downtime would mean to them, then show them what each option costs. Set the values per system, as your payroll, website and data warehouse will have different needs. Also check they are consistent: if the finance application has an RTO of four hours, everything it depends on must be back in less than that.

Why restores fail

Backup failures tend to be discovered in predictable places:

  • Missing keys. The data is safely encrypted, and the key was in the system you lost.
  • Partial scope. You have the database, but not the secrets, configuration, DNS, certificates or the deployment tooling.
  • Slow transfer. The restore works, but pulling many terabytes over the available link takes days, well outside the RTO.
  • Inconsistent data. A snapshot taken while a database was writing doesn't start cleanly, or related systems were backed up hours apart.
  • Unprotected SaaS. Your data in a hosted service may have retention limits that don't match your assumptions. Read the contract and check what the provider restores for you.
  • Lost knowledge. Only one person knows the procedure, and they are on leave.

Ransomware changes the design

Attackers who reach your administrator accounts commonly go after backups first, because destroying them makes payment more likely. A backup that your domain administrator can delete won't protect you from someone holding those credentials. Design for that case:

  • Immutability. Use write-once storage, such as object lock with a retention period that nobody, including administrators, can shorten during that window. Check the mode you have selected, because some allow privileged override.
  • Separate credentials. Hold backups in a separate account or tenant, with different administrators and strong authentication, and don't join backup systems to your main domain.
  • An offsite or offline copy. The old 3-2-1 rule remains a sensible baseline: three copies, on two types of media, with one offsite.
  • Enough history. Attackers may be present for weeks before acting. If you only keep 14 days of backups and the intrusion is older, every copy you hold may contain their foothold.
  • Monitoring. Alert on deleted backups, shortened retention and sudden changes in backup size or compression, which can signal encryption in progress.

Restore into an isolated network, and check for persistence before reconnecting anything. Plan for this taking longer than a normal recovery, because it will. Note too that being locked out of personal information can count as a privacy breach under New Zealand's Privacy Act 2020, even where nothing was copied.

Run real drills

Treat a restore drill as a timed test with a written result. Rotate the systems, and include a full-service recovery at least once a year, not only single files. Have someone other than the usual administrator follow the runbook, so that gaps in it show up. For each drill, record:

  • What was restored, and from which backup.
  • The age of the data recovered, compared with the RPO.
  • The elapsed time from start to working service, compared with the RTO.
  • How correctness was checked, such as application logins, record counts and reconciliation against a known figure.
  • What broke, and who is fixing it.

Starting a database process is a poor test of success. A person using the application to complete a real task is a much better one.

What to do this week

  1. Pick your most important database, and restore last night's backup to an isolated environment.
  2. Time it from the moment you begin, including finding the keys and credentials.
  3. Run a few checks that prove the data is usable.
  4. Create a disposable test backup, then see whether an ordinary administrator account can delete it. If it can, you have your next priority.

Then give the result to whoever owns the risk. A measured number is more useful than any assurance.