Secure Backup: The Offline and Immutable Version That Ransomware Can't Delete

Files are showing up with the .locked extension, there's a README.txt on the desktop, and the first thing your mind does is: "I'll just restore from backup." That's the moment you realize the backup drive was also mounted to the same server and got encrypted. This article is about that moment; not about why backups matter.

Three things need to be right at the same time for a backup to be secure: a copy outside network reach, a copy that can't be modified even with root access during the retention window, and a recovery that has actually been executed in the past 90 days. If any one of these three is missing, the rest is decoration.

Why a backup on the same network isn't a backup

Today's ransomware looks for the backup destination first, before encrypting anything. Their scripts read mount points, credentials stored in ~/.aws/credentials, the restic config, and passwords inside .env files. If the backup server is reachable with the same SSH key or the same S3 token, the attacker does exactly what your script does: delete the previous version, encrypt the new one.

The sign always shows up late. The backup server log is full of successful jobs, the dashboard is green, and you only find out when you try to restore and see that the last seven days of snapshots are all zero bytes or encrypted. This is where people go wrong: they assume "successful job" means "healthy backup." A successful job means the file was transferred; it says nothing about its contents.

A real test, not a checklist

Once a month, on a separate machine, restore the latest snapshot and open a specific file. The numbers you should write down are two things: time to first healthy byte, and the number of files whose checksum doesn't match the source. If you haven't done this, you have an untested backup, and an untested backup isn't a backup.

restic -r s3:s3.example.com/bucket restore latest \
  --target /tmp/restore-test --include /var/www/html/index.php
sha256sum /tmp/restore-test/var/www/html/index.php

Immutable version: how to actually implement immutability

"Immutable" means that during the retention window, no one — including you with the admin key — can delete or overwrite the object. There are three practical approaches, and choosing between them comes down to budget and pain tolerance.

If I had to pick one, for production sites I'd choose Object Lock in Compliance mode, because it's automatic and doesn't depend on human memory. I wouldn't choose Governance mode; in that mode, anyone with the s3:BypassGovernanceRetention permission can unlock it, and that permission is exactly what an attacker is looking for.

Pull-only pattern with SSH

In this pattern, the backup server holds the key and connects to the production server. The production server has no key to write to the backup. If an attacker gets root on the production server, they can't see the backup destination.

# On the backup server, in /etc/ssh/sshd_config on the source:
Match User backupuser
    AllowTcpForwarding no
    PermitOpen none

# On the backup server, daily cron:
rsync -a --delete --link-dest=/backup/current \
  backupuser@prod:/var/www/html/ /backup/$(date +\%F)/

A subtle point: --link-dest saves space, but if the source overwrites a file, the old version stays untouched. That's exactly what saves you in a ransomware attack.

Database backup: where most failures happen

Copying /var/lib/mysql files from a running database produces a corrupt backup that looks healthy until the moment you restore it. For MySQL, you need to take a logical dump and verify it with a checksum.

mysqldump --single-transaction --quick --routines --triggers \
  --all-databases | gzip > /backup/db-$(date +\%F).sql.gz
gzip -t /backup/db-$(date +\%F).sql.gz && echo OK

The --single-transaction flag only makes sense on InnoDB. If you have MyISAM tables, this flag doesn't take a lock and the dump becomes inconsistent; there you need to use --lock-tables and shorten your maintenance window. That's the trade-off you rarely see in the documentation.

Backup encryption and key management

An unencrypted backup on cloud storage means that if the bucket leaks, everything leaks. But encryption has its own trap: where do you keep the key? If the key sits next to the backup, encryption is just a decorative layer. The key must be somewhere else, in a team password manager or KMS, and accessible to at least two people.

For restic and borg, read the password from a separate file and don't put that file in the backup:

export RESTIC_PASSWORD_FILE=/etc/restic/pass
restic -r s3:... backup /var/www --exclude-caches

If you lose the key, the backup is unrecoverable forever. Write this in the runbook, not in your head.

The 3-2-1 rule and the offline copy

Three copies, two different media, one copy off-site. The offline copy is what saves you in the worst-case scenario: when the attacker has gained access to both the production server and the cloud account. An NVMe or HDD that's connected once a week and then physically disconnected is stronger than any software setting, because a network attack on a disk that isn't connected is impossible.

Let me also mention the cost: this method is manual and in practice gets forgotten. My solution is to assign it to one person with a calendar reminder and log the result of each run in a file with a date. If no one logs it, that week there is no offline backup.

Monitoring: how to find out a backup is broken, before the incident

Monitor three signals. First, the age of the last successful snapshot; if it's past 26 hours, alert. Second, snapshot size; a sudden drop, say from 40 GB to 2 GB, means something is broken at the source. Third, the result of the monthly test restore.

A small script is enough:

restic snapshots --json | jq -r '.[-1].time'

If the output is older than 24 hours ago, send an alert in cron. To check the network path and permissions, you can also use DNS and network lookup tools to make sure the backup destination isn't reachable from the outside.

Permissions: backup with least privilege only

The account that takes the backup shouldn't be able to delete. In S3, write the policy so that only PutObject and ListBucket are allowed and DeleteObject is explicitly denied. On the server, the backup user shouldn't be a member of the sudo group and should preferably run in a chroot or separate container.

If you're working on your own infrastructure, separating the backup network from the production network is one of the most effective things you can do. For the network layer and defending against malicious traffic on the backup server, also take DDoS protection seriously; a backup server that's visible from the outside is a good target.

Alongside that, keep the audit log untouched. If one day you find that snapshots have been deleted, your first question is: "Who and from where?" Without logs, you have no answer. We've written about keeping it properly in audit logs and how to keep them untouched.

Recovery: the drill that actually works

Run the recovery on an environment that has no connection to production. The order of operations: first the database, then the files, then the configs and keys. If you reverse the order, the site comes up but has no data, and that's the worst case, because you think you succeeded.

1. Spin up a fresh machine with the same OS version and the same MySQL version.
2. Restore the dump and compare the row counts of the three main tables with production.
3. Restore the files and take the checksum of ten random files.
4. Bring the site up with a temporary DNS and test a real transaction.
5. Record the total time; that number is your real RTO, not the number you wrote in the document.

If the recovery time exceeds business tolerance, the problem isn't the backup, it's the architecture. There you need to think about a warm replica or faster snapshots.

Mistakes that get repeated over and over

• Keeping the backup password inside the same script that's on the production server.
• Retention shorter than the time to detect a breach; if you keep 7 days and the attack started 20 days ago, you have no healthy copy.
• Backing up from a path that is itself a symlink to somewhere else, and restoring the wrong files.
• Relying on provider snapshots as the only copy; a snapshot also disappears when the account is deleted.

If you work with WordPress, part of the attack surface is the application itself. Read separately about closing user-leak paths in securing the WordPress REST API and about general hardening in WordPress security from wp-config to plugins; a good backup on a site that's already wide open only pushes back the date of the incident.

And if your server's admin access is open to the outside, close that first before anything else: SSH security from changing the port to fail2ban and public keys.

Frequently asked questions

Is a cloud snapshot enough for a secure backup?

No, if it's the only copy. A snapshot is usually in the same account and the same control plane; an attacker who gains access to the account can delete it. Keep the snapshot as a speed layer, but have at least one separate immutable or offline copy.

What does immutable backup mean and how is it enabled?

It means that during the retention window, deleting or overwriting the object isn't possible. In S3 it's enabled with Object Lock and setting the retention mode. Choose Compliance mode, because in Governance mode anyone with the bypass permission can unlock it.

How often should I test recovery?

At least once a month for a single file and once every three months for a full recovery on a separate environment. The monthly test is quick and reveals early failures; the quarterly test determines the real RTO.

What happens to an encrypted backup if the key is lost?

It becomes unrecoverable forever; there's no way around it. So keep the key somewhere separate from the backup, accessible to at least two people, and with a printed copy in a safe.

What you can do today is this: restore a snapshot on a separate machine and write down the time. If it doesn't work, the rest of the discussion is pointless.