rsync for Server File Synchronization: A Practical Guide

An scp transfer got cut off midway and now you don't know which files made it and which didn't. Or you copied a 40-gigabyte folder with cp -r and it starts from scratch every time. The problem is you picked the wrong tool. rsync was built for exactly this moment: incremental, resumable transfers that can be repeated without re-copying things that are already in place.

In this guide I'll only cover things I've actually run into on real servers; from the difference a single trailing slash makes that changes the entire result, to flags that will wipe the destination server's files if you get them wrong.

What rsync actually transfers and what it doesn't

Before any transfer, rsync builds and compares a list of source and destination files. The default comparison criterion is a combination of file size and modification time (mtime), not a content hash. That means if a file is the same size and its mtime matches too, rsync assumes it's identical and skips it. This decision is correct for 99% of tasks because it multiplies speed several times over, but it has one important exception: when a file's content has changed but its size and timestamp have stayed the same. For example, a script that manually resets the timestamp. In that case you need --checksum so rsync computes an MD5 hash. The cost is that on several hundred thousand small files, the initial scan takes several minutes and disk I/O goes up.

For transfers over the network, rsync usually uses SSH as a tunnel. If you've set SSH on a non-default port, you have to say so explicitly:

rsync -avz -e "ssh -p 2222" /var/www/html/ deploy@203.0.113.10:/var/www/html/

The -z flag compresses data before sending. On text and code files, it pays off. On already-compressed files like jpg, zip, or mp4, it just burns CPU and gains you nothing. If you're taking backups over an internal network, drop -z.

The trailing slash difference; where the most mistakes happen

This is where people slip up: they see the trailing slash on the source path and think it's decorative. It isn't. These two commands produce completely different results:

rsync -av /var/www/html/  user@host:/backup/
rsync -av /var/www/html   user@host:/backup/

In the first line, the contents of the html folder are placed inside backup. In the second line, the html folder itself is created as a subfolder inside backup. If you get this wrong, after a few days of backups you'll notice the folder structure has become nested and the restore path is a mess. The practical sign is that you see a folder named html/html/html on the destination.

The rule to remember: a trailing slash means "put what's inside this folder." Without the slash it means "take this folder itself." For the destination path this difference doesn't exist; there a trailing slash has no effect.

Options that actually earn their keep

The full man rsync list has more than a hundred flags. These are the ones that come up in practice on web servers:

The combination I use for a daily backup of a WordPress site is this:

rsync -avz --partial --bwlimit=8000 \
  --exclude='wp-content/cache/' \
  --exclude='*.log' \
  /var/www/html/ backup@10.0.0.5:/srv/backups/site1/

Don't underestimate the bandwidth limit. If the backup runs over the same network link that carries user traffic, an unlimited rsync can push the site's TTFB up for minutes. A value of 8000 kilobytes per second on a 100-megabit link takes about 60 percent of capacity and leaves the rest for live traffic.

Resuming an interrupted transfer without starting from zero

If a 20-gigabyte file's transfer gets cut off midway, by default rsync deletes the incomplete destination file and starts over from the beginning next time. With --partial the half-finished file stays in place and the next run continues from there. For large files, --append-verify is also an option: it assumes the already-transferred portion is intact and only sends the tail. But if that first portion is corrupted, the final file stays corrupted. For sensitive backups I prefer --partial and, once it's done, I run rsync -avnc once to confirm with checksums that everything matches.

For long transfers, run rsync under screen or tmux. If SSH drops and rsync dies as its child, all those hours are wasted.

Safe deletion: how not to destroy yourself with --delete

--delete is what turns rsync from a simple copier into a real synchronization tool. Without it, files deleted from the source stay on the destination forever and your backup gradually becomes worthless. With it, one mistake in the source path means data gets wiped on the destination.

The safe approach is to always run with -n first and read the output:

rsync -avn --delete /var/www/html/ backup@10.0.0.5:/srv/backups/site1/

Lines starting with deleting are the files that are going to be removed. If their count is unexpectedly high, you've written the path wrong. Another protective layer is --max-delete=100: if rsync wants to delete more than this many files, it stops. This flag prevents a disaster like wiping the entire destination because of a missing slash.

The second point is that --delete interacts with --exclude. Excluded files on the destination are "protected" from rsync's point of view and won't be deleted, unless you use --delete-excluded. If you have files on the destination that should only live there (like a local cache), this behavior is exactly what you want.

When rsync doesn't give the right answer

Three common symptoms, each with a different cause:

• "Permission denied (publickey)" at the start: the SSH key isn't installed for the destination user. rsync uses the same SSH mechanism, so first ssh user@host needs to work without a password.
• "rsync: mkstemp ... failed: Permission denied" mid-transfer: the destination user doesn't have write permission on the target folder. rsync creates a temporary file for each file and then moves it; if the folder isn't writable, you see this error.
• The transfer sends all files every time: mtime probably isn't being preserved. If you didn't use -a, or the destination filesystem like FAT stores times with two-second precision, rsync always thinks the file has changed. Add -a.

If your destination server is shared hosting and you don't have full SSH access, rsync isn't an option and you'll need to look at other transfer methods. For servers where you have full control, Linux hosting with SSH access and rsync pre-installed is the logical starting point.

A daily backup script with cron

A short script that both logs and emails on error beats a long command you type by hand every time:

#!/bin/bash
LOG=/var/log/rsync-backup.log
rsync -az --partial --delete --max-delete=200 \
  --exclude='wp-content/cache/' \
  /var/www/html/ backup@10.0.0.5:/srv/backups/site1/ \
  >> "$LOG" 2>&1
echo "$(date '+%F %T') exit=$?" >> "$LOG"

Don't remove the last line. Without recording the exit code, if rsync fails and cron just stays silent, you'll find out weeks later that your backups have been incomplete. An exit code of zero means success; any other number means you need to open the log.

If you're running this on the same WordPress server, watch the timing. If you schedule the backup cron to run exactly when wp-cron is also running, both put pressure on disk and CPU. I've written about this conflict and its fix in WordPress wp-cron and replacing it with real cron.

Frequently asked questions

What's the difference between rsync and scp, and which should I pick?

scp transfers the whole file from the start every time and can't resume an interrupted transfer or remove extra files on the destination. rsync sends only the differences and, with --delete, can make the destination an exact mirror of the source. For a one-off transfer of a small file, scp is simpler; for anything that's going to be repeated, rsync is the right choice.

Why does rsync re-transfer all the files every time?

In most cases because file modification times aren't being preserved. You forgot the -a flag, or the destination filesystem is a type that stores mtime with low precision. Run with rsync -avni and see which files are in the transfer list; if they're all there, the problem is time, not content.

Can rsync encrypt files on the destination server?

rsync itself doesn't encrypt content. The transfer over SSH is encrypted, but the file is stored on the destination disk in its original form. If you want encrypted backups, you need to encrypt the files after transfer with a tool like gpg or openssl enc, or use a backup tool that does this internally.

How do I transfer only files newer than a specific date?

Use --newer: rsync -av --newer='2024-01-01' /src/ user@host:/dst/. This flag only sends files whose mtime is after the given date. For a time range, combine --newer and --older. Note that this filter operates on mtime, not on the file's creation date.

If you want to get the path structure and permissions right once before setting up automatic backups, ServerNet's documentation and knowledge base is a good starting point. And if you just want to know what the real transfer speed between your two servers is, the free tools were built for exactly that measurement.