Transferring files with scp and rsync

Why is file transfer with scp and rsync still essential?

When working with Linux servers, sooner or later you will reach a point where you need to move files between different machines. You might want to transfer a database backup from the production server to a secure destination, or send application logs to another server for analysis. In such situations, graphical tools like FTP are not only insufficiently secure, but are also very slow and inefficient for large and frequent transfers.

This is where scp and rsync come in. These two command-line tools are the real standard for file transfer in the Linux world. scp is great for simple and quick transfers, while rsync, with its incremental synchronization capability and the ability to resume interrupted transfers, is the top choice for backups and transferring large amounts of data. In this article, we will examine both tools with practical examples and cover important points that are rarely mentioned in official documentation.

Getting to know scp; simple and secure transfer

The scp (Secure Copy) command uses the SSH protocol for file transfer. This means all data is encrypted, and your authentication is done via an SSH key or password. Its basic syntax is as follows:

scp [options] source destination

For example, if you want to transfer the file backup.sql from your local server to the /var/backups/ directory on the destination server with the address 203.0.113.10:

scp backup.sql root@203.0.113.10:/var/backups/

In this command, root is the username and 203.0.113.10 is the server's IP address. If your SSH port is non-standard (for example, 2222), use the -P option:

scp -P 2222 backup.sql root@203.0.113.10:/var/backups/

To transfer an entire directory recursively, add the -r option:

scp -r /var/www/html/ root@203.0.113.10:/var/www/

Limitations of scp you should know

scp is a simple tool, but it has three important limitations that become annoying in real-world projects:

• No support for incremental transfer: If you have already transferred a file and only a few megabytes have changed, scp will resend the entire file.
• Interrupted transfers: If the network connection drops midway, you have to start the entire process from scratch.
• Incomplete preservation of file attributes: In some cases, file ownership and modification times are not preserved correctly.

These limitations are exactly where rsync comes in.

rsync; smart synchronization and resuming interrupted transfers

rsync is a file synchronization tool with a smart algorithm for detecting differences. Instead of sending the entire file, it only transfers the changed parts. This feature is a huge advantage for transferring large files (such as database files or videos).

The basic syntax of rsync is slightly different from scp:

rsync [options] source destination

A simple example of transferring a file:

rsync -avz backup.sql root@203.0.113.10:/var/backups/

The -a (archive) option enables archive mode, which includes preserving ownership, modification times, symbolic links, and recursive transfer. -v (verbose) displays operation details, and -z enables data compression during transfer.

Incremental synchronization; saving time and bandwidth

Suppose you have a directory with 50 GB of data and you need to sync it with the destination server every night. With scp, 50 GB would be sent each time. But with rsync, if only 200 MB has changed, only that 200 MB is transferred. This is done with the following command:

rsync -avz --delete /var/www/html/ root@203.0.113.10:/var/www/html/

The --delete option removes files from the destination that have been deleted from the source. This option is essential for complete synchronization, but if you only want to add or update files, do not use it.

Resuming interrupted transfers with rsync

One of the most important features of rsync is the ability to resume a transfer from where it stopped. If your connection drops during the transfer of a 10 GB file, simply re-running the same command will resume from where it left off. To do this, use the --partial option:

rsync -avz --partial --progress bigfile.iso root@203.0.113.10:/var/tmp/

The --partial option allows rsync to keep the incomplete file on the destination and resume from that point on the next run. The --progress option also displays the progress percentage.

Remote-to-remote transfer and bandwidth limiting

rsync can transfer files between two remote servers without the data passing through your machine:

rsync -avz root@192.0.2.10:/var/www/ root@198.51.100.20:/var/www/

If your bandwidth is limited, you can set a maximum transfer speed with the --bwlimit option. For example, limiting to 5 megabytes per second:

rsync -avz --bwlimit=5120 /data/ root@203.0.113.10:/data/

The value 5120 is in kilobytes per second (i.e., 5 megabytes). This prevents saturating your bandwidth and disrupting other services.

Practical comparison of scp and rsync; which one should you choose?

The choice between these two tools depends on your needs. The table below is a quick guide:

• One-time transfer of a few small files: scp is simpler and faster.
• Periodic backups and synchronization: rsync with incremental transfer is the smart choice.
• Transferring large files over an unstable connection: rsync with --partial is the only logical option.
• Precise preservation of file permissions and ownership: rsync with the -a option is fully superior.
• Quick transfer without needing special settings: scp is sufficient.

Common errors and how to fix them

When working with these tools, there are a few common errors that you should be aware of in advance.

Permission denied error in scp and rsync

This error usually occurs for two reasons: either the password is wrong, or the destination user does not have write access to the destination directory. To fix the second issue, first log into the server via SSH and check the directory permissions:

ls -ld /var/backups/
chown root:root /var/backups/
chmod 755 /var/backups/

If you are using a non-root user, make sure the user has access to the destination directory.

Problem preserving file ownership with rsync

If you see a chown failed error when using -a, it means your user does not have permission to change file ownership on the destination. This issue often occurs when using a non-root user. The solution is to either use the root user or add the --no-owner option to the command so that file ownership is not copied from the source:

rsync -avz --no-owner /data/ user@203.0.113.10:/data/

Frequent transfer interruptions with rsync

If your connection is unstable, try the following combination of options:

rsync -avz --partial --progress --timeout=60 /data/ root@203.0.113.10:/data/

The --timeout=60 option causes the connection to close and rsync to return an error if no data is exchanged for 60 seconds. Then you can run the command in a loop until the transfer completes:

until rsync -avz --partial --progress /data/ root@203.0.113.10:/data/; do echo "Retrying..."; sleep 5; done

This simple script repeats the command until it succeeds completely.

Important security tips for file transfer

Although both tools use SSH, there are a few security points you should not forget:

• Use SSH keys instead of passwords. This is both more secure and enables automation.
• Never compress sensitive files with the -z option if you are concerned about side-channel attacks. Compression can leak information about file contents.
• For highly sensitive file transfers, use the -e "ssh -c aes256-gcm@openssh.com" option to apply a stronger encryption algorithm.
• Limit user access. If you only need file transfer, you can use rsync in daemon mode with restricted permissions, but for most cases, regular SSH is sufficient.

Summary and final recommendation

File transfer with scp and rsync are two fundamental skills for any server administrator. scp is great for quick and simple tasks, but for regular backups, synchronizing large directories, and transferring files over unstable connections, rsync is unrivaled. My recommendation is to keep both tools in your toolbox and choose one based on the situation.

If you are looking for infrastructure where these tools work well, ServerNet's cloud and dedicated server services with adequate bandwidth and network stability are a suitable option for performing these transfers. But regardless of your infrastructure choice, mastering these two tools will significantly improve the quality of your work.

Finally, I suggest practicing these commands in a test environment. Create a multi-gigabyte file, transfer it with scp, and then synchronize it with rsync. The difference in speed and behavior between these two tools in real conditions will be the best lesson.