Using rsync to copy files

Last updated: 29 April 2021

The rsync utility is used to copy files. It can copy files locally as well as from one host to another. You can also use rsync to make backups.

Common options

rsync is almost always run with one or more options. The most common options are -a and -v. We will briefly look at these options before we get to some examples.

Archiving

The -a option is a combination of a bunch of other options:

-r  Recurse into directories  
-l  Copy symlinks as symlinks
-p  Preserve file permissions
-t  Preserve modification times
-g  Preserve group
-o  Preserve owner
-D  Preserve device and special files

This is a convenient shortcut but you don’t necessary want to use all these options. For instance, if you are syncing files to a remote system you may not want to preserve the group and owner data, as the same user and group might not exist on the remote system.

Verbose output

Another commonly used option is -v (--verbose). As the name suggest, this option makes rsync produce lots of output – it prints all the files that are being synced and a summary of the amount of data transferred. You typically want to use this option, as it tells you what rsync is doing.

Copying files

To demonstrate how to use rsync we will copy an entire WordPress site to a different directory. This is something you may want to do after you have installed WordPress via Softaculous. By default Softaculous installs WordPress in a subdirectory with the name wp. Often that is useful, as the install will be separate from any existing files in the public_html directory. However, it does mean that you have to move the files if you prefer to have the install in the public_html directory.

You can sync all the files from the public_html/wp directory to the public_html directory in seconds:

$ rsync -av /home/example/public_html/wp/ /home/example/public_html/
sending incremental file list
./
.htaccess
index.php
...
sent 44,901,625 bytes  received 36,723 bytes  29,958,898.67 bytes/sec
total size is 44,762,896  speedup is 1.00

As we added the -v option the output tells us which files were copied and how much data was transferred. Copying about 45MB took about 1.5 seconds.

rsync vs mv

It is worth mentioning that the data has been copied rather than moved. In other words, the public_html/wp directory still contains all the original files. If you just want to move files it makes more sense to use the mv utility instead. There are two things to be aware of though. The first is that mv doesn’t move dotfiles by default:

$ mv /home/example/public_html/wp/ /home/example/public_html/

$ ls -1A /home/example/public_html/wp/
.htaccess

In the above example we moved all the files, but the .htaccess file is still sitting in the wp subdirectory. In Bash, you can set the dotglob option to get mv to also move hidden files:

$ shopt -s dotglob nullglob
$ mv /home/example/public_html/wp/* /home/example/public_html/
$ ls -lA wp/
$

A second thing to note is that mv doesn’t preserve the SELinux context of files. If SELinux is enforced on your server then it is easier to use rsync, which does preserve the context.

Beware of the trailing slash

It is very important to be aware of the function of the trailing slash in the source. To illustrate the point, let’s create two directories (source and destination) and create a file named testfile in the source directory:

$ mkdir source destination
$ touch source/testfile

If you want to copy all files in the source directory to the destination directory then the source should include the trailing slash, like so:

$ rsync -av source/ destination/

The trailing slash instructs rsync to copy all files (and directories) in the source directory. Can you guess what happens if you omit the trailing slash?

$ rsync -a source destination
$ tree destination/
destination/
└── source
    └── testfile

1 directory, 1 file

Instead of copying the test file it copied the entire source directory to the destination directory. It is easy to see how you can very quickly do a lot of damage with rsync if you get the trailing slash wrong.

Dry run

If you are not quite sure about an rsync command then you can use the -n (--dry-run) option. rsync will go through the motions but not actually copy any files. You can then check the output to see if your command works as it should:

$ rsync -avn source destination
sending incremental file list
source/
source/testfile

sent 108 bytes  received 23 bytes  262.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)

The main thing to note in the above output is that rsync is copying the source/ directory, which is not what we wanted.

Copying files to a remote server

To copy files to a remote server you use pretty much the same syntax. In the below example we are syncing files from the /srv/www/public_html directory on our local computer to the example.net server:

$ rsync -avz /srv/www/public_html/ example@example.net:/home/example/transfers/
sending incremental file list
./
.htaccess
index.php
license.txt
...
sent 12,344,200 bytes  received 36,504 bytes  575,846.70 bytes/sec
total size is 44,750,608  speedup is 3.61

We added one new option:  -z (--compress) compresses file data during the transfer. This is particularly useful when you sync files to and from a remote server, as this will always be slower than syncing files locally.

To connect to the remote server you use the username and hostname, just like you do when you connect to a remote server via SSH. You can specify the directory to which you want to sync the files by adding a colon (:) followed by the path to the destination directory. In this case the files were synced to the /home/example/transfers directory on the remote server.

Copying files from a remote server

The opposite, syncing files from a remote server to your local computer, is of course also possible. You can simply reverse the source and destination, as shown in the below example:

$ rsync -avz example@example.net:/home/example/transfers/ /srv/www/public_html/

Exclude directories

The --exclude option lets you exclude a directory. This can be useful if you are developing websites locally and have, say, a directory for backups in your document root. Using --exclude you can copy all the files, except for the backups directory:

$ rsync -avn --exclude 'backup' /srv/www/public_html/ example@example.net:/var/www/html/ | grep -c ^backup
0

Notice that we used the verbose option (-v) and did a dry run (-n). The output lists all the files that would normally be copied. As that list is very long we piped the output to grep -c ^backup, which gives us a count of the number of lines starting with the string backup. There were zero lines, which means that our command does indeed exclude the backup directory.

You can exclude files in exactly the same way. And if you need to exclude more than one file or directory you can simply use multiple --exclude options.

Using wild cards

As a bonus tip, you can also use wild cards (“globs”). For instance, you may have two directories that are not part of the website itself and which should never be copied. Let’s say you got a __backups directory for backups and a __resources directory for assets. As both directories start with two underscores you can exclude both at the same time:

$ rsync -avn --exclude '__*' /srv/www/public_html/ example@example.net:/var/www/html/ | grep -c ^__
0

The asterisk in --exclude '__*' matches both __backups and __resources. In the same way you can exclude specific file types, such as .tar.gz or .zip files:

$ rsync -avn /srv/www/public_html/ example@example.net:/var/www/html/ \
| grep -c ".*tar.gz$"
21

$ rsync -avn --exclude '*.tar.gz' /srv/www/public_html/ example@example.net:/var/www/html/ \
| grep -c ".*tar.gz$"
0

Deleting files

The --delete option deletes files from the destination that are not in the source. In other words, the file foo.txt is deleted if it exists in the destination but not in the source.

This option is typically used for non-incremental backups. For instance, here we are copying the directory /src/www/ to an external backup drive:

$ rsync -av --delete /srv/www/ /run/media/example/ext_drive/backup/

After the sync the two directories are identical. All files are copied from /src/www/ to the backup drive and any files that exist on the backup drive but not in /srv/www/ directory are deleted.

As an aside, strictly speaking it is not true that all the files in the /src/www/ directory are copied. Any files that already exist on the backup drive are skipped. If you want you can force rsync to overwrite files that already exist at the destination using the -I (--ignore-times) option.

More information

rsync has an impressive amount of options. This article aimed to give you an idea of how you can use the tool. To learn more, it is worth reading the man page and search for more thorough tutorials, such as the one written by Computer Hope.

Also, the best way to learn about utilities such as rsync is to use them. To demonstrate the importance of the trailing slash in the source directory we created two test directories and a test file. Rather than just reading about the utility you can create simple test cases like that yourself.