Using rsync To Synchronize Local/Remote Systems

Rsync is a powerful tool that facilitates the transfer and synchronization of data between both local and remote systems.

 

Basic rsync usage

Let’s create two directories inside /tmp, called ”foo” and ”bar”, and generate a large amount of dummy files inside /tmp/foo
mkdir /tmp/foo /tmp/bar
for i in `seq 1 100`;do touch /tmp/foo/file$i;done


Now, we’ve got 100 files in /tmp/foo; /tmp/bar should still have none. We can use rsync to copy all of the files from /tmp/foo over to /tmp/bar:

rsync /tmp/foo/* /tmp/bar

Using basic file globbing, we can grab all files and copy them to another directory. What if there’s a directory inside /tmp/foo? It won’t be transferred. We will need to use the -r flag (–recursive) to traverse the directory, transferring every file inside:

rsync -r /tmp/foo/ /tmp/bar

This is a very simple example and does not even begin to touch the real power of the rsync command. There are flags to preserve permissions, ownerships, groups, symlinks, and so on. Because these flags are so often used, the -a (–archive) flag acts as an alias to incorporate them all, including -r.

Clear out /tmp/bar, create a symlink to one file in /tmp/foo, and use rsync to recursively copy all files:

find /tmp/bar -delete
ln -s /tmp/foo/file100 /tmp/foo/file101
rsync -r /tmp/foo/ /tmp/bar

We see that rsync has omitted the symlink we created. Clear out /tmp/bar again, and let’s try it again, this time using the -a flag:

find /tmp/bar -delete
rsync -a /tmp/foo/ /tmp/bar

Use chown to change the ownership of a file in /tmp/foo to another user, and copy over the files using -a to /tmp/bar. Run ls -l and note that the ownership moved with the file. Handy stuff!

NOTE: There is a difference between including a forward slash (/) at the end of the source path, and omitting it; the former will transfer all files INSIDE the specified directory, while the latter will transfer the directory itself with all files inside.

The -a Flag

We mentioned earlier that the -a (–archive) flag is an alias for a collection of other flags, -rltpgoD. Broken down, each flag does the following:

r – Recursive

l – Transfer any symlinks encountered

t – Preserve time stamps

p – Preserve permissions

g – Preserve groups

o – Preserve ownership

D – Preserve block and character devices

You may want to add the following to your command for easier to read file sizes:

h – Human-readable format of file sizes

Everyone Loves Feedback

The -v (–verbose) flag will give you more output about the state of the transfer, including a summary at the end, which will look something like this:

$ rsync -av foo/ bar
building file list ... done
sent 1040 bytes received 20 bytes 2120.00 bytes/sec
total size is 7 speedup is 0.01

If you want more statistics, run rsync with the –stats flag. This will give you a detailed list of the total number of files, files transferred, benchmarks, and even an averaged transfer speed. On the other side, -q (–quiet) will suppress all output, which can be used for scripts when feedback is not required.

Remote Transfers Made Simple

The true power of rsync is in its ability to perform not only local transfers but remote transfers as well. If you’ve used scp before, the syntax for remote transfers is quite similar:

rsync [flags] [local path] [user]@[remote server]:[remote path]

As an example, an rsync using this syntax would look like the following:

rsync -avh /tmp/foo/ root@host2:/tmp/bar

Note the : (colon) between the remote server and the remote path; this is required.

More Options

rsync comes with a large list of available options, far too many to go over in a single article. The last flags we will cover are the –exclude, –exclude-from, –update and –delete flags

–exclude

Exclude files based on a pattern. Rsync doesn’t support regex yet, so only standard file matching and globbing work

–exclude-from

Exclude files listed in a line-separated file

–update

Update files at the destination ONLY if the source copy has been modified more recently

–delete

Delete files on the destination ONLY if the source copy no longer exists.

Alternate SSH Ports

If you have changed the SSH port on your server you will need to tell rsync to use the new port number.

Example with normal SSH port:
rsync -azh /local/path/file user@host.com:/remote/path/file

Example with alternate SSH port (22334):
rsync -azh /local/path/file -e 'ssh -p 22334' user@host.com:/remote/path/file

Password-less Remote Transfers

Remote-to-local or local-to-remote file transfers can be made easier with the use of SSH keys. With SSH keys set up on both remote and local servers, synchronization can be scripted effortlessly and without human intervention (not having to enter a password every time). In another article, we discuss how to setup SSH keys.