File Synchronization 101: The Basics
I. Using rsync to Transfer and Synchronize Local and Remote Systems II. How to Setup SSH Keys for Use with File Synchronization
Basic rsync usageLet’s create two directories inside /tmp, called ”foo” and ”bar”, and generate a large amount of dummy files inside /tmp/foo
mkdir /tmp/foo /tmp/bar for i in `seq 1 100`;do touch /tmp/foo/file$i;doneNow, we’ve got 100 files in /tmp/foo; /tmp/bar should still have none. We can use rsync to copy all of the files from /tmp/foo over to /tmp/bar:
rsync /tmp/foo/* /tmp/barUsing basic file globbing, we can grab all files and copy them to another directory. What if there’s a directory inside /tmp/foo? It won’t be transferred. We will need to use the -r flag (–recursive) to traverse the directory, transferring every file inside:
rsync -r /tmp/foo/ /tmp/barThis is a very simple example and does not even begin to touch the real power of the rsync command. There are flags to preserve permissions, ownerships, groups, symlinks, and so on. Because these flags are so often used, the -a (–archive) flag acts as an alias to incorporate them all, including -r. Clear out /tmp/bar, create a symlink to one file in /tmp/foo, and use rsync to recursively copy all files:
find /tmp/bar -delete ln -s /tmp/foo/file100 /tmp/foo/file101 rsync -r /tmp/foo/ /tmp/barWe see that rsync has omitted the symlink we created. Clear out /tmp/bar again, and let’s try it again, this time using the -a flag:
find /tmp/bar -delete rsync -a /tmp/foo/ /tmp/barUse chown to change the ownership of a file in /tmp/foo to another user, and copy over the files using -a to /tmp/bar. Run ls -l and note that the ownership moved with the file. Handy stuff! NOTE: There is a difference between including a forward slash (/) at the end of the source path, and omitting it; the former will transfer all files INSIDE the specified directory, while the latter will transfer the directory itself with all files inside.
The -a FlagWe mentioned earlier that the -a (–archive) flag is an alias for a collection of other flags, -rltpgoD. Broken down, each flag does the following: r – Recursive l – Transfer any symlinks encountered t – Preserve time stamps p – Preserve permissions g – Preserve groups o – Preserve ownership D – Preserve block and character devices You may want to add the following to your command for easier to read file sizes: h – Human-readable format of file sizes
Everyone Loves FeedbackThe -v (–verbose) flag will give you more output about the state of the transfer, including a summary at the end, which will look something like this:
$ rsync -av foo/ bar building file list ... done sent 1040 bytes received 20 bytes 2120.00 bytes/sec total size is 7 speedup is 0.01If you want more statistics, run rsync with the –stats flag. This will give you a detailed list of the total number of files, files transferred, benchmarks, and even an averaged transfer speed. On the other side, -q (–quiet) will suppress all output, which can be used for scripts when feedback is not required.
Remote Transfers Made SimpleThe true power of rsync is in its ability to perform not only local transfers but remote transfers as well. If you’ve used scp before, the syntax for remote transfers is quite similar:
rsync [flags] [local path] [user]@[remote server]:[remote path]As an example, an rsync using this syntax would look like the following:
rsync -avh /tmp/foo/ root@host2:/tmp/barNote the : (colon) between the remote server and the remote path; this is required.
More Optionsrsync comes with a large list of available options, far too many to go over in a single article. The last flags we will cover are the –exclude, –exclude-from, –update and –delete flags –exclude
Exclude files based on a pattern. Rsync doesn’t support regex yet, so only standard file matching and globbing work–exclude-from
Exclude files listed in a line-separated file–update
Update files at the destination ONLY if the source copy has been modified more recently–delete
Delete files on the destination ONLY if the source copy no longer exists.
Alternate SSH PortsIf you have changed the SSH port on your server you will need to tell rsync to use the new port number. Example with normal SSH port:
rsync -azh /local/path/file email@example.com:/remote/path/fileExample with alternate SSH port (22334):
rsync -azh /local/path/file -e 'ssh -p 22334' firstname.lastname@example.org:/remote/path/file