Rsync is a powerful tool that facilitates the transfer and synchronization of data between both local and remote systems.
II. How to Setup SSH Keys for Use with File Synchronization
Basic rsync usage
Let’s create two directories inside /tmp, called ”foo” and ”bar”, and generate a large amount of dummy files inside /tmp/foo
mkdir /tmp/foo /tmp/bar
for i in `seq 1 100`;do touch /tmp/foo/file$i;done
Now, we’ve got 100 files in /tmp/foo; /tmp/bar should still have none. We can use rsync to copy all of the files from /tmp/foo over to /tmp/bar:
rsync /tmp/foo/* /tmp/bar
Using basic file globbing, we can grab all files and copy them to another directory. What if there’s a directory inside /tmp/foo? It won’t be transferred. We will need to use the -r flag (–recursive) to traverse the directory, transferring every file inside:
rsync -r /tmp/foo/ /tmp/bar
This is a very simple example and does not even begin to touch the real power of the rsync command. There are flags to preserve permissions, ownerships, groups, symlinks, and so on. Because these flags are so often used, the -a (–archive) flag acts as an alias to incorporate them all, including -r.
Clear out /tmp/bar, create a symlink to one file in /tmp/foo, and use rsync to recursively copy all files:
find /tmp/bar -delete
ln -s /tmp/foo/file100 /tmp/foo/file101
rsync -r /tmp/foo/ /tmp/bar
We see that rsync has omitted the symlink we created. Clear out /tmp/bar again, and let’s try it again, this time using the -a flag:
find /tmp/bar -delete
rsync -a /tmp/foo/ /tmp/bar
Use chown to change the ownership of a file in /tmp/foo to another user, and copy over the files using -a to /tmp/bar. Run ls -l and note that the ownership moved with the file. Handy stuff!
NOTE: There is a difference between including a forward slash (/) at the end of the source path, and omitting it; the former will transfer all files INSIDE the specified directory, while the latter will transfer the directory itself with all files inside.
The -a Flag
We mentioned earlier that the -a (–archive) flag is an alias for a collection of other flags, -rltpgoD. Broken down, each flag does the following:
r – Recursive
l – Transfer any symlinks encountered
t – Preserve time stamps
p – Preserve permissions
g – Preserve groups
o – Preserve ownership
D – Preserve block and character devices
You may want to add the following to your command for easier to read file sizes:
h – Human-readable format of file sizes
Everyone Loves Feedback
The -v (–verbose) flag will give you more output about the state of the transfer, including a summary at the end, which will look something like this:
$ rsync -av foo/ bar
building file list ... done
sent 1040 bytes received 20 bytes 2120.00 bytes/sec
total size is 7 speedup is 0.01
If you want more statistics, run rsync with the –stats flag. This will give you a detailed list of the total number of files, files transferred, benchmarks, and even an averaged transfer speed. On the other side, -q (–quiet) will suppress all output, which can be used for scripts when feedback is not required.
Remote Transfers Made Simple
The true power of rsync is in its ability to perform not only local transfers but remote transfers as well. If you’ve used scp before, the syntax for remote transfers is quite similar:
rsync [flags] [local path] [user]@[remote server]:[remote path]
As an example, an rsync using this syntax would look like the following:
rsync -avh /tmp/foo/ root@host2:/tmp/bar
Note the : (colon) between the remote server and the remote path; this is required.
rsync comes with a large list of available options, far too many to go over in a single article. The last flags we will cover are the –exclude, –exclude-from, –update and –delete flags
Exclude files based on a pattern. Rsync doesn’t support regex yet, so only standard file matching and globbing work
Exclude files listed in a line-separated file
Update files at the destination ONLY if the source copy has been modified more recently
Delete files on the destination ONLY if the source copy no longer exists.
Alternate SSH Ports
If you have changed the SSH port on your server you will need to tell rsync to use the new port number.
Example with normal SSH port:
rsync -azh /local/path/file email@example.com:/remote/path/file
Example with alternate SSH port (22334):
rsync -azh /local/path/file -e 'ssh -p 22334' firstname.lastname@example.org:/remote/path/file
Password-less Remote Transfers
Remote-to-local or local-to-remote file transfers can be made easier with the use of SSH keys. With SSH keys set up on both remote and local servers, synchronization can be scripted effortlessly and without human intervention (not having to enter a password every time). In another article, we discuss how to setup SSH keys.
- What is CentOS? Everything You Need to Know
- Check Apache Status with systemctl status and apachectl status Commands
- How to Find the Server Name Indication (SNI) Supporting Details
- How to Install Pip on CentOS 7
- Get Kernel Version for Linux: A Guide
- What Is a Time Series Database? How It Works & Use Cases
Our Sales and Support teams are available 24 hours by phone or e-mail to assist.