rsync (remote synchronization) is a command-line tool for transferring and synchronizing files and directories locally and remotely in Linux. In other words, rsync is used for transferring files and directories, creating backups, mirroring files, and transferring large files between two locations.
rsync identifies the differences between the source and destination files and directories using its delta-transfer algorithm, and only transfers the difference between both files and directories. This makes rsync efficient and further differentiates it from other file synchronization and transfer tools.
In addition, rsync supports partial file transfers, which allows it to resume interrupted file transfers rather than starting over from the beginning. This is particularly useful when sending large files.
In this article, we'll explore the most common and useful rsync commands and discuss examples of how to use them in different scenarios.
Installing rsync
Most Linux distributions come with rsync preinstalled. However, you can check whether or not rsync is present on your system by running the following command:
rsync --version
Output:
rsync version 3.2.7 protocol version 31
This output indicates the version of rsync installed.
Nonetheless, if rsync isn't installed on your system, you can install it using the following commands, depending on your Linux distribution.
For Ubuntu, Debian, and Mint:
sudo apt install rsync
For Fedora:
sudo dnf install rsync
For CentOs:
sudo yum install rsync
For Arch Linux:
sudo pacman -S rsync
Basic rsync Syntax
rsync commands usually take the following form:
rsync [options...][<source>][destination...]
- options: These are optional arguments that modify the behavior of the rsync command. They usually denote the action to be taken when rsync is executed. Some commonly used options include -a for archive mode, -z for file compression, and -v for verbose output.
- source: This indicates the path to the source file or directory, which may be local or remote.
- destination: This indicates the path to the destination directory, which may be local or remote. If one doesn't exist, rsync will create it.
Examples
1. Copying/Syncing Data Locally
The primary use of rsync is for transferring files to a different location. To transfer files locally, we can execute the following command:
rsync example.zip ../dest-dir/test/
In this example, we simply tell rsync to copy the source file "example.zip" to the destination directory "../dest-dir/test" without specifying additional options. However, executing rsync without additional options doesn't utilize its full capabilities such as copying subdirectories and file metadata.
To copy the content of a directory and its metadata, we can execute the command below:
rsync -avh ./ ../dest-dir/test/
In the example above, we're copying the content of the current directory to the destination directory, "../dest-dir/test/". We have also included three options:
- -a (--archive), archive mode. This option ensures that subdirectories are copied recursively, and symlinks, file permissions, modification times, file ownership, and block devices are preserved accordingly. It is equivalent to combining the "-r", "-l", "-p", "-t", "-g", "-o", and "-D" options.
- -v (--verbose), verbose output. This option ensures that the files and directories being copied are displayed in the shell output.
- -h (--human readable). This option displays numbers such as file sizes in a human-readable format.
Output:
sending incremental file list ./ example.zip photoFive.png photoFour.png photoOne.png photoThree.png photoTwo.png backups/ backups/index.js backups/database/ sent 15.28M bytes received 183 bytes 30.56M bytes/sec total size is 15.27M speedup is 1.00
It is important to note that rsync behaves differently depending on whether a trailing slash, "/" is added or omitted from the source directory's path. When the trailing slash is added, like in the previous example, rsync transfers the content of that directory to the destination directory. Whereas, when the trailing slash is omitted, rsync creates a directory with the name of the source directory where the content is transferred into.
2. Copying/Syncing Data with a Remote Host
In the previous examples, we've only transferred data between local directories. However, rsync supports sending files to a remote directory and vice versa.
To transfer files to a remote destination, the remote machine should also have rsync installed. rsync uses SSH as the default remote shell. However, rsync also supports using a rsync daemon for remote communication via TCP.
The following command can be used to transfer data to a remote directory:
rsync -avh ./ [email protected]:/home/admin/dest-example/
As we can see from the example above, the difference between how data is sent locally and remotely is in how the destination directory is defined. The remote destination is made up of the following parts:
- User's name: This refers to the name of the user on the remote server that owns the home directory to which data is transferred. In this example, the user's name is admin.
- Remote host: This is the name or IP address of the server being accessed.
- Destination directory: This comes after the ":" separator and describes the path where files are to be stored on the remote host.
After executing the command, you may be prompted to enter a password for the user if SSH passwordless login isn't set up.
Output:
sending incremental file list ./ example.zip photoFive.png photoFour.png photoOne.png photoThree.png photoTwo.png backups/ backups/index.js backups/database/ sent 15.28M bytes received 183 bytes 4.37M bytes/sec total size is 15.27M speedup is 1.00
Alternatively, you can copy files from a remote directory to a local directory, as shown in the example below:
rsync -avh [email protected]:/home/admin/source-example/ ./dest-dir/
The command above copies files and subdirectories from the "/home/admin/source-example" remote source directory to the local "./dest-dir" destination directory.
Output:
receiving incremental file list ./ example.zip photoFive.png photoFour.png photoOne.png photoThree.png photoTwo.png backups/ backups/index.js backups/database/ sent 172 bytes received 15.28M bytes 4.37M bytes/sec total size is 15.27M speedup is 1.00
3. Including and Excluding Files/Directories
rsync provides options that give us precise control over files being copied, which is particularly useful when we want to exclude a large number of files and directories, or only include a specific set of files.
To copy a certain set of files or directories only, we can use the "--include" and "--exclude" options with rsync, followed by the file, directory, or pattern of files to be included.
For example, to exclude "example.zip" when transferring files to the destination directory, we can use the following command:
rsync -avh --exclude "example.zip" ./ ../dest-dir/
In this example, the "--exclude" option tells rsync to ignore "example.zip" when transferring files to the "../dest-dir" directory.
Output:
sending incremental file list ./ photoFive.png photoFour.png photoOne.png photoThree.png photoTwo.png backups/ backups/index.js backups/database/ sent 7.66M bytes received 145 bytes 15.32M bytes/sec total size is 7.66M speedup is 1.00
In addition, we can use wildcards(*) to exclude multiple files or directories that match a certain pattern. For example, to exclude all ".png" files in the source directory, we can use the command below:
rsync -avh --exclude "*.png" ./ ../dest-dir/
Output:
sending incremental file list ./ example.zip backups/ backups/index.js backups/database/ sent 7.62M bytes received 69 bytes 15.24M bytes/sec total size is 7.62M speedup is 1.00
Although rsync supports the use of multiple "--exclude" options when trying to omit multiple files or directories, this is not practical if you want to exclude a large number of files. Fortunately, rsync also provides the "--exclude-from" option, which allows us to specify the files and directories to be excluded in a single place.
The "--exclude-from" option takes a file argument, which contains the list of files and directories to be ignored. For instance, given a file called "exclusion.txt" with the following entries:
*.zip /backups exclusion.txt
We can use "--exclude-from" to tell rsync to ignore the entries in the file during data transfer as shown in the example below:
rsync -avh --exclude-from "exclusion.txt" ./ ../dest-dir/
Output:
sending incremental file list ./ photoFive.png photoFour.png photoOne.png photoThree.png photoTwo.png sent 7.66M bytes received 114 bytes 15.31M bytes/sec total size is 7.66M speedup is 1.00
However, it's important to note that each entry to be excluded should be on a single line and there shouldn't be any whitespaces to enable rsync to successfully read the file's content.
Alternatively, we can use the "--include" option together with "--exclude" to only transfer a certain set of files. For example, to copy only ".png" files, we can use the command below:
rsync -avh --include "*.png" --exclude "*" ./ ../dest-dir/
In this example, the "--include" option is passed a "*.png" argument, which tells rsync to only include files with the ".png" extension, while the wildcard argument "*" passed to the "--exclude" option tells rsync to ignore every other file.
Output:
sending incremental file list ./ photoFive.png photoFour.png photoOne.png photoThree.png photoTwo.png sent 7.66M bytes received 114 bytes 15.31M bytes/sec total size is 7.66M speedup is 1.00
Additionally, rsync provides an "--include-from" option, which is similar to "--exclude-from". The "--include-from" option accepts a file argument, which contains a list of files and directories to be included in the transfer.
4. Removing Source Files After Data Transfer
After successfully transferring files to the destination directory, we can tell rsync to delete the files from the source directory by adding the "--remove-source-files" option to the rsync command. This is useful in situations where you want to immediately free up space after completing large file transfers.
To remove transferred files from the source directory, we specify the "--remove-source-files" option, as shown in the example below:
rsync -avh --remove-source-files ./*.png ../dest-dir/
Output:
sending incremental file list photoFive.png photoFour.png photoOne.png photoThree.png photoTwo.png sent 7.66M bytes received 231 bytes 15.31M bytes/sec total size is 7.66M speedup is 1.00
However, this option should be used with caution to avoid data loss, as it permanently erases files from the source directory.
5. Showing Progress During Data Transfer
rsync provides the "--progress" or "-P" option for displaying progress information during data transfer, which is particularly useful when transferring large files or directories. It shows information about how much data has been sent, giving you an idea of how long the transfer will take.
To show transfer progress, add the "--progress" or "-P" option to the rsync command as shown in the examples below:
rsync -avhP ./ ../dest-dir/
rsync -avh --progress ./ ../dest-dir/
Output:
sending incremental file list
./
example.zip
7.62M 100% 241.18MB/s 0:00:00 (xfr#1, to-chk=8/10)
photoFive.png
753.88K 100% 20.54MB/s 0:00:00 (xfr#2, to-chk=7/10)
photoFour.png
1.25M 100% 29.74MB/s 0:00:00 (xfr#3, to-chk=6/10)
photoOne.png
1.49M 100% 32.95MB/s 0:00:00 (xfr#4, to-chk=5/10)
photoThree.png
1.25M 100% 25.31MB/s 0:00:00 (xfr#5, to-chk=4/10)
photoTwo.png
2.92M 100% 35.71MB/s 0:00:00 (xfr#6, to-chk=3/10)
backups/
backups/index.js
0 100% 0.00kB/s 0:00:00 (xfr#7, to-chk=1/10)
backups/database/
sent 15.28M bytes received 164 bytes 30.56M bytes/sec
total size is 15.27M speedup is 1.00
The output will look similar to the one above, which shows the following information:
- The names and sizes of the files or directories being transferred.
- The percentage of the transfer completed.
- The transfer rate of each file or directory being transferred, expressed in MB/s and kB/s.
- The estimated time left for each transfer to be completed.
- A message indicating the transfer number, the number of files to be checked (x), and the total number of files (y).
6. Deleting Data
To remove extra files or directories in the destination directory that are not present in the source directory, rsync provides the
"--delete"
option, which ensures both directories are synchronized.
Given a source directory with the following items:
backups photoFive.png photoFour.png photoThree.png
Then, having a destination directory with the following items:
backups photoFive.png photoOne.png photoTwo.png example.zip photoFour.png photoThree.png
The example below shows how to remove extra files in the destination directory with rsync using the "--delete" option.
rsync -avh --delete ./ ../dest-dir
After executing the command, we get the following output:
sending incremental file list deleting photoTwo.png deleting photoOne.png deleting example.zip ./ sent 215 bytes received 68 bytes 188.67 bytes/sec total size is 3.25M speedup is 11,479.72
The shell output above shows that the three files that were not present in the source directory have been deleted.
7. Compressing Data
rsync supports compression during data transfer, which is particularly useful for speeding up transfers, especially when sending large files over a slow network connection.
To enable compression during data transfer, we can use the "-z" or "--compress" options with rsync, as shown in the example below:
rsync -avhz ./ [email protected]:/home/admin/dest-dir/
In the command above, the "-z" option is added to enable data compression, while transferring data from the current directory to the remote destination directory.
Output:
sending incremental file list ./ example.zip photoFive.png photoFour.png photoOne.png photoThree.png photoTwo.png backups/ backups/index.js backups/database/ sent 13.99M bytes received 164 bytes 3.11M bytes/sec total size is 15.27M speedup is 1.09
Prior to version 3.2.0, rsync used zlib as the default compression algorithm. Newer versions of rsync support different compression algorithms, including lz4 and zstd, which offer better performance and compression ratio. rsync provides the "--compress-choice" or "--zc" options, which allows us to specify the compression algorithm used during compression.
The example below illustrates how to use the "lz4" algorithm with rsync for better data compression:
rsync -avh --compress-choice=lz4 ./ [email protected]:/home/admin/dest-dir/
Output:
sending incremental file list ./ example.zip photoFive.png photoFour.png photoOne.png photoThree.png photoTwo.png backups/ backups/index.js backups/database/ sent 555 bytes received 53.05K bytes 15.31K bytes/sec total size is 15.27M speedup is 284.96
Additionally, we can adjust the compression level used during compression by adding the "--compress-level" or "--zl" options. The compress level specified depends on the compression method used by rsync. For instance, zlib requires a value between 0 - 9, where 0 deactivates compression, and 9 represents the highest compression level.
Higher compression levels generally result in better data compression although data transfer may be slower and CPU and memory usage may increase. By default, zlib uses a compression level of 6, which provides a good balance between the compression rate and data transfer speed.
The example below sets a compression level of 8 while using the zlib compression algorithm:
rsync -avh --zc=zlib --compress-level=8 ./ [email protected]:/home/admin/dest-dir/
Output:
sending incremental file list ./ example.zip photoFive.png photoFour.png photoOne.png photoThree.png photoTwo.png backups/ backups/index.js backups/database/ sent 13.99M bytes received 164 bytes 4.00M bytes/sec total size is 15.27M speedup is 1.09
In addition, the "--skip-compress" option can be used to skip compression for a specific set of files. This is particularly useful when the data being transferred, such as media files, are already compressed, thus, reducing transfer time and CPU overhead.
The "--skip-compress" option takes a list of file extensions without the dot, where each extension is separated with a forward slash (/) as shown in the example below:
rsync -avhz --skip-compress=png/zip ./ [email protected]:/home/admin/dest-dir
In the command above, both ".png" and ".zip" files are excluded from compression using the "--skip-compress" option.
Output:
sending incremental file list ./ example.zip photoFive.png photoFour.png photoOne.png photoThree.png photoTwo.png backups/ backups/index.js backups/database/ sent 13.99M bytes received 164 bytes 3.11M bytes/sec total size is 15.27M speedup is 1.09
Note that rsync inherently has a list of file extensions for "--skip-compress" that are excluded by default during compression, which includes 3gp, 7z, apk, mp3, mp4, ogg, rar, zip, webm, webp, and so on. Specifying the "--skip-compress" option will override this list.
8. Performing a Dry Run
rsync supports performing a dry run, which is a way to test a command without actually making any changes. This produces a similar output to the actual command execution and is particularly useful for checking if a rsync command does what you intend to do.
To perform a dry run, we can use the "--dry-run" option with rsync, as shown in the example below:
rsync -avh --delete --dry-run ./ ../dest-dir/
In this example, rsync will execute a dry run of the "--delete" command, showing the files that could be deleted from the destination directory without actually deleting the files.
Output:
sending incremental file list deleting photoTwo.png deleting photoThree.png deleting photoOne.png deleting example.zip ./ sent 198 bytes received 86 bytes 568.00 bytes/sec total size is 2.00M speedup is 7,046.90 (DRY RUN)
9. Set File Size Limits
rsync provides the "--max-size" and "--min-size" options for putting a limit on the maximum and minimum sizes of files that can be transferred respectively. Both options take values expressed in units such as B (bytes), K (kilo), M (mega), G (giga), and P (peta).
For example, the command below sets the maximum size of files that will be transferred to 2 MB:
rsync -avh --max-size=2M ./ ../dest-dir/
Output:
sending incremental file list ./ photoFive.png photoFour.png photoOne.png photoThree.png backups/ backups/index.js backups/database/ sent 4.74M bytes received 126 bytes 9.47M bytes/sec total size is 15.27M speedup is 3.23
Alternatively, we can use the "--min-size" option to set the minimum size of files that will be transferred as shown below:
rsync -avh --min-size=2M ./ ../dest-dir/
Output:
sending incremental file list ./ example.zip photoTwo.png backups/ backups/database/ sent 10.54M bytes received 69 bytes 21.09M bytes/sec total size is 15.27M speedup is 1.45
10. Set a Bandwidth Limit
We can limit the bandwidth used by rsync during data transfer by adding the "--bwlimit" option. This option takes the maximum transfer rate expressed in units such as K (kilo), M (mega), G (giga), and P (peta) and defaults to K (kilo) if no unit is specified. This is particularly useful in situations where you want to limit the impact of data transfer on other network users.
The following example illustrates how to set a bandwidth limit using the "--bwlimit" option with rsync:
rsync -avhP --bwlimit=1M ./ ../dest-dir/
In the command above, the bandwidth limit is set to 1MB/s.
Output:
sending incremental file list
./
example.zip
7.62M 100% 1007.75kB/s 0:00:07 (xfr#1, to-chk=8/10)
photoFive.png
753.88K 100% 457.27kB/s 0:00:01 (xfr#2, to-chk=7/10)
photoFour.png
1.25M 100% 751.98kB/s 0:00:01 (xfr#3, to-chk=6/10)
photoOne.png
1.49M 100% 772.59kB/s 0:00:01 (xfr#4, to-chk=5/10)
photoThree.png
1.25M 100% 570.32kB/s 0:00:02 (xfr#5, to-chk=4/10)
photoTwo.png
2.92M 100% 1.01MB/s 0:00:02 (xfr#6, to-chk=3/10)
backups/
backups/index.js
0 100% 0.00kB/s 0:00:00 (xfr#7, to-chk=1/10)
backups/database/
sent 15.28M bytes received 164 bytes 985.74K bytes/sec
total size is 15.27M speedup is 1.00
Conclusion
rsync is a valuable tool for transferring and synchronizing files and directories between local and remote systems. Its delta-transfer algorithm makes data transfer incredibly efficient and its support for different options makes it a flexible transfer solution for a variety of use cases, including routine backups and large data migration.
System administrators and other users can leverage rsync to better manage their file transfers.
Further Reading
Here are some more resources that you can check out to learn more about how to use the rsync command for tranfering files and backup.
https://wiki.archlinux.org/title/Rsync
https://www.pointsoftware.ch/2012/09/12/howto-local-and-remote-snapshot-backup-using-rsync-with-hard-links/
