Rsync and Linux

Rsync and Linux


This post was last updated on July 9th, 2020 at 04:32 pm

This is how the rsync command works:
cd /var/ftp/pub/rsync/; rsync -azvx dest_server::ftp .

This is how to generate keys and populate them on the server so you can login to it without a password and run a command:
ssh-keygen -t dsa
cat .ssh/ | ssh [email protected]_ip “(cd .ssh && cat – >> authorized_keys2)”

Here’s how to verify that the local rsync files match the remote, then remove the remote rsync files if they are the same (moving from remote to local using rsync to copy and diff/nfs/ssh to remove)
for X in *; do diff -Naurq $X /mnt/nfs/$X && ssh [email protected]_ip “(cd /var/ftp/pub && rm $X)”; done
Here’s an improvement to handle filenames w/ spaces:
(IFS=,; for X in `find . -type f -printf “%h/%f,”`; do echo _________$X; cat $X | ssh [email protected] “(cd \”`pwd`\” && diff -Naurq – \”$X\” && rm -rf \”$X\” )”; done)

Linux can do some nice snapshot backups.

Just make a nice little script, adjust the date stamp to your liking and run as often as you like. Of course this script can be improved (such as it’s detection of the last rsync run) I’ll take any improvements to the script or I’ll update it as I make changes to my script:

## New rsync style
mkdir -p $BACKUPDIR
DATE=`date +%Y%m%d`
# Note if you will be doing snapshots more frequently (than daily) 
# then you might use the full date stamp of +%Y%m%d%H%M%S
LAST=`ls $BACKUPDIR | grep "^[0-9][0-9]*$" | sort | tail -1`
if [ -n "$LAST" ]; then
	rsync -a --relative --link-dest=../$LAST `find /tftpboot ! -type d | grep -v img$` /var/ftp/pub/scripts /var/ftp/pub/kickstart /var/ftp/pub/testing $BACKUPDIR/$DATE
	rsync -a --relative `find /tftpboot ! -type d | grep -v img$` /var/ftp/pub/scripts /var/ftp/pub/kickstart /var/ftp/pub/testing $BACKUPDIR/$DATE
REMOTE_LAST=`ssh [email protected] "ls -d /var/storage/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]" | tail -1`
if [ -n "$REMOTE_LAST" ]; then
	rsync -e ssh -a --relative --link-dest=../$REMOTE_LAST `find /tftpboot ! -type d | grep -v img$` /var/ftp/pub/scripts /var/ftp/pub/kickstart /var/ftp/pub/testing [email protected]:/var/storage/$DATE
	rsync -e ssh -a --relative `find /tftpboot ! -type d | grep -v img$` /var/ftp/pub/scripts /var/ftp/pub/kickstart /var/ftp/pub/testing [email protected]:/var/storage/$DATE

## Old Style
#DATE=`date +%Y%m%d`
#mkdir -p /root/backups/$DATE
#cd /
#tar -czvf /root/backups/$DATE/$DATE.tgz `find tftpboot | grep -v img$ | grep -v ^tftpboot$` var/ftp/pub/scripts var/ftp/pub/kickstart var/ftp/pub/testing

On the above script I probably have the ssh key thing set up too.

Rsyncing files from behind a firewall without doing extra work:

rsync -e "ssh -X [email protected] ssh -X [email protected]" -avx --progress --relative --bwlimit=10 :/pictures/20050420 .
[email protected]'s password:

You will get the above command line prompt for your first ssh password. You will get the X11 popup for each of the subsequent password prompts (you can do multiple hops). This saves me from having to copy the files to my gateway, then locally, then delete from the gateway, not to mention needing extra space on the gateway.

Selective rsync
Using rsync to copy only a certain file type or file extension, or pattern:

Note: Lots of folks on different web pages keep saying you must match ALL the top level directories but that just is not so, you just have to include all directories with a pattern match (+ */):

Contents of include file (or exclude file — it is the same thing either way you go)
+ */
+ *mp3
– *

Rsync command:
rsync -avx –include-from above-file /source /destination
(You can use either of –include-from or –exclude-from when using the + – patterns)

For rsync you have to include before you exclude. Once you match a pattern then pattern matching stops. If you don’t match a pattern then it falls through until it matches or runs out of patterns. In the case above, the last pattern excludes everything that wasn’t matched above. So:

1. Include all directories (*/), so if it’s a directory it automatically adds it and moves on
2. Includes all patterns (only files left since we added all directories above) that end in mp3
3. Excludes everything that didn’t match either of the 2 above includs.

The only possible downside is that you make a copy of the directory tree on the destination, whether those directories contain *mp3 files or not. Oh well, I can live with that. Here’s a way to quickly remove empty directories (note that rmdir doesn’t remove directories that have contents unless you force it – so we will use the “politeness” of rmdir to our advantage):

find /destination -type d | tac | while read; do rmdir “$REPLY”; done

The above should account for filenames with spaces as well as a large quantity of directory names. Here is a shorter way but works on a smaller list of directories names as well as doesn’t handle spaces well. So if you know that the directory names have no spaces in them and you have less than maybe 1000 directories then the following works good:

rmdir `find /destination -type d | tac`

Let us say you want to match mp3 anywhere in the filename and not just the end then it looks like the following will work for your include file (I didn’t fully test the following but it looks like it works good enough):

+ */
+ *mp3*
– *

Then to match the beginning of a filename you could probably use mp3* or /mp3* (untested).

If you want to use rsync to do multiple files on the source part:

rsync -e “ssh -C [email protected]” -avx –progress “:/source1 /src2 /src3” /destination

So I’m sure you could combine this with the above –include-from pattern matching to.


+ */
+ *mp3*
– *


+ */
+ *mp3*
– /*


+ /mydir
+ /mydir1/
– /*

Previous Converting from .PST to mbox format
Next Capturing text and images

About author

shibaa987 275 posts

Linux kernel developer and a firmware developer with an experience of 10+ years.

View all posts by this author →

You might also like

System Internals 0 Comments

Linux and Memory

This post was last updated on May 26th, 2020 at 10:10 pmContents1 Determining if your entire memory is being detected:2 Specifying Undetected Memory – Method #12.0.1 The following format is

System Internals 0 Comments

Basic Operating System Services: A beginner’s guide

This post was last updated on November 3rd, 2020 at 05:20 pmOperating System Services are the basic functionalities or the responsibilities of an Operating System that it provides to the

System Internals 0 Comments

Linux Boot Process

This post was last updated on June 3rd, 2020 at 03:34 pmThe following is from /usr/src/linux-2.4/Documentation/initrd.txt When using initrd, the system typically boots as follows: 1) the boot loader loads


No Comments Yet!

You can be first to comment this post!

Leave a Reply