Rsync and Linux

Rsync and Linux

 

This post was last updated on July 9th, 2020 at 04:32 pm

This is how the rsync command works:
cd /var/ftp/pub/rsync/; rsync -azvx dest_server::ftp .

This is how to generate keys and populate them on the server so you can login to it without a password and run a command:
ssh-keygen -t dsa
cat .ssh/id_dsa.pub | ssh [email protected]_ip “(cd .ssh && cat – >> authorized_keys2)”

Here’s how to verify that the local rsync files match the remote, then remove the remote rsync files if they are the same (moving from remote to local using rsync to copy and diff/nfs/ssh to remove)
for X in *; do diff -Naurq $X /mnt/nfs/$X && ssh [email protected]_ip “(cd /var/ftp/pub && rm $X)”; done
Here’s an improvement to handle filenames w/ spaces:
(IFS=,; for X in `find . -type f -printf “%h/%f,”`; do echo _________$X; cat $X | ssh [email protected] “(cd \”`pwd`\” && diff -Naurq – \”$X\” && rm -rf \”$X\” )”; done)


Linux can do some nice snapshot backups.

Just make a nice little script, adjust the date stamp to your liking and run as often as you like. Of course this script can be improved (such as it’s detection of the last rsync run) I’ll take any improvements to the script or I’ll update it as I make changes to my script:

## New rsync style
BACKUPDIR=/root/backups/ro
mkdir -p $BACKUPDIR
DATE=`date +%Y%m%d`
# Note if you will be doing snapshots more frequently (than daily) 
# then you might use the full date stamp of +%Y%m%d%H%M%S
LAST=`ls $BACKUPDIR | grep "^[0-9][0-9]*$" | sort | tail -1`
if [ -n "$LAST" ]; then
	rsync -a --relative --link-dest=../$LAST `find /tftpboot ! -type d | grep -v img$` /var/ftp/pub/scripts /var/ftp/pub/kickstart /var/ftp/pub/testing $BACKUPDIR/$DATE
else
	rsync -a --relative `find /tftpboot ! -type d | grep -v img$` /var/ftp/pub/scripts /var/ftp/pub/kickstart /var/ftp/pub/testing $BACKUPDIR/$DATE
fi
REMOTE_LAST=`ssh [email protected] "ls -d /var/storage/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]" | tail -1`
REMOTE_LAST=`basename $REMOTE_LAST`
if [ -n "$REMOTE_LAST" ]; then
	rsync -e ssh -a --relative --link-dest=../$REMOTE_LAST `find /tftpboot ! -type d | grep -v img$` /var/ftp/pub/scripts /var/ftp/pub/kickstart /var/ftp/pub/testing [email protected]:/var/storage/$DATE
else
	rsync -e ssh -a --relative `find /tftpboot ! -type d | grep -v img$` /var/ftp/pub/scripts /var/ftp/pub/kickstart /var/ftp/pub/testing [email protected]:/var/storage/$DATE
fi

## Old Style
#DATE=`date +%Y%m%d`
#mkdir -p /root/backups/$DATE
#cd /
#tar -czvf /root/backups/$DATE/$DATE.tgz `find tftpboot | grep -v img$ | grep -v ^tftpboot$` var/ftp/pub/scripts var/ftp/pub/kickstart var/ftp/pub/testing

On the above script I probably have the ssh key thing set up too.


Rsyncing files from behind a firewall without doing extra work:

rsync -e "ssh -X [email protected] ssh -X [email protected]" -avx --progress --relative --bwlimit=10 :/pictures/20050420 .
[email protected]'s password:

You will get the above command line prompt for your first ssh password. You will get the X11 popup for each of the subsequent password prompts (you can do multiple hops). This saves me from having to copy the files to my gateway, then locally, then delete from the gateway, not to mention needing extra space on the gateway.


Selective rsync
Using rsync to copy only a certain file type or file extension, or pattern:

Note: Lots of folks on different web pages keep saying you must match ALL the top level directories but that just is not so, you just have to include all directories with a pattern match (+ */):

Contents of include file (or exclude file — it is the same thing either way you go)
+ */
+ *mp3
– *

Rsync command:
rsync -avx –include-from above-file /source /destination
(You can use either of –include-from or –exclude-from when using the + – patterns)

For rsync you have to include before you exclude. Once you match a pattern then pattern matching stops. If you don’t match a pattern then it falls through until it matches or runs out of patterns. In the case above, the last pattern excludes everything that wasn’t matched above. So:

1. Include all directories (*/), so if it’s a directory it automatically adds it and moves on
2. Includes all patterns (only files left since we added all directories above) that end in mp3
3. Excludes everything that didn’t match either of the 2 above includs.

The only possible downside is that you make a copy of the directory tree on the destination, whether those directories contain *mp3 files or not. Oh well, I can live with that. Here’s a way to quickly remove empty directories (note that rmdir doesn’t remove directories that have contents unless you force it – so we will use the “politeness” of rmdir to our advantage):

find /destination -type d | tac | while read; do rmdir “$REPLY”; done

The above should account for filenames with spaces as well as a large quantity of directory names. Here is a shorter way but works on a smaller list of directories names as well as doesn’t handle spaces well. So if you know that the directory names have no spaces in them and you have less than maybe 1000 directories then the following works good:

rmdir `find /destination -type d | tac`

Let us say you want to match mp3 anywhere in the filename and not just the end then it looks like the following will work for your include file (I didn’t fully test the following but it looks like it works good enough):

+ */
+ *mp3*
– *

Then to match the beginning of a filename you could probably use mp3* or /mp3* (untested).


If you want to use rsync to do multiple files on the source part:

rsync -e “ssh -C [email protected]” -avx –progress “:/source1 /src2 /src3” /destination

So I’m sure you could combine this with the above –include-from pattern matching to.


???

+ */
+ *mp3*
– *


???

+ */
+ *mp3*
– /*


???

+ /mydir
+ /mydir1/
– /*


http://optics.ph.unimelb.edu.au/help/rsync/rsync_pc1.html

Previous Converting from .PST to mbox format
Next Capturing text and images
 

About author

shibaa987
shibaa987 241 posts

Linux kernel developer and a firmware developer with an experience of 10+ years.

View all posts by this author →

You might also like

System Internals 0 Comments

Static Routes in RH73 Linux

This post was last updated on June 3rd, 2020 at 04:36 pm(Other routing page: routes.html) Here’s how to set up Red Hat 7.3 /etc/sysconfig/static-routes file, I mention this here because

System Internals 0 Comments

The Concept Of RAID

This post was last updated on August 3rd, 2020 at 01:50 pmOriginally the term RAID defined as “Redundant Array of Inexpensive Disks” but now it is referred to as “Redundant

System Internals 0 Comments

What’s an Operating system & why do we need one?

This post was last updated on November 3rd, 2020 at 05:15 pmBasically an operating system (OS) is a software system (collection of interdependent software code) on its own that allows

0 Comments

No Comments Yet!

You can be first to comment this post!

Leave a Reply