Rsync and Linux
This post was last updated on July 9th, 2020 at 04:32 pm
This is how the rsync command works:
cd /var/ftp/pub/rsync/; rsync -azvx dest_server::ftp .
This is how to generate keys and populate them on the server so you can login to it without a password and run a command:
ssh-keygen -t dsa
cat .ssh/id_dsa.pub | ssh [email protected]_ip “(cd .ssh && cat – >> authorized_keys2)”
Here’s how to verify that the local rsync files match the remote, then remove the remote rsync files if they are the same (moving from remote to local using rsync to copy and diff/nfs/ssh to remove)
for X in *; do diff -Naurq $X /mnt/nfs/$X && ssh [email protected]_ip “(cd /var/ftp/pub && rm $X)”; done
Here’s an improvement to handle filenames w/ spaces:
(IFS=,; for X in `find . -type f -printf “%h/%f,”`; do echo _________$X; cat $X | ssh [email protected] “(cd \”`pwd`\” && diff -Naurq – \”$X\” && rm -rf \”$X\” )”; done)
Linux can do some nice snapshot backups.
Just make a nice little script, adjust the date stamp to your liking and run as often as you like. Of course this script can be improved (such as it’s detection of the last rsync run) I’ll take any improvements to the script or I’ll update it as I make changes to my script:
## New rsync style BACKUPDIR=/root/backups/ro mkdir -p $BACKUPDIR DATE=`date +%Y%m%d` # Note if you will be doing snapshots more frequently (than daily) # then you might use the full date stamp of +%Y%m%d%H%M%S LAST=`ls $BACKUPDIR | grep "^[0-9][0-9]*$" | sort | tail -1` if [ -n "$LAST" ]; then rsync -a --relative --link-dest=../$LAST `find /tftpboot ! -type d | grep -v img$` /var/ftp/pub/scripts /var/ftp/pub/kickstart /var/ftp/pub/testing $BACKUPDIR/$DATE else rsync -a --relative `find /tftpboot ! -type d | grep -v img$` /var/ftp/pub/scripts /var/ftp/pub/kickstart /var/ftp/pub/testing $BACKUPDIR/$DATE fi REMOTE_LAST=`ssh [email protected] "ls -d /var/storage/[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]" | tail -1` REMOTE_LAST=`basename $REMOTE_LAST` if [ -n "$REMOTE_LAST" ]; then rsync -e ssh -a --relative --link-dest=../$REMOTE_LAST `find /tftpboot ! -type d | grep -v img$` /var/ftp/pub/scripts /var/ftp/pub/kickstart /var/ftp/pub/testing [email protected]:/var/storage/$DATE else rsync -e ssh -a --relative `find /tftpboot ! -type d | grep -v img$` /var/ftp/pub/scripts /var/ftp/pub/kickstart /var/ftp/pub/testing [email protected]:/var/storage/$DATE fi ## Old Style #DATE=`date +%Y%m%d` #mkdir -p /root/backups/$DATE #cd / #tar -czvf /root/backups/$DATE/$DATE.tgz `find tftpboot | grep -v img$ | grep -v ^tftpboot$` var/ftp/pub/scripts var/ftp/pub/kickstart var/ftp/pub/testing
On the above script I probably have the ssh key thing set up too.
Rsyncing files from behind a firewall without doing extra work:
rsync -e "ssh -X [email protected] ssh -X [email protected]" -avx --progress --relative --bwlimit=10 :/pictures/20050420 . [email protected]'s password:
You will get the above command line prompt for your first ssh password. You will get the X11 popup for each of the subsequent password prompts (you can do multiple hops). This saves me from having to copy the files to my gateway, then locally, then delete from the gateway, not to mention needing extra space on the gateway.
Using rsync to copy only a certain file type or file extension, or pattern:
Note: Lots of folks on different web pages keep saying you must match ALL the top level directories but that just is not so, you just have to include all directories with a pattern match (+ */):
Contents of include file (or exclude file — it is the same thing either way you go)
rsync -avx –include-from above-file /source /destination
(You can use either of –include-from or –exclude-from when using the + – patterns)
For rsync you have to include before you exclude. Once you match a pattern then pattern matching stops. If you don’t match a pattern then it falls through until it matches or runs out of patterns. In the case above, the last pattern excludes everything that wasn’t matched above. So:
1. Include all directories (*/), so if it’s a directory it automatically adds it and moves on
2. Includes all patterns (only files left since we added all directories above) that end in mp3
3. Excludes everything that didn’t match either of the 2 above includs.
The only possible downside is that you make a copy of the directory tree on the destination, whether those directories contain *mp3 files or not. Oh well, I can live with that. Here’s a way to quickly remove empty directories (note that rmdir doesn’t remove directories that have contents unless you force it – so we will use the “politeness” of rmdir to our advantage):
find /destination -type d | tac | while read; do rmdir “$REPLY”; done
The above should account for filenames with spaces as well as a large quantity of directory names. Here is a shorter way but works on a smaller list of directories names as well as doesn’t handle spaces well. So if you know that the directory names have no spaces in them and you have less than maybe 1000 directories then the following works good:
rmdir `find /destination -type d | tac`
Let us say you want to match mp3 anywhere in the filename and not just the end then it looks like the following will work for your include file (I didn’t fully test the following but it looks like it works good enough):
Then to match the beginning of a filename you could probably use mp3* or /mp3* (untested).
If you want to use rsync to do multiple files on the source part:
rsync -e “ssh -C [email protected]” -avx –progress “:/source1 /src2 /src3” /destination
So I’m sure you could combine this with the above –include-from pattern matching to.
You might also like
This post was last updated on June 1st, 2020 at 07:00 pmBy default you have 8 loop devices (loop0 – loop7). You can extend this number up to 255 (maybe
This post was last updated on August 3rd, 2020 at 01:50 pmOriginally the term RAID defined as “Redundant Array of Inexpensive Disks” but now it is referred to as “Redundant