Using Exclude Files in rsync - watch those spaces!

Summary

When using filter or exclude files ensure each entry has no spaces at the end of it or it will not match correctly e.g.

- Music/
- downloads/

and not

- Music/[space]
- downloads/[space]

Use two -v options on the command line to get output about what was skipped or included and why.

Use the -n option to simulate what rsync will do without actually copying any files.

Detail

I’ve recently started using rsync to copy files to an online backup server (rsync.net - the name is co-incidental).

Initially I started by using exclude options on the command line for the big directories that I didn’t want uploaded, things like my Music files etc. The command line ended up something like:-

rsync -n -v -a -z --exclude=Music/ --exclude=downloads/ --exclude=".*" --exclude=macky/ --exclude=XBMC/ -r /home/simon/ blah@rsync.net:blahblah

I decided it would make more sense to use an exclude file or filter file rather than an ever expanding command line so the I changed the command to be

rsync -v -a -z --filter='merge /home/simon/rsync_exclude' /home/simon/ blah@rsync.net:blahblah

I then created the file /home/simon/rsync_exclude by cutting and pasting from the old command line and editing the file to put one entry per line with the - sign in front to indicate it should not be included:-

- downloads/
- Music/
- macky/
- XBMC/
- ktorrent/
- /.*

However when I ran the rsync command with the filter file it started to copy up all my Music etc. after much fiddling and use of the very helpful -n option to just simulate what happening and -v -v to show what was being selected I discovered the problem.

In spliting the lines in my exclude file I had left a space at the end of each line and rsync was doing a literal match and so looking for something that matched “Music/[space]” and not finding anything.

So make sure any filter or exclude files don’t have any spaces at the end of the line.

The -n option for rsync is very useful it just simulates what rsync would do if you ran it for real. Also if you run rsync with -v -v ( yes two -v options ) you get detailed output about what is skipped etc.

rsync -n -v -v blah blah
building file list …
[sender] hiding directory .Skype because of pattern /.*
[sender] hiding directory .kchmviewer because of pattern /.*
[sender] hiding directory .gnupg because of pattern /.*
[sender] hiding file .gtk-bookmarks because of pattern /.*
[sender] hiding file .profile because of pattern /.*
[sender] hiding file .dmrc because of pattern /.*
…….