Harsh but fair

Open source chicanery and the battle with my inner geek

Using Exclude Files in rsync – watch those spaces!

Posted by raetsel on May 15, 2008

Summary

When using filter or exclude files ensure each entry has no spaces at the end of it or it will not match correctly e.g.

- Music/
- downloads/

and not

- Music/[space]
- downloads/[space]

Use two -v options on the command line to get output about what was skipped or included and why.

Use the -n option to simulate what rsync will do without actually copying any files.

Detail

I’ve recently started using rsync to copy files to an online backup server (rsync.net – the name is co-incidental).

Initially I started by using exclude options on the command line for the big directories that I didn’t want uploaded, things like my Music files etc. The command line ended up something like:-

rsync -n -v -a -z --exclude=Music/ --exclude=downloads/ --exclude=".*" --exclude=macky/ --exclude=XBMC/ -r /home/simon/ blah@rsync.net:blahblah

I decided it would make more sense to use an exclude file or filter file rather than an ever expanding command line so the I changed the command to be

rsync -v -a -z --filter='merge /home/simon/rsync_exclude' /home/simon/ blah@rsync.net:blahblah

I then created the file /home/simon/rsync_exclude by cutting and pasting from the old command line and editing the file to put one entry per line with the – sign in front to indicate it should not be included:-

- downloads/
- Music/
- macky/
- XBMC/
- ktorrent/
- /.*

However when I ran the rsync command with the filter file it started to copy up all my Music etc. after much fiddling and use of the very helpful -n option to just simulate what happening and -v -v to show what was being selected I discovered the problem.

In spliting the lines in my exclude file I had left a space at the end of each line and rsync was doing a literal match and so looking for something that matched “Music/[space]” and not finding anything.

So make sure any filter or exclude files don’t have any spaces at the end of the line.

The -n option for rsync is very useful it just simulates what rsync would do if you ran it for real. Also if you run rsync with -v -v ( yes two -v options ) you get detailed output about what is skipped etc.

rsync -n -v -v blah blah
building file list ...
[sender] hiding directory .Skype because of pattern /.*
[sender] hiding directory .kchmviewer because of pattern /.*
[sender] hiding directory .gnupg because of pattern /.*
[sender] hiding file .gtk-bookmarks because of pattern /.*
[sender] hiding file .profile because of pattern /.*
[sender] hiding file .dmrc because of pattern /.*
.......

About these ads

6 Responses to “Using Exclude Files in rsync – watch those spaces!”

  1. himuraken said

    Very useful, thanks for the breakdown.

  2. aj said

    Thank you!!! I spent the past hour+ trying to figure out why my –exclude-from option wasn’t working and this was it. Amazing.

  3. Jozsef said

    Same here!
    Spent long time to get exclude option to work but was not able to make it till I read this page.
    Thanks!

    Best,
    Jozsef

  4. […] Posts Mac OS X Snow Leopard & Cisco AnyConnect VPNUsing Exclude Files in rsync – watch those spaces!Snmpd filling up /var/log/messagesLaptop Build: apt-get behind a proxyEscape from TelnetKDE4 Save […]

  5. memeLab said

    so good! this is the first mention i’ve heard of the -vv option (i missed it in the man page, too!) so helpful, thanks!

  6. Martin said

    Sitting 2 hours to find the same what you found….

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
Follow

Get every new post delivered to your Inbox.

%d bloggers like this: