I have a very large picture archive that needs backup "off-site". Week to week this archive sees a lot of action, whether it be picture addition or reorganization. The pictures are organized by directory, and directories are often moved or renamed. Nothing ever gets deleted. I wanted:
- daily backups so I could go back and see the archive on a particular day.
- hard-links to the previous backup so that only "new" files use disk space.
- graceful handling of moved, renamed, and duplicate files without additional storage or transfer bandwidth.
- backup format of a directory tree that can be directly copied when needed.
- remote backups using ssh.
I didn't find exactly this tool so I created Link-Backup ("lb"). lb keeps track of file stats and content and is able to find and hard-link to identical files independent of filename or path. A side benefit of this is that it'll find and hard-link identical files within your archive. In addition, if there is a duplicate file with different stats (can't be hard-linked) lb will perform a copy at the destination instead of transferring a new copy. lb requires python to be installed at both the source and destination. It sends itself to the remote end, so it only needs to be installed where it is invoked.