Howto: backup to hpcloud.com object store using duplicity

Warning: this is only a demonstration, and the hpcloud is only a beta trial! dont rely on this to backup your important data. However...

I managed to get duplicity working with the hpcloud.com object store without any changes, just some close reading of the hpcloud documentation on legacy authentication support.



Basically hpcs are moving onto the identity style of authentication by passing the credentials or account keys as part of the body of the authentication request. However most tools currently available work with the old style of passing X-Auth headers. (including the python-cloudfiles libs for duplicity)

To configure the required duplicity ENV variables, you need to look at your API keys page and make a note of the tenant-id value. Then add your account email address, and your login password, and they are composed together like this;  

X-Auth-User: <tenant-id>:<email-address> 
X-Auth-Key: <hpcloud-password>


The legacy version 1.0 AUTHURL is

https://region-a.geo-1.identity.hpcloudsvc.com:35357/auth/v1.0/

duplicity

This requires a recent version of duplicity and python-cloudfiles (https://github.com/rackspace/python-cloudfiles), so if you experience any problems its likely that the first step is to update to recent cloudfiles using the docs on github.

On fedora 15, the following command worked fine for me;

 sudo yum -y install duplicity python-cloudfiles

Once duplicity (http://www.nongnu.org/duplicity/) is installed you can then add the following to your ~/.bashrc file to enable backup to your hpcloud object store in a one-liner.

======= add to /home/<username>/.bashrc=======

export UPLOAD_TO_CONTAINER="my-backup-container-2012" #adjust it as you like 
export CLOUDFILES_AUTHURL="https://region-a.geo-1.identity.hpcloudsvc.com:35357/auth/v1.0/" 
export CLOUDFILES_USERNAME="<tenant-id>:<email-address>" 
export CLOUDFILES_APIKEY="<hpcloud-password>" 
export PASSPHRASE="" 
alias duplicity="duplicity --no-encryption"

======= end /home/<username>/.bashrc=======

You can then backup your data using the following command style;  

[tomhodder@workstation001 ~]$ touch ~/notes/cycling/testfile_deletme.txt 
[tomhodder@workstation001 ~]$ duplicity ~/notes/cycling/ cf+http://my-back-up-full-data-etc
 Local and Remote metadata are synchronized, no sync needed. Last full backup date: none No signatures found, switching to full backup. 
--------------[ Backup Statistics ]-------------- 
StartTime 1329330905.94 (Wed Feb 15 18:35:05 2012) 
EndTime 1329330906.01 (Wed Feb 15 18:35:06 2012) 
ElapsedTime 0.07 (0.07 seconds) SourceFiles 2 
SourceFileSize 170871 (167 KB) NewFiles 2 
NewFileSize 170871 (167 KB) DeletedFiles 0 
ChangedFiles 0 ChangedFileSize 0 (0 bytes) 
ChangedDeltaSize 0 (0 bytes) DeltaEntries 2 
RawDeltaSize 166775 (163 KB) 
TotalDestinationSizeChange 155070 (151 KB) 
Errors 0 --------------------------
-----------------------


you can verify and restore files using the alternative syntax, with the source and destination reversed.  

[tomhodder@workstation001 ~]$ rm ~/notes/cycling/testfile_deletme.txt 
[tomhodder@workstation001 ~]$ duplicity verify cf+http://my-back-up-stufftesty11 ~/notes/cycling 
Local and Remote metadata are synchronized, no sync needed. Last full backup date: Wed Feb 15 18:35:05 2012 Difference found: 
File . has mtime Wed Feb 15 18:48:01 2012, expected Wed Feb 15 18:37:40 2012 Difference found: 
File testfile_deletme.txt is missing Verify complete: 3 files compared, 2 differences found.  
[tomhodder@workstation001 ~]$ duplicity restore --file-to-restore testfile_deletme.txt cf+http://my-back-up-stufftesty11 ~/notes/cycling/testfile_deletme.txt [tomhodder@workstation001 ~]$ ls -lah ~/notes/cycling/testfile_deletme.txt -rw-rw-r-- 1 tomhodder tomhodder 0 Feb 15 18:37 /home/tomhodder/notes/cycling/testfile_deletme.txt


Notes =====

you will need to source your bashrc file after making changes to it to see this stuff if you dont the the no-encryption option on duplicity you will need to create gpg keys for that.

see duplicty man pages for encrypted backups if you reference a non-existing container, openstack will create it for you.

I managed to do some stupid stuff using "duplicity restore --force" do it was lucky I had a backup available! so I would recommend restoring to another folder and copying over, or renaming your local directory rather than restorting to --force

http://www.nongnu.org/duplicity/
 https://github.com/rackspace/python-cloudfiles
 http://api-docs.hpcloud.com/hpcloud-identity-service/1.0/content/authenticate.html

back compatibility for authentication;
 https://connect.hpcloud.com/article/hp-cloud-object-storage-users-identity-changes