Urgent need to extract data from duplicity backup files

Maintenance
1

I am starting a new topic because there seems to be no interest in the old one:
Upgrade restore from backup errors

Because the restore failed, I have been forced to start the upgraded MIAB server with empty database. When I began using MIAB, about 18 months ago, I had the same situation, as my previous mail data were in a Maildir hierarchy of around 100 “folders”. But then, at least I was able to open two email clients and refer to old mail, cut and paste from it when needed, and even to log on to the server and use grep etc to search the old mails. In MIAB the mails are buried in a database and only accessible through the server. This means I am unable to carry on existing correspondence unless I can unlock the data hidden in the backup.

To recap, the backup consists of 78 .gpg files. I attempted the restoration as instructed, and here is the resultant error (the same as reported by by Talbot in the thread referred above.)

export PASSPHRASE=$(cat /path/to/miab-bak/secret_key.txt)
$ sudo -E duplicity restore --force file:///miab-bak/encrypted /home/user-data/
gpg: Attention : le propriétaire du répertoire personnel « /home/richard/.gnupg »
            n'est pas sûr
Les métadonnées locales et distantes sont déjà synchronisées. Aucune synchronisation nécessaire.
Date de la dernière sauvegarde complète : aucune
Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1541, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1393, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1472, in do_backup
    restore(col_stats)
  File "/usr/bin/duplicity", line 728, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/usr/bin/duplicity", line 750, in restore_get_patched_rop_iter
    backup_chain = col_stats.get_backup_chain_at_time(time)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 974, in get_backup_chain_at_time
    raise CollectionsError("No backup chains found")
 CollectionsError: No backup chains found

So now I need to get the data from the backup in readable form so that I can use it to continue work and so that I retain a full archive of past correspondence. It is now too late to back it up to the server, since I had to restart the server and it now contains current data in the mail database. It is a pity it is not in Maildir format, as then I would be able to merge the two sets of data manually, but any readable format would do. I could even boot up yet another MIAB box to be able to read it.

Does anyone here know how to do this?

2

I suspect that you could spin up a droplet with Ubuntu 14.04 and reinstall MiaB v0.30 and restore the data base to it. Note, if you do this you would not want to do the normal install procedure … you will want to use a different hostname and ignore glue records, etc.

Looking at your comment in the other thread and what you have posted here, you do not have the same error. My French is lacking, but there is a message about ownership of a .gnupg directory in what you posted.

Just a hunch, but check owner and permissions of the backup and see if they are the same as on the new box.

3

alento: Thank you.

I had already checked the ownership and permissions.
The .gnupg directory is owned by my username 700. It contains only an empty directory ‘private-keys-v1.d’, also 700. The backup directory is owned by me 755 and its contents and its key are all owned by me and all 600.

I have just tried setting it all 777 but the result is unchanged. I am beginning to suspect corruption of some key file during the slow process of uploading the backup over an ADSL link. I am thinking of installing ubuntu 14.04 on spare space on my home box and trying the restore here, though I could start another VM either with Bytemark in UK (where the MiaB server is), or here in France with OVH.

If I do get it working on MiaB 0.30, are you suggesting I then try the upgrade all over again, or just keep it as an archive to view and copy from as needed?

If I try unencrypting all the 78 files, how would I use them to extract the data? Are they SQL or in any way readable outside of the MiaB package of applications?

4

Either, or. Attempt a new backup and try the upgrade again. Though my line of thinking was to then use imapsync to transfer the files if that is reasonable if attempting another upgrade/restore didn’t work.

The backup is of the /home/user-data/ directory. You can look at the current running box on Ubuntu 18.04 to get an idea of what it looks like. Mail is stored in maildir format, not in the database itself.

5

Goodness! I don’t know why I never thought to look. There is something about the presentation of MiaB that discourages tinkering.

Thank you. I will investigate…

6

After several attempts, success. I ended up zipping up the entire /home/user-data/backup/ folder recursively, copying it over to the new droplet, and pointing the restore to the /backup/encrypted/ folder.

7

I have also reached a degree of success. I set up a new Ubuntu 14.04 server and installed MiaB. Copied the backup there and tried restoring from that. At first, I had the same errors, but, by chance, I tried adding a trailing slash to the directory containing the .pgp files – that is “file:///miab-bak/encrypted/”. This time, it restored successfully. Now, I wonder whether merely adding the trailing slash would have been enough to avoid the problem in the new version of MiaB 0.40.

I am now in the process of scp-ing the extracted data to the MiaB box, where I hope I can easily insert and/or merge it.

FINAL UPDATE: I was able to swap the whole restored user-data back in to the new installation. I had to manually adjust all the ownerships which had been lost in transmission, and then run sudo mailinabox. Also, I needed to retain the new user-data/ssl, rather than that from the backup otherwise nginx failed to restart. All I have left is to merge a handful of emails from the last two or three days.

So, problem solved. But I think MiaB developers should confirm whether that trailing slash ought to be included.

Thanks to alento for the help.

8

I got a very similar error message and felt quite kerfuffled about it for some time. After looking at my .gpg file set, I found that the duplicity encryption/backup process had quit because it hit a set of target files that were corrupted. All my backups were bad/incomplete because my duplicity process abruptly stopped before writing the duplicity-full-signatures.xxx.sigtar.gpg file. In my findings, it appears this file index is necessary/critical.

Duplicity reported which file/folder it was choking on. I removed it, and the backups ran to completion. After that, I was able to perform successful restorations.