Hi all
Ever since upgrading to MiaB v60, I’ve suspected that the hard disk I’m using with Ubuntu 22.04 probably has block errors. As a precaution, before it’s too late, I tried to clone the hard disk to a reliable SSD. I ran the clone process using Ubuntu command “sudo dd if=/dev/sda of=/dev/sdb bs=4M conv=sync,noerror status=progress”, both, the sda (current HDD), and sdb (new SSD) connected to an USB 3.0 drive each. My suspicion has been confirmed: the hard disk actually has errors. And the
cloned SSD did NOT boot at all. So I have to store the current version 62 with three domains with the following command “sudo rsync -aAXv /home/user-data/ /media/backup/MiaB-v62-User-Data-on-Ubuntu-22-2023-06-12/” to an external backup medium. And install a fresh Ubuntu 22.04 and MiaB v62 on the SSD. Question: Can I restore the entire /home/user-data/ directory or do I have to restore each domain individually?
I haven’t had to handle a disk failure, but MiaB does do nightly backups to for such a situation. I would use the backup to recreate your system, and not try to do it manually with rsync. There are probably docs around on how to restore a newly built system from backup, but if there are not, then you can probably follow the same procedure that was documented for upgrading from Ubuntu 18.04 to 22.04 (which also included restoring the new system from the backup).
During the upgrade procedure (from 18.04 to 22.04) I tried to follow the procedure that was documented for upgrading from Ubuntu 18.04 to 22.04 (which also included restoring the new system from the backup). But that didn’t work. Because with SCP I got the backup data (/home/user-data/backup/encrypted) from MiaB-v57a with the secret_key.txt over to the freshly installed MiaB v60.1 also in the same directory /home/user-data /backup/encrypted. When restoring I took the secret_key.txt from MiaB v57a. The command was as follows: “sudo -E duplicity restore --force file:///home/user-data/backup/encrypted /home/user-data/”. The result showed the following:
Entfernte Metadaten werden zum lokalen Puffer synchronisiert …
GnuPG passphrase for decryption:
duplicity-full-signatures.20230114T023913Z.sigtar.gpg wird zum lokalen Puffer kopiert.
…
5/WfKbdWwdkfL3SANSS/X+GMhhWEv5LOTvz24MOBdl18MIuuh2dhs7jMLv7I4G4u
WmyKRh0aUEZRJR4sPI0kUX7n2MFkB13GhQTghxOXOD6YpZ8lPYj9m3VMoEsxLZJP
6x4vEoLXKP1CAQ2CxK+avfqyILvxWK/y33gqgbGYVjy2YkS0jirrkWzVS/2nuvXV
…
GPGError: GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: AES256.CFB encrypted data
gpg: encrypted with 1 passphrase
gpg: decryption failed: Bad session key
===== End GnuPG log =====
Due to this error I successfully copied an “old” mailbox over from MiaB v57a with rsync: “sudo rsync -Aavx /media/backup/MiaB-v57a-on-Ubuntu-18-2023-01-03/home/user-data/mail/mailboxes/example.com /home/user-data/mail/mailboxes/”. Question: Is there a clear documentation on how to use the restore to bring back a saved installation?
I’m not sure why you got those errors on the first try, unless it’s trying to use the wrong passphrase or maybe your data was corrupted. The command to restore looks fine to me. Here’s the steps I followed when I upgraded to v61. The one main difference was that I didn’t place the backup directory under /home/user-data, I put it under /root. BTW, putting it under /root requires that you do all of this as the root user. If you’re not comfortable with that, then you can also put it under your home directory and use sudo, but I wouldn’t put it under user-data:
- I have a tarball of the
backupdirectory and everything below it and I copy this tar file to the root home directory,/root, on the mail server I want to restore. - I untar that file in a workspace I created under
/rootcalled/root/miab-recover, so I wind up with/root/miab-recover/backup - Put the secret key into an environment variable for duplicity to use later.
export PASSPHRASE=$(cat /root/miab-recover/backup/secret_key.txt) - Finally, restore using the duplicity command (if you’re not root, then you’ll need to prepend
sudo -Eto the command):
duplicity restore --force file:///root/miab-recover/backup/encrypted/ /home/user-data/
When you do a backup of the backup, it’s crucial that you copy the secret_key.txt along with the backup. So, for example, I have a nightly cron that creates a compressed tarball of the entire /home/user-data directory (which will include the backup), and copies this tarball to an offsite linux server for safe storage.
Thanks for the efforts. Although I’ll soon be 70, I’m still trying to manage my entire IT infrastructure myself. Many things that I used to do with self-confidence I now have to dwell on for a little longer. I’m very sorry! - So, I have the following situation: In the directory /home/user-data/backup/encrypted I have my backups (that’s really all what I have):
lion@box:/home/user-data/backup/encrypted$ ls -la
total 1303420
drwxr-xr-x 2 user-data root 20480 Jun 13 03:36 .
drwxr-xr-x 4 root root 4096 Jan 17 17:11 ..
-rw-r--r-- 1 user-data root 523282 Mai 25 03:47 duplicity-full.20230525T013538Z.manifest.gpg
-rw-r--r-- 1 user-data root 262124272 Mai 25 03:38 duplicity-full.20230525T013538Z.vol1.difftar.gpg
-rw-r--r-- 1 user-data root 262131101 Mai 25 03:41 duplicity-full.20230525T013538Z.vol2.difftar.gpg
-rw-r--r-- 1 user-data root 262183340 Mai 25 03:43 duplicity-full.20230525T013538Z.vol3.difftar.gpg
-rw-r--r-- 1 user-data root 262133116 Mai 25 03:44 duplicity-full.20230525T013538Z.vol4.difftar.gpg
-rw-r--r-- 1 user-data root 174600762 Mai 25 03:47 duplicity-full.20230525T013538Z.vol5.difftar.gpg
-rw-r--r-- 1 user-data root 45434055 Mai 25 03:47 duplicity-full-signatures.20230525T013538Z.sigtar.gpg
-rw-r--r-- 1 user-data root 1618 Mai 26 03:36 duplicity-inc.20230525T013538Z.to.20230526T013531Z.manifest.gpg
-rw-r--r-- 1 user-data root 4036664 Mai 26 03:36 duplicity-inc.20230525T013538Z.to.20230526T013531Z.vol1.difftar.gpg
-rw-r--r-- 1 user-data root 1241 Mai 27 03:35 duplicity-inc.20230526T013531Z.to.20230527T013527Z.manifest.gpg
-rw-r--r-- 1 user-data root 3058628 Mai 27 03:35 duplicity-inc.20230526T013531Z.to.20230527T013527Z.vol1.difftar.gpg
-rw-r--r-- 1 user-data root 833 Mai 28 03:36 duplicity-inc.20230527T013527Z.to.20230528T013530Z.manifest.gpg
-rw-r--r-- 1 user-data root 2281627 Mai 28 03:36 duplicity-inc.20230527T013527Z.to.20230528T013530Z.vol1.difftar.gpg
-rw-r--r-- 1 user-data root 1023 Mai 29 03:37 duplicity-inc.20230528T013530Z.to.20230529T013643Z.manifest.gpg
-rw-r--r-- 1 user-data root 419561 Mai 29 03:37 duplicity-inc.20230528T013530Z.to.20230529T013643Z.vol1.difftar.gpg
-rw-r--r-- 1 user-data root 838 Mai 30 03:35 duplicity-inc.20230529T013643Z.to.20230530T013517Z.manifest.gpg
-rw-r--r-- 1 user-data root 2018949 Mai 30 03:35 duplicity-inc.20230529T013643Z.to.20230530T013517Z.vol1.difftar.gpg
-rw-r--r-- 1 user-data root 1778 Mai 31 03:36 duplicity-inc.20230530T013517Z.to.20230531T013530Z.manifest.gpg
-rw-r--r-- 1 user-data root 3828571 Mai 31 03:35 duplicity-inc.20230530T013517Z.to.20230531T013530Z.vol1.difftar.gpg
-rw-r--r-- 1 user-data root 1447 Jun 1 03:36 duplicity-inc.20230531T013530Z.to.20230601T013529Z.manifest.gpg
-rw-r--r-- 1 user-data root 3158370 Jun 1 03:36 duplicity-inc.20230531T013530Z.to.20230601T013529Z.vol1.difftar.gpg
-rw-r--r-- 1 user-data root 1348 Jun 2 03:36 duplicity-inc.20230601T013529Z.to.20230602T013529Z.manifest.gpg
-rw-r--r-- 1 user-data root 4189568 Jun 2 03:36 duplicity-inc.20230601T013529Z.to.20230602T013529Z.vol1.difftar.gpg
-rw-r--r-- 1 user-data root 1280 Jun 3 03:35 duplicity-inc.20230602T013529Z.to.20230603T013526Z.manifest.gpg
-rw-r--r-- 1 user-data root 3905266 Jun 3 03:35 duplicity-inc.20230602T013529Z.to.20230603T013526Z.vol1.difftar.gpg
-rw-r--r-- 1 user-data root 998 Jun 4 03:36 duplicity-inc.20230603T013526Z.to.20230604T013527Z.manifest.gpg
-rw-r--r-- 1 user-data root 2406674 Jun 4 03:35 duplicity-inc.20230603T013526Z.to.20230604T013527Z.vol1.difftar.gpg
-rw-r--r-- 1 user-data root 889 Jun 5 03:36 duplicity-inc.20230604T013527Z.to.20230605T013618Z.manifest.gpg
-rw-r--r-- 1 user-data root 663505 Jun 5 03:36 duplicity-inc.20230604T013527Z.to.20230605T013618Z.vol1.difftar.gpg
-rw-r--r-- 1 user-data root 1437 Jun 6 03:35 duplicity-inc.20230605T013618Z.to.20230606T013524Z.manifest.gpg
-rw-r--r-- 1 user-data root 2803594 Jun 6 03:35 duplicity-inc.20230605T013618Z.to.20230606T013524Z.vol1.difftar.gpg
-rw-r--r-- 1 user-data root 995 Jun 7 03:36 duplicity-inc.20230606T013524Z.to.20230607T013543Z.manifest.gpg
-rw-r--r-- 1 user-data root 3101453 Jun 7 03:36 duplicity-inc.20230606T013524Z.to.20230607T013543Z.vol1.difftar.gpg
-rw-r--r-- 1 user-data root 1704 Jun 8 03:36 duplicity-inc.20230607T013543Z.to.20230608T013527Z.manifest.gpg
-rw-r--r-- 1 user-data root 7508036 Jun 8 03:36 duplicity-inc.20230607T013543Z.to.20230608T013527Z.vol1.difftar.gpg
-rw-r--r-- 1 user-data root 1051 Jun 9 03:36 duplicity-inc.20230608T013527Z.to.20230609T013528Z.manifest.gpg
-rw-r--r-- 1 user-data root 2868245 Jun 9 03:36 duplicity-inc.20230608T013527Z.to.20230609T013528Z.vol1.difftar.gpg
-rw-r--r-- 1 user-data root 1505 Jun 10 03:36 duplicity-inc.20230609T013528Z.to.20230610T013527Z.manifest.gpg
-rw-r--r-- 1 user-data root 8481433 Jun 10 03:36 duplicity-inc.20230609T013528Z.to.20230610T013527Z.vol1.difftar.gpg
-rw-r--r-- 1 user-data root 1176 Jun 12 03:37 duplicity-inc.20230610T013527Z.to.20230612T013625Z.manifest.gpg
-rw-r--r-- 1 user-data root 3019667 Jun 12 03:37 duplicity-inc.20230610T013527Z.to.20230612T013625Z.vol1.difftar.gpg
-rw-r--r-- 1 user-data root 1620 Jun 13 03:36 duplicity-inc.20230612T013625Z.to.20230613T013539Z.manifest.gpg
-rw-r--r-- 1 user-data root 2888350 Jun 13 03:36 duplicity-inc.20230612T013625Z.to.20230613T013539Z.vol1.difftar.gpg
-rw-r--r-- 1 user-data root 304338 Mai 26 03:36 duplicity-new-signatures.20230525T013538Z.to.20230526T013531Z.sigtar.gpg
-rw-r--r-- 1 user-data root 258947 Mai 27 03:35 duplicity-new-signatures.20230526T013531Z.to.20230527T013527Z.sigtar.gpg
-rw-r--r-- 1 user-data root 229514 Mai 28 03:36 duplicity-new-signatures.20230527T013527Z.to.20230528T013530Z.sigtar.gpg
-rw-r--r-- 1 user-data root 206631 Mai 29 03:37 duplicity-new-signatures.20230528T013530Z.to.20230529T013643Z.sigtar.gpg
-rw-r--r-- 1 user-data root 226929 Mai 30 03:35 duplicity-new-signatures.20230529T013643Z.to.20230530T013517Z.sigtar.gpg
-rw-r--r-- 1 user-data root 303090 Mai 31 03:36 duplicity-new-signatures.20230530T013517Z.to.20230531T013530Z.sigtar.gpg
-rw-r--r-- 1 user-data root 283133 Jun 1 03:36 duplicity-new-signatures.20230531T013530Z.to.20230601T013529Z.sigtar.gpg
-rw-r--r-- 1 user-data root 304139 Jun 2 03:36 duplicity-new-signatures.20230601T013529Z.to.20230602T013529Z.sigtar.gpg
-rw-r--r-- 1 user-data root 279399 Jun 3 03:35 duplicity-new-signatures.20230602T013529Z.to.20230603T013526Z.sigtar.gpg
-rw-r--r-- 1 user-data root 228529 Jun 4 03:35 duplicity-new-signatures.20230603T013526Z.to.20230604T013527Z.sigtar.gpg
-rw-r--r-- 1 user-data root 224628 Jun 5 03:36 duplicity-new-signatures.20230604T013527Z.to.20230605T013618Z.sigtar.gpg
-rw-r--r-- 1 user-data root 261822 Jun 6 03:35 duplicity-new-signatures.20230605T013618Z.to.20230606T013524Z.sigtar.gpg
-rw-r--r-- 1 user-data root 230758 Jun 7 03:36 duplicity-new-signatures.20230606T013524Z.to.20230607T013543Z.sigtar.gpg
-rw-r--r-- 1 user-data root 297534 Jun 8 03:36 duplicity-new-signatures.20230607T013543Z.to.20230608T013527Z.sigtar.gpg
-rw-r--r-- 1 user-data root 241565 Jun 9 03:36 duplicity-new-signatures.20230608T013527Z.to.20230609T013528Z.sigtar.gpg
-rw-r--r-- 1 user-data root 325141 Jun 10 03:36 duplicity-new-signatures.20230609T013528Z.to.20230610T013527Z.sigtar.gpg
-rw-r--r-- 1 user-data root 258952 Jun 12 03:37 duplicity-new-signatures.20230610T013527Z.to.20230612T013625Z.sigtar.gpg
-rw-r--r-- 1 user-data root 256586 Jun 13 03:36 duplicity-new-signatures.20230612T013625Z.to.20230613T013539Z.sigtar.gpg
And the secret key is in the directory above: /home/user-data/backup
lion@box:/home/user-data/backup$ ls -la
total 44
drwxr-xr-x 4 root root 4096 Jan 17 17:11 .
drwxr-xr-x 9 user-data user-data 4096 Jan 7 18:11 ..
drwxr-xr-x 3 root root 4096 Jan 8 04:24 cache
drwxr-xr-x 2 user-data root 20480 Jun 13 03:36 encrypted
-rw------- 1 root root 2775 Jan 17 17:31 secret_key.txt
-rw------- 1 root root 2775 Jan 17 17:09 secret_key.txt.new
-rw------- 1 root root 2775 Jan 17 17:11 secret_key.txt.old
Well, if I understand correctly, I need to back up all of these files somewhere, e.g. on an external storage. OK? Really all or the latest only?
If I have now set up a new box with the reliable SSD under Ubuntu 22.04 and MiaB v62 all with the same credentials, hopefully also with the same IP address, then I save the previous backup data in the /root/miab-recover/backup directory. Since I didn’t make a tarball, I don’t need to use the untar command either. So far so good?
Now, which secret key should I put into an environment variable for duplicity to use later (the one previously stored with the backup)?
-rw------- 1 root root 2775 Jan 17 17:31 secret_key.txt
export PASSPHRASE=$(cat /root/miab-recover/backup/secret_key.txt)
Then the final shot:
sudo -E duplicity restore --force file:///root/miab-recover/backup/encrypted/ /home/user-data/
Did I do something wrong or do I get a placet (lat. like it) from you? Thanks a lot in advance!
Another question: Are all settings, e.g. all three domains, all mailboxes (there are quite a few), DNSSEC keys, etc. adopted after the restore?
First off, there should only be one secret_key.txt file in backup. It is concerning that you also have the .old and .new files. As far as I know, duplicity will encrypt with the secret_key.txt file when it does the backup, and that file is needed to do the restore. The file should never change during the life of your mail sever, or if it does, then you probably want to just delete the entire backup directory and let duplicity do a new backup from scratch. I believe it will just create a new secret_key.txt file when it does this.
duplicity is actually rather sophisticated in that it is able to do full and incremental backups when it runs each night, and all the files in the encrypted/ directory constitute duplicity’s data set that it manages when doing backups. So, you need the entire directory to do a restore. So, don’t worry about how many files are in there and which ones you may need. You need everything in there, because the directory is a black box managed by duplicity.
At this point I’m not sure if you still have a running system that is reliable and if the data on your mail server is corrupted or not, because if your system is running and the data is “sound”, I would recommend (like in the upgrade docs) that you start by manually doing one more backup, then stopping you mail server’s services. Then tarball the backup directory (I suggest tarball because it preserves the entire directory structure, ownership, permissions, links, etc) and you only have to worry about copying one file around from system to system, but that’s just my preference.
Also, you can always experiment with restoring the backup outside of the area of the actual user-data directory to better understand how it works before applying it to the final destination. For example, I tested extracting my backup tarball and restoring the backup with duplicity on my linux workstation days before I actually began my 18.04 → 22.04 upgrade, just to make sure I understood how it works to minimize downtime.
One more thing, yes, the backup restores everything related to the email. Users, mail boxes, domains, certificates, etc. I can’t speak to DNSSEC because at the time I did the restore I was using external DNS and did not have DNSSEC enabled. But everything else related to email and such was exactly the way it was when I ran the last backup.
Hope this info helps.
FYI, here’s the command I run nightly to get backups from my mail server (at least the salient part):
ssh root@mail.example.com 'tar c -C / -z -f - home/user-data/' > $HOME/mail.example.com/tarballs/home-backup-`date -Iseconds -u`.tgz
This command depends on ssh keys being loaded to allow root access to the box. You can always just use a normal user and add sudo to the tar command. If you put that into a cron job, try to run it after the mail server’s nightly cron has done the backups.
You wrote:
It is concerning that you also have the
.oldand.newfiles.
I wrote above:
During the upgrade procedure (from 18.04 to 22.04) I tried to follow the procedure that was documented for upgrading from Ubuntu 18.04 to 22.04 (which also included restoring the new system from the backup). But that didn’t work. Because with SCP I got the backup data (/home/user-data/backup/encrypted) from MiaB-v57a with the secret_key.txt over to the freshly installed MiaB v60.1 also in the same directory /home/user-data /backup/encrypted. When restoring I took the secret_key.txt from MiaB v57a. The command was as follows: “sudo -E duplicity restore --force file:///home/user-data/backup/encrypted /home/user-data/”. The result showed …
above failed as I said. I’m not sure which key is the current. But I guess the one which named secret_key.txt, otherwise the backups could not have taken place 100% every day since May 25, 2023. Please let me assume it is so, otherwise I would have to delete the entire backup directory as stated in the following cite (should I?):
The file should never change during the life of your mail sever, or if it does, then you probably want to just delete the entire backup directory and let duplicity do a new backup from scratch. I believe it will just create a new secret_key.txt file when it does this.
What shall I do now?
I’m a bit confused because as you wrote, you tried to restore the backup and the result showed a GPGError and the restore failed. If that was while using the secret_key.txt file, then I wouldn’t be confident that things are okay with your backup (even if it is able to continue doing nightly backups). I’m not that familiar with duplicity.
I’m also confused about the state of your current system. Have you moved to a new disk and got everything restored to your satisfaction? Or is the system still running on a bad drive?
Here’s what I would recommend (IF you have already moved everything to a new drive, and it’s been restored, manually or otherwise, to your satisfaction):
Move the backup directory (don’t delete just in case you need it back) to another place on disk for safe keeping. For example, move it under your user’s home directory. If you do that, I believe duplicity will create a fresh, full backup with a new secret_key.txt tonight when the nightly cron runs the backup. If that works, then you could archive the old backup directory for safe keeping until you’re sure you won’t need it again. Also, you should regularly save your backups (the entire backup directory with the secret key) on another system somewhere, so if you have another disk problem and need to rebuild your system again, you’ll have a good, recent backup to use for restoring.
A few hours ago I have done the following with the defective HDD:
Perform one last backup on the old machine.
And a few minutes ago the trial
curl -s https://mailinabox.email/bootstrap.sh | sudo bash
has successfully passed.
Am I now able to copy today’s last backup to the new /root/miab-recover/backup directory and successfully start the restore from there? Wish me luck!
It’s worked out! I don’t know how to say thank you for the help I’m getting in this user forum? Everything looks like yesterday on the defective HDD. The restore worked:
Summary: 56 ✓ OK, 1
Error, 3 ? Warning
1
3 ? Warnings (3 domains): Web has been disabled for this domain because you have set a custom DNS record.
BTW: I’ve already tried several times to make it clear to the ISP provider that they should correct reverse DNS for me. But they have deaf ears and don’t even know what that is!!! Nevertheless, the mail server has been working without reverse DNS for years!
But at least all three DNSSEC ‘DS’ records (3 domains) were correctly transferred during this migration (when restoring).
So far, thanks to everyone, especially @dms for being patient with an “old” man.
Great to hear. I’m glad it’s worked out. Usually a VPS provider has a way for you to setup the reverse DNS. May I ask who your provider is? My provider is linode and the UI for my VPS systems has a way for me to setup the reverse DNS myself.
This topic was automatically closed 40 days after the last reply. New replies are no longer allowed.