Backup script failing

YakShaver · September 5, 2023, 9:48pm

I reverted my manual changes by doing a git reset --hard in /root/mailinabox and then curl -s https://mailinabox.email/setup.sh | sudo bash successfully upgraded my installation to v64.

amazzi44 · September 5, 2023, 10:48pm

Unfortunately, neither a restart nor trying to force duplicity to upgrade worked.

In particular, I still got the following error (which is the one I originally saw on today’s overnight email):

Error processing remote manifest (duplicity-inc.20230903T012907Z.to.20230904T012926Z.manifest.gpg): GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: Note: '--decrypt' is not considered an option
gpg: WARNING: no command supplied.  Trying to guess what you mean ...
usage: gpg [options] [filename]
===== End GnuPG log =====

GPGError: GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: Note: '--force-mdc' is not considered an option
gpg: Note: '--symmetric' is not considered an option
gpg: WARNING: no command supplied.  Trying to guess what you mean ...
usage: gpg [options] [filename]
===== End GnuPG log =====

Traceback (most recent call last):
  File "/root/mailinabox/management/./backup.py", line 647, in <module>
    perform_backup(full_backup)
  File "/root/mailinabox/management/./backup.py", line 318, in perform_backup
    shell('check_call', [
  File "/root/mailinabox/management/utils.py", line 126, in shell
    ret = getattr(subprocess, method)(cmd_args, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/duplicity', 'incr', '--verbosity', 'warning', '--no-print-statistics', '--archive-dir', '/home/user-data/backup/cache', '--exclude', '/home/user-data/backup', '--volsize', '250', '--gpg-options', "'--cipher-algo=AES256'", '--allow-source-mismatch', '/home/user-data', 'file:///home/user-data/backup/encrypted']' returned non-zero exit status 31.

and this is the result of ls -al /etc/apt/sources.list.d/:

total 16
drwxr-xr-x 2 root root 4096 Jun 18 13:02 .
drwxr-xr-x 8 root root 4096 Aug 10  2022 ..
-rw-r--r-- 1 root root  192 Sep  6 00:28 duplicity-team-ubuntu-duplicity-release-git-jammy.list
-rw-r--r-- 1 root root  140 Sep  6 00:28 ondrej-ubuntu-php-jammy.list

but still no upgrades available (could the fact that I am on arm64 be an issue?).

What did work was replacing both the root and the non-root backup.py files with the old one (the one without the single quotes): than I ran the backup.py as root and it completed without issues.

On the Backup Status web page I never saw an error - only today’s backup missing.

However, I don’t consider it a robust solution.

SHAdmin · September 6, 2023, 12:09am

That worked very well. Thanks much. For other readers, the particular commands run as the root user were:

cd /root/mailinabox
git reset --hard
curl -s https://mailinabox.email/setup.sh | sudo bash

mokiejovis · September 6, 2023, 2:48am

You need to get the latest duplicity installed before you can expect the v64 backup.py to work. This should be what you focus on to resolve the problem because in the long-term you’re going to want to easily upgrade MIAB and simply use whatever the latest release of duplicity is available.

Without more troubleshooting, it’s difficult to figure out exactly what is going wrong with getting the latest version installed. Maybe you have it version-locked, maybe something is up with the PPA config somehow, it could be anything. It doesn’t appear to be your arm64 chip - that’s a supported release from the PPA.

britannic · September 11, 2023, 2:06pm

@amazzi44, I have the exact same error. Here’s what I already tried:

Upgraded to v64
Removed the duplicity package (full purge)
Reboot
Installed duplicity 2.1.1-ppa202309021553~ubuntu22.04.1
Reboot
Backup still fails
Downgraded duplicity to 0.8.21-1build1
Reboot
Back up still fails
Upgraded duplicity to 2.1.1-ppa202309021553~ubuntu22.04.1
Confirmed backup.py matches latest

Not sure what else to try - last successful backup was 8 days ago.

barry · September 19, 2023, 2:19pm

I’m in the same boat and tried the same actions. Is it time to wipe the machine and restore from backup?

Barry

darond · September 19, 2023, 4:47pm

For me I had the same error after upgrading until I re-ran the install scripts which appeared to force the change and all good from that point forwards for me:

curl -s https://mailinabox.email/setup.sh | sudo bash

Since the install is idempotent there’s no harm in giving this a go.

britannic · September 19, 2023, 5:18pm

Unfortunately, that didn’t work for me. I also completely removed the ~/mailinabox directory before running the install.

barry · September 23, 2023, 8:53pm

I solved the backup issue by moving to a new box. I can only assume I screwed up the box when I installed a local copy of duplicity. The only trouble was I obvs have no good backup. I ended up moving the user-data directory to the new instance and everything just worked…

Thanks mailinabox!

xenophonf · October 7, 2023, 9:46pm

If, like me, you’re deploying Mail-in-a-Box v64 on Ubuntu 22.04/arm64 because t4g EC2 instances are cheap, note that the duplicity-team PPA doesn’t build binary packages for you. Instead, build Duplicity from source:

sudo sed -i'' -e 's/^# *//' /etc/apt/sources.list.d/duplicity-team-ubuntu-duplicity-release-git-jammy.list
sudo apt-get update
sudo apt-get build-dep duplicity
apt-get --build source duplicity
sudo apt-get install lftp
sudo dpkg -i duplicity*.deb

To confirm, note the empty contents of https://ppa.launchpadcontent.net/duplicity-team/duplicity-release-git/ubuntu/dists/jammy/main/binary-arm64/Packages.gz and the output of apt-cache madison duplicity, the latter of which will output something like:

 duplicity | 0.8.21-1build1 | http://us-east-1.ec2.ports.ubuntu.com/ubuntu-ports jammy/main arm64 Packages

See also: https://www.cyberciti.biz/faq/how-to-get-source-code-of-package-using-the-apt-command-on-debian-or-ubuntu/

And for reference, this is the error I’m getting from the nightly backup script:

GPGError: GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: Note: '--force-mdc' is not considered an option
gpg: Note: '--symmetric' is not considered an option
gpg: WARNING: no command supplied.  Trying to guess what you mean ...
usage: gpg [options] [filename]
===== End GnuPG log =====

Traceback (most recent call last):
  File "/home/ubuntu/mailinabox/management/backup.py", line 647, in <module>
    perform_backup(full_backup)
  File "/home/ubuntu/mailinabox/management/backup.py", line 318, in perform_backup
    shell('check_call', [
  File "/home/ubuntu/mailinabox/management/utils.py", line 126, in shell
    ret = getattr(subprocess, method)(cmd_args, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/duplicity', 'full', '--verbosity', 'warning', '--no-print-statistics', '--archive-dir', '/home/user-data/backup/cache', '--exclude', '/home/user-data/backup', '--volsize', '250', '--gpg-options', "'--cipher-algo=AES256'", '--allow-source-mismatch', '/home/user-data', 'file:///home/user-data/backup/encrypted']' returned non-zero exit status 31.

0zul · October 8, 2023, 11:17pm

Can you explain a bit more since I’m experiencing the same issue just as you are.

xenophonf · October 9, 2023, 2:38am

Per the above thread, Mail-in-a-Box v64 requires Duplicity version 2 or newer. The Mail-in-a-Box setup script adds the Duplicity team’s personal package archive (PPA), which includes binary packages for Duplicity 2.1.3. Unfortunately, they do not build arm64 binaries, which you can verify as described above. Furthermore, the Mail-in-a-Box setup script does not check the version of Duplicity that gets installed, so what happens is that a much older version of Duplicity gets installed from the official Ubuntu package repositories instead of the newest version from the PPA as intended, and no error is raised during the Mail-in-a-Box setup. Specifically, Duplicity 0.8.21 on the arm64 version of Ubuntu 22.04 isn’t compatible with Mail-in-a-Box v64, hence the error message GPGError: GPG Failed even when APT claims that everything is up to date and Mail-in-a-Box successfully completes installation.

Copy and paste the six commands I specified to enable the duplicity-team PPA’s source repo, download the latest source for the Duplicity package, install any build-time prerequisites, build the binary package for your server, install any run-time prerequisites manually (since you aren’t running APT), and install the binary package you built directly using dpkg.

0zul · October 9, 2023, 7:55pm

I was able to upgrade to latest duplicity version successfully thanks to you. But now I’m getting an error related to --gpg-options: expected one argument. When I try to manually trigger the backup to s3

britannic · October 10, 2023, 12:21am

I’m running MIAB v64 on an AMD VM and now get this error after upgrading duplicity to 2.1.3 (I also tried compiling duplicity source, just for fun, but no difference):

root@box:~# /root/mailinabox/management/backup.py --full
GPGError: GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: Note: '--force-mdc' is not considered an option
gpg: Note: '--symmetric' is not considered an option
gpg: WARNING: no command supplied.  Trying to guess what you mean ...
usage: gpg [options] [filename]
===== End GnuPG log =====

Traceback (most recent call last):
  File "/root/mailinabox/management/backup.py", line 647, in <module>
    perform_backup(full_backup)
  File "/root/mailinabox/management/backup.py", line 318, in perform_backup
    shell('check_call', [
  File "/root/mailinabox/management/utils.py", line 126, in shell
    ret = getattr(subprocess, method)(cmd_args, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/duplicity', 'full', '--verbosity', 'warning', '--no-print-statistics', '--archive-dir', '/home/user-data/backup/cache', '--exclude', '/home/user-data/backup', '--volsize', '250', '--gpg-options', "'--cipher-algo=AES256'", '--allow-source-mismatch', '/home/user-data', 'file:///home/user-data/backup/encrypted']' returned non-zero exit status 31.

0zul · October 10, 2023, 5:29am

I got the local backup working when it automatically makes the backup. But when I try to trigger the backup, then I am getting this error.

ubuntu@mail:~$ sudo mailinabox/management/backup.py --full
CommandLineError: argument --gpg-options: expected one argument
Traceback (most recent call last):
  File "/home/ubuntu/mailinabox/management/backup.py", line 595, in <module>
    perform_backup(full_backup)
  File "/home/ubuntu/mailinabox/management/backup.py", line 299, in perform_backup
    shell('check_call', [
  File "/home/ubuntu/mailinabox/management/utils.py", line 123, in shell
    ret = getattr(subprocess, method)(cmd_args, **kwargs)
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/duplicity', 'full', '--verbosity', 'warning', '--no-print-statistics', '--archive-dir', '/home/user-data/backup/cache', '--exclude', '/home/user-data/backup', '--volsize', '250', '--gpg-options', '--cipher-algo=AES256', '/home/user-data', 's3://xxxx/backups', '--allow-source-mismatch', '--s3-endpoint-url', 'https://xxx.digitaloceanspaces.com']' returned non-zero exit status 23.

So I don’t know what might be the issue since it fixed for me.

jusoaresf · October 10, 2023, 5:58am

I believe it would work had you run /root/mailinabox/management/backup.py.

I am not sure whether /home/user/mailinabox/ should still be there or not.

tyler · October 11, 2023, 4:49pm

I am experiencing issues with the backup script as well. t3a (AMD) instance so no ARM stuff to worry about. Ubuntu 22.04 and the latest MIAB.

Issue: My nightly backup job emails me with a body that reads: ‘Contents’.

Manually running the backup script produces:

ubuntu@mail01:~/mailinabox/management$ sudo ./backup.py
CommandLineError: argument --gpg-options: expected one argument
Something is wrong with the backup: ERROR 23 CommandLineError
. CommandLineError: argument --gpg-options: expected one argument


ubuntu@mail01:~/mailinabox/management$ sudo ./backup.py --full
Failed to stop php7.2-fpm.service: Unit php7.2-fpm.service not loaded.

ubuntu@mail01:~/mailinabox/management$

The above is interesting, since PHP is now version 8.2.11.

This is not a huge deal for me, as I take full machine backups nightly using AWS Backup. I would still like this to work though.

Duplicity info:

ubuntu@mail01:~$ which duplicity
/usr/bin/duplicity
ubuntu@mail01:~$ /usr/bin/duplicity --version
duplicity 2.1.3

EDIT: ok, I manually modified ./backup.py and commented out the php7.2 service_command lines, and replaced them with the equivalent php8.2 service_command commands. This is my new error:

ubuntu@mail01:~/mailinabox/management$ sudo ./backup.py --full
CommandLineError: argument --gpg-options: expected one argument

britannic · October 13, 2023, 1:03am

So I already have python3-future and duplicity 2.1.3 installed on my MIAB v64 VM:

pip3 install future Requirement already satisfied: future in /usr/local/lib/python3.10/dist-packages (0.18.3)

root@box:~# apt list duplicity Listing... Done duplicity/jammy,now 2.1.3-ppa202310081632~ubuntu22.04.1 amd64 [installed]

Manually running the generated command line from the backup.py error works just fine:

Extracted command line from this error:
subprocess.CalledProcessError: Command '['/usr/bin/duplicity', 'incr', '--verbosity', 'warning', '--no-print-statistics', '--archive-dir', '/home/user-data/backup/cache', '--exclude', '/home/user-data/backup', '--volsize', '250', '--gpg-options', "'--cipher-algo=AES256'", '--allow-source-mismatch', '/home/user-data', 'file:///home/user-data/backup/encrypted']' returned non-zero exit status 31.

Extracted command line (works without error):
root@box:~# /usr/bin/duplicity incr --verbosity warning --no-print-statistics --archive-dir /home/user-data/backup/cache --exclude /home/user-data/backup --volsize 250 --gpg-options "--cipher-algo=AES256" --allow-source-mismatch /home/user-data file:///home/user-data/backup/encrypted GnuPG passphrase for decryption: Retype passphrase for decryption to confirm:

Running backup.py still results in this error:
root@box:/home/user-data/backup/cache# /root/mailinabox/management/backup.py Error processing remote manifest (duplicity-full.20231013T004920Z.manifest.gpg): GPG Failed, see log below: ===== Begin GnuPG log ===== gpg: Note: '--decrypt' is not considered an option gpg: WARNING: no command supplied. Trying to guess what you mean ... usage: gpg [options] [filename] ===== End GnuPG log =====

IMHO it seems to be an issue with the backup.py generated command line.

0zul · October 13, 2023, 11:55am

Refer this for help Backup script failing - #77 by jusoaresf

SHAdmin · October 14, 2023, 9:34pm

The python3-future package should have been removed by an Ubuntu system update. Our system reported:

The following package was automatically installed and is no longer required:
python3-future
Use ‘apt autoremove’ to remove it.

For some reason, on our system, running “apt autoremove” and “apt-get autoremove” did not remove python3-future, so I had to run “apt purge python3-future” to successfully remove that obsolete package. The following is the pertinent portion of the terminal shell session which on our system was run when duplicity 2.1.2 was the latest version:

root@mail:~# apt purge python3-future
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
The following additional packages will be installed:
duplicity
Suggested packages:
ncftp python3-boto3 python3-kerberos
The following packages will be REMOVED:
python3-future*
The following packages will be upgraded:
duplicity
1 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
Need to get 313 kB of archives.
After this operation, 1609 kB disk space will be freed.
Do you want to continue? [Y/n] Y
Get:1 Index of /duplicity-team/duplicity-release-git/ubuntu jammy/main amd64 duplicity amd64 2.1.2-ppa202309271547~ubuntu22.04.1 [313 kB]
Fetched 313 kB in 1s (476 kB/s)
(Reading database … 115182 files and directories currently installed.)
Preparing to unpack …/duplicity_2.1.2-ppa202309271547~ubuntu22.04.1_amd64.deb …
Unpacking duplicity (2.1.2-ppa202309271547~ubuntu22.04.1) over (2.1.1-ppa202309021553~ubuntu22.04.1) …
(Reading database … 115182 files and directories currently installed.)
Removing python3-future (0.18.2-5ubuntu0.1) …
Setting up duplicity (2.1.2-ppa202309271547~ubuntu22.04.1) …
Processing triggers for man-db (2.10.2-1) …

Notice that duplicity was automatically upgraded by running “apt purge python3-future” which fixed all problems on our system.