Cannot Backup since Duplicity update

dfowlkes · May 17, 2022, 3:11am

I got the following backup status email:

------
Giving up after 5 attempts. ModuleNotFoundError: No module named 'boto3'
Something is wrong with the backup: Attempt of list Nr. 1 failed. ModuleNotFoundError: No module named 'boto3'
WARNING 1
. Attempt of list Nr. 1 failed. ModuleNotFoundError: No module named 'boto3'

Attempt of list Nr. 2 failed. ModuleNotFoundError: No module named 'boto3'
WARNING 1
. Attempt of list Nr. 2 failed. ModuleNotFoundError: No module named 'boto3'
Attempt of list Nr. 3 failed. ModuleNotFoundError: No module named 'boto3'
WARNING 1
. Attempt of list Nr. 3 failed. ModuleNotFoundError: No module named 'boto3'
Attempt of list Nr. 4 failed. ModuleNotFoundError: No module named 'boto3'
WARNING 1
. Attempt of list Nr. 4 failed. ModuleNotFoundError: No module named 'boto3'
ERROR 50 list
. Giving up after 5 attempts. ModuleNotFoundError: No module named 'boto3'
----------------------

When I manually run from SSH:
cd mailinabox
sudo management/backup.py
I get the same errors.
Ubuntu pushed out an update for Duplicity last week and it was installed on my Ubuntu 18.04 instance:

--------------------------
The following packages will be upgraded:
  duplicity
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 330 kB of archives.
After this operation, 4096 B of additional disk space will be used.
Get:1 http://ppa.launchpad.net/duplicity-team/duplicity-release-git/ubuntu bionic/main amd64 duplicity amd64 0.8.23-ppa202205151528~ubuntu18.04.1 [330 kB]
Fetched 330 kB in 1s (542 kB/s)
(Reading database ... 134021 files and directories currently installed.)
Preparing to unpack .../duplicity_0.8.23-ppa202205151528~ubuntu18.04.1_amd64.deb ...
Unpacking duplicity (0.8.23-ppa202205151528~ubuntu18.04.1) over (0.8.22-ppa202203041847~ubuntu18.04.1) ...
Setting up duplicity (0.8.23-ppa202205151528~ubuntu18.04.1) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
----------------------------------------------

On several cloned instances of this mail server, I have tried:

Remove and re-install Duplicity
Install Boto3 using: sudo python3 -m pip install boto3 or
sudo -H pip install boto3
or
sudo apt install boto3
or
python3 -m pip install boto3

My understanding is that the boto3 module is only used for transfers to Amazon AWS S3. Thus other backup methods should continue to function.

I have one question:
Does anyone have a solution for this?
Note: I changed the backup method from the admin page to backups on the machine running MIAB. I then successfully ran the backup.py script and confirmed full backups were in /home/user-data/backup/encrypted

In the meantime, I will make daily snapshots of the instance. All help is appreciated.

ge8Hooe · May 17, 2022, 12:00pm

Hi, I ran in to same problem, same error messages etc…

Here is what I did to revert back to previous version of duplicity:

cd /tmp
sudo apt-get remove duplicity
wget 'https://launchpad.net/~duplicity-team/+archive/ubuntu/duplicity-release-git/+files/duplicity_0.8.22-ppa202203041847~ubuntu18.04.1_amd64.deb'
sudo apt install ./duplicity_0.8.22-ppa202203041847~ubuntu18.04.1_amd64.deb 
sudo apt-mark hold duplicity
rm duplicity_0.8.22-ppa202203041847~ubuntu18.04.1_amd64.deb

I’ve explained somewhat how I ended up at the solution, that should work.

How to rollback unfortunately this didn’t work in this case, as the previous version has been removed from repository, hence the wget above.
How to install specific version this didn’t work, as the version requested does not exist in the repository.
Howto keep a package in current version.

Here is the link to i386 version of duplicity version 0.8.22-ppa202203041847~ubuntu18.04.1, just in case someone is running MIAB on i386 instead of amd64.

Edit:
I ran “sudo management/backup.py” which did not show any error or other messages and return value was also zero.

dfowlkes · May 18, 2022, 2:47am

Many thanks! You instructions were great. I only needed to add “sudo” for the wget line.
I manually ran management/backup.py and S3 had a full backup in minutes.

Cheers,
Dana

bmach · May 19, 2022, 10:22pm

Solution worked perfectly for me - my backups are back up and working. Thank you very much for sharing.

I’m new to MIAB so I’m just trying to work out what issues this tactical fix could do down the track.

I’m assuming the permanent fix for this will be a new version of MIAB - specifically an updated ~/mailinabox/management/backup.py that works with the latest version of duplicity. When that happens would the steps be that anyone who has applied this tactical fix would need to remove the hold, update to the latest version of duplicity then upgrade MIAB?

sudo apt-mark unhold duplicity
sudo apt-get update
sudo apt-get upgrade

ge8Hooe · May 20, 2022, 8:34am

The duplicity is not installed from any of the repositories maintained by Ubuntu.
It is installed directly from a repo maintained by Duplicity developers, which is defined in file

/etc/apt/sources.list.d/duplicity-team-ubuntu-duplicity-release-git-bionic.list

file in my box.

AFAIK yes the “unhold” will enable update of duplicity and if boto3 dependency is not fixed, again AFAIK, in MIAB code base, duplicity will not work.

makeitgo · June 5, 2022, 2:23pm

This doesn’t work as of at least today – the older deb for 0.8.22 is completely gone from the archives now.

Is this a duplicity team problem, I guess? Should we be filing bug reports with them to “unbreak” duplicity on Ubuntu 18.04?

pupsncats · June 5, 2022, 7:37pm

It’s not working for me either. Been working on this problem for the past four hours.

bmach · June 5, 2022, 10:18pm

It looks like duplicity has moved from launchpad.net to here: duplicity / duplicity · GitLab

Might require some digging but maybe you could find the DEB binaries there and update the WGET above to the new URL.

makeitgo · June 6, 2022, 1:43pm

I poked around a bit at that new duplicity site, but didn’t find any built packages there. But I admit I didn’t try all --that-- hard, I will probably just run without backups for a while and hope the 22.04 miab upgrade comes before I encounter a catastrophic crash, and that the 22.04 upgrade fixes things, of course.

ion · June 6, 2022, 3:20pm

/usr/local/lib/python3.6/dist-packages/boto3/compat.py:88: PythonDeprecationWarning: Boto3 will no longer support Python 3.6 starting May 30, 2022. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.7 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/
  warnings.warn(warning, PythonDeprecationWarning)

I’m having this issue as well when I was trying to back up to S3

Based on aws sites
On May 30, 2022, the AWS SDK for Python (Boto3 and Botocore) and the AWS Command Line Interface (AWS CLI) v1 will no longer support Python 3.6.

End of support for Python 3.6.9 for AWS SDK.

pupsncats · June 6, 2022, 4:44pm

Exactly. For now, what I did was just moved back to local backups. I installed Python3.7 on my box. Created a simple python script that’s triggered by a cron that will move the local backups over to AWS S3 until I have more time to find a solution.

Update:
I rewrote the backup.py script to work with Python3.7. I only had time to test it twice and seems to be working. I’ll work on it more tonight. If I feel that’s working, I’ll provide the script. Hopefully, when the next MIAB version comes out it will fix the problem and just overwrite the backup.py file.

dbogatov · June 10, 2022, 8:36pm

This worked for me:

sudo apt-get remove duplicity
sudo apt-get install duplicity=0.7.17-0ubuntu1
sudo apt-mark hold duplicity

JoshData · June 10, 2022, 8:56pm

Ah good idea!

I’d like to include something in the next release of Mail-in-a-Box to fix this. I use S3 backups so I’m facing this issue also.

I’m concerned though that this rollback might have unintended consequences since it’s going back a major version. We were probably all using 0.8.21-ppa202111091602~ubuntu1 from the duplicity PPA until June 5, which is when my box automatically updated to 0.8.23-ppa202205151528~ubuntu18.04.1.

We switched to the duplicity PPA in Implement Backblaze for Backup by hija · Pull Request #1812 · mail-in-a-box/mailinabox · GitHub with adding support for Backblaze backups. I assume that took us from a 0.7 version to an 0.8 version but I haven’t checked.

Hopefully someone has some ideas for how we can implement a fix that works for everyone.

thedesi · June 11, 2022, 1:27pm

Thanks for the tip, watching this space to see if we have more elegant way to fix this long term.

JoshData · June 11, 2022, 7:10pm

I think I’ve figured out how to get things working with the latest duplicity. I’ll post more in S3 Backups failing with `ModuleNotFoundError: No module named 'boto3'` · Issue #2123 · mail-in-a-box/mailinabox · GitHub when it’s ready.

kevaug · June 13, 2022, 6:08am

Can confirm this fixed the “boto3” s3 backup issue with my install. Thanks @JoshData for the quick fix. No noticeable issues after the update FWIW.

bmach · June 13, 2022, 10:51am

Same here. Version 57 works perfectly for me for S3 compatible backups to Wasabi. Thanks so much @JoshData.

In case it helps others I did have the tactical downgrade fix in place prior to upgrade. So I made sure I did the unhold and upgrade prior to MIAB upgrade to 57.

sudo apt-mark unhold duplicity
sudo apt-get update
sudo apt-get upgrade
curl -s https://mailinabox.email/setup.sh | sudo bash

curtiser · June 13, 2022, 12:47pm

I unmarked duplicity, then did an update, and upgrade duplicity.

I upgraded MiaB to v57.

I went to the admin and did a system check. It reported that I have more than one backup cache.

I went to admin / Backup Status. I got this message:

#### Backup Error
not enough values to unpack (expected 2, got 1)

Did it loose the backup configuration settings because the values displayed on the page is completely incorrect?

Did I set something wrong with the MiaB?

I’m using Wasabi which is a S3 service.

(Kudos to @JoshData for working on a fix!)

makeitgo · June 13, 2022, 12:56pm

To fix the “not enough values to unpack” backup error, I edited mailinabox/management/backup.py and changed line 211 from:

                target[1], target[2] = target[2].lstrip('/').split('/', 1)

to:

                try:
                        target[1], target[2] = target[2].lstrip('/').split('/', 1)
                except Exception as e:
                        target[1] = target[2].lstrip('/')
                        target[2] = ""

My Python’s pretty rusty, so I’m sure a proper fix will be better than my quick hack. If you make these temporary changes yourself, please note the leading whitespace are all tabs, NOT individual spaces.

After that change, running “mailinabox/management/backup.py --full” worked fine for me.

I did have to reboot to get the admin page to load up correctly. Reboot probably wasn’t necessary, but seemed safest.

PS (should’ve been first thing I wrote): Thank you @JoshData for all your work!

SHAdmin · June 13, 2022, 8:08pm

Thank you for your “quick hack.” It allowed me to temporarily work around the “not enough values to unpack” backup error.