Third party S3 and AWS SDK changes. How to fix?

1

As you may have heard, the S3 SDK has received a breaking change in recent weeks [1], this means if you are using a third party S3 service to backup MIAB you will see the error

An error occurred (MissingContentLength)

The solution according to many online posts [2] is to add the following option to either your ~/.aws/config profile or to boto3 config

request_checksum_calculation=WHEN_REQUIRED

Is there a way to make this happen in MIAB? I don’t see ~/.aws/config and I don’t know where boto3 config is located (boto3 config might be created on the fly based on values you enter in the MIAB admin backup page so maybe there is no config file at all?)

Alternatively it seems another solution would be a boto3 downgrade using pip [3] but then you would need to pin it by making a change in requirements.txt or whatever is used in your python codebase for pinning dependencies. Where would this file be located in MIAB?

[1] AWS S3 SDK breaks its compatible services

[2] e.g. amazon web services - AWS CLI copy local file to S3 cloud giving errors - Stack Overflow

[3] e.g. If you’ve upgraded boto3 or the JavaScript S3 client in the last week, uploading files won’t work. Here’s how to fix it. | Tigris Object Storage

2

For now I downgraded boto3 with

$ pip install --upgrade boto3==1.35.95

then launched a backup with

$ sudo management/backup.py

Just to be on the safe side since backup failed for a few days in a row already.

3

Hey,
To fix this, set these environment variables in get_duplicity_env_vars:

env["AWS_REQUEST_CHECKSUM_CALCULATION"] = "WHEN_REQUIRED"
env["AWS_RESPONSE_CHECKSUM_VALIDATION"] = "WHEN_REQUIRED"

This ensures checksums are only used when necessary. I’ve submitted a pull request to fix this upstream. See: Add Configurations to Handle AWS SDK Checksum Changes for Third-Party S3-Compatible Services by MrWinux · Pull Request #2490 · mail-in-a-box/mailinabox · GitHub

2 Likes
4

thank you @Winux for submitting the PR! I did see get_duplicity_env_vars and thought about changing it but didn’t because I figured that a mailinabox update would overwrite my change. Now that you made the PR the problem can be solved for good!