AWS S3 Policy blocking duplicity restore? - SOLVED

Maintenance
1

Just testing the duplicity restore process - in this case to a new, unused MIAB on 22.04. Backup is to an AWS S3 bucket.

Initial backup and incremental backups work fine from the box and I can see files written in the bucket and the backup status page on the MIAB is updated accordingly.

So at this point I know the bucket is working and my access to that bucket works (S3 Access Key/S3 Secret Access Key).

Now, coming to testing the restore:

export AWS_ACCESS_KEY_ID=my access key same as used in backup configuration
export AWS_SECRET_ACCESS_KEY=my access key same as used in backup configuration
export PASSPHRASE=$(cat /home/user-data/backup/secret_key.txt)
sudo -E duplicity restore --force s3://amazonregionusedinbackupconfig.amazonaws.com/bucketname/folder /home/user-data/

I get:

Attempt of list Nr. 1 failed. ClientError: An error occurred (403) when calling the HeadBucket operation: Forbidden
Attempt of list Nr. 2 failed. AttributeError: 'NoneType' object has no attribute 'objects'
Attempt of list Nr. 3 failed. AttributeError: 'NoneType' object has no attribute 'objects'
Attempt of list Nr. 4 failed. AttributeError: 'NoneType' object has no attribute 'objects'
Giving up after 5 attempts. AttributeError: 'NoneType' object has no attribute 'objects'

Which, given the ‘403’ suggests either permission issues or the objects do not exist.

Given I can backup (so my access keys must be correct) my assumption is that it’s permissions related and I must be missing an action in the S3 policy I have applied:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketPublicAccessBlock",
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::bucketname/*",
                "arn:aws:s3:::bucketname",
            ]
        }
    ]
}

But even applying a blanket S3 policy:

            "Action": [
                "s3:*"
            ],

Has no impact.

The policy is applied to a group and the user (with the access keys) is a member of the group.

So I can write to the bucket using duplicity when backing up but appear to be unable to read from the bucket for a restore.

Any suggestions would be welcome.

1 Like
2

So after two days of failing to resolve and amending policies left right and centre I’ve identified the issue was down to the use of the URI recommended in the duplicity restore statement here:

sudo -E duplicity restore --force s3://s3.amazonaws.com/your-bucket-name/your-backup-path /home/user-data/

It was only by accident by reviewing the properties for the ‘Object’ in the AWS console i.e. /your-bucket-name/your-backup-path that I noticed the URI was showing as s3://your-bucket-name/your-backup-path not the full s3://s3.region.amazonaws.com/…

So using the Uri specified for the object:

sudo -E duplicity restore --force s3://your-bucket-name/your-backup-path /home/user-data/

It worked straight away.

So in conclusion the 403 wasn’t permissions related it was that the object was inaccessible as the URI was malformed for my bucket.

Leaving my learning here in case it helps anyone else from tearing their hair out

:grinning:

3 Likes
3

Thank you so much for posting the updated S3 URI. I ran into the same issues attempting to restore from S3 bucket. I followed your suggested URI and was able to successfully restore.

1 Like
4

LEGEND

I have been pulling my hair out and had done some searches. After giving up and rolling back to the old machine and enabling a free security maintenance from loosely following this article:

I found your article.

I have been at it for 6 hours and will take a break. I have the other state ready to go and will go for a restore tomorrow when the family is asleep again.

1 Like
5

It definitely could have been better explained in the instructions.

6

This topic was automatically closed 40 days after the last reply. New replies are no longer allowed.