Unable to setup S3 Backups

goldielocks · December 12, 2022, 3:01pm

Just upgraded my server to Ubuntu 22 and version 60.1. Figured now is a good time to setup S3 backups which I’ve been meaning to do for ages.

I’ve created my S3 bucket, generated my access key with “S3 Full Access” and input all the details into mailinabox on the Backup Status screen, however when I try to save this I get an error:

<!doctype html> 500 Internal Server Error

Internal Server Error

The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

Relevant extract from syslog attached:

Dec 12 14:58:36 mail01 start[80869]: [2022-12-12 14:58:36,728] ERROR in app: Exception on /system/backup/config [POST]
Dec 12 14:58:36 mail01 start[80869]: Traceback (most recent call last):
Dec 12 14:58:36 mail01 start[80869]: File “/usr/local/lib/mailinabox/env/lib/python3.10/site-packages/flask/app.py”, line 2525, in wsgi_app
Dec 12 14:58:36 mail01 start[80869]: response = self.full_dispatch_request()
Dec 12 14:58:36 mail01 start[80869]: File “/usr/local/lib/mailinabox/env/lib/python3.10/site-packages/flask/app.py”, line 1822, in full_dispatch_request
Dec 12 14:58:36 mail01 start[80869]: rv = self.handle_user_exception(e)
Dec 12 14:58:36 mail01 start[80869]: File “/usr/local/lib/mailinabox/env/lib/python3.10/site-packages/flask/app.py”, line 1820, in full_dispatch_request
Dec 12 14:58:36 mail01 start[80869]: rv = self.dispatch_request()
Dec 12 14:58:36 mail01 start[80869]: File “/usr/local/lib/mailinabox/env/lib/python3.10/site-packages/flask/app.py”, line 1796, in dispatch_request
Dec 12 14:58:36 mail01 start[80869]: return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
Dec 12 14:58:36 mail01 start[80869]: File “/root/mailinabox/management/daemon.py”, line 75, in newview
Dec 12 14:58:36 mail01 start[80869]: return viewfunc(*args, **kwargs)
Dec 12 14:58:36 mail01 start[80869]: File “/root/mailinabox/management/daemon.py”, line 635, in backup_set_custom
Dec 12 14:58:36 mail01 start[80869]: return json_response(backup_set_custom(env,
Dec 12 14:58:36 mail01 start[80869]: File “/root/mailinabox/management/backup.py”, line 515, in backup_set_custom
Dec 12 14:58:36 mail01 start[80869]: list_target_files(config)
Dec 12 14:58:36 mail01 start[80869]: File “/root/mailinabox/management/backup.py”, line 471, in list_target_files
Dec 12 14:58:36 mail01 start[80869]: bucket_objects = s3.list_objects_v2(Bucket=bucket, Prefix=path)[‘Contents’]
Dec 12 14:58:36 mail01 start[80869]: KeyError: ‘Contents’
Dec 12 14:58:36 mail01 Exception on /system/backup/config [POST]#012Traceback (most recent call last):#012 File “/usr/local/lib/mailinabox/env/lib/python3.10/site-packages/flask/app.py”, line 2525, in wsgi_app#012 response = self.full_dispatch_request()#012 File “/usr/local/lib/mailinabox/env/lib/python3.10/site-packages/flask/app.py”, line 1822, in full_dispatch_request#012 rv = self.handle_user_exception(e)#012 File “/usr/local/lib/mailinabox/env/lib/python3.10/site-packages/flask/app.py”, line 1820, in full_dispatch_request#012 rv = self.dispatch_request()#012 File “/usr/local/lib/mailinabox/env/lib/python3.10/site-packages/flask/app.py”, line 1796, in dispatch_request#012 return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)#012 File “/root/mailinabox/management/daemon.py”, line 75, in newview#012 return viewfunc(*args, **kwargs)#012 File “/root/mailinabox/management/daemon.py”, line 635, in backup_set_custom#012 return json_response(backup_set_custom(env,#012 File “/root/mailinabox/management/backup.py”, line 515, in backup_set_custom#012 list_target_files(config)#012 File “/root/mailinabox/management/backup.py”, line 471, in list_target_files#012 bucket_objects = s3.list_objects_v2(Bucket=bucket, Prefix=path)[‘Contents’]#012KeyError: ‘Contents’

latinhypercube · December 12, 2022, 3:52pm

Can’t say whether this is your problem but it is necessary to create a folder in your bucket to save backups. The backup mechanism can’t create it itself in my experience.

Parameters are then something like this
S3 Host/Endpoint: s3.fr-par.scw.cloud [I’m using Scaleway]
S3 Path; bucketname/foldername

There are also restrictions on allowed characters in names.

syslarper · October 14, 2023, 2:37pm

I had to create an account to share my victory over the past four hours of relentlessness with MiaB and Minio.

To run Minio like me, in a firewalled and SSL self-signed way, you either need to:

Figure out how to import/accept the SSL cert using python on your box (gave up)
Use certbot to set up legit certs using dns verification (to keep firewalled/use alternate ports).

I got my Minio set up with proper certs. And then the fun really began. Working through the annoying flaws in both MiaB and Minio collectively.

It is possible to get an “OK” success after your efforts. But here are some tips for future readers:

As of current, MiaB is bugged for S3. It will not parse or handle http(s) or port numbers in Host/Endpoint. And will screw up its POST if you use any URI such as http. You must use “yourminio.com” as one simple example.
do not enter any region name
It does not allow you to accept self signed certs currently. ( I’m using the post hook method with certbot to run the post hook script, rename, replace and chown minio:minio my outdated certs to minios cert folder when re-generated (/etc/letsencrypt/renewal-hooks/post/copykeys.sh - look it up)
It will always connect on a 443 port. Do not try using iptables to forward 443 to 9000. Websockets will not work with this. Wasted timefiguring that out. Instead, reconfigure minio to run the API on 443 using your delicious accepted LE cert.
https://serverfault . com/questions/1081607/access-minio-api-on-port-80
https:// github . com/minio/minio-service/tree/master/linux-systemd
You need to create a miab bucket. AND THEN (*important) you need to create a folder within the bucket (e.g. mail) AND THEN (!) go inside that folder and upload some temp file to keep make the folder persist as a minio bucket generated directory. There’s no other way around it.

Good luck.
Hopefully MiaB addresses this tedious stuff once and for all.
ps - i had to break some urls with spaces because new new users can never be trusted to contribute

miabuser · October 17, 2023, 6:54am

Look on the bright side. Now you have a properly configured Minio instance instead of the improvised thing with self-signed certs and port mapping you had before