Mailbackup B2 Storage

I’ve managed to get my backup to Backblaze B2 storage and I thought I would share it here…

Shamelessly edited the script found over at autoize > original one found here, which is intended for Nextcloud.

Made some adjustments and got it working for Mailinabox.

#!/bin/sh
Path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

# EDITED by Mellow for use with Mailinabox
# Originally a NextCloud to BackBlaze B2 Backup Script
# Original Author: Autoize (autoize.com)

# This script creates an incremental backup of your Mailinabox instance at BackBlaze's off-site location.
# BackBlaze B2 is an object storage service that is much less expensive than using Amazon S3 for the same purpose, with similar versioning and lifecycle management features.
# Uploads are free, and storage costs only $0.005/GB/month compared to S3's $0.022/GB/month.

# Requirements
# - BackBlaze B2 account (10 GB Free) - Create one at https://www.backblaze.com/b2/sign-up.html
# - BackBlaze B2 CLI installed from PyPI - sudo pip3 install b2

# Instructions
# 1. Create a bucket and obtain your Account ID and Application Key from your B2 account.
# 2. Authenticate your CLI using the b2 authorize_account command.
# 4. Save this script to a safe directory such as /srv/backupToB2.sh and make it executable with the following command.
#      sudo chmod +x backupToB2.sh
# 5. This script must be run as root. To run a backup now:
#      sudo ./backupToB2.sh
# 6. Set up a cron job to run this backup on a predefined schedule (optional).
#      sudo crontab -u root -e
#    Add the following line to the crontab to conduct a weekly backup every Saturday at 2:00am. 
#      0 2 * * sat /srv/backupToB2.sh > /srv/backupToB2.log 2>&1
#    Save, quit and check that the crontab has been installed using the following command.
#      sudo crontab -u root -l

# Name of BackBlaze B2 Bucket
b2_bucket='yourbucketname'

# Path to box backupdata directory
data_dir='/home/user-data/backup/encrypted'

# Check if running as root

if [ "$(id -u)" != "0" ]; then
   echo "This script must be run as root" 1>&2
   exit 1
fi

echo 'Started'
date +'%a %b %e %H:%M:%S %Z %Y'

# Sync data to B2

b2 sync $data_dir b2://$b2_bucket$data_dir

date +'%a %b %e %H:%M:%S %Z %Y'
echo 'Finished'
6 Likes

Awesome.

I didn’t realize they have a free tier.

To-do list updated.

There is an open PR to add Backblaze as a backup destination here. Hope it’ll get merged eventually, B2 is a nice service :slight_smile:

3 Likes

Been meaning to enable this on my box while we wait for maybe having it implemented in vanilla MiaB. This is great to share a detailed step by step manual setup and a script for it!

I would appreciate having B2 support as well. I currently run a cron job that runs independent of MiaB and syncs MiaB’s backups to a B2 bucket using rclone. This has worked well for 6 months, but having B2 as an option in MiaB would be the preferred way to go.

1 Like

Is it not possible to back up to B2 using the S3 option in the MiaB control panel? B2 has S3 compatibility.

I tried that a few weeks back and it did not work. It seems like some s3 api calls that duplicity uses were missing/different enough in b2 that it did not work properly. Might have changed already though

1 Like

Ah, I see. I’ve not tried it yet, was just something I thought about since the new(ish) B2 S3 API should make it a drop-in replacement in most cases.

I looked into why the S3 option of MiaB wouldn’t work with B2 back when B2 offered S3 compatibility. From what I could see, the API calls were the same, but the endpoint URLs to B2 nodes are slightly different that S3.

https://s3.us-west-001.backblazeb2.com/bucketname
vs
https://s3.us-west-1.amazonaws.com/bucketname

The scripts MiaB uses hardcodes parts of the AWS endpoint, the “us-west-1” part, and thus it can’t talk to B2, simply because of the missing ‘00’.

1 Like

yes that is a problem. We would require post-processing in order to detect whether b2 or amz is used in the backend. This came up in the PR as well and I agree with the author’s stance that a separate b2 option would be preferable both from a usabilty and code quality point of view.

There is another problem however:
the duplicity version used in miab relies on boto2 s3 bindings which do not support b2’s s3-compatible API due to it requiring the v4 signature algorithm.
So at the minimum, we would have to upgrade duplicity so that we can use boto3. As mentioned, I tried all that on my dev box a few weeks back and still ran into trouble when verifying backups.

This is what I’ve also been doing… Would love to see it baked in though…

Ok, so update from me: I use the PR since I made it and I also run into verification problems. However, I feel that this is a problem with the verification process or duplicity, but not with the implementation.

If I look into the logs I see for example:

“Difference found: File mail/mailboxes/hilko.eu/mail/.Mailinabox has mtime Thu Nov 19 23:21:00 2020, expected Wed Nov 18 03:46:08 2020”

For me that states that the version on backblaze is newer than the local version. So this is no problem for me (since newer should be better) and I think the reason is probably that we are comparing the backup with an outdated cache. Has anyone tried running the verification command on an amazon aws backup? I would expect the same behaviour since I did (at least to my knowledge) no changes to the backup procedure, i.e. which files are updates, how the cache is stored etc.

Besides, I don’t know if Josh is interested in implementing B2 backups, since he never commented (positively nor negatively) about the PR.

Just had a look again. There is an older issue with exactly the same problem (backup data is newer than the data compared against):

As mentioned earlier, i guess this has something to do with the way we verify our backup or how we create the cache directory against we which compare our backup.

@fspoettel are you using AWS Backup on your miab production system? Would you mind running “sudo .management/backup.py --verify”? Do you get verification errors or does it run fine?