Nightly backup to a full secondary standby server


#1

Hi all

I’ve slightly hacked the backup script to include a full rsync backup to a secondary server that has a full MIAB install, and simply is offline [in terms of dovecot, postfix, and php7.2-fpm] and is ready to be enabled at the click of a button should your primary server go down.

This could happen for any number of reasons, and since it only adds about 20 to 30 seconds to the nightly backup script [or slightly more if your server is user by a bunch of people], I thought I’d share it here.

This guide is for MIAB v0.4 / Ubuntu 18.

1] Fully install an empty MIAB server on a new machine in a remote location [as independent as possible from the main one of course]

2] on the newly created empty secondary node:

systemctl disable dovecot
systemctl disable postfix
systemctl disable php7.2-fpm
mv /home/user-data /home/user-data-initial

3] on the primary node:

enter the user where you installed MIAB as and locate the “mailinabox” folder [with contents such as Changelog.MD, etc…] and cd into it

nano +254 ./management/backup.py

change the pre-backup portion of the script so that you get this:

        pre_script = os.path.join(backup_root, 'before-backup')
        if os.path.exists(pre_script):
                shell('check_call',
                        ['bash', pre_script, config["target"]],
                        env=env)
nano /home/user-data/backup/before-backup

and paste the following into it:

#!/usr/bin/env bash

rsync -Pave "ssh -i /home/user-data/backup/push-backups-to-standby-server" /home/user-data/ root@SECONDARY-NODE-IP:/home/user-data/ &> /dev/null

changing SECONDARY-NODE-IP with the public ipv4 of the secondary node of course.

ssh-keygen -t rsa -b 8192

and at the prompt specify to save it into /home/USER/.ssh/push-backups-to-standby-server changing USER with the username that performed the MIAB install, the same one we located earlier in these instructions.

cat /home/USER/.ssh/push-backups-to-standby-server.pub

and copy the output on the second server’s /root/.ssh/authorized-keys file with nano. If there’s no /root/.ssh/ dir, simply mkdir /root/.ssh before you do the nano part.

Now back on your primary node sudo -i or log in as root.

ssh -i /home/user-data/backup/push-backups-to-standby-server root@SECONDARY-NODE-IP

and press yes if you get prompted to continue with the initial connection.

That should do it!

If you ever need to make the secondary node live because your first one has any problems, simply [from the secondary node as root]:

systemctl enable php7.2-fpm
systemctl enable dovecot
systemctl enable postfix

then stay on the secondary server and log in as the user that has performed the empty MIAB initially [where the “mailinabox” folder with Changelog.MD is], and then execute:

sudo mailinabox

to have MIAB pick up the new data. Then update your main MIAB domain’s DNS accordingly [glue records and such]. Remember to keep an eye on the status checks page of your secondary server to see if everything is running ok.

Back to the initial guide, before anything bad ever happens, the first rsync will take a long time so you might want to execute one manually [with services running, it’s ok, as it’s just an initial sync, a real one will follow later starting on the first night] by running:

bash /home/user-data/backup/before-data

as root from the primary node.

The command won’t output anything, but you’ll see the progress happening on your secondary node by executing [on the secondary node of course]:

du -sh /home/user-data

This complexity is due to this “issue

You’ll need to probably re-run part of these steps on the primary node when you upgrade mailinabox, most likely only the step where you modify the backup.py file though, everything else should be persistent.

I’ve been running this for a few days now and works amazingly well! Let me know what you think!


#2

I skimmed through this. Thank you for sharing your hack!

Do you think you could add necessary parts to the admin interface and make a pull request to the official repository? Maybe this could be an official feature you could share with everyone!


#3

I don’t think it would ever be accepted except for the part about enabling before-backup to run as root. MIAB already has an rsync backup option, pulling this would mean allowing the user to use multiple backup mechanisms, seems to me like more work that anyone would really use, but hey, @JoshData what do you think?


#4

I think you should use stop/start instead of disable/enable. As far as I know disable does not shut down the server, it just means it won’t start automatically on the next boot.


#5

That’s what I’d say, yep.


#6

cool, so can we have them run as root at least? ;]


#7

Let’s take the discussion to the issue you opened (https://github.com/mail-in-a-box/mailinabox/issues/1521).