Mail-in-a-Box version v0.40 and moving to Ubuntu 18.04


#1

Hello everyone! After nearly a year of work, Mail-in-a-Box is ready to be installed on Ubuntu 18.04.

Introduction

For those who aren’t familiar with Ubuntu’s release schedule. each version of the Ubuntu linux operating system is supported by Ubuntu with security updates for a certain amount of time. For the last five years Mail-in-a-Box has been built on top of Ubuntu 14.04. Ubuntu’s support window for Ubuntu 14.04 ends in April, however. After that point, as time goes on, there is a growing risk that a security vulnerability will be discovered in Ubuntu 14.04 and no fix will be posted.

Going forward, Mail-in-a-Box will be based on Ubuntu 18.04, which is last year’s release of Ubuntu and is supported by Ubuntu through April 2023. Version v0.40 is the first version of Mail-in-a-Box to work on Ubuntu 18.04. Version v0.30 will be the last version of Mail-in-a-Box to work with Ubuntu 14.04. There are no major functional changes to Mail-in-a-Box in v0.40 and actually not many updates to the underlying software packaged with Ubuntu either. As always, our full release notes are posted in our CHANGELOG.

If you are installing Mail-in-a-Box for the first time, use Ubuntu 18.04. (The setup guide and instructions are updated. The only change is creating a server with the newer version of Ubuntu.) If you are already running a Mail-in-a-Box, please keep reading about how to upgrade.

Thanks in particular to deflume1 who kicked off the work on switching to Ubuntu 18.04 last year, and thanks to everyone else who posted fixes, reported issues, tested it out, and volunteered time here on the forum to help out other users.

About upgrading

The only supported way to update an existing Mail-in-a-Box to Ubuntu 18.04 and version 0.40 is to backup + restore to a new, clean Ubuntu 18.04 box. In-place upgrades of Ubuntu are not supported — in-place upgrades are simply too flakey for us to try to support. (If you do so, you do so at your own risk.)

Upgrades must follow these four steps:

  1. Upgrade your existing Mail-in-a-Box to version v0.30 first. This is mandatory and relates to limitations in the PHP versions supported by ownCloud/Nextcloud database migrations.
  2. Run a final backup of your data on your existing box.
  3. Setup a new machine running Ubuntu 18.04 and install Mail-in-a-Box on it.
  4. Restore your data from your backup to your new machine.

Details follow.

Detailed upgrade instructions

Here’s what you need to do step by step. Thanks to illume who posted their notes on the forum, which these steps are adapted from.

Let your users know what’s happening

Your box is likely to be off-line for some time during the upgrade process. If you change IP addresses, DNS may take a few days to update. Let your users know ahead of time what your plan is.

Update your existing box to the latest version of Mail-in-a-Box for Ubuntu 14.04

Update your existing Mail-in-a-Box to version v0.30. This is mandatory and relates to limitations in the PHP versions supported by ownCloud/Nextcloud database migrations.

Log into your machine using SSH in exactly the same manner as when you were setting up the box (see the setup guide section called Setting Up The Box for a reminder of what that looked like). Then, once logged in, run:

curl -s https://mailinabox.email/setup.sh | sudo bash

This is actually the same command you ran when setting up the box. It does upgrades too. (These are the usual steps for upgrading to a new version of Mail-in-a-Box described in Upgrading Mail-in-a-Box in the maintenance guide.)

Take a snapshot of your VM

If your box is hosted at a service that allows you to take snapshots of your machine’s hard disk, do that now. Digital Ocean and Amazon Web Services do, among others. It’s always good to have a backup before big changes.

Digital Ocean automatically makes periodic backups of machine hard drives. But these backups are deleted if you delete or rebuild your droplet. So if you are on Digital Ocean, convert your droplet backups to snapshots now to make sure you don’t lose them.

Perform a final backup

Stop mail services

You will be performing a backup + restore to move your data to the new box. To ensure you have a final backup, first block access to your box to all services besides SSH so that no new emails are sent or received, no calendar or contacts data is changed, and so on, after the backup completes — you don’t want to have any data on the box that isn’t backed up.

Log into your box using SSH and run the following commands to change the software firewall rules to block everything except SSH:

sudo ufw reset
sudo ufw allow 22
sudo ufw enable

(If you are using a nonstandard SSH port, update accordingly.)

Make a backup

Then perform a backup to ensure you have everything backed up since the last nightly backup run. Run:

cd mailinabox
sudo management/backup.py

If using local backups, make a copy.

If you haven’t configured your Mail-in-a-Box backup settings, you are probably creating backups that are stored on the box itself. Copy the backups and the encryption key used to encrypt the backups off of your box and onto your desktop or somewhere else.

Using any SFTP program like FileZilla (or scp if you know how) copy your backups from /home/user-data/backup/encrypted and the encryption key at /home/user-data/backup/secret_key.txt on your box to somewhere safe.

If using S3 or other remote backups, check it

If you are backing up your Mail-in-a-Box to Amazon S3 or another remote backup location, give a quick look at that location to make sure there are new backup files there created just now.

Create a new Ubuntu 18.04 machine for your new Mail-in-a-Box

Spin up a new machine following the setup guide’s section The Machine. Use as many of the same settings as on your original box as makes sense. But of course now choose Ubuntu 18.04 LTS x64 server edition as your operating system.

If your VM provider has a way to do it, try to preserve your IP address. In the email world, IP addresses have a “reputation” and mail from IP addresses with a bad (or nonexistent) reputation tend to get rejected or sent to spam folders more often. So you want to keep your reputation if you can.

  • On Digital Ocean, you have the option to preserve your IP address by using Digital Ocean’s Rebuild feature. However, make sure your Digital Ocean droplet backups are converted to snapshots and that you made a snapshot of the droplet first. Use Droplet -> Destroy -> Rebuild to create a new Droplet with the same IP address. If you don’t use Rebuild, make sure that you name your droplet the same name as your existing droplet — your box’s primary hostname — because the name is crucial for having reverse DNS set correctly.
  • On Amazon Web Services, if you’re using an Elastic IP address then you an re-assign it to a new box. If you’re not using one, now is the time to start.
  • Otherwise, you might want to ask your VM provider if they can swap the IP addresses on your old and new machine so that you can take your old IP address with you to the new machine.
  • If you absolutely can’t keep your IP address when moving to a new box, and if you have been using your IP address for a long time, you may want to… sorry…attempt an in-place upgrade to Ubuntu 18.04. This is the only circumstance where we’ll try to support in-place upgrades if you ask here on the forum.

Then follow the steps in the setup guide’s section Setting Up The Box to set up your new machine. When you are prompted for the box’s hostname, you will need to use the same hostname that you are currently using. Basically you just SSH into your new machine and run:

curl -s https://mailinabox.email/setup.sh | sudo -E bash

Restore your backup

Still logged into your new machine, you will now restore your backup.

First, move aside the new machine’s empty Mail-in-a-Box user-data directory.

sudo mv /home/user-data /tmp/user-data.empty

Local backups

If you were using the default backup method, which stores backups on the box itself, then you should have already copied the backup files and encryption key off the old box to somewhere safe. Using any SFTP program like FileZilla (or scp if you know how) copy your backups from wherever you stored them to somewhere on your new machine.

Then restore the files:

export PASSPHRASE=$(cat /path/to/secret_key.txt)
sudo -E duplicity restore --force file:///path/to/backup /home/user-data/
Amazon S3 backups

If your backups are stored in Amazon S3, get your AWS account credentials handy. You should have already copied your backup encryption key to a safe place (it was at /home/user-data/backup/secret_key.txt on your old machine) — copy that onto your new box somewhere as secret_key.txt. Then run:

export AWS_ACCESS_KEY_ID=paste your AWS access key ID here
export AWS_SECRET_ACCESS_KEY=paste your AWS secret access key here
export PASSPHRASE=$(cat /path/to/secret_key.txt)
sudo -E duplicity restore --force s3://s3.amazonaws.com/your-bucket-name/your-backup-path /home/user-data/

You may have to adjust the S3 URL depending on what AWS region you use. You can find the AWS Regions and Endpoints here

The whole backup + restore process is also documented at https://mailinabox.email/maintenance.html#moving-boxes in our maintenance guide.

Re-run Mail-in-a-Box setup

Your files are back in place, but you must re-run Mail-in-a-Box setup one more time to set up all of your domains and other settings.

Run:

sudo mailinabox

After that, your new box should be operating correctly.

Updating DNS

If your IP address changed, you’ll need to update DNS. If you were using your Mail-in-a-Box for DNS (the recommended way), you’ll need to go to your domain name registrar and update the “glue records” that you created during setup. (If you are serving mail for more than one domain, glue records only exist for your box’s primary hostname — you do not need to make any changes for other domains.)

If you are using external DNS, compare the External DNS page of the Mail-in-a-Box control panel at /admin on the new box to your DNS settings.

Wrapping up

Log into your new box at /admin and look over the system status checks. If your IP address changed, you’ll have a lot of errors — but some might be fixed automatically once DNS changes update, which can take hours or a few days. Look over the final steps in the setup guide for additional tips.

Test your new system: send/get emails, look at the calendar, admin section, etc. Look at some of the test services listed at the end of the setup guide.

Once everything is working

After you’re sure everything is working — maybe a month later — go back and delete your old machine (unless you used Digital Ocean’s Rebuild feature). You may want to test that you are able to do anther backup + restore to a third machine to make sure you have that process down, in case you need it again.

Need help?

If anything seems wrong with these instructions, please reply to this post with the problem you encountered — I will try to update this post with the best information.

If you encounter other problems, please start a new topic.


Help: Waiting for the Mail-in-a-Box management daemon to start
Pip error with fresh install of v.40 on Umbuntu 18.04 using upgrade steps
#2

Congratulations and THANK YOU! for the work to get this release done. I’ve only been using Mailinabox for a few months now, and I’m grateful for how this community works to produce an outstanding mail server product.


Ray Frush (Fort Collins, Colorado)


#3

Thanks, Josh!

If you don’t have a ton of data on your MiAB installation, one potentially easier upgrade option with at least some cloud providers is to simply resize your old disk (if needed to gain enough space), create a new disk on the same machine, and then install Ubuntu 18.04 onto that new disk.

At least with Linode, this is really easy to do, and you can set up a configuration profile to boot with the new 18.04 install without having to destroy the old 14.04 disk or profile (and you can easily revert to the old install if things go wrong).

This eliminates any hassle with switching IP addresses since it’s the same machine, you don’t have to mess with DNS, and since you can directly mount the old Ubuntu 14.04 disk while running 18.04, it’s really easy to copy your old data over too.

I admit that I haven’t tried this with MiAB yet, but I’ve done this in many other situations with other applications on Linode and it works great. I assume DO has similar capability.

Even the lowest-tier plan on Linode has 25 Gb of space, which is enough to allow for two 12 Gb disks simultaneously, which is presumably more than a lot of people have in their mailboxes. (And even if you have too much data, you can easily upgrade your server to a higher plan for a couple of hours while performing the upgrade to get enough disk space, and then downgrade once you’ve performed the migration.)

Scott


#4

That’s a great idea for folks comfortable doing that. To riff on that, some VM providers might allow you to swap out the drive entirely for a different one - and that would be an even cleaner way to do it.


#5

My VPS provider allows me to simply reinstall a new OS … which is my plan. I can copy backups to some place safe and then request a reinstall of the OS to Ubuntu 18.04. This preserves the IP address. Seems that this would likely be the simplest path? The only caveat is that you destroy the Ubuntu 14.04 VPS so do not have that to fall back on. But it should be rare that the process fails so badly that one would give up? Am I missing anything here?


#6

When I run the setup script it looks to be failing ont he z-push setup

Installing OpenDKIM/OpenDMARC…
Installing SpamAssassin…
Installing Nginx (web server)…
Installing Roundcube (webmail)…
Installing Nextcloud (contacts/calendar)…
Nextcloud is already latest version
Installing Z-Push (Exchange/ActiveSync server)…

FAILED: wget -O /tmp/z-push.zip https://stash.z-hub.io/rest/api/latest/projects/ZP/repos/z-push/archive?at=refs%2Ftags%2F2.4.4&format=zip

–2019-01-12 12:39:52-- https://stash.z-hub.io/rest/api/latest/projects/ZP/repos/z-push/archive?at=refs%2Ftags%2F2.4.4&format=zip
Resolving stash.z-hub.io (stash.z-hub.io)… failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘stash.z-hub.io

Bad luck this repo is down ?


#7

Looks like a DNS error someplace … I suggest just trying again a bit later.

If you would rather edit your hosts file at /etc/hosts you could add the line:

195.201.215.164 shash-z.hub.io

If you do this I suggest that you remove that entry immediately after you have successfully completed installation as the IP address may change at anytime in the future.


#8

ya for now i’ve reverted to my snapshot and figured try again later :slight_smile:


#9

Shouldn’t that be a sudo-prefaced command?


#10

Thanks @cliff1976, fixing!


#11

Thank you so much, Josh and all contributors. My stuff is working. The upgrade to 18.04 and v0.40 was very smooth. I am so happy with it!

Something I just thought of concerning a final step for the migration – the secret_key.txt file changes as part of the reinstallation under 18.04 and v0.40, and I think a reminder to store the new post-upgrade secret_key.txt safely would not go amiss.


#12

Thanks Josh and team!!

Followed the instructions above and it was perfectly smooth sailing: did final backup to S3, rebuilt machine (Linode) to 18.04 so IP address etc stays the same, installed miab, pulled backup down from S3…PERFECT! Everything was there as if I’d never changed a thing.

Possible Gotcha Some people may overlook that even though their config is restored, it is not truly restored in it’s entirety i.e. the backup configs (from /home/user-data/backup) are not restored since they are not part of the backup in the first place. Thus you need to manually reconfigure backups via the admin control panel.

Thanks again!

Brett


#13

Perhaps it goes without saying, but a new machine (even if you use Digital Ocean’s rebuild option instead provisioning a new VPS) means a new ssh host key on the MiaB, and if you’re fetching local backups from the MiaB instance to somewhere else, you gotta update the known_hosts file, too, else your offsite backup scripts will get the man-in-the-middle warning.

So ALL aspects of the backup routine (local and remote) need to be checked and updated.


#14

Hi,
thank you very much for the hard work!

Just want to add that since a year I use a separate disk mounted on /home/user-data , it is a nice approach as running out of space won’t harm your system root partition and makes backups with snapshot easier.

Now I can just create a new instance with Ubuntu 18.04 and attach the same disk to the new machine without having to restore or copy (a process that can be slow), making total downtime just a minute longer than the setup script

Maybe a good thing to do since a lot of users will switch
Cheers!


#15

I’m trying to setup MIAB on a fresh server, but stuck like this. Tried 2 times same result.

Waiting for the Mail-in-a-Box management daemon to start…


#16

Looks like you are at UpCloud, too … Try this: Ubuntu 18 install issue and confirm if solved.


#17

Hey @JoshData - you may want to recommend WinSCP to users instead of FileZilla. Earlier this year the dev for FileZilla started shipping releases with “bundled offers,” i.e. adware. Some of the bundled stuff led to more bundled stuff that was straight-up malware.
Link:


#18

That’s correct. And Thank you so much. The above solution worked me. MIAB is running perfectly fine. :sunglasses:


#19

Just wanted to say hi and a big thank you!

I have just upgraded and moved to ubuntu 18.04 without experiencing any issues.:+1:


#20

Another option for Linode users is to go to “Remote Access” in the Linode Manager and select IP Swap.