Speed up backups or migration proccess

[marcosmatas]

I need to upgrade the OS in my mail-in-a-box installation (Still with 18.04). But i have big troubles with the speed to make the backup of the machine and restore it.

My Mail-in-a-box lives in a virtual machine, but I have 588 Gb of mails. Making a backup using the procedure indicated on the web takes me a long time (approximately 18 hours). However, making a backup of the virtual machine with veeam takes just over 30 minutes. Keep in mind that it would take 18 hours just to make the backup and then you have to restore it to the machine, so it takes another 18 hours to restore it.

I guess the slowness of the backup is due to two factors:

• Data is compressed
• The data is encrypted.

Is there a way to make a backup without compressing or encrypting? Or to make a migration between the two machines? I am going to have the old machine and the new one working at the same time so perhaps I don’t need the backup. If I can transfer everything from one to the other without a backup between both servers, I could save a lot of time (I already have veeam backups).

[marcosmatas]

Another question. ¿All the troubles with the z-push are solved in the new versions?

[alento]

@marcosmatas

Another question. ¿All the troubles with the z-push are solved in the new versions?

I believe so, but don’t take my word on it …

I answered the simple question first.

@marcosmatas

Is there a way to make a backup without compressing or encrypting? Or to make a migration between the two machines?

Yeah, if you really know what you’re doing it is possible. But a couple of questions first … are you not using the MiaB backup function to create a backup now as it is? I am going to assume not since it should not take 18 hours to update potentially a few hours worth of emails.
What VPS provider are we working with? For that insane volume of email, you should be using external storage. If you were, your path to success would be much simpler.

I don’t know if I’d want to be the one to take on this project. 588gb of emails? How many users? But, if you’d like to hire me for a hour or two to figure this out, get in touch.

[marcosmatas]

@alento

I believe so, but don’t take my word on it …

Perfect. That reassures me. It’s a feature that I know wasn’t ready in the migration and that I consider very necessary.

@alento

are you not using the MiaB backup function to create a backup now as it is?

No. Let me explain a little how we have set up the MIAB server. I get the impression that you may be interested in the way we are using it and how we have it set up.

I’ve never liked hosting things on external “clouds.” I always preferred “on-premise” installations, meaning with our own hardware. Our first MIAB server was a small Pentium IV, but gradually, we made it grow. First, we upgraded it to a better machine, and later, when we virtualized some servers with VMWare ESXi, we included it as another virtual machine. Now, it coexists with a few other servers on the same machine. Currently, we perform backups using Veeam Backup to a FreeNAS that is connected to our network (as I told you, we don’t like “the cloud” and we want everything local). We have the entire machine fully backed up, so there is no need to use the mail-in-a-box backup tool (besides, the initial and incremental backups are significantly faster with Veeam).

Currently, we have assigned 4 processor cores, 4 gigabytes of memory, and 700 GB of hard disk space to the MIAB server. There are 184 email accounts on it, spread across 10 different domains. In reality, they are third-level subdomains of the type “correo.domain.com.” We use second-level domains for the main email, which are hosted by our regular email provider. The issue is that the space that our email provider offer is limited. It’s not a problem for computers using “POP” accounts, but people using mobile devices need IMAP accounts, so we use a workaround explained on this website:

Correo empresarial en el móvil: el método de las dos cuentas. - Flopy.es

Basically, what we do in these cases is the following: for the person who needs IMAP email on their mobile device, we forward a copy of their received emails to the account hosted on mail-in-a-box. This way, incoming emails can be viewed on the mobile device. As for outgoing emails, we configure the SMTP settings on the mobile device to use our regular email provider.

The problem with this method is that sometimes a user requests full synchronization of incoming and outgoing emails. In such cases, what I do is create an “alias” on our domain provider’s server, which is a “virtual” account that directly forwards received emails to the mail-in-a-box. I configure both the computer and the mobile device with IMAP settings pointing to the mail-in-a-box server, and for sending emails, I set up a generic account that allows sending emails through our regular provider. Essentially, the third-level domains on mail-in-a-box remain hidden from view, and the user is unaware that they exist in the background.

This may seem tedious and crazy. You’re probably thinking, “Why not let MIAB handle all the emails directly?” Well, it’s because of the issues with blacklists, Microsoft bans, and similar problems. Matters like the ones explained on this website:

#Tutorial para desbloquear un #servidor de #correo del bloqueo de #Microsoft

With this method, I let my domain provider handle those issues, and we only have to ensure that the communication between the domain provider and our server is flawless. We never use MIAB to send emails; we only use it to receive emails from our domain provider, and always we use it via IMAP. This way, we can focus on seamless email reception while leaving the complexities of deliverability and blacklisting to the domain provider to handle.

Another issue we face is the disk space: It might seem a lot of disk space, but if we don’t put a stop to it, it would be around five times more (I’m not exaggerating). We receive enormous amounts of email, some of which are very large in size. So, I’ve been forced to make a small “hack” to MIAB and enable email compression. Well… since this is becoming lengthy, I’ll also explain how I do it:

1. Change the file /etc/dovecot/dovecot.conf to add this lines (I added all of them at top of the file)

mail_plugins = $mail_plugins zlib

plugin {
zlib_save_level = 6 # 1…9; default is 6
zlib_save = gz # or bz2, xz or lz4
}

2. change the file /etc/dovecot/conf.d/20-pop3.conf . You have to find this line

mail_plugins = $mail_plugins antispam

And change it to this line

mail_plugins = $mail_plugins antispam zlib

3. Edit the file /etc/dovecot/conf.d/20-imap.conf

In the file you have to put the following:

protocol imap {
mail_plugins = $mail_plugins antispam imap_zlib
}

4. Edit the file /etc/dovecot/conf.d/20-lmtp.conf
   and put:
   protocol lmtp {
   mail_plugins = $mail_plugins sieve zlib
   }
   And then restart the server

But even so, I have some “challenging” users who have an extremely high volume of emails. Since most of them fetch their emails via POP and keep a local copy on their computers, and they don’t need to access emails older than a year, what I do is run a script periodically (launched with cronab) that deletes emails older than a year for these users. Here’s how I do it:

cd /home/user-data/mail/mailboxes/correo.mydomain.com/mailoftheuser
find * -mtime +365 -exec rm {} \;

Now we are planning to take advantage of setting up a new server with Proxmox to host MIAB there, allocating it more resources and disk space. However, the issue is that migrating “only” those nearly 600 gigabytes is proving to be incredibly slow (we have some virtual machines that occupy much more, and managing them is much faster).

I assume that if I move the content of the “/home/user-data/” folder from one server to another, restore the secret key and enable email compression again, I should have everything working, but I would rather make sure before I start doing things that will probably take a long time. I have previously performed updates and migrated machines with MIAB (from this same installation with everything I have commented) and everything has always gone well, but I have always used the official MIAB backup tool for it. And back then the machine didn’t weigh as much as it does now. That’s why my question about whether there is any other method I can use for the migration.

By the way… I am very clear that MIAB is focused on personal use, but I have to be honest: it works wonderfully and does not fail at all when using it in this way for larger projects.

[alento]

Wow, what an interesting read @marcosmatas. If you have followed this forum at all over the years you’ll know that I am highly critical of hosting MiaB locally as you have, but I do have to say that it can be done when properly done by someone who clearly knows what they are doing. You have a perfect, beautiful example of how it can be done and for that I commend you.

But you now have the issue of migrating 700gb of emails for 184 users. That does add some complexity to the problem.

My first thought would be to rsync the ‘/home/user-data/mail/mailboxes/’ directory to the new server after adding the users in the normal way, but that is not at all practical with 184 users. So that begs the question of "Is the structure of the sqlite database different between v 0.57a and the current version of MiaB? I cannot answer that, but perhaps @davness can offer some insight on that topic, for if the database layout is identical, you should be able to just rsync it over as well. It does sound to me from the way that you explained things that you are not using NextCloud, can you confirm this?

I guess I will have to leave this here for now and hopefully David will chime in, or someone else with the relevant experience/knowledge about the database issue will chime in.

But, let me ask you something a bit different … if you could resolve the deliverability issues that cause you to have to maintain email service off-prem with your “domain provider” and instead host it all locally – and probably eliminate the need to actually jump through these hoops, would you be interested? I guess the first question would be, how much are you paying extra for the service for the root domain’s email and what would be within budget to eliminate them, and at the same time eliminate the need for the subdomains that you are using in the manner described? Please feel free to take this to PM if you’d prefer to answer privately.

[marcosmatas]

@alento

if the database layout is identical, you should be able to just rsync it over as well

That’s what I think too. I even believe that instead of using “rsync” in the /home/user-data/mail/mailboxes/ directory, I should do it directly in /home/user-data. In any case, before making a mistake in a lengthy process, I prefer to ask. It wouldn’t be a bad idea to have somewhere documented how a “manual” migration could be done. I’m not suggesting implementing it as an official migration method in Mail-in-a-Box, but at least having it documented somewhere for cases like this (being objective, I don’t think there will be many people using the software in this particular way).

@alento

It does sound to me from the way that you explained things that you are not using NextCloud, can you confirm this?

Indeed, we don’t use Nextcloud using the Mail-in-a-Box installation. We do have a separate Nextcloud instance on a Virtualmin running on Debian (also virtualized in Proxmox), but we don’t use the Nextcloud installation provided by Mail-in-a-Box because we need to use additional Nextcloud plugins besides contacts and calendars. As far as I know, tweaking the Nextcloud of Mail-in-a-Box is strongly discouraged, so we prefer to use it separately. (And in case you’re wondering, our Nextcloud files also take up around 700 gigabytes). However, we do utilize that Virtualmin for various other tasks (proxy, hosting some pages, etc.). At one point, we considered migrating the email from Mail-in-a-Box to Virtualmin, but given the amount of work it would involve and since Mail-in-a-Box is working perfectly for us, we adhere to the policy of “If it works, don’t touch it.”

@alento

if you could resolve the deliverability issues…

It doesn’t seem at all straightforward to solve these kinds of problems because they don’t solely depend on how well or poorly you implement the mail server. On a personal level, I host my personal email on a small server at home with Virtualmin (which doesn’t have the scale of everything we’ve been discussing). Occasionally, you encounter situations where you try to deliver an email to someone, and despite having everything properly configured (DKIM, DNS, DMARC, etc.), sometimes there’s no way to deliver that email. It’s only later that you realize, for example, that the recipient’s domain is hosted on Microsoft servers, and for “security reasons,” Microsoft wants you to enroll in their “Junk Mail Reporting Partner Program.” Until you do so, you can’t send them emails. With so many users, it wouldn’t be surprising if, for any reason, several are sending emails to a particular domain, and that domain wrongly perceives it as spam, putting us on blacklists… In fact, I can think of several scenarios where it’s better to let our hosting provider handle email delivery. After all, with that provider, we have several large websites, so the cost of having the email service with them is negligible (as the hosting of those websites is what matters to us the most).

For now, the third-level domain scheme is incredibly convenient for us. In practice, the people using it are satisfied and completely unaware that there are two email accounts involved in receiving emails. For us, it’s all advantages: we eliminate unnecessary costs with the main email provider, have ample storage capacity, avoid problems and time wasted resolving blacklist issues or similar matters, and basically have no more worries than updating the mail server when needed. Perhaps for someone not accustomed to this type of servers, it may seem confusing or complex, but ultimately, you can install and make a Mail-in-a-Box functional in no time. Therefore, for us, this scheme is straightforward to manage and implement.

It’s a shame, but nowadays, having a static IP and a properly configured mail server doesn’t guarantee at all that emails will reach their destination. Particularly, Microsoft’s policy of forcing you to enroll in their program to send emails to their users seems like one of the sneakiest tactics. If all email providers did the same, we wouldn’t be able to send emails between them.

That’s why we don’t like the idea of sending emails directly from our mail server. I do it with my personal one because, in the end, I might send a maximum of 10 emails per day, but in the Mail-in-a-Box installation, it could be several thousand daily (some even to many recipients). It would be enough for one user to mistakenly send an email to many recipients, and the company’s IP could end up blacklisted. Dealing with those issues would indeed be much more complex.

[alento]

@marcosmatas

That’s what I think too. I even believe that instead of using “rsync” in the /home/user-data/mail/mailboxes/ directory, I should do it directly in /home/user-data. In any case, before making a mistake in a lengthy process, I prefer to ask.

Yes, of course … also it needs to be noted that the database is in a different location that the maildir’s, so I used “rsync” generically to mean copy from the old server to the new server. Now, I tried this for a migration from Ubuntu 14 to Ubuntu 18 for someone who had put it off and never had done any maintenance to his MiaB. It did not go well, but that was due to the fact that he had never updated, so the database format was different due to a change. This is why I stress that the database format needs to be the same.

@marcosmatas

It wouldn’t be a bad idea to have somewhere documented how a “manual” migration could be done. I’m not suggesting implementing it as an official migration method in Mail-in-a-Box, but at least having it documented somewhere for cases like this (being objective, I don’t think there will be many people using the software in this particular way).

In some ways I agree, but then in others there is no one-size fits all solution. So it is really impractical to document.

@marcosmatas

Indeed, we don’t use Nextcloud using the Mail-in-a-Box installation. We do have a separate Nextcloud instance on a Virtualmin running on Debian (also virtualized in Proxmox), but we don’t use the Nextcloud installation provided by Mail-in-a-Box because we need to use additional Nextcloud plugins besides contacts and calendars.

Excellent!

@marcosmatas

It doesn’t seem at all straightforward to solve these kinds of problems because they don’t solely depend on how well or poorly you implement the mail server.

Actually, that is not the case at all. You just need to relay through someone who does have a good network. All of your comments and thoughts are valid, but they go out the window when someone is relaying their outgoing mail through a SMTP relay. I ask as I created this product simply to address this need within the community:

AnyMXRelay - AnyDomain LLC.

@marcosmatas

For now, the third-level domain scheme is incredibly convenient for us. In practice, the people using it are satisfied and completely unaware that there are two email accounts involved in receiving emails.

As you said earlier, “If it isn’t broken, don’t fix it” (well that is not exactly what you said, but the same concept applies) but I did want to at least make you aware that it is absolutely possible to remove the level of complexity that you have.

[marcosmatas]

Finally, I have completed the migration… using the traditional method (creating a backup with the MIAB tool and then restoring it on the new machine).

Everything went well, but let me share what I did. What you mentioned about the database format was what worried me the most (and indeed, what you said about database versions makes a lot of sense). I couldn’t find out if the database versions vary between the installation on Ubuntu 18 and the installation on Ubuntu 22, so I preferred to play it safe.

The problem, as I feared, was the copying time. I tried to minimize the volume of what I was going to copy (deleted old emails from many accounts and even removed some that were no longer in use), but still, it took 12 hours for the backup and nearly the same amount of time for the restoration on the new machine. In total, it took 24 hours.

I know such a large-scale installation is not common, but as a suggestion (and I say this with the humility of understanding that this is an open-source project, and the developers will have to evaluate if this proposal is beneficial for the community or not), it might be good to have an option to perform a direct copy between two machines. Perhaps utilizing the existing script, but instead of compressing everything into multiple files, it would directly copy to the correct folders on the destination machine. This way, it would save time on file compression (copying is always faster than compressing), and there would be no need for decompression. Such an approach would accelerate migration by more than twice the time.

Regarding particularities of the installation, let me share some anecdotes of things that happened to me, so if anyone else encounters these issues, they know how to solve them.

The first issue I encountered was that immediately after restoring the backup, nothing was working (not even the system’s configuration page). I simply had to run the installation script again, and everything was resolved with a restart.

But immediately after, I noticed that no emails were showing up in the mailboxes of the accounts. I remembered this from previous migrations. After migrating, I always have to remember to reconfigure the compression of the emails (as I mentioned in my third message in this thread), because otherwise, the system will encounter compressed mails it can’t read. Once those changes were made, the system was functioning correctly.

A couple of days after completing the migration, the certificate of one of the domains was about to expire. However, when the system attempted to renew the domain, the renewal process failed and presented the following message:

Provisioning TLS certificates for correo.domain.com, autoconfig.correo.domain.com, autodiscover.correo.domain.com, mta-sts.correo.domain.com, [www.correo.domain.com](http://www.correo.domain.com).
error: correo.domain.com, autoconfig.correo.domain.com, autodiscover.correo.domain.com, mta-sts.correo.domain.com, [www.correo.domain.com:](http://www.correo.domain.com:)
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Missing command line flag or config entry for this setting:
Please choose an account
Choices: ['box.otherdomain.com@2018-10-24T07:09:56Z (1665)', 'box.otherdomain.com@2019-03-16T16:19:18Z (235f)', 'box.otherdomain.com@2018-10-24T08:18:53Z (7cfb)']
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

I tried to manually renew the certificate through the terminal, but I couldn’t manage to renew it. I found the solution on these same forums at this link:

https://discourse.mailinabox.email/t/missingcommandlineflag-msg-certbot-errors-missingcommandlineflag-missing-command-line-flag-or-config-entry-for-this-setting-please-choose-an-account/7281

I simply had to go to the folder /home/user-data/ssl/lets_encrypt/accounts/acme-v02.api.letsencrypt.org/directory and delete the older certificate folders (there were three folders, and I kept only the latest one). With this change made, I restarted the machine, launched the certificate renewal through the web again, and the certificate was successfully renewed without any issues.

Another thing that I always forget (yes… my memory is a mess) is that I must always modify fail2ban after any MIAB update (after running the installation script). Many people who use the server access it from the same location, so they access it from the same public IP address. Fail2ban interprets this as an attack, and after a while, the server gets blocked, preventing anyone from that IP from accessing it. To fix this, I just need to access these two files:

/etc/fail2ban/jail.d/mailinabox.conf
/etc/fail2ban/jail.conf

…and in the line “ignore IP,” I just need to put the public IP addresses from which we access the server.

Lastly, I had to make another small adjustment (I forgot to mention this in my third message as well). Since most messages are received through aliases from other domains, it’s common for some messages to end up in the SPAM folder incorrectly (because the mail server sending the message to MIAB is one of our servers, but the sender doesn’t use that server). With this setup, sometimes SPF and DMARC records get marked as incorrect by SpamAssassin, causing the message to be sent to the spam folder. This can be resolved by editing the miab_spf_dmark_.cf file located in the /etc/spamassassin directory and modifying the scores assigned when there are issues. I’ve left the file as follows:

# Evaluate DMARC Authentication-Results
header DMARC_PASS Authentication-Results =~ /box\.correo\.tabigal\.com; dmarc=pass/
describe DMARC_PASS DMARC check passed
score DMARC_PASS -0.1

header DMARC_NONE Authentication-Results =~ /box\.correo\.tabigal\.com; dmarc=none/
describe DMARC_NONE DMARC record not found
score DMARC_NONE 0.1

header DMARC_FAIL_NONE Authentication-Results =~ /box\.correo\.tabigal\.com; dmarc=fail \(p=none/
describe DMARC_FAIL_NONE DMARC check failed (p=none)
score DMARC_FAIL_NONE 1.0

header DMARC_FAIL_QUARANTINE Authentication-Results =~ /box\.correo\.tabigal\.com; dmarc=fail \(p=quarantine/
describe DMARC_FAIL_QUARANTINE DMARC check failed (p=quarantine)
score DMARC_FAIL_QUARANTINE 1.0

header DMARC_FAIL_REJECT Authentication-Results =~ /box\.correo\.tabigal\.com; dmarc=fail \(p=reject/
describe DMARC_FAIL_REJECT DMARC check failed (p=reject)
score DMARC_FAIL_REJECT 0.0

# Evaluate SPF Authentication-Results
header SPF_PASS Authentication-Results =~ /box\.correo\.tabigal\.com; spf=pass/
describe SPF_PASS SPF check passed
score SPF_PASS -0.1

header SPF_NONE Authentication-Results =~ /box\.correo\.tabigal\.com; spf=none/
describe SPF_NONE SPF record not found
score SPF_NONE 1.0

header SPF_FAIL Authentication-Results =~ /box\.correo\.tabigal\.com; spf=fail/
describe SPF_FAIL SPF check failed
score SPF_FAIL 1.0

And yes… I know you’re going to say that this will cause more actual SPAM to go to the inbox. I know. But over time, I’ve observed that a message that is genuinely SPAM is usually detected correctly by spamassassin through other means. At least, no one has complained about having an excessive volume of spam in the inbox (especially when the emails are already filtered and marked at the main domain on our hosting).
Perhaps someone may find some of the things I’ve done useful (sharing knowledge is always best).