Mail-in-a-Container: proof-of-concept, pros, cons, technical issues

[hughsw]

Hi Folks,

What is the overall interest and the maintainers’ perspective on migrating MIAB to run in a containerized environment as a fully supported, if not the recommended way to deploy Mail-in-a-Box?

TL;DR It’s easy to run the current MIAB in a proof-of-concept containerized environment – let’s discuss the interest and issues involved in fully supporting a containerized MIAB.

Background

As part of preparing for the migration to v60.1, and following my first false start, I got MIAB running in a podman container, hoping it would make the migration simpler to test and rollback if necessary. Running MIAB in a rootless, systemd-aware podman-container was an easy proof-of-concept for containerized MIAB. However, it raised a number of design, migration, and long-term support issues, so I set the containerization idea aside as my second false start.

Eventually my traditional migration to a new Ubuntu 22.04 server was successful, but it took a bit of planning and obsessing. It is documented as the final couple of entries in the Discourse discussion for the v60 migration. Version 60 for Ubuntu 22.04 is released - #104 by hughsw

As a result of the containerization POC I learned a lot about MIAB internals, and I’ve made PRs for a few fixes and UI improvements to MIAB.

At this point I’m trying to figure out if a containerized MIAB could

• make the end-user’s installation and maintenance easier
• make the maintainers’ development and testing cycles easier
• avoid the frictions of the v60 upgrade to Ubuntu 22.04
• address the common theme of wanting to run other services alongside MIAB (search for Discourse entries with ‘docker’ or ‘container’).

Proof-of-concept 1

I have a package that builds and runs a new MIAB installation in a rootless container using podman. The image-building steps control the exact Ubuntu 22.04 environment that MIAB gets inside the container. It is refreshing to have complete control over the Ubuntu, unlike when installing on one of the varied Ubuntus one gets on different cloud providers or the (unsupported) situations where MIAB is shoehorned into older or non-exclusive Ubuntu environments. That is, containerization eliminates a whole class of at-present-uncontrollable environmental issues.

On the host side, for rootless use of podman, I had to tweak the sysctl config to allow unprivileged users to open low-numbered ports, so that the rootless podman could map the relevant ports into the container. This seemed preferable and more cgroups embracing than running podman as root. The command-line for running podman specifies the necessary port mappings and directory bindings so that MIAB can do its things.

On the MIAB side (running in the container), for the POC, I only had to make two un-intrusive one-line changes to MIAB’s setup code. By “un-intrusive” I mean that the changes would not affect the traditional hosted MIAB installation and usage. With those two one-line tweaks, containerized MIAB installation and maintenance is essentially identical to running the traditional (hosted) way. A minor difference is that you have to ssh to a high-numbered port that podman maps to the container’s port 22. After the curl-based setup steps, Email, DNS, certificates, web access, administration, backups, cli, etc all “just work”. I have done this using podman hosted on Ubuntu, Debian, and CentOS.

Proof-of-concept 2

I took the containerizing process a step further. I un-intrusively factored the setup script code, separating the logic into three semantics:

• generic installation of Ubuntu packages
• shared MIAB setup, i.e. MIAB-specific package configuration that does not depend on the contents of /etc/mailinabox.conf
• deployment specific MIAB setup, i.e. package and MIAB configuration that does depend on the contents of /etc/mailinabox.conf

The first two of these can be done when building the image, i.e. once, by the MIAB maintainer in the release build process. No end-user would have to wait for Ubuntu apt-get installations and their hiccups again, and maintainers would not have to field Issues stemming from the package installation problems, e.g. transient repository failures, botched upgrades, etc!

The third step, configuration that depends on the contents of /etc/mailinabox.conf (mostly PRIMARY_HOSTNAME), obviously has to be run by the end-user when setting up their deployment. Actually, it’s essentially the only thing an end-user has to do. It’s quicker and simpler compared the current MIAB installation process.

That’s where things stand with POC work, and the lessons of that effort informs the following list of discussion points.

Discussion points

There is the important question of clarifying and weighing the long-term gains and costs of moving MIAB to a containerized structure. Does it have significant benefits? And, to have a supportable, containerized MIAB package there are a number of high-level design and other technical issues to address.

Here’s my quick outline of the major technical matters to be sorted out:

• which containerization environment to support – I used podman for several reasons; Kubernetes, Docker, LXC/LXD, etc are other options
• how to manage the overall state of the MIAB deployment – at present this state is scattered in
  • the /home/user-data tree (configuration, credentials, email, webmail databases)
  • the /etc tree (configuration managed by packages and by MIAB)
  • the /var/lib tree (configuration managed by packages and by MIAB)
  • /root/.ssh/id_rsa_miab (credentials)
• which parts of this state should be stored where:
  • in the container’s file system, or is the container ephemeral (meaning there is no expectation of or reliance on the persistence of the container)
  • in the containerization’s file store, e.g. podman volumes
  • in host file trees
• how to migrate an existing MIAB installation into the containerized environment
• how to do a fresh MIAB installation into the containerized environment
• how to structure MIAB’s ongoing migrations, both its internals and when Ubuntu packages need to be reconfigured
• supporting development and testing work for maintainers
• release process and a CI/CD framework
• documentation
• And, once the above are reasoned-out, how to structure the incremental changes needed to get to the containerized goal without breaking things and with minimal friction

AFAICT none of these has any show-stopper qualities, but I’m not a veteran of the MIAB maintenance experience, and some of the webmail/web-app support in MIAB has torturous migration logic

And obviously, there would have to be agreement among the maintainers that this is the right direction for MIAB to move.

So, please provide high-level input on the pros and cons of moving to a containerized MIAB, and please add to the list and discuss the lower-level, implementation issues outlined above.

Thanks,
-Hugh

[ Some of the points (and prose) above appears in my comments in this issue: Any official docker installation? · Issue #1934 · mail-in-a-box/mailinabox · GitHub ]

[marko]

I would be happy if Mail-in-a-box would be moved into container. But I can judge from user POV, not as Mail-in-a-box developer.

For me, moving MIAB into container would allow to have emails on host FS. As we use ZFS on the hosts of containers, it would properly use backup and snapshots integrated with the rest of the systems.

Dockerized mail servers exist though, see https://mailcow.email/ . However, they don’t include DNS server as a part of a setup, unfortunately. But it is one of the possible pathways to consider as an alternative to dockerize this project. When comparing with Mailcow, I find MIAB easier to use. Although, being MIAB user it is biased comparison. So, having MIAB in container would be preferred by me.

I would expect that having MIAB in a container would allow us to easily add functionality by those who want it. For example, maybe we can get back decent mail indexing and search. Mailcow is currently using Solr - this can be done here as well and later switch to other schemes when they get supported.

It is possible that over time, MIAB if in a container, could start partitioning its services into separate containers which would allow us later to pick and choose useful bits when setting up the server. But that could be done over longer development cycle.

Re your questions:

• we went for docker in our setup as it is widely used. that also includes freeIPA which has systemd running inside the container
• container should be ephemeral, please. so, config and user files should be out of it
• I would expect migration to be similar as we have between 20.04 and 22.04 base - new install
• new install for containerized environment - probably corresponding README would be sufficient with the commands to run

[andrew]

I know MIAB is not a democracy, but from me a strong NO – Nnooooo. I acknowledge their utility but I try to avoid containerised anything as much as I can.

Resource usage – a containerised app is typically 10 x larger than a “native” app. In general, system performance is bad enough (1GB min RAM to run a useful Linux server) without making things 10 x worse! (Puts on old-guy voice: “I was running servers when you couldn’t even buy 1GB of RAM” )

Development - I like the ability to see inside MIAB. I can trust something more when I can see inside. And I can help out – make minor fixes or additions, and submit pull requests. That might be possible with containerised apps but it just adds a whole new layer of pain and discourages people from helping out. I don’t see that as the open source way.

Morality - Containers permit and encourage poor OS and system design … the library/app/version situation is bad enough without just giving up and saying “do whatever you want, wherever you want, I give you free reign” to both OS and app developers!

Security – our boxes are exposed to the rest of the world and need to be up-to-date and secure. MIAB keeps everything reasonably up-to-date using Ubuntu’s normal mechanisms. Wouldn’t stuff distributed by container need very regular container updates, or a horrible update-in-place mechanism, plus normal Ubuntu updates

[marko]

Good points, let me try to address some of them (keeping away from “morality”).

Resource usage: as containerized app is running natively, its size differences are related to bundled libraries that would be loaded in your docker host as well. Memory (RAM) overhead otherwise should be minimal.

Now, overall resource usage would depend on how you are running MIAB. We run it in a separate VM that is inside physical server responsible for several other services as well. In our case, dropping VM would reduce overall memory consumption by MIAB significantly - it will just use as much as it needs and I don’t have to factor in memory for other aspects of OS. In our case, all filesystem caches and other services can be shared by all container services running in the system directly.

Development: You can easily get inside running container shell through your host. That would allow you to peek inside and change there too.

When you compare to current deployment of MIAB, you would have way better control of what is inside. As it is, MIAB installations are drifting one from another due to our administration of MIAB server. In the case of container, you would know that you and I are running the same base and our differences are only due to a handful config files that are stored separately.

Development is also very simple (and open source way) - you describe in a script (or scripts) how your changes are applied to the base container image. So, you ad your apt-get and other commands into a single file and then submit PRs to that file or other appropriate scripts. Basically, similar to https://mailinabox.email/setup.sh

Security: Yes, our boxes are exposed and security is important. To keep things tidy, MIAC has to be updated as well as the host (let’s say docker host). Now, Docker host would have to be updated via host regular OS mechanisms, whatever your OS of choice is. If your OS supports automated updates that can be employed here as well.

MIAC would have to be updated through container update. For updating containers, on user side, I am using GitHub - containrrr/watchtower: A process for automating Docker container base image updates. to keep all containers updated. It keeps an eye on the running containers and once in 24h checks if an update was published. If published, new container is pulled, old removed, new started. Has worked for me, as a user, very well for few years now. There is an occasional hiccup, but that’s usually related to some major change in the container (app specific).

I presume (speculation as not involved in it) that it requires some work on the developer side. On developer side, when container base image (let’s say Ubuntu 2204) is updated or any of the used packages are updated, you would have to rebuild and publish new version of the container. I would expect that this is automated as well.

Now, running in container, could also let you do few more things to improve security. You can run them rootless (root of container is mapped to some other UID), reduce number of used services (no need for SSH, for example). You can also link your container only to the services that it needs and separate network from the host network.

[hughsw]

Clarifying the proof-of-concept configuration

The setup should perhaps be called Mail-in-a-Box in an Ubuntu container. That is, the containerized MIAB is a complete MIAB with all its security configuration in place and operating. As I said, I only made two un-intrusive one-line changes to the setup code. It’s a complete MIAB, same code, same setup procedures, etc, but it’s running in an Ubuntu container rather than Ubuntu on the metal or the cloud host. You ssh into the system and run sudo mailinabox etc as usual. It automatically updates packages as usual. It does backup, etc etc.

What’s different from a developer and tester perspective is that you can snapshot the running container into a new image. So you have an image with a complete MIAB including test users etc. You can then run multiple test containers that start from that same image snapshot.

One way to view the containerization is that it makes an entire Ubuntu with its MIAB installation into a stand-alone entity that has object-like semantics: you can create it, you can look inside it, you can copy it, you can edit each copy separately, you can run and rerun tests from the same starting point. Etc.

When you start and stop the container you even see the logging of the systemd startup and shutdown. So, you’ve taken that single Ubuntu with MIAB that runs on the metal, and instead made it an object that can be manipulated, snapshotted, copied, etc.

[marko]

@hughsw, thank you for clarification. I guess, if MIAB will move or be available in container, even in such configuration as you describe, it would make sense to work towards cleaner container design. So, for example, the security updates would be handled through mechanisms that are usually used by containers. Same goes for partitioning config and user data into volumes and so on. It doesn’t have to be from a get-go, but it seems to be a logical step if MIAB will move in that direction.

[dwomack]

I am interested in this.
What is the memory overhead of Docker?