Version 60 for Ubuntu 22.04 is released

mouse · October 21, 2022, 7:05pm

Just wanted to add another confirmation, in place upgrade worked, had to run LTS upgrade twice to get to 22.04 and the above operation mentioned by Wherewolf before running setup script and it all went smoothly. Didn’t have to restore backup with duplicity or anything.

Full operation I performed

# backed up as mentioned in box migration instructions just incase
curl -s https://mailinabox.email/setup.sh | sudo bash
sudo apt update
sudo apt upgrade 
reboot 
sudo nano /etc/update-manager/release-upgrades
  - uncomment Prompt=lts , and comment Prompt=never
sudo do-release-upgrade
  - select no for every option on replacing configs and agree to remove outdated packages, confirm reboot
sudo do-release-upgrade
  - select no for every option on replacing configs and agree to remove outdated packages, confirm reboot
sudo mv /usr/local/lib/mailinabox/env/bin/pip3 /usr/local/lib/mailinabox/env/bin/pip3-old
sudo mv /usr/local/lib/mailinabox/env/bin/python3 /usr/local/lib/mailinabox/env/bin/python3-old
sudo ln -s /usr/bin/pip3 /usr/local/lib/mailinabox/env/bin/pip3
sudo ln -s /usr/bin/python3 /usr/local/lib/mailinabox/env/bin/python3
curl -s https://mailinabox.email/setup.sh | sudo bash
  - just smash that enter button
sudo nano /etc/update-manager/release-upgrades
  - uncomment Prompt=never , and comment Prompt=lts

Love this mail management service, thanks to all the people past and present who have contributed to the project.

mouse · October 21, 2022, 7:07pm

Had no issue with NextCloud retaining files and contacts with in place upgrade risk I took, all seems to be there.

gaobo · October 22, 2022, 5:37am

A little bit buggy. Still struggling the way out.

tdahbura · October 22, 2022, 3:05pm

I tried a test upgrade (using a new ubuntu 22.04 host). After the install of MIAB 60 and the restore I found the following error message shows up after the final sudo mailinabox command was run:
All tables already up to date!
Nextcloud is already latest version
Error: stepping, UNIQUE constraint failed: oc_users_external.uid, oc_users_external.backend (19)

This error continues to show up anytime the mailinabox installer runs… Any ideas what this could be? I did not see this message on the initial install on the clean host. It appeared after the restore and subsequent run.

I also think it is critical to let folks know that they should run a sudo rm -rf /home/user-data/ssl/* command on the new host host prior to trying to use the duplicity restore command as it made a mess with the existing SSL files from the initial install and not being able to replace them.

Just wanted to report it out for folks.

alento · October 22, 2022, 3:11pm

I actually submitted a pull request to add this to the “Moving to a new Box” section in the maintenance guide.

Mandrake · October 22, 2022, 6:19pm

Well, I managed to get z-push to work, but I had to hack my way around.

The bottom line is, z-push 2.6.2 requires php 7.4, there’s no way around it, (see this comment for an effort to migrate to 8.x on another project).

So, what I did was to install the required 7.4 packages on the box, and manually change some configurations.

I installed these packages:

php7.4-cli
php7.4-common
php7.4-curl
php7.4-fpm
php7.4-imap
php7.4-json
php7.4-mbstring
php7.4-opcache
php7.4-readline
php7.4-soap
php7.4-xml

After that, I manually added a 7.4 upstream to nginx’s local.conf (yes, I’m aware this is not ideal):

upstream php7.4-fpm {
        server unix:/var/run/php/php7.4-fpm.sock;
}

And modified any reference to php-fpm for any z-push location (example):

        # Z-Push (Microsoft Exchange ActiveSync)
        location /Microsoft-Server-ActiveSync {
                include /etc/nginx/fastcgi_params;
                fastcgi_param SCRIPT_FILENAME /usr/local/lib/z-push/index.php;
                fastcgi_param PHP_VALUE "include_path=.:/usr/share/php:/usr/share/pear:/usr/share/awl/inc";
                fastcgi_read_timeout 630;
                fastcgi_pass php7.4-fpm;

                # Outgoing mail also goes through this endpoint, so increase the maximum
                # file upload limit to match the corresponding Postfix limit.
                client_max_body_size 128M;
        }
        location ~* ^/autodiscover/autodiscover.xml$ {
                include fastcgi_params;
                fastcgi_param SCRIPT_FILENAME /usr/local/lib/z-push/autodiscover/autodiscover.php;
                fastcgi_param PHP_VALUE "include_path=.:/usr/share/php:/usr/share/pear:/usr/share/awl/inc";
                fastcgi_pass php7.4-fpm;
        }

Lastly, I manually modified z-push-admin.php and z-push-top.php to use php7.4 by changing the first line to:

#!/usr/bin/env php7.4

This was only needed to be able to run the tools directly, otherwise you can still run them using php7.4:

$ php7.4 /usr/local/lib/z-push/z-push-admin.php -a lastsync

EDIT: Take this changes with a grain of salt. So far, everything seems to be working, but I’m not sure if I broke anything else on my box, or what are the consequences of running multiple PHP versions in the long run

Bore_God_of_Blunder · October 22, 2022, 6:30pm

Thank you to everyone who made this happen!

Everything went quite smoothly with my upgrade process. My only couple of comments are:

First after my upgrade I had to set AAA records before the Let’s Encrypt script would work (I use external dns), whereas before I had only had A records and that was working successfully. I have a faint recollection of fighting with it a little on my first install and starting with the box doing the DNS before changing, so perhaps the order I did it in was unusual and allowed that to work somehow.

Second (and maybe this is too obvious to bother mentioning, but it didn’t occur to me until a day later, so I figure just in case someone else overlooked it too), your secret_key.txt file will change as part of the process and needs to be retained securely again.

I wonder if it makes sense to recommend that people copy their old files to /home/user-data first and then run the install script (which I assume might solve the ssl directory needing to be replaced as well as preserve the old secret_key.txt).

Thanks again for all your hard work. It is very much appreciated.

cinergi · October 22, 2022, 8:18pm

I managed to get Z-Push to work with a few changes to the PHP files, without resorting to installing PHP 7.4:

Edit /usr/local/lib/z-push/backend/ipcsharedmemory/ipcsharedmemoryprovider.php as per this post.
Edit /usr/local/lib/z-push/policies.ini as per the above post.
Add the symbolic links as per this post.
Edit /usr/local/lib/z-push/backend/caldav/caldav.php as per the end of this thread.

Another (minor) issue that I encountered is that on the admin GUI’s External DNS page, the table of DNS records doesn’t load. Not a big deal since the raw zonefile can still be displayed, but just want to highlight it for @JoshData for a future release.

Thanks for all the hard work! I love mailinabox.

jbdubbs · October 23, 2022, 12:56am

So I did the actual upgrade a couple days ago. Everything ended up working except for DNS, which turned out to be a nightmare. I haven’t quite figured it out, but even though I didn’t change IPs, my DNS was being propogated as “not set” across the whole of the internet. That won’t do.

Short term fix was to take an hour and manually type all the DNS settings into Namecheap directly and remove my custom glue servers.

I’ll go back into it later so I can hopefully switch back.

Anybody else have any strange DNS issues?

amp51 · October 23, 2022, 1:19am

TL;DR - Encountered a few issues during the restore process on the new Ubuntu 22.04 LTS instance and issues with Roundcube web mail

Issue 1: During restore, had issues with writing to /home/user-data/ssl sub dir; seems Duplicity (backup application) can not backup or is not set to backup symbolic links. As a result, the restore throws an error, and the missing “file” (symbolic link) seems to break Nginx, causing a failure to start after running MIAB setup after the restore
To over come this issue, after initial MIAB install and before restoring, deleted the contents of the /home/user-data/ssl sub directory and completed the restore with no error messages
Issue 2: Restore corruption. After completing restore with no errors and running the MIAB setup again, email was not showing for any clients in Thunderbird (email client) or Roundcube (web mail access)
To overcome this issue, before running restore, had to copy the secret_key.txt from the old instance. After completing this, it was possible to see the email for various user accounts using Thunderbird but not Roundcube web mail
Issue 3: After completing the restore from backup without errors, and after completing the 2nd MIAB setup (seemingly without errors), when using Roundcube web mail, the session times out and eventually displays a 504 Gateway Time-out nginx error.
This issue was self-inflicted due to where my MIAB is located on my network
To overcome this issue by modifying the host file and adding local IP address with FQDN and “pretty” (aka short) name
Edited the host file at /etc/hosts (sudo nano /etc/hosts) and added the following to the existing entries
127.0.0.1 localhost
127.0.1.1 box
(local IP address) box.<domain.name> box
(public IP address) box.<domain.name>
Reboot required for this change to take affect

Expanded Info

Environment - MIAB 57a (aka old MIAB) is updated to the latest version with current security updates

Old MIAB OS is Ubuntu Server 18.04 LTS and was not modified during initial setup or during it’s operation in production. This is a “vanilla” install.
MIAB 60 (aka new MIAB) has been setup with Ubuntu server 22.04 LTS with a “vanilla” install, using the minimal install option during initial OS setup
Both the new and old instances run in-house on my own hardware, each as a dedicated virtual machine instances with dedicated CPU, RAM and HDD space
The VM host is XCP-ng current version (8.2.1) with a vanilla install, enterprise-grade configuration

Expanded Error Info

Issue 1 - During restore, had issues with writing to /home/user-data/ssl subdir; seems Duplicity (backup application) can not backup or is not set to backup symbolic links. As a result, the restore throws an error during restore and after running MIAB setup again after restore (sudo mailinabox)

Restore Error
Error ‘[Errno 17] File exists: b’/home/user-data/ssl/(redacted name)-20221207-4b64c829.pem’ → b’/home/user-data/ssl/ssl_certificate.pem’’ processing ssl/ssl_certificate.pem

Nginx Error after restore and running MIAB setup again
FAILED: service nginx restart
Job for nginx.service failed because the control process exited with error code.
See “systemctl status nginx.service” and “journalctl -xeu nginx.service” for details.

Additional Issue 1 Symptoms

Thunderbird email client will see the MIAB server but will display a dialog box related to an unknown SSL identity error
Roundcube web mail portal will not open as Nginx was not able to restart
Nextcloud calendar and contacts will not open as Nginx was not able to restart
No access to the MIAB Control Panel

Correction of Issue 1 - Steps Taken
After completing the first run of MIAB setup on the new system, delete the contents of the SSL sub directory in /home/user-data/ssl/ with command sudo rm -rf /home/user-data/ssl/* before completing the restore

Issue 2 - Restore data corruption. After completing restore with no errors and running the MIAB setup again and completing successfully, email was not showing for any clients in Thunderbird (email client) or Roundcube (web mail access).

Issue 2 Symptoms

When attempting to log into a user account via Roundcube web mail, in the lower right corner a yellow wheel spins and message says there is no data
When attempting to log into Nextcloud contacts or calendar, nothing is displayed
When using Thunderbird email client, when pressing the “Get Messages” button is pushed, no data is found
Can successfully log into the MIAB Admin Panel
When running MIAB setup (sudo miailinabox) for a 3rd time, during the upgrade process, a critical error message is thrown halting the install process. The error message is displayed stating that the data structure was faulty or had been corrupted (was not able to get the exact verbiage)

Suspected Cause of Issue 2

During initial setup of the new MIAB (v60 with Ubuntu 22.04 LTS), all the necessary files are created including the secret_key.txt with new secret key data. As this data is not the same as the secret_key.txt from the old MIAB instance, during the restore process and/or during normal operation, the related email and other files can not be properly decrypted and read. As a result is is necessary to import the secret_key.txt from the old MIAB instance before running the data recovery from backup files. It seems that the backup process essentially assumes a restore will happen on the same instance.

Correction of Issue 2

On the new MIAB instance (v60 with Ubuntu 22.04 LTS), after running the initial first time setup of MIAB and before running the restore process, it is necessary to copy the secret_key.txt from the old MIAB to the new MIAB.
Commands used to correct this issue
Preserve the current secret_key.txt file by making a local copy (using the move command) with a different file name sudo mv /home/user-data/backup/secret_key.txt /home/user-data/backup/secret_key.txt.oem.bak
Copy the secret_key.txt from the old MIAB to the new MIAB (using the move command) sudo mv /path/to/backup/location/secret_key.txt /home/user-data/backup/secret_key.txt Note this is a destructive command and will overwrite the current secret_key.txt file
Continue with the restore and finally running MIAB install for the 2nd time after the restore has completed

Additional Note on Issue 2 and secret_key.txt

Remembered there was an issue from 2020 when the secret_key.txt was being parsed for restore, only the first part of the data was necessary for recovery of the data. The “extra” data not used in the secret_key.txt is included for backward compatibility. Not sure if this is still the case, but as a result of this issue, I have always used just the first, top line of the secret_key.txt to restore the backups.
Related issue here at Backups are encrypted using the wrong password #1810

Feature Request Based on Issue 2

Request the necessary logic be added to the restore process to check the existing key against what is about to be restored. Seems to be especially necessary when upgrading an existing installation to a new OS version. Basically call duplicity to extract just the secret_key.txt from the backup, cat and compare the current installed secret_key.txt and then decide if it needs to be overwritten first before restoring the rest of the files.
Request the necessary logic be added to detect and compare the contents of the ssl sub directory located in /home/user-data/ssl/ with the ability to deal with either deleting or overwriting the contents and if possible either editing, overwriting or regenerating existing symbolic links. If possible with Duplicity, have it grab symbolic links and restore them properly.

Issue 3 - After completing the restore from backup without errors, and after completing the 2nd MIAB setup (seemingly without errors), when using Roundcube web mail, the session times out and eventually displays a 504 Gateway Time-out Nginx error.

Issue 3 Symptoms

Can’t use Roundcube web mail to access email, when attempting to log on, the Roundcube page is displayed properly, but after entering the username and password, the page eventually times out and the Nginx error is displayed “504 Gateway Time-out”
Can access the MIAB Control Panel
Can access the Nextcloud contacts and calendar web pages with out errors
Can send and receive email when using an email client like Thunderbird
Nginx error logs say " *[error] (number) upstream timed out (110: Unknown error) while reading response header upstream, client: (ip address of the client), server: (FQDN of the MIAB), request: “POST /mail/?_task=login HTTP/2.0”, upstream: “fastcgi://unix:/var/run/php/php8.0-fpm.sock”, host: (FQDN of the MIAB), referrer: “https://(FQDN of MIAB)/mail/”
Nginx error logs also contain a warning related to ssl stapling [warn] 1004#1004: “ssl_stapling” ignored, issuer certificate not found for certificate “/home/user-data/ssl/ssl_certificate.pem”
Seems the symbolic link recreated after running the restore process and MIAB install the 2nd time should have taken care of this one, not sure why we are seeing this one

Cause of Issue 3

Issue has to do with my particular configuration of my internal network, DNS, where my MIAB sits in my network and how apparently Roundcube handles upstream (internal) calls to local MIAB resources. Because my particular MIAB instance sits on an internal private network, behind a firewall, with different split DNS for internal and external clients, the issue was revealed. This issue is essentially self-inflicted

Indepth Workings Causing the Issue: During login from Roundcube web mail, the session tries to pass information upstream (internally) to another part of the system, but can’t complete the upstream communications as the FQDN is called. In my particular case, the MIAB machine is sitting on a private network behind a firewall with the necessary ports to facilitate function are forwarded through the firewall. DNS on the firewall for the MIAB instance is configured to point to the external IP address (public IP address), while clients from the local network use DNS pointing to the local IP address (internal IP address). This does not allow the Roundcube part of MIAB to correctly resolve and connect to other internal resources to facilitate the correct functions. Essentially this issue was caused because of the configuration of MIAB in my system as it does not stationed on the internet like other instances hosted on VPS services like OVH, Digital Ocean, Linode, etc…

Error log file from the Nginx provided the necessary clue as the issue is caused upstream when trying to call the FQDN of the MIAB (error log snipit) *[error] 12427#12427: 1 upstream timed out (110: Unknown error) while reading response header from upstream, client: (local client IP), server: box.<domain.name>, request: “POST /mail/?_task=login HTTP/2.0”, upstream: “fastcgi://unix:/var/run/php/php8.0-fpm.sock”, host: “box.<domain.name>”, referrer: “https://box.<domain.name>/mail/” and because the FQDN is called instead of a local pointer such as 127.0.0.1 or localhost, it’s proper function is dependent on external DNS configuration. In my case using split DNS for internal clients and external clients with different resolutions depending on where the traffic originates from.
Assuming contact and calendar from Nextcloud and MIAB Control Panel make upstream (internal) calls in a different fashion or rely on another, different internal method for name and/or resource resolution and is why these could be opened and viewed successfully, where Roundcube past the initial login screen could not be opened and times out.
Other Notes: Possibly a deeper issue because of the use of TLS certificates that would be considered valid when accessing from external (because TLS cert was generated based on the external (public) IP address. If Roundcube relies on TLS + FQDN = external IP address, it will always break

Correction of Issue 3

Corrected this issue by modifying the host file and adding local IP address with FQDN and “pretty” (aka short) name
Edited the host file at /etc/hosts (sudo nano /etc/hosts) and added the following to the existing entries
127.0.0.1 localhost
127.0.1.1 box
(local IP address) box.<domain.name> box
(public IP address) box.<domain.name>
Reboot required for this change to take affect
Can access Roundcube web mail from external and internal IP addresses

Feature Request or Issue Mitigation or System Reliability Improvement Based on Issue 3

If possible, change the way upstream (internal) resource calls are made by Roundcube within MIAB to mitigate this issue and improve system reliability

bobaxx · October 23, 2022, 7:31am

thx for the answer, and im waiting for dns propagation

bobaxx · October 23, 2022, 7:39am

Update from ubuntu 18.04 to 22.04 seems to be good except 2 things at the end of installation/update script :

Just after nextcloud installation/update :

Error: stepping, UNIQUE constraint failed: oc_users_external.uid, oc_users_external.backend (19)

and at the very end of the installation/update :

Command ‘[‘ssh-keyscan’, ‘-t’, ‘rsa,dsa,ecdsa,ed25519’, ‘-p’, ‘XXXX’, ‘localhost’]’ returned non-zero exit status 1.

(ED25519 is my type of ssh key)

Should i ignore those warnings ? like i said, im still waiting for dns propagation (cant test everything now)

alento · October 23, 2022, 10:44am

Make sure that you restart NSD, or it will be an endless wait.

sudo service nsd restart

alento · October 23, 2022, 11:22am

2 posts were split to a new topic: How is this possible?

bobaxx · October 23, 2022, 11:15am

sure, some restart has been done since installation, it is marked as “active running”

i’am approaching the 48h dns resolving deadlines given by ovh. If not successufl, im going to do another dns resolving approach (using ovh dns i think instead of MiaB box dns)

alento · October 23, 2022, 11:18am

It doesn’t matter what the status page says. A bug was introduced into the latest version. Your DNS will NOT propagate until you issue the command sudo service nsd restart from the command line. You also need to do the same if you make any changes such as adding a domain, custom DNS record, etc.

bobaxx · October 23, 2022, 11:20am

Ok, i have just done it, i will see what happen next

bobaxx · October 23, 2022, 4:01pm

Something must be wrong with dns resolving on my box, when i read syslog in /var/log :

Oct 23 17:49:25 box named[18014]: timed out resolving ‘box.DOMAIN_NAME/A/IN’: OLD_PUBLIC@IP#53
Oct 23 17:49:30 box named[18014]: REFUSED unexpected RCODE resolving ‘_.box.DOMAIN_NAME/A/IN’: NEW_PUBLIC@IP#53
Oct 23 17:50:06 box named[18014]: REFUSED unexpected RCODE resolving ‘autodiscover.DOMAIN_NAME/AAAA/IN’: NEW_PUBLIC@IP#53
Oct 23 17:50:06 box named[18014]: timed out resolving ‘autodiscover.DOMAIN_NAME/A/IN’: OLD_PUBLIC@IP#53
Oct 23 17:50:11 box named[18014]: REFUSED unexpected RCODE resolving ‘mta-sts.DOMAIN_NAME/A/IN’: NEW_PUBLIC@IP#53
Oct 23 17:50:11 box named[18014]: timed out resolving ‘autodiscover.DOMAIN_NAME/AAAA/IN’: OLD_PUBLIC@IP#53
Oct 23 17:50:16 box named[18014]: REFUSED unexpected RCODE resolving ‘mta-sts.DOMAIN_NAME/AAAA/IN’: NEW_PUBLIC@IP#53
Oct 23 17:50:16 box named[18014]: timed out resolving ‘mta-sts.metalbox.ovh/A/IN’: OLD_PUBLIC@IP#53
Oct 23 17:50:21 box named[18014]: REFUSED unexpected RCODE resolving ‘www.DOMAIN_NAME/A/IN’: NEW_PUBLIC@IP#53
Oct 23 17:50:22 box named[18014]: timed out resolving ‘mta-sts.DOMAIN_NAME/AAAA/IN’: OLD_PUBLIC@IP#53
Oct 23 17:50:26 box named[18014]: REFUSED unexpected RCODE resolving ‘www.DOMAIN_NAME/AAAA/IN’: NEW_PUBLIC@IP#53
Oct 23 17:50:27 box named[18014]: timed out resolving ‘www.DOMAIN_NAME/A/IN’: OLD_PUBLIC@IP#53
Oct 23 17:50:32 box named[18014]: timed out resolving ‘www.DOMAIN_NAME/AAAA/IN’: OLD_PUBLIC@IP#53

i deactivate UFW to be sure, just for a moment, i see that bind9 is activated beside nsd, is there any conflict with that ?

i tried to stop bind9, but i dont get anymore dns resolving then (cant get apt update so …)

i am nearly 48h dns update resolving with my DNS ISP, i didnt remember last time it took so much time

it seems my DNS resolving is refused according to the log file

jbdubbs · October 23, 2022, 8:28pm

Had the same issue, just ended up going with my registrar DNS manually. Waiting to see how this plays out.

cinergi · October 25, 2022, 4:05am

I just noticed that fail2ban was disabled post-upgrade for some reason, at least on my box. I re-enabled and started it, and it seems to be working fine. Folks may want to check - I had some Iranian-based IPs hammering on my SMTP port every 10 seconds, which is what led me to investigate.