TL;DR - Encountered a few issues during the restore process on the new Ubuntu 22.04 LTS instance and issues with Roundcube web mail
-
Issue 1: During restore, had issues with writing to /home/user-data/ssl sub dir; seems Duplicity (backup application) can not backup or is not set to backup symbolic links. As a result, the restore throws an error, and the missing “file” (symbolic link) seems to break Nginx, causing a failure to start after running MIAB setup after the restore
-
To over come this issue, after initial MIAB install and before restoring, deleted the contents of the /home/user-data/ssl sub directory and completed the restore with no error messages
-
Issue 2: Restore corruption. After completing restore with no errors and running the MIAB setup again, email was not showing for any clients in Thunderbird (email client) or Roundcube (web mail access)
-
To overcome this issue, before running restore, had to copy the secret_key.txt from the old instance. After completing this, it was possible to see the email for various user accounts using Thunderbird but not Roundcube web mail
-
Issue 3: After completing the restore from backup without errors, and after completing the 2nd MIAB setup (seemingly without errors), when using Roundcube web mail, the session times out and eventually displays a 504 Gateway Time-out nginx error.
-
This issue was self-inflicted due to where my MIAB is located on my network
-
To overcome this issue by modifying the host file and adding local IP address with FQDN and “pretty” (aka short) name
-
Edited the host file at /etc/hosts (sudo nano /etc/hosts) and added the following to the existing entries
127.0.0.1 localhost
127.0.1.1 box
(local IP address) box.<domain.name> box
(public IP address) box.<domain.name>
-
Reboot required for this change to take affect
Expanded Info
Environment - MIAB 57a (aka old MIAB) is updated to the latest version with current security updates
- Old MIAB OS is Ubuntu Server 18.04 LTS and was not modified during initial setup or during it’s operation in production. This is a “vanilla” install.
- MIAB 60 (aka new MIAB) has been setup with Ubuntu server 22.04 LTS with a “vanilla” install, using the minimal install option during initial OS setup
- Both the new and old instances run in-house on my own hardware, each as a dedicated virtual machine instances with dedicated CPU, RAM and HDD space
- The VM host is XCP-ng current version (8.2.1) with a vanilla install, enterprise-grade configuration
Expanded Error Info
Issue 1 - During restore, had issues with writing to /home/user-data/ssl subdir; seems Duplicity (backup application) can not backup or is not set to backup symbolic links. As a result, the restore throws an error during restore and after running MIAB setup again after restore (sudo mailinabox)
Restore Error
Error ‘[Errno 17] File exists: b’/home/user-data/ssl/(redacted name)-20221207-4b64c829.pem’ → b’/home/user-data/ssl/ssl_certificate.pem’’ processing ssl/ssl_certificate.pem
Nginx Error after restore and running MIAB setup again
FAILED: service nginx restart
Job for nginx.service failed because the control process exited with error code.
See “systemctl status nginx.service” and “journalctl -xeu nginx.service” for details.
Additional Issue 1 Symptoms
- Thunderbird email client will see the MIAB server but will display a dialog box related to an unknown SSL identity error
- Roundcube web mail portal will not open as Nginx was not able to restart
- Nextcloud calendar and contacts will not open as Nginx was not able to restart
- No access to the MIAB Control Panel
Correction of Issue 1 - Steps Taken
After completing the first run of MIAB setup on the new system, delete the contents of the SSL sub directory in /home/user-data/ssl/ with command sudo rm -rf /home/user-data/ssl/* before completing the restore
Issue 2 - Restore data corruption. After completing restore with no errors and running the MIAB setup again and completing successfully, email was not showing for any clients in Thunderbird (email client) or Roundcube (web mail access).
Issue 2 Symptoms
- When attempting to log into a user account via Roundcube web mail, in the lower right corner a yellow wheel spins and message says there is no data
- When attempting to log into Nextcloud contacts or calendar, nothing is displayed
- When using Thunderbird email client, when pressing the “Get Messages” button is pushed, no data is found
- Can successfully log into the MIAB Admin Panel
- When running MIAB setup (sudo miailinabox) for a 3rd time, during the upgrade process, a critical error message is thrown halting the install process. The error message is displayed stating that the data structure was faulty or had been corrupted (was not able to get the exact verbiage)
Suspected Cause of Issue 2
- During initial setup of the new MIAB (v60 with Ubuntu 22.04 LTS), all the necessary files are created including the secret_key.txt with new secret key data. As this data is not the same as the secret_key.txt from the old MIAB instance, during the restore process and/or during normal operation, the related email and other files can not be properly decrypted and read. As a result is is necessary to import the secret_key.txt from the old MIAB instance before running the data recovery from backup files. It seems that the backup process essentially assumes a restore will happen on the same instance.
Correction of Issue 2
- On the new MIAB instance (v60 with Ubuntu 22.04 LTS), after running the initial first time setup of MIAB and before running the restore process, it is necessary to copy the secret_key.txt from the old MIAB to the new MIAB.
- Commands used to correct this issue
- Preserve the current secret_key.txt file by making a local copy (using the move command) with a different file name sudo mv /home/user-data/backup/secret_key.txt /home/user-data/backup/secret_key.txt.oem.bak
- Copy the secret_key.txt from the old MIAB to the new MIAB (using the move command) sudo mv /path/to/backup/location/secret_key.txt /home/user-data/backup/secret_key.txt Note this is a destructive command and will overwrite the current secret_key.txt file
- Continue with the restore and finally running MIAB install for the 2nd time after the restore has completed
Additional Note on Issue 2 and secret_key.txt
- Remembered there was an issue from 2020 when the secret_key.txt was being parsed for restore, only the first part of the data was necessary for recovery of the data. The “extra” data not used in the secret_key.txt is included for backward compatibility. Not sure if this is still the case, but as a result of this issue, I have always used just the first, top line of the secret_key.txt to restore the backups.
- Related issue here at Backups are encrypted using the wrong password #1810
Feature Request Based on Issue 2
- Request the necessary logic be added to the restore process to check the existing key against what is about to be restored. Seems to be especially necessary when upgrading an existing installation to a new OS version. Basically call duplicity to extract just the secret_key.txt from the backup, cat and compare the current installed secret_key.txt and then decide if it needs to be overwritten first before restoring the rest of the files.
- Request the necessary logic be added to detect and compare the contents of the ssl sub directory located in /home/user-data/ssl/ with the ability to deal with either deleting or overwriting the contents and if possible either editing, overwriting or regenerating existing symbolic links. If possible with Duplicity, have it grab symbolic links and restore them properly.
Issue 3 - After completing the restore from backup without errors, and after completing the 2nd MIAB setup (seemingly without errors), when using Roundcube web mail, the session times out and eventually displays a 504 Gateway Time-out Nginx error.
Issue 3 Symptoms
- Can’t use Roundcube web mail to access email, when attempting to log on, the Roundcube page is displayed properly, but after entering the username and password, the page eventually times out and the Nginx error is displayed “504 Gateway Time-out”
- Can access the MIAB Control Panel
- Can access the Nextcloud contacts and calendar web pages with out errors
- Can send and receive email when using an email client like Thunderbird
- Nginx error logs say " *[error] (number) upstream timed out (110: Unknown error) while reading response header upstream, client: (ip address of the client), server: (FQDN of the MIAB), request: “POST /mail/?_task=login HTTP/2.0”, upstream: “fastcgi://unix:/var/run/php/php8.0-fpm.sock”, host: (FQDN of the MIAB), referrer: “https://(FQDN of MIAB)/mail/”
- Nginx error logs also contain a warning related to ssl stapling [warn] 1004#1004: “ssl_stapling” ignored, issuer certificate not found for certificate “/home/user-data/ssl/ssl_certificate.pem”
- Seems the symbolic link recreated after running the restore process and MIAB install the 2nd time should have taken care of this one, not sure why we are seeing this one
Cause of Issue 3
- Issue has to do with my particular configuration of my internal network, DNS, where my MIAB sits in my network and how apparently Roundcube handles upstream (internal) calls to local MIAB resources. Because my particular MIAB instance sits on an internal private network, behind a firewall, with different split DNS for internal and external clients, the issue was revealed. This issue is essentially self-inflicted
Indepth Workings Causing the Issue: During login from Roundcube web mail, the session tries to pass information upstream (internally) to another part of the system, but can’t complete the upstream communications as the FQDN is called. In my particular case, the MIAB machine is sitting on a private network behind a firewall with the necessary ports to facilitate function are forwarded through the firewall. DNS on the firewall for the MIAB instance is configured to point to the external IP address (public IP address), while clients from the local network use DNS pointing to the local IP address (internal IP address). This does not allow the Roundcube part of MIAB to correctly resolve and connect to other internal resources to facilitate the correct functions. Essentially this issue was caused because of the configuration of MIAB in my system as it does not stationed on the internet like other instances hosted on VPS services like OVH, Digital Ocean, Linode, etc…
- Error log file from the Nginx provided the necessary clue as the issue is caused upstream when trying to call the FQDN of the MIAB (error log snipit) *[error] 12427#12427: 1 upstream timed out (110: Unknown error) while reading response header from upstream, client: (local client IP), server: box.<domain.name>, request: “POST /mail/?_task=login HTTP/2.0”, upstream: “fastcgi://unix:/var/run/php/php8.0-fpm.sock”, host: “box.<domain.name>”, referrer: “https://box.<domain.name>/mail/” and because the FQDN is called instead of a local pointer such as 127.0.0.1 or localhost, it’s proper function is dependent on external DNS configuration. In my case using split DNS for internal clients and external clients with different resolutions depending on where the traffic originates from.
- Assuming contact and calendar from Nextcloud and MIAB Control Panel make upstream (internal) calls in a different fashion or rely on another, different internal method for name and/or resource resolution and is why these could be opened and viewed successfully, where Roundcube past the initial login screen could not be opened and times out.
-
Other Notes: Possibly a deeper issue because of the use of TLS certificates that would be considered valid when accessing from external (because TLS cert was generated based on the external (public) IP address. If Roundcube relies on TLS + FQDN = external IP address, it will always break
Correction of Issue 3
- Corrected this issue by modifying the host file and adding local IP address with FQDN and “pretty” (aka short) name
- Edited the host file at /etc/hosts (sudo nano /etc/hosts) and added the following to the existing entries
127.0.0.1 localhost
127.0.1.1 box
(local IP address) box.<domain.name> box
(public IP address) box.<domain.name>
- Reboot required for this change to take affect
- Can access Roundcube web mail from external and internal IP addresses
Feature Request or Issue Mitigation or System Reliability Improvement Based on Issue 3
- If possible, change the way upstream (internal) resource calls are made by Roundcube within MIAB to mitigate this issue and improve system reliability