Ssl trouble upgrading to v65 on ubuntu 22.04 breaks nginx

gris · October 30, 2023, 4:53pm

systemctl status nginx.service

Oct 30 17:33:15 some.domain.eu systemd[1]: Starting A high performance web server and a reverse proxy server…
Oct 30 17:33:15 some.domain.eu nginx[14625]: nginx: [warn] “ssl_stapling” ignored, issuer certificate not found for certificate "/home/use>
Oct 30 17:33:15 some.domain.eu nginx[14625]: nginx: [emerg] cannot load certificate "/home/user-data/ssl/some.domain.eu-20240128>
Oct 30 17:33:15 some.domain.eu nginx[14625]: nginx: configuration file /etc/nginx/nginx.conf test failed
Oct 30 17:33:15 some.domain.eu systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Oct 30 17:33:15 some.domain.eu systemd[1]: nginx.service: Failed with result ‘exit-code’.
Oct 30 17:33:15 some.domain.eu systemd[1]: Failed to start A high performance web server and a reverse proxy server.

Yes, I did do

sudo rm -rf /home/user-data/ssl/*

obviously, some.domain.eu/admin is not available if nginx will not even start.

Any tips?

dms · October 30, 2023, 5:24pm

I have a few questions for clarification.

What are you upgrading from?
What steps did you perform to get to this point?
Why did you delete the SSL directory contents?

gris · October 30, 2023, 5:31pm

I ran mail-in-a-box on ubuntu 18 without trouble for a couple of years.
I installed a “clean” ubuntu 22.04 using the instructions on

then I restored the backup of the ubuntu 18 mail-in-a-box environment onto ubuntu 22.04.

Thanks for potentially helping!

dms · October 30, 2023, 6:24pm

So, you (1) built a new 22.04 box, (2) installed MiaB, (3) deleted the ssl directory, (4) restored your duplicity backup from 18.04, (5) re-ran sudo mailinbox to complete configuration.

And after all that you’re getting this error? If I remember correctly, the SSL certs should have been part of the backup and should have been restored in step 4. Are there any files in your SSL directory, or is it empty?

gris · October 30, 2023, 6:39pm

Thanks!

Yes, that is correct. Restoring the backup does restore old certificates, but now after the “new” install of MIAB and after having done

sudo rm -rf /home/user-data/ssl/* as per instructions

gris@some:/home/user-data/ssl$ ls -la
total 24
drwxr-xr-x 2 root root 4096 Oct 30 17:32 .
drwxr-xr-x 10 user-data user-data 4096 Oct 30 16:20 …
-rw-r–r-- 1 root root 424 Oct 30 17:32 dh2048.pem
lrwxrwxrwx 1 root root 68 Oct 30 17:32 ssl_certificate.pem → /home/user-data/ssl/some.domain.eu-selfsigned-20231030.pem
-rw------- 1 root root 1704 Oct 30 17:32 ssl_private_key.pem
-rw-r–r-- 1 root root 1034 Oct 30 17:32 some.domain.eu.eu-selfsigned-20231030.pem
gris@some:/home/user-data/ssl$

dms · October 30, 2023, 6:44pm

Did you delete the SSL directory before or after restoring the backup?

gris · October 30, 2023, 6:46pm

I have tried sudo mailinabox several times after restoring the backup. Each time I have done the recommended

sudo rm -rf /home/user-data/ssl/*

just before sudo mailinabox

dms · October 30, 2023, 6:49pm

What I don’t understand is why your nginx log message is trying to load /home/user-data/ssl/some.domain.eu-20240128, but your directory listing shows that you only have /home/user-data/ssl/some.domain.eu-selfsigned-20231030.pem present. Your current file is apparently “self-signed” which is incorrect. That’s what was in your duplicity backup?

dms · October 30, 2023, 6:50pm

I’m working off of 6-month old memory, but I believe that is incorrect. You are supposed to remove the contents of the SSL directory just once, before restoring the backup.

Like in the steps above that I asked about; 3) delete ssl directory, 4) restore backup.

gris · October 30, 2023, 6:51pm

No, MIAB on ubintu 18 had “proper” let’s encrypt certs

gris · October 30, 2023, 6:55pm

Thanks for your tips!

It looks as if I will have to

do a fresh ubuntu 22.04 install again
install MIAB from scratch
delete ssl directory contents
restore backup

Thanks again for your help!
I will post the result of this in about twelve hours or so.

dms · October 30, 2023, 6:56pm

Here’s what I recommend you try. Restore the backup to a different location (somewhere on this server, away from /home/user-data, or on another linux system). Remove the contents of the ssl directory on your MiaB server. manually copy the contents of SSL directory from the backup you just restored somewhere else to the ssl directory on your MiaB server. Run sudo mailinabox.

In other words, I don’t believe all is lost. But you need to get a copy of the ssl directory from your backup onto the new server in the correct location. Hopefully, if that’s all that’s wrong, the server should start working.

gris · October 30, 2023, 7:10pm

That is a good tip! As I will not have to do another fresh ubuntu 22.04 for now.

I will

restore the backup into (my own) /home/gris/temp-restored-backup
manually copy the contents of SSL directory from that restored backup to /home/user-data/ssl/
run sudo mailinabox
post the results

gris · October 30, 2023, 7:27pm

restoring backup as we speak.

the trouble could be caused by the let’s encrypt certs having expired and the old instance of MIAB failing to get new ones. That is why I decided to do a fresh ubuntu 22.04

gris · October 30, 2023, 7:35pm

Alas!

Mail-in-a-Box Version: v65

Updating system packages…
Installing system packages…
Initializing system random number generator…
Firewall is active and enabled on system startup
Synchronizing state of fail2ban.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable fail2ban
Installing nsd (DNS server)…
Installing Postfix (SMTP server)…
Installing Dovecot (IMAP server)…
Installing OpenDKIM/OpenDMARC…
Installing SpamAssassin…

Installing Nginx (web server)…

FAILED: service nginx restart

Job for nginx.service failed because the control process exited with error code.
See “systemctl status nginx.service” and “journalctl -xeu nginx.service” for details.

root@some:/home#

so same old unfortunately

dms · October 30, 2023, 7:49pm

what is the reason for not starting from the logs?
did you begin by deleting the contents of your /home/user-data/ssl directory?
what is in the ssl directory now?
did you confirm the new ssl directory is “identical” to the directory from the backup?
how old was the backup? (I ask because you were concerned the certs were expired)
what MaiB version on 18.04 is that backup from?

gris · October 30, 2023, 8:08pm

nginx: [emerg] cannot load certificate "/home/user-data/ssl/some.domain.eu-20240128…
nginx: configuration file /etc/nginx/nginx.conf test failed
no.

root@some:/home/user-data/ssl# total 144
drwxr-xr-x 3 root drwxr-xr-x -rw-r–r-- 1 root drwxr-xr-x 5 root lrwxrwxrwx 1 root -rw------- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root -rw-r–r-- 1 root ls -la
root 4096 Oct 30 20:31 .
10 user-data user-data 4096 Oct 30 16:20 …
root 424 Oct 30 20:31 dh2048.pem
root 4096 Oct 30 20:31 lets_encrypt
root 66 Oct 30 20:31 ssl_certificate.pem → /home/user-data/ssl/some.domain.eu-20231028-0303b551.pem
root 1679 Oct 30 20:31 ssl_private_key.pem
root 5600 Oct 30 20:31 domain.eu-20220801-5a503057.pem
root 5746 Oct 30 20:31 domain.eu-20221016-0300a4aa.pem
root 5746 Oct 30 20:31 domain.eu-20221230-5287dae4.pem
root 5742 Oct 30 20:31 domain.eu-20230315-a97c48f8.pem
root 5742 Oct 30 20:31 domain.eu-20230529-5dea455d.pem
root 5746 Oct 30 20:31 domain.eu-20230813-c43f7f08.pem
root 5669 Oct 30 20:31 domain.eu-20231027-4543b047.pem
root 5811 Oct 30 20:31 some.domain.eu-20220801-abdeb3ad.pem
root 5665 Oct 30 20:31 some.domain.eu-20221017-a705efce.pem
root 5665 Oct 30 20:31 some.domain.eu-20221231-ce7ce2d4.pem
root 5665 Oct 30 20:31 some.domain.eu-20230316-818e94f7.pem
root 5665 Oct 30 20:31 some.domain.eu-20230530-ecf464d9.pem
root 5661 Oct 30 20:31 some.domain.eu-20230814-ce953234.pem
root 5588 Oct 30 20:31 some.domain.eu-20231028-0303b551.pem
root 1034 Oct 30 20:31 some.domain.eu-selfsigned-20220503.pem
root 1034 Oct 30 17:32 some.domain.eu-selfsigned-20231030.pem

yes.
20230922 was the latest
version v57a

dms · October 30, 2023, 8:23pm

no

Why not? Just to be clear, you MUST make sure the ssl directory is empty before restoring the files from the backup. There may be no files in this directory. I still see selfsigned certs in that directory, for example. At this point you may have to start over with the installation to make sure you’ve cleared out all the cruft that should not be there. Fortunately, you have your backup. Guard it safely. I’m curious as to why your backup is so old, but that’s neither here nor there at this point.

gris · October 30, 2023, 8:26pm

Thanks a bunch for all your efforts to help me!

I will install ubuntu 22.04 from scratch.

I will report again “tomorrow” or some 12 hours from now.

Thanks again!

dms · October 30, 2023, 8:31pm

I do have another question about the nginx error. It’s trying to load a file that does not exist in your restored directory, “/home/user-data/ssl/some.domain.eu-20240128…”. Which I don’t understand and makes me wonder if there’s another underlying problem here. I looks as though the system is configured to use certs that have since been deleted.

Also, would it be possible to make a newer backup to use, or is the old system no longer functioning properly?

Ssl trouble upgrading to v65 on ubuntu 22.04 breaks nginx

FAILED: service nginx restart

Job for nginx.service failed because the control process exited with error code. See “systemctl status nginx.service” and “journalctl -xeu nginx.service” for details.

Job for nginx.service failed because the control process exited with error code.
See “systemctl status nginx.service” and “journalctl -xeu nginx.service” for details.