Upgraded to Ubuntu 20.04 -- nginx stopped working

ecnal · December 17, 2020, 8:33pm

I stupidly upgraded to Ubuntu 20.04. I didn’t do my homework and just did it. Mail-in-a-box worked fine for a week or so, but now the webserver has stopped working and it will not start. I even brought up a new install, moved over the backup, and nginx still won’t start. I realize that this is my fault. You can chastise me if you want. I was stupid. I would still like to know if there is anything that I can do to rectify this situation.

I tried to include the nginx error logs, but it throws me a notification that “new users can only include two links in posts” or something along those lines.

ecnal · December 17, 2020, 8:35pm

I removed the URLs, so hopefully this posts.

error logs:

ecnal@box:~$ systemctl status nginx.service

● nginx.service - A high performance web server and a reverse proxy server

Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)

Active: failed (Result: exit-code) since Thu 2020-12-17 12:04:56 MST; 10s ago

Docs: man:nginx(8)

Process: 1588 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)

Process: 1599 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)

Dec 17 12:04:54 nginx[1599]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)

Dec 17 12:04:54 box.ecnal.me nginx[1599]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)

Dec 17 12:04:55 nginx[1599]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)

Dec 17 12:04:55 nginx[1599]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)

Dec 17 12:04:55 nginx[1599]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)

Dec 17 12:04:55 nginx[1599]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)

Dec 17 12:04:56 nginx[1599]: nginx: [emerg] still could not bind()

Dec 17 12:04:56 systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE

Dec 17 12:04:56 systemd[1]: nginx.service: Failed with result ‘exit-code’.

Dec 17 12:04:56 systemd[1]: Failed to start A high performance web server and a reverse proxy server.

ecnal · December 17, 2020, 8:38pm

And just for clarification: the mail server itself is still operational – sends and receives email just fine. That I can tell, it is only nginx that is having issues.

alento · December 18, 2020, 2:18am

Is anything else besides MiaB installed on the server?

Also what is the output of apache2 -v ? @ecnal

ecnal · December 18, 2020, 2:32am

It is a Proxmox VM and it only runs Mail-in-a-box.

ecnal@box:~$ apache2 -v

Server version: Apache/2.4.41 (Ubuntu)

Server built: 2020-08-12T19:46:17

I can turn up the new VM and get the output from that as well, if you want it? But it is literally a brand new install of Ubuntu 18.04 (also a VM) with only Mail-in-a-box installed, and the backup moved over.

alento · December 18, 2020, 2:34am

On the Ubuntu 20.04 @ecnal
sudo apt remove apache2
sudo systemctl start nginx.service

I am going to guess that the issue on the 18.04 is different??? What OS version did you install specifically? (ISO image filename?)

ecnal · December 18, 2020, 2:42am

That worked. nginx is back online and web access is working.

Image: ubuntu-20.04.1-live-server-amd64.iso

alento · December 18, 2020, 2:43am

On second thought that may be the issue with the 18.04 VM as well … check with the same command and if you get a response take the same actions.

ecnal · December 18, 2020, 2:44am

Wait, that is not true. Sorry. I did a do-release-upgrade.

alento · December 18, 2020, 2:45am

Moot point anyways … your end goal is to get it running again on the 18.04 VM.

ecnal · December 18, 2020, 2:45am

But let me shutdown this VM and turn up the new install and see if that works. They are on the same IP.

alento · December 18, 2020, 2:46am

Ok. It is going on 4AM for me and I am fading fast, so I may not respond again until tomorrow.

ecnal · December 18, 2020, 3:01am

apache2 is not installed on the other machine.

ecnal@box:~$ systemctl status nginx.service

● nginx.service - A high performance web server and a reverse proxy server

Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)

Active: failed (Result: exit-code) since Thu 2020-12-17 19:55:50 MST; 11s ago

Docs: man:nginx(8)

Process: 2920 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)

Dec 17 19:55:50 systemd[1]: Starting A high performance web server and a reverse proxy server…

Dec 17 19:55:50 nginx[2920]: nginx: [emerg] BIO_new_file("/home/user-data/ssl/ssl_certificate.pem") failed (SSL: error:02001002:system library:f

Dec 17 19:55:50 nginx[2920]: nginx: configuration file /etc/nginx/nginx.conf test failed

Dec 17 19:55:50 systemd[1]: nginx.service: Control process exited, code=exited status=1

Dec 17 19:55:50 systemd[1]: nginx.service: Failed with result ‘exit-code’.

Dec 17 19:55:50 systemd[1]: Failed to start A high performance web server and a reverse proxy server.

ecnal · December 18, 2020, 3:08am

ecnal@box:~$ journalctl -xe

Dec 17 19:55:50 sudo[2916]: pam_unix(sudo:session): session opened for user root by ecnal(uid=0)

Dec 17 19:55:50 systemd[1]: Starting A high performance web server and a reverse proxy server…

– Subject: Unit nginx.service has begun start-up

– Defined-By: systemd

– Support:

–

– Unit nginx.service has begun starting up.

Dec 17 19:55:50 nginx[2920]: nginx: [emerg] BIO_new_file("/home/user-data/ssl/ssl_certificate.pem") failed (SSL: error:02001002:system library:f

Dec 17 19:55:50 nginx[2920]: nginx: configuration file /etc/nginx/nginx.conf test failed

Dec 17 19:55:50 systemd[1]: nginx.service: Control process exited, code=exited status=1

Dec 17 19:55:50 systemd[1]: nginx.service: Failed with result ‘exit-code’.

Dec 17 19:55:50 systemd[1]: Failed to start A high performance web server and a reverse proxy server.

– Subject: Unit nginx.service has failed

– Defined-By: systemd

– Support:

–

– Unit nginx.service has failed.

–

– The result is RESULT.

Dec 17 19:55:50 sudo[2916]: pam_unix(sudo:session): session closed for user root

Dec 17 19:55:51 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=33:33:00:00:00:01:ce:8d:0e:a4:af:1c:86:dd SRC=fe80:0000:0000:0000:e976:9186:00b9:db6e DST=

Dec 17 19:56:01 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=45.129.33.152 DST=72.250.213.12 LEN=40 TOS=0

Dec 17 19:56:24 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=33:33:00:00:00:01:ce:8d:0e:a4:af:1c:86:dd SRC=fe80:0000:0000:0000:e976:9186:00b9:db6e DST=

Dec 17 19:56:27 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=45.129.33.53 DST=72.250.213.12 LEN=40 TOS=0x

Dec 17 19:56:31 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=45.129.33.179 DST=72.250.213.12 LEN=40 TOS=0

Dec 17 19:56:35 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=142.93.17.77 DST=72.250.213.12 LEN=40 TOS=0x

Dec 17 19:56:46 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=193.27.229.47 DST=72.250.213.12 LEN=40 TOS=0

Dec 17 19:56:55 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=123.59.120.136 DST=72.250.213.12 LEN=33 TOS=

Dec 17 19:56:59 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=33:33:00:00:00:01:ce:8d:0e:a4:af:1c:86:dd SRC=fe80:0000:0000:0000:e976:9186:00b9:db6e DST=

Dec 17 19:57:06 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=193.32.8.76 DST=72.250.213.12 LEN=438 TOS=0x

Dec 17 19:57:33 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=33:33:00:00:00:01:ce:8d:0e:a4:af:1c:86:dd SRC=fe80:0000:0000:0000:e976:9186:00b9:db6e DST=

Dec 17 19:57:43 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=89.248.165.7 DST=72.250.213.12 LEN=40 TOS=0x

Dec 17 19:57:46 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=92.53.65.52 DST=72.250.213.12 LEN=40 TOS=0x0

Dec 17 19:57:51 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=162.142.125.28 DST=72.250.213.12 LEN=44 TOS=

Dec 17 19:57:57 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=193.93.62.58 DST=72.250.213.12 LEN=40 TOS=0x

Dec 17 19:57:57 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=45.129.33.8 DST=72.250.213.12 LEN=40 TOS=0x0

Dec 17 19:58:05 sshd[2940]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.187.232.73 user=root

Dec 17 19:58:06 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=33:33:00:00:00:01:ce:8d:0e:a4:af:1c:86:dd SRC=fe80:0000:0000:0000:e976:9186:00b9:db6e DST=

Dec 17 19:58:06 sshd[2940]: Failed password for root from 222.187.232.73 port 52028 ssh2

Dec 17 19:58:09 sshd[2940]: Failed password for root from 222.187.232.73 port 52028 ssh2

Dec 17 19:58:10 sshd[2940]: Failed password for root from 222.187.232.73 port 52028 ssh2

Dec 17 19:58:11 sshd[2940]: Received disconnect from 222.187.232.73 port 52028:11: [preauth]

Dec 17 19:58:11 sshd[2940]: Disconnected from authenticating user root 222.187.232.73 port 52028 [preauth]

Dec 17 19:58:11 sshd[2940]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.187.232.73 user=root

Dec 17 19:58:14 sshd[2942]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.187.232.73 user=root

Dec 17 19:58:17 sshd[2942]: Failed password for root from 222.187.232.73 port 59929 ssh2

Dec 17 19:58:20 sshd[2942]: Failed password for root from 222.187.232.73 port 59929 ssh2

Dec 17 19:58:22 sshd[2942]: Failed password for root from 222.187.232.73 port 59929 ssh2

Dec 17 19:58:22 sshd[2942]: Received disconnect from 222.187.232.73 port 59929:11: [preauth]

Dec 17 19:58:22 sshd[2942]: Disconnected from authenticating user root 222.187.232.73 port 59929 [preauth]

Dec 17 19:58:22 sshd[2942]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.187.232.73 user=root

Dec 17 19:58:26 sshd[2944]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.187.232.73 user=root

Dec 17 19:58:29 sshd[2944]: Failed password for root from 222.187.232.73 port 32545 ssh2

Dec 17 19:58:30 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=89.248.160.139 DST=72.250.213.12 LEN=40 TOS=

Dec 17 19:58:31 sshd[2944]: Failed password for root from 222.187.232.73 port 32545 ssh2

Dec 17 19:58:34 sshd[2944]: Failed password for root from 222.187.232.73 port 32545 ssh2

Dec 17 19:58:34 sshd[2944]: Received disconnect from 222.187.232.73 port 32545:11: [preauth]

Dec 17 19:58:34 sshd[2944]: Disconnected from authenticating user root 222.187.232.73 port 32545 [preauth]

Dec 17 19:58:34 sshd[2944]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.187.232.73 user=root

Dec 17 19:58:39 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=33:33:00:00:00:01:ce:8d:0e:a4:af:1c:86:dd SRC=fe80:0000:0000:0000:e976:9186:00b9:db6e DST=

Dec 17 19:58:43 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=116.228.181.106 DST=72.250.213.12 LEN=40 TOS

Dec 17 19:58:46 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=45.129.33.13 DST=72.250.213.12 LEN=40 TOS=0x

Dec 17 19:59:13 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=33:33:00:00:00:01:ce:8d:0e:a4:af:1c:86:dd SRC=fe80:0000:0000:0000:e976:9186:00b9:db6e DST=

Dec 17 19:59:37 postfix/qmgr[2090]: EF50D6C0924: from=<double-bounce@>, size=936, nrcpt=1 (queue active)

Dec 17 19:59:37 spampd[2331]: WARNING!! Error in process_request eval block: /usr/sbin/spampd: socket connect failure: Connection refused

Dec 17 19:59:37 postfix/lmtp[2952]: EF50D6C0924: to=ecnal@ecnal.me, orig_to=, relay=127.0.0.1[127.0.0.1]:10025, delay=376, delays=

Dec 17 19:59:37 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=190.224.126.107 DST=72.250.213.12 LEN=40 TOS

Dec 17 19:59:46 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=33:33:00:00:00:01:ce:8d:0e:a4:af:1c:86:dd SRC=fe80:0000:0000:0000:e976:9186:00b9:db6e DST=

Dec 17 20:00:01 CRON[2955]: pam_unix(cron:session): session opened for user root by (uid=0)

Dec 17 20:00:01 CRON[2956]: pam_unix(cron:session): session opened for user root by (uid=0)

Dec 17 20:00:01 CRON[2958]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [

Dec 17 20:00:01 CRON[2959]: (root) CMD (sudo -u www-data php -f /usr/local/lib/owncloud/cron.php)

Dec 17 20:00:01 CRON[2957]: pam_unix(cron:session): session opened for user munin by (uid=0)

Dec 17 20:00:01 CRON[2955]: pam_unix(cron:session): session closed for user root

Dec 17 20:00:01 CRON[2960]: (munin) CMD (if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi)

Dec 17 20:00:01 sudo[2961]: root : TTY=unknown ; PWD=/root ; USER=www-data ; COMMAND=/usr/bin/php -f /usr/local/lib/owncloud/cron.php

Dec 17 20:00:01 sudo[2961]: pam_unix(sudo:session): session opened for user www-data by (uid=0)

Dec 17 20:00:02 sudo[2961]: pam_unix(sudo:session): session closed for user www-data

Dec 17 20:00:02 CRON[2956]: pam_unix(cron:session): session closed for user root

Dec 17 20:00:03 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=45.129.33.177 DST=72.250.213.12 LEN=40 TOS=0

Dec 17 20:00:04 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=146.185.25.177 DST=72.250.213.12 LEN=40 TOS=

Dec 17 20:00:06 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=89.248.165.7 DST=72.250.213.12 LEN=40 TOS=0x

Dec 17 20:00:11 CRON[2957]: pam_unix(cron:session): session closed for user munin

Dec 17 20:00:20 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=33:33:00:00:00:01:ce:8d:0e:a4:af:1c:86:dd SRC=fe80:0000:0000:0000:e976:9186:00b9:db6e DST=

Dec 17 20:00:41 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=45.129.33.162 DST=72.250.213.12 LEN=40 TOS=0

Dec 17 20:00:48 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=192.241.237.32 DST=72.250.213.12 LEN=40 TOS=

Dec 17 20:00:55 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=33:33:00:00:00:01:ce:8d:0e:a4:af:1c:86:dd SRC=fe80:0000:0000:0000:e976:9186:00b9:db6e DST=

Dec 17 20:00:59 kernel: [UFW BLOCK] IN=ens18 OUT= MAC=b6:f2:12:93:e6:e6:24:b6:57:f1:9f:c4:08:00 SRC=141.98.80.242 DST=72.250.213.12 LEN=40 TOS=0

ecnal · December 18, 2020, 3:12am

Thank you for the assistance, by the way. The new VM nginx won’t start, as per the logs I just posted. I would honestly prefer to move to the new VM, seeing as it is running 18.04 and that is what current Mail-in-a-box is designed for. But if I just have to stick to the currently working VM, I am alright with that.

alento · December 18, 2020, 3:12am

Ok, you need to delete the contents of the /home/user-data/ssl directory (including subdirectories) and rerun sudo mailinabox, then run the /tools/ssl_cerficates.py script. However when rerunning sudo mailinabox if at the very end there is not a message that Let’s Encrypt is installed, there is another step - which I can look up for you tomorrow.

ecnal · December 18, 2020, 3:13am

Okay. I will do that and post the results. Thanks for the help. Go to bed. It can wait.

ecnal · December 18, 2020, 8:50pm

Sorry for the delayed response. The system wouldn’t let me post any more replies – it said I had reached my daily quota as a new member. But that worked. Mail-in-a-box is up and running on the new VM. Just had to run through the letsencrypt stuff again – but that makes sense, considering it is a different install, so it would need new TLS certificates. Thank you so much for the assistance. I really appreciate it.