500 Internal Server Errors

jeffshead · October 20, 2024, 9:36pm

Everything has been working for a couple of years. I started having issues ever since I updated to v70. I’d get 500 errors when trying to add new users, add custom DNS and when accessing the External DNS page. I also noticed some DNS zone files were missing while others would not display.

After a few hours of trying to figure out what was causing the issues, I decided to just re-install Ubuntu and start all over. After doing so, I get the following error right out of the gate:

Okay. I'm about to set up jeff@xxxxx.com for you. This account will also
have access to the box's control panel.
password:
 (again):
<!doctype html>
<html lang=en>
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>

So now, with a fresh install, I get a 500 error during setup, when it’s time to add my admin account credentials for the web UI.

I don’t know where to begin troubleshooting this. Any ideas?

jeffshead · October 21, 2024, 12:02am

root@mail:~# systemctl status nsd.service
× nsd.service - Name Server Daemon
     Loaded: loaded (/lib/systemd/system/nsd.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sun 2024-10-20 19:59:49 EDT; 14s ago
       Docs: man:nsd(8)
    Process: 4350 ExecStart=/usr/sbin/nsd -d -P  (code=exited, status=1/FAILURE)
   Main PID: 4350 (code=exited, status=1/FAILURE)
        CPU: 90ms

Oct 20 19:59:49 mail.xxx.com systemd[1]: nsd.service: Main process exited, code=exited, status=1/FAILURE
Oct 20 19:59:49 mail.xxx.com systemd[1]: nsd.service: Failed with result 'exit-code'.
Oct 20 19:59:49 mail.xxx.com systemd[1]: Failed to start Name Server Daemon.
Oct 20 19:59:49 mail.xxx.com systemd[1]: nsd.service: Scheduled restart job, restart counter is at 5.
Oct 20 19:59:49 mail.xxx.com systemd[1]: Stopped Name Server Daemon.
Oct 20 19:59:49 mail.xxx.com systemd[1]: nsd.service: Start request repeated too quickly.
Oct 20 19:59:49 mail.xxx.com systemd[1]: nsd.service: Failed with result 'exit-code'.
Oct 20 19:59:49 mail.xxx.com systemd[1]: Failed to start Name Server Daemon.

jeffshead · October 21, 2024, 12:08am

root@mail:~# journalctl -xeu nsd.service
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit nsd.service has begun execution.
░░
░░ The job identifier is 2236.
Oct 20 20:05:47 mail.xxx.com nsd[5358]: [2024-10-20 20:05:47.361] nsd[5358]: error: Cannot open /var/log/nsd.log for appending (Read-only file system), logging to stderr
Oct 20 20:05:47 mail.xxx.com nsd[5358]: [2024-10-20 20:05:47.363] nsd[5358]: notice: nsd starting (NSD 4.3.9)
Oct 20 20:05:47 mail.xxx.com nsd[5358]: [2024-10-20 20:05:47.367] nsd[5358]: error: can't bind tcp socket: Cannot assign requested address
Oct 20 20:05:47 mail.xxx.com nsd[5358]: [2024-10-20 20:05:47.368] nsd[5358]: error: cannot open control interface ::1 8952
Oct 20 20:05:47 mail.xxx.com nsd[5358]: [2024-10-20 20:05:47.369] nsd[5358]: error: could not open remote control port
Oct 20 20:05:47 mail.xxx.com nsd[5358]: [2024-10-20 20:05:47.370] nsd[5358]: error: could not perform remote control setup
Oct 20 20:05:47 mail.xxx.com systemd[1]: nsd.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ An ExecStart= process belonging to unit nsd.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Oct 20 20:05:47 mail.xxx.com systemd[1]: nsd.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit nsd.service has entered the 'failed' state with result 'exit-code'.
Oct 20 20:05:47 mail.xxx.com systemd[1]: Failed to start Name Server Daemon.
░░ Subject: A start job for unit nsd.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit nsd.service has finished with a failure.
░░
░░ The job identifier is 2236 and the job result is failed.
Oct 20 20:05:47 mail.xxx.com systemd[1]: nsd.service: Scheduled restart job, restart counter is at 5.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ Automatic restarting of the unit nsd.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Oct 20 20:05:47 mail.xxx.com systemd[1]: Stopped Name Server Daemon.
░░ Subject: A stop job for unit nsd.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A stop job for unit nsd.service has finished.
░░
░░ The job identifier is 2319 and the job result is done.
Oct 20 20:05:47 mail.xxx.com systemd[1]: nsd.service: Start request repeated too quickly.
Oct 20 20:05:47 mail.xxx.com systemd[1]: nsd.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit nsd.service has entered the 'failed' state with result 'exit-code'.
Oct 20 20:05:47 mail.xxx.com systemd[1]: Failed to start Name Server Daemon.
░░ Subject: A start job for unit nsd.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit nsd.service has finished with a failure.
░░
░░ The job identifier is 2319 and the job result is failed.
lines 2911-2974/2974 (END)

jeffshead · October 21, 2024, 12:19am

root@mail:~# ufw status
Status: active

To                         Action      From
--                         ------      ----
22/tcp                     LIMIT       Anywhere
53                         ALLOW       Anywhere
25/tcp                     ALLOW       Anywhere
465/tcp                    ALLOW       Anywhere
587/tcp                    ALLOW       Anywhere
993/tcp                    ALLOW       Anywhere
995/tcp                    ALLOW       Anywhere
4190/tcp                   ALLOW       Anywhere
80/tcp                     ALLOW       Anywhere
443                        ALLOW       Anywhere
22/tcp (v6)                LIMIT       Anywhere (v6)
53 (v6)                    ALLOW       Anywhere (v6)
25/tcp (v6)                ALLOW       Anywhere (v6)
465/tcp (v6)               ALLOW       Anywhere (v6)
587/tcp (v6)               ALLOW       Anywhere (v6)
993/tcp (v6)               ALLOW       Anywhere (v6)
995/tcp (v6)               ALLOW       Anywhere (v6)
4190/tcp (v6)              ALLOW       Anywhere (v6)
80/tcp (v6)                ALLOW       Anywhere (v6)
443 (v6)                   ALLOW       Anywhere (v6)

Player · October 21, 2024, 12:50am

can’t bind tcp socket: Cannot assign requested address

This is usually due to the port (53 probably) already being used by a different program, like systemd-networkd. Run ss -lptun as root to see the programs listening on ports. For port 53 it should be bind9’s named on 127.0.0.1 and nsd on the external ip.

Make sure you are on Ubuntu 22.04.

jeffshead · October 21, 2024, 1:07am

Thank you!

However, after more testing, I believe the issue is caused by not having an IPv6 address.

I get nothing back when I run ip -6 addr

If I run the following commands…

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=0
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=0
sudo service nsd start

…the NSD service starts but I run into the same problem when the server is rebooted.

I never manually disabled IPv6 so I don’t know why I have to run those commands to get a v6 IP address. Where should I look to see why I’m not getting a v6 IP address until I run those commands?

Player · October 21, 2024, 1:13am

These settings through sysctl -w don’t persist, that’d be in /etc/sysctl.conf (and others, see man sysctl.conf). You should find something there disabling it since the default is on.

Maybe the miab scripts don’t configure bind9/nsd to be separate properly if ipv6 is disabled, they should be isolated by the former listening on localhost and the latter on the exposed ip.

jeffshead · October 21, 2024, 1:48am

Thanks, again.

It seems the Ubuntu 22.04 template the VPS provider uses has IPv6 disabled by default.

I had to change all three from 1 to 0

net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.lo.disable_ipv6 = 0

I still get the following error:

and MTA-STS policy is missing: STSFetchResult.NONE

Player · October 21, 2024, 1:46pm

Check if this works from an external machine: dig @box.example.com mta-sts.box.example.com TXT - replace the domain ofc

jeffshead · October 21, 2024, 4:08pm

I ended running the curl -s https://mailinabox.email/setup.sh | sudo -E bash command and that seems to have solved the MTA-STS policy is missing: STSFetchResult.NONE issue.

However, I’m still getting Something went wrong, sorry. error when accessing the MIAB External DNS page. I also get 500 Internal Server Errors when adding users but it seems they are being added. I get the same results when I let MIAB handle the DNS on the box and when I use external DNS on a third-party provider. I have, of course, allowed enough time for the DNS to propagate when switching back and forth between internal and external DNS.

Player · October 21, 2024, 4:29pm

The first thing to check with http 500 is the nginx error log at /var/log/nginx/error.log for more detail

jeffshead · October 22, 2024, 9:30am

I’m not seeing any associated errors in the nginx error log.

This is what I see when adding an alias:

If I refresh the page after getting the error, it seems the alias does get added.

KiekerJan · October 23, 2024, 1:13pm

When you get this error, check the mailinabox logs: sudo journalctl -u mailinabox

jeffshead · October 23, 2024, 5:31pm

External DNS page:

Oct 23 13:24:14 mail.example.com start[1956]: [2024-10-23 13:24:14,064] ERROR in app: Exception on /dns/dump [GET]
Oct 23 13:24:14 mail.example.com start[1956]: Traceback (most recent call last):
Oct 23 13:24:14 mail.example.com start[1956]:   File "/usr/local/lib/mailinabox/env/lib/python3.10/site-packages/flask/app.py", line 1473, in wsgi_app
Oct 23 13:24:14 mail.example.com start[1956]:     response = self.full_dispatch_request()
Oct 23 13:24:14 mail.example.com start[1956]:   File "/usr/local/lib/mailinabox/env/lib/python3.10/site-packages/flask/app.py", line 882, in full_dispatch_request
Oct 23 13:24:14 mail.example.com start[1956]:     rv = self.handle_user_exception(e)
Oct 23 13:24:14 mail.example.com start[1956]:   File "/usr/local/lib/mailinabox/env/lib/python3.10/site-packages/flask/app.py", line 880, in full_dispatch_request
Oct 23 13:24:14 mail.example.com start[1956]:     rv = self.dispatch_request()
Oct 23 13:24:14 mail.example.com start[1956]:   File "/usr/local/lib/mailinabox/env/lib/python3.10/site-packages/flask/app.py", line 865, in dispatch_request
Oct 23 13:24:14 mail.example.com start[1956]:     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
Oct 23 13:24:14 mail.example.com start[1956]:   File "/root/mailinabox/management/daemon.py", line 74, in newview
Oct 23 13:24:14 mail.example.com start[1956]:     return viewfunc(*args, **kwargs)
Oct 23 13:24:14 mail.example.com start[1956]:   File "/root/mailinabox/management/daemon.py", line 396, in dns_get_dump
Oct 23 13:24:14 mail.example.com start[1956]:     return json_response(build_recommended_dns(env))
Oct 23 13:24:14 mail.example.com start[1956]:   File "/root/mailinabox/management/dns_update.py", line 1072, in build_recommended_dns
Oct 23 13:24:14 mail.example.com start[1956]:     for (domain, _zonefile, records) in build_zones(env):
Oct 23 13:24:14 mail.example.com start[1956]:   File "/root/mailinabox/management/dns_update.py", line 174, in build_zones
Oct 23 13:24:14 mail.example.com start[1956]:     records = build_zone(domain, domains, additional_records, env)
Oct 23 13:24:14 mail.example.com start[1956]:   File "/root/mailinabox/management/dns_update.py", line 221, in build_zone
Oct 23 13:24:14 mail.example.com start[1956]:     subzone = build_zone(subdomain, domain_properties, additional_records, env, is_zone=False)
Oct 23 13:24:14 mail.example.com start[1956]:   File "/root/mailinabox/management/dns_update.py", line 213, in build_zone
Oct 23 13:24:14 mail.example.com start[1956]:     records.extend((None, "SSHFP", value, "Optional. Provides an out-of-band method for verifying an SSH key before connecting. Use 'VerifyHostKeyDNS yes' (or 'VerifyHostKeyDNS ask') when connecting with>
Oct 23 13:24:14 mail.example.com start[1956]:   File "/root/mailinabox/management/dns_update.py", line 213, in <genexpr>
Oct 23 13:24:14 mail.example.com start[1956]:     records.extend((None, "SSHFP", value, "Optional. Provides an out-of-band method for verifying an SSH key before connecting. Use 'VerifyHostKeyDNS yes' (or 'VerifyHostKeyDNS ask') when connecting with>
Oct 23 13:24:14 mail.example.com start[1956]:   File "/root/mailinabox/management/dns_update.py", line 457, in build_sshfp_records
Oct 23 13:24:14 mail.example.com start[1956]:     keys = shell("check_output", ["ssh-keyscan", "-4", "-t", "rsa,dsa,ecdsa,ed25519", "-p", str(port), "localhost"])
Oct 23 13:24:14 mail.example.com start[1956]:   File "/root/mailinabox/management/utils.py", line 127, in shell
Oct 23 13:24:14 mail.example.com start[1956]:     ret = getattr(subprocess, method)(cmd_args, **kwargs)
Oct 23 13:24:14 mail.example.com start[1956]:   File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
Oct 23 13:24:14 mail.example.com start[1956]:     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
Oct 23 13:24:14 mail.example.com start[1956]:   File "/usr/lib/python3.10/subprocess.py", line 526, in run
Oct 23 13:24:14 mail.example.com start[1956]:     raise CalledProcessError(retcode, process.args,
Oct 23 13:24:14 mail.example.com start[1956]: subprocess.CalledProcessError: Command '['ssh-keyscan', '-4', '-t', 'rsa,dsa,ecdsa,ed25519', '-p', '22', 'localhost']' returned non-zero exit status 1.
Oct 23 13:24:14 mail.example.com gunicorn[1956]: Exception on /dns/dump [GET]
                                                   Traceback (most recent call last):
                                                     File "/usr/local/lib/mailinabox/env/lib/python3.10/site-packages/flask/app.py", line 1473, in wsgi_app
                                                       response = self.full_dispatch_request()
                                                     File "/usr/local/lib/mailinabox/env/lib/python3.10/site-packages/flask/app.py", line 882, in full_dispatch_request
                                                       rv = self.handle_user_exception(e)
                                                     File "/usr/local/lib/mailinabox/env/lib/python3.10/site-packages/flask/app.py", line 880, in full_dispatch_request
                                                       rv = self.dispatch_request()
                                                     File "/usr/local/lib/mailinabox/env/lib/python3.10/site-packages/flask/app.py", line 865, in dispatch_request
                                                       return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
                                                     File "/root/mailinabox/management/daemon.py", line 74, in newview
                                                       return viewfunc(*args, **kwargs)
                                                     File "/root/mailinabox/management/daemon.py", line 396, in dns_get_dump
                                                       return json_response(build_recommended_dns(env))
                                                     File "/root/mailinabox/management/dns_update.py", line 1072, in build_recommended_dns
                                                       for (domain, _zonefile, records) in build_zones(env):
                                                     File "/root/mailinabox/management/dns_update.py", line 174, in build_zones
                                                       records = build_zone(domain, domains, additional_records, env)
                                                     File "/root/mailinabox/management/dns_update.py", line 221, in build_zone
                                                       subzone = build_zone(subdomain, domain_properties, additional_records, env, is_zone=False)
                                                     File "/root/mailinabox/management/dns_update.py", line 213, in build_zone
                                                       records.extend((None, "SSHFP", value, "Optional. Provides an out-of-band method for verifying an SSH key before connecting. Use 'VerifyHostKeyDNS yes' (or 'VerifyHostKeyDNS ask') when connecting w>
                                                     File "/root/mailinabox/management/dns_update.py", line 213, in <genexpr>
                                                       records.extend((None, "SSHFP", value, "Optional. Provides an out-of-band method for verifying an SSH key before connecting. Use 'VerifyHostKeyDNS yes' (or 'VerifyHostKeyDNS ask') when connecting w>
                                                     File "/root/mailinabox/management/dns_update.py", line 457, in build_sshfp_records
                                                       keys = shell("check_output", ["ssh-keyscan", "-4", "-t", "rsa,dsa,ecdsa,ed25519", "-p", str(port), "localhost"])
                                                     File "/root/mailinabox/management/utils.py", line 127, in shell
                                                       ret = getattr(subprocess, method)(cmd_args, **kwargs)
                                                     File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
                                                       return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
                                                     File "/usr/lib/python3.10/subprocess.py", line 526, in run
                                                       raise CalledProcessError(retcode, process.args,
                                                   subprocess.CalledProcessError: Command '['ssh-keyscan', '-4', '-t', 'rsa,dsa,ecdsa,ed25519', '-p', '22', 'localhost']' returned non-zero exit status 1.

dms · October 23, 2024, 6:22pm

You haven’t by chance changed your ssh port have you? Try running this command by hand on your server:
ssh-keyscan -4 -t "rsa,dsa,ecdsa,ed25519" -p 22 localhost

KiekerJan · October 23, 2024, 6:22pm

Could you check the output of the following command:

sudo ssh-keyscan -4 -t rsa,dsa,ecdsa,ed25519 -p 22 localhost

jeffshead · October 23, 2024, 8:30pm

Now I know where I screwed up… I wanted to limit SSH access to the server so I ran the following command: nano /etc/hosts.allow. I added my public IP’s but neglected to allow localhost. So I just added sshd: 127.0.0.1 and no more errors.

I never would have thought to check SSH connectivity. Thanks, guys!

system · October 30, 2024, 8:30pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.