Can't load certificates nor can they be recreated

I’ve been banging my head on my desk for the last few weeks and at this point I’ve got not a single clue of where to go from here.

At some point, NGINX failed on me entirely, taking down the admin site and I’ve not been able to restore it since then. I’ve gotten through a couple of certificate errors but now I’m stuck with this one (and I suspect multiple others behind it):

Jun 06 17:59:31 box.maldor.tech nginx[43845]: nginx: [emerg] cannot load certificate "/home/user-data/ssl/mta-sts.box.maldor.tech-202605>

ssl_certificates.py is fried:

Traceback (most recent call last):
  File "/root/mailinabox/management/./ssl_certificates.py", line 682, in <module>
    provision_certificates_cmdline()
  File "/root/mailinabox/management/./ssl_certificates.py", line 393, in provision_certificates_cmdline
    status = provision_certificates(env, limit_domains=domains)
  File "/root/mailinabox/management/./ssl_certificates.py", line 369, in provision_certificates
    ret.extend(post_install_func(env))
  File "/root/mailinabox/management/./ssl_certificates.py", line 479, in post_install_func
    if cert and os.readlink(system_ssl_certificate) != cert['certificate']:
OSError: [Errno 22] Invalid argument: '/home/user-data/ssl/ssl_certificate.pem'

status_checks.py:

======
✖  Dovecot LMTP LDA is not running (port 10026).
✖  IMAPS (dovecot) is not running (port 993).
✖  Mail Filters (Sieve/dovecot) is not running (port 4190).
✖  HTTP Web (nginx) is not running (port 80).
   nginx: [emerg] cannot load certificate "/home/user-data/ssl/mta-sts.box.maldor.tech-20260528-baf6a44e.pem": BIO_new_file() failed
   (SSL: error:80000002:system library::No such file or directory:calling
   fopen(/home/user-data/ssl/mta-sts.box.maldor.tech-20260528-baf6a44e.pem, r) error:10000080:BIO routines::no such file)
   nginx: configuration file /etc/nginx/nginx.conf test failed
✖  HTTPS Web (nginx) is not running (port 443).
   nginx: [emerg] cannot load certificate "/home/user-data/ssl/mta-sts.box.maldor.tech-20260528-baf6a44e.pem": BIO_new_file() failed
   (SSL: error:80000002:system library::No such file or directory:calling
   fopen(/home/user-data/ssl/mta-sts.box.maldor.tech-20260528-baf6a44e.pem, r) error:10000080:BIO routines::no such file)
   nginx: configuration file /etc/nginx/nginx.conf test failed
✖  The SSH server on this machine permits password-based login. A more secure way to log in is using a public key. Add your SSH public
   key to $HOME/.ssh/authorized_keys, check that you can log in without a password, set the option 'PasswordAuthentication no' in
   /etc/ssh/sshd_config, and then restart the openssh via 'sudo service ssh restart'.
✓  System software is up to date.
✖  A new version of Mail-in-a-Box is available. You are running version v75. The latest version is v76. For upgrade instructions, see
   https://mailinabox.email.
✓  System administrator address exists as a mail alias. [administrator@box.maldor.tech ↦ admin@maldor.tech]
✓  The disk has 14.38 GB space remaining.
✓  System memory is 55% free.
✓  Backups are enabled

Network
=======
✓  Firewall is active.
✓  Outbound mail (SMTP port 25) is not blocked.
✓  IPv4 address is not blacklisted by zen.spamhaus.org.
✓  IPv6 address is not blacklisted by zen.spamhaus.org.

box.maldor.tech
===============
?  Nameserver glue records (ns1.box.maldor.tech and ns2.box.maldor.tech) should be configured at your domain name registrar as having
   the IP address of this box (45.56.82.60). They currently report addresses of [Not Set]/[Not Set]. If you have set up External DNS,
   this may be OK.
✖  This domain must resolve to this box's IP address (45.56.82.60 / 2600:3c01::f03c:93ff:fe4d:7ebc) in public DNS but it currently
   resolves to 45.56.82.60 / 2600:3c0a::2000:6ff:fe20:e188. It may take several hours for public DNS to update after a change. This
   problem may result from other issues listed above.
✖  This box's reverse DNS is currently box.maldor.tech (IPv4) and [Not Set] (IPv6), but it should be box.maldor.tech. Your ISP or cloud
   provider will have instructions on setting up reverse DNS for this box.
✓  Hostmaster contact address exists as a mail alias. [hostmaster@box.maldor.tech ↦ administrator@box.maldor.tech]
✓  Domain's email is directed to this domain. [box.maldor.tech has no MX record, which is ok]
✓  Postmaster contact address exists as a mail alias. [postmaster@box.maldor.tech ↦ administrator@box.maldor.tech]
✓  Domain is not blacklisted by dbl.spamhaus.org.
✖  The TLS (SSL) certificate for this domain is currently self-signed. You will get a security warning when you check or send email and
   when visiting this domain in a web browser (for webmail or static site hosting).

Other logs can be available upon request, but honestly I think I might need a fresh pair of keys, but I don’t know how to force the system to use the new keys if I were to generate them because of how many places they could be possibly stored in.

There are several dns related remarks on your status page that might well prevent certificate renewal. e.g.

Questions:

  • Does the box provide dns or do you use an external dns provider?
  • Can you provide the output of ls -al /home/user-data/ssl? Specifically, ssl_certificate.pem should be a symlink to a certificate file.
  • Did you change anything on the box? It might be a while before the certificate error pops up, because you only encounter it once the certificate expires.

To narrow the ipv6 issue:

  • Is there one ipv6 address assigned to your box?
  • If yes, check /etc/mailinabox.conf, modify if needed. It should contain the ipv6 address that is assigned to the box. Then rerun mailinabox
  • Keep an eye on the reverse DNS setting for the ipv6 address. This should be set correctly for mail delivery.

Does the box provide dns or do you use an external dns provider?

External DNS, Specifically Linodes DNS networking

Can you provide the output of ls -al /home/user-data/ssl? Specifically, ssl_certificate.pem should be a symlink to a certificate file.

root@box:/home/user-data/ssl# ls -al
total 28
drwxr-xr-x  3 root      root      4096 Jun  6 17:50 .
drwxr-xr-x 10 user-data user-data 4096 Jun  6 17:48 ..
-rw-r--r--  1 root      root      1009 Jun  6 17:38 box.maldor.tech-selfsigned-20260606.pem
-rw-r--r--  1 root      root       424 Jun  6 17:39 dh2048.pem
drwxr-xr-x  5 root      root      4096 Jun  7 01:03 lets_encrypt
-rw-r--r--  1 root      root      1009 Jun  6 17:38 ssl_certificate.pem
-rw-r--r--  1 root      root      1704 Jun  6 17:38 ssl_private_key.pem

It does not look like its a symlink, will work on that now…

Did you change anything on the box? It might be a while before the certificate error pops up, because you only encounter it once the certificate expires.

There were no changes leading up to the this issue and eventual NGINX failure that I am aware of. I was looking into cleaning it up in terms of what websites it hosted when I discovered that it was offline.

Concerning IPv6. The address there was for a different box. That issue has since been corrected to be the for the same box as the ipv4 address

If you compare the self signed and the ssl_certificate.pem files you’ll probably find they’re the same. Any way, first step I think is to mv the ssl_certificate.pem file away. Then make a symmlink from the self signed file to the ssl_certificate.pem file. From there, try to renew certificates.

Ok, so far its making sense, when I first started looking into this, one of the certificates was not present. Certs are now reprovisioned and I’m right now seeing if

sudo mailinabox

Is going to behave now… And its looking like it will…

Edit: looks like everything is alive again, but I need to have new certificates for everything. Time to look into that DNS01 challenge problem again
Thanks for your help!

Normally, MiaB should obtain these certificates for you. It uses the HTTP challenge, not the DNS challenge.