TLS Certificate Auto-Renewal

Maintenance
21

@backpackhasjetz & @gabeotisbenson:

Huh. Might be that somehow you have multiple cron jobs?

take a look at /etc/cron.d/. There should only be one of these:
mailinabox-nightly

and it’s contents should look like:

# Mail-in-a-Box --- Do not edit / will be overwritten on update.
# Run nightly tasks: backup, status checks.
0 3 * * *       root    (cd /root/mailinabox && management/daily_tasks.sh)
```

Additionally, would you mind posting the output of this command:

` /root/mailinabox/management/ssl_certificates.py -v `
22

I see only one mailinabox-nightly under /etc/cron.d/ - so good there, and contents match what you posted.

The output of the command:

root@box:/etc/cron.d# /root/mailinabox/management/ssl_certificates.py -v
> Reading account key from /home/user-data/ssl/lets_encrypt/account.pem.
> Validating existing account saved to /home/user-data/ssl/lets_encrypt/registration.json.
> Reusing existing challenges for box.hireahelpermail.com.
> The challenges for box.hireahelpermail.com have been accepted.
> Reusing existing challenges for hireahelpermail.com.
> The challenges for hireahelpermail.com have been accepted.
> Reusing existing challenges for www.hireahelpermail.com.
> The challenges for www.hireahelpermail.com have been accepted.
> Generating a new certificate signing request.
> Requesting a certificate.
box.hireahelpermail.com, hireahelpermail.com, www.hireahelpermail.com:
Something unexpected went wrong: Error creating new cert :: Too many certificates already issued for exact set of domains: box.hireahelpermail.com,hireahelpermail.com,www.hireahelpermail.com
root@box:/etc/cron.d#
23

What are the permissions on the files in /home/user-data/ssl/?

Can root both read and write? Should look something like this:

drwxr-xr-x 3 root      root      4096 Dec  5 09:43 ./
drwxr-xr-x 9 user-data user-data 4096 Dec  4 21:51 ../
-rw------- 1 root      root      3466 Dec  5 09:43 box.example.tld-20170305-a0fcd12a.pem
-rw-r--r-- 1 root      root      1013 Dec  4 21:32 box.example.tld-selfsigned-20161204.pem
-rw-r--r-- 1 root      root       424 Dec  4 21:36 dh2048.pem
-rw------- 1 root      root      3453 Dec  5 09:17 example.tld-20170305-313ff4df.pem
drwxr-xr-x 3 root      root      4096 Dec  4 22:48 lets_encrypt/
lrwxrwxrwx 1 root      root        64 Dec  5 09:43 ssl_certificate.pem -> /home/user-data/ssl/box.example.tld-20170305-a0fcd12a.pem
-rw------- 1 root      root      1679 Dec  4 21:32 ssl_private_key.pem
-rw------- 1 root      root      3466 Dec  5 03:00 www.example.tld-20170305-a4f2gf1d.pem
24

Looks like it?

root@box:/home/user-data/ssl# ls -la
total 44
drwxr-xr-x 3 root      root      4096 Oct 17 03:00 .
drwxr-xr-x 9 user-data user-data 4096 Dec 19 12:17 ..
-rw------- 1 root      root      3559 Mar 18  2016 box.hireahelpermail.com-20160616-2a386255.pem
-rw------- 1 root      root      3612 May 18  2016 box.hireahelpermail.com-20160816-78fd2b67.pem
-rw------- 1 root      root      3531 Aug  2 03:00 box.hireahelpermail.com-20161031-5e6d1511.pem
-rw------- 1 root      root      3531 Oct 17 03:00 box.hireahelpermail.com-20170115-02effd85.pem
-rw-r--r-- 1 root      root      1013 Mar 18  2016 box.hireahelpermail.com-selfsigned-20160318.pem
-rw-r--r-- 1 root      root       424 Mar 18  2016 dh2048.pem
drwxr-xr-x 3 root      root      4096 Mar 18  2016 lets_encrypt
lrwxrwxrwx 1 root      root        65 Oct 17 03:00 ssl_certificate.pem -> /home/user-data/ssl/box.hireahelpermail.com-20170115-02effd85.pem
-rw------- 1 root      root      1679 Mar 18  2016 ssl_private_key.pem
25

@backpackhasjetz: Have you tried the solution posted by @NatCC below?

26

Thanks, @cromulus - even though I’m not getting an SSL error, I tried those steps and it still spits out same message:

root@box:~/mailinabox# ./management/ssl_certificates.py
box.hireahelpermail.com, hireahelpermail.com, www.hireahelpermail.com:
Something unexpected went wrong: Error creating new cert :: Too many certificate                   s already issued for exact set of domains: box.hireahelpermail.com,hireahelperma                   il.com,www.hireahelpermail.com

I’m wondering if this all started around the time there was the SSL upgrade/weird bugs in MIAB. I did log in to both my MIAB boxes and run commands (I believe from @joshdata elsewhere on this forum) to get things fixed after upgrading.

I’ve also tried re-running the MIAB install/upgrade command, rebooting, no change. Both of my boxes now have expired certs :-/

27

Searching crt.sh shows numerous certs issued: https://crt.sh/?q=box.hireahelpermail.com

28

I’m seeing the same thing as backpackhasjetz on my domain: https://crt.sh/?q=mail.seanwatson.io

Since about the start of January the box has been requesting new certs every day. It looks like they are being issued properly which is why Lets Encrypt is starting to rate limit the requests.

29

Did you try this and reboot?

sudo pip3 install pyOpenSSL --upgrade

30

Or perhaps the steps outlined here:

31

Yes, and I get the same error message as backpackhasjetz. I’m going to disable SSL renewal for a week and then try it again.

32

Yes, I tried that comment’s instructions, didn’t see a change.Something really does seem to be bugged in MIAB with LetsEncrypt, though :-/

I’m not sure how to stop the auto-LetsEncrypt stuff from firing? There isn’t a way in the admin interface. What are you doing to disable it, @swatson?

33

What makes this worse is that I tried moving to a new box to see if that fixes it, and ran into issues with the guide: https://github.com/mail-in-a-box/mailinabox/issues/1071 - and I don’t know how to proceed from there. So I’m just stuck. Both of my boxes now have expired certs, so I may need to buy a real SSL cert or something.

34

I just commented out command to renew certs from the cron script. Open /home/root/mailinabox/management/daily_tasks.sh (or something like that, I’m writing from memory) and comment out the line with ssl_certificates.py (or something like that). I don’t see any new certificates on https://crt.sh today so I think its working.

35

FYI- Let’s Encrypt limit is 5 certificates for a given domain over a 7 day period.

Have you been able to diagnose why the certificates that are issued aren’t being saved?

You could try diagnosing by switching to the Let’s Encrypt staging server. To do so, edit management/ssl_certificates.py and modify the two places that client.issue_certificate is called – in my checkout of 0.21b the functions are called on line 332 and on line 351.

They should look something like this:

                                cert = client.issue_certificate(              
                                        domain_list,                          
                                        account_path,                         
                                        agree_to_tos_url=agree_to_tos_url,    
                                        private_key=private_key,              
                                        logger=my_logger)

You’d want to add a new acme_server argument:

                                cert = client.issue_certificate(              
                                        domain_list,                          
                                        account_path,
                                        acme_server=client.LETSENCRYPT_STAGING_SERVER,                         
                                        agree_to_tos_url=agree_to_tos_url,    
                                        private_key=private_key,              
                                        logger=my_logger)

Then try running ./management/ssl_certificates.py again and see if you can determine why the updated certificates aren’t being saved.

Once you’re done, you can reset that file back to the clean state:

$ git checkout -- ./management/ssl_certificates.py

36

Thanks. Very strange. I updated the code and re-ran ssl_certificates.py - and I’m getting the same error! Seems unlikely since I haven’t do anything regarding their staging server, ever. Maybe this means there’s something amiss with the actual ACME client.

37

What’s the output of:
pip3 list
and
dpkg --list

FYI: the staging server doesn’t produce valid certificates nor does it have a rate limit. That’s why @benschumacher suggested it. If the staging server produced the rate limit error, it’s not actually letsencrypt rate limiting you.

38

@cromulus I put the outputs here: https://gist.github.com/nicholashead/0a700cb43289093d7336d742d3935197

Very odd. Why would I be getting a rate error? Possible bug in ACME client or something else? It definitely changed the source code of the ssl_certificates.py file - I even put a print(“testing”) statement in there to make sure it was working.

39

I disabled the nightly ssl_certificates.py call two nights ago - and now just manually ran ssl_certificates (pointed at LetsEncrypt staging still supposedly) and it gave me… a totally valid certificate.

So I guess my problem is “fixed” for now, but that doesn’t answer:

  • How did MIAB get in this state to begin with?
  • What’s the long term fix (since I should be able to run the ssl_certificates.py nightly?)
  • Why isn’t the ACME client changing the server URL when passed into the method call?

I’m a C# programmer by trade, so I’m not really familiar with python enough to diagnose.

40

How did MIAB get in this state to begin with?

So, my hunch is that the pyOpenSSL version was outdated and incorrectly validating the ssl certificate downloaded from letsencrypt. Because validation failed, MIAB didn’t save it and thus when the cron job ran again the next day, MIAB requested another certificate. This caused MIAB to hit the rate limit.

Upgrading your pyOpenSSL package and then waiting for the rate limit to expire resulted in your ability to get a new, valid cert.

What’s the long term fix (since I should be able to run the ssl_certificates.py nightly?)

Everything should just work now. I think we might need to be more rigorous about the version of our python packages and the packages that are installed on a MIAB install.

Why isn’t the ACME client changing the server URL when passed into the method call?

Not sure what you are asking here. There is only one Lets Encrypt production URL. The staging url is only meant really for developing ACME clients or testing an implementation.