Error during Provision TLS, missing flags for certbot

phred · August 12, 2018, 4:00am

When I hit the “Provision” button on the TSL (SSL) Certificates page, instead of getting the expected result, I get this message:

Log:
Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator webroot, Installer None You should register before running non-interactively, or provide --agree-tos and --email <email_address> flags.

I’d sure like to get this working, so if there’s any advice on what to try, I’d appreciate it. Thanks.

The contents of the log:

2018-08-11 21:54:29,208:DEBUG:certbot.main:certbot version: 0.26.1
2018-08-11 21:54:29,208:DEBUG:certbot.main:Arguments: ['--non-interactive', '-d', 'box.rflm.net,rflm.net,www.rflm.net', '--csr', '/tmp/tmphena12yl', '--cert-path', '/tmp/tmpipeha3bt/cert', '--chain-path', '/tmp/tmpipeha3bt/chain', '--fullchain-path', '/tmp/tmpipeha3bt/cert_and_chain.pem', '--webroot', '--webroot-path', '/home/user-data/ssl/lets_encrypt/webroot', '--config-dir', '/home/user-data/ssl/lets_encrypt']
2018-08-11 21:54:29,209:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2018-08-11 21:54:29,218:DEBUG:certbot.log:Root logging level set at 20
2018-08-11 21:54:29,219:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2018-08-11 21:54:29,219:DEBUG:certbot.plugins.selection:Requested authenticator webroot and installer None
2018-08-11 21:54:29,220:DEBUG:certbot.plugins.selection:Single candidate plugin: * webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot.plugins.webroot:Authenticator
Initialized: <certbot.plugins.webroot.Authenticator object at 0x7ff6da893470>
Prep: True
2018-08-11 21:54:29,220:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.webroot.Authenticator object at 0x7ff6da893470> and installer None
2018-08-11 21:54:29,220:INFO:certbot.plugins.selection:Plugins selected: Authenticator webroot, Installer None
2018-08-11 21:54:29,221:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/certbot/display/ops.py", line 50, in get_email
    force_interactive=True)
  File "/usr/lib/python3/dist-packages/certbot/display/util.py", line 529, in input
    self._interaction_fail(message, cli_flag)
  File "/usr/lib/python3/dist-packages/certbot/display/util.py", line 474, in _interaction_fail
    raise errors.MissingCommandlineFlag(msg)
certbot.errors.MissingCommandlineFlag: Missing command line flag or config entry for this setting:
Enter email address (used for urgent renewal and security notices)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.26.1', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1364, in main
    return config.func(config, plugins)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1238, in certonly
    le_client = _init_le_client(config, auth, installer)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 641, in _init_le_client
    acc, acme = _determine_account(config)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 517, in _determine_account
    config.email = display_ops.get_email()
  File "/usr/lib/python3/dist-packages/certbot/display/ops.py", line 54, in get_email
    raise errors.MissingCommandlineFlag(msg)
certbot.errors.MissingCommandlineFlag: You should register before running non-interactively, or provide --agree-tos and --email <email_address> flags.

–
Ray Frush

alento · August 12, 2018, 4:33am

Looking at the index or using the search function of the forum is usually a good first step!

phred · August 12, 2018, 7:46pm

Thanks for the quick reply. I did do the ‘sudo mailinabox’, and got the prompt for the hostname, and then it proceeded with a repeat of the install steps. Things went fine up through the Munin install, and then went south for the TLS step. I got the following:

-----------------------------------------------
Mail-in-a-Box uses Let's Encrypt to provision free certificates
to enable HTTPS connections to your box. You'll now be asked to agree
to Let's Encrypt's terms of service.

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Registering without email!
From cffi callback <function _verify_callback at 0x7f914b91c268>:
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/SSL.py", line 309, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: 'module' object has no attribute 'X509_up_ref'
An unexpected error occurred:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 438, in wrap_socket
    cnx.do_handshake()
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/SSL.py", line 1907, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/SSL.py", line 1639, in _raise_ssl_error
    _raise_current_error()
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 846, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 326, in connect
    ssl_context=context)
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 325, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 445, in wrap_socket
    raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    raise SSLError(e)
urllib3.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.26.1', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1364, in main
    return config.func(config, plugins)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 721, in register
    _determine_account(config)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 520, in _determine_account
    config, account_storage, tos_cb=_tos_cb)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 180, in register
    acme = acme_from_config_key(config, key)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 50, in acme_from_config_key
    return acme_client.BackwardsCompatibleClientV2(net, key, config.server)
  File "/usr/lib/python3/dist-packages/acme/client.py", line 744, in __init__
    directory = messages.Directory.from_json(net.get(server).json())
  File "/usr/lib/python3/dist-packages/acme/client.py", line 1078, in get
    self._send_request('GET', url, **kwargs), content_type=content_type)
  File "/usr/lib/python3/dist-packages/acme/client.py", line 1027, in _send_request
    response = self.session.request(method, url, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 502, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 612, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)
Please see the logfiles in /var/log/letsencrypt for more details.

-----------------------------------------------

–
Ray Frush

alento · August 12, 2018, 8:42pm

try this …

If you want backup the /home/user-data/ssl directory somewhere safe then remove ALL it’s contents and run the ssl_certificates.py under the ~/mailinabox/management/ directory.

Note: I copied this from elsewhere … it is not necessary to make the backup of the ~/ssl directory in your situation, just delete any contents that are in there.

phred · August 13, 2018, 3:16am

Alento-

I like your approach, scrape ssl and start over, however I’m not really getting anywhere with the suggestion to move /home/user-data/ssl out of the way:

So I moved the directory:
root@box:/home/user-data# mv ssl/ ssl.borked

Then ran the ssl_certificates.py script, which immediately complains.

root@box:~/mailinabox/management# ./ssl_certificates.py 
Traceback (most recent call last):
  File "./ssl_certificates.py", line 660, in <module>
    provision_certificates_cmdline()
  File "./ssl_certificates.py", line 372, in provision_certificates_cmdline
    status = provision_certificates(env, limit_domains=domains)
  File "./ssl_certificates.py", line 274, in provision_certificates
    os.mkdir(account_path)
FileNotFoundError: [Errno 2] No such file or directory: '/home/user-data/ssl/lets_encrypt'

manually adding the missing path leads to another error about a missing file:

root@box:~/mailinabox/management# ./ssl_certificates.py 
Traceback (most recent call last):
  File "./ssl_certificates.py", line 660, in <module>
    provision_certificates_cmdline()
  File "./ssl_certificates.py", line 372, in provision_certificates_cmdline
    status = provision_certificates(env, limit_domains=domains)
  File "./ssl_certificates.py", line 348, in provision_certificates
    ret.extend(post_install_func(env))
  File "./ssl_certificates.py", line 448, in post_install_func
    cert = get_domain_ssl_files(env['PRIMARY_HOSTNAME'], ssl_certificates, env, use_main_cert=False)
  File "./ssl_certificates.py", line 153, in get_domain_ssl_files
    "certificate_object": load_pem(load_cert_chain(ssl_certificate)[0]),
  File "./ssl_certificates.py", line 600, in load_cert_chain
    with open(pemfile, "rb") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/user-data/ssl/ssl_certificate.pem'

It feels like there must be another script that does the initial setup of the /home/user-data/ssl directory first before the ssl_certificates.py script can run properly.

alento · August 13, 2018, 4:26am

Move the CONTENTS, not the directory itself. I made the same mistake! And had the same results!

You can recreate the directory with the mkdir command.

alento · August 13, 2018, 4:42am

Reviewing the thread that I linked in the previous post … you want to do the following:

mkdir /home/user-data/ssl

rerun mailinabox

then run ./ssl_certificates.py from the ~/mailinabox/management directory.

phred · August 13, 2018, 10:19am

So, following your suggestion, which helped me better understand how the SSL directory gets provisioned in the first place, I still ran into issues:

When I ran ‘sudo mailinabox’:

Mail-in-a-Box Version:  v0.28

Updating system packages...
Installing system packages...
Initializing system random number generator...
Firewall is active and enabled on system startup
Creating initial SSL certificate and perfect forward secrecy Diffie-Hellman parameters...
Generating DH parameters, 2048 bit long safe prime, generator 2
This is going to take a long time    ........................................+................................................................................+...............+.................................................................................................++*++*
Installing nsd (DNS server)...
Installing Postfix (SMTP server)...
Installing Dovecot (IMAP server)...
Installing OpenDKIM/OpenDMARC...
Installing SpamAssassin...
Installing Nginx (web server)...
Installing Roundcube (webmail)...
Installing Nextcloud (contacts/calendar)...
Nextcloud is already latest version
Installing Z-Push (Exchange/ActiveSync server)...
Installing Mail-in-a-Box system management daemon...
Installing Munin (system monitoring)...
updated DNS: rflm.net
web updated

-----------------------------------------------
Mail-in-a-Box uses Let's Encrypt to provision free certificates
to enable HTTPS connections to your box. You'll now be asked to agree
to Let's Encrypt's terms of service.

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Registering without email!
From cffi callback <function _verify_callback at 0x7ff588689bf8>:
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/SSL.py", line 309, in wrapper
    _lib.X509_up_ref(x509)
AttributeError: 'module' object has no attribute 'X509_up_ref'
An unexpected error occurred:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 438, in wrap_socket
    cnx.do_handshake()
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/SSL.py", line 1907, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/SSL.py", line 1639, in _raise_ssl_error
    _raise_current_error()
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 846, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 326, in connect
    ssl_context=context)
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 325, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 445, in wrap_socket
    raise ssl.SSLError('bad handshake: %r' % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    raise SSLError(e)
urllib3.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.26.1', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1364, in main
    return config.func(config, plugins)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 721, in register
    _determine_account(config)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 520, in _determine_account
    config, account_storage, tos_cb=_tos_cb)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 180, in register
    acme = acme_from_config_key(config, key)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 50, in acme_from_config_key
    return acme_client.BackwardsCompatibleClientV2(net, key, config.server)
  File "/usr/lib/python3/dist-packages/acme/client.py", line 744, in __init__
    directory = messages.Directory.from_json(net.get(server).json())
  File "/usr/lib/python3/dist-packages/acme/client.py", line 1078, in get
    self._send_request('GET', url, **kwargs), content_type=content_type)
  File "/usr/lib/python3/dist-packages/acme/client.py", line 1027, in _send_request
    response = self.session.request(method, url, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 502, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 612, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)
Please see the logfiles in /var/log/letsencrypt for more details.

-----------------------------------------------

Your Mail-in-a-Box is running.

Please log in to the control panel for further instructions at:

https://ip.address/admin

You will be alerted that the website has an invalid certificate. Check that
the certificate fingerprint matches:

89:...:FB

Then you can confirm the security exception and continue.

Note that the contents of /var/log/letsencrpyt/letsencrypt.log

2018-08-13 03:58:05,780:DEBUG:certbot.main:certbot version: 0.26.1
2018-08-13 03:58:05,781:DEBUG:certbot.main:Arguments: ['--register-unsafely-without-email', '--config-dir', '/home/user-data/ssl/lets_encrypt']
2018-08-13 03:58:05,781:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2018-08-13 03:58:05,797:DEBUG:certbot.log:Root logging level set at 20
2018-08-13 03:58:05,798:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2018-08-13 03:58:05,799:INFO:certbot.client:Registering without email!
2018-08-13 03:58:06,056:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2018-08-13 03:58:06,059:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
2018-08-13 03:58:06,085:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 438, in wrap_socket
    cnx.do_handshake()
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/SSL.py", line 1907, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/SSL.py", line 1639, in _raise_ssl_error
    _raise_current_error()
  File "/usr/local/lib/python3.4/dist-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]

Then running the ssl_certificates.py script:

root@box:/home/user-data# cd ~/mailinabox/management/
root@box:~/mailinabox/management# ./ssl_certificates.py 
Provisioning TLS certificates for box.rflm.net, rflm.net, www.rflm.net.
error: box.rflm.net, rflm.net, www.rflm.net:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
You should register before running non-interactively, or provide --agree-tos and --email <email_address> flags.

Which has left me right back where I started from. So at least it’s consistent in how the script is broken.

It seems the main issue is:

OpenSSL.SSL.Error: [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]

A seach leads to this: https://community.letsencrypt.org/t/certificate-verify-failed/64848 which suggests re-installing the ca-certificates, but that didn’t help.

Another key error seems to be:

AttributeError: 'module' object has no attribute 'X509_up_ref'

A search on that error turns up a number of hits suggesting either the ‘pyOpenssl’ package or the ‘cryptography’ package are out of date. (see https://github.com/pyca/pyopenssl/issues/728 for example) The locally delivered python from MailInABox is actually pretty up to date, ‘pyOpenssl’ is the most recent, and ‘cryptography’ was 2.2.2 and upgraded to 2.3. Both of which should satisfy the X509_up_ref issue according to the various links I found on this error.

root@box:~/mailinabox/management# sudo /usr/local/lib/mailinabox/env/bin/pip3.4 install cryptography --upgrade
...
Successfully installed cryptography-2.3

However that didn’t seem to have any impact on this issue at all, which makes me wonder if the problem is when ‘certbot’ is called, it may be using the system python, which is running ‘cryptography’ 1.9. I haven’t found a good way to force an update of the system’s ‘cryptography’ package for python. A pip install cryptography --upgrde fails because the module can’t be uninstalled (owned by the OS).

Anyways, I appreciate any further insights or recommendations. I want to rule out any human error in this before I open an issue on github.

–
Ray Frush

phred · August 14, 2018, 12:39am

I found a non-optimal workaround for this issue. I added the following line to /etc/letsencrypt/cli.ini

no-verify-ssl = true

This allows the certbot to skip the SSL verification that’s throwing the ‘SSL3_GET_SERVER_CERTIFICATE’ error.

–
Ray Frush

davinian · August 17, 2018, 2:44pm

Running ‘sudo mailinabox’ (e.g. updating mib) worked and fixed the problem for me - I think the process re-confirms Let’s Encrypt’s T&Cs.

system · August 24, 2018, 2:44pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.