I build a new mailserver and rsynced /home/user-data across to it, then re-ran setup.
This initially failed due to owncloud install issues, so I commented that out and re-ran startup.
This also failed at the LetsEncrypt stage with the same error:
OpenSSL.SSL.Error: [('asn1 encoding routines', 'ASN1_CHECK_TLEN', 'wrong tag'), ('asn1 encoding routines', 'ASN1_D2I_EX_PRIMITIVE', 'nested asn1 error'), ('asn1 encoding routines', 'ASN1_TEMPLATE_NOEXP_D2I', 'nested asn1 error'), ('rsa routines', 'OLD_RSA_PRIV_DECODE', 'RSA lib')]
I tried running management/ssl_certificates.py for some individual domains, and got the same error.
Then I removed the /home/user-data/ssl/ directory completely and re-ran setup. This failed with a different error.
In desperation I ran management/ssl_certificates.py for my mailserver domain and that worked. At which point I was able to use the web admin interface again. Then I used the web UI to generate the missing SSL certs.
At this point, it all seems to be working again. I’m not sure what step made things work this time - I’d removed /home/user-data/ssl/ on my original mailserver and that didn’t get things working.