I run a couple of Openfire servers which are set to require encryption and uses the default self-signed certificates. I use Trillian as my client.
When Trillian got updated to version 6 it would no longer connect to the Openfire servers. Trillian 6 uses an updated SSL engine compared to version 5.
After diagnosing the problem with the Trillian devs they gave me this answer:
"When connecting, it looks like Openfire is adding an extension: "id-ce-subjectAltName' with a type of 'id-on-xmppAddr', but it has an error with it. The particular error we see is:
BER Error: Wrong field in SEQUENCE expected class:CONTEXT(2) tag:0 but found class:UNIVERSAL(0) tag: 12"
I tried generating my own self signed certificates using easyrsa and openssl and those work fine with Trillian 6. I'll add the method I used to generate them below. Publicly signed certificates using LetEncrypt or others should also work.
Can someone look into this and see if it's possible to create the certificates so they validate properly? I'll gladly ask the Trillian devs if you want some more info from their end.
Here's how I created my own self signed certificates. Run on Fedora 24. Replace <servername> with suitable name (without brackets) if you want to run the commands yourself.
cp vars.example vars
I uncommented the following in vars file. Probably not needed though...
set_var EASYRSA_KEY_SIZE 2048
set_var EASYRSA_CA_EXPIRE 3650
set_var EASYRSA_CERT_EXPIRE 3650
Make RSA cert
./easyrsa gen-req <servername>_rsa
./easyrsa sign-req server <servername>_rsa
Convert to x509
openssl x509 -in pki/issued/<servername>_rsa.crt -out <servername>_rsa.crt
Decrypt private key since Openfire doesn't seem to import it otherwise
openssl rsa -in pki/private/<servername>_rsa.key -out rsapriv.key
Make DSA cert
openssl dsaparam -out dsaparam.pem 2048
openssl gendsa -out privdsakey.pem dsaparam.pem
openssl req -new -key privdsakey.pem -out pki/reqs/<servername>_dsa.req
./easyrsa sign-req server <servername>_dsa
Convert to x509
openssl x509 -in pki/issued/<serverame>_dsa.crt -out <servername>_dsa.crt
In Openfire web GUI
- Import ca.crt to truststore
- Import DSA and RSA cert using the decrypted keys and x509 certs