AnsweredAssumed Answered

"DHPublicKey does not comply to algorithm constraints"

Question asked by spacegoose on Jul 26, 2017
Latest reply on Jul 27, 2017 by spacegoose

Our LDAP team upgraded the endpoint for connections removing insecure protocols and ciphers.

I was unable to login to the admin console (and likely users were unable to authenticate to chat) until pointing to another LDAP node (in the cluster) with an endpoint that had not yet been upgraded. This was done via an /etc/hosts mapping. The non-upgraded IP address was given for the updated LDAP server name.

 

I updated the JRE cipheres with:

Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files for JDK/JRE 8 Download

 

noticed these new ciphers (in addition to others):

 

 

     > *       TLS_DHE_RSA_WITH_AES_256_CBC_SHA

 

     > *       TLS_DHE_RSA_WITH_AES_256_CBC_SHA256

     > *       TLS_DHE_RSA_WITH_AES_256_GCM_SHA384

 

and made them available via the admin console within Openfire (4.0.3).

 

I then removed the /etc/hosts mapping.

 

I was unable to login to the admin console until downgrading the DH keySize to < 1024

 

#jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 2048

jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 1024

 

within jdk1.8.0_72/jre/lib/security/java.security (the JRE our Openfire uses).

 

I am confused as to why this was necessary? The error I could see in the log until doing so was:

 

"DHPublicKey does not comply to algorithm constraints"

 

openssl s_connect to both new and old LDAP systems (both 636) show 2048 public keys, though with different ciphers, depending on the machine I run openssl s_connect from ... I guess due to differing openssl versions or cipher config differences ...

 

The DH keySize < 2048 setting had worked until the LDAP endpoint was upgraded, and continued to work when pointing to the non-upgraded LDAP endpoint. It seems unusual that apparently newer more secure protocols / ciphers would require this seeming downgrade to the security settings.

 

A java program that tests SSL gave the following info (from the Openfire machine, using same JRE Openfire uses):

 

Testing server - upgraded endpoint

Given this client's capabilities ([SSLv3, TLSv1, TLSv1.1, TLSv1.2]), the server prefers protocol=TLSv1.2, cipher=TLS_DHE_RSA_WITH_AES_256_GCM_SHA384

 

Testing server - non-upgraded endpoint

Given this client's capabilities ([SSLv3, TLSv1, TLSv1.1, TLSv1.2]), the server prefers protocol=TLSv1.2, cipher=TLS_RSA_WITH_AES_128_CBC_SHA

 

I am also unsure if the error refers to the the LDAP SSL or the SSL on Openfire itself? Our instance XMPP and admin console require SSL.

 

 

Here are some of the stack traces showing the issue for the admin console access and LDAP connection:

 

 

 

 

2017.07.25 18:54:32 WARN  [Jetty-QTP-AdminConsole-98]: org.jivesoftware.admin.LoginLimitManager - Failed admin console login attempt by <myuserid> from <myIP>

2017.07.25 18:54:33 INFO  [Server SR - 881162561]: org.jivesoftware.openfire.net.SocketReadingMode - STARTTLS negotiation (with: org.jivesoftware.openfire.net.SocketConnection@1248cf94 socket: Socket[addr=/78.46.93.108,port=57984,localport=5269] session: org.jivesoftware.openfire.session.LocalIncomingServerSession@2843cab status: 1 address: <xyz>.com/5c3gn5yu6p id: 5c3gn5yu6p) failed.

javax.net.ssl.SSLHandshakeException: DHPublicKey does not comply to algorithm constraints

        at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1431)

        at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)

        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)

        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)

        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)

        at org.jivesoftware.openfire.net.TLSStreamHandler.doHandshake(TLSStreamHandler.jav a:241)

        at org.jivesoftware.openfire.net.TLSStreamHandler.start(TLSStreamHandler.java:178)

        at org.jivesoftware.openfire.net.SocketConnection.startTLS(SocketConnection.java:1 95)

        at org.jivesoftware.openfire.net.SocketReadingMode.negotiateTLS(SocketReadingMode. java:87)

        at org.jivesoftware.openfire.net.BlockingReadingMode.readStream(BlockingReadingMod e.java:138)

        at org.jivesoftware.openfire.net.BlockingReadingMode.run(BlockingReadingMode.java: 76)

        at org.jivesoftware.openfire.net.SocketReader.run(SocketReader.java:145)

        at java.lang.Thread.run(Thread.java:745)

Caused by: javax.net.ssl.SSLHandshakeException: DHPublicKey does not comply to algorithm constraints

        at sun.security.ssl.DHCrypt.checkConstraints(DHCrypt.java:237)

        at sun.security.ssl.ServerHandshaker.clientKeyExchange(ServerHandshaker.java:1599)

        at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:269)

        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)

        at sun.security.ssl.Handshaker$1.run(Handshaker.java:919)

        at sun.security.ssl.Handshaker$1.run(Handshaker.java:916)

        at java.security.AccessController.doPrivileged(Native Method)

        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369)

        at org.jivesoftware.openfire.net.TLSStreamHandler.doTasks(TLSStreamHandler.java:34 4)

        at org.jivesoftware.openfire.net.TLSStreamHandler.doHandshake(TLSStreamHandler.jav a:254)

        ... 7 more

 

 

 

 

 

 

 

 

 

 

 

 

 

2017.07.25 18:54:36 INFO  [Server SR - 1692736043]: org.jivesoftware.openfire.net.SocketReadingMode - STARTTLS negotiation (with: org.jivesoftware.openfire.net.SocketConnection@7b80ac6e socket: Socket[addr=/136.243.42.223,port=47704,localport=5269] session: org.jivesoftware.openfire.session.LocalIncomingServerSession@330ab9e3 status: 1 address: <xyz>.com/5rljrbkums id: 5rljrbkums) failed.

javax.net.ssl.SSLHandshakeException: DHPublicKey does not comply to algorithm constraints

        at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1431)

        at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)

        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)

        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)

        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)

        at org.jivesoftware.openfire.net.TLSStreamHandler.doHandshake(TLSStreamHandler.jav a:241)

        at org.jivesoftware.openfire.net.TLSStreamHandler.start(TLSStreamHandler.java:178)

        at org.jivesoftware.openfire.net.SocketConnection.startTLS(SocketConnection.java:1 95)

        at org.jivesoftware.openfire.net.SocketReadingMode.negotiateTLS(SocketReadingMode. java:87)

        at org.jivesoftware.openfire.net.BlockingReadingMode.readStream(BlockingReadingMod e.java:138)

        at org.jivesoftware.openfire.net.BlockingReadingMode.run(BlockingReadingMode.java: 76)

        at org.jivesoftware.openfire.net.SocketReader.run(SocketReader.java:145)

        at java.lang.Thread.run(Thread.java:745)

Caused by: javax.net.ssl.SSLHandshakeException: DHPublicKey does not comply to algorithm constraints

        at sun.security.ssl.DHCrypt.checkConstraints(DHCrypt.java:237)

        at sun.security.ssl.ServerHandshaker.clientKeyExchange(ServerHandshaker.java:1599)

        at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:269)

        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)

        at sun.security.ssl.Handshaker$1.run(Handshaker.java:919)

        at sun.security.ssl.Handshaker$1.run(Handshaker.java:916)

        at java.security.AccessController.doPrivileged(Native Method)

        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369)

        at org.jivesoftware.openfire.net.TLSStreamHandler.doTasks(TLSStreamHandler.java:34 4)

        at org.jivesoftware.openfire.net.TLSStreamHandler.doHandshake(TLSStreamHandler.jav a:254)

        ... 7 more

 

 

 

 

 

2017.07.25 19:03:00 ERROR [Jetty-QTP-AdminConsole-56]: org.jivesoftware.openfire.ldap.LdapAuthProvider - Error connecting to LDAP server

javax.naming.CommunicationException: <myLDAP>:636 [Root exception is javax.net.ssl.SSLHandshakeException: DHPublicKey does not comply to algorithm constraints]

        at com.sun.jndi.ldap.Connection.<init>(Connection.java:226)

        at com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:137)

        at com.sun.jndi.ldap.LdapClientFactory.createPooledConnection(LdapClientFactory.ja va:64)

        at com.sun.jndi.ldap.pool.Connections.<init>(Connections.java:115)

        at com.sun.jndi.ldap.pool.Pool.getPooledConnection(Pool.java:132)

        at com.sun.jndi.ldap.LdapPoolManager.getLdapClient(LdapPoolManager.java:329)

        at com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1606)

        at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2746)

        at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:319)

        at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:192)

        at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:210)

        at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:153)

        at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:83)

        at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)

        at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313)

        at javax.naming.InitialContext.init(InitialContext.java:244)

        at javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:154)

        at org.jivesoftware.util.JiveInitialLdapContext.<init>(JiveInitialLdapContext.java :43)

        at org.jivesoftware.openfire.ldap.LdapManager.getContext(LdapManager.java:568)

        at org.jivesoftware.openfire.ldap.LdapManager.findUserDN(LdapManager.java:975)

        at org.jivesoftware.openfire.ldap.LdapManager.findUserDN(LdapManager.java:928)

        at org.jivesoftware.openfire.ldap.LdapAuthProvider.authenticate(LdapAuthProvider.j ava:126)

        at org.jivesoftware.openfire.auth.AuthFactory.authenticate(AuthFactory.java:217)

        at org.jivesoftware.openfire.admin.login_jsp._jspService(login_jsp.java:175)

        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1669)

        at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:39)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1652)

        at org.jivesoftware.util.LocaleFilter.doFilter(LocaleFilter.java:76)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1652)

        at org.jivesoftware.util.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingF ilter.java:53)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1652)

        at org.jivesoftware.admin.PluginFilter.doFilter(PluginFilter.java:80)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1652)

        at org.jivesoftware.admin.AuthCheckFilter.doFilter(AuthCheckFilter.java:162)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.ja va:1652)

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:22 3)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:11 27)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185 )

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:106 1)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandler Collection.java:215)

        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.jav a:110)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

        at org.eclipse.jetty.server.Server.handle(Server.java:499)

        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)

        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)

        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635 )

        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

        at java.lang.Thread.run(Thread.java:745)

Caused by: javax.net.ssl.SSLHandshakeException: DHPublicKey does not comply to algorithm constraints

        at sun.security.ssl.DHCrypt.checkConstraints(DHCrypt.java:237)

        at sun.security.ssl.ClientHandshaker.serverKeyExchange(ClientHandshaker.java:712)

        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:268)

        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)

        at sun.security.ssl.Handshaker.process_record(Handshaker.java:914)

        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)

        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)

        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)

        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)

        at com.sun.jndi.ldap.Connection.createSocket(Connection.java:376)

        at com.sun.jndi.ldap.Connection.<init>(Connection.java:203)

        ... 56 more

 

 

 

 

Thanks for any help!

 

sg

Outcomes