Server federation is the fancy name for the old concept of server-to-server communication. Communication between servers has evolved in XMPP thus we have a few different ways to achieve it. In the document Server To Server HowTo's we explain how to configure your firewall and manually test that servers can be reached by other servers. This document will explain the different options supported by Openfire 3.6.0 for doing server-to-server.
Old Server Dialback method
Ths old server dialback method is the oldest one and is specified in the rfc3920 document. This method is currently probably the most widely used method when connecting Openfire to other XMPP servers. Communication between servers occurs over plain (i.e. not encrypted) sockets thus it is not secure. Servers rely on a DNS server to verify the remote server and as a way to protect against domain spoofing.
TLS + SASL EXTERNAL
As you may know TLS is the evolution of SSL which is a way to verify identities and encrypt communications. Certificates are needed when using TLS. You can read the Certificates in Openfire to learn how to create certificates in Openfire. After certificates were verified the communication is then secured and traffic is encrypted. After TLS was negotiated SASL EXTERNAL will be performed as a way to authenticate the server.
Openfire by default will reject self-signed certificates. Self-signed certificates are certificates that were not signed by a trusted CA but instead by the same issuer (note: that is not a technically correct way to describe it but it is an easy way to explain it). When TLS fails to be negotiated then Openfire will check if the Old Server Dialback method is available and will try to use it. If Old Server Dialback method is not available then communication between servers will fail.
It is possible to configure Openfire to trust self-signed certificates. However, by doing that you can no longer trust the identity of the remote server. To overcome this limitation server dialback over TLS is enabled when using self-signed certificates. For controlled environments this may be good enough but that is not always the case.
SASL EXTERNAL is only available when using trusted certificates. That means that when using self-signed certificates, even when accepted, will not be used.
TLS + Server Dialback (new in Openfire 3.6.0)
This method will be offered by Openfire when TLS was negotiated and self-signed certificates are being used. By default Openfire will not accept self-signed certificate so this option is not offered. TLS + SASL EXTERNAL is preferred over this method since it is more secure. If self-signed certificates were found and are not accepted then server dialback over TLS will not be used and Openfire will check if the Old Server Dialback method is available and will try to use it. If Old Server Dialback method is not available then communication between servers will fail.
Server dialback over TLS is a good way to keep using encrypted connections and have a relatively secure way to verify who is the remote server.