SSL Keys and Keystores

Overview

The following diagram shows how the various files involved in TLS/SSL are created and installed, and the relationships between them:

Organization

Each server has a directory :

  • etc/sipxpbx/ssl

This contains the public and private keys for that server and the java
trust store and keystore. Here is what that directory contains:

  • ssl.crt : The SSL certificate for XML RPC.
  • ssl.keystore : The java keystore used for XML RPC communication. This is different for each server. Only the XML RPC server for a given node in the cluster that runs the xml rpc server using that key will need to know this. It is dervied from ssl.key.
  • ssl.key : The SSL Key that is used for XML RPC. This is loaded into ssl.keystore and used by java. There is only one key in the keystore.
  • ssl.p12 : PKCS 12 certificate of ssl.crt. This is currently not used.
  • ssl-web.crt : The SSL Certificate for web communication (https from the browser). It can be signed by an external trusted signer. We do not have the CA certificate for this signer.
  • ssl-web.p12 : PKCS 12 format certificate of ssl-web.crt.
  • ssl-web.keystore : The keystore for the HTTPS server that is run in sipxconfig.
  • authorities.jks : The trust store for this server. This contains all the public keys that are used for XML RPC communication. Public keys can be transmitted in the clear so this information can be shared. Each server must have the certificate for the master server and its own certificate that can be transmitted in the clear to any other server that needs it (we never need to do this for the moment).
  • ssl-web.key : The key that is used ( loaded into the ssl-web.keystore) for HTTPS communication.

There is only one CA for the cluster. The certificate for that is located in etc/sipxpbx/ssl/authorities. Each certificate in authorites.jks is signed by the CA for the cluster. If you list the keys using

keytool -list -keystore authorities.jks

You will see that there is a certificate ca.masterhost which is the CA cert for the cluster.

How keys are installed on the primary (master) server

When you do an install, SSL Keys are generated by :

bin/ssl-cert/gen-ssl-keys.sh

This generates the ssl keys in a staging area /var/sipxdata/certdb

This staging area is present only on the master server.

There you will notice each key/cert pair is prefixed by the host name to which it belongs. Certificates are generated in several forms for different tools that need them for input. Each host in the cluster has its own key that is not known to the other servers and this cache can be deleted from the master at any time after the cluster has been installed.

There is also a authorities.jks file there. Which contains public keys for use by xml rpc over https. Java works relaibly with der format certificates and does not like the .crt certificates. Hence a conversion step is done in gen-ssl-keys.sh before the certificate is put into the certificate database.

The keys, keystore and authorities.jks file are copied over to the etc/ssl/ directory by invoking another script

bin/ssl-cert/install-cert.sh

A given server will only use the information present in etc/ssl/. The Staging area is never consulted except for adding and removing certificates.

HA Install

When an HA install is under way, the secondary server contacts sipxconfig. sipXconfig calls the following script:

libexec/sipXecs/initial-config secondaryServerHostName

This invokes gen-ssl-keys.sh and install-ssl-keys.sh passing the secondaryServerHostName as argument.
You will notice that as nodes are added to the cluster, the var/sipdata/certdb/authorities.jks file is augmented with the public
keys (certificates) of the new servers added. As before, a format conversion to DER format is done before adding the certificate to the java trust store. Each host has a Java keystore alias of its own host name.
You can re-run sipxecs-setup as many times as you wish on any server. However, if you run it on the master server, note that a new CA certificate will be generated and hence you will need to run it on all the other servers of the cluster.

Each host also has the common CA certificate for the cluster. XML RPC certificates are signed with the key of the common XML RPC CA. Thus any host can talk to any other host in the cluster using XMLRPC over SSL. The certificate for the server it is trying to contact is shipped to it via SSL handshake and is verified upon receipt to ensure it has been signed by the common CA key given that the CA certificate is available to the client. This is part of standard SSL handshake.

The tar file that is generated by the install script contains the authorities.jks database and the private xml rpc key for the specific
server to which that tar file is headed. It does not contain the private keys of any other server in the cluster. It is untarred upon arrival and the secondary server is good to go at this point.
The SSL handshake takes care of

Web Keys

Starting with sipxecs 4.0 the web and xml RPC certificates are distinct. For the initial install they start out being the same but subsequently, a new web certificate may be installed. The CA certificate of the web certificate is not assumed to be available at the time the web cert is installed. Hence the web cert is not verified prior to installation. A web cert is identified by a hostname-web.p12. The CA cert should be available to the web server for verification of the SSL Cert that is shipped to it.

Important : The information here pertains to all version 3.11.12. 3.10 did things differently.