SSL: Regenerating all certificates in an OpenVox deployment

In some cases, you might need to regenerate the certificates and security credentials (private and public keys) generated by OpenVox’s built-in certificate authority (CA). For example, you might need to move your Puppet Server to a different network, or you might have experienced a security incident that makes existing credentials untrustworthy.

Note that this process destroys the certificate authority and all other certificates. It is meant for use in the event of a total compromise of your site, or some other unusual circumstance. If you just need to replace a few agent certificates, you can run puppetserver ca clean --certname <NAME> on your Puppet Server and then follow step 3 for any agents that need to be replaced.

Regenerating all certs involves the following steps:

  1. On your Puppet Server (CA host), clear and regenerate the CA and server certificate.
  2. Clear and regenerate certificates for extensions (e.g. OpenVoxDB).
  3. Clear and regenerate certificates for all agent nodes.

Step 1: Clear and regenerate certs on your Puppet Server

On the Puppet Server hosting the CA:

  1. Back up the following directories. If something goes wrong, you may need to restore them to keep your deployment functional.

    • SSL directory: /etc/puppetlabs/puppet/ssl/
    • Puppet Server CA directory: /etc/puppetlabs/puppetserver/ca/
    • OpenVoxDB SSL directory (if applicable): /etc/puppetlabs/puppetdb/ssl/

  2. Stop the OpenVox agent service:

    sudo puppet resource service puppet ensure=stopped

  3. Stop the Puppet Server service:

    sudo puppet resource service puppetserver ensure=stopped

  4. Clear the Puppet Server’s SSL state:

    sudo puppet ssl clean --localca
sudo rm -rf /etc/puppetlabs/puppetserver/ca

    puppet ssl clean --localca removes the server’s certificate, private key, and its local copy of the CA certificate. The Puppet Server CA directory must be removed with rm as there is no dedicated Puppet command to do so.

  5. Regenerate the CA and Puppet Server certificate:

    sudo puppetserver ca setup

    You should see: Generation succeeded. Find your files in /etc/puppetlabs/puppetserver/ca

  6. Start the Puppet Server service:

    sudo puppet resource service puppetserver ensure=running

    Wait for Puppet Server to finish initializing before proceeding. You can poll the status endpoint:

    until curl -sk https://puppet:8140/status/v1/simple > /dev/null 2>&1; do sleep 5; done

  7. Start the OpenVox agent service:

    sudo puppet resource service puppet ensure=running

At this point:

  • You have a brand new CA certificate and key.
  • Your Puppet Server has a certificate from the new CA and can accept new certificate requests.
  • The Puppet Server will reject catalog requests from nodes that haven’t replaced their certificates (which at this point is all of them except itself).
  • If you are using any extensions that rely on Puppet certificates, like OpenVoxDB, the Puppet Server won’t be able to communicate with them until their certificates are also replaced.

Step 2: Clear and regenerate certs for extensions

If you are running extensions that use Puppet CA certificates to communicate with Puppet Server, you need to refresh their certificates as well. Many tools have their own setup scripts or documentation for this.

OpenVoxDB

OpenVoxDB maintains its own SSL directory at /etc/puppetlabs/puppetdb/ssl/, separate from Puppet’s. This directory holds its own copies of the CA certificate and the OpenVox agent certificate and key that OpenVoxDB uses to authenticate with Puppet Server. After CA regeneration, refresh it using the puppetdb ssl-setup tool, then restart the service:

sudo puppetdb ssl-setup -f
sudo puppet resource service puppetdb ensure=stopped
sudo puppet resource service puppetdb ensure=running

Note: Running puppetdb ssl-setup without -f detects the cert mismatch but will not overwrite the files. The -f flag is required to update them. The service must be restarted (not just ensured running) to load the new certificates.

Step 3: Clear and regenerate certs for OpenVox agents

To replace certificates on agent nodes, log into each agent and do the following:

  1. Stop the OpenVox agent service.

    On *nix nodes:

    sudo puppet resource service puppet ensure=stopped

    On Windows nodes, run the same command without sudo with Administrator privileges.

  2. Clean the agent’s SSL state, including the locally cached CA certificate:

    On *nix nodes:

    sudo puppet ssl clean --localca

    On Windows nodes, run the same command without sudo with Administrator privileges.

    Note: The --localca flag is required. Without it, only the agent’s own certificate and key are removed. The agent retains the old CA certificate and fails to connect to the Puppet Server, which now presents a certificate signed by the new CA.

  3. Start the OpenVox agent service.

    On *nix nodes:

    sudo puppet resource service puppet ensure=running

    On Windows nodes, run the same command without sudo with Administrator privileges.

    Once the agent starts, it fetches the new CA certificate, generates a new key pair, and submits a certificate signing request to the Puppet Server.

  4. If you are not using autosigning, sign each agent’s certificate request from the Puppet Server:

    sudo puppetserver ca list
sudo puppetserver ca sign --certname <NAME>

Once an agent’s new certificate is signed, it fetches it automatically and begins a Puppet run.

Once you have regenerated all agents’ certificates, everything should now be back to normal and fully functional under the new CA.

Last updated: