DEBOPS.PKI(5)

NAME
debops.pki - PKI management using Ansible

SYNOPSIS

debops service/pki [--limit group,host,...] [--diff] [--check] [--tags tag1,tag2,...] [--skip-tags tag1,tag2,...] [<ansible-playbook options>] ...

DESCRIPTION

The debops.pki role provides a standardized management of the X.509 certificates on hosts controlled by Ansible. Other Ansible roles can utilize the environment created by debops.pki to automatically enable TLS/SSL encrypted connections.

Using this role, you can bootstrap a Public Key Infrastructure in your environment using an internal Certificate Authority, easily switch the active set of certificates between internal and external Certificate Authorities, or use the ACME protocol to automatically obtain certificates from CA that support it (for example Let's Encrypt <https://en.wikipedia.org/wiki/Let's_Encrypt>).

GETTING STARTED

  Ansible Controller requirements

Some operations performed by the debops.pki role are done on the Ansible Controller. However, DebOps roles are not designed to manage the Ansible Controller host directly, so they cannot automatically install the required software.

Software packages required by the role on the Ansible Controller:

bash >= 4.3.0
openssl >= 1.0.1

  Initial configuration

The default configuration of the debops.pki role and the environment it creates is designed to automatically provide valid X.509 certificates within the DebOps cluster – a central Certificate Authority located on the Ansible Controller automatically signs certificate requests from remote hosts, which are uploaded by Ansible to the secret/ directory. Signed certificates are downloaded to the remote hosts and automatically enabled, when necessary.

All of the remote hosts managed by DebOps will have the internal Root Certificate Authority certificate installed in the system CA store. This will ensure that the hosts within the cluster will automatically trust each other and encrypted communication within the cluster should be possible without further intervention.

Because of the nature of how X.509 certificates are created and maintained (they cannot be "updated", per se, only new ones can be issued by the CA), it's a good practice to create a test deployment of the PKI, and check if the generated certificates are acceptable (correct Distinguished Name, certificate validity times, etc.). There are some important role variables which might need to be set in the inventory for best results.

Maintaining a secure Certificate Authority is a complicated process and to ensure secure operation additional steps might be needed. You should read the rest of the debops.pki documentation to find out which parts of the created environment are sensitive and require additional steps to ensure secure operation.

  Useful global parameters

The environment managed by the debops.pki role is distributed between the Ansible Controller and remote hosts. Due to how task delegation in Ansible is designed, some of the variables that are important for the operation of the Ansible Controller are "sourced" on remote hosts. Therefore it's a good practice to define them in the Ansible inventory all group (usually ansible/inventory/group_vars/all/pki.yml) for consistency between different remote hosts.

Most of these variables are related to Certificate Authority operation, the ones you will likely want to change are:

pki_ca_domain
    This is the DNS domain used as a base for the internal Certificate Authority Distinguished Names. If you use more than one domain in your environment, you should set this variable to your preferred domain on all hosts, through Ansible's inventory.

    If you use VPS hosts provided by an external organization, they might be configured with no default domain, or the provider domain might be set up by default. Make sure that you check what domain is used by your remote hosts.

pki_ca_organization
    This is the organizations name used as a base for the internal Certificate Authority Distinguished Names.

pki_ca_root_dn, pki_ca_domain_dn, pki_ca_service_dn
    These variables define the Distinguished Name or Subject of the Root Certificate Authority and Domain Certificate Authority. The value is a list of DN entries which define the subject.

pki_authorities
    This is the list of internal Certificate Authorities managed on an Ansible Controller.

    debops.pki now supports the X.509 Name Constraints certificate extension by default. This may break software using old version of OpenSSL and multi-domain environments. Please see name_constraints and name_constraints_critical under pki_authorities for more information.

  Example inventory

In DebOps, the debops.pki role is included in the common.yml playbook and is run automatically on all of the managed hosts. You don't need to specifically enable it in Ansible's inventory.

  Example playbook

The debops.pki role requires a specialized Ansible playbook for correct operation. Additional directories required by the role are created in the secret/ directory on Ansible Controller, and this requires use of the special debops.pki/env role provided within the main role.

---
- name: Manage Public Key Infrastructure
  collections: [ 'debops.debops', 'debops.roles01',
                 'debops.roles02', 'debops.roles03' ]
  hosts: [ 'debops_all_hosts', 'debops_service_pki' ]
  become: True
  environment: '{{ inventory__environment | d({})
                   | combine(inventory__group_environment | d({}))
                   | combine(inventory__host_environment  | d({})) }}'
  pre_tasks:
    - name: Prepare pki environment
      import_role:
        name: 'pki'
        tasks_from: 'main_env'
      tags: [ 'role::pki', 'role::pki:secret', 'role::secret' ]
  roles:
    - role: secret
      tags: [ 'role::secret', 'role::pki', 'role::pki:secret' ]
      secret_directories:
        - '{{ pki_env_secret_directories }}'
    - role: cron
      tags: [ 'role::cron', 'skip::cron' ]
    - role: pki
      tags: [ 'role::pki', 'skip::pki' ]

  Renewing Certificates

For renewing certificates, just

1.  remove /etc/pki/realms/<realm>/ from the remote host and

2.  re-run debops.pki against it, e.g. this:

    debops service/pki  --limit=$REMOTE_HOST

    This will create a new key for the remote host. If you use external keys, they will be preserved

    NOTE: You can use ansible-console to delete the file on multiple hosts at once.

PKI REALMS STRUCTURE

The concept of PKI realms is designed to provide a standardized way for various applications to access X.509 certificates and private keys. The management of the keys and certificates is moved outside of the application into a fixed directory and file structure, which other applications can access.

  PKI realm overview

A "PKI realm" is a placeholder name for a bundle of the private key, X.509 certificate and Root CA certificate. This bundle has a certain directory structure, rules for naming files and what symlinks are present. It is designed so that from the outside of the debops.pki role other Ansible roles, or services they manage, have a standardized, uniform location where they can find X.509 certificates and private keys.

In different guides that describe setting up TLS for different services like webservers, mail servers, databases, etc. the private keys and X.509 certificates are usually put in different directories - for example /etc/nginx/ssl/, /etc/postfix/certs/, /etc/ssl/certs/, and so on. The debops.pki role turns this around by setting up an uniform set of directories split into "PKI realms", so that a host can have multiple sets of certificates, each for different purposes. Then, various services can be configured to get the private key and certificate files from those specific directories, including privileged access to the private keys when needed.

PKI realms have a concept of multiple certificate authorities - there's one set of private keys which can be signed by different CAs - internal CA, external CA, ACME CA and self-signed when everything else is disabled. There can be an "example.org" PKI realm which has certificates signed by both internal CA and the Let's Encrypt CA (via ACME), and the pki-realm script used to manage the realms on the remote hosts will automatically switch between them after checking the validity of their X.509 certificates.

  The application view

Different applications have different requirements for X.509 certificates and private keys. Some of them support keys and certificates in separate files, others require them combined in a single file. The PKI realm is designed to support both schemes at once.

The realms are located in the /etc/pki/realms/ directory. Each realm is contained in it's own subdirectory. By default a domain realm is configured by the role, and it's simplified directory structure looks like this:

/etc/pki/realms/
└── domain/
    ├── CA.crt
    ├── default.crt
    ├── default.key
    ├── default.pem
    └── trusted.crt

Each of these symlinks point to a file contained in another subdirectory (see the next section). The contents of these files are:

CA.crt
    This is the "trust anchor" or Root Certificate Authority certificate used by the application to check the validity of client certificates. This file is publicly readable.

    CA.crt may contain a CA certificate different than the one that issued the server certificate. In this case, this can be used to have separate client and server Certificate Authorities.

default.crt
    This is the server certificate with optionally bundled Intermediate Certificate Authorities and Diffie-Hellman parameters. It is sent to the clients during connection establishment by the application. This file is publicly readable. Note that not all software appreciates embedded DH parameters in this file. Some Java-based applications, or at least Graylog <https://www.graylog.org/> 4.1 and other software using recent versions of the Bouncy Castle library, throw exceptions when trying to parse this file. Consider using public/cert_intermediate.pem when that happens.

default.key
    This is the server private key. It's readable only by the root account and by pki_private_group – this can be used to limit access to different private keys by different UNIX accounts.

default.pem
    This file contains the private key, server certificate and Intermediate CA certificate(s). It has the same restrictions as the private key – can be read only by the root account and by pki_private_group.

trusted.crt
    This is the complete trust chain of intermediate and root CA certificates, without the server certificate, similar to CA.crt. It is used for automatic OCSP stapling verification by the server, and works with the primary CA in case the alternative Certificate Authority is used for client certificates.

    All of the above filenames are static, which means that the only thing you need to change to select a different PKI realm is the realm directory name.

  Example nginx configuration

To use the domain realm in your nginx configuration, you can add something similar to this example in your configuration file:

server {
    listen [::]:443 ssl;
    # HTTPS support
    ssl_certificate         /etc/pki/realms/domain/default.crt;
    ssl_certificate_key     /etc/pki/realms/domain/default.key;
    # OCSP Stapling support
    ssl_stapling            on;
    ssl_stapling_verify     on;
    ssl_trusted_certificate /etc/pki/realms/domain/trusted.crt;
    # X.509 Client certificate support
    ssl_verify_client       optional;
    ssl_verify_depth        2;
    ssl_trusted_certificate /etc/pki/realms/domain/CA.crt;
}

This configuration explains where each certificate is used, but this is not sufficient to enable HTTPS for the webserver. Refer to the nginx documentation for the rest of the required configuration options.

If you use the debops.nginx Ansible role provided with the project, it has extensive integration with the debops.pki role and can configure the webserver automatically. Usually all you need to do is to make sure the default realm matches the one you would like to use for each server configuration.

  The PKI realm directory structure

This is an example domain realm directory, created on each remote host managed by debops.pki. The current set of certificates active in this realm is provided by the internal debops.pki Certificate Authority:

/etc/pki/realms/
└── domain/
    ├── acme/
    ├── config/
    │   └── realm.conf
    ├── external/
    ├── internal/
    │   ├── alt_intermediate.pem
    │   ├── alt_root.pem
    │   ├── cert.pem
    │   ├── gnutls.conf
    │   ├── intermediate.pem
    │   ├── request.pem
    │   └── root.pem
    ├── private/
    │   ├── key_chain_dhparam.pem
    │   ├── key_chain.pem
    │   ├── key.pem
    │   └── realm_key.pem
    ├── public/
    │   ├── alt_intermediate.pem -> ../internal/alt_intermediate.pem
    │   ├── alt_intermediate_root.pem
    │   ├── alt_root.pem -> ../internal/alt_root.pem
    │   ├── alt_trusted.pem -> alt_intermediate_root.pem
    │   ├── cert_intermediate_dhparam.pem
    │   ├── cert_intermediate.pem
    │   ├── cert.pem -> ../internal/cert.pem
    │   ├── cert.pem.sig
    │   ├── chain.pem -> cert_intermediate_dhparam.pem
    │   ├── full.pem
    │   ├── intermediate_root.pem
    │   ├── root.pem -> ../internal/root.pem
    │   └── trusted.pem -> intermediate_root.pem
    ├── selfsigned/ (optional)
    │   ├── cert.pem
    │   ├── gnutls.conf
    │   ├── request.pem
    │   └── root.pem -> cert.pem
    ├── CA.crt -> public/alt_trusted.pem
    ├── default.crt -> public/chain.pem
    ├── default.key -> private/key.pem
    ├── default.pem -> private/key_chain_dhparam.pem
    └── trusted.crt -> public/trusted.pem

On the Ansible Controller, there's a corresponding directory structure located in the secret/ directory maintained by the debops.secret Ansible role:

secret/pki/
├── realms/
│   ├── by-group/
│   │   └── all/
│   │       └── domain/
│   │           ├── external/
│   │           └── private/
│   └── by-host/
│       └── hostname.example.com/
│           └── domain/
│               ├── external/
│               ├── internal/
│               │   ├── alt_intermediate.pem
│               │   ├── alt_root.pem
│               │   ├── cert.pem
│               │   ├── intermediate.pem
│               │   └── root.pem
│               └── private/
└── requests/
    └── domain/
        └── hostname.example.com/
            └── domain/
                └── request.pem

Your version might not contain all of the shown files and symlinks, for example the alt_*.pem versions of intermediate and root CA certificates are only present if an alternative CA is configured.

Both directories are maintained and kept in sync using two Bash scripts provided by the role, pki-realm and pki-authority. Ansible tasks are used to copy files to and from Ansible Controller to remote hosts.

  How a PKI realm is created

Each PKI realm starts with a simple directory structure created on the Ansible Controller in the secret/ directory:

secret/pki/
└── realms/
    ├── by-group/
    │   └── all/
    │       └── domain/
    │           ├── external/
    │           └── private/
    └── by-host/
        └── hostname.example.com/
            └── domain/
                ├── external/
                ├── internal/
                └── private/

These directories are created at the beginning, so that Ansible can copy private files before the actual PKI realm creation on remote hosts. This can be used to provide a set of identical private RSA keys to multiple hosts at once (using the directories in private/ subdirectories), custom scripts that access external Certificate Authorities (using external/ subdirectories), or DNS challenge API keys.

Next, PKI realm directories are created on the remote host:

/etc/pki/realms/
└── domain/
    ├── acme/
    ├── config/
    │   └── realm.conf
    ├── external/
    ├── internal/
    ├── private/
    └── public/

The config/realm.conf file contains a set of Bash variables that define different parameters of the PKI realm, for example the default DNS domain used to generate the certificates, owner and group of various directories and files, permissions applied to various directory and file types, and so on.

The acme/, external/ and internal/ subdirectories hold data files for different Certificate Authorities. Each CA is described in more detail in a separate document, here is a brief overview:

acme/
    This directory is for certificates issued using ACME <https://en.wikipedia.org/wiki/Automated_Certificate_Management_Environment> (for example Let's Encrypt <https://en.wikipedia.org/wiki/Let's_Encrypt>). It will be activated and used automatically when a host has a public IP address and the nginx webserver is installed and configured to support ACME Challenges (see the debops.nginx <https://github.com/debops/ansible-nginx> role for more details).

external/
    This directory is used to manage certificates signed by an external Certificate Authority. To do this, you need to provide a special script file, which will be executed with a set of environment variables. This can be used to request a certificate from an external CA, like Active Directory or FreeIPA, or download a certificate from an external location.

    An alternative is to provide an already signed cert.pem file and optionally the intermediate.pem and root.pem files.

internal/
    This directory is used by the internal debops.pki Certificate Authority to transfer certificate requests as well as certificates.

    If the internal CA is disabled either globally for a host, or for a particular PKI realm, an alternative directory, selfsigned/ will be created. It will hold a self-signed certificate, not trusted by anything else (not even the host that has created it). This is done, so that services depending on the existence of the private keys and certificates can function correctly at all times.

    The pki-realm script, located in /usr/local/lib/pki on remote hosts, checks which of these directories have valid certificates in order of pki_authority_preference, and the first valid one is used as the "active" directory. Files from the active directory are symlinked to the public/ directory.

    The public/ directory holds the currently active certificates which are symlinks to the real certificate files in one of the active directories mentioned above. Some additional files are also created here by the pki-realm script, namely the certificate chain (server certificate + intermediate certificates) and the trusted chain (intermediate certificates + root certificate). The full certificate contains server certificate + intermediate certificates + root certificate, which might be required by some applications.

    The private/ directory holds the private key of a given realm. Access to this directory and files inside is restricted by UNIX permissions and only a specific system group (usually ssl-cert, but it can be configured) is allowed to access the files inside.

    The next step is the creation of all necessary files, like private/public keys, certificate requests, etc. At this point, if Ansible was provided with a private RSA key to use, it will copy it to the private/ directory. After that, all necessary files are created by the pki-realm script on the remote host. The directory structure changes a bit:

    /etc/pki/realms/
    └── domain/
        ├── acme/
        ├── config/
        │   └── realm.conf
        ├── external/
        ├── internal/
        │   ├── gnutls.conf
        │   └── request.pem
        ├── private/
        │   ├── key.pem
        │   └── realm_key.pem
        ├── public/
        ├── CA.crt -> /etc/ssl/certs/ca-certificates.crt
        └── default.key -> private/key.pem

    As you can see, the configuration of a Certificate Request for an internal CA has been created, and the internal/request.pem file has been generated, using the private/key.pem RSA key. By default, if no root.pem certificate is provided, the system CA certificate store is symlinked as CA.crt.

    Afterwards, Ansible uploads the generated Certificate Signing Request <https://en.wikipedia.org/wiki/Certificate_signing_request> (CSR <https://en.wikipedia.org/wiki/Certificate_signing_request>) to the Ansible Controller for the internal CA to sign (if it's enabled). The CSR is uploaded to the secret/ directory:

    secret/pki/
    ├── realms/
    │   ├── by-group/
    │   │   └── all/
    │   │       └── domain/
    │   │           ├── external/
    │   │           └── private/
    │   └── by-host/
    │       └── hostname.example.com/
    │           └── domain/
    │               ├── external/
    │               ├── internal/
    │               └── private/
    └── requests/
        └── domain/
            └── hostname.example.com/
                └── domain/
                    └── request.pem

    To avoid possible confusion, the secret/pki/requests/domain/ directory points to the "domain" internal CA which is an intermediate CA located under the "root" CA. The hostname.example.com/domain/ directory inside the domain/ directory points to the "domain" realm on the hostname.example.com host.

    When all of the requests from the remote hosts are uploaded to the Ansible Controller, the pki-authority script inside the secret/lib directory takes over and performs certificate signing for all of the currently managed hosts. The certificate named cert.pem is placed in the internal/ directory of each host according to the realm the request came from.

    In addition to the certificates, the CA intermediate and root certificates are also symlinked to the internal/ directory, so that Ansible can automatically copy their contents to the remote hosts. If a particular Certificate Authority indicates that an alternative CA should be present, the alt_*.pem versions of intermediate and root certificates are also symlinked there:

    secret/pki/
    ├── realms/
    │   ├── by-group/
    │   │   └── all/
    │   │       └── domain/
    │   │           ├── external/
    │   │           └── private/
    │   └── by-host/
    │       └── hostname.example.com/
    │           └── domain/
    │               ├── external/
    │               ├── internal/
    │               │   ├── alt_intermediate.pem
    │               │   ├── alt_root.pem
    │               │   ├── cert.pem
    │               │   ├── intermediate.pem
    │               │   └── root.pem
    │               └── private/
    └── requests/
        └── domain/
            └── hostname.example.com/
                └── domain/
                    └── request.pem

    When all of the requests have been processed, Ansible copies the content of the directories to remote hosts. The content of the by-host/ directory is copied first and overwrites all files that are present on remote hosts, the by-group/ directory content is copied only when the corresponding files are not present. This allows the administrator to provide the shared scripts or private keys/certificates as needed, per host, per group or for all managed hosts.

    After certificates signed by the internal CA are downloaded to remote hosts, the directory structure might look similar to:

    /etc/pki/realms/
    └── domain/
        ├── acme/
        ├── config/
        │   └── realm.conf
        ├── external/
        ├── internal/
        │   ├── alt_intermediate.pem
        │   ├── alt_root.pem
        │   ├── cert.pem
        │   ├── gnutls.conf
        │   ├── intermediate.pem
        │   ├── request.pem
        │   └── root.pem
        ├── private/
        │   ├── key.pem
        │   └── realm_key.pem
        ├── public/
        ├── CA.crt -> /etc/ssl/certs/ca-certificates.crt
        └── default.key -> private/key.pem

    Other authority directories (acme/ and external/) might also contain various files.

    After certificates are copied from the Ansible Controller, the pki-realm script is executed again for each PKI realm configured on a given host. It checks which authority directories have valid certificates, picks the first viable one according to pki_authority_preference and activates them.

    Certificate activation entails symlinking the certificate, intermediate and root files to the public/ directory and generation of various chain files: certificate + intermediate, intermediate + root and key + certificate + intermediate (which is stored securely in the private/ directory).

    Some applications do not support a separate dhparam file, and instead expect that the DHE parameters are present after the X.509 certificate chain. If the debops.dhparam <https://github.com/debops/ansible-dhparam> role has been configured on a host and Diffie-Hellman parameter support is enabled in a given PKI realm, DHE parameters will be appended to the final certificate chains (both public and private). When debops.dhparam <https://github.com/debops/ansible-dhparam> regenerates the parameters, the pki-realm script will automatically detect the new ones and update the certificate chains.

    The end result is a fully configured PKI realm with a set of valid certificates available for other applications and services:

    /etc/pki/realms/
    └── domain/
        ├── acme/
        ├── config/
        │   └── realm.conf
        ├── external/
        ├── internal/
        │   ├── alt_intermediate.pem
        │   ├── alt_root.pem
        │   ├── cert.pem
        │   ├── gnutls.conf
        │   ├── intermediate.pem
        │   ├── request.pem
        │   └── root.pem
        ├── private/
        │   ├── key_chain_dhparam.pem
        │   ├── key_chain.pem
        │   ├── key.pem
        │   └── realm_key.pem
        ├── public/
        │   ├── alt_intermediate.pem -> ../internal/alt_intermediate.pem
        │   ├── alt_intermediate_root.pem
        │   ├── alt_root.pem -> ../internal/alt_root.pem
        │   ├── alt_trusted.pem -> alt_intermediate_root.pem
        │   ├── cert_intermediate_dhparam.pem
        │   ├── cert_intermediate.pem
        │   ├── cert.pem -> ../internal/cert.pem
        │   ├── cert.pem.sig
        │   ├── chain.pem -> cert_intermediate_dhparam.pem
        │   ├── full.pem
        │   ├── intermediate_root.pem
        │   ├── root.pem -> ../internal/root.pem
        │   └── trusted.pem -> intermediate_root.pem
        ├── CA.crt -> public/alt_trusted.pem
        ├── default.crt -> public/chain.pem
        ├── default.key -> private/key.pem
        ├── default.pem -> private/key_chain_dhparam.pem
        └── trusted.crt -> public/trusted.pem

    During this process, at various stages special "hook" scripts might be run, which can react to events like realm creation, activation of new certificates and so on.

INTERNAL CERTIFICATE AUTHORITIES

One of the problems in the deployment of a Public Key Infrastructure is the need for the certificates to be signed by a third party, called a Certificate Authority. By using a trusted CA, different entities that communicate with each other can ensure that certificates in use are valid and genuine. A common solution to that problem is the creation of a "self-signed certificate" which can be used by various applications as any other certificate. Unfortunately, this is not a sufficient way to ensure validity of a given certificate in a distributed environment, where various services need to ensure trust without a human intervention because other nodes in the cluster don’t trust the self-signed certificate.

There are various solutions that let you set up a Certificate Authority which then can use an automated API to receive Certificate Signing Requests and issue certificates, however there always is an issue of bootstrapping such a solution. To ensure that the communication between requesting host and the CA is confidential, you need to provide the service over HTTPS, which requires a set of certificates, which require a CA, and so on, and so forth. An alternative is to request a certificate in an already existing Certificate Authority and configure them manually on your own CA server, however this requires human interaction. A proposed solution to this problem is ACME, used for example by the Let's Encrypt, however this solution cannot be used with internal hosts, which still need to be protected.

The debops.pki role solves this problem by creating it's own set of internal Certificate Authorities, located on the Ansible Controller in the secret/ directory (see debops.secret for more details). These Certificate Authorities can be used to bootstrap a new PKI environment, which can then be passed over to a stand-alone CA server located on the network. Alternatively, certificates signed by the internal CA can be used for internal communication between hosts in the cluster, and external certificates can be enabled for public services like websites or mail servers.

  Layout of Certificate Authorities

By default, the debops.pki role creates two Certificate Authorities:

•   a Root Certificate Authority which is used as the "trust anchor" by intermediate Certificate Authorities;

•   a Domain Certificate Authority which issues certificates based on incoming CSRs from remote hosts;

    The directory structure of the Certificate Authorities stored in the secret/ directory on the Ansible Controller:

    secret/pki/
    ├── authorities/
    │   ├── domain/
    │   │   ├── certs/
    │   │   │   └── BDD39DBA82436F2685A67FABD01519B8.pem
    │   │   ├── config/
    │   │   │   ├── authority.conf
    │   │   │   ├── openssl-request.conf
    │   │   │   └── openssl-sign.conf
    │   │   ├── database/
    │   │   │   ├── index
    │   │   │   ├── index.attr
    │   │   │   ├── index.attr.old
    │   │   │   ├── index.old
    │   │   │   ├── serial
    │   │   │   └── serial.old
    │   │   ├── issuer -> ../root/
    │   │   ├── private/
    │   │   │   └── key.pem
    │   │   ├── requests/
    │   │   ├── signed/
    │   │   └── subject/
    │   │       ├── cert.pem
    │   │       └── request.pem
    │   └── root/
    │       ├── certs/
    │       │   ├── F6F915290E08AB1A804E4092A9FEA4C9.pem
    │       │   └── F6F915290E08AB1A804E4092A9FEA4CA.pem
    │       ├── config/
    │       │   ├── authority.conf
    │       │   ├── openssl-request.conf
    │       │   ├── openssl-selfsign.conf
    │       │   └── openssl-sign.conf
    │       ├── database/
    │       │   ├── index
    │       │   ├── index.attr
    │       │   ├── index.attr.old
    │       │   ├── index.old
    │       │   ├── serial
    │       │   └── serial.old
    │       ├── private/
    │       │   └── key.pem
    │       ├── requests/
    │       ├── signed/
    │       └── subject/
    │           ├── cert.pem
    │           └── request.pem
    ├── ca-certificates/
    │   └── by-group/
    │       └── all
    │           └── root-ca.example.com.crt -> ../../../authorities/root/subject/cert.pem
    ├── realms/
    │   └── by-host/
    │       └── hostname.example.com/
    │           └── domain/
    │               ├── external/
    │               ├── internal/
    │               │   ├── cert.pem
    │               │   ├── intermediate.pem -> ../../../../../authorities/domain/subject/cert.pem
    │               │   └── root.pem -> ../../../../../authorities/domain/issuer/subject/cert.pem
    │               └── private/
    └── requests/
        └── domain/
            └── hostname.example.com/
                └── domain/
                    └── request.pem

    The incoming certificate requests are placed in subdirectories of the secret/pki/requests/ directory. Each subdirectory is related to a Certificate Authority, on above directory tree you can see that a request has been uploaded from hostname.example.com host for the domain Certificate Authority.

    The certificates are placed in subdirectories of the secret/pki/realms/ directory. The intermediate CA certificate and root CA certificate files are symlinked in the same subdirectory as the leaf certificate, so that Ansible can copy their contents as regular files to remote host and correct certificate chains can be created in the PKI realm.

  Security of an internal CA

The Certificate Authority is a very vulnerable element of the Private Key Infrastructure. Hosts that have a Root CA certificate in their system certificate store will trust any certificates signed by that CA and it's intermediate Certificate Authorities, therefore protection of the Root CA private key should be taken care of as soon as possible.

The default Root Certificate Authority private key can be found in:

secret/pki/authorities/root/private/key.pem

This key allows you to create new Intermediate Certificate Authorities as well as revoke existing ones (although enforcement of the revocation in the form or distribution of Certificate Revocation Lists or an OCSP service is not currently implemented). You should protect access to it by moving the file to a secure location (preferably an encrypted, offline filesystem) and replacing it with an empty key.pem file (otherwise the debops.pki role will replace the private key and regenerate all of the CA certificates).

Unfortunately, private keys of the Domain Certificate Authority and any other Intermediate Certificate Authority or "Service CA", which is a Root CA used to sign service certificates cannot be protected by taking them offline - the private keys are required to sign certificates. Therefore, it is strongly recommended to store the secret/ directory encrypted, and use it on an encrypted filesystem during use. In DebOps, you can use the EncFS filesystem together with the debops-padlock script to keep the secret/ directory encrypted at rest. You should make sure that access to the plaintext files in secret/ is only possible when it is really needed by unmounting the encrypted filesystem as soon as possible after usage and that only programs which need access can read the files by setting up restrictions like Mandatory Access Control <https://en.wikipedia.org/wiki/Mandatory_access_control> and compartmentalization/sandboxing, to avoid leakage of private keys.

The Certificate Signing Requests created by debops.pki Ansible role contain a random challenge password (different on each run) which is then checked on the Ansible Controller, and only the CSR with correct passwords are signed by the Certificate Authorities. This should prevent signing of Certificate Signing Requests modified by a third party, unless the challenge password can be intercepted (it's currently passed using environment variables).

If for any reason CSR signing cannot be completed, you will need to remove the internal/gnutls.conf and internal/request.pem files on remote hosts to re-initialize the certificate signing.

ACME INTEGRATION

Automated Certificate Management Environment (ACME <https://en.wikipedia.org/wiki/Automated_Certificate_Management_Environment>) is a protocol that allows automated certificate requests, retrieval of certificates and certificate renewal. It was designed to enable easy deployment of X.509 certificates from Let's Encrypt <https://en.wikipedia.org/wiki/Let's_Encrypt>.

The debops.pki Ansible role provides support for the ACMEv2 protocol which is used by default with the Let's Encrypt (there is a possibility to integrate other similar services in the future). Interaction with the ACME Certificate Authority is performed using the acme-tiny <https://github.com/diafygi/acme-tiny> alternative client written in Python.

  Let's Encrypt rate limits

The Let's Encrypt ACME Certificate Authority has different rate limits <https://letsencrypt.org/docs/rate-limits/> related to the number of certificate requests and the number of domains permitted per certificate.

  Certificate renewal

The debops.pki role creates a cron entry for the pki-realm script to be executed periodically for all realms. When a realm has the ACME configuration active, it will check for validity of the certificate, and about a month before the expiration date it will try to renew the certificate automatically.

  Example: Certificate for apex domain and subdomains

The apex domain is the "root" level of your domain. In this example a X.509 certificate for the apex domain example.com is going to be issued. example.com will be listed in the certificate Subject DN. The certificate will also be valid for the subdomains www.example.com, blog.example.com and mail.example.com which are included in the certificate as Subject Alternative Names <https://en.wikipedia.org/wiki/SubjectAltName>.

pki_realms:
  - name: 'example.com'
    acme: True
    acme_subdomains: [ 'www', 'blog', 'mail' ]
    # acme_ca: 'le-staging-v2'

For testing it's strongly advised to uncomment acme_ca with le-staging-v2 to use the staging environment of Let's Encrypt. It does not create a trusted certificate and allows you to avoid problems with the rate limits in the production environment. When you are sure that everything works correctly, comment the staging environment out again to get yourself a valid and trusted X.509 certificate.

  Example: Certificate for subdomains excluding the apex domain

In the example we create a certificate for logs.example.com (certificate Subject) and for mon.example.com (certificate Subject Alternative Names <https://en.wikipedia.org/wiki/SubjectAltName>), which does not include the example.com apex (root) domain.

pki_realms:
  - name: 'logs.example.com'
    acme: True
    acme_default_subdomains: []
    # Can also include different domains like 'mail.example.org'
    # in the same realm.
    acme_domains: [ 'mon.example.com' ]
    # acme_ca: 'le-staging-v2'

Again, for testing it's strongly advised to uncomment acme_ca: le-staging-v2. See above for details.

  ACME configuration variables

The debops.pki role has several default variables which can be used to control ACME support. The most important are:

pki_acme
    Boolean. When True, support for ACME Certificate Authority will be configured for all PKI realms unless disabled on the realm level. By default the role checks if a public IP address is available and a default domain is configured, otherwise the support is disabled automatically.

pki_acme_install
    Boolean. Enable or disable installation of acme-tiny and configuration of ACME support without enabling it for all realms. When this variable is set to True and pki_acme is set to False, ACME support can be enabled independently in each PKI realm. By default, it is set to the same value as pki_acme.

pki_acme_ca
    Name of the ACME Certificate Authority API endpoint to use. Dictionary with endpoints is defined in the pki_acme_ca_api_map variable. By default, le-live-v2 is used which points to the Let's Encrypt Live CA. For testing you can switch the default CA to le-staging-v2 which points to Let's Encrypt Staging CA.

pki_acme_default_subdomains
    List of subdomains which will be added to the default ACME domain and all other domains configured for ACME certificate by default, can be overridden by item.acme_subdomains parameter. By default, the www. subdomain will be added to each domain configured in the realm. Remember that all subdomains need to be correctly configured in the DNS for the Certificate Authority to sign the request.

    Each PKI realm configured in the pki_realms or pki_*_realms variables can have several parameters related to the ACME certificates:

item.name
    Name of the PKI realm. If it has at least one dot, the realm name will be treated as the apex (root) domain to configure for this realm.

item.acme
    Boolean. Enable or disable ACME support per realm.

item.acme_domains
    List of additional apex (root) domains to add in ACME Certificate Signing Request. Each domain will have the default or custom subdomains added to it.

item.acme_default_subdomains
    List of subdomains that should be added to all of the ACME apex (root) domains. If you want to create an ACME certificate only with the apex domain, you might need to set this parameter to an empty list using [] to override pki_acme_default_subdomains.

item.acme_subdomains
    List of subdomains added to each apex (root) domain configured in the ACME certificate. Overrides list of default ACME subdomains.

item.acme_subject
    List of Distinguished Name entries which define the ACME certificate Subject.

ACME TINY INTEGRATION

  Prerequisites

To request and renew ACME certificates with acme-tiny <https://github.com/diafygi/acme-tiny>, a host needs to meet several requirements enforced by this Ansible role:

•   A webserver configured to handle ACME challenges needs to be installed on the host (currently this role supports only http-01 challenges). The debops.nginx <https://github.com/debops/ansible-nginx> role configures ACME support for all servers by default when other conditions are met.

•   A publicly routable IPv4 or IPv6 address is required, so that the Certificate Authority can contact the webserver and check the challenge responses. The debops.pki role detects if a suitable IP address is present, and disables the ACME support otherwise. This can be overridden if necessary for example to allow ACME on an internal server which can handle challenges forwarded through the gateway.

•   Each domain or subdomain requested in a particular certificate needs to be correctly configured in the DNS to point to the host that requests the certificate. This is currently not done automatically and requires intervention by the administrator. If any domain specified in the request is not authorized by the correct ACME challenge, the certificate request won't be successful.

    To request and renew ACME certificates with certbot and a DNS challenge, a host needs to meet several requirements enforced by this Ansible role:

•   A registered domain name.

•   API credentials that enable the DNS challenge to succeed.

    Due to above requirements, the default domain PKI realm configured by the role does not request ACME certificates automatically. Other realms created by the debops.pki role might have ACME support enabled, depending on presence of a public IP address and a configured nginx server.

  Let's Encrypt rate limits

When a certificate request fails, useful error output will be written to acme/error.log. This file will also prevent the pki-realm script from quickly retrying the request and potentially hitting a rate limit. If this file exists and it was modified less than two days ago, the pki-realm script will not perform the request. If the file is older than two days, it will move the file out of the way and perform the request as usual. If you want to retry the request straightaway, you can just move acme/error.log out of the way yourself.

  How ACME certificates are managed

When a new PKI realm is created and support for ACME Certificate Authority is enabled, a separate configuration for a Certificate Request will be created in the acme/ directory. The acme-tiny request does not use a wildcard certificate, instead the default domain and a set of subdomains will be requested (see below for configuration variables). The directory structure at this time looks like this:

/etc/pki/realms/
└── example.com/
    ├── acme/
    │   ├── account_key.pem
    │   ├── openssl.conf
    │   └── request.pem
    ├── config/
    │   └── realm.conf
    ├── external/
    ├── internal/
    │   ├── gnutls.conf
    │   └── request.pem
    ├── private/
    │   ├── key.pem
    │   └── realm_key.pem
    ├── public/
    ├── CA.crt -> /etc/ssl/certs/ca-certificates.crt
    └── default.key -> private/key.pem

When the pki-realm detects the acme/request.pem file, it automatically calls the acme-tiny script using the pki-acme unprivileged account to request the certificate. When the request has completed successfully and an external/cert.pem certificate is found, the certificate will be activated in the public/ directory. The script automatically downloads Let's Encrypt intermediate certificate as well as links the Root CA certificate from the system certificate store provided by the ca-certificates package.

The realm directory after the process is complete:

/etc/pki/realms/
└── example.com/
    ├── acme/
    │   ├── account_key.pem
    │   ├── cert.pem
    │   ├── openssl.conf
    │   ├── intermediate.pem
    │   ├── request.pem
    │   └── root.pem -> /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt
    ├── config/
    │   └── realm.conf
    ├── external/
    ├── internal/
    │   ├── cert.pem
    │   ├── gnutls.conf
    │   ├── intermediate.pem
    │   ├── request.pem
    │   └── root.pem
    ├── private/
    │   ├── key_chain_dhparam.pem
    │   ├── key_chain.pem
    │   ├── key.pem
    │   └── realm_key.pem
    ├── public/
    │   ├── cert_intermediate_dhparam.pem
    │   ├── cert_intermediate.pem
    │   ├── cert.pem -> ../acme/cert.pem
    │   ├── cert.pem.sig
    │   ├── chain.pem -> cert_intermediate_dhparam.pem
    │   ├── intermediate_root.pem
    │   ├── root.pem -> ../acme/root.pem
    │   └── trusted.pem -> intermediate_root.pem
    ├── CA.crt -> public/trusted.pem
    ├── default.crt -> public/chain.pem
    ├── default.key -> private/key.pem
    ├── default.pem -> private/key_chain_dhparam.pem
    └── trusted.crt -> public/trusted.pem

If the request is not successful, you will find a acme/error.log file with log of the acme-tiny session. Check and fix the issue, and remove the log file to re-enable the process again. Otherwise, pki-realm will not request the certificates to avoid rate limit issues explained above.

ACME CERTBOT INTEGRATION

The Certbot <https://certbot.eff.org/about/> application, developed by Electronic Frontier Foundation <https://www.eff.org/> is an ACME client that gives users the ability to request and renew X.509 certificates from Let's Encrypt <https://letsencrypt.org/> or another provider that supports the ACME protocol. The debops.pki role includes support for certbot to allow the X.509 certificates obtained via the service to be used by Ansible roles and other applications integrated with DebOps PKI infrastructure.

The certbot integration with the role is focused specifically on supporting the DNS-01 <https://letsencrypt.org/docs/challenge-types/#dns-01-challenge> ACME challenge, which allows users to request wildcard X.509 certificates and doesn't require a standalone webserver to operate. If you plan to use the HTTP-01 challenge instead, you should refer to the ACME Tiny Integration documentation.

  Prerequisites

To request and renew ACME certificates with certbot, certain requirements must be met. You need to have a registered DNS domain which is served by one of supported DNS providers <https://certbot.eff.org/docs/using.html#dns-plugins>. Each DNS plugin requires a set of API credentials, either a token or an user/password combination - refer to the specific plugin documentation for details about obtaining them.

Due to the requirements above, the default domain PKI realm does not include support for ACME certificates. You will have create a separate PKI realm with name based either on the domain you plan to use or the host's FQDN, which will then hold the private key and X.509 certificate. Other DebOps roles can then use such PKI realm, either automatically or when configured.

  Example ACME certificates with CloudFlare

This is an example debops.pki configuration which enables the certbot integration and uses CloudFlare <https://www.cloudflare.com/> to manage the DNS domain. We will be using the CloudFlare DNS plugin <https://certbot-dns-cloudflare.readthedocs.io/en/stable/> to manage the DNS entries.

The role will request a X.509 certificate for the example.org DNS domain as well as the *.example.org wildcard to cover any potential subdomains. The configuration will be applied only on specific hosts in the cluster, included in the [pki_acme_cloudflare] Ansible inventory group, that way DebOps can configure multiple hosts with the X.509 certificates later on. There's no need to get ACME certificates on backend hosts, the cluster can use the DebOps Internal CA for encrypted communication between nodes. The PKI realm will be named example.org, based on the DNS domain.

The example Ansible inventory looks like this:

[debops_all_hosts]
hostname
[pki_acme_cloudflare]
hostname

You will need to obtain the API access token from CloudFlare web interface. The role expects the credentials in specific files, named after the certbot DNS plugins. On the remote hosts the files should be located in the /etc/pki/realms/<realm>/private/dns-*-credentials.key file. To ensure that the credentials file will be put there by the role, you should place it on the Ansible Controller host in the path (relative to the project directory):

ansible/secret/pki/realms/by-group/<inventory_group>/<realm_name>/private/dns-<dns_plugin>-credentials.key

So, in our case, that would be:

ansible/secret/pki/realms/by-group/pki_acme_cloudflare/example.org/private/dns-cloudflare-credentials.key

In the CloudFlare's case, the actual credentials file will look like:

dns_cloudflare_api_token = <token>

On older OS releases where the certbot version is not recent enough, you might need to use the CloudFlare global API key instead. In such case, the credentials file will look like:

dns_cloudflare_email = <cloudflare account email>
dns_cloudflare_api_key = <secret key>

The last bit is the inventory configuration. The variables will be set on the group level, in the ansible/inventory/group_vars/pki_acme_cloudflare/pki.yml file:

---
# ansible/inventory/group_vars/pki_acme_cloudflare/pki.yml
# Use CloudFlare as the DNS provider for ACME. Changing this variable
# implicitly enables 'certbot' support.
pki_acme_type: 'dns-cloudflare'
# Certbot requires a working e-mail account (it will be validated), you
# might need to specify it if the role uses a non-existent e-mail address.
pki_acme_contacts: [ 'admin@example.org' ]
# If you want to try the staging Let's Encrypt CA to test if the
# certificates are obtained correctly, uncomment this variable.
#pki_acme_ca: 'le-staging-v2'
# Tell the 'pki' role to manage X.509 certificates for these Ansible
# inventory groups.
pki_inventory_groups: [ 'debops_service_pki', 'pki_acme_cloudflare' ]
# Here we define the actual PKI realm, which will be created on each host in
# this inventory group.
pki_group_realms:
  - name: 'example.org'
    acme: True
    # You can define your preferred certificate subject here, the 'CN='
    # attribute is ignored in certificate verification, only SANs are
    # important.
    subject: [ 'CN=example.org' ]
    # This is a list of domains which should be signed by Let's Encrypt CA.
    # Here we get the base domain as well as the wildcard for subdomains.
    domains: [ 'example.org', '*.example.org' ]
    # These parameters are required right now to override the defaults used
    # by the 'pki' role, otherwise you might get a wrong set of domains in the
    # certificate request.
    subdomains: [ '' ]
    default_subdomains: [ '' ]

After the inventory is set up, you can apply the configuration on the host using the debops.pki role, by executing the command:

debops service/pki -l hostname --diff

After it finishes, the certbot aplication should be installed, and symlinks to the /etc/letsencrypt/ directory should be present in the configured PKI realm. You can see an example realm directory structure below:

/etc/pki/realms/
└── example.org
    ├── acme
    │   ├── account_key.pem
    │   ├── cert.pem -> /etc/letsencrypt/live/example.org/cert.pem
    │   ├── intermediate.pem -> /etc/letsencrypt/live/example.org/chain.pem
    │   ├── openssl.conf
    │   └── request.pem
    ├── CA.crt -> /etc/ssl/certs/ca-certificates.crt
    ├── config
    │   └── realm.conf
    ├── default.crt -> public/chain.pem
    ├── default.key -> private/key.pem
    ├── default.pem -> private/key_chain_dhparam.pem
    ├── external
    ├── internal
    │   ├── cert.pem
    │   ├── gnutls.conf
    │   ├── intermediate.pem
    │   ├── request.pem
    │   └── root.pem
    ├── private
    │   ├── dns-cloudflare-credentials.key
    │   ├── key_chain_dhparam.pem
    │   ├── key_chain.pem
    │   ├── key.pem -> /etc/letsencrypt/live/example.org/privkey.pem
    │   └── realm_key.pem
    └── public
        ├── cert_intermediate_dhparam.pem
        ├── cert_intermediate.pem
        ├── cert.pem -> ../acme/cert.pem
        ├── cert.pem.sig
        ├── chain.pem -> cert_intermediate_dhparam.pem
        └── intermediate.pem -> ../acme/intermediate.pem

If there are any issues, certbot command will log everything in the /var/log/letsencrypt/letsencrypt.log log file. you should review it to find what caused the problem and fix it. After that you need to re-initialize the PKI realm by removing its directory (/etc/pki/realms/example.org/) from the remote host and re-running the debops.pki role again. The certificates will be safely stored in the /etc/letsencrypt/ directory so this shouldn't case issues with Let's Encrypt rate limits <https://letsencrypt.org/docs/rate-limits/> if the certificates themselves were registered successfully, but take care otherwise. To avoid rate limiting, you might want to consider enabling the staging CA for testing purposes.

EXTERNAL CERTIFICATES

The PKI realms managed by the debops.pki role support management of private keys and certificates from external Certificate Authorities. You can either provide a set of valid certificates with corresponding private keys, or use a script with a custom environment to request a certificate remotely from an external Certificate Authority.

  Required files

For the pki-realm script to correctly recognize and enable external certificates, you need to provide a set of specific files, either statically through the secret/ directory or by creating them using a script (see below). All paths are relative to the main PKI realm directory, for example /etc/pki/realms/example.com/:

private/key.pem
    Private key used by a specific PKI realm. If not present, the pki-realm script will generate one automatically before executing the external script.

external/cert.pem
    Required. The certificate signed by an external Certificate Authority, in PEM format.

external/intermediate.pem
    Set of intermediate CA certificates which signed the realm certificate. They will be chained together with the realm certificate automatically.

external/root.pem
    The certificate of the Root Certificate Authority. It will be chained with the intermediate CA certificates for OCSP stapling purposes.

external/script
    A custom script (any language should work, however you need to take care of additional dependencies) which will be executed on the remote host if found, with a set of environment variables. The script will be executed inside the external/ directory of a given realm.

  Static private keys and certificates

When the debops.pki Ansible role is run, it creates a set of directories on the Ansible Controller in the secret/ directory:

secret/pki/
└── realms/
    ├── by-group/
    │   ├── all/
    │   │   └── domain/
    │   │       ├── external/
    │   │       └── private/
    │   └── inventory_group/
    │       └── domain/
    │           ├── external/
    │           └── private/
    └── by-host/
        └── hostname.example.com/
            └── domain/
                ├── external/
                ├── internal/
                └── private/

As you can see, the directory structure reflects the Ansible inventory model:

•   realms/by-group/all/ -> inventory/group_vars/all/

•   realms/by-group/inventory_group/ -> inventory/group_vars/inventory_group/

•   realms/by-host/hostname.example.com/ -> inventory/host_vars/hostname.example.com/

    Each of those directories has a set of subdirectories for configured PKI realms, with the external/, internal/ and private/ directories corresponding to the same ones on the remote hosts. Ansible at different stages of the debops.pki role run will copy contents of these directories to remote hosts, in a specific order:

•   contents of the realms/by-host/<hostname> directories for each host will be copied and overwrite already present files;

•   contents of the realms/by-group/<group_name>/ directories will be copied next, but will not overwrite already existing files. Only hosts that are in a given inventory group will receive the corresponding files;

•   and finally, contents of the realms/by-group/all/ directory will be copied to all currently managed remote hosts, but won't overwrite already present files;

    You can use this to distribute already issued certificates with their private keys. Putting them in realms/by-group/all/ directory will ensure that all hosts will have the same set of keys and certificates. If you put them in a specific group directory, only hosts in that group will receive the files. Files put in a specific host directory will only be copied to that host.

    The private keys will be copied to remote hosts before the PKI realm is created, which means that any potential ACME or internal certificates will use them instead of automatically generated ones. This might be useful if you need to have several hosts which use the same set of private keys.

    The above mechanism is used to distribute certificates from internal Certificate Authorities, using the internal/ directory.

    Because files copied from by-group/all/ and by-group/inventory_group/ directories are not overwritten automatically, you will need to remove the corresponding files on remote hosts yourself if you want to update them.

    The pki_inventory_groups default variable is a list of Ansible inventory groups that will have their corresponding directories. You need to specify your custom inventory groups in order to have them "active".

  Certificates managed by a custom script

You can create a custom script and store it in above directories as external/script (permissions are not important). It will be copied to the remote host, made executable and run by the pki-realm script with the external/ directory as the current working directory. You can use this to provide additional files needed by the Certificate Authority. The expected output of the script is a set of files mentioned above.

The script will be executed under the root account, with a set of $PKI_SCRIPT_* environment variables:

$PKI_SCRIPT_REALM
    Contains the name of the current PKI realm, set in item.name parameter.

$PKI_SCRIPT_FQDN
    Contains Fully Qualified Domain Name used as the default domain if the realm does not specify one in it's name.

$PKI_SCRIPT_SUBJECT
    Contains the Distinguished Name, or subject of the certificate, each element separated by the / character, similar to the format of the openssl req -subj option.

$PKI_SCRIPT_DOMAINS
    List of apex (root) domains configured for the realm, separated by the / character.

$PKI_SCRIPT_SUBDOMAINS
    List of subdomains which should be added to each apex domain, each one separated by the / character. The special _wildcard_ name means a wildcard subdomain (*.example.com).

$PKI_SCRIPT_PRIVATE_KEY
    Absolute path to the private key of the current PKI realm.

$PKI_SCRIPT_DEFAULT_CRT
    Absolute path to the current PKI realm certificate chain, expected to be used in the application configuration files.

$PKI_SCRIPT_DEFAULT_KEY
    Absolute path to the current PKI realm private key, expected to be used in the application configuration files.

$PKI_SCRIPT_DEFAULT_PEM
    Absolute path to the current PKI realm combined private key and certificate chain, expected to be used in the application configuration files.

$PKI_SCRIPT_STATE
    A list of PKI realm states separated by the , character. You can inspect this variable to determine the current state of the current realm (initialization, activation of new certificates, changed files) and react to it in the script.

    Because the operation of the PKI realm is stateless, the external script will be executed multiple times during debops.pki run. The state in which the realm is in will be present in the $PKI_SCRIPT_STATE variable and using that you can perform various operations, like issuing a new certificate request when the realm is created.

SYSTEM CA CERTIFICATES

On Debian GNU/Linux and it's derivative distributions, management of Root Certificate Authority certificates is performed by the ca-certificates package. This package maintains a list of active Root CA certificates in the /etc/ca-certificates.conf configuration file, and stores the certificates themselves in the /etc/ssl/certs/ directory.

The debops.pki role has several variables which can be used to control what Root Certificate Authorities are trusted by the system, as well as supports easy installation of local or custom Root Certificate Authorities.

  Configuration of system CA certificates

The /etc/ca-certificates.conf configuration file specifies which certificates will be trusted by the system. This is done by specifying names of certificate files located in the /usr/share/ca-certificates/ directory. The specified certificate files will be included in the system CA store. If a given filename is prefixed with !, a given certificate will be excluded from the system CA store.

By default, Debian hosts automatically trust new Root Certificate Authorities added in the ca-certificates package. To control this, you can use pki_system_ca_certificates_trust_new boolean variable. Setting this variable to True will ensure that new Root CA certificates are trusted. Setting it to False will not enable new CA certificates automatically.

You can use pki_system_ca_certificates_blacklist and pki_system_ca_certificates_whitelist list variables to define which certificates will be excluded/included in the CA store. Each list element is a regexp of the certificate file name. If a given file is found in both lists, it will be excluded from the certificate store.

To find out the names of the certificate files you can use, check the contents of the /etc/ca-certificates.conf configuration file.

  Examples

Blacklist all certificates:

pki_system_ca_certificates_blacklist:
  - '.*'

Blacklist all certificates provided in the Mozilla CA list:

pki_system_ca_certificates_blacklist:
  - 'mozilla/.*'

Blacklist all VeriSign certificates:

pki_system_ca_certificates_blacklist:
  - '.*VeriSign.*'

  Local Root CA certificates

Contents of the secret/pki/ca-certificates/ directory located on the Ansible Controller will be copied to all of the remote hosts by the debops.pki role, to the /usr/local/share/ca-certificates/pki/ directory. After that, they will be automatically added to the system Root CA store using the update-ca-certificates script.

Note that already present CA certificates on remote hosts with the same name are not updated after the first download unless the CA certificates are by host.

The internal Root Certificate Authorities created by the debops.pki role will have their certificates automatically symlinked in the secret/pki/ca-certificates/ directory. You can prevent that by adding an item.system_ca: False parameter in the CA configuration variable.

CUSTOM FILE MANAGEMENT

If you need to, you can copy custom files or even create files with content in YAML templates using a set of list variables. You can use this feature to install private keys and certificates stored as YAML text blocks in a file encrypted with ansible-vault which is unlocked during Ansible run.

Each element of the file list is a dict with specific parameters:

src Required, unless content is specified. Path to a file on the Ansible Controller, which will be copied to the remote host.

content
    Required, unless src is specified. A YAML text block or a Jinja variable, contents of which will be copied to the specified file on the remote host.

dest
    Required. Path to the destination file on the remote host.

owner
    Owner of the created file, by default root.

group
    File group, depending on the file type it will be root (for public files) or a group specified by the pki_private_group variable, usually ssl-cert (for private files).

mode
    File permissions, by default public files are copied with permissions 644 and private files will have 640 permissions.

force
    Boolean. If True (default), an existing file will be replaced with the specified file or contents.

    There are multiple list variables which can be used on multiple inventory levels:

•   all hosts in the inventory: - pki_private_files - pki_public_files

•   hosts in specific inventory group: - pki_group_private_files - pki_group_public_files

•   specific hosts: - pki_host_private_files - pki_host_public_files

    The private files will be copied before PKI realms are created, so that you can provide private keys if you want to. Public files will be copied after PKI realms are created, and internal certificates are signed.

  Examples

Install a custom private key from a Jinja variable on all hosts:

pki_private_files:
  - content: '{{ custom_variable }}'
    dest: '/etc/pki/realms/domain/private/key.pem'

CUSTOM HOOK SCRIPTS

The pki-realm script supports usage of a custom hook scripts located in the /etc/pki/hooks/ directory. These scripts will be executed in alphabetical order (see run-parts(8) <https://manpages.debian.org/run-parts(8)> for more details) within a special environment. You can use that to perform operations after certain actions like creation of a new PKI realm, or activation of a new certificate.

  Execution environment

The hook scripts will be executed by the root account inside the PKI hook directory (/etc/pki/hooks/), with a set of $PKI_SCRIPT_* environment variables:

$PKI_SCRIPT_REALM
    Contains the name of the current PKI realm.

$PKI_SCRIPT_FQDN
    Contains Fully Qualified Domain Name used as the default domain if the realm does not specify one in it's name.

$PKI_SCRIPT_SUBJECT
    Contains the Distinguished Name, or subject of the certificate, each element separated by the / character, similar to the format of the openssl req -subj option.

$PKI_SCRIPT_DOMAINS
    List of apex (root) domains configured for the realm, separated by the / character.

$PKI_SCRIPT_SUBDOMAINS
    List of subdomains which should be added to each apex domain, each one separated by the / character. The special _wildcard_ name means a wildcard subdomain (*.example.com).

$PKI_SCRIPT_PRIVATE_KEY
    Absolute path to the private key of the current PKI realm.

$PKI_SCRIPT_DEFAULT_CRT
    Absolute path to the current PKI realm certificate chain, expected to be used in the application configuration files.

$PKI_SCRIPT_DEFAULT_KEY
    Absolute path to the current PKI realm private key, expected to be used in the application configuration files.

$PKI_SCRIPT_DEFAULT_PEM
    Absolute path to the current PKI realm combined private key and certificate chain, expected to be used in the application configuration files.

$PKI_SCRIPT_STATE
    A list of PKI realm states separated by the , character. You can inspect this variable to determine the current state of the realm (initialization, activation of new certificates, changed files) and react to it in the script.

  Known script states

You can use the $PKI_SCRIPT_STATE variable to check current state of the PKI realm. This variable should always be non-empty, otherwise hook scripts are not executed. Each state can repeat multiple times on the list, but you should avoid multiple execution due to a particular state.

List of known states:

new-realm
    A new PKI realm has been initialized, there are no private keys or certificates present.

new-private-key
    A private key has been generated.

new-internal-request
    A new internal CA certificate signing request has been generated.

new-acme-request
    A new ACME certificate signing request has been generated.

changed-certificate
    A new certificate has been activated, or there has been change of the active Certificate Authority (internal, external, acme).

changed-dhparam
    Diffie-Hellman parameters in the certificate chain have been added/updated.

file-change
    A generic file change notification.

file-deletion
    A file has been deleted.

changed-public-file
    Some of the files in public/ directory have been changed/replaced.

changed-private-file
    Some of the files in private/ directory have been changed/replaced.

  Example nginx hook

This is an example hook script which detects if a given PKI realm is currently used by the nginx server and if so, when a certificate change is detected it reloads the nginx daemon so that new certificate can be activated.

#!/bin/bash
# Reload or restart nginx on a certificate state change
set -o nounset -o pipefail -o errexit
nginx_config="/etc/nginx/nginx.conf"
nginx_sites="/etc/nginx/sites-enabled"
nginx_action="reload"
# Check if current PKI realm is used by the 'nginx' webserver
certificate=$(grep -r "${PKI_SCRIPT_DEFAULT_CRT:-}" ${nginx_sites}/* || true)
# Get list of current realm states
states=( $(echo "${PKI_SCRIPT_STATE:-}" | tr "," " ") )
if [ -n "${certificate}" -a "${#states[@]}" -gt 0 ] ; then
    for state in "${states[@]}" ; do
        if [ "${state}" = "changed-certificate" -o "${state}" = "changed-dhparam" ] ; then
            # Check if current init is systemd
            if $(pidof systemd > /dev/null 2>&1) ; then
                nginx_state="$(systemctl is-active nginx.service)"
                if [ ${nginx_state} = "active" ] ; then
                    if $(/usr/sbin/nginx -c ${nginx_config} -t > /dev/null 2>&1) ; then
                        systemctl ${nginx_action} nginx.service
                    fi
                fi
            else
                nginx_pidfile="$(grep -E '^pid\s+' ${nginx_config} | awk '{print $2}' | cut -d\; -f1)"
                if $(kill -0 $(<${nginx_pidfile}) > /dev/null 2>&1) ; then
                    if $(/usr/sbin/nginx -c ${nginx_config} -t > /dev/null 2>&1) ; then
                        service nginx ${nginx_action}
                    fi
                fi
            fi
            break
        fi
    done
fi

ANSIBLE INTEGRATION

The debops.pki role creates Ansible local facts on each managed remote host. These local facts can be used by other Ansible roles as an idempotent source of PKI-related configuration – they should be accessible from any playbook executed on that host without the requirement of the debops.pki role being a part of it or a role dependency.

The local facts are saved in ansible_local.pki.* namespace. List of available facts:

ansible_local.pki.enabled
    Boolean. Can be used to determine if PKI is enabled on a given host. This doesn't mean that a particular PKI realm has a correctly configured set of private keys and certificates.

ansible_local.pki.acme
    Boolean. Specifies if an ACME environment is enabled on a given host, which means that the acme-tiny script is installed and any PKI realm that is not configured otherwise will try to register an ACME certificate.

ansible_local.pki.internal
    Boolean. Specifies that PKI realms configured on a given host will request certificates in internal Certificate Authority.

ansible_local.pki.path
    Directory where PKI realms are located, by default /etc/pki/realms/.

ansible_local.pki.hooks
    Directory where PKI hooks are located, by default /etc/pki/hooks/.

ansible_local.pki.realm
    Default server realm name configured for this system, should be used as the provider of private key and certificate for a given service.

ansible_local.pki.ca_realm
    Default client realm name configured for this system, should be used as the provider of the CA certificate for a given service.

ansible_local.pki.known_realms
    List which contains names of all PKI realms that might be present on a given remote host. Contents of the list are never removed, only appended. This list can be used to check on the Ansible playbook/role level if a given PKI realm is available to be used.

    The facts listed below are currently static, but are planned to be used in the future for better control over PKI realm directory structure:

ansible_local.pki.crt
    Name of the default certificate symlink located in the PKI realm main directory.

ansible_local.pki.key
    Name of the default private key symlink located in the PKI realm main directory.

ansible_local.pki.pem
    Name of the default private key and certificate bundle symlink located in the PKI realm main directory.

ansible_local.pki.ca
    Name of the default CA certificate symlink located in the PKI realm main directory.

ansible_local.pki.trusted
    Name of the trusted CA chain symlink located in the PKI realm main directory.

DEFAULT VARIABLE DETAILS

Some of debops.pki default variables have more extensive configuration than simple strings or lists, here you can find documentation and examples for them.

  pki_private_groups_present

This list can be used to create system groups that otherwise could be not present when the PKI realm is managed. For example another role creates custom user/group that maintains its own service certificates, but in order to do that, debops.pki is used to manage the PKI realm but at the moment that the debops.pki role is run by Ansible, custom group does not exist, so the Ansible run stops. Therefore, you can create system groups beforehand using this list.

You can define the system groups as simple items, or dictionary values with parameters:

name
    The name of the group to create.

system
    Boolean, by default True. Specify if a given group is a system group.

when
    The value of this variable is checked as a boolean (True/False) to determine if a given system group should be created or not. You can use this as a condition to, for example, create groups only on specific hosts.

  Examples

Ensure two system groups exist, one with a condition:

pki_private_groups_present:
  - 'group1'
  - name: 'group2'
    when: '{{ inventory_hostname in specific_inventory_group }}'

  pki_realms

The set of pki_realms lists can be used to define the configuration of PKI realms located on remote hosts. Each realm keeps a set of private keys and certificates which are signed by the various Certificate Authorities.

Each entry is a dictionary variable with specific parameters. Most of the parameters are optional, and if they are not specified, the pki-realm script that manages the PKI realms should pick the correct options by itself.

List of parameters related to the entire PKI realm:

name
    Required. This is the name of the PKI realm, used as the name of the directory which contains the realm subdirectories, by default stored in the /etc/pki/realms/ directory. The name parameter is interpreted by the role in various ways:

•   the single string name, like "domain", "xxxxaaaayyyy" or similar strings. These PKI realm names can be thought of as "handles" and they don't have any impact on the domains stored in the X.509 certificates. The role will by default use the host's FQDN and DNS domain name to generate such realms, but it can be overridden. Users can create multiple such PKI realms for various purposes.

•   the DNS-based name, which contains dots, like "example.com", "host.example.com" and the like. These PKI realms base their X.509 certificates after the realm name by default. This is also the reason the default PKI realm is named "domain" and not "{{ ansible_domain }}" - users can create a new PKI realm for their default DNS domain on hosts that are reachable publicly and they will automatically get the Let's Encrypt certificates when possible, or can easily use external certificates grabbed from some other CA.

    Some DebOps roles like debops.nginx can check the list of available PKI realms via the local facts and use some other PKI realm rather than the default one automatically. For example, if an user creates an "example.com" PKI realm and then uses the "example.com" DNS domain, standalone or with a subdomain like "sub.example.com", the debops.nginx role will check if a PKI realm named after a given FQDN or DNS domain exists and will use it instead of the "domain" PKI realm used by default. It can be thought of as a shortcut to easily manage X.509 certificates for multiple websites, each one with its own FQDN domain name.

•   the mail-based name, like "user@example.org <user@example.org>" - any PKI realm name which contains the '@' character qualifies as one. These PKI realms were meant to keep the client certificates used to authenticate to services, but this idea was not developed further, so far.

    If the subject parameter is not specified, name parameter is checked to see if it might be a DNS domain (at least 1 dot present in the value). If yes, it will be used as a default domain for a given PKI realm.

    Example:

    pki_realms:
      # Default PKI realm
      - name: 'domain'
      # Custom PKI realm
      - name: 'example.org'

authority
    Specify name of the internal Certificate Authority to send the internal certificate requests to instead of the default one configured in pki_default_authority variable. This should be the "normal" name of the authority, not its subdomain name.

acme
    Optional, boolean. Enable or disable support for ACME Certificate Authority. Can be used to invert the global pki_acme setting per PKI realm if needed, but support for ACME needs to be present on the remote host for it to work (see pki_acme_install variable).

acme_ca
    Optional, name of the ACME API endpoint used by the ACME client. Defaults to pki_acme_ca.

acme_ca_api
    Optional, the URL of the ACME API endpoint. Defaults to the value in pki_acme_ca_api_map (whose key defaults to item.acme_ca).

acme_challenge_dir
    Optional, directory where the ACME client should store responses to ACME CA challenges. Defaults to pki_acme_challenge_dir.

acme_contacts
    Optional, list of (mailto:) URLs that the ACME server can use to contact you for issues related to your account. For example, the server may wish to notify you about server-initiated revocation or certificate expiration. If not specified, the list defined in pki_acme_contacts will be used.

internal
    Optional, boolean. Enable or disable support for internal CA certificates in a given realm. If you disable internal CA support, an alternative, self-signed certificate will be created and enabled automatically.

authority_preference
    Optional. List of directory names (external, acme, internal, selfsigned) which determines the order in which the PKI realm looks for valid certificates. The first found valid certificate is enabled. If not specified, the order configured in pki_authority_preference will be used.

library
    Optional. Specify name of the crypto library used to generate private key and internal certificate requests in a given PKI realm. Either gnutls (default) or openssl.

acme_library
    Optional. Specify name of the crypto library used to generate ACME certificate requests in a given PKI realm. Either openssl (default) or gnutls.

private_dir_group
    Optional. System group which will be set as the group of the private/ directory of a given PKI realm. By default, ssl-cert. It needs to exist, and can be created using pki_private_groups_present list.

private_file_group
    Optional. System group which will be set as the group of the private keys inside of the private/ directory. It needs to exist, and can be created using pki_private_groups_present list.

private_dir_acl_groups
    Optional. List of groups which should be allowed execute (X) permission to the private/ realm directory. The access will be granted using filesystem ACL table. If not specified, the list defined in pki_private_dir_acl_groups will be applied.

private_file_acl_groups
    Optional. List of groups which should be allowed read (r) permission to the files in the private/ realm directory. The access will be granted using filesystem ACL table. If not specified, the list defined in pki_private_file_acl_groups will be applied.

realm_key_size
    Optional. The size of the private key that is to be generated. Defaults to pki_realm_key_size.

dhparam
    Optional, boolean. Enable or disable support for adding the Diffie-Hellman parameters at the end of the certificate chain.

dhparam_file
    Optional. Path to the Diffie-Hellman parameters to include in the certificate chain. If not specified, DHE parameters managed by the debops.dhparam role will be used automatically, if they're available.

selfsigned_sign_days
    Optional. Number of days a selfsigned certificate will be valid for. The default is 365 days.

enabled, when
    Optional, boolean. Enable or disable management of a given realm. If disabled, Ansible will not execute commands related to that realm. The when parameter is meant for automated processing, and enabled should be used as an user option, exposed through the inventory.

    These parameters are related to internal certificates and ACME certificates, respectively:

default_domain
    Optional. Change the default domain used by a given PKI realm. If not specified, the default domain is based on the name parameter if it has at least 1 dot, or it will be taken from pki_default_domain variable which is populated by the ansible_domain variable.

default_subdomains, acme_default_subdomains
    Optional. List of subdomains added to each domain configured in a given PKI realm. A special value _wildcard_ can be used to indicate that a wildcard domain should be present in the certificate.

    If not specified, pki_default_subdomains (for internal CA) and pki_acme_default_subdomains (for ACME CA) will be used. The PKI parameters can be set to empty to override the default variables.

subject, acme_subject
    Optional. The Distinguished Name of the certificate, specified as a list of DN elements. If not specified, a CommonName based on the default domain of the given PKI realm will be used. Empty string elements of the list will be ignored.

    Example:

    pki_realms:
      - name: 'domain'
        subject: [ 'o=Organization Name', 'ou=IT', 'cn=example.org' ]

domains, acme_domains
    Optional. List of domains which should be included in a given certificate. Each domain will include a set of subdomains specified by the other parameters. This can be used to easily create certificates that use multiple domains with similar set of subdomains.

subdomains, acme_subdomains
    Optional. List of subdomains which will be added to each domain specified by the above parameters. The special value _wildcard_ indicates that a wildcard domain should be included in the certificate.

subject_alt_names, acme_alt_names
    Optional. Specify a custom set of SubjectAltNames included in a certificate, as a list. Each element of a list needs to indicate its type in a special format. Currently supported types:

•   a DNS record: [ 'dns:example.org', 'DNS:example.com' ]

•   an IP address: [ 'ip:192.0.2.1', 'IP:2001:db8::dead:beef' ]

•   an URI path: [ 'uri:http://example.org/', 'URI:https://example.com/' ]

•   an email address: [ 'email:root@example.org', 'EMAIL:staff@example.com' ]

    If an element of the list does not specify its type, it will not be included in the certificate request. Different element types can be used in the same list.

    Example:

    pki_realms:
      - name: 'domain'
        subject_alt_names:
          - 'ip:{{ ansible_default_ipv4.address }}'
          - 'uri:https://{{ ansible_domain }}/'
          - 'dns:*.{{ ansible_domain }}'
          - 'dns:{{ ansible_domain }}'

  pki_authorities

The set of pki_authorities lists can be used to define internal Certificate Authorities managed on an Ansible Controller.

List of supported parameters:

name
    Required, a short name of the CA (used for role internals).

subdomain
    Required, the subdomain to be prepended to item.domain.

subject
    Required, list of the X.509 subject elements of the CA certificate.

alt_authority
    Optional, item.name of a cross-signed CA. Only use this if the alternative CA certificate can also be used to form a trust chain. The alternative CA certificates will be stored in the PKI realm directory with the alt_*.pem filename.

ca_sign_days
    Optional, the number of days that the intermediate CA certificate will be valid. If left empty, this value will be calculated by multiplying pki_default_sign_base with pki_default_ca_sign_multiplier.

cert_sign_days
    Optional, the number of days that the client/server certificates will be valid. If left empty, this value will be calculated by multiplying pki_default_sign_base with pki_default_cert_sign_multiplier.

crl Optional, the CRL URL to include in certificates which can be used for certificate status checking. The default is True which will result in http://\$name.\$domain_suffix/crl/. It can be set to False to not include a CRL URL in certificates. Any other value (not matching ^(?:[Tt]rue|[Ff]alse)$) will be included as is as CRL URL.

domain
    Optional, the DNS domain used for the CA. Defaults to pki_ca_domain.

issuer_name
    Optional, name of the higher-authority CA (item.name) that signs this CA. Defaults to an empty string, meaning that this CA is self-signed (and can thus become the root CA).

key_size
    Optional, size of the CA private key in bits. Defaults to 4096.

ocsp
    Optional, the OCSP URL to include in certificates which can be used for certificate status checking. The default is True which will result in http://\$name.\$domain_suffix/ocsp/. It can be set to False to not include a OCSP URL in certificates. Any other value (not matching ^(?:[Tt]rue|[Ff]alse)$) will be included as is as OCSP URL.

name_constraints
    Optional, the X.509 Name Constraints certificate extension to include in certificates which will be used during certificate verification to ensure that the CA is authorized to issue a certificate for the name in question. The default is True which will result in critical, permitted;DNS:${config_domain} (the 'critical, ' part is omitted when item.name_constraints_critical is set to False). It can be set to False to not include X.509 Name Constraints in certificates. Any other value (not matching ^(?:[Tt]rue|[Ff]alse)$) will be included as is as X.509 Name Constraint.

name_constraints_critical
    Optional, boolean for specifying whether to mark the default Name Constraints extension as critical or not. The default is True. The CA/Browser forum recommends this to be enabled (REQUIRING X.509 libraries to support it or to return an error), but mentions that the extension may be disabled for compatibility reasons (ref: Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates (v1.6.4) <https://cabforum.org/wp-content/uploads/CA-Browser-Forum-BR-1.6.4.pdf>).

root_sign_days
    Optional, the number of days that the root CA certificate will be valid. If left empty, this value will be calculated by multiplying pki_default_sign_base with pki_default_root_sign_multiplier.

system_ca
    Optional, boolean that specifies whether the root CA certificate will be added to the system CA certificates. This happens by creating a symlink in the ansible/secret/pki/ca-certificates directory tree. Defaults to True.

type
    Optional, the CA type. Either root, service, server or an empty string. Sets various CA type specific options when running the pki-authority script. These options are not documented, you will have to read the pki-authority script to find out what exactly they do.

    Example:

    pki_authorities:
      - name: 'root'
        subdomain: 'root-ca'
        subject: [ 'c={{ pki_ca_country }}', 'o={{ pki_ca_organization }}',
                   'cn={{ pki_ca_organization }} Internal Root CA' ]
        key_size: '4096'
        crl: False
        ocsp: False
        name_constraints: '{{ "permitted;DNS:example.net,"
                              + "permitted;DNS:.example.net,"
                              + "permitted;DNS:example.com,"
                              + "permitted;DNS:.example.com" }}'

  pki_certbot_configuration

The pki_certbot_*_configuration variables can be used to define the contents of the /etc/letsencrypt/cli.ini configuration file, which contains global certbot configuration <https://certbot.eff.org/docs/using.html#configuration-file>.

  Examples

Recreate the official example configuration file using DebOps:

pki_certbot_configuration:
  - name: 'key-type'
    comment: 'Use ECC for the private key'
    value: 'ecdsa'
  - 'elliptic-curve': 'secp384r1'
  - name: 'rsa-key-size'
    comment: 'Use a 4096 bit RSA key instead of 2048'
    value: 4096
  - name: 'email'
    comment: 'Uncomment and update to register with the specified e-mail address'
    value: 'foo@example.com'
    state: 'comment'
  - name: 'authenticator-standalone'
    option: 'authenticator'
    comment: 'Uncomment to use the standalone authenticator on port 443'
    value: 'standalone'
    state: 'comment'
  - name: 'authenticator-webroot'
    option: 'authenticator'
    comment: |
      Uncomment to use the webroot authenticator. Replace webroot-path with the
      path to the public_html / webroot folder being served by your web server.
    value: 'webroot'
    state: 'comment'
  - name: 'webroot-path'
    value: '/usr/share/nginx/html'
    state: 'comment'
  - name: 'agree-tos'
    comment: 'Uncomment to automatically agree to the terms of service of the ACME server'
    value: True
    state: 'comment'
  - name: 'server'
    comment: 'An example of using an alternate ACME server that uses EAB credentials'
    value: 'https://acme.sectigo.com/v2/InCommonRSAOV'
    state: 'comment'
  - name: 'eab-kid'
    value: 'somestringofstuffwithoutquotes'
    state: 'comment'
  - name: 'eab-hmac-key'
    value: 'yaddayaddahexhexnotquoted'
    state: 'comment'

Define manual hook scripts used for certificate renewal and cleanup:

# Use "certbot" in manual mode, with custom authentication and cleanup
# scripts
pki_acme_type: 'manual'
pki_certbot_configuration:
  - 'authenticator': 'manual'
  - 'manual-auth-hook': '/etc/letsencrypt/scripts/authenticator.php'
  - 'manual-cleanup-hook': '/etc/letsencrypt/scripts/cleanup.php'
  - 'manual-public-ip-logging-ok': True

  Syntax

The contents of the configuration file are defined using the Universal Configuration system. The variables are list of YAML dictionaries. You can use a dictionary key to define a parameter name, and a dictionary value as the value.

If you use name as the key, the configuration options can be defined in a more extended format using specific parameters:

name
    Required. Name of the configuration option. Entries with the same name are merged together and can affect each other; this can be used to activate configuration options conditionally.

option
    Optional. If you need to specify the same configuration option multiple times, you need to use unique name parameters. In that case you can add the option parameter to define the actual option name which will be used in the configuration file.

value
    The value of a configuration option, added in the configuration file as-is.

state
    Optional. If not specified or present, a given configuration option will be included in the generated configuration file. If absent, a given option will not be included in the configuration file. If comment, the option will be included but commented out. if ignore, a given entry will not be processed by the role during execution.

comment
    Optional. String or YAML text block with a comment about a given configuration option added in the generated file.

AUTHOR
Maciej Delmanowski, Robin Schneider

COPYRIGHT
2014-2022, Maciej Delmanowski, Nick Janetakis, Robin Schneider and others