DEBOPS.ICINGA(5)

NAME
debops.icinga - Manage Icinga 2 Agent using Ansible

SYNOPSIS

debops service/icinga [--limit group,host,...] [--diff] [--check] [--tags tag1,tag2,...] [--skip-tags tag1,tag2,...] [<ansible-playbook options>] ...

DESCRIPTION

The Icinga <https://www.icinga.com/> project is a monitoring solution which can be used to monitor infrastructure hosts and services. Icinga can use variety of remote agents as well as SNMP to monitor remote hosts.

DebOps provides support for Icinga 2, split into several roles:

•   the debops.icinga role is the Icinga 2 Agent role, usually installed on all hosts in the cluster that should be monitored. Different instances can function as masters, satellites or clients depending on the configuration.

•   the debops.icinga_db role can be used to configure the database support for Icinga 2, used by the web interface. Both PostgreSQL and MariaDB databases are supported.

•   the debops.icinga_web role manages the Icinga 2 Web interface, including installation of external modules like Icinga 2 Director.

GETTING STARTED

  Example inventory

The debops.icinga role is not included in the common.yml playbook and needs to be activated by adding a host to a specific Ansible inventory group. The role is designed to be used on all hosts in the cluster, therefore the easiest way to do this is to include the main DebOps group in the Icinga inventory group.

[debops_all_hosts]
icinga-master
hostname1
hostname2
[debops_service_icinga:children]
debops_all_hosts

See the Deployment guide documentation for more details about deploying Icinga in DebOps.

  Example playbook

If you are using this role without DebOps, here's an example Ansible playbook that uses the debops.icinga role:

---
- name: Configure Icinga service
  collections: [ 'debops.debops', 'debops.roles01',
                 'debops.roles02', 'debops.roles03' ]
  hosts: [ 'debops_service_icinga' ]
  become: True
  environment: '{{ inventory__environment | d({})
                   | combine(inventory__group_environment | d({}))
                   | combine(inventory__host_environment  | d({})) }}'
  pre_tasks:
    - name: Prepare icinga environment
      import_role:
        name: 'icinga'
        tasks_from: 'main_env'
      tags: [ 'role::icinga', 'role::secret' ]
  roles:
    - role: keyring
      tags: [ 'role::keyring', 'skip::keyring', 'role::icinga' ]
      keyring__dependent_apt_keys:
        - '{{ icinga__keyring__dependent_apt_keys }}'
    - role: secret
      tags: [ 'role::secret', 'role::icinga' ]
      secret__directories:
        - '{{ icinga__secret__directories | d([]) }}'
    - role: apt_preferences
      tags: [ 'role::apt_preferences', 'skip::apt_preferences' ]
      apt_preferences__dependent_list:
        - '{{ icinga__apt_preferences__dependent_list }}'
    - role: etc_services
      tags: [ 'role::etc_services', 'skip::etc_services' ]
      etc_services__dependent_list:
        - '{{ icinga__etc_services__dependent_list }}'
    - role: unattended_upgrades
      tags: [ 'role::unattended_upgrades', 'skip::unattended_upgrades' ]
      unattended_upgrades__dependent_origins: '{{ icinga__unattended_upgrades__dependent_origins }}'
    - role: ferm
      tags: [ 'role::ferm', 'skip::ferm' ]
      ferm__dependent_rules:
        - '{{ icinga__ferm__dependent_rules }}'
    - role: icinga
      tags: [ 'role::icinga', 'skip::icinga' ]

  Ansible tags

You can use Ansible --tags or --skip-tags parameters to limit what tasks are performed during Ansible run. This can be used after a host was first configured to speed up playbook execution, when you are sure that most of the configuration is already in the desired state.

Available role tags:

role::icinga
    Main role tag, should be used in the playbook to execute all of the role tasks as well as role dependencies.

role::icinga:register
    This tag can be used to register an Icinga 2 instance in the Icinga Director via the REST API. Already registered nodes are not registered again.

DEPLOYMENT GUIDE

  Deployment design

The Icinga 2 support in DebOps is designed around installation of Icinga Agents on all hosts and presence of 1 master/director node. The Icinga 2 Director is not mandatory, without it the debops.icinga role can be used to configure Icinga 2 nodes manually, either through Ansible inventory or via role dependent variables. However, with Icinga Director, the role can register new Icinga nodes automatically as long as the specified host templates are prepared beforehand.

It's advisable to test the deployment in a development environment before applying it in production. This should let you find out possible issues with DNS and PKI configuration you might encounter.

  Icinga 2 inter-node communication and PKI

You might need to allow connections to the Icinga API interface, by default on port 5665, through the firewall. You can do this by setting the icinga__allow or its group or host equivalent in the inventory. Usually only the master host needs the access opened, unless you plan to initiate connections from the master to the clients.

Icinga 2 uses X.509 certificates for internal communication between the nodes, therefore the correct DNS records for the hosts are required. When DNS is not configured properly beforehand, communication between the cluster nodes can be disrupted.

At the moment, the debops.icinga role uses the PKI infrastructure maintained by the debops.pki role to provide X.509 certificates for Icinga 2 agents. Due to that, automatic registration of the Icinga 2 agents external to the cluster in the Icinga 2 CA is not possible at this time. It can be implemented later if there's demand for it.

  DNS SRV records

The role detects the master and director servers by using DNS SRV Records for the following services:

_icinga-master._tcp.{{ icinga__domain }} (default port 5665)
_icinga-director._tcp.{{ icinga__domain }} (default port 443)

There can be multiple master and director DNS SRV records. The role will configure multiple master nodes in the zones.conf configuration file, however only one director node will be used.

WARNING: The role uses the DNS SRV resource records to determine if a given host should be configured as the "master" host, or a client host that should register itself with the "master". Therefore, you should create the DNS SRV records beforehand, as all the Icinga 2 nodes will see themselves as "master" nodes otherwise, and won't connect to the "master" node.

For details on how to configure DNS SRV records, see DNS SRV Records.

  Manual override

If SRV records are not feasible or you don't want to publish them everywhere, you can specify the required information using Ansible inventory. For example, to define the master and director nodes for all hosts in the Ansible inventory, you can add the variables below in the ansible/inventory/group_vars/all/icinga.yml configuration file:

---
# Icinga Masters
icinga__master_nodes:
  - target: 'icinga.example.org'
    port: '5665'
# Icinga Directors
icinga__director_nodes:
  - target: 'icinga.example.org'
    port: '443'

With this information, the role can correctly resolve the needed host and port details to use in other variables.

  Initial deployment

This is an example Ansible inventory for deployment of the full Icinga "stack" in DebOps environment. The debops.icinga role is applied on all hosts in the environment, however the debops.icinga_db and debops.icinga_web are applied only on the master host.

[debops_all_hosts]
icinga-master
hostname1
hostname2
[debops_service_icinga:children]
debops_all_hosts
[debops_service_postgresql_server]
icinga-master
[debops_service_icinga_db]
icinga-master
[debops_service_icinga_web]
icinga-master

By default the web interface is configured on the icinga. subdomain, you can change this by setting the icinga_web__fqdn variable.

It's best to start the deployment on the Icinga master node, by setting up the local Icinga 2 Agent, and the web interface with the Icinga Director. You can login to the web interface using the root username and the password stored in the secret/icinga_web/auth/<inventory_hostname>/credentials/root/password file (see debops.secret for more details).

After logging in, and if you haven't already done so, try applying the debops.icinga role to some other host which is to be monitored. If everything is configured correctly, the role should automatically register the new host in Icinga via the Director REST API. Subsequent execution of the role will not change the status of the host in Icinga, but if you remove the host from the web interface and re-run the debops.icinga role, the host will be registered again.

By default, the role will automatically create two host templates, generic-host and icinga-agent-host (the latter depending on the former) as part of the host registration process and will register new hosts using the icinga-agent-host template. See icinga_web__director_templates for more details. Note that if you delete these templates they will, by default, be recreated every time a host is (re-)registered with the Director.

You can define the list of templates automatically applied during registration using the icinga__director_register_*_templates default variables.

USAGE AS A ROLE DEPENDENCY

The debops.icinga role can be used as a dependency by other Ansible roles to manage contents of the /etc/icinga2/ directory idempotently. Configuration options from multiple roles can be merged together and included in the Icinga 2 configuration, or removed conditionally.

  Dependent role variable

The role exposes the icinga__dependent_configuration variable which can be used to define Icinga configuration files by other Ansible roles through the role dependent variables.

The variable is an YAML list with YAML dictionaries as entries. A short format of the configuration uses the dictionary key as a name of the dependent role and dictionary value as that role's configuration, in the format defined by icinga__configuration variable, respectively (see playbook excerpt below):

roles:
  - role: icinga
    icinga__dependent_configuration:
      - role_name: '{{ role_name__icinga__dependent_configuration }}'

The extended version of the configuration uses YAML dictionaries with specific parameters:

role
    Required. Name of the role, used to save its configuration in a YAML dictionary on the Ansible Controller. Shouldn't be changed once selected, otherwise the configuration will be desynchronized.

config
    Required. YAML list with definition of the Icinga configuration files in the same format defined by icinga__configuration variables.

state
    Optional. If not specified or present, the configuration will be included in the generated configuration files. If absent, the configuration will be removed from the configuration files. If ignore, a given configuration entries will be skipped during data evaluation and won't affect any existing entries.

    An example extended configuration (playbook excerpt):

    roles:
      - role: icinga
        icinga__dependent_configuration:
          - role: 'role_name'
            config: '{{ role_name__icinga__dependent_configuration }}'

    The above configuration layout allows for use of the multiple role dependencies in one playbook by providing configuration of each role in a separate configuration entry.

  Dependent configuration storage and retrieval

The dependent configuration from other roles is stored in the secret/ directory on the Ansible Controller (see debops.secret for more details) in a JSON file (one for each variable), with each role configuration in a separate dictionary. The debops.icinga role reads these files when Ansible local facts indicate that the Icinga support is installed, otherwise empty files are created. This ensures that the stale configuration is not present on a new or re-installed host.

The YAML dictionaries from different roles are merged with the main configuration in the icinga__combined_configuration variable that are used to generate the final configuration. The merge order of the different icinga__*_configuration variables allows to further affect the dependent configuration through Ansible inventory if necessary, therefore the Ansible roles that use this method don't need to provide additional variables for this purpose themselves.

DEFAULT VARIABLE DETAILS

Some of debops.icinga default variables have more extensive configuration than simple strings or lists, here you can find documentation and examples for them.

  icinga__configuration

The icinga__*_configuration variables specify the contents of the Icinga 2 configuration located in the /etc/icinga2/ directory. The variables are combined together using the icinga__combined_configuration variable, which means that users don't need to copy entire values to the inventory to change it.

Because Icinga 2 configuration language is extensive, the role is focused only on conditional activation of the specific files and sections inside. You will want to read the upstream documentation <https://www.icinga.com/docs/icinga2/latest/doc/04-configuring-icinga-2/> to learn how to configure Icinga 2 using its DSL.

The vriables contain lists of YAML dictionaries, each dictionary can be defined using specific parameters:

name
    Required. The name of the file located in the /etc/icinga2/ directory, for example icinga2.conf. This can also include subdirectories, for example conf.d/templates.conf. Missing subdirectories will be automatically created by the role.

    This parameter is used as a handle for merging multiple configuration entries together.

filename
    Optional. Alternative filename with optional subdirectories. Can be used to override the filename specified in the name parameter. Usually not used.

divert
    Optional, boolean. When defined and True, this parameter marks the file as belonging to a .deb package. The original file will be diverted or reverted depending on the state parameter to allow for unobtrusive package upgrades. Diverted files have .dpkg-divert suffix and are ignored by Icinga 2.

state
    Optional. Specify the desired state of a given configuration file. Possible states:

•   present: default if not defined. The configuration file will be generated, any original files will be diverted to preserve them.

•   absent: the file will be removed. Any diverted files will be returned to their original state.

•   init: the configuration of a given file will be primed, but will not be actually implemented by the role. This can be used to prepare configuration files to be activated conditionally.

•   ignore: a given configuration entry will be ignored during template generation. This can be used to disable specific configuration entries conditionally.

•   divert: only divert a given configuration file without generating a custom one. The files will be reverted back when the state is set to absent.

•   feature: only enable/disable the feature state in the /etc/icinga2/features-enabled/ directory.

feature_name
    Optional. Specify name of the symlink managed in the /etc/icinga2/features-enabled/ directory, without the .conf suffix. This should be only used with configuration files located in the features-available/ subdirectory, otherwise the generated symlinks will be broken.

    The name parameter is not correlated with the feature_name, and is used only for configuration merging.

feature_state
    Optional. If set and present, the symlink to a particular feature file will be created. If absent, the symlink to a particular feature will be removed, thus disabling it.

owner
    Optional. Specify the UNIX account owner of the configuration file. If not specified, root will be the owner.

group
    Optional. Specify the UNIX group of the configuration file. If not specified, root will be the group.

mode
    Optional. Specify the file attributes. If not specified, 0644 will be used by default.

no_log
    Optional, boolean. If set and True, Ansible will not log the generation of a given configuration file. This might be useful for files with sensitive data like passwords.

comment
    Optional. String or YAML text block with a comment, included in the beginning of the configuration file.

value
    Optional. String or YAML text block that contains the Icinga 2 configuration, specified using Icinga 2 DSL <https://www.icinga.com/docs/icinga2/latest/doc/17-language-reference/>. It will be included in the configuration file as-is.

options
    Optional. List of configuration snippets that will be included in the file. It's an alternative to a single value entry which can be used to conditionally enable or disable parts of the configuration file. Options lists from different configuration entries are merged together and can affect each other.

    Each list element is a YAML dictionary with specific parameters:

name
    An element identifier, it is used for merging options lists from different configuration entries and is ignored otherwise. It should be an unique string.

value
    Required. String or YAML text block with Icinga 2 configuration written in is DSL. Will be included as-is in the configuration file.

comment
    Optional. String or YAML text block with a comment which will be added before a given element.

state
    Optional. If not set or present, the configuration option will be included in the generated file. If absent, the configuration option will not be included in the generated file. If ignore, a given list element is not evaluated by Ansible and will be ignored. If comment, the configuration option will be included in the configuration file, but commented out.

weight
    Optional. A positive or negative number that affects the order of the elements in the options list. It can be used to move configuration lower or higher in the configuration file.

  Examples

Many examples can be found in the role defaults/main.yml file.

Add simple host checks in separate directory:

icinga__configuration:
  - name: 'conf.d/hosts/host1.{{ ansible_domain }}/host.conf'
    comment: 'Custom host configuration'
    options:
      - name: 'host'
        value: |
          object Host "host1.{{ ansible_domain }}" {
            address = "host1.{{ ansible_domain }}"
            check_command = "hostalive"
          }
        state: 'present'
  - name: 'conf.d/hosts/host2.{{ ansible_domain }}/host.conf'
    value: |
      object Host "host2.{{ ansible_domain }}" {
        address = "host2.{{ ansible_domain }}"
        check_command = "hostalive"
      }
    state: 'present'

Define a set of services and apply them to hosts in a specific zone:

icinga__configuration:
  - name: 'zones.d/master/services.conf'
    state: 'present'
    options:
      - name: 'service_load'
        value: |
          apply Service "load" {
            import "generic-service"
            check_command = "load"
            command_endpoint = host.vars.client_endpoint
            assign where host.vars.client_endpoint
          }
        state: 'present'
      - name: 'service_procs'
        value: |
          apply Service "procs" {
            import "generic-service"
            check_command = "procs"
            command_endpoint = host.vars.client_endpoint
            assign where host.vars.client_endpoint
          }
        state: 'present'
  - name: 'zones.d/master/host1.{{ ansible_domain }}.conf'
    options:
      - name: 'object_zone'
        value: |
          object Zone "host1.{{ ansible_domain }}" {
            endpoints = [ "host1.{{ ansible_domain }}" ]
            parent = "master"
          }
        state: 'present'
      - name: 'object_endpoint'
        value: |
          object Endpoint "host1.{{ ansible_domain }}" {
            host = "host1.{{ ansible_domain }}"
          }
        state: 'present'
      - name: 'object_host'
        value: |
          object Host "host1.{{ ansible_domain }}" {
            import "generic-host"
            address = "host1.{{ ansible_domain }}"
            vars.notification["mail"] = {
              groups = [ "icingaadmins" ]
            }
            vars.client_endpoint = name
          }
        state: 'present'

  icinga__custom_files

The icinga__*_custom_files variables can be used to copy additional hosts to hosts managed with the debops.icinga role. The variables are lists, each list entry is a YAML dictionary with specific parameters:

content
    String or YAML text block with file contents. Cannot be set with the src parameter at the same time.

src Absolute path to the file located on the Ansible Controller which will be copied to the remote host. Cannot be set with the content parameter at the same time.

dest
    Required. Absolute path where the file will be placed on the remote host.

owner
    Optional. Specify the owner of the file. If not specified, root will be the owner.

group
    Optional. Specify the default group of the file. If not specified, root will be the default group.

mode
    Optional. Specify the file attributes. If not specified, 0755 will be set (by default the role assumes that the managed custom files are scripts).

force
    Optional, boolean. If True (default), the role will override already existing file. If False, the role will not override an existing file.

state
    Optional. If not set or present, the file will be copied to the remote host. This can be used to conditionally copy files depending on other factors.

  Examples

Add a simple hello world script in Icinga 2 scripts/ directory:

icinga__custom_files:
  - content: |
      #!/bin/sh
      echo "Hello, world!"
    dest: '/etc/icinga2/scripts/hello-world.sh'

AUTHOR
Maciej Delmanowski

COPYRIGHT
2014-2022, Maciej Delmanowski, Nick Janetakis, Robin Schneider and others