DEBOPS.KIBANA(5)

NAME
debops.kibana - Install and manage Kibana dashboard using Ansible

SYNOPSIS

debops service/kibana [--limit group,host,...] [--diff] [--check] [--tags tag1,tag2,...] [--skip-tags tag1,tag2,...] [<ansible-playbook options>] ...

DESCRIPTION

Kibana <https://en.wikipedia.org/wiki/Kibana> is a web interface which can be used to display and analyze data stored in an Elasticsearch cluster. It is a part of the Elastic Stack. The software is developed by Elastic <https://www.elastic.co/>.

The debops.kibana Ansible role can be used to deploy Kibana behind an nginx reverse proxy which connects to a local or remote Elasticsearch cluster. The role can be used as a dependency by other Ansible roles to allow control over their configuration options in the Kibana configuration file.

GETTING STARTED

  Upstream package is used by default

The Debian and Ubuntu archives contain the elasticsearch packages, however they are very old releases (1.x). The security support for the package in Debian has been discontinued <https://lists.debian.org/debian-security-announce/2015/msg00290.html> due to issues with upstream security policy (Debian Bug #803713 <https://bugs.debian.org/803713>). Debian Developers suggest that in production environments the upstream version should be used <https://bugs.debian.org/829078> to provide adequate security updates.

Due to the above reasons, the role is focused on the upstream Kibana release only (5.x+) to keep compatibility with upstream Elasticsearch. The package installation itself is handled by the debops.elastic_co Ansible role which also manages upstream APT repository configuration.

  Kibana is insecure by default

The Kibana dashboard relies on Elasticsearch to ensure secure communication between cluster nodes. The default configuration expects an Elasticsearch instance installed on the same host, preferably in a "load balancer" mode, that is with no master possibility, no data or ingest functions enabled on the node.

Kibana by default does not include any authentication or ACL support; however the role by default does not configure any access restrictions. If you want to have a simple security solution, you can use kibana__webserver_* variables to configure HTTP Basic Auth in the nginx frontend. Keep in mind that once a user is authenticated using this method, he/she has full access to the Elasticsearch cluster.

You can install additional plugins that provide encrypted connections, authentication, authorization and access control:

•   X-Pack <https://www.elastic.co/products/x-pack>, an Elastic commercial plugin with free trial period. Supports encryption, authentication, access control, integrates with Elasticsearch and Logstash. Since the plugin is developed by the same team, its releases are in parallel with Kibana.

•   Search Guard <http://floragunn.com/searchguard/>, an open source third party plugin with commercial support. Has features comparable with X-Pack, with more basic features like HTTP and transport encryption, basic authentication and access control available free of charge.

  Use as a role dependency

The Kibana main configuration file does not support an include statement or conf.d directory <https://github.com/elastic/elasticsearch/issues/11362>. To mitigate that and allow multiple Kibana configuration sources from other Ansible roles, the debops.kibana role supports operation as a dependent role. This functionality can be used by other Ansible roles to better manage Kibana plugins or extend the configuration without the need to implement the entire role again and with preserved idempotency.

See the Usage as a role dependency for more details.

  Example inventory

To deploy Kibana, you can add the host to the [debops_service_kibana] Ansible inventory group. By default Kibana expects an Elasticsearch instance on the same host to leverage the cluster load balancing. To install an ES node in a load balancer configuration, include the host in the [debops_service_elasticsearch_lb] inventory group:

[debops_service_kibana]
hostname
[debops_service_elasticsearch_lb]
hostname

See the documentation of the debops.elasticsearch role about different Ansible inventory group types and their usage.

  Example playbook

If you are using this role without DebOps, here's an example Ansible playbook that uses the debops.kibana role:

---
- name: Manage Kibana service
  collections: [ 'debops.debops', 'debops.roles01',
                 'debops.roles02', 'debops.roles03' ]
  hosts: [ 'debops_service_kibana' ]
  become: True
  environment: '{{ inventory__environment | d({})
                   | combine(inventory__group_environment | d({}))
                   | combine(inventory__host_environment  | d({})) }}'
  pre_tasks:
    - name: Prepare kibana environment
      import_role:
        name: 'kibana'
        tasks_from: 'main_env'
      tags: [ 'role::kibana', 'role::secret', 'role::kibana:config' ]
  roles:
    - role: keyring
      tags: [ 'role::keyring', 'skip::keyring', 'role::nginx', 'role::elastic_co' ]
      keyring__dependent_apt_keys:
        - '{{ nginx__keyring__dependent_apt_keys }}'
        - '{{ elastic_co__keyring__dependent_apt_keys }}'
    - role: secret
      tags: [ 'role::secret', 'role::kibana', 'role::kibana:config' ]
      secret__directories:
        - '{{ kibana__secret__directories }}'
    - role: apt_preferences
      tags: [ 'role::apt_preferences', 'skip::apt_preferences' ]
      apt_preferences__dependent_list:
        - '{{ elastic_co__apt_preferences__dependent_list }}'
        - '{{ nginx__apt_preferences__dependent_list }}'
    - role: etc_services
      tags: [ 'role::etc_services', 'skip::etc_services' ]
      etc_services__dependent_list:
        - '{{ kibana__etc_services__dependent_list }}'
    - role: ferm
      tags: [ 'role::ferm', 'skip::ferm' ]
      ferm__dependent_rules:
        - '{{ nginx__ferm__dependent_rules }}'
    - role: python
      tags: [ 'role::python', 'skip::python' ]
      python__dependent_packages3:
        - '{{ nginx__python__dependent_packages3 }}'
      python__dependent_packages2:
        - '{{ nginx__python__dependent_packages2 }}'
    - role: nginx
      tags: [ 'role::nginx', 'skip::nginx' ]
      nginx__dependent_servers:
        - '{{ kibana__nginx__dependent_servers }}'
      nginx__dependent_upstreams:
        - '{{ kibana__nginx__dependent_upstreams }}'
    - role: elastic_co
      tags: [ 'role::elastic_co', 'skip::elastic_co' ]
      elastic_co__dependent_packages:
        - '{{ kibana__elastic_co__dependent_packages }}'
    - role: kibana
      tags: [ 'role::kibana', 'skip::kibana' ]

  Ansible tags

You can use Ansible --tags or --skip-tags parameters to limit what tasks are performed during Ansible run. This can be used after a host was first configured to speed up playbook execution, when you are sure that most of the configuration is already in the desired state.

Available role tags:

role::kibana
    Main role tag, should be used in the playbook to execute all of the role tasks as well as role dependencies.

role::kibana:config
    Generate the Kibana configuration taking into account different configuration sources.

USAGE AS A ROLE DEPENDENCY

The debops.kibana role can be used as a dependency by other Ansible roles to manage Kibana main configuration file idempotently. Configuration options from multiple roles can be merged together and included in the configuration file, or removed conditionally.

  Dependent role variables

The role exposes three default variables that can be used by other Ansible roles as dependent variables:

kibana__dependent_name
    Required. Name of the role that uses the debops.kibana as a dependency. This will be used to store the configuration in its own YAML dictionary. The selected name shouldn't be changed, otherwise configuration will be desynchronized.

kibana__dependent_configuration
    Required. List of the Kibana configuration options defined in the same format as the main configuration. See kibana__configuration for more details.

kibana__dependent_state
    Optional. If not specified or present, the configuration will be included in the /etc/kibana/kibana.yml configuration file and stored in the secret/ directory on the Ansible Controller. if absent, the configuration will be removed from the generated configuration file.

  Dependent configuration storage and retrieval

The dependent configuration from other roles is stored in the secret/ directory on the Ansible Controller (see debops.secret for more details) in a JSON file, with each role configuration in a separate dictionary. The debops.kibana role reads this file when Ansible local facts indicate that the Kibana service is installed, otherwise a new empty file is created. This ensures that the stale configuration is not present on a new or re-installed host.

The YAML dictionaries from different roles are be merged with the main configuration in the kibana__combined_configuration variable that is used to generate the final configuration. The merge order of the different kibana__*_configuration variables allows to further affect the dependent configuration through Ansible inventory if necessary, therefore the Ansible roles that use this method don't need to provide additional variables for this purpose themselves.

  Example role variables

This file shows an example set of default variables included in a role that uses the debops.kibana role as a dependency:

---
# State of the application deployment
application__deploy_state: 'present'
# Kibana configuration for application
application__kibana__dependent_configuration:
  - name: 'application.option'
    value: True
  - 'application.other.option': False

  Example role playbook

This file shows an example playbook for a role that uses the debops.kibana role as a dependency:

---
- name: Manage application
  collections: [ 'debops.debops' ]
  hosts: [ 'debops_service_kibana_application' ]
  become: True
  environment: '{{ inventory__environment | d({})
                   | combine(inventory__group_environment | d({}))
                   | combine(inventory__host_environment  | d({})) }}'
  pre_tasks:
    - name: Prepare kibana environment
      import_role:
        name: 'kibana'
        tasks_from: 'main_env'
      tags: [ 'role::kibana', 'role::secret', 'role::kibana:config' ]
  roles:
    - role: secret
      tags: [ 'role::secret', 'role::kibana', 'role::kibana:config' ]
      secret__directories:
        - '{{ kibana__secret__directories }}'
    - role: kibana
      tags: [ 'role::kibana' ]
      kibana__dependent_role: 'application'
      kibana__dependent_state: '{{ application__deploy_state }}'
      kibana__dependent_configuration:
        - '{{ application__kibana__dependent_configuration }}'
    - role: application
      tags: [ 'role::application' ]

DEFAULT VARIABLE DETAILS

Some of debops.kibana default variables have more extensive configuration than simple strings or lists, here you can find documentation and examples for them.

  kibana__configuration

The kibana__*_configuration variables define the Kibana configuration options that are set in the /etc/kibana/kibana.yml configuration file.

The main Kibana configuration file format is YAML. The reference documentation <https://www.elastic.co/guide/en/kibana/current/settings.html> defines two YAML formats recognized by Kibana, hierarchical (YAML dictionary keys are indented), or flat (YAML dictionary keys are separated by dots). This role focuses only on the latter, flat format since it's used everywhere in the Kibana documentation and seems to be the preferred method for majority of the configuration options.

For quick reference, Kibana configuration file contains options in the following format (similar to Elasticsearch):

cluster.name: example-cluster
node.name: node-1
network.host: [ _local_, _site_ ]
bootstrap.memory_lock: true
discovery.zen.minimum_master_nodes: 3

The kibana__*_configuration variables are a YAML lists of dictionaries. Each YAML dictionary defines an option, or redefines a previously defined option (the variables are flattened and then processed in order).

The first YAML dictionary key of each option (in above case, cluster, node, network, bootstrap, discovery is significant, and is used to separate configuration options into sections defined by the kibana__configuration_sections variable.

Configuration options can be defined as YAML dictionaries directly, with the key being the name of the option, and value being its value:

kibana__configuration:
  - 'cluster.name': 'example-cluster'
  - 'node.name': 'node-1'
  - 'network.host': [ '_local_', '_site_' ]
  - 'bootstrap.memory_lock': True
  - 'discovery.zen.minimum_master_nodes': 3

The extended YAML dictionary format is detected if a YAML dictionary contains a name key. The dictionaries support specific parameters:

name
    String. The name of the Kibana option.

value
    The value of the Kibana option. Can be a string, a number, a boolean or a YAML list.

comment
    An optional comment added to the option, either as a string or a YAML text block.

state
    If not specified or present, the option will be included in the configuration. If absent, the option will not be included. If comment, the option will be present but commented out (it's an internal feature and may not work reliably for all cases).

options
    Optional, a YAML dictionary with keys being the "leaf" configuration names of the primary key, and value being their values. This parameter can be used to group several similar configuration options together in the generated configuration file, for readability. When this parameter is used, the "leaf" part of the main configuration name is discarded, and only used as a marker for these parameters. An example configuration:

    kibana__configuration:
      - name: 'node.meta.host_type'
        comment: 'Node type'
        options:
          'master': True
          'data':   True
          'ingest': True

    The above configuration should result in:

    # Node type
    node.master: true
    node.data: true
    node.ingest: true

raw Optional, a YAML text block. The name of the configuration option will be discarded and used only as a marker for these parameters. The contents of the raw key will be added as-is to the configuration file. You can use this to include more extensive configuration defined as a hierarchical YAML structure. An example configuration which should be equivalent to the previous example:

    kibana__configuration:
      - name: 'node.meta.host_type'
        raw: |
          # Node type
          node.master: true
          node.data: true
          node.ingest: true

    You should make sure that the identation of the YAML parameters is consistent through the configuration file.

  kibana__configuration_sections

The /etc/kibana/kibana.yml configuration file is structured in informal 'sections", each section contains configuration options from a specific group (node, cluster, etc.). The kibana__configuration_sections contains a YAML list of sections and option types to associate with them. The order of the entries on the list determines the order of the sections in the finished configuration file.

Each section definition is a YAML dictionary with specific parameters:

name
    Name of the section, stored as a comment.

part or parts
    A string or a YAML list of configuration option prefixes (first YAML dictionary key of a given configuration option). Only the parts defined for a given section will be included in that section.

    After all of the sections are processed, any left over configuration options not matched with a particular section will be added at the end of the configuration file.

  kibana__plugins

The kibana__*_plugins variables are YAML lists that can be used to install or remove Kibana plugins. Support for plugin management using these variables is minimalistic; you can install plugins known by the Elastic plugin repository <https://www.elastic.co/guide/en/kibana/current/kibana-plugins.html>, or from an URL. More involved management can be done by creating a separate role and using debops.kibana as a role dependency to manage configuration if necessary. See Usage as a role dependency for more details.

Each element of the list is a YAML dictionary with specific parameters:

name
    Required. Name of the plugin that shows up in the output of the

    bin/kibana-plugin list

    command, without the version information included. This parameter will be used to check the state of the plugin.

url Optional. If the plugin is distributed via an URL, you can provide it here for the plugin management script to use instead of the plugin name.

state
    Optional. If not specified or present, the plugin and its configuration will be installed. If absent the plugin and its configuration will be removed.

state
    Optional. The system user used for plugin management. Defaults to kibana__user. Certain plugins like X-Pack generate files on installation which Kibana needs to have write permissions to.

configuration or config
    Optional. Custom configuration for a given plugin, in the format recognized by the main configuration template.

    See kibana__configuration for more details.

  Examples

Install a LogTrail plugin:

kibana__plugins:
  - name: 'logtrail'
    url: 'https://github.com/sivasamyk/logtrail/releases/download/0.1.13/logtrail-5.4.0-0.1.13.zip'

AUTHOR
Maciej Delmanowski

COPYRIGHT
2014-2022, Maciej Delmanowski, Nick Janetakis, Robin Schneider and others