DEBOPS.RSYSLOG(5)

NAME
debops.rsyslog - Manage syslog daemon using Ansible

SYNOPSIS

debops service/rsyslog [--limit group,host,...] [--diff] [--check] [--tags tag1,tag2,...] [--skip-tags tag1,tag2,...] [<ansible-playbook options>] ...

DESCRIPTION

The rsyslog <https://www.rsyslog.com/> package is used to read, process, store and forward system logs in different ways, on local or remote systems. The debops.rsyslog role can be used to easily configure log forwarding to a central log server, as well as store logs on the filesystem or other storage backends.

GETTING STARTED

  Initial configuration

The debops.rsyslog default configuration is designed to closely resemble the Debian rsyslog package defaults. The same system logs will be generated, although with slightly longer log rotation. If the operating system is Debian, rsyslog will be run on a privileged root account; if the system is Ubuntu, an unprivileged syslog account will be used by default.

  Configuration filename extensions

The configuration order is important, and to aid support of configuration from other roles, the /etc/rsyslog.conf configuration file includes other configuration files with different filename extensions at certain points of the configuration:

/etc/rsyslog.d/*.input
    These files define configuration of the rsyslog input modules <https://www.rsyslog.com/doc/v8-stable/configuration/modules/idx_input.html> which can be used as data sources.

/etc/rsyslog.d/*.template
    These configuration files can be used to create custom templates used by rsyslog in different parts of the configuration.

/etc/rsyslog.d/*.conf
    These files are included by default. They are meant to be used for configuration of the local system logs, the extension is used to preserve compatibility with Debian package conventions.

/etc/rsyslog.d/*.output
    These files define configuration of the rsyslog output modules <https://www.rsyslog.com/doc/v8-stable/configuration/modules/idx_output.html> which can be used as targets by various local and remote rulesets defined later on.

/etc/rsyslog.d/*.ruleset
    These configuration files are meant to be used to define log matching rules specific to a given system, to store logs in different files.

/etc/rsyslog.d/*.remote
    These configuration files are meant to store configuration for logs coming from other systems over the network. These rules will be defined in a separate "ruleset" called remote which is used by the UDP and TCP input modules. This way the local (system) logs and remote logs from other hosts can be managed separately and shouldn't mix with each other.

  Quick start: log forwarding

Log forwarding <https://www.rsyslog.com/sending-messages-to-a-remote-syslog-server/> tells rsyslogd server to send all, or selected, syslog messages to remote syslog server(s). The debops.rsyslog role is tailored to perform log forwarding over TLS to a central syslog server using DNS SRV Records for the following service:

_syslog._tcp.{{ rsyslog__domain }} (default port 6514)

If:

•   the resource records are found; and

•   the host is not configured to receive logs via the rsyslog__remote_enabled variable; and

•   the debops.pki role has been configured on the host

    Then the debops.rsyslog role will configure each target server to send syslog messages over TLS. This configuration can be found and changed in the rsyslog__default_forward and the rsyslog__default_rules variables.

  Quick start: receiving remote logs

The role does not configure rsyslogd service to receive log messages from the network by default. To enable this, you can specify a list of allowed IP addresses and/or CIDR subnets which are allowed to send syslog messages using the rsyslog__allow, rsyslog__group_allow and/or rsyslog__host_allow variables. Defining these in the inventory will tell the role to configure rsyslog to accept remote logs and store them in subdirectories under the /var/log/remote/ directory. The debops.ferm and the debops.logrotate roles will be used to configure the IPTables firewall and log rotation respectively.

This behaviour is controlled by the rsyslog__remote_enabled variable.

  Example inventory

The debops.rsyslog role is included by default in the DebOps common.yml playbook and does not need to be specifically enabled.

To enable the debops.rsyslog role on a given host or group of hosts not managed by DebOps, you need to add that host to the [debops_service_rsyslog] Ansible inventory group:

[debops_service_rsyslog]
hostname

  Example playbook

Here's an example playbook which uses debops.rsyslog role:

---
- name: Manage rsyslog service
  collections: [ 'debops.debops', 'debops.roles01',
                 'debops.roles02', 'debops.roles03' ]
  hosts: [ 'debops_all_hosts', 'debops_service_rsyslog' ]
  become: True
  environment: '{{ inventory__environment | d({})
                   | combine(inventory__group_environment | d({}))
                   | combine(inventory__host_environment  | d({})) }}'
  roles:
    - role: etc_services
      tags: [ 'role::etc_services', 'skip::etc_services' ]
      etc_services__dependent_list:
        - '{{ rsyslog__etc_services__dependent_list }}'
    - role: apt_preferences
      tags: [ 'role::apt_preferences', 'skip::apt_preferences' ]
      apt_preferences__dependent_list:
        - '{{ rsyslog__apt_preferences__dependent_list }}'
    - role: ferm
      tags: [ 'role::ferm', 'skip::ferm' ]
      ferm__dependent_rules:
        - '{{ rsyslog__ferm__dependent_rules }}'
    - role: cron
      tags: [ 'role::cron', 'skip::cron' ]
    - role: logrotate
      tags: [ 'role::logrotate', 'skip::logrotate' ]
      logrotate__dependent_config:
        - '{{ rsyslog__logrotate__dependent_config }}'
    - role: rsyslog
      tags: [ 'role::rsyslog', 'skip::rsyslog' ]

UNPRIVILEGED SYSLOG AND ENCRYPTED CONNECTIONS

The rsyslog daemon can be used in a privileged or an unprivileged mode. In a privileged mode the daemon is run on the root account, has access to all required files, sockets, etc. In the unprivileged mode, rsyslog daemon is started in a privileged mode first, opens required sockets/ports and then drops all of its privileges and supplementary UNIX groups.

The debops.rsyslog role allows you to select which mode is used by configuring the rsyslog__unprivileged boolean variable. By default, to preserve original configuration, the role enables unprivileged mode on Ubuntu hosts, leaving the configuration privileged on Debian hosts.

The unprivileged operation places certain restrictions on the system configuration. In particular, the rsyslog process only uses its primary system group, dropping any additional groups the user is in. This means, that using TLS with the default configuration maintained by debops.pki role becomes problematic - unprivileged rsyslog process uses only its own primary group, so it cannot access private keys to allow encrypted connections.

There are multiple solutions to this problem, which you can use. Each one has pros and cons, and you should evaluate the selected method in a development environment before implementing it in production to avoid issues.

  Run the daemon in privileged mode

This method is the default on Debian hosts. Ubuntu hosts use the unprivileged mode by default, and reverting to the privileged mode should work, but that hasn't been evaluated yet.

The daemon will be run with the root permissions, and there shouldn't be any issues with file access. Enabling TLS connections should work out of the box. On the downside, an externally accessible service is running with root permissions, so you should be careful what hosts have access to it, this is controlled using the firewall.

To enable this mode, set the following in the Ansible inventory:

rsyslog__unprivileged: False

This will enforce the privileged operation.

  Grant access to private keys by additional groups

The debops.pki role that maintains the DebOps X.509 infrastructure, allows you to specify additional system groups, which should have access to the private keys. This should be configured before the role creates the private keys, because the permissions are not enforced afterwards - this means that you will need to recreate the private keys and certificates, or update the permissions manually. Additional permissions are granted using the filesystem ACL support.

To enable rsyslog to get access to the private keys in unprivileged mode by the syslog system groups, configure in the Ansible inventory:

# Ensure that needed system group is present
pki_private_groups_present:
  - name: 'syslog'
    system: True
# Add custom ACL groups to private files and directories for all PKI realms
pki_private_dir_acl_groups:  [ 'syslog' ]
pki_private_file_acl_groups: [ 'syslog' ]
# Or, add custom ACL groups to private files only in default PKI realm
pki_default_realms:
  - name: 'domain'
    acme: False
    private_dir_acl_groups:  [ 'syslog' ]
    private_file_acl_groups: [ 'syslog' ]

After the PKI realm is recreated, you can check the result using command:

root@logs:~# getfacl /etc/pki/realms/domain/private
root@logs:~# getfacl /etc/pki/realms/domain/private/key.pem

You should see the syslog entry on the list of groups that can access the respective files and directories. When the rsyslog process is restarted, it should be able to access the private keys without issues. To enable the unprivileged mode on Debian hosts, you might want to enforce it through the Ansible inventory. Here it is, with example log forwarding to remote host with TCP over TLS:

# Enable unprivileged operation
rsyslog__unprivileged: True
# Enable TLS support
rsyslog__capabilities: [ 'tls' ]
# Forward logs over encrypted TCP connection
rsyslog__forward: [ '*.* @@logs.{{ ansible_domain }}:6514' ]

  Create custom PKI realm for syslog

The debops.pki role allows you to create multiple PKI realms with different purposes and configuration. If you don't want to modify and existing infrastructure in place, creating a separate internal realm just for syslog might be an easy alternative.

To create new PKI realm, add this to the Ansible inventory for all involved hosts:

# Ensure that needed system group is present
pki_private_groups_present:
  - name: 'syslog'
    system: True
# Create custom realm for syslog
pki_realms:
  - name: 'syslog'
    acme: False
    private_dir_group:  'syslog'
    private_file_group: 'syslog'

When the new PKI realm is created, the private directory and files inside should be owned by the syslog group. This should ensure that the rsyslog daemon in the unprivileged mode, running as syslog user, should have access to them. The certificates should be signed by existing debops.pki Certificate Authority, so they should be trusted by all hosts in the cluster.

When the new PKI realm is ready, you can tell debops.rsyslog role to use it:

# Enable unprivileged operation
rsyslog__unprivileged: True
# Enable TLS support
rsyslog__capabilities: [ 'tls' ]
# Change the default PKI realm used by rsyslog
rsyslog__pki_realm: 'syslog'
# Forward logs over encrypted TCP connection
rsyslog__forward: [ '*.* @@logs.{{ ansible_domain }}:6514' ]

When the new configuration is applied, you should see in the rsyslog configuration files that the daemon is using the correct private key and certificate.

  Testing encrypted connections

To make sure that the logs are sent over an encrypted connection, you can check the traffic using the tshark command. On the receiving server, run the command:

root@logs:~# tshark -i eth0 -f "dst port 514 or dst port 6514" \
             -d tcp.port==514,syslog -d tcp.port==6514,syslog

This will output packets that are sent to TCP ports 514 (plaintext traffic) and 6514 (TLS traffic). Afterwards, on remote hosts try sending some test log messages:

user@host:~$ logger Test log message, please ignore

If the connection is not encrypted, you should see something similar to this (notice the unencrypted contents of the packet):

9 132.751792 192.0.2.2 -> 192.0.2.1 Syslog 133 USER.NOTICE: May 16 14:06:05 host user: Test log message, please ignore\n

If the connection is encrypted, output should look similar to this:

9 132.751792 192.0.2.2 -> 192.0.2.1 Syslog 164 \027\003\003\000]\000\000\000\000\000\000\000\037\257\301,\030\365\311\324\023qR9\b\352\203\256\306\260T\023\022\016g\271\220\325\031\250\326\323\0045\3549\270\277>\205\301\256\325\234\246\tzt\333\255\002\006K"\254\334\021wB1\353\f\356,u\344\220\207d\024o\305\234\b\201\003Js[\2533\261\207\231?k\230J

Of course, the contents of the logs should appear normally in the log files, for example in /var/log/remote/hosts/host/syslog you should see:

May 16 14:06:05 host user: Test log message, please ignore

DEFAULT VARIABLE DETAILS

Some of debops.rsyslog default variables have more extensive configuration than simple strings or lists, here you can find documentation and examples for them.

  rsyslog__forward

The rsyslog__default_forward, rsyslog__forward, rsyslog__group_forward and rsyslog__host_forward variables are lists used to define forwarding rules for rsyslog. Because the daemon configuration is ordered, the forward statements should be set in a specific place in the configuration. You can of course define your own forwarding rules instead of using these specific variables, if you wish.

You can check the rsyslog remote forward documentation <https://www.rsyslog.com/sending-messages-to-a-remote-syslog-server/> to see how to forward logs to other hosts. Each configuration entry should be specified in a separate YAML list element. The entries can be simple FQDN hostnames which will be configured to use TCP connections over TLS and port 6514; alternatively you can define more detailed configuration using specific parameters:

selector
    What type of logs to forward to another server, by default *.* (all facilities, all priorities).

target
    The FQDN of the syslog server where logs will be forwarded.

port
    The port to which rsyslog will connect, by default 6514.

protocol
    The protocol which should be used for connections, by default tcp.

resume_retry_count
    Number of times rsyslog should try to reconnect to the syslog server when connection is lost, by default 100.

queue_type
    The type of the internal queue to use for this server, by default linkedList.

queue_size
    The size of the message queue, by default 10000.

netstream_driver
    The NetStream driver used by the imtcp module, defaults to the value of rsyslog__default_netstream_driver.

driver_mode
    The NetStream driver mode, defaults to the value of rsyslog__default_driver_mode.

driver_authmode
    The NetStream driver authentication mode, defaults to the value of rsyslog__default_driver_authmode.

  rsyslog__configuration

The rsyslog__*_configuration variables define the contents of the /etc/rsyslog.conf configuration file. This is the main rsyslog configuration, additional config snippets can be found in the /etc/rsyslog.d/ directory, which can be managed using the rsyslog__rules variables.

  Examples

Enable kernel log input module in rsyslog service:

rsyslog__configuration:
  - name: 'module_imklog'
    raw: |
      module(load="imklog")

Other examples can be found in the rsyslog__original_configuration default variable. The rsyslog__default_configuration contains changes to the original options applied by the role.

  Syntax

The variables are lists of YAML dictionaries with specific parameters:

name
    Required. An identification of the configuration entry, not used otherwise. Multiple configuration entries with the same name parameter are merged together; this can be used to modify already defined entries.

state
    Optional. If not defined or present, a given configuration entry will be present in the generated config file. If absent, the entry will not be included in the configuration file. If comment, the entry will be present, but commented out. If ignore, a given configuration entry will not be evaluated during role execution.

comment
    Optional. String or a YAML text block with additional comments about a given configuration entry, included in the generated file.

raw Optional. String or YAML text block with the rsyslog.conf(5) configuration options or RainerScript definitions <https://www.rsyslog.com/doc/v8-stable/rainerscript/index.html>, included in the generated file as-is.

section
    Optional. Specify the configuration section in which a given entry should be included. The sections are defined using the rsyslog__configuration_sections variables; the default sections available are: modules, global, templates, output, rules and unknown. If the section is not defined, the entry will be added to the unknown section.

  rsyslog__configuration_sections

The rsyslog__*_configuration_sections variables define what sections are present in the /etc/rsyslog.conf configuration file. Sections will be included in the file in the order they appear in the configuration variables.

The default set of configuration sections, defined in the rsyslog__default_configuration_sections variable, is based on the recommendations from the rsyslog.conf(5) manual page.

  Examples

Define a section with a custom title:

rsyslog__configuration_sections:
  - name: 'custom_section'
    title: 'Example configuration'

  Syntax

Each configuration entry is a YAML dictionary with specific parameters:

name
    Required. The name of the section, used in the section parameter of the /etc/rsyslog.conf configuration. Multiple entries with the same name parameter are merged together.

title
    Optional. This parameter can be used to override the section name which is used in the generated configuration file.

state
    Optional. If not specified or present, a given section will be included in the generated configuration file. If absent, the section will not be included in the file. If ignore, a given configuration entry will not be evaluated during role execution. If hidden, the section's title comment will be hidden in the generated configuration file.

weight
    Optional. A positive or negative number which can be used to affect the order of sections in the generated configuration file. Positive numbers add more "weight" to the section making it appear "lower" in the file; negative numbers substract the "weight" and therefore move the section upper in the file.

  rsyslog__rules

The rsyslog__*_rules variables define the configuration stored in the /etc/rsyslog.d/ directory. Configuration files in the directory can be named with different "extensions", each one imported at different point in the /etc/rsyslog.conf configuration file. The supported extensions are: .module, .template, .conf, .output, .ruleset, .remote.

  Examples

See the rsyslog__default_rules variable for example configurations.

  Syntax

Each configuration file is described using YAML dictionaries with specific parameters:

name
    Required. Name of the configuration file in the /etc/rsyslog.d/ directory. Multiple configuration entries with the same name parameter will be merged together.

divert
    Optional, boolean. If specified and True, the debops.rsyslog role will use the dpkg-divert command to move specified originaL configuration file out of the way before generating the configuration from a template. This parameter can be used to modify the rsyslogd configuration provided by the system packages.

divert_to
    Optional. If the divert parameter is enabled, using this parameter you can specify the filename to divert the file to. The diversion will be confined to /etc/rsyslog.d/ directory. This can be used to change the order of the packaged configuration files if needed.

comment
    Optional. A comment added at the beginning of the file.

raw A string or YAML text block with the rsyslog.conf(5) <https://manpages.debian.org/rsyslog.conf(5)> configuration, included in the generated config files as-is.

state
    Optional. Either present or absent. If undefined or present a given configuration file present, if absent, given configuration file will be removed. If ignore, a given configuration entry will not be evaluated during execution. This parameter can be used to conditionally enable or disable parts of the configuration.

options
    Optional. This is a list of YAML dictionaries with configuration definition which should be included in the given file. If the raw parameter is present on the file level, the options list is ignored.

    Each configuration entry in the options list needs to be defined as a YAML dictionary with parameters:

name
    Required. An identifier for a particular section of the configuration file, not used otherwise. The options with the same name parameter from different configuration file entries are merged together.

comment
    Optional. A comment added at the beginning of a given section.

raw A string or YAML text block with the rsyslog.conf(5) <https://manpages.debian.org/rsyslog.conf(5)> configuration, included in the generated config file section as-is.

state
    Optional. Either present or absent. If undefined or present a given configuration file or configuration section will be present, if absent, given configuration file or section will be removed. This parameter can be used to conditionally enable or disable parts of the configuration.

AUTHOR
Maciej Delmanowski

COPYRIGHT
2014-2022, Maciej Delmanowski, Nick Janetakis, Robin Schneider and others