# Confmap | Status | | | ------------- |-----------| | Stability | [stable]: logs, metrics, traces | | Issues | [![Open issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector?query=is%3Aissue%20is%3Aopen%20label%3Apkg%2Fconfmap%20&label=open&color=orange&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector/issues?q=is%3Aopen+is%3Aissue+label%3Apkg%2Fconfmap) [![Closed issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector?query=is%3Aissue%20is%3Aclosed%20label%3Apkg%2Fconfmap%20&label=closed&color=blue&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector/issues?q=is%3Aclosed+is%3Aissue+label%3Apkg%2Fconfmap) | | [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@mx-psi](https://www.github.com/mx-psi), [@evan-bradley](https://www.github.com/evan-bradley) | [stable]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#stable # High Level Design ## Conf The [Conf](confmap.go) represents the raw configuration for a service (e.g. OpenTelemetry Collector). ## Provider The [Provider](provider.go) provides configuration, and allows to watch/monitor for changes. Any `Provider` has a `` associated with it, and will provide configs for `configURI` that follow the ":" format. This format is compatible with the URI definition (see [RFC 3986](https://datatracker.ietf.org/doc/html/rfc3986)). The `` MUST be always included in the `configURI`. The scheme for any `Provider` MUST be at least 2 characters long to avoid conflicting with a driver-letter identifier as specified in [file URI syntax](https://datatracker.ietf.org/doc/html/rfc8089#section-2). ## Converter The [Converter](converter.go) allows implementing conversion logic for the provided configuration. One of the most common use-case is to migrate/transform the configuration after a backwards incompatible change. ## Resolver The `Resolver` handles the use of multiple [Providers](#provider) and [Converters](#converter) simplifying configuration parsing, monitoring for updates, and the overall life-cycle of the used config providers. The `Resolver` provides two main functionalities: [Configuration Resolving](#configuration-resolving) and [Watching for Updates](#watching-for-updates). ### Configuration Resolving The `Resolver` receives as input a set of `Providers`, a list of `Converters`, and a list of configuration identifier `configURI` that will be used to generate the resulting, or effective, configuration in the form of a `Conf`, that can be used by code that is oblivious to the usage of `Providers` and `Converters`. `Providers` are used to provide an entire configuration when the `configURI` is given directly to the `Resolver`, or an individual value (partial configuration) when the `configURI` is embedded into the `Conf` as a values using the syntax `${configURI}`. **Limitation:** - When embedding a `${configURI}` the uri cannot contain dollar sign ("$") character unless it embeds another uri. - The number of URIs is limited to 100. ```terminal Resolver Provider Resolve │ │ ────────────────►│ │ │ │ ┌─ │ Retrieve │ │ ├─────────────────────────►│ │ │ Conf │ │ │◄─────────────────────────┤ foreach │ │ │ configURI │ ├───┐ │ │ │ │Merge │ │ │◄──┘ │ └─ │ │ ┌─ │ Retrieve │ │ ├─────────────────────────►│ │ │ Partial Conf Value │ │ │◄─────────────────────────┤ foreach │ │ │ embedded │ │ │ configURI │ ├───┐ │ │ │ │Replace │ │ │◄──┘ │ └─ │ │ │ Converter │ ┌─ │ Convert │ │ │ ├───────────────►│ │ foreach │ │ │ │ Converter │ │◄───────────────┤ │ └─ │ │ │ │ ◄────────────────┤ │ ``` The `Resolve` method proceeds in the following steps: 1. Start with an empty "result" of `Conf` type. 2. For each config URI retrieves individual configurations, and merges it into the "result". 3. For each embedded config URI retrieves individual value, and replaces it into the "result". 4. For each "Converter", call "Convert" for the "result". 5. Return the "result", aka effective, configuration. #### (Experimental) Append merging strategy for lists You can opt-in to experimentally combine slices instead of discarding the existing ones by enabling the `confmap.enableMergeAppendOption` feature flag. Lists are appended in the order in which they appear in their configuration sources. This will **not** become the default in the future, we are still deciding how this should be configured and want your feedback on [this issue](https://github.com/open-telemetry/opentelemetry-collector/issues/8754). ##### Example Consider the following configs, ```yaml # main.yaml receivers: otlp/in: processors: attributes/example: actions: - key: key value: "value" action: upsert exporters: otlp/out: extensions: file_storage: service: pipelines: traces: receivers: [ otlp/in ] processors: [ attributes/example ] exporters: [ otlp/out ] extensions: [ file_storage ] ``` ```yaml # extra_extension.yaml extensions: healthcheckv2: service: extensions: [ healthcheckv2 ] pipelines: traces: ``` If you run the Collector with following command, ``` otelcol --config=main.yaml --config=extra_extension.yaml --feature-gates=confmap.enableMergeAppendOption ``` then the final configuration after config resolution will look like following: ```yaml # main.yaml receivers: otlp/in: processors: attributes/example: actions: - key: key value: "value" action: upsert exporters: otlp/out: extensions: file_storage: healthcheckv2: service: pipelines: traces: receivers: [ otlp/in ] processors: [ attributes/example ] exporters: [ otlp/out ] extensions: [ file_storage, healthcheckv2 ] ``` Notice that the `service::extensions` list is a combination of both configurations. By default, the value of the last configuration source passed, `extra_extension`, would be used, so the extensions list would be: `service::extensions: [healthcheckv2]`. > [!NOTE] > By enabling this feature gate, only the extensions, receivers and exporters under the `service` section are merged. ### Watching for Updates After the configuration was processed, the `Resolver` can be used as a single point to watch for updates in the configuration retrieved via the `Provider` used to retrieve the “initial” configuration and to generate the “effective” one. ```terminal Resolver Provider │ │ Watch │ │ ───────────►│ │ │ │ . . . . . . │ onChange │ │◄────────────────────┤ ◄───────────┤ │ ``` The `Resolver` does that by passing an `onChange` func to each `Provider.Retrieve` call and capturing all watch events. Calling the `onChange` func from a provider triggers the collector to re-resolve new configuration: ```terminal Resolver Provider │ │ Watch │ │ ───────────►│ │ │ │ . . . . . . │ onChange │ │◄────────────────────┤ ◄───────────┤ │ | | Resolve │ │ ───────────►│ │ │ │ │ Retrieve │ ├────────────────────►│ │ Conf │ │◄────────────────────┤ ◄───────────┤ │ ``` An example of a `Provider` with an `onChange` func that periodically gets notified can be found in provider_test.go as UpdatingProvider ## Troubleshooting ### Null Maps Due to how our underlying merge library, [koanf](https://github.com/knadh/koanf), behaves, configuration resolution will treat configuration such as ```yaml processors: ``` as null, which is a valid value. As a result if you have configuration `A`: ```yaml receivers: nop: processors: nop: exporters: nop: extensions: nop: service: extensions: [nop] pipelines: traces: receivers: [nop] processors: [nop] exporters: [nop] ``` and configuration `B`: ```yaml processors: ``` and do `./otelcorecol --config A.yaml --config B.yaml` The result will be an error: ``` Error: invalid configuration: service::pipelines::traces: references processor "nop" which is not configured 2024/06/10 14:37:14 collector server run finished with error: invalid configuration: service::pipelines::traces: references processor "nop" which is not configured ``` This happens because configuration `B` sets `processors` to null, removing the `nop` processor defined in configuration `A`, so the `nop` processor referenced in configuration `A`'s pipeline no longer exists. This situation can be remedied 2 ways: 1. Use `{}` when you want to represent an empty map, such as `processors: {}` instead of `processors:`. 2. Omit configuration like `processors:` from your configuration.