Metadata-Version: 2.1
Name: websites-metrics-consumer
Version: 0.0.1
Summary: An application that consumes metrics from Kafka messages and store the results into ta postgres db
Home-page: https://github.com/antoniodimariano/metrics_consumer
Author: Antonio Di Mariano
Author-email: antonio.dimariano@gmail.com
License: Apache 2.0
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: psycopg2
Requires-Dist: confluent-kafka
Requires-Dist: avro-python3
Requires-Dist: requests

[![CircleCI](https://circleci.com/gh/antoniodimariano/metrics_consumer/tree/master.svg?style=svg&circle-token=f7da2eeb3c4a1dbbe2705fda45fdffe02e8ef1ce)](https://circleci.com/gh/antoniodimariano/metrics_consumer/tree/master)
# Websites Metrics Consumer

#### Author: Antonio Di Mariano - antonio.dimariano@gmail.com

## Description

This service is part of the Websites Monitoring Application, a small distributed application 
that aims to produce and collect metrics about the availability of one or more websites. 

This service is responsible for consuming messages about metrics being produced to an Apache Kafka Avro Topic by a different service.
The main action of this service is to store the incoming message into a PostgreSQL database.

# Run

If you want to run from the source code, go to the directory `websites_metrics_consumer` and run `python main`

If you want to use it as package (suggested method) install  `pip3 install websites_metrics_consumer`

Then

1. Set the ENV variables as show in this README.md
2. Then use it this way

```python
def consume_message():
    from websites_metrics_consumer.main import start_consumer
    start_consumer()
    


```


# Requirements

* Python >=3.8

# Run

`python main`

# Dependencies

* confluent_kafka==1.7.0
* psycopg2==2.9.1

# Run test

**NOTE**: for simplicity I am assuming you have a running local instance of PostgreSQL with the following information

* database='test'
* user='postgresql'
* password='test123'
* host='localhost'

`python -m unittest tests/test_db_operations.py`
`python -m unittest test/test_events_handler_class.py`


# Service ENV configuration 

If you are using a Broker that uses `SSL`, like Aiven.com, you need to download the following certificates files and copy them to a folder
 * Access Key
 * Access Certificate
 * CA Certificate

for more information about SSL and Kafka please read https://docs.confluent.io/3.0.0/kafka/ssl.html

| ENV Variable  | VALUE | DESCRIPTION                                                                       |
|---------------|------|------------------------------------------------------------------------------------|
| brokers    | string   | Required. The FQDN of the Apache Kafka Brokers.|
| schema_registry    | string   | Required. The FQDN of the Service Registry.|
| ssl_ca_location    | string   | Required. he relative path to your ca.pem|
| ssl_certificate_location    | string   | Required. The relative path to your service.cert|
| ssl_key_location    | string   | Required. The relative path to your service.key |
| security_protocol    | string   | Required. SSL.|
| persistence_conf    | string   | Required. The full string for connecting to your PostgreSQL database.|
| group_id    | string   | Optional. The consumer group id to assign to the consumer(s) . The default value is `metrics_consumer`|
| consumer_topics    | string or list  | Optional. The Kafka Avro Topic where to consume messages from. The defaul value is `websites_metrics` |
| logging_level    | string   | Optional. The level of logging to use for the built-in `logging` package. The default is `logging.INFO`  |


**Example of mandatory ENV variables to use** 

* brokers=kafka-xxx-yyy-abc.com:12026
* schema_registry=https://user:password@kafka-xxx-yyy-abc.com:12029
* ssl_ca_location=configuration/ca.pem
* ssl_certificate_location=configuration/service.cert
* ssl_key_location=configuration/service.key
* security_protocol=SSL
* persistence_conf=postgres://user:password@mydbhost.com:12024/metrics?sslmode=require



If your broker requires SASL authentication, like Confluent Cloud, these are the ENV variables to include

| ENV Variable  | VALUE | DESCRIPTION                                                                       |
|---------------|------|------------------------------------------------------------------------------------|
| brokers    | string   | Required. The FQDN of the Apache Kafka Brokers.|
| schema_registry    | string   | Required. The FQDN of the Service Registry.|
| sasl_username    | string   | Required. YOUR USERNAME HERE|
| sasl_password    | string   | Required. YOUR PASSWORD HERE|
| schema_registry_basic_auth_user_info    | string   | Required. AUTH HERE |
| schema_registry_basic_auth_credentials_source    | string   | Required. USER_INFO.|
| sasl_mechanisms    | string   | Required. PLAIN.|
| security_protocol    | string   | Required. SASL_SSL.|
| persistence_conf    | string   | Required. The full string for connecting to your PostgreSQL database.|
| group_id    | string   | Optional. The consumer group id to assign to the consumer(s) . The default value is `metrics_consumer`|
| consumer_topics    | string or list  | Optional. The Kafka Avro Topic where to consume messages from. The defaul value is `websites_metrics` |
| logging_level    | string   | Optional. The level of logging to use for the built-in `logging` package. The default is `logging.INFO`  |


