Metadata-Version: 2.1
Name: pyspark-prometheus
Version: 0.1.2
Summary: 
Author: Zachary King
Author-email: zach@makewithdata.tech
Requires-Python: >=3.9,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Dist: prometheus-client (>=0.21.0)
Requires-Dist: pyspark (>=3.5.0)
Description-Content-Type: text/markdown

# Spark Streaming Metrics Prometheus Instrumentation

This project provides a seamless integration between PySpark and Prometheus for monitoring Spark **Structured Streaming** applications.

> *Note:* this project focuses on better metrics for Spark Structured Streaming specifically. If you would like other Spark metrics such as executor memory, CPU, GC times, etc. in Prometheus please refer to Spark's monitoring guide and its support for Prometheus using JMX (Java Management Extensions).

## Features

- Collects metrics from PySpark Streaming Queries
- Exposes metrics in Prometheus format
- Easy integration with existing PySpark applications

## Installation

To install the required dependencies, run:

```bash
pip install -r requirements.txt
```

## Usage

1. Import the necessary modules in your PySpark application:

    ```python
    from pyspark_prometheus import with_prometheus_metrics
    ```

2. Initialize the Prometheus metrics:

    ```python
    spark = SparkSession.builder.master("local").appName("MySparkApp").getOrCreate()
    spark = with_prometheus_metrics(spark, 'http://localhost:9091')
    ```

3. Start your PySpark job as usual. Metrics will be collected and exposed automatically.

## Contributing

Contributions are welcome! Please submit a pull request or open an issue to discuss your ideas.

## License

This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.

## Contact

For any questions or support, please open an issue in the repository.
****
