Metadata-Version: 2.4
Name: challengerteco
Version: 0.2.0
Summary: Clase para manejo de challenger en la nube
Author: Eduardo Pagnone
Author-email: mlopsaa@teco.com.ar
Requires-Python: >=3.7,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Description-Content-Type: text/markdown

# ChallengerTeco

`challengerteco` is a Python library designed to streamline the process of training, evaluating, and comparing machine learning models in a distributed environment. It provides a flexible and configurable pipeline for data scientists and ML engineers to run model "challenger" competitions.

## Description

This project encapsulates the end-to-end machine learning workflow, from data preparation to model evaluation. It is built to work with large datasets and can be executed on different platforms like Google Cloud Platform (GCP) or on-premise clusters.

The main goal of this library is to provide a standardized way to test different models and hyperparameters, making it easier to select the best-performing model for a given task.

## Features

- **Multi-platform support**: Runs on Google Cloud Platform and on-premise environments.
- **Multiple ML Models**: Supports popular classification algorithms like:
    - Random Forest
    - Gradient Boosting
    - Logistic Regression
    - LightGBM
    - XGBoost
- **Data Preprocessing**: Includes steps for:
    - Target balancing (undersampling)
    - Feature scaling
    - Correlation analysis and removal
    - Data type casting
- **Hyperparameter Tuning**: Uses `RandomizedSearchCV` and `GridSearchCV` to find the best model parameters.
- **Model Evaluation**: Evaluates models using AUC and other metrics.
- **Persistence**: Saves trained models and evaluation results.

## Classes

### `query_manager`

This class is a wrapper for database interactions. It abstracts the boilerplate code needed to connect to and query different data sources.

- **`execute_query(query)`**: Executes a SQL query.
- **`to_table(df, mode, nombre)`**: Saves a DataFrame to a table.
- **`from_query(query)`**: Creates a DataFrame from a SQL query.

### `Challenger`

The main class of the library. It orchestrates the entire ML pipeline. It takes a large number of configuration parameters to control every aspect of the process.

The main method is `EjecutarChallenger()`, which runs the complete challenger process.

## Installation

```bash
pip install challengerteco
```

## Usage

The intended use is to create a JSON configuration file and instantiate the `Challenger` class from it.

**`config.json`**
```json
{
    "BALANCEAR_TARGET": true,
    "ELIMINAR_CORRELACIONES": true,
    "CASTEAR_BIGINT": true,
    "REDONDEAR_DECIMALES": true,
    "CON_SCALER": true,
    "TIENE_TESTING": true,
    "CORRER_RF": true,
    "CORRER_GB": true,
    "CORRER_LR": true,
    "CORRER_LGBM": true,
    "CORRER_XGB": true,
    "CORRER_PRODUCTIVO": false,
    "CAMPO_CLAVE": "client_id",
    "TARGET": "purchase",
    "modelo": "propensity_to_buy",
    "ABT_VARIABLES": "var1, var2, var3",
    "ABT_TABLA": "my_project.my_dataset.abt_table",
    "TGT_TABLA": "my_project.my_dataset.target_table",
    "TGT_VARIABLES": "target_variable",
    "TGT_BALENCEO": 20,
    "DECIMALES_VARIABLES_NUMERICAS": 4,
    "COTA_CORRELACIONES": 0.9,
    "REGISTROS_X_PARTICION": 100000,
    "PORCENTAJE_TRAINING": 0.8,
    "GB_param_test": {},
    "LGBM_param_test": {},
    "XGB_param_test": {},
    "RF_param_test": {},
    "LR_param_test": {},
    "PERIODO": "202301",
    "PERIODO_TRAIN1": "202207",
    "PERIODO_TRAIN2": "202208",
    "PERIODO_TRAIN3": "202209",
    "PERIODO_TRAIN4": "202210",
    "PERIODO_TRAIN5": "202211",
    "PERIODO_TRAIN6": "202212",
    "PERIODO_TEST1": "202301",
    "PERIODO_TEST2": "202302",
    "PERIODO_TEST3": "202303",
    "MODELO_PRODUCTIVO": "RF",
    "MODELO_PRODUCTIVO_param_test": {},
    "GRABAR_BINARIOS": true,
    "Tabla_Performance_Modelos": "my_project.my_dataset.performance_table"
}
```

**`main.py`**
```python
from challengerteco import Challenger
from pyspark.sql import SparkSession
# You will need to provide your own bq and spark objects
# bq = bigquery.Client()
# spark = SparkSession.builder.getOrCreate()

challenger = Challenger.instanciar_desde_json('config.json')
challenger.bq = bq
challenger.spark = spark
challenger.reset_query_manager()
challenger.EjecutarChallenger()

```

## TODOs

The following `TODO`s were found in the code and could be addressed in future versions:

- Clean up library imports.
- Add assertions to validate input parameters.
- Improve error handling and logging.
- Refactor duplicated code in `RandomForest`, `GradientBoosting`, and `LogisticRegression` methods.
- Review memory pressure in pandas-based methods.
- Improve output formatting for better readability.
- Remove hardcoded values and unused variables.
- Update deprecated commands (`hdfs dfs -copyFromLocal`).
- Add docstrings to methods.

This `README.md` provides a good overview of the project. It explains what the project does, its main features, how to install and use it, and what can be improved.

