Metadata-Version: 2.1
Name: rearden
Version: 0.0.2
Summary: Library of useful data structures for data preprocessing, vizualizations and grid search
Author-email: Sergey Polivin <s.polivin@gmail.com>
License: MIT License
        
        Copyright (c) 2023 Sergey Polivin
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/spolivin/rearden
Project-URL: Bug Tracker, https://github.com/spolivin/rearden/issues
Project-URL: Pull requests, https://github.com/spolivin/rearden/pulls
Project-URL: Source, https://github.com/spolivin/rearden/tree/master/src/rearden
Classifier: Development Status :: 1 - Planning
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: matplotlib (>=3.3.4)
Requires-Dist: pandas (>=1.2.4)
Requires-Dist: numpy (>=1.24.3)
Requires-Dist: scikit-learn (>=1.1.3)
Requires-Dist: seaborn (>=0.11.1)
Requires-Dist: statsmodels (>=0.13.2)
Provides-Extra: formatters
Requires-Dist: autoflake (>=2.1.1) ; extra == 'formatters'
Requires-Dist: isort (>=5.12.0) ; extra == 'formatters'
Requires-Dist: black (>=23.3.0) ; extra == 'formatters'
Provides-Extra: linters
Requires-Dist: darglint (>=1.8.1) ; extra == 'linters'
Requires-Dist: flake8 (>=5.0.0) ; extra == 'linters'
Requires-Dist: flake8-bandit (>=4.1.1) ; extra == 'linters'
Requires-Dist: flake8-isort (>=6.0.0) ; extra == 'linters'
Requires-Dist: flake8-black (>=0.3.6) ; extra == 'linters'
Requires-Dist: flake8-pyproject (>=1.2.3) ; extra == 'linters'
Provides-Extra: precommit
Requires-Dist: pre-commit (>=3.3.2) ; extra == 'precommit'

# Rearden

**Rearden** is a Python package that provides a faster and more convenient way of carrying out data science and running machine learning algorithms. Making use of the functionality of the most popular libraries for data analysis (`pandas`, `numpy`, `statsmodels`), data vizualization (`matplotlib`, `seaborn`) and grid search (`scikit-learn`), it enables reaching the conclusions about the data in a quicker and clearer manner.

----

## Modules and API

The package is designed to aid data scientists in quickly getting insights about the data during the following stages of data analysis/machine learning:

* Data preprocessing
* Data vizualization
* Time-series analysis
* Grid search

Hence, the data structures which make up the **Rearden** package have been logically divided into Python modules based off of the above respective parts:

* `preprocessings.py`
* `vizualizations.py`
* `time_series.py`
* `grid_search.py`

### Data preprocessing

Functions included in `preprocessings.py` are basically programmed to help with missing and anomalous values, duplicates and data preparation for machine learning algorithms (e.g. data split into sets). For instance, currently the following functions are included in the module:

| Name | Kind | Description |
| :---------------------- | :---------------------- | :---------------------- |
| `identify_missing_values` | *function* | Display of the number and share of missing values |
| `preprocess_duplicates` | *function* | Deletion of duplicated rows with a message |
| `filter_data` | *function* | Filters data according to the predetermined ranges |
| `prepare_sets`| *function* | Data split into sets depending on target name and sets proportions |

The module and the associated functions can be called like so:

```python
from rearden.preprocessings import prepare_sets
```

### Data vizualization

Enhanced data vizualizations tools are located in `vizualizations.py` module. The functions here are as follows:

| Name | Kind | Description |
| :---------------------- | :---------------------- | :---------------------- |
| `plot_model_comparison` | *function* | Vizualization of ML models performances based on their names and scores |
| `plot_corr_heatmap` | *function* | Plotting correlation matrix heatmap in one go|
| `plot_class_structure`| *function* | Plotting the shares of different classes for a target vector in classification problems |

#### Models performance comparison

Using `plot_model_comparison` function, it is very easy to conveniently showcase how models perform according to some metric. One would just run:

```python
import seaborn as sns

from rearden.vizualizations import plot_model_comparison

sns.set_theme()

models_performance = [
    ("Decision Tree", 30.8343),
    ("Random Forest", 29.3127),
    ("Catboost", 26.4651),
    ("Xgboost", 26.7804),
    ("LightGBM", 26.6084),
]

plot_model_comparison(
    results=models_performance,
    metric_name="RMSE",
    title_name="Grid search results",
)
```

The result is the following figure:

![Models performance comparison](https://github.com/spolivin/rearden/raw/test/images/model_comparison.png)

#### Correlation matrix heatmap

It is possible to quickly plot the heatmap of the correlation matrix for the data using `plot_corr_heatmap` function. Here is how we would do that:

```python
import pandas as pd
import seaborn as sns

from rearden.vizualizations import plot_corr_heatmap

sns.set_theme()

test_data = pd.read_csv("datasets/test_data.csv")

plot_corr_heatmap(
    data=test_data,
    heatmap_coloring="Oranges",
    annotation=True,
    lower_triangle=True,
)
```

The code above results in the following plot:

![Correlation matrix heatmap](https://github.com/spolivin/rearden/raw/test/images/corr_mat_heatmap.png)

### Time-series analysis

Tools for time-series analysis from `time_series.py` are pretty straightforward:

| Name | Kind | Description |
| :---------------------- | :---------------------- | :---------------------- |
| `FeaturesExtractor` | *class* | Extraction of time variables from a one-dimensional time-series depending on lag and rolling mean order values |
| `prepare_ts` | *function* | Data split of a time-series data into sets depending on target name and sets proportions |
| `plot_time_series` | *function* | Plotting the original time-series or a decomposed one |

One can, for example, want to firstly generate the data by `FeaturesExtractor`, then look at the graph via `plot_time_series` and then divide the data into sets with `prepare_ts`.

#### Time-series plot and its decomposition

`plot_time_series` function provides two additional ways we could plot a time-series:

* Plain time-series
* Decomposed time-series (trend, seasonality, residual)

Take, for example:

```python
import pandas as pd
import seaborn as sns

from rearden.time_series import plot_time_series

sns.set_theme()

ts_data_test = pd.read_csv("datasets/ts_data_test.csv", parse_dates=[0], index_col=[0])
ts_data_test_resampled = ts_data_test.resample("1H").sum()

plot_time_series(
    data=ts_data_test_resampled,
    col="num_orders",
    period_start="2018-03-01",
    period_end="2018-03-03",
    ylabel_name="Number of orders",
    title_name="Time-series plot",
)
```

In this case we plot the evolution of the number of orders against time. We obtain the following plot:

![Time-series plot](https://github.com/spolivin/rearden/raw/test/images/ts_plot.png)

We could also decompose this time series by just adding `kind="decomposed"` to the above function:

```python
plot_time_series(
    data=ts_data_test_resampled,
    col="num_orders",
    kind="decomposed",
    period_start="2018-03-01",
    period_end="2018-03-03",
    ylabel_name="Number of orders",
)
```

The result is as follows:

![Decomposed time-series](https://github.com/spolivin/rearden/raw/test/images/ts_decomposed.png)

### Grid search

In `grid_search.py` module, `RandomizedSearchCV` base class from `sklearn.model_selection` was used, which has been wrapped with two additional classes with some additional methods, custom defaults and other functionality:

| Name | Kind | Description |
| :---------------------- | :---------------------- | :---------------------- |
| `RandomizedHyperoptRegression` | *class* | Wrapper for `RandomizedSearchCV` with functionality to quickly compute regression metrics and conveniently display tuning process |
| `RandomizedHyperoptClassification` | *class* | Wrapper for `RandomizedSearchCV` with functionality to quickly compute classification metrics, conveniently display tuning process and fastly plot confusion matrix |

#### Confusion matrix

We can use `RandomizedHyperoptClassification` wrapper for quickly making conclusions about the results of the grid search. For instance, let's imagine that we have managed to split the data into `features_train` and `features_test` as well as `target_train` and `target_test`. We can now run the grid search algorithms and immediately get the plot of the confusion matrix:

```python
from sklearn.tree import DecisionTreeClassifier

from rearden.grid_search import RandomizedHyperoptClassification

dtc_model = DecisionTreeClassifier(random_state=12345)
param_grid_dtc = {"max_depth": np.arange(1, 12)}

dtc_grid_search = RandomizedHyperoptClassification(
    estimator=dtc_model,
    param_distributions=param_grid_dtc,
    train_dataset=(features_train, target_train),
    eval_dataset=(features_test, target_test),
    random_state=12345,
    cv=5,
    n_iter=5,
    scoring="f1",
    n_jobs=None,
)
dtc_grid_search.train_crossvalidate()

dtc_grid_search.plot_confusion_matrix(label_names=("label_1", "label_2"))
```

Thanks to the additional `eval_dataset` attribute, the resulting plot is already a confusion matrix for the best model after cross-validation which has been used for making predictions on the test data:

![Confusion matrix](https://github.com/spolivin/rearden/raw/test/images/confusion_matrix.png)

## Installation

### Package dependencies

**Rearden** library requires the following dependencies:

| Package | Version |
| :---------------------- | :---------------------- |
| Matplotlib | >= 3.3.4|
| Pandas | >= 1.2.4|
| NumPy| >= 1.24.3|
| Scikit-learn| >= 1.1.3|
| Seaborn| >= 0.11.1|
| Statsmodels| >= 0.13.2|

> **_NOTE:_**  The package currently requires Python 3.7 or higher.

### Installation using `pip`

The package is available on [PyPI Index](https://pypi.org/project/rearden/) and can be easily installed using `pip`:

```
pip install rearden
```

The dependencies are automatically downloaded when executing the above command or can be installed manually using (after cloning the repo):

```
pip install -r requirements.txt
```

## Building the package

Thanks to the build system requirements and other metadata specified in `pyproject.toml` it is easy to build and install the package. Firstly, clone the repository:

```
git clone https://github.com/spolivin/rearden.git

cd rearden
```

Then, one can simply run the following:

```
pip install -e .
```

## Automatic code style checks

### Installation of `pre-commit`
Before pushing the changed code to the remote Github repository, the code undergoes numerous checks conducted with the help of *pre-commit hooks* specified in `.pre-commit-config.yaml`. Before making use of this feature, it is important to first download `pre-commit` package to the system:

```
pip install pre-commit
```

or if `rearden` package has already been installed:

```
pip install rearden[precommit]
```

Afterwards, in the git-repository run the following command for installation:

```
pre-commit install
```

Now, the *pre-commit hooks* can be easily used for verifying the code style.

### Pre-commit hooks

After running `git commit -m "<Commit message>"` in the terminal, the file to be committed goes through a few checks before being enabled to be committed. As specified in `.pre-commit-config.yaml`, the following hooks are used:

| Hooks | Version |
| :---------------------- | :---------------------- |
| Pre-commit-hooks | 4.3.0 |
| Autoflake | 2.1.1 |
| Isort | 5.12.0 |
| Black | 23.3.0 |
| Flake8 | 5.0.0|

> **_NOTE:_** Check `.pre-commit-config.yaml` for more information about the repos and hooks used.

It is also possible to download the required dependencies for *pre-commit hooks*:

```
pip install -r requirements-dev.txt
```

or:

```
pip install rearden[formatters]

pip install rearden[linters]
```
