Metadata-Version: 2.1
Name: pandas-cleaner
Version: 0.0.1
Summary: A pandas extension for cleaning datasets.
Home-page: https://github.com/eurodecision/pandas-cleaner
Author: Renan Hilbert
Author-email: renan.hilbert@eurodecision.com
License: UNKNOWN
Project-URL: Documentation, https://github.com/eurodecision/pandas-cleaner/blob/master/README.rst
Project-URL: Source Code, https://github.com/eurodecision/pandas-cleaner
Project-URL: Bug Tracker, https://github.com/eurodecision/pandas-cleaner/issues
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: License :: OSI Approved :: BSD License
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python
Classifier: Topic :: Software Development
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.6
Description-Content-Type: text/x-rst
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: matplotlib
Requires-Dist: seaborn
Requires-Dist: scikit-learn
Requires-Dist: requests


.. image:: https://github.com/eurodecision/pandas-cleaner/blob/master/docs/source/pandas-cleaner.png

------

pandas-cleaner
==============


What is it ?
------------

Pandas-cleaner is a Python package, built on top of pandas, that provides methods detect, analyze and clean errors in datasets with different types of data (numerical, categorical, text, datetimes...).


Features
--------
Pandas-cleaner offers functionnalities to automatically :

+ **detect** different kind of potential errors in datasets such as outliers, inconsistencies, typos, wrong-typed ..., given predefined rules or statistiscal estimations,  via an easy-to-use API extending pandas,

+ **analyze** these errors, via reports and plots, to check the validity of the set and/or decide if any correction is needed,

+ **clean** the datasets, either by dropping the lines with errors, emptying, correcting or replacing bad values,

+ **reapply** the same rules to any other incoming fresh data.


Usage
-----

Import the package

.. code-block:: python

   import pandas as pd
   import pdcleaner

Create an example data series

.. code-block:: python

   series = pd.Series([1, 5, -6, 100, 10])

Detect the errors in the series with a given method (such as `bounded`, `iqr`, `zscore` and many more depending the type of data...)

.. code-block:: python

   detector = series.cleaner.detect('bounded', lower=0, upper=10)

Inspect the result:

.. code-block:: none

   detector.report()

.. code-block:: none

                                    Detection report                               
   ==============================================================================
   Method:                      bounded      Nb samples:                        5
   Date:                January 24,2022      Nb errors:                         2
   Time:                       16:06:08      Nb rows with NaN:                  0
   ------------------------------------------------------------------------------
   lower                              0      upper                             10
   inclusive                       both      sided                           both
   ==============================================================================

Check the potential errors that have been detected

.. code-block:: python

   detector.detected

.. parsed-literal::

    2     -6
    3    100
    dtype: int64

Clean the detected errors from the series using the chosen method among `drop`, `to_na`, `clip`
, `replace`...

.. code-block:: python

   series.cleaner.clean("drop", detector, inplace=True)
   series

.. parsed-literal::

    0      1
    1      5
    4     10
    dtype: int64

Documentation
-------------

The documentation is still a **work in progress**. 

* Clone the project

* Build the documentation using :

.. code-block:: bash

    cd docs
    make html

* Open `docs/build/html/index.html` in your browser

Contributing to pandas-cleaner
------------------------------

All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.

Issues and bugs can be reported at https://github.com/eurodecision/pandas-cleaner/issues


