Metadata-Version: 2.1
Name: hbac-bias-detection
Version: 0.2.5
Summary: UNKNOWN
Home-page: UNKNOWN
License: UNKNOWN
Platform: UNKNOWN
Description-Content-Type: text/markdown
Requires-Dist: anyio (==3.5.0)
Requires-Dist: appnope (==0.1.3)
Requires-Dist: argon2-cffi (==21.3.0)
Requires-Dist: argon2-cffi-bindings (==21.2.0)
Requires-Dist: asttokens (==2.0.5)
Requires-Dist: attrs (==21.4.0)
Requires-Dist: Babel (==2.9.1)
Requires-Dist: backcall (==0.2.0)
Requires-Dist: beautifulsoup4 (==4.11.1)
Requires-Dist: bleach (==5.0.0)
Requires-Dist: certifi (==2021.10.8)
Requires-Dist: cffi (==1.15.0)
Requires-Dist: charset-normalizer (==2.0.12)
Requires-Dist: cloudpickle (==2.0.0)
Requires-Dist: cycler (==0.11.0)
Requires-Dist: debugpy (==1.6.0)
Requires-Dist: decorator (==5.1.1)
Requires-Dist: defusedxml (==0.7.1)
Requires-Dist: docutils (==0.18.1)
Requires-Dist: entrypoints (==0.4)
Requires-Dist: executing (==0.8.3)
Requires-Dist: fastjsonschema (==2.15.3)
Requires-Dist: fonttools (==4.32.0)
Requires-Dist: idna (==3.3)
Requires-Dist: ipykernel (==6.13.0)
Requires-Dist: ipython (==8.2.0)
Requires-Dist: ipython-genutils (==0.2.0)
Requires-Dist: jedi (==0.18.1)
Requires-Dist: Jinja2 (==3.1.1)
Requires-Dist: joblib (==1.1.0)
Requires-Dist: json5 (==0.9.6)
Requires-Dist: jsonschema (==4.4.0)
Requires-Dist: jupyter-client (==7.2.2)
Requires-Dist: jupyter-core (==4.9.2)
Requires-Dist: jupyter-server (==1.16.0)
Requires-Dist: jupyterlab (==3.3.3)
Requires-Dist: jupyterlab-pygments (==0.2.1)
Requires-Dist: jupyterlab-server (==2.12.0)
Requires-Dist: kiwisolver (==1.4.2)
Requires-Dist: llvmlite (==0.38.0)
Requires-Dist: MarkupSafe (==2.1.1)
Requires-Dist: matplotlib (==3.5.1)
Requires-Dist: matplotlib-inline (==0.1.3)
Requires-Dist: mistune (==0.8.4)
Requires-Dist: nbclassic (==0.3.7)
Requires-Dist: nbclient (==0.6.0)
Requires-Dist: nbconvert (==6.5.0)
Requires-Dist: nbformat (==5.3.0)
Requires-Dist: nest-asyncio (==1.5.5)
Requires-Dist: notebook (==6.4.10)
Requires-Dist: notebook-shim (==0.1.0)
Requires-Dist: numba (==0.55.1)
Requires-Dist: numpy (==1.21.6)
Requires-Dist: packaging (==21.3)
Requires-Dist: pandas (==1.4.2)
Requires-Dist: pandocfilters (==1.5.0)
Requires-Dist: parso (==0.8.3)
Requires-Dist: pexpect (==4.8.0)
Requires-Dist: pickleshare (==0.7.5)
Requires-Dist: Pillow (==9.1.0)
Requires-Dist: prometheus-client (==0.14.1)
Requires-Dist: prompt-toolkit (==3.0.29)
Requires-Dist: psutil (==5.9.0)
Requires-Dist: ptyprocess (==0.7.0)
Requires-Dist: pure-eval (==0.2.2)
Requires-Dist: pycparser (==2.21)
Requires-Dist: Pygments (==2.11.2)
Requires-Dist: pyparsing (==3.0.8)
Requires-Dist: pyrsistent (==0.18.1)
Requires-Dist: python-dateutil (==2.8.2)
Requires-Dist: pytz (==2022.1)
Requires-Dist: pyzmq (==22.3.0)
Requires-Dist: requests (==2.27.1)
Requires-Dist: scikit-learn (==1.0.2)
Requires-Dist: scipy (==1.8.0)
Requires-Dist: seaborn (==0.11.2)
Requires-Dist: Send2Trash (==1.8.0)
Requires-Dist: shap (==0.40.0)
Requires-Dist: six (==1.16.0)
Requires-Dist: sklearn (==0.0)
Requires-Dist: slicer (==0.0.7)
Requires-Dist: sniffio (==1.2.0)
Requires-Dist: soupsieve (==2.3.2)
Requires-Dist: stack-data (==0.2.0)
Requires-Dist: statistics (==1.0.3.5)
Requires-Dist: terminado (==0.13.3)
Requires-Dist: threadpoolctl (==3.1.0)
Requires-Dist: tinycss2 (==1.1.1)
Requires-Dist: tornado (==6.1)
Requires-Dist: tqdm (==4.64.0)
Requires-Dist: traitlets (==5.1.1)
Requires-Dist: urllib3 (==1.26.9)
Requires-Dist: wcwidth (==0.2.5)
Requires-Dist: webencodings (==0.5.1)
Requires-Dist: websocket-client (==1.3.2)
Requires-Dist: xgboost (==0.80)

# Using HBAC to detect biased data segments
- Hierarchical Bias-Aware Clustering (HBAC) on regression models.
- Input: a trained model and a model's test data.
- Output: analysis of biased/discriminated data segment according to HBAC:
  - Comparing distributions of discriminated and remaining data.
  - Segment predictor: trains a XGBoost binary classifier to evaluate distinguishability of discriminated and remaining data with descriptive features. 
  
![github_workflow drawio](https://user-images.githubusercontent.com/59140255/163182069-22a51b22-2b3b-41c0-93c9-3ec60b48ce9c.png)


```python
# Initialize HBAC 
hbac = HBAC_analyser()

# In this case, input includes model path, X data and Y data
hbac.hbac_on_model(model_path, X_test, y_test) 

hbac.pca_plot()
discrimated_cluster, bias =  hbac.get_max_bias_cluster(print_results=True)

# Get discriminated data in panda df
hbac.all_unscaled_discriminated

# Displaying results in dataframes
hbac.clustered_data

# Mean per feature 'discrimnated' cluster vs 'remaining' clusters
hbac.mean_clusters

# Plot 3 most different features' distributions
hbac.plot_distributions(plot_top_features = 3)

# Train XGBoost a binary classifier to predict whther a datapoint will be discrimnated or not, without using error as feature.
hbac.segment_predictor(plot_roc_auc=True,shap_analysis=True)
```

Also see **example.ipynb**.

For the use of HBAC on classification models, see https://github.com/Sm2468/msc_thesis/tree/master/hbac%20scripts, on which this project was based.


