Metadata-Version: 2.4
Name: dt_feature_importance
Version: 0.1.0
Summary: High-performance feature importance, IV, PSI, and monotonicity tools for decision tree models like CHAID
Author-email: Vrukshya <vrukshyaai@gmail.com>
Maintainer-email: Vrukshya <vrukshyaai@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/vrukshya/dt_feature_importance
Project-URL: Documentation, https://github.com/vrukshya/dt_feature_importance#readme
Project-URL: Repository, https://github.com/vrukshya/dt_feature_importance
Project-URL: Bug Tracker, https://github.com/vrukshya/dt_feature_importance/issues
Keywords: decision-tree,chaid,cart,feature-importance,feature-ranking,information-value,iv,psi,population-stability-index,monotonicity,credit-risk,scorecard,binning,pandas,numpy,fintech,machine-learning
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Office/Business :: Financial
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: numpy>=1.20.0
Requires-Dist: pandas>=1.3.0
Provides-Extra: optbinning
Requires-Dist: optbinning>=0.17.0; extra == "optbinning"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: flake8>=6.0.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# dt_feature_importance

[![PyPI version](https://img.shields.io/pypi/v/dt_feature_importance.svg)](https://pypi.org/project/dt-feature-importance/)
[![Python Version](https://img.shields.io/badge/python-3.8%20%7C%203.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue.svg)](https://pypi.org/project/dt-feature-importance/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

**High-performance feature importance, Information Value (IV), Population Stability Index (PSI), and monotonicity tools for decision tree models like CHAID and credit risk scorecards.**

---

## Overview

When developing decision tree models (such as **CHAID**, **CART**, and shallow gradient boosted trees) or regulatory credit scoring scorecards (Basel II/III, IFRS 9), practitioners need fast, reliable diagnostic tools to:
- Screen and rank candidate features prior to tree splitting.
- Measure predictive strength using **Information Value (IV)** / Chi-Square.
- Track distribution drift across development and out-of-time samples using **Population Stability Index (PSI)**.
- Enforce business and regulatory logic by ensuring **monotonic event rate trends** across bin intervals.

`dt_feature_importance` delivers high-performance NumPy/Pandas implementations of these essential utilities with **zero heavy machine-learning dependencies**.

---

## Key Features

- ⚡ **Zero-Dependency Single-Split Tree Importance (`lgbm_top_features`)**: Pure NumPy gradient/hessian split-gain ranking. Evaluates numeric and categorical splits, missing value directions, and min-sample/min-event constraints without installing or compiling LightGBM.
- 📊 **Vectorized Information Value (`iv_from_ids`)**: Ultra-fast IV computation from pre-binned indices using `np.bincount` and Laplace smoothing.
- 📈 **Population Stability Index (`psi_from_ids`)**: Quantify population distribution shift between reference/baseline and validation datasets.
- 🔄 **Monotonicity Trend Detection (`monotonic_direction`, `monotonic_direction_from_ids`)**: Identifies whether target event rates across bins are `'increasing'`, `'decreasing'`, or `'non_monotonic'`, supporting both raw arrays and `optbinning` objects.
- 🏷️ **Automated Feature Categorization (`infer_feature_types`)**: Automatically splits candidate columns into numeric and categorical features for downstream tree binning.

---

## Installation

Install the stable release from PyPI:

```bash
pip install dt_feature_importance
```

For optional integration with `optbinning` (to extract monotonicity directly from `OptimalBinning` objects):

```bash
pip install "dt_feature_importance[optbinning]"
```

---

## Quickstart Guide

### 1. Feature Importance for Decision Trees (`lgbm_top_features`)

Quickly rank hundreds of candidate variables by single-split gain to select optimal features for CHAID or decision tree nodes:

```python
import numpy as np
import pandas as pd
from dt_feature_importance import lgbm_top_features

# Sample dataset
df = pd.DataFrame({
    "bureau_score": [650, 720, 580, 790, 610, 800, 540, 690, 710, 600] * 100,
    "income": [45000, 80000, 32000, 120000, 40000, 110000, 28000, 75000, 85000, 39000] * 100,
    "employment_type": ["Salaried", "Self-Employed", "Salaried", "Salaried", "Other"] * 200,
    "default_flag": [1, 0, 1, 0, 1, 0, 1, 0, 0, 1] * 100
})

importance_df, top_features = lgbm_top_features(
    df=df,
    target="default_flag",
    features=["bureau_score", "income", "employment_type"],
    top_n=10,
    min_bin_size_node_g=20,  # Minimum samples per child leaf
    min_bin_n_event=5        # Minimum target events per child leaf
)

print(importance_df)
#           feature  importance
# 0    bureau_score   82.415024
# 1          income   64.309118
# 2 employment_type   12.184910

print("Selected top features:", top_features)
```

---

### 2. Information Value (IV) from Bin IDs (`iv_from_ids`)

Calculate Information Value directly from pre-binned categories or quantile bins:

```python
import numpy as np
from dt_feature_importance import iv_from_ids

# Pre-assigned bin IDs (e.g. from quantile cuts, CHAID leaf nodes, or optbinning)
bin_ids = np.array([0, 0, 0, 1, 1, 1, 2, 2, 2, 2])
y = np.array([1, 1, 0, 1, 0, 0, 0, 0, 0, 1])

iv = iv_from_ids(bin_ids=bin_ids, y=y)
print(f"Information Value: {iv:.4f}")
```

#### Rule of Thumb for IV Interpretation:
| Information Value (IV) | Predictive Power |
| :--- | :--- |
| `< 0.02` | Unpredictable / Not useful |
| `0.02 to 0.1` | Weak predictive power |
| `0.1 to 0.3` | Medium predictive power |
| `0.3 to 0.5` | Strong predictive power |
| `> 0.5` | Suspicious / Overfitting check needed |

---

### 3. Population Stability Index (`psi_from_ids`)

Detect sample drift between your development (baseline) population and recent/validation data:

```python
import numpy as np
from dt_feature_importance import psi_from_ids

# Bin allocations for development vs. validation populations
dev_bins = np.array([0, 0, 1, 1, 2, 2, 2, 3, 3, 3])
rec_bins = np.array([0, 1, 1, 1, 2, 2, 3, 3, 3, 3])

psi = psi_from_ids(dev_ids=dev_bins, rec_ids=rec_bins)
print(f"Population Stability Index: {psi:.4f}")
```

#### Rule of Thumb for PSI Interpretation:
| Population Stability Index (PSI) | Action / Stability |
| :--- | :--- |
| `< 0.10` | **Stable**: No significant shift in population distribution |
| `0.10 to 0.25` | **Moderate Drift**: Slight shift; investigate bins |
| `> 0.25` | **Significant Drift**: Major population change; recalibration required |

---

### 4. Monotonicity Direction Extraction

Validate that the relationship between bin intervals and event rates strictly ascends or descends:

#### From Raw Bin IDs:
```python
import numpy as np
from dt_feature_importance import monotonic_direction_from_ids

bin_ids = np.array([0, 0, 1, 1, 2, 2])
y = np.array([0, 0, 0, 1, 1, 1])

direction = monotonic_direction_from_ids(bin_ids=bin_ids, y=y)
print(direction)  # Output: 'increasing'
```

#### From an `OptimalBinning` Model:
```python
from optbinning import OptimalBinning
from dt_feature_importance import monotonic_direction

optb = OptimalBinning(name="bureau_score", dtype="numerical")
optb.fit(df["bureau_score"], df["default_flag"])

direction = monotonic_direction(optb)
print("Monotonicity trend:", direction)  # Output: 'decreasing'
```

---

### 5. Feature Type Categorization (`infer_feature_types`)

Separate numeric columns from categorical variables automatically:

```python
from dt_feature_importance import infer_feature_types

numeric_cols, categorical_cols = infer_feature_types(
    df=df,
    features=["bureau_score", "income", "employment_type"]
)

print("Numeric:", numeric_cols)          # ['bureau_score', 'income']
print("Categorical:", categorical_cols)  # ['employment_type']
```

---

## Decision Tree / CHAID Workflow Integration

Here is how `dt_feature_importance` fits into a standard decision tree or scorecard model lifecycle:

```
                  Raw Candidate Feature Space (100+ Features)
                                       │
                                       ▼
                       infer_feature_types(df, features)
                                       │
                                       ▼
                   lgbm_top_features(df, target, features)
                (Rank features by single-split gain quickly)
                                       │
                                       ▼
                     Top Candidate Variables Selected
                                       │
                                       ▼
                         Binning / Tree Discretization
                            (e.g., CHAID / Binning)
                                       │
                ┌──────────────────────┴──────────────────────┐
                ▼                                             ▼
     iv_from_ids(bin_ids, y)               monotonic_direction_from_ids(bin_ids, y)
 (Verify predictive strength)                   (Verify monotonic risk trend)
                │                                             │
                └──────────────────────┬──────────────────────┘
                                       ▼
                     psi_from_ids(dev_ids, rec_ids)
                   (Confirm out-of-time stability)
                                       │
                                       ▼
               Final Regulatory Decision Tree / Scorecard Nodes
```

---

## API Reference

### `lgbm_top_features(df, target, features, top_n=20, min_bin_size_node_g=None, min_bin_n_event=None, **kwargs)`
- **Parameters**:
  - `df` (*pd.DataFrame*): Input dataset.
  - `target` (*str*): Binary target column name (0/1).
  - `features` (*List[str]*): Candidate feature names to evaluate.
  - `top_n` (*int*, default `20`): Number of top features to return.
  - `min_bin_size_node_g` (*float | int*, optional): Minimum samples per child leaf. If float in `(0, 1)`, interpreted as proportion of samples.
  - `min_bin_n_event` (*int*, optional): Minimum target events (1s) required in each child leaf.
- **Returns**: `(pd.DataFrame, List[str])` - Sorted importance table (`['feature', 'importance']`) and list of top feature names.

### `iv_from_ids(bin_ids, y, eps=1e-9)`
- **Parameters**:
  - `bin_ids` (*np.ndarray*): 1D array of bin identifiers.
  - `y` (*np.ndarray*): Binary target values (0 or 1).
  - `eps` (*float*, default `1e-9`): Smoothing parameter.
- **Returns**: `float` - Information Value (IV), or `np.nan` if invalid.
- *Alias*: `chi_square_from_ids` is available as a backward-compatible alias.

### `psi_from_ids(dev_ids, rec_ids, eps=1e-8)`
- **Parameters**:
  - `dev_ids` (*np.ndarray*): Bin indices from development/reference population.
  - `rec_ids` (*np.ndarray*): Bin indices from validation/recent population.
  - `eps` (*float*, default `1e-8`): Constant to avoid division/log of zero.
- **Returns**: `float` - Population Stability Index.

### `monotonic_direction_from_ids(bin_ids, y, missing_id=-1, special_id=-2)`
- **Parameters**:
  - `bin_ids` (*np.ndarray*): Array of bin indices.
  - `y` (*np.ndarray*): Binary target array.
  - `missing_id` (*int*, default `-1`): Bin index for missing values (excluded from monotonicity check).
  - `special_id` (*int*, default `-2`): Bin index for special values (excluded from monotonicity check).
- **Returns**: `Optional[str]` - `'increasing'`, `'decreasing'`, `'non_monotonic'`, or `None`.

### `monotonic_direction(optb)`
- **Parameters**:
  - `optb` (*Any*): Fitted `OptimalBinning` instance with a `binning_table` attribute.
- **Returns**: `Optional[str]` - `'increasing'`, `'decreasing'`, or `None`.

### `infer_feature_types(df, features)`
- **Parameters**:
  - `df` (*pd.DataFrame*): Input DataFrame.
  - `features` (*List[str]*): List of column names.
- **Returns**: `Tuple[List[str], List[str]]` - `(numeric_features, categorical_features)`.

---

## Contributing & Development

To contribute or run tests locally:

1. Clone the repository:
   ```bash
   git clone https://github.com/vrukshya/dt_feature_importance.git
   cd dt_feature_importance
   ```

2. Install in editable mode with development dependencies:
   ```bash
   pip install -e ".[dev]"
   ```

3. Run the test suite:
   ```bash
   pytest tests/
   ```

---

## License

This project is licensed under the MIT License - see the [LICENSE.txt](LICENSE.txt) file for details.
