Metadata-Version: 2.4
Name: drishtipy
Version: 0.6.1
Summary: drishtipy -Lightweight pandas data profiling, data quality, ML readiness, ETL analysis, PII detection, correlation analysis, comparison, and HTML reporting.
Author: M Jain
License: MIT
Project-URL: Homepage, https://github.com/drishtipy/drishtipy
Project-URL: Repository, https://github.com/drishtipy/drishtipy
Project-URL: Issues, https://github.com/drishtipy/drishtipy/issues
Keywords: pandas,data-profiling,data-quality,eda,etl,machine-learning,pii,data-analysis,drishtipy,drishti,pandas-profiling,pii-detection,data-cleaning,pandas-accessor
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Intended Audience :: Developers
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pandas>=1.3
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

### drishtipy

### Lightweight Data Profiling & Data Quality for pandas (Fast Pandas Profiling Alternative)

**Understand your DataFrame in seconds.**

drishtipy is a lightweight, dependency-minimal pandas profiling toolkit for data quality, statistics, ML readiness, ETL analysis, PII detection, correlations, dataset comparison, and HTML reporting. It extends the native pandas workflow to catch schema errors, trace data quality metrics, and automate privacy checks in a single line of code.

The goal is simple:

> **Load your DataFrame → profile it → find problems → understand the data → improve it.**

[![PyPI](https://img.shields.io/pypi/v/drishtipy.svg)](https://pypi.org/project/drishtipy/)
[![Python](https://img.shields.io/pypi/pyversions/drishtipy.svg)](https://pypi.org/project/drishtipy/)
[![Downloads](https://img.shields.io/pypi/dm/drishtipy.svg)](https://pypi.org/project/drishtipy/)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Tests](https://img.shields.io/badge/tests-pytest-green.svg)](#development)
[![GitHub](https://img.shields.io/badge/GitHub-drishtipy-black.svg)](https://github.com/drishtipy/drishtipy)

---

## ⚡ See It in Action

```python
import pandas as pd
import drishtipy
or
from drishtipy import profile_of  # ✅ full autocomplete + type checking

df = pd.read_csv("data.csv")

# Complete profile
df.profile.info() or profile_of(df).info()

# Data structure
df.profile.schema()

# Statistical analysis
df.profile.statistics()

# Data quality analysis
df.profile.quality()

# ML readiness & suggestions
df.profile.ml()

# ETL readiness & cleaning issues
df.profile.etl()

# Overall data quality score
df.profile.quality_score()

# Data quality alerts
df.profile.alerts()

# Potential PII detection
df.profile.pii()

# Correlation analysis
df.profile.correlations(by="pairs")

# Generate a shareable HTML report
df.profile.html("report.html")

# Compares two DataFrames and generates an HTML before/after report
df.profile.compare_html(df2, path="compare.html")

✨ **NEW:** This feature has been added recently.
# Auto-discover relationships across every column pair
df.profile.relationships()
or 
df.profile.relationships().to_html("relationships.html")  
or
df.profile.relationships().to_html("relationships.html", style="table")

✨ **NEW:** This feature has been added recently.
#utoML: benchmark regression models, no sklearn import needed
result = df.ml.auto_predict(target="Sales")
✨ **NEW:** This feature has been added recently.
# Lazy, composable ETL pipeline
clean = (       
    drishtipy.ETL()
    .extract("data.csv")
    .text("name").strip().title()
    .fill_missing("age", strategy="median")
    .load("clean.csv", report="etl_report.html")
```

That's it.
No complex configuration. No wrapper object required.

Just import `drishtipy` and use:

```python
df.profile
```

---

## 🎯 What Can drishtipy Tell You?

Give `drishtipy` a pandas DataFrame and quickly answer questions like:

- What columns does my dataset contain?
- Which columns have missing values?
- Where are the duplicates?
- Which numeric columns contain outliers?
- How good is the overall data quality?
- Which columns may contain PII?
- Is this dataset ready for machine learning?
- What problems could affect an ETL pipeline?
- Which numeric features are strongly correlated?
- What changed after cleaning my data?
- Can I generate a report to share with someone else?

---

## 🚀 One DataFrame. Multiple Insights.

```text
                         pandas DataFrame
                                │
                                ▼
                         ┌─────────────┐
                         │  drishtipy  │
                         └─────────────┘
                                │
          ┌─────────────┬───────┼────────┬──────────────┐
          ▼             ▼       ▼        ▼              ▼
       Schema       Statistics Quality   ML             ETL
          │             │       │        │              │
          │             │       ▼        ▼              ▼
          │             │   Quality    Readiness     Cleaning
          │             │    Score                   Issues
          │             │       │
          │             ▼       ▼
          │        Correlations Alerts
          │
          ├──────────────► PII Detection
          │
          ├──────────────► Before / After Comparison
          │
          └──────────────► HTML Report
```

---

## ⭐ Core Features

| Feature                          | Purpose                                                                                        |
| -------------------------------- | ---------------------------------------------------------------------------------------------- |
| **Schema**                 | Understand columns, dtypes, missing values, uniqueness and memory                              |
| **Statistics**             | Descriptive statistics, quantiles, skewness, kurtosis and more                                 |
| **Quality**                | Missing values, duplicates, zeros and outliers                                                 |
| **Quality Score**          | Overall and per-column data quality scoring                                                    |
| **Alerts**                 | High, medium and low severity data-quality issues                                              |
| **PII Detection**          | Detect potentially sensitive and identifiable data                                             |
| **Correlations**           | Find relationships and multicollinearity candidates                                            |
| **ML Readiness**           | Analyze feature suitability for machine learning                                               |
| **ETL Readiness**          | Identify common cleaning and transformation problems                                           |
| **Comparison**             | Compare DataFrames before and after transformations                                            |
| **HTML Reports**           | Generate standalone, shareable reports                                                         |
| **Large CSV**              | Profile large CSV files using chunking and sampling                                            |
| **Relationship Discovery** | Auto-detect, score, and rank relationships between every pair of columns                       |
| **AutoML Regression**      | Benchmark multiple regression models automatically, no scikit-learn import required            |
| **ETL Pipeline**           | Lazy, composable, pandas-native pipeline — extract, clean, transform, validate, quality-score |

---

## 📦 Installation

```bash
pip install drishtipy
```

Then:

```python
import pandas as pd
import drishtipy

df = pd.read_csv("data.csv")

df.profile.quality()
```

### Development installation

```bash
pip install -e .
```

For development dependencies:

```bash
pip install -e ".[dev]"
```

---

## 🐼 Designed for pandas

`drishtipy` extends the familiar pandas workflow with a `.profile` accessor.

Instead of:

```python
DataProfiler(df).info_dataframe(
    section="quality"
)
```

you can simply write:

```python
df.profile.quality()
```

The profiling results remain pandas-friendly, so you can continue to use normal pandas operations:

```python
report = df.profile.quality()

report.sort_values(
    "Missing Count",
    ascending=False
)
```

Filter them:

```python
report[
    report["Missing Count"] > 0
]
```

Or export them:

```python
report.to_csv(
    "quality_report.csv",
    index=False
)
```

---

## 📋 Schema Profiling

Inspect the structure of your DataFrame:

```python
df.profile.schema()
```

The schema report includes one row per column with:

- Column name and pandas dtype
- Non-null count, missing count, missing %
- Unique count, unique %
- Memory usage (bytes)

---

## 📊 Statistical Profiling

Generate descriptive statistics:

```python
df.profile.statistics()
```

One row per column, with (for numeric columns) count, mean, median, std,
min/Q1/Q2/Q3/95th/99th percentile/max, range, IQR, skewness, and kurtosis —
plus mode and mode frequency for every column regardless of dtype.

---

## 🔍 Data Quality Profiling

Analyze common data-quality problems:

```python
df.profile.quality()
```

The quality report includes:

- Duplicate value count (per column)
- Missing value count
- Zero count (numeric columns)
- IQR-based outlier count and outlier % (numeric columns)

Example:

```text
Column    Duplicate Count    Missing Count    Zero Count    Outlier Count    Outlier %
------------------------------------------------------------------------------------
age                     0                1             0                0        0.00
salary                  0                0             0                1       25.00
city                    1                0          None             None        None
```

---

## ⭐ Quality Score

Get a quick, weighted assessment of your dataset across five dimensions —
Missing Values, Duplicates, Outliers, Data Types, and Invalid Values:

```python
df.profile.quality_score()
```

Example output (a `Metric` / `Score` DataFrame):

```text
                 Metric  Score
 Overall Quality Score   88.4
         Missing Values  92.0
             Duplicates 100.0
               Outliers  71.0
             Data Types  95.0
         Invalid Values  84.0
```

For a column-level breakdown (one row per column, plus a per-column
`Quality Score`):

```python
df.profile.quality_score(by="column")
```

Filter by column type:

```python
df.profile.quality_score(column_type="numeric")
```

Custom scoring weights (unspecified dimensions keep their default weight of
20; values are normalized automatically, so they don't need to sum to 100):

```python
df.profile.quality_score(
    weights={
        "missing": 40,
        "outliers": 30,
    }
)
```

> **Note:** `Duplicates` at the overall level measures fully duplicated
> *rows*. At the column level (`by="column"`), it measures repeated
> *values* within that column instead — the two aren't the same metric.

---

## 🚨 Data Quality Alerts

Find important data-quality problems automatically, ranked by severity, in
one table instead of reading every section separately:

```python
df.profile.alerts()
```

Returns a DataFrame with columns `Severity`, `Column`, `Alert`, and
`Details` (table-wide issues like duplicate rows or correlated pairs use
`"(table)"` as the column value). Detected alert types include: Missing
Values, Duplicate Rows, Outliers, Constant Column, Near-Constant Column,
Skewed Distribution, High Cardinality, Possible ID Column, Wrong Data Type,
Empty Strings, Extra Whitespace, Zero-Heavy Column, Highly Correlated Pair,
and Potential PII.

Filter for critical issues only:

```python
df.profile.alerts(min_severity="high")
```

`min_severity` accepts `"high"` (High only), `"medium"` (High + Medium), or
`"low"` (everything — the default).

---

## 🔐 PII Detection

Detect potentially personally identifiable information:

```python
df.profile.pii()
```

Returns a DataFrame — `Column`, `PII Type`, `Confidence %` — sorted by
confidence, descending. Detected types:

| Type                 | Meaning                                           |
| -------------------- | ------------------------------------------------- |
| `EMAIL`            | Email addresses                                   |
| `PHONE`            | Phone numbers (Indian mobile pattern + general)   |
| `POSSIBLE_ID`      | Aadhaar-shaped 12-digit numbers                   |
| `POSSIBLE_PAN`     | Indian PAN format (`ABCDE1234F`)                |
| `IP`               | IPv4 addresses                                    |
| `POSSIBLE_NAME`    | Free-text columns that look like personal names   |
| `POSSIBLE_ADDRESS` | Free-text columns that look like postal addresses |

The `POSSIBLE_` prefix is a reminder that ID/PAN/Name/Address detections are
pattern/heuristic guesses, not verified PII — always review before acting on
them (e.g. before dropping or publishing columns). `EMAIL`, `PHONE`, and
`IP` use stricter pattern matching.

Filter by confidence:

```python
df.profile.pii(min_confidence=80)
```

For large DataFrames, only the first `sample` non-null values per column
are scanned by default (for speed):

```python
df.profile.pii(sample=2000)   # default
df.profile.pii(sample=None)   # scan every value instead
```

Generate a masked copy instead of a report:

```python
masked_df = df.profile.pii(mask=True)
```

`mask=True` returns a masked **copy** of the DataFrame — the original is
never modified. Masking is type-aware (emails keep the domain, phone/ID
numbers keep the last few digits, addresses are fully redacted, etc.).

## 🔐 PII & Sensitive Data

DrishtiPy includes local PII detection and analysis capabilities to help users identify potentially sensitive or personally identifiable information (PII) in datasets.

⚠️ **Privacy & Legal Notice:** DrishtiPy processes data locally on the user's device and does not transmit user data to external servers. The PII detection provided by DrishtiPy is an analytical aid and should not be considered a legal determination of whether data is personal, sensitive, regulated, or lawful to process.

Users are solely responsible for ensuring that their use, storage, processing, disclosure, and handling of personal, sensitive, or confidential data complies with all applicable laws, regulations, contractual obligations, and organizational policies in their jurisdiction.

Users should obtain appropriate authorization and apply suitable security and privacy controls before processing personal or sensitive data.

**DrishtiPy does not provide legal advice and does not guarantee compliance with any specific law or regulation.**

---

## 🧮 Correlation Analysis

Analyze relationships between numerical columns (Pearson by default):

```python
df.profile.correlations()
```

Returns the full column-by-column correlation matrix. Get a tidy,
one-row-per-pair view instead, sorted by strength:

```python
df.profile.correlations(by="pairs")
```

Find only strongly correlated pairs — useful for spotting multicollinearity
candidates before modeling:

```python
df.profile.correlations(by="pairs", threshold=0.8)
```

Supported methods:

```python
df.profile.correlations(method="pearson")
df.profile.correlations(method="spearman")
df.profile.correlations(method="kendall")
```

Raises `ValueError` if fewer than two numeric columns are available.

---

## 🤖 ML Readiness

Analyze whether your DataFrame is suitable for machine-learning workflows:

```python
df.profile.ml()
```

One row per column, including: feature type, unique count/%, missing %,
variance, skewness, outlier %, cardinality, encoding suggestion, scaling
suggestion, recommended transformation, and an overall feature status
(e.g. `"Good"`) flagging columns that may need attention before modeling.

---

## 🔄 ETL Readiness

Analyze your DataFrame for common ETL and cleaning problems:

```python
df.profile.etl()
```

One row per column, including: missing/duplicate/unique counts and %,
zero count, negative count, empty-string count, whitespace-issue count,
special-character count, outlier count/%, an `Issue`/`Issue Count` summary,
an `ETL Status` (e.g. `"Ready"`), and a `Recommended Action`.

---

## 🔬 Before / After DataFrame Comparison

`drishtipy` includes `DataComparator` for comparing DataFrames before and after transformations.

Useful for:

- ETL pipelines
- Data cleaning
- Data transformation
- Feature engineering
- Data validation
- Preprocessing workflows

```python
from drishtipy import DataComparator

comparator = DataComparator()

comparison = comparator.compare_dataframe(
    before_df,
    after_df
)
```

Or through the pandas accessor:

```python
df.profile.compare(df2)
```

### Comparison levels

Supported levels:

```text
All
Dataset
Schema
Quality
ML
```

(`"quality"` and `"ml"` currently produce the same column-level report.)

Example:

```python
comparator.compare_dataframe(
    before_df,
    after_df,
    level="quality"
)
```

The result is a long-format `DataFrame` with columns `Section`, `Metric`,
`Before`, `After`, `Change`, and `Status` (not every row populates every
column). Raises `TypeError` if either argument isn't a DataFrame, and
`ValueError` for an unrecognized `level`.

### Changes only

`compare_dataframe` can return dozens of rows on wide DataFrames. Keep only
the rows where something actually changed:

```python
comparator.compare_dataframe(
    before_df,
    after_df,
    changes_only=True
)
```

### Compact summary

A one-row-per-column view instead of the long-format table:

```python
comparator.summary(
    before_df,
    after_df
)
```

```text
   Column     Status                          Summary
0     age    Changed  missing 1->0; outliers 1->0; mean 122.5->35.17
1    city  Unchanged                         No change
2 new_col      Added                     Column added
3 old_col    Removed                   Column removed
```

### Specific columns

```python
comparator.compare_dataframe(
    before_df,
    after_df,
    level="quality",
    columns=["age", "salary"]
)
```

A column missing from one side is still reported as added/removed. Raises
`ValueError` if a name in `columns` isn't present in either DataFrame. Note:
`level="dataset"` metrics (row/column counts, memory, etc.) always reflect
the full `before`/`after` DataFrames, not the selection.

---

## 🌐 HTML Reports

Generate a standalone, dark-themed HTML report instead of raw DataFrames —
handy for sharing a report without a notebook:

```python
df.profile.html(
    "profile_report.html"
)
```

Comparison reports:

```python
df.profile.compare_html(
    df2,
    path="comparison_report.html"
)
```

Explicit classes also support HTML output:

```python
DataProfiler(df).to_html(
    path="profile_report.html"
)

DataComparator().to_html(
    before_df,
    after_df,
    path="comparison_report.html"
)
```

Both accept the same filtering arguments as their DataFrame-returning
counterparts (`section`/`column_type` for `DataProfiler`, `level`/
`changes_only`/`columns` for `DataComparator`), plus `title` for a custom
page heading. Omitting `path` returns the HTML as a string instead of
writing a file.

---

## 🕸️ Relationship Discovery

For a 50-column dataset, there are 50×49/2 = 1,225 possible column pairs —
nobody is manually checking all of them. `relationships()` scans every pair
automatically, picks a statistically appropriate method based on each
column's *semantic* type (not just its pandas dtype), filters out pairs
that can't be meaningfully compared, and ranks what's left:

```python
result = df.profile.relationships()
result          # DRISHTIPY — RELATIONSHIP DISCOVERY dashboard, like info()
```

```text
DRISHTIPY — RELATIONSHIP DISCOVERY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Columns analyzed:       15
Possible pairs:         105
Pairs analyzed:         91
Ignored pairs:          20

Meaningful relationships: 27
Strong:                    8
Moderate:                  7
Weak:                      12

TOP RELATIONSHIPS

  Region             ↔ City               1.00 🔥
  Category           ↔ Product            1.00 🔥
  Sales_Amount       ↔ Cost               1.00 🔥
  Sales_Amount       ↔ Profit             0.98 🔥
  Cost               ↔ Profit             0.96 🔥

result.top(20) / .matrix() / .graph() / .insights() / .dependencies()
```

### What gets compared, and how

Each pair is routed to a method based on **semantic type**
(`drishtipy.semantic.detect_semantic_type` — id, numeric, categorical,
boolean, date/datetime, email, phone, currency, percentage, text,
constant), not raw pandas dtype:

| Pair type                  | Method                                      | Notes                                                                                                         |
| -------------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| numeric ↔ numeric         | Pearson**and** Spearman               | The stronger of the two wins — a monotonic non-linear relationship that Pearson underrates still gets caught |
| categorical ↔ numeric     | ANOVA effect size (η²)                    | Also reports a p-value from an F-test when scipy is installed                                                 |
| categorical ↔ categorical | Cramér's V + normalized Mutual Information | Built from a contingency table with pandas/numpy only — no scipy required for the effect size itself         |
| date ↔ numeric            | Spearman trend                              | Direction reported as Increasing / Decreasing / Flat                                                          |

Pairs that can't be meaningfully compared are **filtered out before any
statistics run**: ID ↔ anything, constant columns, free text, and
email/phone columns are excluded by default (`RelationshipConfig(include_id_pairs=True)`
opts back in for advanced use).

### Configuring thresholds

```python
from drishtipy import RelationshipConfig

result = df.profile.relationships(
    RelationshipConfig(
        strong_threshold=0.80,
        moderate_threshold=0.50,
        min_sample_size=30,
        max_categories=50,     # categorical columns above this are skipped
        max_pairs=5_000,       # safety cap for very wide DataFrames
        sample_size=100_000,   # rows; None disables sampling
    )
)
```

If a pair doesn't have enough valid (non-null, overlapping) observations,
it's reported as `"Not enough valid observations"` rather than a
misleading score.

### Views on the result

```python
result.top(20)               # ranked DataFrame of the strongest relationships
result.matrix()               # symmetric strength matrix (method varies per cell — see below)
result.graph(top=20)          # RelationshipGraph: nodes + weighted edges
result.graph(threshold=0.70)  # only edges at/above a strength cutoff
result.insights()             # auto-generated, human-readable observations
result.dependencies()         # potential functional dependencies (farmer_id -> farmer_name)
result.redundant_columns()    # column groups that look like duplicates/derived copies
result.to_dataframe()         # the full, unfiltered results table
result.to_json()
result.to_html("relationships.html")
```

`result.to_html()` renders a standalone KPI-card dashboard by default — top
relationships, an inline-SVG relationship graph, color-coded insight cards,
and a dependencies list, all computed from the actual result (not a static
mockup):

```python
result.to_html("relationships.html")                    # dashboard style (default)
result.to_html("relationships.html", style="table")      # plain multi-section tables instead
result.to_html("relationships.html", top_n=10, graph_top=15)
```

The graph is real SVG with a computed circular layout (works for any number
of nodes), not a fixed set of hand-placed positions — the most-connected
column gets a highlighted "hub" style, matching how it's usually the one
worth investigating first.

`result.matrix()` is **not** a plain correlation matrix — each cell's value
comes from whichever method fit that pair's semantic types, so don't treat
the whole matrix as linear-correlation strength.

`result.graph()` returns nodes/edges and a readable adjacency-list view —
not a rendered image. Building an actual force-directed layout needs a
graph/plotting stack (networkx, matplotlib) this library intentionally
doesn't depend on; `result.graph().edges` is meant to be handed to one of
those directly if you want a visual layout.

### Functional dependencies and redundant columns

Beyond correlation, `relationships()` checks whether one column's values
consistently determine another's (e.g. `farmer_id -> farmer_name`), and
flags column groups that appear to store the same information twice (exact
duplicates, or near-perfectly correlated numeric/categorical pairs) —
language is deliberately hedged ("Potential functional dependency",
consistency %) rather than claiming certainty the data may not fully
support.

### Multiple testing and significance

With hundreds of pairs analyzed at once, some correlations will look
"significant" by chance alone. When [scipy](https://scipy.org) is
installed (`pip install drishtipy[relationships]`), p-values are computed
per pair and corrected for multiple testing via Benjamini-Hochberg FDR
by default (`RelationshipConfig(fdr_correction=False)` to disable). Without
scipy, effect sizes (Pearson/Spearman/η²/Cramér's V) are still computed —
`Significance` just reports `"Unknown (scipy not installed)"` instead of
guessing.

### Integration

```python
df.profile.relationships()          # via the accessor
df.profile.info().relationships()   # chained from the full profile report
DataProfiler(df).relationships()    # via the explicit class API
```



## 🤖 AutoML Regression

Train and benchmark multiple regression models with one call — no
scikit-learn import required on your end. `scikit-learn`, `numpy`, and
`pandas` are all installed automatically with `pip install drishtipy`.

```python
result = df.ml.auto_predict(target="Sales")
result       # DrishtiPy AutoML Regression summary, printed like df.profile.info()
```

```text
DrishtiPy AutoML Regression
──────────────────────────────────────────────

Target       : Sales
Models Tested: 6
CV Folds     : 5

Best Model   : Gradient Boosting

R²           : 0.9200
Adjusted R²  : 0.9100
RMSE         : 610.21
MAE          : 430.18
MAPE         : 5.10%
CV R²        : 0.9100
CV Std       : 0.0180
Overfit Gap  : 0.0200
Score        : 94.70/100

result.leaderboard() / .best_model() / .predict(new_data) / .report()
```

> **Not the same thing as `df.profile.ml()`.** `df.profile.ml()` is the
> read-only ML-*readiness* report (encoding suggestions, no training).
> `df.ml.auto_predict()` actually trains and benchmarks models. Similar
> names, different jobs.

### What it does automatically

- Detects numeric vs. categorical columns, imputes missing values
  (median for numeric, most-frequent for categorical), scales numeric
  features (matters a lot for Ridge/Lasso), and one-hot encodes
  categoricals — all inside a single scikit-learn `Pipeline`, fit only
  on the training split, so there's no leakage into the test set.
- Unseen categories at prediction time map to all-zero indicator
  columns instead of raising.
- Benchmarks 6 models by default — Linear Regression, Ridge, Lasso,
  Random Forest, Extra Trees, Gradient Boosting — plus XGBoost/
  LightGBM/CatBoost automatically if they're installed
  (`pip install drishtipy[advanced-ml]`; skipped with a one-line notice
  otherwise, never an error).
- Computes 11 regression metrics per model (R², Adjusted R², RMSE, MAE,
  MSE, MAPE, sMAPE, Median AE, Max Error, Explained Variance, RMSLE —
  all with safe handling for zero targets, negative targets, and
  too-few-samples), train/test overfitting gap, and 5-fold
  cross-validation (mean + std of R²/RMSE/MAE).
- Ranks models by a weighted **Composite Score** (0-100) — R² 30%,
  RMSE 20%, MAE 20%, CV R² 15%, CV stability 5%, overfitting 5%,
  training speed 5% — not by R² alone, and not thrown off by models
  tying on a metric or a metric being unavailable for one model.

### Configuring a run

```python
result = df.ml.auto_predict(
    target="Sales",
    test_size=0.20,
    cv=5,
    random_state=42,
    include_optional=True,   # use XGBoost/LightGBM/CatBoost if installed
    n_jobs=-1,
)
```

### Views on the result

```python
result.summary()               # dict of headline numbers
result.leaderboard()           # every model tested, ranked by Composite Score
result.best_model()            # the fitted sklearn Pipeline (preprocessing + model)
result.predict(new_data)       # same preprocessing applied automatically
result.prediction_table()      # Actual / Predicted / Residual / Absolute Error (test split)
result.feature_importance()    # Feature / Importance, one-hot names mapped back, sorted
result.residual_analysis()     # mean/median/std/skew, outlier count, % within ±5/10/20%
result.overfitting_check()     # Train R² / Test R² / Gap / Status (Low/Moderate/High)
result.report("automl.html")   # standalone HTML dashboard
```

`result.feature_importance()` uses `feature_importances_` for tree-based
models and absolute coefficient magnitude for linear ones, with one-hot
encoded categories mapped back to readable names (e.g. `cat__city_Mumbai`
rather than an opaque column index).


## 🔄 ETL Pipeline

A lazy, composable, pandas-native ETL system — `pandas` + `numpy` only,
no heavy dependencies. Every method below only **registers** an
operation; nothing runs until `.run()`, `.to_dataframe()`, or `.load()`.

```python
import drishtipy as dp

pipeline = (
    dp.ETL()
    .extract("customers.xlsx")
    .text("name").strip().title()
    .numeric("age").to_numeric()
    .condition("age", "<", 0).replace(value=None)
)
# nothing has executed yet — no file read, no output written

result = pipeline.run()      # now it actually runs
df = result.data
```

### Reusable, composable pipelines

Build small pipelines once, reuse them anywhere:

```python
missing_pipeline = (
    dp.ETL()
    .fill_missing("age", strategy="median")
    .fill_missing("city", value="Unknown")
    .drop_missing(["email"])
)

text_pipeline = (
    dp.ETL()
    .text("name").strip().title()
    .text("email").strip().lower()
)

numeric_pipeline = (
    dp.ETL()
    .numeric("age").to_numeric()
    .condition("age", "<", 0).replace(value=None)
    .condition("age", ">", 120).replace(strategy="median")
)

date_pipeline = dp.ETL().date("signup_date").to_datetime()
```

Combine them with `.include()` (mutates the pipeline it's called on —
built for assembling one master pipeline step by step):

```python
master_pipeline = (
    dp.ETL()
    .extract("customers.xlsx")
    .include(text_pipeline)
    .include(numeric_pipeline)
    .include(missing_pipeline)
    .include(date_pipeline)
    .validate()
    .quality_report()
)
```

...or with `+` (returns a **new** pipeline, leaving both operands
untouched — safe to combine the same reusable pipeline into several
different masters):

```python
pipeline_a = base_pipeline + numeric_pipeline
pipeline_b = base_pipeline + missing_pipeline
# base_pipeline, pipeline_a, and pipeline_b are all independent
```

> **Operator precedence note:** `dp.ETL().extract(x) + a + b.validate()`
> calls `.validate()` on `b` alone, not on the combined pipeline —
> Python binds the method call to the nearest preceding expression.
> Wrap the `+` chain in parentheses before calling further methods:
> `(dp.ETL().extract(x) + a + b).validate()`.

### Text / numeric / date column API

```python
.text("name").strip().lower().title()      # strip, lower, upper, title, capitalize
.numeric("age").to_numeric().round(2).abs()
.date("signup_date").to_datetime().year().month().day().day_name()
```

Date-part extraction (`.year()`, `.month()`, etc.) adds new columns
(`signup_date_year`, ...) rather than overwriting the original.

### Conditional replace

```python
.condition("age", "<", 0).replace(value=None)
.condition("age", ">", 120).replace(strategy="median")
```

Operators: `==`, `!=`, `<`, `<=`, `>`, `>=`, `in`, `not_in`, `between`,
`is_null`, `not_null`. Strategies: `median`, `mean`, `mode`, `min`,
`max`, `zero`. `replace(value=None)` explicitly nulls out matching
values — it is *not* treated as "no value passed" (an internal
sentinel distinguishes the two).

### Missing values, duplicates, outliers

```python
.fill_missing("age", strategy="median")     # median/mean/mode/min/max/zero/ffill/bfill
.fill_missing("city", value="Unknown")
.drop_missing(["email"])

.drop_duplicates(["customer_id"])           # or drop_duplicates() for all columns

.outliers("total_spent", method="iqr", action="flag")
# methods: iqr, zscore, modified_zscore, percentile
# actions: flag (adds total_spent_outlier, doesn't touch values), remove, replace, clip
```

### Validation and quality scoring

```python
.validate(required_columns=["customer_id"], ranges={"age": (0, 120)})
.quality_report()
```

`quality_report()` doesn't reimplement scoring — it calls the same
`DataProfiler.quality_score()` used by `df.profile.quality_score()`,
once on the data right after `extract()` and once on the final result,
so "before ETL" vs. "after ETL" numbers are directly comparable to
anything else in drishtipy.

`validate()` never turns a soft issue into a fatal error on its own —
missing values, duplicates, and invalid coercions default to
`WARNING`; only explicitly-configured `required_columns`/`unique_columns`
checks can produce `FAIL`.

### Execution and output

```python
df = pipeline.to_dataframe()                     # run() and just take .data

result = pipeline.load("clean.csv")               # run + write output
result = pipeline.load("clean.xlsx", report="etl_report.html")  # + HTML report
```

`.load()` never touches the source file — every `.run()`/`.load()`
call re-extracts from scratch, so running the same pipeline twice (or
composing it into two different masters) always starts from the
original data.

### Inspecting a pipeline without running it

```python
pipeline.describe()
```

```text
DRISHTIPY ETL PIPELINE
----------------------

1. Extract customers.xlsx
2. Text: name → strip → title
3. Numeric: age → to_numeric
4. Condition: age < 0 → None

Status: CONFIGURED
Operations: 4
```

### The result object

```python
result.data                  # final pandas DataFrame
result.status                # PASS / WARNING / FAIL / COMPLETED
result.statistics             # dict: input/output records, missing fixed, duplicates, ...
result.warnings                # list of WARNING-level validation messages
result.execution_time
result.audit_log()            # DataFrame: one row per executed operation
result.validation_report()    # DataFrame: Check / Status / Details
result.execution_log()        # DataFrame: Timestamp / Operation / Duration / Status
result.to_html("report.html")
result.to_csv("out.csv")
result.to_excel("out.xlsx")
```


## 📁 Large CSV Profiling

Profile large CSV files without loading the entire source into memory:

```python
from drishtipy import DataProfiler

profiler = DataProfiler.from_csv(
    "huge_dataset.csv",
    sample_size=100_000,
    chunksize=50_000,
    random_state=42
)
```

Then:

```python
profiler.info_dataframe()
```

Check whether sampling was used:

```python
profiler.is_sampled
```

Get the exact source row count:

```python
profiler.total_rows_in_source
```

### How sampling works

```text
CSV file
   │
   ├── Pass 1
   │     └── Count total rows
   │
   ├── Pass 2
   │     └── Select a random sample
   │
   └── Profile the sample
```

The file is streamed in chunks twice — once to count exact total rows, once
to keep each row with probability `sample_size / total_rows` (a random,
roughly-uniform sample). If the file has fewer rows than `sample_size`,
it's loaded in full and `is_sampled` stays `False`.

When sampling is used, the source row count (`total_rows_in_source`) is
exact, while statistics/ML sections computed from the sample are estimates
of the complete dataset. `to_html()` on a sampled profiler automatically
notes the sample size vs. total rows in the report.

Additional keyword arguments can be forwarded to `pandas.read_csv()`:

```python
DataProfiler.from_csv(
    "data.csv",
    usecols=["age", "salary"],
    dtype={"age": "float64"},
    sep=",",
    parse_dates=["date"]
)
```

---

## 🧱 Explicit Class API

The pandas accessor is recommended, but explicit classes are also available
— useful if you want to keep the profiler object around for repeated calls,
or prefer not to rely on accessor registration:

```python
import pandas as pd

from drishtipy import DataProfiler

df = pd.DataFrame({
    "age": [25, 32, None, 47],
    "city": ["Delhi", "Mumbai", "Delhi", "Pune"]
})

profiler = DataProfiler(df)

profiler.info_dataframe(
    section="schema"
)

profiler.quality_score()
profiler.alerts()
profiler.pii()
profiler.correlations()
```

Raises `TypeError` if `df` isn't a `pandas.DataFrame`.

---

## ⚙️ `info_dataframe()`

The underlying profiler API provides:

```python
profiler.info_dataframe(
    section="All",
    column_type="All"
)
```

### `section`

Supported values (case-insensitive):

```text
All
Schema
Statistics
Quality
ML
ETL
```

`"All"` returns a `dict` mapping section name -> `DataFrame`:

```python
{
    "Schema": ...,
    "Statistics": ...,
    "Quality": ...,
    "ML": ...,
    "ETL": ...
}
```

Any other value returns just that section's `DataFrame`.

### `column_type`

Supported values (case-insensitive):

```text
All
Numeric
Categorical
```

Example:

```python
profiler.info_dataframe(
    section="statistics",
    column_type="numeric"
)
```

Raises `ValueError` if `section` or `column_type` isn't recognized, or if
`column_type` filters out every column.

---

## 🧠 Design Philosophy

### pandas-first

`drishtipy` extends pandas rather than replacing it.

### Lightweight

The core package intentionally keeps dependencies minimal — pandas is the
only required dependency.

### Non-destructive

Profiling operations analyze the DataFrame without modifying it. The one
exception is explicit: `df.profile.pii(mask=True)` returns a masked
**copy**, never mutating the original.

### Composable

Reports remain compatible with normal pandas operations — sort, filter,
export, or display them however you like.

### Human-readable

HTML reports and summaries are designed to make data problems easy to
understand at a glance.

---

## 🧪 Development

Install development dependencies:

```bash
pip install -e ".[dev]"
```

Run tests:

```bash
pytest
```

Recommended workflow:

```bash
git clone <repository>
cd drishtipy

pip install -e ".[dev]"

pytest
```

---

## 🗺️ Roadmap

Potential future capabilities include:

- Advanced validation rules
- Automatic cleaning suggestions
- Smart semantic data-type/data-dictionary detection
- Time-series profiling
- Advanced ML-readiness checks
- Data drift detection
- Statistical distribution analysis
- Excel / JSON / Parquet / SQL database profiling
- Expanded interactive HTML dashboards
- Additional data-quality checks

---

## 📦 Package Information

| Property           | Value          |
| ------------------ | -------------- |
| Package            | `drishtipy`  |
| Import             | `drishtipy`  |
| Current Version    | `0.6.1`      |
| Python             | `>=3.8`      |
| License            | MIT            |
| Primary Dependency | pandas (>=1.3) |

Install:

```bash
pip install drishtipy
```

Import:

```python
import drishtipy
```

Use:

```python
df.profile.quality()
```

---

## 📄 License

`drishtipy` is released under the MIT License.

⚠️ **Privacy Notice:** DrishtiPy processes data locally on the user's device and does not transmit user data to external servers. Users are responsible for ensuring that their use of personal, sensitive, or confidential data complies with applicable laws, regulations, and organizational policies.
