Metadata-Version: 2.4
Name: dpdp-scrub
Version: 0.0.3
Summary: Detect, validate, and redact Indian PII (Aadhaar, PAN, GSTIN, UPI, and more) — self-hosted, checksum-validated, DPDP-ready.
Author: Palkush Dave
License: MIT
Keywords: pii,dpdp,aadhaar,pan,gstin,privacy,redaction,india
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Security
Classifier: Topic :: Text Processing
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Provides-Extra: server
Requires-Dist: fastapi>=0.110; extra == "server"
Requires-Dist: uvicorn>=0.29; extra == "server"
Provides-Extra: redis
Requires-Dist: redis>=5; extra == "redis"
Dynamic: license-file

# dpdp-scrub

[![PyPI](https://img.shields.io/pypi/v/dpdp-scrub)](https://pypi.org/project/dpdp-scrub/)
[![CI](https://github.com/ramcharanteja0307/dpdp-scrub/actions/workflows/ci.yml/badge.svg)](https://github.com/ramcharanteja0307/dpdp-scrub/actions)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue)](https://pypi.org/project/dpdp-scrub/)

**Detect, validate, and redact Indian PII before it reaches an LLM — self-hosted, checksum-validated, DPDP-ready.**

Your org keeps the keys; the LLM gets the placeholders.

```
IN  : Sir aadhaar 2345 6789 0124 hai, call karo 9876543210 pe, PAN ABCPE1234F
LLM : Sir aadhaar XXXX XXXX 0124 hai, call karo <PHONE_IN_1> pe, PAN <PAN_1>
BACK: Done! 9876543210 pe call schedule ho gaya.        ← rehydrated, agent-side
```

## Why this exists

Every Indian company wiring LLMs into support, sales, and ops is streaming
Aadhaar numbers, PANs, GSTINs, UPI IDs, and phone numbers to third-party model
providers — and copying them into logs, vector DBs, and fine-tuning sets.
The DPDP Act's substantive obligations become enforceable in **May 2027**
(penalties up to ₹250 crore); UIDAI's Aadhaar-masking rules and RBI's card
rules already apply.

The redaction loop itself is well-established engineering (Presidio, Google
DLP). What didn't exist is **eyes that can see Indian PII** — validated,
context-aware, and open. Google Cloud DLP ships 3 Indian infoTypes, AWS
Comprehend 4, Presidio 6. dpdp-scrub detects **15**, most with mathematical
or reference-data validation:

| Entity | Validation |
|---|---|
| Aadhaar / Aadhaar VID | **Verhoeff checksum** + first-digit rule |
| GSTIN | **mod-36 check character** + embedded-PAN structure + state code |
| Credit/debit card | **Luhn checksum** |
| PAN | holder-type structure rule |
| UPI ID | handle verified against **known PSP list** (`@ybl`, `@oksbi`, …) |
| IFSC, phone, email, voter ID, passport | structural + context scoring |
| Vehicle registration | state/RTO code verified against reference list (incl. BH series) |
| Bank account, EPF UAN, ABHA | **context-required** (surface only with evidence like "A/c", "UAN") |

The context engine is what separates a phone number from an order ID:

```
order 9876543210 ka status batao   →  nothing        (suppressed by "order")
call karo 9876543210 pe            →  PHONE_IN 0.99  (boosted by "call")
mera a/c 34512345678901 hai        →  BANK_ACCOUNT   (earned by "a/c")
code 34512345678901 likha hai      →  nothing
```

## Quick start

```bash
pip install dpdp-scrub
```

```python
from dpdp_scrub import Scrubber

s = Scrubber()

result = s.scrub("mera PAN ABCPE1234F hai, call 9876543210", session_id="ticket-42")
result.text    # "mera PAN <PAN_1> hai, call <PHONE_IN_1>"
result.spans   # typed spans: entity, offsets, confidence

reply = call_your_llm(result.text)          # provider never sees real values
s.rehydrate(reply, "ticket-42")             # real values restored, your side only
```

Detection only:

```python
from dpdp_scrub import detect
detect("payment 9876543210@ybl pe karo")
# [Span(entity='UPI_ID', start=8, end=22, ..., confidence=0.92)]
```

Zero runtime dependencies. Python 3.9+.

## Trust model

```
Ring 0: the vault (your Redis/memory)   — real values, session-scoped, droppable
Ring 1: your org's services             — placeholders + scoped right to resolve
Ring 2: LLM providers, logs, vector DBs — placeholders only, no resolution path
```

- Same value → same placeholder within a session (multi-turn consistency).
- `vault.drop(session)` makes every old copy of its placeholders permanently
  unresolvable — retroactive redaction for logs you already wrote.
- No cloud, no telemetry, no phoning home. This is a library, not a service:
  **your data never leaves your infrastructure.**

Per-entity actions are policy: `tokenize` (reversible), `mask` (UIDAI-format
`XXXX XXXX 0124` for Aadhaar, last-4 for cards), `redact` (ABHA health IDs by
default), `ignore`.

## Benchmark

IndicPII-Bench v0 (in `bench/`): 500 synthetic messages — Hinglish, English,
Devanagari, OCR-noise spacing, multi-entity — with 20% traps (order IDs,
invoice refs, phones-in-txn-context). Presidio runs at its best
configuration — its IN_* recognizers explicitly registered (they are shipped
but **not loaded by default**). F1, value-level matching:

| entity | dpdp-scrub | presidio |
|---|---|---|
| AADHAAR | **1.000** | 1.000 |
| GSTIN | **1.000** | 0.971 |
| PAN | **1.000** | 0.383 |
| PHONE_IN | **1.000** | 0.602 |
| UPI_ID | **1.000** | 0.000 |
| IFSC | **1.000** | 0.000 |
| EMAIL | **1.000** | 1.000 |
| **OVERALL** | **1.000** | 0.595 |
| trap false positives | **0/100** | 200/100 |

Honest caveats: this is our own v0 corpus (templated, synthetic, Hinglish-
heavy) — a perfect self-score mainly means the corpus is still too easy, and
it will get harder (Devanagari, OCR noise, multi-entity). Presidio was not
built for code-mixed text; the 107 trap FPs come from it having no context
suppression (e.g. any 10-digit number near "txn" flags as phone). Reproduce:
`python3 bench/compare.py 500`.

## Status

Pre-alpha, moving fast. Built so far: 15 entities, checksum validators,
context-scoring engine, scrub/rehydrate loop with session vault — 86 tests,
all synthetic data. Roadmap:

- [ ] YAML policy profiles (`dpdp-strict`, `uidai-mask`, `audit-only`) + audit log
- [ ] `scrub_json()` for structured payloads
- [ ] Redis vault backend
- [ ] LiteLLM / LangChain / Presidio adapters
- [ ] **IndicPII-Bench** — a public benchmark for Indian PII detection
- [ ] Indic-script & code-mixed NER for names/addresses (AI4Bharat IndicNER)

## Contributing

The reference-data files are designed for one-line PRs: new UPI PSP handles,
RTO codes, context words (English/Hindi/regional). Entity requests welcome —
include format, any checksum, and public documentation.

All test data must be synthetic (checksum-generated) or published samples.
Never commit real PII, including your own.

## License

MIT
