Metadata-Version: 2.4
Name: leadmatch
Version: 0.1.0
Summary: Fast lead/contact deduplication: blocking + fuzzy matching, benchmarked against naive all-pairs
Project-URL: Homepage, https://github.com/humza210/LeadDeDupe
Project-URL: Repository, https://github.com/humza210/LeadDeDupe
Author: Hamza Amjad
License: MIT
License-File: LICENSE
Keywords: crm,deduplication,entity-resolution,fuzzy-matching,leads,record-linkage
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.10
Requires-Dist: datasketch>=1.6
Requires-Dist: faker>=24.0
Requires-Dist: numpy>=1.24
Requires-Dist: pandas>=2.0
Requires-Dist: rapidfuzz>=3.9
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Description-Content-Type: text/markdown

# leadmatch

[![CI](https://github.com/humza210/LeadDeDupe/actions/workflows/ci.yml/badge.svg)](https://github.com/humza210/LeadDeDupe/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/leadmatch)](https://pypi.org/project/leadmatch/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

Fast, standalone lead/contact deduplication (identity resolution) for messy
CRM data — with a reproducible benchmark proving the speedup over naive
all-pairs fuzzy matching is **algorithmic**, not implementation slop.

## Why

Leads arrive from web forms, ad campaigns, calls, and SMS. The same person
shows up as `Bob Smith / bob.smith+promo@gmail.com` and
`Robert Smith / bobsmith@gmail.com / (555) 123-4567`. Every duplicate lead
wastes ad spend on re-targeting, splits outreach history across records, and
double-counts pipeline. Deduplicating naively means comparing every record
with every other — 1.25 **billion** pairs at 50k records — which stops scaling
long before a real lead database does.

`leadmatch` fixes the asymptotics with **blocking**: candidate pairs come from
cheap indexes (normalized email, phone, phonetic name keys, MinHash LSH), and
only those candidates get scored. It has no external services and makes no
network calls, so it can run in a batch job, a worker, or a microservice.

## Install

```bash
pip install leadmatch       # from PyPI
# or from a checkout:
pip install -e .
# runtime deps: rapidfuzz, datasketch, faker, typer, pandas, numpy
```

## Quickstart

```bash
# 1. Generate a synthetic dataset with ground truth (or bring your own CSV)
leadmatch gen --n 50000 --dup-rate 0.3 --seed 42 --out leads.csv

# 2. Deduplicate: one canonical record per cluster
leadmatch run --input leads.csv --out merged.csv --clusters-out clusters.csv

# 3. Reproduce the benchmark below
leadmatch benchmark --sizes 10000,50000 --seed 42 --out BENCHMARK.md
```

Input CSV columns (all optional, but each record needs at least one of
email / phone / full name):
`first_name, last_name, email, phone, company, city, zip, source, created_at`.

As a library:

```python
from leadmatch import Lead, MatchConfig, dedupe_leads

leads = [
    Lead("a", first_name="Bob", last_name="Smith", email="bob.smith+ads@gmail.com"),
    Lead("b", first_name="Robert", last_name="Smith", email="bobsmith@gmail.com"),
]
result = dedupe_leads(leads, MatchConfig(threshold=0.85))
result.canonical    # one merged record per cluster
result.assignment   # record_id -> cluster_id
```

## How it works

1. **Normalize** — emails are lowercased, `+tag` suffixes stripped, Gmail dots
   removed; phones reduced to digits with US country code dropped; names
   lowercased, punctuation-stripped, nicknames canonicalized via a built-in
   ~190-entry map (`bob→robert`, `liz/beth→elizabeth`, …), and Double Metaphone
   codes computed by a from-scratch implementation.
2. **Block** — candidate pairs are the union of: exact normalized email,
   phone (last 7 digits), `(metaphone(last), zip)`, `(metaphone(last), city)`,
   `(first_initial, metaphone(last))`, and MinHash LSH (via `datasketch`) over
   character 3-gram shingles of `"first last email phone"` at Jaccard ≈ 0.5 as
   a fuzzy catch-all.
3. **Score** — only candidate pairs, vectorized with `rapidfuzz`:
   Jaro-Winkler name similarity on canonicalized full names (max over swapped
   first/last), structure-aware email similarity (digit runs in the local part
   are identity-bearing: `jsmith12` ≠ `jsmith87`), exact phone, and
   company+city similarity, combined as a weighted composite
   (email 0.35, phone 0.25, name 0.30, company+city 0.10 — renormalized over
   fields present in both records). Exact email auto-matches; exact phone with
   a strong name auto-matches; otherwise composite ≥ threshold (default 0.85)
   with a minimum-evidence floor so a shared common name + city alone never
   merges two different people.
4. **Cluster** — matched pairs feed a from-scratch union-find (path
   compression + union by rank); each cluster is merged into one canonical
   record using the most complete value per field.

## Benchmark

Latest committed run (see [BENCHMARK.md](BENCHMARK.md) for the full report and
`results.json` for machine-readable numbers):

| records | pipeline | wall time (s) | pairs scored | peak mem (MB) | precision | recall | F1 |
|---:|---|---:|---:|---:|---:|---:|---:|
| 10,000 | naive | 52.8 | 49,995,000 | 747 | 0.9857 | 0.9525 | 0.9688 |
| 10,000 | blocked | 3.2 | 41,481 | 228 | 0.9858 | 0.9493 | 0.9672 |
| 50,000 | naive | ~2,762.0 (estimated) | 1,249,975,000 | 480 | — | — | — |
| 50,000 | blocked | 18.6 | 953,066 | 897 | 0.9661 | 0.9437 | 0.9548 |

- **10,000 records:** blocking scores **1,205× fewer pairs** and is
  **16.6× faster** end-to-end, at the same accuracy as naive (F1 0.9672 vs
  0.9688, identical precision).
- **50,000 records:** blocking scores **1,312× fewer pairs** and is
  **~148× faster** (the naive run was projected past the time cap and its
  wall time extrapolated from a same-scorer random pair sample, clearly
  flagged above).

Machine: Linux x86_64, Python 3.11, 4 CPU cores; seed 42, dup rate 0.3,
threshold 0.85, naive time cap 900 s.

Regenerate with `leadmatch benchmark --sizes 10000,50000 --seed 42`.

## Methodology & honesty notes

The benchmark is designed so the delta can only come from the algorithm:

- **Same scorer both paths.** The naive baseline and the blocked pipeline call
  the *same* normalizer, the *same* rapidfuzz scoring functions, and the
  *same* thresholds. The only difference is candidate generation: naive
  scores all O(n²) pairs; blocked scores pairs produced by blocking + LSH.
  The wall-time and pairs-scored gaps are purely algorithmic.
- **Seeded synthetic data.** Datasets come from `leadmatch gen`, which is
  fully deterministic per seed and ships in this repo, including the hidden
  `entity_id` ground-truth column used for pairwise precision/recall/F1.
- **Honest timing.** Each benchmark leg runs in its own subprocess: wall time
  is measured around the end-to-end pipeline, and peak memory is the
  process-lifetime `ru_maxrss` of that leg alone.
- **Time-capped naive runs are labeled.** If a naive run is projected (from a
  random 1M-pair sample scored with the same scorer) to exceed the time cap
  (default 30 min), its wall time is extrapolated and clearly marked
  *estimated*; accuracy metrics are not reported for extrapolated runs.
- **Reproduce it:** `pip install -e . && leadmatch benchmark --sizes 10000,50000 --seed 42`.

## Limitations

- **US-centric phone handling.** Normalization assumes NANP numbers (drops a
  leading `1`, matches on the last 7–10 digits). International numbers work
  only incidentally.
- **Synthetic ≠ production.** The generator's corruption distribution (typos,
  nicknames, tag/domain changes, reformatting) is realistic but stylized; real
  lead streams have different noise. Treat the benchmark as a controlled
  comparison of candidate-generation strategies, not a production accuracy
  claim.
- **Precision-first design choices.** Records sharing only a name and
  geography never merge (common-name homonyms outnumber true matches there),
  and a shared phone with two clearly different names is treated as a
  household/office line, not a duplicate. Tune `MatchConfig` if your data
  says otherwise.
- **English nickname map.** The built-in canonicalization covers common
  English given names only.
- Blocking is approximate by construction: pairs sharing no block key and
  falling under the LSH threshold are never scored. The benchmark quantifies
  the resulting recall gap against the naive baseline (≈0.3% of gold pairs at
  10k in the committed run).

## Development

```bash
pip install -e ".[dev]"
ruff check .
pytest
```

CI runs lint, the full test suite, and a 5k-record mini-benchmark on every
push.

## License

[MIT](LICENSE)
