Metadata-Version: 2.4
Name: gibsondedup
Version: 0.1.1
Summary: A high-performance search result deduplication engine that groups duplicate results by canonical URL and semantic similarity, returning clean batches with the most authoritative source and full traceability to all original sources.
Author: Gibson Kwabena Aseda Mensah
Author-email: logosyninc@gmail.com
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Dynamic: author-email
Dynamic: requires-python

  ![logo](./Gemini_Generated_Image_cx7jpwcx7jpwcx7j.png)


# 🧹 gibsondedup
  
  ### *High‑Performance Search Result Deduplication Engine*
  
  [![Python Version](https://img.shields.io/badge/python-3.8%2B-blue.svg)](https://python.org)
  [![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
  [![Tests](https://img.shields.io/badge/tests-59%20passing-brightgreen.svg)](#testing)
  [![Coverage](https://img.shields.io/badge/coverage-94%25-success.svg)](#testing)
  [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/kwabenaaseda/gibsondedup)


---

## 📌 The Problem

When aggregating search results from multiple providers (Google, Bing, DuckDuckGo), **40–60%** of results are duplicates — same content, different URLs, different titles, different tracking parameters.

```
Source 1 (Google):
  url: https://www.python.org/docs?utm_source=google
  title: "Python Official Documentation"

Source 2 (Bing):
  url: http://python.org/docs/
  title: "Python Docs"

Source 3 (DuckDuckGo):
  url: https://python.org/docs?utm_medium=cpc
  title: "Learn Python - Official Docs"
```

These are the same resource. Without deduplication, users see **noise instead of signal**.

---

## ✨ The Solution

`gibsondedup` processes raw search results through a **multi‑stage pipeline**:

```mermaid
flowchart TD
    A[Raw Results] --> B[Parse & Validate]
    B --> C[URL Normalization]
    C --> D[Exact URL Grouping]
    D --> E[Semantic Merge]
    E --> F[Canonicalization]
    F --> G[Clean Output + Metadata]
```

**Result:** 100 noisy inputs → **60–70 clean, canonical results**.

---

## 🚀 Installation

```bash
pip install gibsondedup
```

---

## 🏃 Quick Start

```python
from gibsondedup import DeduplicationEngine

engine = DeduplicationEngine()

results = engine.process([
    {
        "title": "Python Official Documentation",
        "url": "https://www.python.org/docs?utm_source=google",
        "description": "Official Python docs",
        "source": "google"
    },
    {
        "title": "Python Docs",
        "url": "http://python.org/docs/",
        "description": "Python documentation",
        "source": "bing"
    },
    {
        "title": "Stack Overflow Python",
        "url": "https://stackoverflow.com/questions/tagged/python",
        "description": "Python questions",
        "source": "google"
    },
])

print(results)
```

**Output:**

```json
{
  "results": [
    {
      "title": "Python Official Documentation",
      "url": "https://www.python.org/docs?utm_source=google",
      "canonical_url": "python.org/docs",
      "description": "Official Python docs",
      "sources": ["google", "bing"],
      "duplicates_removed": 1
    },
    {
      "title": "Stack Overflow Python",
      "url": "https://stackoverflow.com/questions/tagged/python",
      "canonical_url": "stackoverflow.com/questions/tagged/python",
      "description": "Python questions",
      "sources": ["google"],
      "duplicates_removed": 0
    }
  ],
  "meta": {
    "total_input": 3,
    "total_parsed": 3,
    "total_output": 2,
    "duplicates_removed": 1,
    "processing_time_ms": 0.09,
    "similarity_threshold": 0.8
  }
}
```

---

## ⚙️ Configuration

### Custom Similarity Threshold

Control how aggressively similar‑titled results are merged:

```python
# Conservative (default) — only merge highly similar titles
engine = DeduplicationEngine(similarity_threshold=0.8)

# Moderate — merge titles with moderate overlap
engine = DeduplicationEngine(similarity_threshold=0.5)

# Aggressive — merge titles with minimal overlap
engine = DeduplicationEngine(similarity_threshold=0.3)
```

### Input Contract

| Field         | Type   | Required | Description                        |
|---------------|--------|----------|------------------------------------|
| `title`       | string | ✅       | Result title                       |
| `url`         | string | ✅       | Result URL                         |
| `description` | string | ❌       | Result description                 |
| `source`      | string | ❌       | Source provider (google, bing, etc.) |

> **Note:** Malformed records (missing title or URL) are **skipped gracefully** — the pipeline continues processing valid records.

---

## 🧱 Architecture

### Pipeline Stages

| Stage | Description |
|-------|-------------|
| **1. Parser** | Converts raw JSON payloads into structured internal contracts. Validates required fields. Skips malformed records without crashing. |
| **2. URL Normalizer** | Converts URLs to canonical form by removing protocol, `www`, tracking params (`utm_*`, `fbclid`, `gclid`), trailing slashes, default ports, and lowercasing everything. |
| **3. Exact Grouping** | Groups results by normalized canonical URL using a hash map. **O(n)** complexity – avoids O(n²) pairwise comparison. |
| **4. Semantic Merger** | Within groups sharing the same domain, compares titles using **Jaccard similarity**. Groups exceeding threshold are merged. Only compares within the same exact domain. |
| **5. Canonicalizer** | Selects the single best result per group using a combined title+description length heuristic. Collects source traceability from all merged results. |

### Data Contracts

```python
# Input
RawSearchResult(title, url, description?, source?)

# Internal
NormalizedSearchResult(title, url, canonical_url, description, source)

# Output
CanonicalResult(title, url, canonical_url, description, sources[], duplicates_removed)
```

### System Invariants

1. Same normalized URL → same duplicate group (always)
2. Canonical results preserve source traceability (always)
3. Pipeline stages do not mutate upstream data (always)
4. Normalization happens once per result (always)

---

## 🧠 Engineering Decisions

| Question | Answer |
|----------|--------|
| Why hash‑based grouping (O(n)) over pairwise (O(n²))? | With 1000 results, O(n²) → 1M comparisons. O(n) → 1000. Milliseconds vs seconds. |
| Why exact domain matching for semantic merging? | `python.org/docs` and `docs.python.org` serve different content. Exact domain prevents false merges. |
| Why Jaccard similarity over Levenshtein distance? | Jaccard operates on word tokens – robust to word order changes & additions. Levenshtein is sensitive to trivial differences like "Docs" vs "Documentation". |
| Why keep normalization deterministic? | Same input → same output. Enables reproducible runs, safe caching, reliable tests. |

---

## 📊 Performance

| Input Size | Processing Time | Memory |
|------------|----------------|--------|
| 100 results | ~1ms           | ~1MB   |
| 1,000 results | ~10ms        | ~5MB   |
| 10,000 results | ~100ms       | ~50MB  |

*Benchmarked on Ubuntu 22.04, Python 3.10, Intel i5*

---

## 🧪 Testing

```bash
# Install dev dependencies
pip install pytest pytest-cov

# Run all tests
pytest tests/ -v

# Run with coverage
pytest tests/ --cov=app --cov-report=term-missing
```

**Test coverage: 59 tests across 5 modules**

| Module | Tests | Focus |
|--------|-------|-------|
| `test_normalizer.py` | 14 | URL normalization |
| `test_grouping.py` | 4  | Exact grouping |
| `test_canonicalizer.py` | 7 | Best result selection |
| `test_merger.py` | 11 | Semantic merging |
| `test_similarity.py` | 13 | Jaccard similarity |
| `test_pipeline.py` | 10 | End‑to‑end pipeline |

---

## 🗺️ Roadmap

| Phase | Status | Features |
|-------|--------|----------|
| Phase 1 | ✅ Complete | URL normalization, exact duplicate grouping, canonicalization with source traceability |
| Phase 2 | ✅ Complete | Jaccard similarity engine, domain‑based semantic merging, configurable threshold |
| Phase 3 | 🚧 Planned | Persistent caching (Redis), database storage (PostgreSQL), REST API, Rails wrapper |
| Phase 4 | 🔮 Planned | Semantic embeddings (vector similarity), distributed processing, production observability |

---

## 📄 License

MIT License. See `LICENSE` file.

---

## 👤 Author

**Aseda Gibson**  
Computer Engineering, University of Energy and Natural Resources (UENR), Ghana.  
Backend systems, distributed architecture, infrastructure tooling.

[![GitHub](https://img.shields.io/badge/GitHub-kwabenaaseda-black?logo=github)](https://github.com/kwabenaaseda)

---

<div align="center">
  <sub>Built with ❤️ and correctness‑oriented engineering.<br>Every architectural decision is documented. Every invariant is tested.</sub>
</div>
```
