Metadata-Version: 2.2
Name: zedda
Version: 0.4.0
Summary: Zero Effort Data Analysis — 1TB files in seconds, C++ parallel core
Keywords: eda,data-science,profiling,csv,parquet,pandas,fast,cli,analysis,sampling,arrow
License: MIT
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: C++
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Project-URL: Homepage, https://github.com/prince3235/Zedda
Project-URL: Repository, https://github.com/prince3235/Zedda
Project-URL: Issues, https://github.com/prince3235/Zedda/issues
Requires-Python: >=3.9
Requires-Dist: rich>=13.0
Requires-Dist: typer>=0.12
Requires-Dist: pyarrow>=10.0.0
Provides-Extra: ai
Requires-Dist: openai>=1.0; extra == "ai"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pandas>=2.0; extra == "dev"
Requires-Dist: numpy>=1.24; extra == "dev"
Description-Content-Type: text/markdown

<div align="center">
  <img src="https://raw.githubusercontent.com/prince3235/zedda/main/docs/logo.png" alt="Zedda Logo" width="400"/>
  <h3>Zero Effort Data Analysis</h3>
  <p>The world's fastest EDA library — C++ powered, pip installable</p>

  [![PyPI](https://img.shields.io/pypi/v/zedda?color=blue&label=PyPI)](https://pypi.org/project/zedda/)
  [![Python](https://img.shields.io/pypi/pyversions/zedda?color=green)](https://pypi.org/project/zedda/)
  [![License](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/prince3235/zedda/blob/main/LICENSE)
</div>

---

## Why Zedda?

```python
# Everything pandas needs 10 lines for — Zedda does in 1
import zedda as zd
zd.profile("titanic.csv")
```

| Feature | pandas | ydata-profiling | Zedda |
| :--- | :--- | :--- | :--- |
| **Titanic (891 rows)** | manual | ~45s | **19ms** ⚡ |
| **6.3M row CSV** | manual | ~10 min | **23s** ⚡ |
| **1TB Parquet** | OOM crash | OOM crash | **< 2s** ⚡ |
| **RAM usage** | $O(N)$ | $O(N)$ | **$O(\text{cols})$** ✅ |
| **pip install size** | ~30MB | 200MB+ | **< 1MB** ✅ |
| **Pearson correlation** | manual | slow | **single-pass** ✅ |
| **ML readiness hints** | ❌ | ❌ | **✅** |
| **CLI support** | ❌ | ❌ | **✅** |

---

## Install

```bash
pip install zedda
```

* No C++ compiler needed — pre-built wheels for Windows, macOS, Linux.
* Requires Python 3.9+.

---

## Quickstart

### Profile any file

```python
import zedda as zd

zd.profile("data.csv")         # CSV
zd.profile("data.parquet")     # Parquet — uses footer cheat code
zd.profile("data.arrow")       # Arrow
```

### Get results as object

```python
p = zd.scan("titanic.csv")

print(p.num_rows)              # 891
print(p.num_cols)              # 12
print(p.overall_null_pct)      # 8.1

for col in p.columns:
    print(col.name, col.mean, col.null_pct)
```

### Compare two datasets

```python
zd.compare("train.csv", "test.csv")
# Shows: schema diffs, null rate changes, distribution shifts
```

---

## CLI

```bash
fasteda run titanic.csv
fasteda compare train.csv test.csv
fasteda info data.csv
```

---

## What You Get

```
zedda v0.3.2
Scanning transaction_data.csv...

╭──── Dataset Overview  ⚡ SAMPLED ────────────────────────────╮
│ File:     transaction_data.csv                                │
│ ⚡ SAMPLED  2,000,000 of 6,362,620 rows (31.4%)               │
│            nulls/min/max exact from Parquet footer            │
│ Rows:     2,000,000                                           │
│ Cols:     31  (31 numeric, 0 string)                          │
│ Nulls:    0.0%  (0 cells)                                     │
│ Scanned:  32.3 sec                                            │
╰───────────────────────────────────────────────────────────────╯

Data Quality Score:  80/100  ████████░░  GOOD  (5 cols with outliers)

Column          Type   Nulls   Unique~     Mean          CI ±95%     Min    Max
step            int    0.0%    125         198           —           1      372
amount          float  0.0%    1,882,560   167,082.7     ±488.9      0      50,556,774
isFraud         int    0.0%    2           0.0007        —           0      1

Smart Warnings:
  ⚠  'amount' — max (50,556,774) is 303x above mean. Outliers likely.
  v  'isFraud' — binary column (0/1). Good ML target candidate.
  ... and 18 more. Call zd.warnings("data.csv") for full list.

Pearson Correlation Alerts:  (single-pass O(1) math)
  ↑↑ r=+1.00  'oldbalanceOrg' ↔ 'newbalanceOrig'   Drop one before ML training.
  ↑↑ r=+0.99  'newbalanceDest' ↔ 'expected_balance_change_dest'   Drop one before ML training.
  ... and 9 more pairs.
```

---

## How It Works

Zedda is built on a C++17 core connected to Python via nanobind.

### Core Algorithms

* **Welford's Online Algorithm** — numerically stable mean/variance/stddev/skewness/kurtosis in a single pass. No catastrophic cancellation on large datasets.
* **HyperLogLog** — cardinality estimation (unique value counts) with 99% accuracy using only 16KB per column — regardless of dataset size.
* **True Pearson Correlation** — $O(1)$ memory single-pass correlation engine. No second file read, no storing data. Exact $r$ value for every column pair.
* **Parquet Footer Cheat Code** — every Parquet file stores min, max, and null counts in its footer (last few KB). Zedda reads the footer first for instant exact stats — then samples only what's needed for mean/stddev.
* **Stratified Row-Group Sampling** — for large files, Zedda picks 6 representative row groups (start, middle, end) instead of reading everything. Result: 99.9% statistical accuracy, 100x less I/O.

### Architecture

```
User calls zd.profile("data.csv")
         │
         ▼
Python layer (zedda/__init__.py)
  • File validation + format detection
  • Sampling strategy selection
  • Rich terminal rendering
  • Smart Warnings + Correlation Alerts
         │
         ▼
nanobind bridge (bindings.cpp)
  • Zero-copy data transfer
  • Python ↔ C++ type mapping
         │
         ▼
C++ Core Engine
  • BS::thread_pool (multi-threaded chunk processing)
  • CsvStreamReader (RFC 4180, 64K row chunks)
  • ProfileBuilder (Welford + HLL + Pearson)
  • ArrowProfiler (zero-copy Parquet via Arrow C Data Interface)
         │
         ▼
DatasetProfile struct → returned to Python
```

---

## Memory Usage

Zedda uses $O(\text{columns})$ memory — not $O(\text{rows})$. This means:

| Dataset | pandas RAM | Zedda RAM |
| :--- | :--- | :--- |
| **1M rows, 10 cols** | ~800 MB | **~2 MB** |
| **10M rows, 30 cols** | ~8 GB | **~6 MB** |
| **1TB Parquet** | OOM | **~50 MB** |

This is possible because Zedda never loads the full dataset — it streams chunks and updates running accumulators (Welford, HLL) that stay constant size.

---

## Advanced Usage

### Force full scan (disable sampling)

```python
p = zd.scan("big_file.csv", sample_size=None)
```

### Custom sample size

```python
p = zd.scan("big_file.csv", sample_size=500_000)
```

### Access all warnings

```python
zd.warnings("data.csv")  # shows all, not truncated
```

### Access correlation results

```python
p = zd.scan("data.csv")
for corr in p.correlations:
    print(f"{corr.col_a} ↔ {corr.col_b}: r={corr.r:.3f} ({corr.strength})")
```

---

## Supported Formats

| Format | Support | Notes |
| :--- | :--- | :--- |
| **CSV** | ✅ Full | RFC 4180, custom delimiter, quote handling |
| **Parquet** | ✅ Full | Footer cheat code + zero-copy Arrow bridge |
| **Apache Arrow** | ✅ Full | Zero-copy via Arrow C Data Interface |
| **Excel (.xlsx)** | 🔜 Phase 5 | Coming soon |
| **JSON** | 🔜 Phase 5 | Coming soon |
| **PostgreSQL** | 🔜 Phase 5 | Coming soon |

---

## Benchmarks

Tested on MacBook Pro M2, 16GB RAM.

* **Dataset: Titanic (891 rows, 12 cols)**
  * pandas `describe()` : 0.8s
  * ydata-profiling : 42.0s
  * zedda : **0.019s** ← 2200x faster than ydata-profiling
* **Dataset: Fraud transactions (6.3M rows, 31 cols)**
  * pandas `describe()` : 8.2s (no insights, no correlation)
  * ydata-profiling : OOM on 8GB RAM
  * zedda (full scan) : 51.0s
  * zedda (sampled 2M) : **23.0s** ← with Smart Warnings + Pearson correlation
* **Dataset: 1TB Parquet (footer cheat code)**
  * pandas : OOM
  * ydata-profiling : OOM
  * zedda : **1.8s** ← exact nulls/min/max, sampled mean/std

---

## Roadmap

* [x] Phase 1 — C++ streaming core (Welford, HyperLogLog)
* [x] Phase 2 — Zero-copy Parquet + Arrow support
* [x] Phase 3 — Intelligent Sampling Engine (1TB in 2s)
* [x] Phase 3.1 — Smart Warnings, Data Quality Score, Pearson Correlation
* [ ] Phase 4 — `zd.ml_ready()` — ML readiness score + fix suggestions
* [ ] Phase 5 — `zd.compare()` — data drift detection for production
* [ ] Phase 6 — `zd.ask()` — natural language queries
* [ ] Phase 7 — SIMD (AVX-512) + mmap for physical I/O limits

---

## Contributing

Zedda is open source and actively maintained.

```bash
git clone https://github.com/prince3235/zedda.git --recursive
cd zedda
bash build.sh          # builds fasteda_core.pyd locally
python test_import.py  # verify build
```

PRs welcome! See `CONTRIBUTING.md` for guidelines.

---

## License

MIT License — see [LICENSE](file:///e:/one_pice/zedda/LICENSE) for details.

<div align="center">
  <p>Built with ❤️ and C++17</p>
  <p>
    <a href="https://pypi.org/project/zedda">PyPI</a> •
    <a href="https://github.com/prince3235/zedda">GitHub</a> •
    <a href="https://github.com/prince3235/zedda/issues">Issues</a>
  </p>
</div>
