Metadata-Version: 2.4
Name: sas2parquet
Version: 1.0.8
Summary: SAS → Parquet Hybrid Converter & Validator
License: MIT
License-File: LICENSE
Keywords: sas,parquet,etl,data,pyarrow
Author: Zaman Ziabakhshganji
Author-email: zaman.ganji@gmail.com
Requires-Python: >=3.11,<4.0
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Dist: narwhals (==2.13.0)
Requires-Dist: numpy (==2.3.5)
Requires-Dist: pandas (==2.3.3)
Requires-Dist: polars (==1.36.1)
Requires-Dist: pyarrow (==22.0.0)
Requires-Dist: pyreadstat (==1.3.2)
Requires-Dist: python-dateutil (==2.9.0.post0)
Requires-Dist: pytz (==2025.2)
Requires-Dist: requests (>=2.32.5,<3.0.0)
Requires-Dist: six (==1.17.0)
Requires-Dist: tzdata (==2025.2)
Project-URL: Homepage, https://ecda.eur.nl/erasmus-data-collaboratory/
Project-URL: Repository, https://git.ecda.ai/EDC-IXLab/sas2parquet
Description-Content-Type: text/markdown

# sas2parquet

[![PyPI version](https://badge.fury.io/py/sas2parquet.svg)](https://pypi.org/project/sas2parquet/)
[![Python versions](https://img.shields.io/pypi/pyversions/sas2parquet.svg)](https://pypi.org/project/sas2parquet/)
[![License](https://img.shields.io/pypi/l/sas2parquet.svg)](LICENCE)

**The ultimate SAS (.sas7bdat) to Parquet converter** — built to handle files that fail with standard tools.

`sas2parquet` automatically detects encodings, repairs schemas, infers correct data types, and performs **pixel-perfect validation** between SAS and Parquet outputs.

---

## ✨ Features

| Feature | Description |
|-------|-------------|
| 🔄 **Auto Encoding** | Detects UTF-8, Latin1, CP1252 from metadata or fallback |
| 🧠 **Smart Types** | Infers datetime, numeric, string with 20+ retry strategies |
| ✅ **Validation** | Chunk-by-chunk comparison (metadata, counts, values) |
| 📊 **Memory Safe** | Chunked processing (96GB RAM optimized, configurable) |
| 💾 **ZSTD Compression** | Level-6 ZSTD for efficient Parquet storage |
| 📝 **Detailed Logs** | Full conversion trace + mismatch reports |
| 🎯 **Two Modes** | Single file or recursive directory processing |

---

## 🚀 Quick Start

### Install
```bash
pip install sas2parquet
```

---

## ✅ Usage

### Convert a directory (recommended)

```bash
sas2parquet path/to/sasdata/
```

- Converts **all `.sas7bdat` files recursively**
- Creates `parquetdata/` and `logging/` next to `sasdata/`

---

### Convert a single file

```bash
sas2parquet path/to/file.sas7bdat
```

Output (default):
```text
path/to/file.parquet
```

---

### Specify output location

#### Directory mode — custom output directory
```bash
sas2parquet path/to/sasdata/ --out path/to/parquetdata/
```

#### File mode — custom output file
```bash
sas2parquet path/to/file.sas7bdat --out path/to/output.parquet
```

---

### Custom log directory (directory mode)

```bash
sas2parquet path/to/sasdata/ --log-dir path/to/logs/
```

---

## 📁 Directory Mode Behavior

```text
your-project/
├── sasdata/
│   ├── file1.sas7bdat
│   └── subfolder/
│       └── nested.sas7bdat
├── parquetdata/
│   ├── file1.parquet
│   └── subfolder_parquet/
│       └── nested.parquet
└── logging/
    └── conversion_20260205_1145.log
```

---

## 🛠️ CLI Reference

```bash
sas2parquet --help
```

---

## ⚙️ Configuration (Advanced)

Edit constants in:

```text
src/sas2parquet/convert.py
```

---


## 👥 Authors

This package was built collaboratively.
- Zaman Ziabakhshganji — creator and maintainer
- Farshad Radman — creator and maintainer
- Jos van Dongen — co-author and contributor

## 📄 License

MIT License
