Metadata-Version: 2.4
Name: redditdumps
Version: 0.1.0
Summary: Lightweight utilities for processing Reddit data dumps in ZST format
Project-URL: Homepage, https://github.com/pssachdeva/redditdumps
Project-URL: Repository, https://github.com/pssachdeva/redditdumps
Author-email: Pratik Sachdeva <pratik.sachdeva@berkeley.edu>
License-Expression: MIT
License-File: LICENSE
Keywords: data-processing,pushshift,reddit,zstandard
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.12
Requires-Dist: pandas>=1.5.0
Requires-Dist: tqdm>=4.64.0
Requires-Dist: typer>=0.9.0
Requires-Dist: zstandard>=0.21.0
Provides-Extra: dev
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Description-Content-Type: text/markdown

# redditdumps

Lightweight Python utilities for processing Reddit data dumps in ZST format.

These dumps are commonly found on Academic Torrents (Pushshift archives) and contain newline-delimited JSON compressed with Zstandard.

## Installation

```bash
uv add redditdumps
```

Or with pip:

```bash
pip install redditdumps
```

## Usage

### Read a ZST file into a DataFrame

```python
import redditdumps as rd

# Read entire file
df = rd.read_zst("RC_2024-01.zst")

# Filter by subreddit
df = rd.read_zst("RC_2024-01.zst", subreddit="science")

# Filter by multiple subreddits
df = rd.read_zst("RC_2024-01.zst", subreddit=["science", "askscience"])

# Select specific columns
df = rd.read_zst("RC_2024-01.zst", columns=["author", "body", "score"])

# Combine filters
df = rd.read_zst(
    "RC_2024-01.zst",
    subreddit="python",
    columns=rd.MINIMAL_COMMENT_COLUMNS,
    max_lines=100000,
)
```

### Inspect file schema

```python
# Discover columns in a file
schema = rd.inspect_schema("RC_2024-01.zst", sample_size=1000)
for col, info in schema.items():
    print(f"{col}: {info['type']} ({info['count']} records)")
```

### Built-in column schemas

```python
# Common column sets for convenience
rd.COMMENT_COLUMNS       # All standard comment fields
rd.SUBMISSION_COLUMNS    # All standard submission fields
rd.MINIMAL_COMMENT_COLUMNS   # Lightweight subset for comments
rd.MINIMAL_SUBMISSION_COLUMNS  # Lightweight subset for submissions
```

## File naming conventions

- `RC_*.zst` - Reddit Comments
- `RS_*.zst` - Reddit Submissions
