Metadata-Version: 2.4
Name: qfin-datasets
Version: 0.1.0
Summary: A helper library to manage all kinds of datasets.
Project-URL: Homepage, https://github.com/siddharthskulkarni/datasets
Project-URL: Issues, https://github.com/siddharthskulkarni/datasets/issues
Author: Siddharth Kulkarni
License-Expression: MIT
License-File: LICENSE
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX
Classifier: Operating System :: Unix
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.9
Requires-Dist: numpy>=1.23
Provides-Extra: data
Requires-Dist: pandas>=2.0; extra == 'data'
Requires-Dist: pyarrow>=14.0; extra == 'data'
Requires-Dist: python-dotenv>=1.0; extra == 'data'
Requires-Dist: requests>=2.31; extra == 'data'
Requires-Dist: yfinance>=0.2; extra == 'data'
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Requires-Dist: twine>=5.0; extra == 'dev'
Provides-Extra: viz
Requires-Dist: dash>=2.14; extra == 'viz'
Requires-Dist: pandas>=2.0; extra == 'viz'
Requires-Dist: plotly>=5.0; extra == 'viz'
Description-Content-Type: text/markdown

# datasets

A helper library to manage all kinds of datasets. Part of the QFIN workspace alongside `equity`, `derivatives`, `fixed-income`, and `risk`.

## Install

```bash
python3 -m pip install -e .
```

Optional extras:

```bash
python3 -m pip install -e '.[data]'   # pandas, pyarrow, requests, python-dotenv, yfinance
python3 -m pip install -e '.[dev]'    # pytest, ruff, build
```

Massive providers require the `[data]` extra and `MASSIVE_API_KEY` in `.env` (see `.env.example`).
FRED providers also need `FRED_API_KEY`.

## DataSource and Dataset

- **DataSource** — a provider/backend (FRED, Massive, Treasury.gov, local CSV).
- **Dataset** — one specific dataset obtainable from that source, identified by a `key`.

Sources expose one or more dataset keys via `source.datasets()`. Use `Dataset(source, key)` as the handle for fetching a single dataset:

```python
from datetime import date

from datasets.data import Dataset, MassiveDailyMarketSummaryRangeSource

source = MassiveDailyMarketSummaryRangeSource(date(2024, 1, 1), date(2024, 1, 5))
summary = Dataset(source, source.datasets()[0]).fetch()
```

`Dataset` also supports export-ready DataFrames and parquet export:

```python
frame = Dataset(source, source.datasets()[0]).to_dataframe()
path = Dataset(source, source.datasets()[0]).export()
```

## Core types

| Type | Description |
|------|-------------|
| `OhlcvBar` / `OhlcvHistory` | Time-series OHLCV bars for any symbol (equity, index, option contract) |
| `MarketBar` / `MarketSummary` | Cross-sectional daily market snapshot |
| `OptionContract` / `OptionContracts` | Options reference data |
| `TreasuryYieldPoint` / `TreasuryYieldCurve` | U.S. Treasury constant-maturity yields (Massive) |
| `CurvePoint` / `ParCurve` | Treasury.gov par yield curve |
| `RateObservation` / `RateSeries` | Rate fixings (SOFR, FRED series) |
| `FuturesSettle` / `FuturesSettleCurve` / `SofrFuturesSettleBundle` | CME SOFR futures settles |

Legacy names (`StockBar`, `StockHistory`, `DailyMarketBar`, `DailyMarketSummary`) remain available as deprecated aliases.

## Rates market data (FRED / NY Fed / Treasury.gov / CME)

| Source | Backend |
|--------|---------|
| `TreasuryParCurveSource` | Treasury.gov daily par yield CSV |
| `NyFedSofrSource` | NY Fed Markets API SOFR |
| `FredSeriesSource` / `FredSofrSource` / … | FRED API (`FRED_API_KEY`) |
| `CmeSofrSettleCsvSource` / `CmeSofrSettleBundleSource` | Local CME settle CSVs under `data/raw/` |

```python
from datetime import date
from datasets.data import NyFedSofrSource, TreasuryParCurveSource

par = TreasuryParCurveSource().fetch()
sofr = NyFedSofrSource().fetch(as_of=date(2026, 5, 22))
```

See [docs/data_access.md](docs/data_access.md). Ingest:

```bash
python3 scripts/ingest_market_data.py --as-of 2026-05-22
```

## Massive.com providers

| Source | Endpoint | Default window |
|--------|----------|----------------|
| `MassiveDailyMarketSummarySource` | [Daily Market Summary](https://massive.com/docs/rest/stocks/aggregates/daily-market-summary) | One trading date |
| `MassiveStockHistorySource` | `/v2/aggs/ticker/{ticker}/range/1/day/...` | 5 calendar years |
| `MassiveOptionContractsSource` | [All Contracts](https://massive.com/docs/rest/options/contracts/all-contracts) | Filtered contract index |
| `MassiveOptionBarsSource` | [Custom Bars](https://massive.com/docs/rest/options/aggregates/custom-bars) | 1 calendar year |
| `MassiveTreasuryYieldsSource` | `/fed/v1/treasury-yields` | 30 calendar days |

**Rate limits:** default client enforces **5 calls/min** (free tier). Downloads print an ETA from pending call count.

| Download | API calls | ~Time at 5/min |
|----------|-----------|----------------|
| 2-year daily market summary | ~504 weekdays | ~100 min |
| N tickers (5-year history) | N calls | N / 5 min |
| Option contracts (per underlying) | 1+ paginated calls | varies |
| Option bars (per contract) | 1 call | 12 sec |

### Options example

```python
from datetime import date

from datasets.data import Dataset, MassiveOptionBarsSource, MassiveOptionContractsSource

contracts_src = MassiveOptionContractsSource(underlying_ticker="AAPL", expired=False)
contracts = Dataset(contracts_src, contracts_src.datasets()[0]).fetch(as_of=date.today())

contract = contracts.contracts[0]
bars_src = MassiveOptionBarsSource(contract, lookback_days=90)
bars = bars_src.fetch()
frame = Dataset(bars_src, bars_src.datasets()[0]).to_dataframe()
```

### Treasury yields example

```python
from datetime import date

from datasets.data import Dataset, MassiveTreasuryYieldsSource

treasury_src = MassiveTreasuryYieldsSource(lookback_days=30)
curve = Dataset(treasury_src, treasury_src.datasets()[0]).fetch(as_of=date.today())
frame = Dataset(treasury_src, treasury_src.datasets()[0]).to_dataframe(as_of=date.today())
```

## Yahoo Finance providers

| Source | Endpoint | Default window |
|--------|----------|----------------|
| `YahooIndexHistorySource` | `yfinance.Ticker(symbol).history(interval="1d")` | 5 calendar years |

**Rate limits:** default client enforces **30 calls/min** for bulk downloads to reduce throttling risk.

### Download scripts

Edit the config block at the top of each script, then run:

```bash
cd datasets
python3 -m pip install -e '.[data]'

# Dry run (no network)
python3 scripts/download_daily_market_summary.py --dry-run
python3 scripts/download_stock_history.py --dry-run
python3 scripts/download_sector_index_history.py --dry-run
python3 scripts/download_option_contracts.py --dry-run
python3 scripts/download_treasury_yields.py --dry-run

# Full download (requires MASSIVE_API_KEY)
python3 scripts/download_daily_market_summary.py --lookback-years 2
python3 scripts/download_stock_history.py
python3 scripts/download_option_contracts.py --underlying SPY
python3 scripts/download_option_contracts.py --underlyings AAPL,MSFT
python3 scripts/download_stock_history.py --tickers AAPL,MSFT
python3 scripts/download_treasury_yields.py

# Full Yahoo download (no API key required)
python3 scripts/download_sector_index_history.py --lookback-years 10
```

**Equity options (AAPL, MSFT):** listed contracts are American-style. Use `EQUITY_OPTION_UNDERLYINGS` / `equity_option_tickers()` from `datasets.data`. Downstream BSM/IV in `derivatives` treats them as a European approximation for empirical study.

Outputs:

- `data/processed/massive/daily_market_summary/{YYYY-MM-DD}.parquet`
- `data/processed/massive/stock_history/{SYMBOL}.parquet`
- `data/processed/massive/option_contracts/{UNDERLYING}_{YYYY-MM-DD}.parquet`
- `data/processed/massive/option_bars/{CONTRACT_TICKER}.parquet`
- `data/processed/massive/treasury_yields/{YYYY-MM-DD}.parquet`
- `data/processed/yahoo/index_history/{slug}.parquet`

`download_stock_history.py` uses a `TICKERS` list at the top of the file (override with `--tickers SPY,AAPL`).
`download_option_contracts.py` accepts `--underlying SPY` or `--underlyings AAPL,MSFT`.
`download_sector_index_history.py` defaults to S&P 500 (`^GSPC`) plus 11 S&P 500 GICS sector indices, and supports overriding via `--slugs sp500,energy,financials`.

## Quickstart

```bash
cd datasets
python3 -m pip install -e .
python3 examples/quickstart.py
```

Notebook example for S&P 500 and sector returns:

```bash
python3 -m pip install -e '.[data]'
python3 scripts/download_sector_index_history.py --lookback-years 10
jupyter notebook examples/sp500_10y_sector_returns.ipynb
```

```python
import datasets
from datasets.data import project_data_dir

print(datasets.__version__, project_data_dir())
```

## Project layout

```
datasets/
├── datasets/               # Python package
│   └── data/
│       ├── massive/        # Massive REST client and sources
│       ├── export.py       # Parquet writers
│       └── ...
├── data/
│   ├── raw/              # Manual CSV fallbacks (gitignored)
│   └── processed/        # Parquet snapshots (gitignored)
├── examples/             # Runnable example scripts and notebooks
├── scripts/              # Build and download helpers
└── tests/
```

## Development

```bash
bash scripts/build_test.sh
```
