Metadata-Version: 2.4
Name: lighter-data
Version: 0.1.0
Summary: Download Lighter (zkLighter) historical data — funding, candles, L2 book, trades — into Parquet, plus a live WebSocket book/trade recorder.
Project-URL: Homepage, https://github.com/bond-labs-dev/lighter-data
Project-URL: Issues, https://github.com/bond-labs-dev/lighter-data/issues
Author-email: bondlabs <hello@bondlabs.dev>
License-Expression: MIT
License-File: LICENSE
Keywords: funding,lighter,market-data,orderbook,perpetuals,zklighter
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Office/Business :: Financial :: Investment
Requires-Python: >=3.10
Requires-Dist: pyarrow>=14
Requires-Dist: websocket-client>=1.6
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.16; extra == 'dev'
Description-Content-Type: text/markdown

# lighter-data

[![ci](https://github.com/bond-labs-dev/lighter-data/actions/workflows/ci.yml/badge.svg)](https://github.com/bond-labs-dev/lighter-data/actions/workflows/ci.yml)

Pull [Lighter](https://lighter.xyz) (zkLighter) market data into Parquet:
funding history, candles, historical L2 book + trades — plus a live WebSocket
recorder for the data no archive sells yet. Everything is free: public REST /
WebSocket, no API key, no account.

Lighter is a young zk CLOB venue with 0%/0% maker/taker fees, perps *and* a
small set of USDC spot markets. There is no official historical-data tooling:
symbols are numeric market indices, history endpoints silently cap a single
response (~750 funding rows, 500 candles), and the funding rate arrives as an
unsigned percent-per-hour magnitude with the sign in a separate field. This
package deals with all of that and writes tidy, manifest-checked Parquet.

## Install

```bash
pip install lighter-data
# or, from a checkout:
pip install -e .
```

Python ≥3.10. Dependencies: `pyarrow`, `websocket-client` (the recorder);
all HTTP is stdlib.

## Quickstart

```bash
# full hourly funding history, per coin, paginated to the floor date
lighter-data funding --coins ETH SOL --root data

# hourly candles for the perp and the (young) USDC spot market
lighter-data candles --coins ETH --kinds perp spot --root data

# historical L2 + trades: Tardis.dev free first-of-month samples, no key
lighter-data tardis --coins XPL,EIGEN,JTO --dates 2026-05-01,2026-06-01 --root data

# live recorder: sampled top-20 book every 500ms + every trade
lighter-data record --coins WLD,ONDO,LINK --root data
```

`--coins` (and `tardis --dates`) accept space- and comma-separated lists
interchangeably on every subcommand.

## Datasets

| command | source | output | notes |
|---|---|---|---|
| `funding` | REST `/api/v1/fundings` | `funding/lighter/<coin>_perp/funding.parquet` | signed hourly fraction, HL-comparable |
| `candles` | REST `/api/v1/candles` | `candles/lighter/<coin>_<kind>/<tf>/candles.parquet` | perp + spot, 500-bar cap windowed |
| `tardis` | datasets.tardis.dev | `l2book/…/date=…/l2book.parquet`, `trades/…/trades.parquet` | first day of each month is free |
| `record` | WebSocket `/stream` | `l2book/…/date=…/part-*.parquet`, `trades/…/part-*.parquet` | top-20 @ 500ms, counterparty account ids |

Candle/funding partitions get a `manifest.json` sidecar (row count, bounds,
cadence, duplicate timestamps, largest gaps), so a half-month hole from a
broken resume shows up at download time, not as a silent bias in a backtest.

## The funding sign, pinned

A Lighter funding row is `{timestamp, value, rate, direction}` where `rate`
is a **non-negative percent-per-hour magnitude** and the sign lives in
`direction` (`"long"` = longs pay shorts). The written `rate` column is the
signed hourly **fraction**:

```
rate = rate_pct / 100 * (+1 if direction == "long" else -1)
```

verified live against `value / index_price`, and on the same scale as
Hyperliquid's `fundingRate` — so cross-venue carry math needs no adapter.
(Sister package: [hyperliquid-data](https://github.com/bond-labs-dev/hyperliquid-data);
the candle and funding schemas are identical across the two.)

## Caps and pagination

A single wide request against Lighter's history endpoints does not error — it
silently returns the most-recent slice (~750 funding rows, 500 candles),
which *looks like* the venue's whole history. The pullers paginate fixed
overlapping windows from a floor date (default `2025-01-01`, before the first
mainnet perp listing) and collapse the seam duplicates, so the written series
reaches the true listing date.

## The recorder

`lighter-data record` maintains each subscribed book from snapshot + deltas
and samples the top-20 every 500 ms (~20x smaller than storing every delta,
and the shape a fill-sim wants), records every trade with **both counterparty
account ids** (Lighter's public feed exposes them — enough to build
maker/taker cohorts offline), flushes Parquet parts every 10 min, and
survives disconnects with backoff. It is a long-running service: run it under
docker-compose/systemd with restart, not from a terminal you plan to close.

```
data/l2book/lighter/wld_perp/date=20260506/part-123005-000100.parquet
data/trades/lighter/wld_perp/date=20260506/part-123005-000100.parquet
```

## Library use

```python
from pathlib import Path

from lighter_data import resolve_perp_ids, fetch_all_funding, funding_rows, write_funding

mid = resolve_perp_ids()["ETH"]
pairs = fetch_all_funding(mid, start_s=1735689600, end_s=1754377200)
write_funding(funding_rows("ETH", pairs), root=Path("data"), instrument="ETH-PERP")
```

The supported surface is `lighter_data.__all__` (pinned by
`tests/test_public_api.py`); schemas are typed in `lighter_data.types` and
pinned as pyarrow schemas in `lighter_data.parquet`.

## License

MIT.
