Metadata-Version: 2.4
Name: streamwright
Version: 0.1.12
Summary: Declarative data connectors — pull from APIs, files and databases with YAML, shape with SQL, write anywhere.
Author-email: Karthick Jaganathan <kjaganathan.sde@gmail.com>
License: Apache-2.0
Project-URL: Homepage, https://streamwright.web.app
Project-URL: Documentation, https://streamwright.web.app/docs/
Project-URL: Repository, https://github.com/karthick-jaganathan/streamwright
Project-URL: Issue Tracker, https://github.com/karthick-jaganathan/streamwright/issues
Keywords: data,pipeline,etl,connector,singer,yaml,duckdb,rest-api
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml==6.0.2
Requires-Dist: requests<3,>=2.25
Requires-Dist: duckdb<2.1,>=2.0
Requires-Dist: pytz
Provides-Extra: dlt
Requires-Dist: dlt>=1.0; extra == "dlt"
Dynamic: license-file

# StreamWright

[![PyPI](https://img.shields.io/pypi/v/streamwright.svg)](https://pypi.org/project/streamwright/)
[![Python](https://img.shields.io/pypi/pyversions/streamwright.svg)](https://pypi.org/project/streamwright/)
[![License](https://img.shields.io/pypi/l/streamwright.svg)](https://github.com/karthick-jaganathan/streamwright/blob/master/LICENSE)
[![Docs](https://img.shields.io/badge/docs-streamwright.web.app-brightgreen.svg)](https://streamwright.web.app/docs/)

Declarative data pipelines in YAML: read from APIs, files, databases and ads platforms, shape with DuckDB SQL,
write to files, DuckDB, DuckLake, Singer or dlt.

- **No per-connector code**: authentication, paging, retries, rate limits and incremental state are configuration.
- **Connectors on demand** from the StreamWright hub: `sw connectors -i <name>`.
- **Jobs**: `kind: DuckDBJob` SQL pipelines with replay-safe writes; `kind: SparkJob` with
  [streamwright-spark](https://pypi.org/project/streamwright-spark/).

## Install

```bash
pip install streamwright
sw connectors -i restapi
```

## Example

A source is a folder: `source.yaml` and one file per stream.

```yaml
# my_api/source.yaml
kind: source
name: my_api
spec:
  secrets:
    api_token: {type: string}
auth:
  provider: restapi
  type: bearer
  token: "{{ secrets.api_token }}"
http:
  base_url: https://api.example.com
```

```yaml
# my_api/streams/items.yaml
requests:
  - name: raw_items
    http: {path: /v1/items}
transform:
  mode: page
  steps:
    - name: items
      select: SELECT record->>'id' AS id, record->>'name' AS name FROM raw_items
export:
  items: {step: items, primary_key: [id]}
```

```bash
sw validate my_api
STREAMWRIGHT_SECRET_API_TOKEN=... sw run my_api --output jsonl:out
```

## Links

[Documentation](https://streamwright.web.app/docs/) ·
[Quickstart](https://streamwright.web.app/docs/quickstart/) ·
[Source code](https://github.com/karthick-jaganathan/streamwright)
