Metadata-Version: 2.4
Name: dlt-community-sources
Version: 0.7.20
Summary: Community-maintained dlt sources for various APIs
Project-URL: Homepage, https://github.com/ota2000/dlt-community-sources
Project-URL: Repository, https://github.com/ota2000/dlt-community-sources
Project-URL: Documentation, https://github.com/ota2000/dlt-community-sources#available-sources
Project-URL: Changelog, https://github.com/ota2000/dlt-community-sources/releases
Project-URL: Issues, https://github.com/ota2000/dlt-community-sources/issues
Author: ota2000
License: MIT
License-File: LICENSE
Keywords: api,data-loading,data-pipeline,dlt,elt,etl
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.12
Requires-Dist: dlt>=1.0.0
Provides-Extra: app-store-connect
Requires-Dist: pyjwt[crypto]>=2.8.0; extra == 'app-store-connect'
Provides-Extra: meta-ads
Provides-Extra: microsoft-ads
Requires-Dist: msal>=1.28.0; extra == 'microsoft-ads'
Provides-Extra: nextdns
Provides-Extra: tiktok-ads
Provides-Extra: twilio
Provides-Extra: yahoo-ads-search
Description-Content-Type: text/markdown

# dlt-community-sources

[![CI](https://github.com/ota2000/dlt-community-sources/actions/workflows/ci.yaml/badge.svg)](https://github.com/ota2000/dlt-community-sources/actions/workflows/ci.yaml)
[![PyPI](https://img.shields.io/pypi/v/dlt-community-sources)](https://pypi.org/project/dlt-community-sources/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)

Community-maintained [dlt](https://dlthub.com/) sources for various APIs. Install only what you need via extras.

## Available Sources

| Source | Extra | Description |
|---|---|---|
| [App Store Connect](dlt_community_sources/app_store_connect/) | `app-store-connect` | Apple App Store Connect API |
| [Twilio](dlt_community_sources/twilio/) | `twilio` | Twilio SMS, Calls, Usage Records |
| [Meta Ads](dlt_community_sources/meta_ads/) | `meta-ads` | Meta (Facebook) Ads campaigns, ad sets, ads, and insights |
| [NextDNS](dlt_community_sources/nextdns/) | `nextdns` | NextDNS logs and analytics |
| [Microsoft Ads](dlt_community_sources/microsoft_ads/) | `microsoft-ads` | Microsoft Advertising (Bing Ads) with certificate auth support |
| [TikTok Ads](dlt_community_sources/tiktok_ads/) | `tiktok-ads` | TikTok Marketing API (Business API) |
| [Yahoo Ads Search](dlt_community_sources/yahoo_ads_search/) | `yahoo-ads-search` | Yahoo Japan Ads Search API (SS) with MCC support |

## Installation

```bash
pip install dlt-community-sources[app-store-connect]
```

Or with uv:

```bash
uv add dlt-community-sources[app-store-connect]
```

## Usage

```python
import dlt
from dlt_community_sources.twilio import twilio_source

pipeline = dlt.pipeline(
    pipeline_name="twilio",
    destination="duckdb",
    dataset_name="twilio_data",
)

source = twilio_source(
    account_sid="your_account_sid",
    api_key_sid="your_api_key_sid",
    api_key_secret="your_api_key_secret",
    resources=["messages", "calls"],
)

load_info = pipeline.run(source)
print(load_info)
```

See each source's README for detailed usage and authentication options.

## Features

All sources share these common features:

- Incremental loading where applicable
- Automatic token/auth refresh
- Rate limit handling with exponential backoff
- Works with any [dlt destination](https://dlthub.com/docs/dlt-ecosystem/destinations/)

## Error handling

Sources distinguish two kinds of resources:

- **Auxiliary resources** (metadata: campaigns, ads, audiences, ...) skip
  expected client errors (HTTP 400/403/404) per endpoint — some APIs return
  4xx for valid requests on accounts without certain features. Every skip is
  logged with the response body.
- **Primary resources** (the fact data a source exists for: `report`,
  `insights`) never skip. Failures raise `PrimaryResourceTerminalError`
  (request rejected — retrying will not help; 4xx other than 408/429) or
  `PrimaryResourceTransientError` (timeout, rate limit, provider-side job
  failure), so a load never "succeeds" with silently missing data. Both mix
  in dlt's `TerminalException` / `TransientException`. Note that dlt wraps
  resource exceptions in `PipelineStepFailed` / `ResourceExtractionError`,
  so retry helpers that inspect only one level (such as
  `dlt.pipeline.helpers.retry_load`) cannot see the classification — use
  `contains_terminal_exception`, which walks the exception chain:

```python
from tenacity import Retrying, retry_if_exception, stop_after_attempt
from dlt_community_sources._utils import contains_terminal_exception

for attempt in Retrying(
    stop=stop_after_attempt(3),
    retry=retry_if_exception(lambda e: not contains_terminal_exception(e)),
    reraise=True,
):
    with attempt:
        pipeline.run(source)
```

To load a subset of resources, either pass the source-specific `resources`
argument or use dlt's native selection: `source.with_resources("report")`.

## Development

```bash
# Fork the repository first, then:
git clone https://github.com/YOUR_USERNAME/dlt-community-sources.git
cd dlt-community-sources
uv sync --group dev
uv run pytest -v
uv run ruff check .
```

## AI-assisted development

This repository is set up for AI coding assistants with two layers of AI context:

- **Project rules** (`.ai/rules.md`) — coding conventions, testing patterns, and structure specific to this repo
- **[dltHub AI workbench](https://dlthub.com/context/)** — dlt ecosystem knowledge, MCP server, and guided skills for pipeline development

Both are synced to tool-specific locations (Claude Code, Cursor, Codex, etc.) via a single command:

```bash
bash scripts/sync-ai-rules.sh
```

The dltHub workbench provides an MCP server for source search (9,700+ APIs), pipeline inspection, and data exploration — plus skills that guide you through the full pipeline development workflow. See [CONTRIBUTING.md](CONTRIBUTING.md#ai-assisted-development-recommended) for details.

Edit `.ai/` only for project rules, then re-run the sync script. CI checks that files are in sync.

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md).

## License

MIT
