Metadata-Version: 2.4
Name: zerodha-data-fetcher
Version: 1.2.0
Summary: A Python package for fetching historical data from Zerodha API Using Account Credentials.
Author-email: Jayam Gupta <guptajayam47@gmail.com>
Maintainer-email: Jayam Gupta <guptajayam47@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/JayceeGupta/Zerodha-Data-Fetcher
Project-URL: Repository, https://github.com/JayceeGupta/Zerodha-Data-Fetcher
Project-URL: Documentation, https://github.com/JayceeGupta/Zerodha-Data-Fetcher#readme
Project-URL: Bug Tracker, https://github.com/JayceeGupta/Zerodha-Data-Fetcher/issues
Keywords: zerodha,trading,data,api,historical,stock-market
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Office/Business :: Financial
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.25.0
Requires-Dist: pandas>=1.3.0
Requires-Dist: python-dateutil>=2.8.0
Requires-Dist: pyotp>=2.6.0
Requires-Dist: keyring>=23.0.0
Requires-Dist: cryptography>=3.4.0
Requires-Dist: Brotli>=1.0.9
Requires-Dist: python-dotenv>=0.19.0
Requires-Dist: platformdirs>=3.0.0
Provides-Extra: dev
Requires-Dist: pytest>=6.0; extra == "dev"
Requires-Dist: pytest-cov>=2.0; extra == "dev"
Requires-Dist: black>=21.0; extra == "dev"
Requires-Dist: flake8>=3.9; extra == "dev"
Requires-Dist: mypy>=0.910; extra == "dev"
Requires-Dist: pre-commit>=2.15; extra == "dev"
Provides-Extra: docs
Requires-Dist: sphinx>=4.0; extra == "docs"
Requires-Dist: sphinx-rtd-theme>=1.0; extra == "docs"
Requires-Dist: myst-parser>=0.15; extra == "docs"
Dynamic: license-file

# Zerodha Data Fetcher

[![PyPI version](https://img.shields.io/pypi/v/zerodha-data-fetcher)](https://pypi.org/project/zerodha-data-fetcher/)
[![Python versions](https://img.shields.io/pypi/pyversions/zerodha-data-fetcher)](https://pypi.org/project/zerodha-data-fetcher/)
[![Tests](https://github.com/JayceeGupta/Zerodha-Data-Fetcher/actions/workflows/test.yml/badge.svg)](https://github.com/JayceeGupta/Zerodha-Data-Fetcher/actions/workflows/test.yml)
[![License](https://img.shields.io/github/license/JayceeGupta/Zerodha-Data-Fetcher)](LICENSE)
[![Downloads](https://img.shields.io/pypi/dm/zerodha-data-fetcher)](https://pypi.org/project/zerodha-data-fetcher/)

Python package for fetching historical market data from Zerodha's Kite web APIs with built-in authentication, rate limiting, instrument lookup, caching, and multi-account workflows.

## Why Zerodha Data Fetcher?

Zerodha's Kite platform provides powerful market data, but programmatic access to historical data requires navigating authentication flows, managing rate limits, and handling session tokens. This library handles all of that so you can focus on analysis:

- **No official API key needed** -- authenticates through the Kite web interface using your existing account credentials
- **Handles the full auth flow** -- password login, TOTP 2FA, encrypted token caching in your OS keyring
- **Production-grade reliability** -- automatic retries, rate limiting, and configurable chunk failure modes
- **Zero-config symbol resolution** -- pass a symbol name like `"RELIANCE"` instead of memorizing instrument tokens
- **Parallel fetching** -- splits large date ranges into chunks and fetches them concurrently

## Table of Contents

- [Features](#features)
- [Prerequisites](#prerequisites)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [TOTP Setup](#totp-setup)
- [Configuration](#configuration)
- [Runtime Configuration Override](#runtime-configuration-override)
- [Historical Fetch Failure Modes](#historical-fetch-failure-modes)
- [Logging](#logging)
- [Multi-Account Parallel Processing](#multi-account-parallel-processing)
- [Instrument Data Caching](#instrument-data-caching)
- [API Reference](#api-reference)
- [Error Handling](#error-handling)
- [Development](#development)
- [Contributing](#contributing)
- [Security](#security)
- [License](#license)
- [Disclaimer](#disclaimer)
- [Support](#support)

## Features

- Fast parallel historical data fetching with request pacing
- Automatic authentication flow with password, TOTP, and token handling
- Support for instrument tokens and symbol-based lookups
- Strict-by-default chunk failure handling for correctness-sensitive workloads
- Runtime overrides for environment-based configuration
- Instrument metadata search and caching
- Multi-account patterns for high-volume retrieval
- Structured logging with separate console and file formats

## Prerequisites

Before using the package, make sure you have:

- Python 3.8 or newer
- A Zerodha account with Kite access
- TOTP enabled on that account
- `pip` or `uv` available in your environment

## Installation

```bash
pip install zerodha-data-fetcher
```

For development:

```bash
uv sync --all-extras
```

## Quick Start

### TOTP Setup

Set up TOTP for your Zerodha account before using the package:

[Zerodha TOTP Setup Guide](https://support.zerodha.com/category/trading-and-markets/general-kite/login-credentials-of-trading-platforms/articles/time-based-otp-setup)

### Environment Variables

Create a `.env` file in your project root:

```env
# Required credentials
ZERODHA_USER_ID=your_user_id
ZERODHA_PASSWORD=your_password
ZERODHA_TOTP_SECRET=your_totp_secret

# Optional configurations
ZERODHA_TYPE=user_id
ZERODHA_BASE_URL=https://kite.zerodha.com
ZERODHA_LOGIN_URL=https://kite.zerodha.com/api/login
ZERODHA_2FA_URL=https://kite.zerodha.com/api/twofa
ZERODHA_HISTORICAL_URL=https://kite.zerodha.com/oms/instruments/historical/{token}/{timeframe}?user_id={userid}&oi=1&from={current_date}&to={next_date}
ZERODHA_KEYRING_TOKEN_KEY=zerodha_auth_token
ZERODHA_KEYRING_ENCRYPTION_KEY=zerodha_encryption_key
ZERODHA_INSTRUMENT_CACHE_TTL=1440
```

### Basic Usage

```python
from datetime import date, timedelta

from zerodha_data_fetcher import ZerodhaDataFetcher, setup_logging

setup_logging(log_level="INFO", log_file="logs/zerodha_fetcher.log")

fetcher = ZerodhaDataFetcher(
    requests_per_second=3,
    chunk_failure_mode="strict",
)

end_date = date.today()
start_date = end_date - timedelta(days=30)

data = fetcher.fetch_historical_data(
    ticker_token=408065,
    start_date=start_date,
    end_date=end_date,
    timeframe="minute",
)

print(f"Retrieved {len(data)} records")
print(data.head())

reliance_data = fetcher.fetch_historical_data(
    ticker_token="RELIANCE",
    start_date=start_date,
    end_date=end_date,
    timeframe="minute",
)

print(f"Retrieved {len(reliance_data)} records for RELIANCE")
print(reliance_data.head())

search_results = fetcher.search_symbols("TATA", limit=5)
print(search_results[["Instrument_Token", "Name", "Exchange"]])

instrument_info = fetcher.get_instrument_info("INFY")
print(instrument_info)
```

Authentication logs are sanitized by default. TOTP values, raw auth response bodies, headers, cookies, and encrypted token material are never written to the logs.

## Configuration

### Environment Variables

| Variable | Description | Required | Default |
| -------- | ----------- | -------- | ------- |
| `ZERODHA_USER_ID` | Zerodha user ID | Yes | - |
| `ZERODHA_PASSWORD` | Zerodha password | Yes | - |
| `ZERODHA_TOTP_SECRET` | TOTP secret for 2FA | Yes | - |
| `ZERODHA_TYPE` | Account type (`user_id` or `corporate`) | No | `user_id` |
| `ZERODHA_BASE_URL` | Zerodha base URL | No | `https://kite.zerodha.com` |
| `ZERODHA_LOGIN_URL` | Login endpoint URL | No | `https://kite.zerodha.com/api/login` |
| `ZERODHA_2FA_URL` | 2FA endpoint URL | No | `https://kite.zerodha.com/api/twofa` |
| `ZERODHA_HISTORICAL_URL` | Historical API URL template | No | Built-in template |
| `ZERODHA_KEYRING_TOKEN_KEY` | Keyring token storage key | No | `zerodha_auth_token` |
| `ZERODHA_KEYRING_ENCRYPTION_KEY` | Keyring encryption key | No | `zerodha_encryption_key` |
| `ZERODHA_INSTRUMENT_CACHE_TTL` | Instrument cache TTL in minutes | No | `1440` |

Only `ZERODHA_USER_ID`, `ZERODHA_PASSWORD`, and `ZERODHA_TOTP_SECRET` are required for normal use.

### Class Parameters

- `requests_per_second`: API rate limit, clamped to the library's supported range
- `token_expiry_hours`: Token validity period
- `cache_ttl_minutes`: Instrument cache TTL in minutes
- `chunk_failure_mode`: `"strict"` or `"partial"`
- `user_id`: Runtime override for `ZERODHA_USER_ID`
- `password`: Runtime override for `ZERODHA_PASSWORD`
- `totp_secret`: Runtime override for `ZERODHA_TOTP_SECRET`
- `user_type`: Runtime override for `ZERODHA_TYPE`

## Runtime Configuration Override

You can override environment variables during class instantiation:

```python
fetcher = ZerodhaDataFetcher(
    requests_per_second=3,
    chunk_failure_mode="strict",
    user_id="override_user_id",
    password="override_password",
    totp_secret="override_totp_secret",
    user_type="user_id",
)
```

This is useful when credentials or account context need to be supplied at runtime instead of through a local `.env` file.

## Historical Fetch Failure Modes

Historical data fetching defaults to strict correctness.

- `chunk_failure_mode="strict"`: any failed chunk raises `DataFetchError` and no partial data is returned.
- `chunk_failure_mode="partial"`: successful chunks are returned, failed ranges are summarized in a warning log, and an exception is raised only if all chunks fail.

You can set the default on the fetcher instance or override it per call:

```python
fetcher = ZerodhaDataFetcher(chunk_failure_mode="strict")

strict_data = fetcher.fetch_historical_data(
    "INFY",
    start_date=start_date,
    end_date=end_date,
)

partial_data = fetcher.fetch_historical_data(
    "INFY",
    start_date=start_date,
    end_date=end_date,
    chunk_failure_mode="partial",
)
```

## Logging

`setup_logging()` uses separate defaults for console and file output.

- Console output is compact and human-readable, for example:

```text
(20:30:29) - [INFO] - zerodha_data_fetcher.core.data_fetcher fetch_historical_data: Data fetch completed successfully
```

- File output includes timestamp, logger, function, line number, and thread name.

Customize each handler independently:

```python
setup_logging(
    log_level="INFO",
    log_file="logs/zerodha_fetcher.log",
    console_format="%(asctime)s %(levelname)s %(message)s",
    file_format="%(asctime)s | %(levelname)s | %(name)s | %(funcName)s:%(lineno)d | %(threadName)s | %(message)s",
)
```

Behavior notes:

- `log_format=...` still overrides both handlers
- `console_format` and `file_format` can be set independently when `log_format` is not provided
- Default console dates use `%H:%M:%S`
- Default file dates use `%Y-%m-%d %H:%M:%S`

## Multi-Account Parallel Processing

For large data retrieval jobs, you can create multiple fetcher instances with different credentials to spread work across accounts.

```python
from concurrent.futures import ThreadPoolExecutor
from datetime import date, timedelta

from zerodha_data_fetcher import ZerodhaDataFetcher

accounts = [
    {
        "user_id": "account1_id",
        "password": "account1_password",
        "totp_secret": "account1_totp_secret",
    },
    {
        "user_id": "account2_id",
        "password": "account2_password",
        "totp_secret": "account2_totp_secret",
    },
    {
        "user_id": "account3_id",
        "password": "account3_password",
        "totp_secret": "account3_totp_secret",
    },
]


def fetch_data_with_account(account_config, symbols_batch):
    fetcher = ZerodhaDataFetcher(
        requests_per_second=3,
        user_id=account_config["user_id"],
        password=account_config["password"],
        totp_secret=account_config["totp_secret"],
    )

    results = {}
    end_date = date.today()
    start_date = end_date - timedelta(days=30)

    for symbol in symbols_batch:
        try:
            data = fetcher.fetch_historical_data(
                ticker_token=symbol,
                start_date=start_date,
                end_date=end_date,
                timeframe="minute",
            )
            results[symbol] = data
            print(f"Fetched {len(data)} records for {symbol} on {account_config['user_id']}")
        except Exception as exc:
            print(f"Failed to fetch {symbol} on {account_config['user_id']}: {exc}")
            results[symbol] = None

    return results


all_symbols = [
    "RELIANCE",
    "HDFC",
    "INFY",
    "TCS",
    "ICICIBANK",
    "SBIN",
    "BAJFINANCE",
    "BHARTIARTL",
    "HDFCBANK",
]

symbols_per_account = len(all_symbols) // len(accounts)
symbol_batches = [
    all_symbols[i:i + symbols_per_account]
    for i in range(0, len(all_symbols), symbols_per_account)
]

if len(symbol_batches) > len(accounts):
    symbol_batches[-2].extend(symbol_batches[-1])
    symbol_batches.pop()

with ThreadPoolExecutor(max_workers=len(accounts)) as executor:
    futures = []
    for i, account in enumerate(accounts):
        if i < len(symbol_batches):
            futures.append(executor.submit(fetch_data_with_account, account, symbol_batches[i]))

    all_results = {}
    for future in futures:
        all_results.update(future.result())

print(f"Completed fetching data for {len(all_results)} symbols across {len(accounts)} accounts")
```

## Instrument Data Caching

The package bundles a snapshot of Zerodha's instrument list and keeps a fresh copy in your local cache directory.

### How It Works

- On first use, the package attempts to download the latest instrument data from `https://api.kite.trade/instruments`.
- If the cached file is younger than the TTL, no download is attempted.
- If the download fails, the bundled snapshot is used as a fallback and a warning is logged.

### Configuration

| Method | Example |
| ------ | ------- |
| Environment variable | `ZERODHA_INSTRUMENT_CACHE_TTL=60` |
| Constructor | `ZerodhaDataFetcher(cache_ttl_minutes=60)` |
| Constructor | `ZerodhaInstrumentManager(cache_ttl_minutes=60)` |

If `ZERODHA_INSTRUMENT_CACHE_TTL` is invalid, the library logs one warning and falls back to `1440` minutes instead of crashing.

### Manual Refresh

```python
from zerodha_data_fetcher import refresh_instruments

refresh_instruments()
```

Cache location:

- Windows: `%LOCALAPPDATA%\zerodha_data_fetcher\Cache\`
- Linux/macOS: `~/.cache/zerodha_data_fetcher/`

## API Reference

### `ZerodhaDataFetcher`

Main class for fetching historical data.

Methods:

- `fetch_historical_data(ticker_token, start_date, end_date, timeframe="minute", chunk_failure_mode=None)`: fetch historical data with strict-by-default chunk failure handling
- `search_symbols(partial_name, limit=10)`: search the instrument data for matching trading symbols
- `get_instrument_info(symbol)`: return instrument metadata for a symbol when available

### Package Helpers

- `setup_logging(...)`: configure console and optional rotating file logging
- `refresh_instruments()`: force-refresh the instrument cache, ignoring TTL

## Error Handling

The package exposes dedicated exception types for common failure modes:

```python
from zerodha_data_fetcher.utils.exceptions import (
    AuthenticationError,
    DataFetchError,
    InvalidTickerError,
    ZerodhaAPIError,
)

try:
    data = fetcher.fetch_historical_data("INVALID", start_date, end_date)
except InvalidTickerError as exc:
    print(f"Invalid ticker: {exc}")
except AuthenticationError as exc:
    print(f"Authentication failed: {exc}")
except DataFetchError as exc:
    print(f"Data fetch failed: {exc}")
except ZerodhaAPIError as exc:
    print(f"Unexpected Zerodha API error: {exc}")
```

## Development

Contributor setup and workflow guidance live in [CONTRIBUTING.md](CONTRIBUTING.md).

Typical local workflow:

```bash
uv sync --all-extras
uv run pytest
uv run black .
uv run flake8
uv run mypy src/
```

## Contributing

Contributions are welcome across bug fixes, docs, tests, and new features.

- Read [CONTRIBUTING.md](CONTRIBUTING.md) before opening a pull request
- Follow the expectations in [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)
- Never include real Zerodha credentials in code, issues, or examples

## Security

Please review [SECURITY.md](SECURITY.md) for supported versions, private vulnerability reporting, and credential exposure guidance.

## License

This project is licensed under the MIT License. See [LICENSE](LICENSE) for details.

## Disclaimer

This package is for educational and research purposes. Make sure your usage complies with Zerodha's terms of service and any applicable laws or regulations.

## Support

- Documentation: https://github.com/JayceeGupta/Zerodha-Data-Fetcher#readme
- Bug reports: https://github.com/JayceeGupta/Zerodha-Data-Fetcher/issues
- Discussions: https://github.com/JayceeGupta/Zerodha-Data-Fetcher/discussions
