Metadata-Version: 2.4
Name: marketforge
Version: 2.0.0
Summary: A professional-grade synthetic OHLCV data generator
Home-page: https://github.com/DamienReichhart/MarketForge
Author: REICHHART Damien
License: MIT
Project-URL: Homepage, https://github.com/DamienReichhart/MarketForge
Project-URL: Documentation, https://github.com/DamienReichhart/MarketForge
Project-URL: Repository, https://github.com/DamienReichhart/MarketForge
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Office/Business :: Financial :: Investment
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: numpy>=2.4.0
Requires-Dist: scipy>=1.16.3
Requires-Dist: pandas>=2.3.3
Requires-Dist: click>=8.3.1
Requires-Dist: tqdm>=4.67.1
Provides-Extra: dev
Requires-Dist: mypy>=1.19.1; extra == "dev"
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: arch>=6.3.0; extra == "dev"
Requires-Dist: statsmodels>=0.14.0; extra == "dev"
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

# MarketForge

A professional-grade synthetic OHLCV (Open-High-Low-Close-Volume) data generator for financial markets. Designed for backtesting, algorithm development, and quantitative research, this tool generates realistic market data with proper statistical properties, correlations, and market dynamics.

## Table of Contents

- [Features](#features)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [Usage](#usage)
- [Architecture](#architecture)
- [Configuration](#configuration)
- [Output Format](#output-format)
- [Validation](#validation)
- [Performance](#performance)
- [Advanced Features](#advanced-features)
- [Examples](#examples)
- [Contributing](#contributing)
- [License](#license)

## Features

### Core Capabilities

- **Multi-Market Support**: Generate data for Forex (81 pairs), Crypto (53 assets), and Stocks (74 symbols)
- **Realistic Price Dynamics**:
  - Multivariate Student-t innovations (heavy tails + tail dependence — assets co-move in extremes)
  - GJR-GARCH(1,1) asymmetric volatility (leverage effect: down-moves raise volatility more)
  - Regime-switching (trend, range, high-vol, crash) with smooth transitions
  - Intraday & weekly volatility seasonality (variance-neutral)
  - Correlated multi-asset returns via Cholesky decomposition
- **Realistic OHLCV Construction**:
  - Intrabar High/Low via a Brownian bridge from open to close scaled by conditional volatility (statistically consistent ranges, High≥max(O,C) and Low≤min(O,C) by construction)
  - Market-specific gap handling (overnight, weekend)
  - Volume via a log-AR(1) process with volume clustering, volume-volatility coupling (Mixture-of-Distributions Hypothesis), and intraday seasonality
  - Time-of-day volume patterns (session-based markets)
- **Anomaly Injection**:
  - Price gaps (overnight/weekend discontinuities)
  - Spikes (fat-tail events with long wicks)
  - Flash crashes (multi-candle V-shaped recoveries)
- **Multiple Timeframes**: Automatic aggregation from m1 to W1 (m1, m5, m15, m30, H1, H4, D1, W1)
- **Batch Processing**: Memory-efficient processing with configurable batch sizes
- **Parallel Processing**: Multi-threaded batch processing for faster generation
- **Reproducibility**: Seed-based random number generation for identical outputs

### Market-Specific Features

- **Crypto**: 24/7 trading, no gaps, higher volatility, flash crash events
- **Forex**: Session-based trading, weekend gaps, lower volatility, session patterns
- **Stocks**: Market hours, overnight/weekend gaps, U-shaped volume patterns

## Installation

### Requirements

- Python 3.10 or higher

### Install from PyPI

For basic users, install the package directly from PyPI:

```bash
pip install marketforge
```

### Install from Source

For developers who want to modify the code or install the latest development version:

```bash
# Clone or download the repository
cd marketforge

# Install in development mode
pip install -e .

# Or install dependencies directly
pip install -r requirements.txt
```

### Verify Installation

```bash
marketforge --help
```

## Quick Start

### Basic Example

Generate one full year of fully realistic EURUSD data across all timeframes
(1 minute through 1 week), for the period 2024-01-01 → 2025-01-01 (UTC):

```bash
marketforge \
    --output-dir ./data \
    --from 1704067200 \
    --to 1735689600 \
    --assets EURUSD \
    --seed 42
```

This produces `./data/forex/EURUSD_{m1,m5,m15,m30,H1,H4,D1,W1}.csv`. The output
reproduces the documented stylized facts of real FX returns — heavy tails,
volatility clustering, the leverage effect, intraday/weekly seasonality, and
realistic intrabar ranges — and is fully reproducible from the `--seed`. Verify
it with the validation harness:

```bash
marketforge validate --input ./data/forex/EURUSD_m1.csv --market forex
```

### Generate Specific Markets

```bash
# Generate only crypto market (2 weeks, all timeframes)
marketforge \
    --output-dir ./data \
    --from 1704067200 \
    --to 1705276800 \
    --market crypto \
    --seed 42

# Generate multiple markets
marketforge \
    --output-dir ./data \
    --from 1704067200 \
    --to 1705276800 \
    --market forex,crypto \
    --seed 42
```

> **Note:** generating the `W1` (weekly) timeframe requires at least one week of
> data. For shorter windows, restrict `--timeframes` (e.g. `m1,m5,H1,D1`).

### With Anomalies

Inject structural anomaly events on top of the base-realistic series:

```bash
marketforge \
    --output-dir ./data \
    --from 1704067200 \
    --to 1705276800 \
    --market crypto \
    --anomalies spikes,flash_crash \
    --seed 42
```

## Usage

### Command-Line Interface

The generator provides a comprehensive CLI with the following options:

#### Required Arguments

- `--output-dir, -o`: Output directory where CSV files will be written
- `--from`: Start timestamp in Unix epoch seconds (UTC)
- `--to`: End timestamp in Unix epoch seconds (UTC)

#### Optional Arguments

- `--market, -m`: Markets to generate (`forex`, `crypto`, `stocks`, `all`). Default: `all`
- `--seed, -s`: Random seed for reproducibility. Default: random
- `--anomalies`: Comma-separated anomaly types (`gaps`, `spikes`, `flash_crash`)
- `--timeframes, -t`: Comma-separated timeframes. Default: `m1,m5,m15,m30,H1,H4,D1,W1`
- `--batch-size`: Number of assets per batch (memory management). Default: 25
- `--threads, --thread-count`: Number of threads for parallel processing. Default: auto-detect
- `--progress/--no-progress`: Show/hide progress bar. Default: enabled

### Getting Unix Timestamps

**Linux/Mac:**
```bash
date +%s                    # Current timestamp
date -d "2024-01-01" +%s    # Specific date
```

**Windows (PowerShell):**
```powershell
[DateTimeOffset]::Parse("2024-01-01").ToUnixTimeSeconds()
```

**Online Tools:**
- Use online Unix timestamp converters
- Example: https://www.epochconverter.com/

### Timeframe Options

Available timeframes:
- `m1`: 1 minute
- `m5`: 5 minutes
- `m15`: 15 minutes
- `m30`: 30 minutes
- `H1`: 1 hour
- `H4`: 4 hours
- `D1`: 1 day
- `W1`: 1 week

All timeframes are aggregated from m1 data using standard OHLCV aggregation rules.

### Anomaly Types

- `gaps`: Price discontinuities between candles (overnight, weekend, news events)
- `spikes`: Sudden intrabar price moves (fat-tail events with long wicks)
- `flash_crash`: Multi-candle sharp declines with V-shaped recovery

**Note**: Gaps are automatically disabled for crypto markets (24/7 trading).

### Batch Processing

For large datasets, adjust batch size based on available memory:

```bash
# Small batch size (less memory, slower)
marketforge ... --batch-size 10

# Large batch size (more memory, faster)
marketforge ... --batch-size 50
```

### Parallel Processing

Enable multi-threaded processing for faster generation:

```bash
# Use 4 threads
marketforge ... --threads 4

# Sequential processing (no threading)
marketforge ... --threads 1

# Auto-detect (default, up to 16 threads)
marketforge ...
```

## Architecture

### Key Components

#### 1. Return Generation (`core/returns.py`)

Generates correlated log-returns using:
- **Correlation Engine**: Cholesky decomposition for multi-asset correlation with Student-t innovations
- **GJR-GARCH(1,1) Model**: Asymmetric volatility clustering with leverage effect (σ²_t = ω + (α + γ·1_{ε<0})·ε²_{t-1} + β·σ²_{t-1})
- **Regime-Switching**: Markov chain for smooth market state transitions (trend, range, high-vol, crash)
- **Volatility Seasonality**: Intraday and weekly variance-neutral seasonality multipliers

#### 2. OHLCV Construction (`generators/ohlcv.py`)

Converts price series to OHLCV candles:
- **Open Prices**: May include gaps for non-crypto markets
- **High/Low**: Brownian-bridge simulation from open to close scaled by conditional volatility (High≥max(O,C) and Low≤min(O,C) by construction)
- **Volume**: Driven by the log-AR(1) MDH volume model

#### 3. Volume Generation (`generators/volume.py`)

Generates realistic trading volumes via a log-AR(1) MDH model:
- **Log-AR(1) Process**: Mean-reverting autoregressive log-volume for volume clustering
- **Volume-Volatility Coupling**: Mixture-of-Distributions Hypothesis — latent information flow drives both returns and volume
- **Intraday Seasonality**: Time-of-day volume multipliers (session-based for stocks and forex)

#### 4. Anomaly Injection (`generators/anomalies.py`)

Injects market anomalies:
- **Gaps**: Price discontinuities at session boundaries
- **Spikes**: Fat-tail events with extended wicks
- **Flash Crashes**: Multi-candle decline and recovery patterns

#### 5. Timeframe Aggregation (`aggregation/timeframes.py`)

Aggregates m1 data to higher timeframes:
- Standard OHLCV rules: Open (first), High (max), Low (min), Close (last), Volume (sum)
- Efficient numpy-based aggregation

#### 6. Batch Processing (`processing/batch.py`)

Manages memory-efficient processing:
- Splits assets into batches
- Maintains correlation within batches
- Supports parallel processing via threading
- Automatic memory management

## Configuration

### Market Configurations

Each market has pre-configured:
- **Asset Definitions**: Symbols, start prices, volatilities, drifts
- **Correlation Matrices**: Realistic inter-asset correlations
- **Volume Parameters**: Base volumes and volume volatilities
- **Market Defaults**: GARCH params, regime params, anomaly configs

### Market-Specific Defaults

#### Crypto Market
- **Assets**: 53 cryptocurrencies (BTC, ETH, SOL, etc.)
- **Volatility**: Higher (0.55-1.00 annualized)
- **Gaps**: Disabled (24/7 trading)
- **Anomalies**: Spikes and flash crashes

#### Forex Market
- **Assets**: 81 currency pairs (EURUSD, GBPUSD, etc.)
- **Volatility**: Lower (0.008-0.015 annualized)
- **Gaps**: Enabled (weekend and session gaps)
- **Anomalies**: Gaps and spikes

#### Stocks Market
- **Assets**: 74 stock symbols (AAPL, MSFT, etc.)
- **Volatility**: Moderate (0.015-0.030 annualized)
- **Gaps**: Enabled (overnight and weekend)
- **Anomalies**: Gaps and spikes

### Custom Configuration

To customize market parameters, modify the configuration files:
- `marketforge/configs/crypto_config.py`
- `marketforge/configs/forex_config.py`
- `marketforge/configs/stocks_config.py`

## Output Format

### Directory Structure

```
output/
├── crypto/
│   ├── BTCUSD_m1.csv
│   ├── BTCUSD_m5.csv
│   ├── BTCUSD_H1.csv
│   ├── BTCUSD_D1.csv
│   ├── ETHUSD_m1.csv
│   └── ...
├── forex/
│   ├── EURUSD_m1.csv
│   ├── EURUSD_m5.csv
│   └── ...
└── stocks/
    ├── AAPL_m1.csv
    ├── AAPL_m5.csv
    └── ...
```

### CSV Format

Each CSV file contains:

```csv
timestamp,open,high,low,close,volume
1704067200,50000.00000000,50125.50000000,49875.25000000,50050.75000000,1250.5000
1704067260,50050.75000000,50100.00000000,50025.00000000,50075.25000000,1180.2500
...
```

**Columns:**
- `timestamp`: Unix timestamp (seconds since epoch)
- `open`: Opening price
- `high`: Highest price in the period
- `low`: Lowest price in the period
- `close`: Closing price
- `volume`: Trading volume

**Precision:**
- Prices: 8 decimal places
- Volume: 4 decimal places

### Data Validation

The generated data ensures:
- `high >= max(open, close)`
- `low <= min(open, close)`
- `high >= low`
- `volume >= 0`
- No negative prices

## Validation

MarketForge ships a validation harness that measures generated data against the
documented "stylized facts" of real financial returns.

### Stylized facts checked

- **Heavy tails** — excess kurtosis of m1 returns within a realistic per-market band.
- **Volatility clustering** — significant, slowly-decaying autocorrelation of |returns|.
- **Leverage effect** — negative correlation between returns and next-period volatility.
- **Range efficiency** — realistic intrabar range relative to candle body.

### Validate a CSV

```bash
marketforge validate --input ./data/crypto/BTCUSD_m1.csv --market crypto
```

This prints a per-metric report with target bands and exits non-zero if a band is
violated. Run the bundled benchmark across all markets with:

```bash
python scripts/benchmark_realism.py
```

## Performance

### Memory Usage

Memory usage depends on:
- Number of assets per batch
- Duration (number of m1 candles)
- Number of timeframes

**Estimation:**
- Per asset (1 month, 8 timeframes): ~5-10 MB
- Batch of 25 assets: ~125-250 MB
- Full market (e.g., 53 crypto assets): ~265-530 MB per batch

**Recommendations:**
- Use `--batch-size 25` for most systems
- Reduce to `--batch-size 10` for limited RAM (< 8 GB)
- Increase to `--batch-size 50` for systems with > 16 GB RAM

## Advanced Features

### Programmatic Usage

The generator can be used programmatically:

```python
from marketforge.configs.loader import ConfigRegistry
from marketforge.processing.batch import generate_market_batched, BatchConfig
from marketforge.configs.base import MarketType
from marketforge.config.settings import AnomalyType

# Load market configuration
registry = ConfigRegistry()
crypto_config = registry.get_config(MarketType.CRYPTO)

# Configure batch processing
batch_config = BatchConfig(
    batch_size=25,
    thread_count=4,
    gc_between_batches=True
)

# Generate data
output_paths = generate_market_batched(
    market_config=crypto_config,
    start_timestamp=1704067200,
    end_timestamp=1704153600,
    output_dir="./data",
    seed=42,
    anomaly_types=frozenset({AnomalyType.SPIKES, AnomalyType.FLASH_CRASH}),
    timeframes=("m1", "m5", "H1", "D1"),
    batch_config=batch_config,
    show_progress=True
)
```

### Custom Asset Generation

To generate data for custom assets:

```python
from marketforge.config.settings import GeneratorConfig, AssetConfig, GARCHParams, MarketType
from marketforge.generators.ohlcv import OHLCVBuilder
from marketforge.utils.random import RandomState
import numpy as np

# Define custom assets
assets = [
    AssetConfig("CUSTOM1", start_price=100.0, volatility=0.02, drift=0.0001),
    AssetConfig("CUSTOM2", start_price=50.0, volatility=0.03, drift=0.0002),
]

# Create correlation matrix
correlation_matrix = np.array([
    [1.0, 0.7],
    [0.7, 1.0]
])

# Create generator config
config = GeneratorConfig(
    assets=assets,
    market_type=MarketType.CRYPTO,
    start_timestamp=1704067200,
    end_timestamp=1704153600,
    correlation_matrix=correlation_matrix,
    seed=42
)

# Generate data
rng = RandomState(42)
builder = OHLCVBuilder(config)
ohlcv_data = builder.build(rng)

# Access data
btc_data = ohlcv_data["CUSTOM1"]
print(f"Generated {len(btc_data)} candles")
```

### Regime Analysis

Access regime information from return generation:

```python
from marketforge.core.returns import ReturnGenerator

generator = ReturnGenerator(config)
result = generator.generate(rng)

# Regime indices: 0=trend_up, 1=trend_down, 2=range, 3=high_vol, 4=crash
regime_indices = result.regime_indices
volatilities = result.volatilities
```

## Examples

### Example 1: Generate 1 Month of Crypto Data

```bash
# January 2024 (Unix timestamps)
marketforge \
    --output-dir ./data/crypto_jan2024 \
    --from 1704067200 \
    --to 1706745599 \
    --market crypto \
    --seed 42 \
    --anomalies spikes,flash_crash \
    --timeframes m1,m5,H1,D1
```

### Example 2: Generate All Markets with Anomalies

```bash
marketforge \
    --output-dir ./data/all_markets \
    --from 1704067200 \
    --to 1705276800 \
    --market all \
    --seed 42 \
    --anomalies gaps,spikes \
    --threads 8 \
    --batch-size 25
```

### Example 3: High-Performance Generation

```bash
# Large dataset with parallel processing
marketforge \
    --output-dir ./data/large_dataset \
    --from 946681200 \
    --to 1767221999 \
    --market all \
    --seed 42 \
    --threads 16 \
    --batch-size 30 \
    --timeframes m1,m5,m15,m30,H1,H4,D1
```

### Example 4: Minimal Generation (Testing)

```bash
# Small dataset for testing
marketforge \
    --output-dir ./data/test \
    --from 1704067200 \
    --to 1704070800 \
    --market crypto \
    --seed 42 \
    --timeframes m1,H1 \
    --batch-size 10 \
    --threads 1
```

### Example 5: Comprehensive Generation with All Features

```bash
# Generate all markets with anomalies, parallel processing, and progress tracking
marketforge \
    --output-dir ./data \
    --from 946681200 \
    --to 1767221999 \
    --seed 42 \
    --anomalies gaps,spikes \
    --market all \
    --threads 8 \
    --batch-size 10 \
    --progress
```

## Contributing

Contributions are welcome! Areas for improvement:

- Additional market types
- More anomaly types
- Custom volume models
- Additional timeframes
- Performance optimizations
- Documentation improvements

### Development Setup

```bash
# Install in development mode
pip install -e ".[dev]"

# Run type checking
mypy marketforge
```

## License

MIT License - see LICENSE file for details.

## Support

For issues, questions, or contributions:
- Open an issue on GitHub
- Check existing documentation
- Review code examples in the repository

---

**Version**: 2.0.0  
**Python**: 3.10+  
**License**: MIT

