Metadata-Version: 2.4
Name: calsyn
Version: 0.1.0
Summary: Calibrated Synthetic Data Generator for Monte Carlo simulations
Author: Aleksandr Burov
License: MIT
Project-URL: Homepage, https://github.com/Saha555339/calsyn
Project-URL: Repository, https://github.com/Saha555339/calsyn
Keywords: synthetic data,monte carlo,catboost,calibration,causal inference
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: polars>=1.0
Requires-Dist: scipy>=1.10
Requires-Dist: catboost>=1.2
Requires-Dist: matplotlib>=3.7
Provides-Extra: tune
Requires-Dist: optuna>=3.0; extra == "tune"
Provides-Extra: dev
Requires-Dist: optuna>=3.0; extra == "dev"
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Dynamic: license-file

# calsyn

**Cal**ibrated **Syn**thetic Data Generator for Monte Carlo simulations.

Generate realistic synthetic trajectories using a piecewise-linear data-generating process calibrated on real data:

```
Y = effect(t) + f(X) + ε
```

where:
- **f(X)** — CatBoost model trained on real features, capturing nonlinear structure
- **effect(t)** — date-based treatment effect (single start date or arbitrary windows with different τ)
- **τ** — user-defined effect size added to the target trajectory during the treatment period
- **ε** — noise sampled from a distribution fitted to calibration residuals

## Installation

```bash
pip install calsyn
```

With Optuna hyperparameter tuning:

```bash
pip install "calsyn[tune]"
```

## Quick Start

```python
import polars as pl
from calsyn import CalibratedGenerator

# df is the source Polars DataFrame with date, feature columns, and target column.
# Split it into X (date + features) and y (target) before fitting.
target_col = "SBER"
X = df.drop(target_col)
y = df[target_col].to_numpy()

gen = CalibratedGenerator(date_col="date", noise="auto")
gen.fit(X, y)

# mode A: effect starts at a given date
result = gen.generate(X, treatment_start="2024-06-01", tau=0.02, n_simulations=500)
result.simulations.shape  # (500, n_samples)
```

## Input Format

The library expects the input data as:

- `df` — source `pl.DataFrame` with one date/datetime column, numeric feature columns,
  and one target column
- `X` — model input: `df` without the target column
- `y` — target array extracted from `df[target_col]`

`X` must contain:

- a date/datetime column (name passed via `date_col`)
- numeric feature columns used to predict the target

`X` must not contain the target column. The order of rows in `X` and `y` must match:
`y[i]` is the target value for row `X[i]`. The date column is used for treatment
windows and stratified validation; all other columns in `X` are passed to CatBoost
as numeric features.

```python
from datetime import date, timedelta
import polars as pl

start = date(2023, 1, 1)
df = pl.DataFrame({
    "date": [start + timedelta(days=i) for i in range(500)],
    "GAZP": gazp_log_returns,
    "LKOH": lkoh_log_returns,
    "USDRUB": usdrub_log_returns,
    "SBER": sber_log_returns,
})

target_col = "SBER"
X = df.drop(target_col)
y = df[target_col].to_numpy()

gen = CalibratedGenerator(date_col="date", noise="auto")
gen.fit(X, y)
```

Equivalent shape requirements:

```python
assert len(X) == len(y)
assert "date" in X.columns
assert target_col not in X.columns
```

## Treatment Modes

`tau` (Greek letter `τ`) is the treatment effect size that you set manually for
the target variable. It is not estimated by the model; it is added directly to
the generated target value for dates where treatment is active:

```
Y_sim(t) = f(X_t) + tau + ε_t
```

For example, if the target is log-returns, `tau=0.02` means an additive shock of
`0.02` to the simulated log-return on treated dates. `tau=0.0` is the no-effect
baseline, and negative values model a negative shock.

### Mode A — single start date

Everything after `treatment_start` gets effect τ. Supports tau grid for sweep.

```python
# single tau
result = gen.generate(X, treatment_start="2024-06-01", tau=0.02)

# tau grid — returns dict[float, GenerationResult]
results = gen.generate(
    X, treatment_start="2024-06-01",
    tau=[0.0, 0.01, 0.02, 0.05],
    n_simulations=500,
)
for tau_val, res in results.items():
    print(f"tau={tau_val}: mean={res.simulations.mean():.4f}")
```

### Mode B — arbitrary windows with different effects

Each window is `(start_date, end_date, tau)`. Dates are inclusive. Outside all windows, effect is zero.

```python
result = gen.generate(
    X,
    treatment=[
        ("2024-03-01", "2024-03-15", 0.03),   # positive shock in early March
        ("2024-06-01", "2024-06-10", -0.01),   # negative shock in June
        ("2024-09-01", "2024-09-30", 0.05),    # larger effect in September
    ],
    n_simulations=500,
)
```

## Feature Noise Simulation

A separate mode for sensitivity analysis: study how noise in individual features propagates to Y.

```
Y_sim = f(X̃) + ε_Y
X̃[j] = X[j] + ε_X[j]   for j in feature_noise
X̃[j] = X[j]             otherwise
```

The model `f` is **not** re-trained — user-specified noise is injected into the features directly. This is independent of treatment effects; do not mix with `generate()`.

```python
result = gen.generate_with_feature_noise(
    X,
    feature_noise={
        "USDRUB": {"distribution": "normal", "scale": 0.01},
        "OIL":    {"distribution": "t", "scale": 0.02, "df": 4},
    },
    n_simulations=500,
)

result.simulations.shape              # (500, n_samples) — Y trajectories
result.X_simulations["USDRUB"].shape  # (500, n_samples) — noisy feature values
result.f_X_mean                       # f(X) on original features, for comparison
```

### `feature_noise` spec

Each entry maps a feature name to a noise spec dict:

| Key | Required | Description |
|-----|----------|-------------|
| `distribution` | yes | `"normal"` or `"t"` |
| `scale` | yes | Std dev (normal) or scale parameter (t) |
| `loc` | no (default 0) | Location shift |
| `df` | yes for `"t"` | Degrees of freedom |

`X_simulations` contains only the noised features — un-noised features remain in the original `X`.

## Validation

The model is trained on ~90% of data (configurable via `val_fraction`). The held-out set is **stratified by time period** — from each period, `val_fraction` of observations go to OOS.

Stratification period is set via `strat_freq`:

| `strat_freq` | Period | Use case |
|---|---|---|
| `"M"` | Month | Daily/weekly financial data (default) |
| `"W"` | Week | Daily data with short history |
| `"d"` | Day | Intraday (hourly/minute) data |
| `"h"` | Hour | Sub-minute tick data |
| `"m"` | Minute | High-frequency tick data |

```python
# daily data — stratify by month (default)
gen = CalibratedGenerator(date_col="date", noise="auto", val_fraction=0.1)

# intraday data — stratify by day
gen = CalibratedGenerator(date_col="date", noise="auto", val_fraction=0.1, strat_freq="d")

gen.fit(X, y)
print(gen.oos_metrics)
# {'r2': 0.82, 'rmse': 0.017, 'bias': -0.0003}
```

## Noise Distribution

Three modes for the residual noise ε:

```python
# auto: pick normal or t by BIC on residuals
gen = CalibratedGenerator(date_col="date", noise="auto")

# force Gaussian
gen = CalibratedGenerator(date_col="date", noise="normal")

# force Student-t (heavier tails)
gen = CalibratedGenerator(date_col="date", noise="t")
```

After fitting, inspect the chosen distribution:

```python
gen.fit(X, y)
print(gen.noise_params)
# {'distribution': 't', 'df': 4.2, 'loc': 0.001, 'scale': 0.012}
```

### Overriding the noise scale

When residual variance is unreliable (e.g. low data variability or poor model fit), you can set the noise scale directly instead of relying on the fitted residuals:

```python
gen = CalibratedGenerator(date_col="date", noise="normal", noise_scale=0.05)
gen.fit(X, y)

print(gen.noise_params)
# {'distribution': 'normal', 'loc': ..., 'scale': 0.05}
```

## Random Seed

By default `random_seed=None`, so simulation outputs vary between runs — which is the correct behavior for Monte Carlo. Pass an integer to fix the seed for reproducibility:

```python
# reproducible
gen = CalibratedGenerator(date_col="date", random_seed=42)

# stochastic (default) — different trajectories each time
gen = CalibratedGenerator(date_col="date")
```

## Diagnostics

```python
result_h0 = gen.generate(X, treatment_start="2024-06-01", tau=0.0, n_simulations=500)
report = gen.diagnose(result_h0)

print(report.oos_metrics)
# {'r2': 0.82, 'rmse': 0.017, 'bias': -0.0003}

print(report.ks)
# KSResult(statistic=0.04, pvalue=0.72)

print(report.coverage)
# CoverageDiagnostic(coverage_1sigma=0.68, coverage_2sigma=0.95, n_points=500)

for c in report.correlations:
    print(f"  X{c.feature_index}: real={c.corr_real:.3f} synth={c.corr_synthetic:.3f}")
```

## Plots

```python
# coverage plot: real trajectory vs 1σ/2σ bands
gen.plot_coverage(result_h0)

# feature correlation comparison
gen.plot_correlations(result_h0, feature_names=["GAZP", "LKOH", "USDRUB"])
```

## Optuna Tuning

```python
gen = CalibratedGenerator(date_col="date", noise="auto", auto_tune=True, n_trials=100)
gen.fit(X, y)
```

## API Reference

### `CalibratedGenerator`

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `date_col` | `str` | `"date"` | Name of date/datetime column in X |
| `noise` | `"auto" \| "normal" \| "t"` | `"auto"` | Noise distribution |
| `noise_scale` | `float \| None` | `None` | Override fitted noise scale |
| `auto_tune` | `bool` | `False` | Optuna hyperparameter search |
| `n_trials` | `int` | `50` | Optuna trials |
| `val_fraction` | `float` | `0.1` | OOS fraction per time period |
| `strat_freq` | `str` | `"M"` | Stratification: "M", "W", "d", "h", "m" |
| `catboost_params` | `dict \| None` | `None` | Custom CatBoost params |
| `random_seed` | `int \| None` | `None` | Random seed (None = varies between runs) |

**Methods:**

- `.fit(X, y)` — fit calibration model (train-only) and noise distribution
- `.generate(X, treatment_start=..., tau=..., n_simulations=...)` — mode A
- `.generate(X, treatment=[...], n_simulations=...)` — mode B
- `.diagnose(result)` — KS, correlations, coverage, OOS metrics
- `.plot_coverage(result)` — coverage plot with 1σ/2σ bands
- `.plot_correlations(result, feature_names=...)` — correlation bar chart
- `.oos_metrics` — dict with R², RMSE, bias
- `.noise_params` — fitted noise distribution parameters

## License

MIT
