Metadata-Version: 2.5
Name: lagkit
Version: 0.1.0
Summary: Sparse Bayesian regression on lagged functional features, with leakage-free subject-level cross-validation.
Project-URL: Homepage, https://github.com/projeet1/lagkit
Project-URL: Issues, https://github.com/projeet1/lagkit/issues
Author: Projeet Bhaumik
License: MIT License
        
        Copyright (c) 2026 Projeet Bhaumik
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: bayesian,cross-validation,functional-data-analysis,horseshoe,time-lagged-features,wearables
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.10
Requires-Dist: arviz>=0.17
Requires-Dist: numpy>=1.24
Requires-Dist: pandas>=2.0
Requires-Dist: patsy>=0.5
Requires-Dist: pymc>=5.10
Requires-Dist: pyyaml>=6.0
Requires-Dist: scikit-learn>=1.3
Provides-Extra: dev
Requires-Dist: matplotlib>=3.7; extra == 'dev'
Requires-Dist: pytest>=7.4; extra == 'dev'
Provides-Extra: plots
Requires-Dist: matplotlib>=3.7; extra == 'plots'
Description-Content-Type: text/markdown

# lagkit

**Sparse Bayesian regression on lagged functional features, with leakage-free subject-level cross-validation.**

Built for the awkward but common setting where a *small* cohort of subjects each contributes a *long* history of lagged measurements — 30 people, 72 hourly lags, 15 sensor-derived features — and you need to answer two questions at once:

1. How well can the outcome be predicted?
2. **Which lags actually carried the signal?**

Treating each of the 1080 lag-feature pairs as an independent covariate answers neither. `lagkit` smooths across lag, shrinks aggressively but selectively, validates without leaking, and can map coefficients back onto the lag axis so the second question has an answer.

```bash
pip install lagkit
```

---

## The method in one pass

```python
from lagkit import LagSpec, LaggedFeatureUnion, GroupedHorseshoe, critical_windows

spec  = LagSpec(max_lag=72)                          # cough_1h ... cough_72h
union = LaggedFeatureUnion(["cough", "breathingRate"], spec)

X_train = union.fit_transform(train_df)              # 144 lag columns -> ~6 scores
model   = GroupedHorseshoe(groups=union.groups_).fit(X_train, y_train)

y_hat = model.predict(union.transform(test_df))

# Which hours mattered?
curves  = union.coefficients_to_lag_domain(model.beta_draws_)
windows = critical_windows(curves["cough"], spec.lags)
```

`windows` is a per-lag table with the posterior mean effect, a 95% band, and flags for lags where the posterior puts ≥95% of its mass on one side of zero.

---

## Why each piece exists

| Component | Problem it solves |
|---|---|
| [`LagSpec`](src/lagkit/lags.py) | Addresses wide `feature_24h` tables without every module hard-coding the naming convention. Swap `template` for `{feature}_lag{lag}` and everything downstream still works. |
| [`WidthPreservingKNNImputer`](src/lagkit/impute.py) | `KNNImputer` **silently drops** columns that are entirely missing. For a lag trajectory about to be projected onto a fixed basis, losing lag 37 is destructive. This one restores such columns and fills them from their neighbours in lag space. |
| [`SplineFPCA`](src/lagkit/fpca.py) | 72 hourly lags → a handful of latent scores, by B-spline projection then PCA. Enforces smoothness across lag rather than pretending hour 24 and hour 25 are unrelated. |
| [`LaggedFeatureUnion`](src/lagkit/fpca.py) | One `SplineFPCA` per feature, concatenated into a design matrix, tracking which columns belong to which feature. |
| [`GroupedHorseshoe`](src/lagkit/horseshoe.py) | Regularized horseshoe with a shrinkage scale per feature group, so a feature switches on or off as a whole. Global scale calibrated from the expected number of relevant coefficients rather than left vague. |
| [`GatedResidualExpert`](src/lagkit/residual.py) | A boosted tree will confidently "correct" a test point sitting nowhere near any training data. This gates every correction by local support and caps what survives. |
| [`crossval`](src/lagkit/crossval.py) | Outer LOSO, inner CV for tuning, cross-fitted residuals — the scaffolding that keeps small-cohort evaluation honest. |
| [`critical_windows`](src/lagkit/windows.py) | Turns posterior effect curves into "hours 18–24 carried a credible negative effect". |
| [`ThresholdGrader`](src/lagkit/grading.py) | Continuous prediction → ordinal grade, with class *probabilities* from the posterior. GOLD staging ships as a preset. |

---

## The leakage discipline

The CV helpers take a **callback that rebuilds the model from index subsets**, rather than accepting a pre-built design matrix:

```python
def fit_predict(fit_idx, val_idx, fold_index):
    ...  # refit imputation, splines, PCA, scaling AND the model on fit_idx alone
    return predictions_for_val_idx

oof = cross_fitted_predictions(len(train_df), fit_predict, n_splits=5)
```

This is deliberate and load-bearing. Fitting the representation once outside the loop leaks validation rows into the imputer, the PCA and the scalers, and quietly inflates every score that follows. It is the single easiest way to get an impressive-looking result that does not replicate.

Three layers of protection, applied by `run_experiment` automatically:

- **Outer LOSO** — no subject informs their own prediction.
- **Inner CV inside the outer training set** — the held-out subject plays no part in choosing hyperparameters.
- **Cross-fitted residuals** — the residual stage trains on out-of-fold residuals, never in-sample ones (which would teach it the backbone's overfitting rather than its systematic errors).

---

## Running a full experiment

```python
from lagkit import ExperimentConfig
from lagkit.experiment import run_experiment

config = ExperimentConfig(
    path="cohort.csv",
    target="FEV1",
    group_col="subject_id",
    features=["cough", "breathingRate", "ie_ratio"],
    sparse_features=["cough"],          # nearest-lag carry, not interpolation
    max_lag=72,
    use_residual_expert=True,
    out_dir="outputs",
)

result = run_experiment(config)
result.save()

print(result.summary["r2"], result.summary["coverage95_predictive"])
print(result.critical_windows["cough"])
```

Or from the shell:

```bash
lagkit template > config.yaml
```

```bash
lagkit run config.yaml --set target="FVC obs" --out-dir results/fvc
```

`run_experiment` is a convenience layer, not the point. Every primitive composes by hand.

### Replaying an existing notebook config

```python
config = ExperimentConfig.from_legacy(BASE_CONFIG)   # upper-case dict from the notebook
```

Unrecognised keys are ignored, so configs that accumulated dead settings still load.

---

## Reading the results

Point accuracy alone is a poor summary when n is small. `result.summary` reports three things together:

- **accuracy** — `r2`, `rmse`, `mae`, `pearson_r`, plus `*_backbone_only` so the residual stage has to justify itself;
- **interval honesty** — `coverage95_predictive` against the nominal 0.95;
- **calibration** — `pred_on_true_slope`. Well below 1 means predictions are being compressed towards the cohort mean. This can coexist with a respectable R² and is invisible without the check.

Convergence is reported per fold: `divergences_total > 0` or `rhat_max > 1.01` means the posterior is not trustworthy, however good the predictions look.

---

## Installation

```bash
pip install lagkit              # core
```

```bash
pip install "lagkit[plots]"     # + matplotlib helpers
```

Requires Python ≥ 3.10. Core dependencies: numpy, pandas, scikit-learn, patsy, pymc, arviz, pyyaml.

## Development

```bash
pip install -e ".[dev]"
```

```bash
pytest -m "not slow"
```

The `slow` marker covers tests that run MCMC. Run the full suite with plain `pytest`.

## Citation

The method was developed for an honours project on predicting spirometry from wearable respiratory sensing. If you use it, please cite the repository.

## License

MIT
