Edit on GitHub

shap_relativities

SHAP Relativities - actuarial-grade multiplicative relativities from tree models.

Converts a trained GBM's SHAP values into the same format as GLM exp(beta) relativities: a table of (feature, level, relativity) triples where the base level is 1.0 and relativities multiply together to give the model's expected prediction.

Typical usage

>>> from shap_relativities import SHAPRelativities
>>> sr = SHAPRelativities(model, X, exposure=df["exposure"],
...                       categorical_features=["area", "ncd_years"])
>>> sr.fit()
>>> rels = sr.extract_relativities(
...     normalise_to="base_level",
...     base_levels={"area": "A", "ncd_years": 0},
... )

Or use the convenience wrapper for one-liners:

>>> from shap_relativities import extract_relativities
>>> rels = extract_relativities(model, X, exposure=df["exposure"],
...                             categorical_features=["area"])

 1"""
 2SHAP Relativities - actuarial-grade multiplicative relativities from tree models.
 3
 4Converts a trained GBM's SHAP values into the same format as GLM exp(beta)
 5relativities: a table of (feature, level, relativity) triples where the base
 6level is 1.0 and relativities multiply together to give the model's expected
 7prediction.
 8
 9Typical usage
10-------------
11>>> from shap_relativities import SHAPRelativities
12>>> sr = SHAPRelativities(model, X, exposure=df["exposure"],
13...                       categorical_features=["area", "ncd_years"])
14>>> sr.fit()
15>>> rels = sr.extract_relativities(
16...     normalise_to="base_level",
17...     base_levels={"area": "A", "ncd_years": 0},
18... )
19
20Or use the convenience wrapper for one-liners:
21
22>>> from shap_relativities import extract_relativities
23>>> rels = extract_relativities(model, X, exposure=df["exposure"],
24...                             categorical_features=["area"])
25"""
26
27from __future__ import annotations
28
29from typing import Any
30
31import polars as pl
32
33from ._core import SHAPRelativities
34
35__all__ = ["SHAPRelativities", "extract_relativities"]
36
37from importlib.metadata import version, PackageNotFoundError
38
39try:
40    __version__ = version("shap-relativities")
41except PackageNotFoundError:
42    __version__ = "0.0.0"  # not installed
43
44
45def extract_relativities(
46    model: Any,
47    X: Any,
48    exposure: Any = None,
49    categorical_features: list[str] | None = None,
50    base_levels: dict[str, str | float | int] | None = None,
51    ci_method: str = "clt",
52) -> pl.DataFrame:
53    """
54    One-shot extraction of SHAP relativities from a tree model.
55
56    Wraps SHAPRelativities.fit() and extract_relativities() for cases where
57    you don't need the intermediate object.
58
59    Args:
60        model: Trained CatBoost model with a log-link objective (Poisson,
61            Tweedie, or Gamma). CatBoost is the recommended choice - it handles
62            categorical features natively without encoding.
63        X: Feature matrix. Accepts a Polars or pandas DataFrame. Polars is
64            preferred; pandas is accepted and converted internally.
65        exposure: Earned policy years. If None, all observations are equally
66            weighted.
67        categorical_features: Features to aggregate by level. If None, all
68            non-numeric columns are treated as categorical.
69        base_levels: Base level for each categorical feature (gets
70            relativity = 1.0).
71        ci_method: "clt" (default) or "none".
72
73    Returns:
74        Polars DataFrame with columns: feature, level, relativity, lower_ci,
75        upper_ci, mean_shap, shap_std, n_obs, exposure_weight.
76    """
77    sr = SHAPRelativities(model, X, exposure, categorical_features)
78    sr.fit()
79    return sr.extract_relativities(base_levels=base_levels, ci_method=ci_method)
class SHAPRelativities: 
 87class SHAPRelativities:
 88    """
 89    Extract multiplicative rating relativities from a tree model via SHAP.
 90
 91    Workflow::
 92
 93        sr = SHAPRelativities(
 94            model=catboost_model,
 95            X=df.select(["area", "ncd_years", "has_convictions"]),
 96            exposure=df["exposure"],
 97            categorical_features=["area", "ncd_years"],
 98        )
 99        sr.fit()
100        rels = sr.extract_relativities(
101            normalise_to="base_level",
102            base_levels={"area": "A", "ncd_years": 0},
103        )
104
105    Args:
106        model: A trained CatBoost model. Must use a log-link objective (Poisson,
107            Tweedie, Gamma). CatBoost is the recommended default - it handles
108            categoricals natively.
109        X: Feature matrix. Use training data for in-sample relativities, or a
110            representative holdout sample for out-of-sample. Polars DataFrames
111            are preferred; pandas DataFrames are accepted and converted
112            internally.
113        exposure: Earned policy years (or other volume measure). Used as
114            observation weights throughout. If None, all observations are
115            weighted equally.
116        categorical_features: Features to aggregate by level (bar-chart style).
117            If None, all non-numeric columns are treated as categorical.
118        continuous_features: Features to leave as per-observation points.
119            If None, all numeric columns are treated as continuous.
120        feature_perturbation: "tree_path_dependent" (default, fast, no
121            background data needed) or "interventional" (corrects for feature
122            correlation, needs background_data).
123        background_data: Required only if feature_perturbation="interventional".
124        n_background_samples: Number of background samples for interventional
125            SHAP. Default 1000.
126        annualise_exposure: If True and exposure is provided, subtract mean
127            log(exposure) from the expected_value to give an annualised
128            baseline. Default True.
129    """
130
131    def __init__(
132        self,
133        model: Any,
134        X: Any,
135        exposure: Any = None,
136        categorical_features: list[str] | None = None,
137        continuous_features: list[str] | None = None,
138        background_data: Any = None,
139        feature_perturbation: str = "tree_path_dependent",
140        n_background_samples: int = 1000,
141        annualise_exposure: bool = True,
142    ) -> None:
143        if not _SHAP_AVAILABLE:
144            raise ImportError(
145                "shap is required for SHAPRelativities. "
146                "Install it with: uv add 'shap-relativities[ml]'"
147            )
148
149        self._model = model
150        self._X: pl.DataFrame = _to_polars(X)
151        self._background_data = (
152            _to_polars(background_data) if background_data is not None else None
153        )
154
155        # Normalise exposure to a numpy array
156        if exposure is None:
157            self._exposure: np.ndarray | None = None
158        elif isinstance(exposure, np.ndarray):
159            self._exposure = exposure
160        elif isinstance(exposure, pl.Series):
161            self._exposure = exposure.to_numpy()
162        else:
163            # pd.Series or similar
164            self._exposure = np.asarray(exposure)
165
166        # Validate exposure length matches X
167        if self._exposure is not None and len(self._exposure) != len(self._X):
168            raise ValueError(
169                f"exposure length ({len(self._exposure)}) does not match "
170                f"X length ({len(self._X)}). Both must have the same number of rows."
171            )
172
173        self._feature_perturbation = feature_perturbation
174        self._n_background_samples = n_background_samples
175        self._annualise_exposure = annualise_exposure
176
177        # Classify features
178        self._categorical_features = categorical_features or self._infer_categorical()
179        self._continuous_features = continuous_features or self._infer_continuous()
180
181        # Populated by fit()
182        self._shap_values: np.ndarray | None = None
183        self._expected_value: float | None = None
184        self._is_fitted: bool = False
185
186    def _infer_categorical(self) -> list[str]:
187        numeric_types = (
188            pl.Int8, pl.Int16, pl.Int32, pl.Int64,
189            pl.UInt8, pl.UInt16, pl.UInt32, pl.UInt64,
190            pl.Float32, pl.Float64,
191        )
192        return [
193            c for c in self._X.columns
194            if not isinstance(self._X[c].dtype, numeric_types)
195        ]
196
197    def _infer_continuous(self) -> list[str]:
198        numeric_types = (
199            pl.Int8, pl.Int16, pl.Int32, pl.Int64,
200            pl.UInt8, pl.UInt16, pl.UInt32, pl.UInt64,
201            pl.Float32, pl.Float64,
202        )
203        return [
204            c for c in self._X.columns
205            if isinstance(self._X[c].dtype, numeric_types)
206            and c not in (self._categorical_features or [])
207        ]
208
209    def fit(self) -> "SHAPRelativities":
210        """
211        Compute SHAP values for all features in X.
212
213        Must be called before extract_relativities(). Calling fit() again
214        recomputes SHAP values (e.g. after changing X or the background data).
215
216        The feature matrix is converted to pandas internally for shap's
217        TreeExplainer. The conversion is a necessary bridge - shap requires
218        pandas for column name handling.
219
220        Returns:
221            Self, for method chaining.
222        """
223        # Convert to pandas for shap - unavoidable bridge
224        X_pd = _to_pandas(self._X)
225
226        bg_data = None
227        if self._feature_perturbation == "interventional":
228            if self._background_data is not None:
229                bg_data = _to_pandas(self._background_data)
230            else:
231                n_bg = min(self._n_background_samples, len(X_pd))
232                bg_data = shap.sample(X_pd, n_bg)
233
234        explainer = shap.TreeExplainer(
235            self._model,
236            data=bg_data,
237            feature_perturbation=self._feature_perturbation,
238            model_output="raw",
239        )
240
241        raw = explainer.shap_values(X_pd)
242
243        # Some models return a list when there is a single output
244        if isinstance(raw, list):
245            if len(raw) == 1:
246                raw = raw[0]
247            else:
248                raise ValueError(
249                    f"Model has {len(raw)} outputs. SHAPRelativities supports "
250                    "single-output models only."
251                )
252
253        self._shap_values = raw
254
255        ev = explainer.expected_value
256        if isinstance(ev, (list, np.ndarray)):
257            ev = float(ev[0])
258        self._expected_value = float(ev)
259
260        self._is_fitted = True
261        return self
262
263    def _check_fitted(self) -> None:
264        if not self._is_fitted:
265            raise RuntimeError("Call fit() before using this method.")
266
267    def shap_values(self) -> np.ndarray:
268        """
269        Raw SHAP values, shape (n_obs, n_features), in log space.
270
271        Returns:
272            Array of shape (n_obs, n_features).
273        """
274        self._check_fitted()
275        return self._shap_values  # type: ignore[return-value]
276
277    def baseline(self) -> float:
278        """
279        exp(expected_value) - the base rate in prediction space.
280
281        If annualise_exposure=True and exposure was provided, this is adjusted
282        for the average log-exposure offset so it represents an annualised rate.
283
284        Returns:
285            Base rate as a float.
286        """
287        self._check_fitted()
288        ev = self._expected_value  # type: ignore[assignment]
289
290        if self._annualise_exposure and self._exposure is not None:
291            mean_log_exp = float(np.mean(np.log(np.clip(self._exposure, 1e-9, None))))
292            ev = ev - mean_log_exp
293
294        return float(np.exp(ev))
295
296    def extract_relativities(
297        self,
298        normalise_to: str = "base_level",
299        base_levels: dict[str, str | float | int] | None = None,
300        ci_method: str = "clt",
301        n_bootstrap: int = 200,
302        ci_level: float = 0.95,
303    ) -> pl.DataFrame:
304        """
305        Extract multiplicative relativities from SHAP values.
306
307        Args:
308            normalise_to: "base_level" (base level for each feature gets
309                relativity = 1.0) or "mean" (exposure-weighted portfolio
310                mean = 1.0).
311            base_levels: Mapping of feature -> base level value. Required for
312                categorical features when normalise_to="base_level". Continuous
313                features automatically use mean normalisation regardless of
314                this setting.
315            ci_method: "clt" (CLT approximation, default, fast) or "none" (no
316                CIs). "bootstrap" is not yet implemented.
317            n_bootstrap: Ignored unless ci_method="bootstrap".
318            ci_level: Two-sided confidence level. Default 0.95.
319
320        Returns:
321            Polars DataFrame with columns: feature, level, relativity,
322            lower_ci, upper_ci, mean_shap, shap_std, n_obs, exposure_weight.
323            One row per (feature, level) combination.
324        """
325        self._check_fitted()
326
327        _VALID_CI_METHODS = {"clt", "bootstrap", "none"}
328        if ci_method not in _VALID_CI_METHODS:
329            raise ValueError(
330                f"Unknown ci_method {ci_method!r}. "
331                f"Valid options are: {sorted(_VALID_CI_METHODS)}."
332            )
333
334        if ci_method == "bootstrap":
335            raise NotImplementedError(
336                "Bootstrap CIs are not yet implemented. Use ci_method='clt'."
337            )
338
339        base_levels = base_levels or {}
340        weights = (
341            self._exposure if self._exposure is not None
342            else np.ones(len(self._X))
343        )
344
345        feature_names = self._X.columns
346        shap_vals = self._shap_values  # type: ignore[assignment]
347
348        parts: list[pl.DataFrame] = []
349
350        for i, feat in enumerate(feature_names):
351            feat_vals = self._X[feat].to_numpy()
352            shap_col = shap_vals[:, i]
353
354            is_categorical = feat in self._categorical_features
355
356            if is_categorical:
357                agg = aggregate_categorical(feat, feat_vals, shap_col, weights)
358            else:
359                agg = aggregate_continuous(feat, feat_vals, shap_col, weights)
360
361            # Normalisation
362            if normalise_to == "base_level" and is_categorical:
363                base = base_levels.get(feat)
364                if base is None:
365                    # Fall back to the level with the smallest mean_shap as
366                    # a sensible default (closest to intercept)
367                    base = agg.sort("mean_shap")["level"][0]
368                    warnings.warn(
369                        f"No base level specified for '{feat}'. "
370                        f"Using '{base}' (lowest mean SHAP) as base.",
371                        UserWarning,
372                        stacklevel=2,
373                    )
374
375                if ci_method == "none":
376                    base_key = str(base)
377                    base_rows = agg.filter(pl.col("level") == base_key)
378                    base_shap = base_rows["mean_shap"][0]
379                    agg = agg.with_columns([
380                        (pl.col("mean_shap") - base_shap).exp().alias("relativity"),
381                        pl.lit(float("nan")).alias("lower_ci"),
382                        pl.lit(float("nan")).alias("upper_ci"),
383                    ])
384                else:
385                    agg = normalise_base_level(agg, base, ci_level=ci_level)
386
387            else:
388                # Mean normalisation for continuous features, or when
389                # normalise_to="mean" for any feature
390                if ci_method == "none":
391                    total_weight = agg["exposure_weight"].sum()
392                    portfolio_mean = float(
393                        (agg["mean_shap"] * agg["exposure_weight"]).sum()
394                        / total_weight
395                    ) if total_weight > 0 else 0.0
396                    agg = agg.with_columns([
397                        (pl.col("mean_shap") - portfolio_mean).exp().alias("relativity"),
398                        pl.lit(float("nan")).alias("lower_ci"),
399                        pl.lit(float("nan")).alias("upper_ci"),
400                    ])
401                else:
402                    agg = normalise_mean(agg, ci_level=ci_level)
403
404            parts.append(agg)
405
406        # Cast the 'level' column to Utf8 in every part before concat.
407        # Categorical features produce level as Utf8; continuous features
408        # produce level as Float64. pl.concat with how="diagonal" cannot
409        # unify mismatched types for the same column name, so we normalise
410        # here rather than requiring callers to pre-cast their feature columns.
411        parts = [
412            p.with_columns(pl.col("level").cast(pl.Utf8))
413            if "level" in p.columns else p
414            for p in parts
415        ]
416
417        result = pl.concat(parts, how="diagonal")
418
419        # Ensure standard column order (wsq_weight is internal, not exported)
420        available = [c for c in _RELATIVITY_COLUMNS if c in result.columns]
421        return result.select(available)
422
423    def extract_continuous_curve(
424        self,
425        feature: str,
426        n_points: int = 100,
427        smooth_method: str = "loess",
428    ) -> pl.DataFrame:
429        """
430        Smoothed relativity curve for a continuous feature.
431
432        Args:
433            feature: Feature name. Must be in continuous_features.
434            n_points: Number of points in the output curve (not the input
435                data).
436            smooth_method: "loess" (locally weighted regression, requires
437                statsmodels), "isotonic" (monotone curve via isotonic
438                regression), or "none" (raw per-observation relativities).
439
440        Returns:
441            Polars DataFrame with columns: feature_value, relativity,
442            lower_ci, upper_ci.
443
444        Raises:
445            ValueError: If feature is not in X or smooth_method is unknown.
446        """
447        self._check_fitted()
448
449        if feature not in self._X.columns:
450            raise ValueError(f"Feature '{feature}' not in X.")
451
452        feat_idx = self._X.columns.index(feature)
453        feat_vals = self._X[feature].to_numpy().astype(float)
454        shap_col = self._shap_values[:, feat_idx]  # type: ignore[index]
455        weights = (
456            self._exposure if self._exposure is not None
457            else np.ones(len(self._X))
458        )
459
460        # Exposure-weighted mean over the actual data distribution
461        portfolio_mean = np.average(shap_col, weights=weights)
462        relativities = np.exp(shap_col - portfolio_mean)
463
464        grid = np.linspace(feat_vals.min(), feat_vals.max(), n_points)
465
466        if smooth_method == "none":
467            order = np.argsort(feat_vals)
468            return pl.DataFrame({
469                "feature_value": feat_vals[order],
470                "relativity": relativities[order],
471                "lower_ci": np.full(len(feat_vals), float("nan")),
472                "upper_ci": np.full(len(feat_vals), float("nan")),
473            })
474
475        elif smooth_method == "isotonic":
476            from sklearn.isotonic import IsotonicRegression
477            ir = IsotonicRegression(out_of_bounds="clip")
478            ir.fit(feat_vals, shap_col, sample_weight=weights)
479            smoothed_shap = ir.predict(grid)
480
481            # P1-4 fix: normalise the smoothed curve so the exposure-weighted
482            # geometric mean of relativities = 1.0.
483            # The smooth is on the data, evaluated on a uniform grid. Subtracting
484            # portfolio_mean (computed on the data distribution) would be correct
485            # only if the grid were distributed like the data — it isn't.
486            # Instead, compute the data-distribution-weighted mean of the smoothed
487            # curve at the original data points, then use that as the reference.
488            smoothed_at_data = ir.predict(feat_vals)
489            weighted_mean_smoothed = np.average(smoothed_at_data, weights=weights)
490            smoothed_rel = np.exp(smoothed_shap - weighted_mean_smoothed)
491
492            return pl.DataFrame({
493                "feature_value": grid,
494                "relativity": smoothed_rel,
495                "lower_ci": np.full(n_points, float("nan")),
496                "upper_ci": np.full(n_points, float("nan")),
497            })
498
499        elif smooth_method == "loess":
500            try:
501                from statsmodels.nonparametric.smoothers_lowess import lowess
502
503                smoothed_shap = lowess(
504                    shap_col, feat_vals, frac=0.3, it=3,
505                    xvals=grid, is_sorted=False,
506                )
507
508                # P1-4 fix: compute the smoothed values at original data points
509                # so the normalisation is data-distribution-weighted, not
510                # grid-uniform. Use the same lowess parameters.
511                smoothed_at_data = lowess(
512                    shap_col, feat_vals, frac=0.3, it=3,
513                    xvals=feat_vals, is_sorted=False,
514                )
515                weighted_mean_smoothed = np.average(smoothed_at_data, weights=weights)
516                smoothed_rel = np.exp(smoothed_shap - weighted_mean_smoothed)
517
518                return pl.DataFrame({
519                    "feature_value": grid,
520                    "relativity": smoothed_rel,
521                    "lower_ci": np.full(n_points, float("nan")),
522                    "upper_ci": np.full(n_points, float("nan")),
523                })
524            except ImportError:
525                warnings.warn(
526                    "statsmodels not installed; falling back to smooth_method='none'.",
527                    UserWarning,
528                    stacklevel=2,
529                )
530                return self.extract_continuous_curve(
531                    feature, n_points=n_points, smooth_method="none"
532                )
533
534        else:
535            raise ValueError(
536                f"Unknown smooth_method '{smooth_method}'. "
537                "Choose from: 'loess', 'isotonic', 'none'."
538            )
539
540    def validate(self) -> dict[str, CheckResult]:
541        """
542        Run diagnostic checks on the SHAP computation.
543
544        Checks performed:
545
546        1. reconstruction: exp(shap.sum(1) + expected_value) should match
547           model predictions within tolerance. Material failure here indicates
548           the explainer was set up incorrectly.
549
550        2. feature_coverage: every feature in X should appear in the SHAP
551           output. Currently always passes given TreeExplainer's API.
552
553        3. sparse_levels: warns if any categorical level has fewer than 30
554           observations. CLT CIs will be unreliable for these levels.
555
556        Returns:
557            Dict with keys "reconstruction", "feature_coverage",
558            "sparse_levels". Each value is a CheckResult(passed, value,
559            message).
560        """
561        self._check_fitted()
562
563        X_pd = _to_pandas(self._X)
564
565        # Get model predictions for reconstruction check
566        preds = None
567        if self._model is not None:
568            try:
569                preds = self._model.predict(X_pd)
570            except Exception:
571                preds = None
572
573        results: dict[str, CheckResult] = {}
574
575        if preds is not None:
576            results["reconstruction"] = check_reconstruction(
577                self._shap_values,  # type: ignore[arg-type]
578                self._expected_value,  # type: ignore[arg-type]
579                preds,
580                tolerance=1e-4,
581            )
582        else:
583            results["reconstruction"] = CheckResult(
584                passed=False,
585                value=float("nan"),
586                message="Could not obtain model predictions for reconstruction check.",
587            )
588
589        feature_names = self._X.columns
590        results["feature_coverage"] = check_feature_coverage(
591            feature_names, feature_names
592        )
593
594        # Check sparse levels for categorical features
595        weights = (
596            self._exposure if self._exposure is not None
597            else np.ones(len(self._X))
598        )
599
600        sparse_parts: list[pl.DataFrame] = []
601        for feat in self._categorical_features:
602            if feat not in self._X.columns:
603                continue
604            feat_idx = self._X.columns.index(feat)
605            agg = aggregate_categorical(
606                feat,
607                self._X[feat].to_numpy(),
608                self._shap_values[:, feat_idx],  # type: ignore[index]
609                weights,
610            )
611            sparse_parts.append(agg)
612
613        if sparse_parts:
614            all_agg = pl.concat(sparse_parts, how="diagonal")
615            results["sparse_levels"] = check_sparse_levels(all_agg)
616        else:
617            results["sparse_levels"] = CheckResult(
618                passed=True, value=0.0,
619                message="No categorical features to check."
620            )
621
622        return results
623
624    def plot_relativities(
625        self,
626        features: list[str] | None = None,
627        show_ci: bool = True,
628        figsize: tuple[int, int] = (12, 8),
629    ) -> None:
630        """
631        Plot relativities as bar charts (categorical) or line charts (continuous).
632
633        Args:
634            features: Subset of features to plot. Defaults to all features.
635            show_ci: Whether to show confidence intervals. Default True.
636            figsize: Overall figure size.
637        """
638        self._check_fitted()
639
640        from ._plotting import plot_relativities as _plot
641
642        rels = self.extract_relativities()
643        _plot(
644            rels,
645            categorical_features=self._categorical_features,
646            continuous_features=self._continuous_features,
647            features=features,
648            show_ci=show_ci,
649            figsize=figsize,
650        )
651
652    def to_dict(self) -> dict[str, Any]:
653        """
654        Serialisable representation of the fitted object.
655
656        Stores SHAP values, expected value, feature names, and feature
657        classification. Does not store the original model or X DataFrame.
658
659        Returns:
660            Dict suitable for JSON serialisation.
661        """
662        self._check_fitted()
663        return {
664            "shap_values": self._shap_values.tolist(),  # type: ignore[union-attr]
665            "expected_value": self._expected_value,
666            "feature_names": self._X.columns,
667            "categorical_features": self._categorical_features,
668            "continuous_features": self._continuous_features,
669            "X_values": {c: self._X[c].to_list() for c in self._X.columns},
670            "exposure": (
671                self._exposure.tolist()
672                if self._exposure is not None else None
673            ),
674            "annualise_exposure": self._annualise_exposure,
675        }
676
677    @classmethod
678    def from_dict(cls, data: dict[str, Any]) -> "SHAPRelativities":
679        """
680        Reconstruct a fitted SHAPRelativities from to_dict() output.
681
682        The reconstructed object has no model attached, so validate() and
683        plot_relativities() still work but fit() cannot be re-run.
684
685        Args:
686            data: Output of to_dict().
687
688        Returns:
689            Fitted SHAPRelativities instance.
690        """
691        # P1-2 fix: use feature_names to control column ordering in X.
692        # Without this, tools that sort JSON object keys (REST APIs, some
693        # pretty-printers) reorder X_values, misaligning columns with the
694        # shap_values matrix columns.
695        feature_names: list[str] = data.get("feature_names", list(data["X_values"].keys()))
696        X = pl.DataFrame({k: data["X_values"][k] for k in feature_names})
697
698        exposure = (
699            np.array(data["exposure"]) if data.get("exposure") is not None
700            else None
701        )
702
703        # Create a minimal instance without a real model
704        instance = cls.__new__(cls)
705        instance._model = None
706        instance._X = X
707        instance._exposure = exposure
708        instance._categorical_features = data.get("categorical_features", [])
709        instance._continuous_features = data.get("continuous_features", [])
710        instance._feature_perturbation = "tree_path_dependent"
711        instance._background_data = None
712        instance._n_background_samples = 1000
713        instance._annualise_exposure = data.get("annualise_exposure", True)
714        instance._shap_values = np.array(data["shap_values"])
715        instance._expected_value = float(data["expected_value"])
716        instance._is_fitted = True
717
718        return instance

Extract multiplicative rating relativities from a tree model via SHAP.

Workflow::

sr = SHAPRelativities(
    model=catboost_model,
    X=df.select(["area", "ncd_years", "has_convictions"]),
    exposure=df["exposure"],
    categorical_features=["area", "ncd_years"],
)
sr.fit()
rels = sr.extract_relativities(
    normalise_to="base_level",
    base_levels={"area": "A", "ncd_years": 0},
)
Arguments:
  • model: A trained CatBoost model. Must use a log-link objective (Poisson, Tweedie, Gamma). CatBoost is the recommended default - it handles categoricals natively.
  • X: Feature matrix. Use training data for in-sample relativities, or a representative holdout sample for out-of-sample. Polars DataFrames are preferred; pandas DataFrames are accepted and converted internally.
  • exposure: Earned policy years (or other volume measure). Used as observation weights throughout. If None, all observations are weighted equally.
  • categorical_features: Features to aggregate by level (bar-chart style). If None, all non-numeric columns are treated as categorical.
  • continuous_features: Features to leave as per-observation points. If None, all numeric columns are treated as continuous.
  • feature_perturbation: "tree_path_dependent" (default, fast, no background data needed) or "interventional" (corrects for feature correlation, needs background_data).
  • background_data: Required only if feature_perturbation="interventional".
  • n_background_samples: Number of background samples for interventional SHAP. Default 1000.
  • annualise_exposure: If True and exposure is provided, subtract mean log(exposure) from the expected_value to give an annualised baseline. Default True.
SHAPRelativities( model: Any, X: Any, exposure: Any = None, categorical_features: list[str] | None = None, continuous_features: list[str] | None = None, background_data: Any = None, feature_perturbation: str = 'tree_path_dependent', n_background_samples: int = 1000, annualise_exposure: bool = True) 
131    def __init__(
132        self,
133        model: Any,
134        X: Any,
135        exposure: Any = None,
136        categorical_features: list[str] | None = None,
137        continuous_features: list[str] | None = None,
138        background_data: Any = None,
139        feature_perturbation: str = "tree_path_dependent",
140        n_background_samples: int = 1000,
141        annualise_exposure: bool = True,
142    ) -> None:
143        if not _SHAP_AVAILABLE:
144            raise ImportError(
145                "shap is required for SHAPRelativities. "
146                "Install it with: uv add 'shap-relativities[ml]'"
147            )
148
149        self._model = model
150        self._X: pl.DataFrame = _to_polars(X)
151        self._background_data = (
152            _to_polars(background_data) if background_data is not None else None
153        )
154
155        # Normalise exposure to a numpy array
156        if exposure is None:
157            self._exposure: np.ndarray | None = None
158        elif isinstance(exposure, np.ndarray):
159            self._exposure = exposure
160        elif isinstance(exposure, pl.Series):
161            self._exposure = exposure.to_numpy()
162        else:
163            # pd.Series or similar
164            self._exposure = np.asarray(exposure)
165
166        # Validate exposure length matches X
167        if self._exposure is not None and len(self._exposure) != len(self._X):
168            raise ValueError(
169                f"exposure length ({len(self._exposure)}) does not match "
170                f"X length ({len(self._X)}). Both must have the same number of rows."
171            )
172
173        self._feature_perturbation = feature_perturbation
174        self._n_background_samples = n_background_samples
175        self._annualise_exposure = annualise_exposure
176
177        # Classify features
178        self._categorical_features = categorical_features or self._infer_categorical()
179        self._continuous_features = continuous_features or self._infer_continuous()
180
181        # Populated by fit()
182        self._shap_values: np.ndarray | None = None
183        self._expected_value: float | None = None
184        self._is_fitted: bool = False
def fit(self) -> SHAPRelativities: 
209    def fit(self) -> "SHAPRelativities":
210        """
211        Compute SHAP values for all features in X.
212
213        Must be called before extract_relativities(). Calling fit() again
214        recomputes SHAP values (e.g. after changing X or the background data).
215
216        The feature matrix is converted to pandas internally for shap's
217        TreeExplainer. The conversion is a necessary bridge - shap requires
218        pandas for column name handling.
219
220        Returns:
221            Self, for method chaining.
222        """
223        # Convert to pandas for shap - unavoidable bridge
224        X_pd = _to_pandas(self._X)
225
226        bg_data = None
227        if self._feature_perturbation == "interventional":
228            if self._background_data is not None:
229                bg_data = _to_pandas(self._background_data)
230            else:
231                n_bg = min(self._n_background_samples, len(X_pd))
232                bg_data = shap.sample(X_pd, n_bg)
233
234        explainer = shap.TreeExplainer(
235            self._model,
236            data=bg_data,
237            feature_perturbation=self._feature_perturbation,
238            model_output="raw",
239        )
240
241        raw = explainer.shap_values(X_pd)
242
243        # Some models return a list when there is a single output
244        if isinstance(raw, list):
245            if len(raw) == 1:
246                raw = raw[0]
247            else:
248                raise ValueError(
249                    f"Model has {len(raw)} outputs. SHAPRelativities supports "
250                    "single-output models only."
251                )
252
253        self._shap_values = raw
254
255        ev = explainer.expected_value
256        if isinstance(ev, (list, np.ndarray)):
257            ev = float(ev[0])
258        self._expected_value = float(ev)
259
260        self._is_fitted = True
261        return self

Compute SHAP values for all features in X.

Must be called before extract_relativities(). Calling fit() again recomputes SHAP values (e.g. after changing X or the background data).

The feature matrix is converted to pandas internally for shap's TreeExplainer. The conversion is a necessary bridge - shap requires pandas for column name handling.

Returns:

Self, for method chaining.

def shap_values(self) -> numpy.ndarray: 
267    def shap_values(self) -> np.ndarray:
268        """
269        Raw SHAP values, shape (n_obs, n_features), in log space.
270
271        Returns:
272            Array of shape (n_obs, n_features).
273        """
274        self._check_fitted()
275        return self._shap_values  # type: ignore[return-value]

Raw SHAP values, shape (n_obs, n_features), in log space.

Returns:

Array of shape (n_obs, n_features).

def baseline(self) -> float: 
277    def baseline(self) -> float:
278        """
279        exp(expected_value) - the base rate in prediction space.
280
281        If annualise_exposure=True and exposure was provided, this is adjusted
282        for the average log-exposure offset so it represents an annualised rate.
283
284        Returns:
285            Base rate as a float.
286        """
287        self._check_fitted()
288        ev = self._expected_value  # type: ignore[assignment]
289
290        if self._annualise_exposure and self._exposure is not None:
291            mean_log_exp = float(np.mean(np.log(np.clip(self._exposure, 1e-9, None))))
292            ev = ev - mean_log_exp
293
294        return float(np.exp(ev))

exp(expected_value) - the base rate in prediction space.

If annualise_exposure=True and exposure was provided, this is adjusted for the average log-exposure offset so it represents an annualised rate.

Returns:

Base rate as a float.

def extract_relativities( self, normalise_to: str = 'base_level', base_levels: dict[str, str | float | int] | None = None, ci_method: str = 'clt', n_bootstrap: int = 200, ci_level: float = 0.95) -> polars.dataframe.frame.DataFrame: 
296    def extract_relativities(
297        self,
298        normalise_to: str = "base_level",
299        base_levels: dict[str, str | float | int] | None = None,
300        ci_method: str = "clt",
301        n_bootstrap: int = 200,
302        ci_level: float = 0.95,
303    ) -> pl.DataFrame:
304        """
305        Extract multiplicative relativities from SHAP values.
306
307        Args:
308            normalise_to: "base_level" (base level for each feature gets
309                relativity = 1.0) or "mean" (exposure-weighted portfolio
310                mean = 1.0).
311            base_levels: Mapping of feature -> base level value. Required for
312                categorical features when normalise_to="base_level". Continuous
313                features automatically use mean normalisation regardless of
314                this setting.
315            ci_method: "clt" (CLT approximation, default, fast) or "none" (no
316                CIs). "bootstrap" is not yet implemented.
317            n_bootstrap: Ignored unless ci_method="bootstrap".
318            ci_level: Two-sided confidence level. Default 0.95.
319
320        Returns:
321            Polars DataFrame with columns: feature, level, relativity,
322            lower_ci, upper_ci, mean_shap, shap_std, n_obs, exposure_weight.
323            One row per (feature, level) combination.
324        """
325        self._check_fitted()
326
327        _VALID_CI_METHODS = {"clt", "bootstrap", "none"}
328        if ci_method not in _VALID_CI_METHODS:
329            raise ValueError(
330                f"Unknown ci_method {ci_method!r}. "
331                f"Valid options are: {sorted(_VALID_CI_METHODS)}."
332            )
333
334        if ci_method == "bootstrap":
335            raise NotImplementedError(
336                "Bootstrap CIs are not yet implemented. Use ci_method='clt'."
337            )
338
339        base_levels = base_levels or {}
340        weights = (
341            self._exposure if self._exposure is not None
342            else np.ones(len(self._X))
343        )
344
345        feature_names = self._X.columns
346        shap_vals = self._shap_values  # type: ignore[assignment]
347
348        parts: list[pl.DataFrame] = []
349
350        for i, feat in enumerate(feature_names):
351            feat_vals = self._X[feat].to_numpy()
352            shap_col = shap_vals[:, i]
353
354            is_categorical = feat in self._categorical_features
355
356            if is_categorical:
357                agg = aggregate_categorical(feat, feat_vals, shap_col, weights)
358            else:
359                agg = aggregate_continuous(feat, feat_vals, shap_col, weights)
360
361            # Normalisation
362            if normalise_to == "base_level" and is_categorical:
363                base = base_levels.get(feat)
364                if base is None:
365                    # Fall back to the level with the smallest mean_shap as
366                    # a sensible default (closest to intercept)
367                    base = agg.sort("mean_shap")["level"][0]
368                    warnings.warn(
369                        f"No base level specified for '{feat}'. "
370                        f"Using '{base}' (lowest mean SHAP) as base.",
371                        UserWarning,
372                        stacklevel=2,
373                    )
374
375                if ci_method == "none":
376                    base_key = str(base)
377                    base_rows = agg.filter(pl.col("level") == base_key)
378                    base_shap = base_rows["mean_shap"][0]
379                    agg = agg.with_columns([
380                        (pl.col("mean_shap") - base_shap).exp().alias("relativity"),
381                        pl.lit(float("nan")).alias("lower_ci"),
382                        pl.lit(float("nan")).alias("upper_ci"),
383                    ])
384                else:
385                    agg = normalise_base_level(agg, base, ci_level=ci_level)
386
387            else:
388                # Mean normalisation for continuous features, or when
389                # normalise_to="mean" for any feature
390                if ci_method == "none":
391                    total_weight = agg["exposure_weight"].sum()
392                    portfolio_mean = float(
393                        (agg["mean_shap"] * agg["exposure_weight"]).sum()
394                        / total_weight
395                    ) if total_weight > 0 else 0.0
396                    agg = agg.with_columns([
397                        (pl.col("mean_shap") - portfolio_mean).exp().alias("relativity"),
398                        pl.lit(float("nan")).alias("lower_ci"),
399                        pl.lit(float("nan")).alias("upper_ci"),
400                    ])
401                else:
402                    agg = normalise_mean(agg, ci_level=ci_level)
403
404            parts.append(agg)
405
406        # Cast the 'level' column to Utf8 in every part before concat.
407        # Categorical features produce level as Utf8; continuous features
408        # produce level as Float64. pl.concat with how="diagonal" cannot
409        # unify mismatched types for the same column name, so we normalise
410        # here rather than requiring callers to pre-cast their feature columns.
411        parts = [
412            p.with_columns(pl.col("level").cast(pl.Utf8))
413            if "level" in p.columns else p
414            for p in parts
415        ]
416
417        result = pl.concat(parts, how="diagonal")
418
419        # Ensure standard column order (wsq_weight is internal, not exported)
420        available = [c for c in _RELATIVITY_COLUMNS if c in result.columns]
421        return result.select(available)

Extract multiplicative relativities from SHAP values.

Arguments:
  • normalise_to: "base_level" (base level for each feature gets relativity = 1.0) or "mean" (exposure-weighted portfolio mean = 1.0).
  • base_levels: Mapping of feature -> base level value. Required for categorical features when normalise_to="base_level". Continuous features automatically use mean normalisation regardless of this setting.
  • ci_method: "clt" (CLT approximation, default, fast) or "none" (no CIs). "bootstrap" is not yet implemented.
  • n_bootstrap: Ignored unless ci_method="bootstrap".
  • ci_level: Two-sided confidence level. Default 0.95.
Returns:

Polars DataFrame with columns: feature, level, relativity, lower_ci, upper_ci, mean_shap, shap_std, n_obs, exposure_weight. One row per (feature, level) combination.

def extract_continuous_curve( self, feature: str, n_points: int = 100, smooth_method: str = 'loess') -> polars.dataframe.frame.DataFrame: 
423    def extract_continuous_curve(
424        self,
425        feature: str,
426        n_points: int = 100,
427        smooth_method: str = "loess",
428    ) -> pl.DataFrame:
429        """
430        Smoothed relativity curve for a continuous feature.
431
432        Args:
433            feature: Feature name. Must be in continuous_features.
434            n_points: Number of points in the output curve (not the input
435                data).
436            smooth_method: "loess" (locally weighted regression, requires
437                statsmodels), "isotonic" (monotone curve via isotonic
438                regression), or "none" (raw per-observation relativities).
439
440        Returns:
441            Polars DataFrame with columns: feature_value, relativity,
442            lower_ci, upper_ci.
443
444        Raises:
445            ValueError: If feature is not in X or smooth_method is unknown.
446        """
447        self._check_fitted()
448
449        if feature not in self._X.columns:
450            raise ValueError(f"Feature '{feature}' not in X.")
451
452        feat_idx = self._X.columns.index(feature)
453        feat_vals = self._X[feature].to_numpy().astype(float)
454        shap_col = self._shap_values[:, feat_idx]  # type: ignore[index]
455        weights = (
456            self._exposure if self._exposure is not None
457            else np.ones(len(self._X))
458        )
459
460        # Exposure-weighted mean over the actual data distribution
461        portfolio_mean = np.average(shap_col, weights=weights)
462        relativities = np.exp(shap_col - portfolio_mean)
463
464        grid = np.linspace(feat_vals.min(), feat_vals.max(), n_points)
465
466        if smooth_method == "none":
467            order = np.argsort(feat_vals)
468            return pl.DataFrame({
469                "feature_value": feat_vals[order],
470                "relativity": relativities[order],
471                "lower_ci": np.full(len(feat_vals), float("nan")),
472                "upper_ci": np.full(len(feat_vals), float("nan")),
473            })
474
475        elif smooth_method == "isotonic":
476            from sklearn.isotonic import IsotonicRegression
477            ir = IsotonicRegression(out_of_bounds="clip")
478            ir.fit(feat_vals, shap_col, sample_weight=weights)
479            smoothed_shap = ir.predict(grid)
480
481            # P1-4 fix: normalise the smoothed curve so the exposure-weighted
482            # geometric mean of relativities = 1.0.
483            # The smooth is on the data, evaluated on a uniform grid. Subtracting
484            # portfolio_mean (computed on the data distribution) would be correct
485            # only if the grid were distributed like the data — it isn't.
486            # Instead, compute the data-distribution-weighted mean of the smoothed
487            # curve at the original data points, then use that as the reference.
488            smoothed_at_data = ir.predict(feat_vals)
489            weighted_mean_smoothed = np.average(smoothed_at_data, weights=weights)
490            smoothed_rel = np.exp(smoothed_shap - weighted_mean_smoothed)
491
492            return pl.DataFrame({
493                "feature_value": grid,
494                "relativity": smoothed_rel,
495                "lower_ci": np.full(n_points, float("nan")),
496                "upper_ci": np.full(n_points, float("nan")),
497            })
498
499        elif smooth_method == "loess":
500            try:
501                from statsmodels.nonparametric.smoothers_lowess import lowess
502
503                smoothed_shap = lowess(
504                    shap_col, feat_vals, frac=0.3, it=3,
505                    xvals=grid, is_sorted=False,
506                )
507
508                # P1-4 fix: compute the smoothed values at original data points
509                # so the normalisation is data-distribution-weighted, not
510                # grid-uniform. Use the same lowess parameters.
511                smoothed_at_data = lowess(
512                    shap_col, feat_vals, frac=0.3, it=3,
513                    xvals=feat_vals, is_sorted=False,
514                )
515                weighted_mean_smoothed = np.average(smoothed_at_data, weights=weights)
516                smoothed_rel = np.exp(smoothed_shap - weighted_mean_smoothed)
517
518                return pl.DataFrame({
519                    "feature_value": grid,
520                    "relativity": smoothed_rel,
521                    "lower_ci": np.full(n_points, float("nan")),
522                    "upper_ci": np.full(n_points, float("nan")),
523                })
524            except ImportError:
525                warnings.warn(
526                    "statsmodels not installed; falling back to smooth_method='none'.",
527                    UserWarning,
528                    stacklevel=2,
529                )
530                return self.extract_continuous_curve(
531                    feature, n_points=n_points, smooth_method="none"
532                )
533
534        else:
535            raise ValueError(
536                f"Unknown smooth_method '{smooth_method}'. "
537                "Choose from: 'loess', 'isotonic', 'none'."
538            )

Smoothed relativity curve for a continuous feature.

Arguments:
  • feature: Feature name. Must be in continuous_features.
  • n_points: Number of points in the output curve (not the input data).
  • smooth_method: "loess" (locally weighted regression, requires statsmodels), "isotonic" (monotone curve via isotonic regression), or "none" (raw per-observation relativities).
Returns:

Polars DataFrame with columns: feature_value, relativity, lower_ci, upper_ci.

Raises:
  • ValueError: If feature is not in X or smooth_method is unknown.
def validate(self) -> dict[str, shap_relativities._validation.CheckResult]: 
540    def validate(self) -> dict[str, CheckResult]:
541        """
542        Run diagnostic checks on the SHAP computation.
543
544        Checks performed:
545
546        1. reconstruction: exp(shap.sum(1) + expected_value) should match
547           model predictions within tolerance. Material failure here indicates
548           the explainer was set up incorrectly.
549
550        2. feature_coverage: every feature in X should appear in the SHAP
551           output. Currently always passes given TreeExplainer's API.
552
553        3. sparse_levels: warns if any categorical level has fewer than 30
554           observations. CLT CIs will be unreliable for these levels.
555
556        Returns:
557            Dict with keys "reconstruction", "feature_coverage",
558            "sparse_levels". Each value is a CheckResult(passed, value,
559            message).
560        """
561        self._check_fitted()
562
563        X_pd = _to_pandas(self._X)
564
565        # Get model predictions for reconstruction check
566        preds = None
567        if self._model is not None:
568            try:
569                preds = self._model.predict(X_pd)
570            except Exception:
571                preds = None
572
573        results: dict[str, CheckResult] = {}
574
575        if preds is not None:
576            results["reconstruction"] = check_reconstruction(
577                self._shap_values,  # type: ignore[arg-type]
578                self._expected_value,  # type: ignore[arg-type]
579                preds,
580                tolerance=1e-4,
581            )
582        else:
583            results["reconstruction"] = CheckResult(
584                passed=False,
585                value=float("nan"),
586                message="Could not obtain model predictions for reconstruction check.",
587            )
588
589        feature_names = self._X.columns
590        results["feature_coverage"] = check_feature_coverage(
591            feature_names, feature_names
592        )
593
594        # Check sparse levels for categorical features
595        weights = (
596            self._exposure if self._exposure is not None
597            else np.ones(len(self._X))
598        )
599
600        sparse_parts: list[pl.DataFrame] = []
601        for feat in self._categorical_features:
602            if feat not in self._X.columns:
603                continue
604            feat_idx = self._X.columns.index(feat)
605            agg = aggregate_categorical(
606                feat,
607                self._X[feat].to_numpy(),
608                self._shap_values[:, feat_idx],  # type: ignore[index]
609                weights,
610            )
611            sparse_parts.append(agg)
612
613        if sparse_parts:
614            all_agg = pl.concat(sparse_parts, how="diagonal")
615            results["sparse_levels"] = check_sparse_levels(all_agg)
616        else:
617            results["sparse_levels"] = CheckResult(
618                passed=True, value=0.0,
619                message="No categorical features to check."
620            )
621
622        return results

Run diagnostic checks on the SHAP computation.

Checks performed:

  1. reconstruction: exp(shap.sum(1) + expected_value) should match model predictions within tolerance. Material failure here indicates the explainer was set up incorrectly.

  2. feature_coverage: every feature in X should appear in the SHAP output. Currently always passes given TreeExplainer's API.

  3. sparse_levels: warns if any categorical level has fewer than 30 observations. CLT CIs will be unreliable for these levels.

Returns:

Dict with keys "reconstruction", "feature_coverage", "sparse_levels". Each value is a CheckResult(passed, value, message).

def plot_relativities( self, features: list[str] | None = None, show_ci: bool = True, figsize: tuple[int, int] = (12, 8)) -> None: 
624    def plot_relativities(
625        self,
626        features: list[str] | None = None,
627        show_ci: bool = True,
628        figsize: tuple[int, int] = (12, 8),
629    ) -> None:
630        """
631        Plot relativities as bar charts (categorical) or line charts (continuous).
632
633        Args:
634            features: Subset of features to plot. Defaults to all features.
635            show_ci: Whether to show confidence intervals. Default True.
636            figsize: Overall figure size.
637        """
638        self._check_fitted()
639
640        from ._plotting import plot_relativities as _plot
641
642        rels = self.extract_relativities()
643        _plot(
644            rels,
645            categorical_features=self._categorical_features,
646            continuous_features=self._continuous_features,
647            features=features,
648            show_ci=show_ci,
649            figsize=figsize,
650        )

Plot relativities as bar charts (categorical) or line charts (continuous).

Arguments:
  • features: Subset of features to plot. Defaults to all features.
  • show_ci: Whether to show confidence intervals. Default True.
  • figsize: Overall figure size.
def to_dict(self) -> dict[str, typing.Any]: 
652    def to_dict(self) -> dict[str, Any]:
653        """
654        Serialisable representation of the fitted object.
655
656        Stores SHAP values, expected value, feature names, and feature
657        classification. Does not store the original model or X DataFrame.
658
659        Returns:
660            Dict suitable for JSON serialisation.
661        """
662        self._check_fitted()
663        return {
664            "shap_values": self._shap_values.tolist(),  # type: ignore[union-attr]
665            "expected_value": self._expected_value,
666            "feature_names": self._X.columns,
667            "categorical_features": self._categorical_features,
668            "continuous_features": self._continuous_features,
669            "X_values": {c: self._X[c].to_list() for c in self._X.columns},
670            "exposure": (
671                self._exposure.tolist()
672                if self._exposure is not None else None
673            ),
674            "annualise_exposure": self._annualise_exposure,
675        }

Serialisable representation of the fitted object.

Stores SHAP values, expected value, feature names, and feature classification. Does not store the original model or X DataFrame.

Returns:

Dict suitable for JSON serialisation.

@classmethod
def from_dict( cls, data: dict[str, typing.Any]) -> SHAPRelativities: 
677    @classmethod
678    def from_dict(cls, data: dict[str, Any]) -> "SHAPRelativities":
679        """
680        Reconstruct a fitted SHAPRelativities from to_dict() output.
681
682        The reconstructed object has no model attached, so validate() and
683        plot_relativities() still work but fit() cannot be re-run.
684
685        Args:
686            data: Output of to_dict().
687
688        Returns:
689            Fitted SHAPRelativities instance.
690        """
691        # P1-2 fix: use feature_names to control column ordering in X.
692        # Without this, tools that sort JSON object keys (REST APIs, some
693        # pretty-printers) reorder X_values, misaligning columns with the
694        # shap_values matrix columns.
695        feature_names: list[str] = data.get("feature_names", list(data["X_values"].keys()))
696        X = pl.DataFrame({k: data["X_values"][k] for k in feature_names})
697
698        exposure = (
699            np.array(data["exposure"]) if data.get("exposure") is not None
700            else None
701        )
702
703        # Create a minimal instance without a real model
704        instance = cls.__new__(cls)
705        instance._model = None
706        instance._X = X
707        instance._exposure = exposure
708        instance._categorical_features = data.get("categorical_features", [])
709        instance._continuous_features = data.get("continuous_features", [])
710        instance._feature_perturbation = "tree_path_dependent"
711        instance._background_data = None
712        instance._n_background_samples = 1000
713        instance._annualise_exposure = data.get("annualise_exposure", True)
714        instance._shap_values = np.array(data["shap_values"])
715        instance._expected_value = float(data["expected_value"])
716        instance._is_fitted = True
717
718        return instance

Reconstruct a fitted SHAPRelativities from to_dict() output.

The reconstructed object has no model attached, so validate() and plot_relativities() still work but fit() cannot be re-run.

Arguments:
  • data: Output of to_dict().
Returns:

Fitted SHAPRelativities instance.

def extract_relativities( model: Any, X: Any, exposure: Any = None, categorical_features: list[str] | None = None, base_levels: dict[str, str | float | int] | None = None, ci_method: str = 'clt') -> polars.dataframe.frame.DataFrame: 
46def extract_relativities(
47    model: Any,
48    X: Any,
49    exposure: Any = None,
50    categorical_features: list[str] | None = None,
51    base_levels: dict[str, str | float | int] | None = None,
52    ci_method: str = "clt",
53) -> pl.DataFrame:
54    """
55    One-shot extraction of SHAP relativities from a tree model.
56
57    Wraps SHAPRelativities.fit() and extract_relativities() for cases where
58    you don't need the intermediate object.
59
60    Args:
61        model: Trained CatBoost model with a log-link objective (Poisson,
62            Tweedie, or Gamma). CatBoost is the recommended choice - it handles
63            categorical features natively without encoding.
64        X: Feature matrix. Accepts a Polars or pandas DataFrame. Polars is
65            preferred; pandas is accepted and converted internally.
66        exposure: Earned policy years. If None, all observations are equally
67            weighted.
68        categorical_features: Features to aggregate by level. If None, all
69            non-numeric columns are treated as categorical.
70        base_levels: Base level for each categorical feature (gets
71            relativity = 1.0).
72        ci_method: "clt" (default) or "none".
73
74    Returns:
75        Polars DataFrame with columns: feature, level, relativity, lower_ci,
76        upper_ci, mean_shap, shap_std, n_obs, exposure_weight.
77    """
78    sr = SHAPRelativities(model, X, exposure, categorical_features)
79    sr.fit()
80    return sr.extract_relativities(base_levels=base_levels, ci_method=ci_method)

One-shot extraction of SHAP relativities from a tree model.

Wraps SHAPRelativities.fit() and extract_relativities() for cases where you don't need the intermediate object.

Arguments:
  • model: Trained CatBoost model with a log-link objective (Poisson, Tweedie, or Gamma). CatBoost is the recommended choice - it handles categorical features natively without encoding.
  • X: Feature matrix. Accepts a Polars or pandas DataFrame. Polars is preferred; pandas is accepted and converted internally.
  • exposure: Earned policy years. If None, all observations are equally weighted.
  • categorical_features: Features to aggregate by level. If None, all non-numeric columns are treated as categorical.
  • base_levels: Base level for each categorical feature (gets relativity = 1.0).
  • ci_method: "clt" (default) or "none".
Returns:

Polars DataFrame with columns: feature, level, relativity, lower_ci, upper_ci, mean_shap, shap_std, n_obs, exposure_weight.