Metadata-Version: 2.4
Name: polartox
Version: 0.5.0
Summary: NLP toolkit for annotator polarization research: synthetic data generation, Polarized Trees, and hyperparameter benchmarking
Author-email: SwkratisCS <swkratisgiannoutsos@gmail.com>
License-Expression: MIT
Project-URL: Repository, https://github.com/Swkratis210204/polartox
Keywords: polarization,annotation,synthetic data,nlp,disagreement,crowdsourcing,demographics,polarized trees,hyperparameter optimization,model selection
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: ndfu
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# polartox

NLP toolkit for **annotator polarization research**. Provides tools for synthetic annotation-data generation, Polarized Trees analysis, and systematic hyperparameter benchmarking.

## Install

```bash
pip install polartox
```

## Tools

| Module | Description | Status |
|---|---|---|
| `polartox.datagen` | Synthetic annotator pool with injected, ground-truth polarization | Stable |
| `polartox.polarized_trees` | Polarized Trees detection algorithm | Stable |
| `polartox.benchmark` | Hyperparameter search and model selection for Polarized Trees | Stable |

## `polartox.datagen`

Builds a pool of annotators with explicit demographic identities and generates annotation datasets where every text independently gets **k active dimensions** that drive its disagreement:

- **k = 0** — no demographic dimension explains the disagreement, providing a true unimodal negative control
- **k ≥ 1** — a subset of dimensions is active, each with its own random toxic/civil lean split and a continuous intensity (`alpha`) controlling how strongly it pulls toward its pole

Identities' rating distributions are built by taking the **elementwise product** of their active-dimension shapes — signal composes rather than averages away, allowing the generated data to span the full nDFU range.

```python
from polartox.datagen import (
    AnnotatorPool,
    DEFAULT_DIMENSIONS,
    DEFAULT_DEPTH_WEIGHTS,
    DEFAULT_INTENSITY_RANGE,
)

pool = AnnotatorPool(
    dimensions=DEFAULT_DIMENSIONS,
    scale=5,
    intensity_range=DEFAULT_INTENSITY_RANGE,
    depth_weights=DEFAULT_DEPTH_WEIGHTS,
    annotators_per_identity=10,
)

result = pool.generate_dataset(
    n_texts=100,
    n_annotators_per_text=150,
    noise=0.05,
    seed=42,
)

dataset, ground_truth = result

# dataset columns:
# text_id, annotator_id, <dimensions>, rating

# ground_truth:
# per-text active_dims, lean, and alpha
```

nDFU scoring is provided by the collaborative [`ndfu`](https://github.com/ipavlopoulos/ndfu) package (Pavlopoulos & Likas, 2024) rather than reimplemented here:

```python
from ndfu import dfu, pdf

text_data = dataset[dataset["text_id"] == 0]
hist = pdf(
    text_data["rating"].tolist(),
    range(1, pool.scale + 1),
)
score = dfu(hist)
```

## `polartox.polarized_trees`

Given an annotation dataset, recursively partitions annotators by demographic dimension to identify the dimensions and intersectional subgroups associated with polarized opinions.

```python
from polartox.polarized_trees import PolarizedTreesPipeline

pipe = PolarizedTreesPipeline(
    dims=list(DEFAULT_DIMENSIONS.keys()),
    scale=5,
    theta_filter=0.3,
    min_size_frac=0.03,
    h=0.15,
    max_depth=6,
    relative_h=True,
)

results = pipe.run_full_evaluation(dataset)

# results["F"], results["C"], results["P"]
# dataset-level polarization summaries

# results["diagnostics"]
# ground-truth-free diagnostics
```

When synthetic ground truth is available, recovery metrics can also be computed:

```python
results = pipe.run_full_evaluation(
    dataset,
    ground_truth=ground_truth,
)

results["recovery"]
# per-text Jaccard, precision, recall, and exact match
```

## `polartox.benchmark`

`PolarizedTreesBenchmark` provides systematic configuration search and model selection when ground truth is available.

The benchmark:

1. takes a `PolarizedTreesPipeline`, annotations, and ground truth;
2. generates valid configurations from a search space;
3. evaluates each configuration using the requested recovery metrics;
4. ranks configurations according to a selected metric;
5. returns the best configuration, best pipeline, top configurations, and complete results;
6. can save benchmark results and a JSON report.

The default search space corresponds to the configuration space used in the paper and contains **3,240 valid configurations**.

The PRG variants use the following valid `beta` combinations:

```text
max  → beta = 1.0
var  → beta = 1.0
beta → beta = 0.5, 1.0, 2.0
```

The benchmark is fully configurable. Users can provide their own search space, choose `full` or `random` search, set the number of sampled configurations and seed, select recovery metrics, and specify the metric and direction used for model selection.

A simple example:

```python
from polartox.benchmark import PolarizedTreesBenchmark

benchmark = PolarizedTreesBenchmark(
    pipeline=pipe,
    annotations=dataset,
    ground_truth=ground_truth,
    strategy="random",
    n_runs=100,
    seed=42,
    metrics=["jaccard", "precision", "recall"],
    selection_metric="jaccard",
)

benchmark.run()

best_config = benchmark.get_best_config()
best_score = benchmark.get_best_score()
top_configs = benchmark.get_top_configs()
best_pipeline = benchmark.get_best_pipeline()
```

The benchmark can also be used with a custom search space:

```python
custom_search_space = {
    "theta_filter": [0.2, 0.3],
    "min_size_frac": [0.03, 0.05],
    "max_depth": [4, 6],
    "variant": ["beta"],
    "h": [0.10, 0.15],
    "relative_h": [True],
    "theta_stop": [0.10, 0.15],
}

benchmark = PolarizedTreesBenchmark(
    pipeline=pipe,
    annotations=dataset,
    ground_truth=ground_truth,
    search_space=custom_search_space,
    strategy="full",
)
```

The selected configuration can subsequently be applied to new annotation data without ground truth. In this inference setting, Polarized Trees reports **F, C, and P** together with the associated diagnostics.

## Reproducibility

The repository includes a complete synthetic benchmark workflow under [`benchmarks/`](https://github.com/Swkratis210204/polartox/tree/main/benchmarks).

The workflow is:

```text
Synthetic data generation
          ↓
Fixed datasets + ground truth
          ↓
Configuration search
          ↓
Recovery-based model selection
          ↓
Selected Polarized Trees pipeline
          ↓
Inference on unseen data
```

The `treesbenchmark.ipynb` notebook serves both as a runnable demonstration of `PolarizedTreesBenchmark` and as the experimental workflow used to obtain the configurations and results reported in the paper.

The benchmark documentation, configuration space, recovery results, and unseen-corpus evaluation are described in [`benchmarks/README.md`](https://github.com/Swkratis210204/polartox/blob/main/benchmarks/README.md).

## Documentation and Paper

Full API documentation and demos are available on [GitHub](https://github.com/Swkratis210204/polartox).

For the full methodology and paper-specific details, see [`polarized_trees.pdf`](https://github.com/Swkratis210204/polartox/blob/main/polarized_trees/polarized_trees.pdf).

## Changelog

See the [CHANGELOG](https://github.com/Swkratis210204/polartox/blob/main/CHANGELOG.md) for release history.
