Metadata-Version: 2.4
Name: polars-list-math
Version: 0.7.0
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Topic :: Scientific/Engineering :: Mathematics
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Dist: polars>=1.39.3
License-File: LICENSE
Summary: List-oriented math and expression helpers for Polars.
Keywords: polars,list,dataframe,rust,pyo3
Home-Page: https://github.com/rufrozen/polars-list-math
License-Expression: MIT
Requires-Python: >=3.12
Description-Content-Type: text/markdown
Project-URL: Documentation, https://github.com/rufrozen/polars-list-math/tree/main/docs
Project-URL: Homepage, https://github.com/rufrozen/polars-list-math
Project-URL: Issues, https://github.com/rufrozen/polars-list-math/issues
Project-URL: Repository, https://github.com/rufrozen/polars-list-math

# polars-list-math

`polars-list-math` helps you work with Polars DataFrames in which every field
of a nested list of structs is represented by its own column. Each field
column preserves the original list nesting, so corresponding values remain
aligned by row and by position at every list level.

For example, a nested list of records with `price` and `qty` fields can be
represented as two nested list columns in the same DataFrame:

```mermaid
flowchart LR
    records["orders: List[List[Struct]]<br/>[[{price: 10, qty: 2}, {price: 20, qty: 1}],<br/>[{price: 30, qty: 4}]]"]
    subgraph columns["Separate field columns — same DataFrame row"]
        price["orders.price: List[List[Int64]]<br/>[[10, 20], [30]]"]
        qty["orders.qty: List[List[Int64]]<br/>[[2, 1], [4]]"]
    end
    records -->|price| price
    records -->|qty| qty
```

Each level of lists keeps its grouping, and each field becomes independently
accessible as a column. The same representation extends to deeper nesting.
The dotted column names above are illustrative.

The library provides Python/Rust helpers for calculations on these list
columns, comparing lists, and combining corresponding fields with
[`list_zip`](docs/list_zip.md) and [`ZipBuilder`](docs/zip_builder.md).
It also includes JSON and URL expressions, null-field discovery, and schema
cleaning for preparing and transforming DataFrames and LazyFrames.

Release notes and upgrade guidance are recorded in [CHANGELOG.md](CHANGELOG.md).

Import the package once to register extra methods on `Expr.list` and `Expr.str`:

```python
import polars as pl
import polars_list_math  # noqa: F401
```

For type-checked code, import the top-level expression helpers described below.

## Install

```bash
pip install polars-list-math
# or
uv add polars-list-math
```

Requires Python 3.12+ and `polars>=1.39.3`.

## Functions

Top-level functions provide the statically typed API for Pyright, Pylance,
and other type checkers. Where available, the optional `Method` column lists
an equivalent runtime method registered when the package is imported.

| Function | Method (optional) | Result | Docs |
| --- | --- | --- | --- |
| `list_zip(...)` | `Expr.list.zip(...)` | Zip lists into `list[struct]` | [List zip](docs/list_zip.md) |
| `list_combinations(...)` | `Expr.list.combinations(...)` | Pair items in one list | [Combinations](https://github.com/rufrozen/polars-list-math/blob/main/docs/combinations.md) |
| `list_combinations_to(...)` | `Expr.list.combinations_to(...)` | Pair items from two lists | [Combinations](https://github.com/rufrozen/polars-list-math/blob/main/docs/combinations.md) |
| `expected_value_of_game(...)` | — | Expected value of a sequential game | [Expected value of a game](https://github.com/rufrozen/polars-list-math/blob/main/docs/expected_value_of_game.md) |
| `list_similarity(...)` | `Expr.list.similarity(...)` | Weighted similarity between two lists | [Similarity](https://github.com/rufrozen/polars-list-math/blob/main/docs/similarity.md) |
| `list_mean_similarity(...)` | `Expr.list.mean_similarity(...)` | Mean similarity inside nested lists | [Similarity](https://github.com/rufrozen/polars-list-math/blob/main/docs/similarity.md) |
| `list_mean_similarity_to(...)` | `Expr.list.mean_similarity_to(...)` | Mean similarity to reference nested lists | [Similarity](https://github.com/rufrozen/polars-list-math/blob/main/docs/similarity.md) |
| `list_containment(...)` | `Expr.list.containment(...)` | Fraction of unique IDs found in another list | [Containment](docs/containment.md) |
| `list_mean_containment(...)` | `Expr.list.mean_containment(...)` | Mean directed containment against peer lists | [Containment](docs/containment.md) |
| `list_mean_containment_to(...)` | `Expr.list.mean_containment_to(...)` | Mean directed containment against reference lists | [Containment](docs/containment.md) |
| `json_object_items(expr, strict=False)` | `Expr.str.json_object_items(...)` | Convert a JSON object to `list[struct[key, value]]` | [JSON object items](https://github.com/rufrozen/polars-list-math/blob/main/docs/json_object_items.md) |
| `json_array_values(expr, strict=False)` | `Expr.str.json_array_values(...)` | Convert a JSON array to `list[str]` | [JSON array values](https://github.com/rufrozen/polars-list-math/blob/main/docs/json_array_values.md) |
| `url_query_encode(expr, doseq=False)` | — | Encode a Struct or key/value list as a URL query string | [URL query encoding](https://github.com/rufrozen/polars-list-math/blob/main/docs/url_query_encode.md) |
| `url_build(...)` | — | Assemble a URL from optional component expressions | [URL building](https://github.com/rufrozen/polars-list-math/blob/main/docs/url_build.md) |
| `dfs_to_lazy_df(...)` | — | Expose DataFrame batches as a pushdown-aware LazyFrame source | [Examples](https://github.com/rufrozen/polars-list-math/tree/main/examples) |
| `find_null_schema(lf)` | — | Discover absent columns and nested fields | [Null schemas](docs/null_schema.md) |
| `clean_schema(schema, subschema)` | — | Remove columns and fields described by a subschema | [Null schemas](docs/null_schema.md) |
| `apply_schema(lf, schema)` | — | Select and cast a LazyFrame to the cleaned schema | [Null schemas](docs/null_schema.md) |

The Python helper `py_list_similarity(...)` computes weighted similarity for
plain Python sequences. `py_list_containment(...)` computes directed set
containment for Python sequences, ignoring ranks and duplicates.

The `Expr.list` and `Expr.str` methods are registered dynamically and remain
available as convenient runtime syntax. Python type checkers cannot safely
augment Polars' own namespace classes from another installed package, so use
the equivalent top-level functions in code checked by Pylance or Pyright.

If a future Polars release ships native methods with the same names, this
package leaves the native implementation untouched.

## Quick Examples

```python
from polars_list_math import (
    list_combinations,
    list_mean_similarity,
    list_similarity,
    list_zip,
)

df = pl.DataFrame(
    {
        "a": [[1, 2, 3]],
        "b": [[2, 1, 3]],
        "groups": [[[1, 2, 3], [1, 2, 3], [4, 5, 6]]],
    }
)

df.with_columns(
    list_similarity("a", "b").alias("similarity"),
    list_mean_similarity("groups").alias("mean_similarity"),
    list_combinations("a").alias("pairs"),
    list_zip("a", "b", fields=["a", "b"]).alias("zipped"),
)
```

Convert JSON strings to nested Polars values. String values stay strings, JSON
null stays null, and other values become compact JSON strings. Invalid JSON
returns null by default; pass `strict=True` to raise instead.

```python
from polars_list_math import json_array_values, json_object_items

json_df = pl.DataFrame(
    {
        "object": ['{"name":"Ada","active":true,"note":null}'],
        "array": ['["python",42,true,null]'],
    }
)

json_df.select(
    json_object_items("object").alias("items"),
    json_array_values("array").alias("values"),
)
```

Encode query parameters and build URLs from expressions:

```python
from polars_list_math import url_build, url_query_encode

pages = pl.DataFrame(
    {
        "host": ["example.com"],
        "path": ["/search"],
        "term": ["rust polars"],
        "page": [2],
    }
)

query = url_query_encode(pl.struct(q=pl.col("term"), page=pl.col("page")))

pages.select(
    url_build(
        scheme=pl.lit("https"),
        netloc="host",
        path="path",
        query=query,
    ).alias("url")
)
```

More complete, runnable examples are available in the
[`examples`](https://github.com/rufrozen/polars-list-math/tree/main/examples)
directory.

## Nested zip builder

`ZipBuilder` combines nested list columns with zip options and ordered eval/filter
operations at each level. `build()` returns a normal Polars expression:

```python
from polars_list_math import ZipBuilder

expr = (
    ZipBuilder("a", "b")
    .level(pad=True)
    .filter(pl.element().struct.field("a").is_not_null())
    .build("nested_zip")
)
```

The constructor configures the outer zip; each `level()` adds one inner zip.
See the [builder guide](docs/zip_builder.md) for transformations, broadcasting,
and deeper nesting.

## Performance experiments

Reproducible performance experiments live in the [`performance`](performance)
directory. Each experiment has its own README with the tested scenario, run
instructions, environment, and recorded results.

The current
[`dataframe_construction`](performance/dataframe_construction) experiment
compares construction of 30-column Polars DataFrames from a regular dataclass,
a slotted dataclass, and a named tuple, including nested structs and lists.

```bash
uv run pytest performance/dataframe_construction -v -s
```

## Development

```bash
make install
make develop
make test
```

Build and check the package locally:

```bash
make check-dist
```

Release steps are in
[docs/publishing.md](https://github.com/rufrozen/polars-list-math/blob/main/docs/publishing.md).

