Metadata-Version: 2.4
Name: dataframework-column-tools
Version: 1.0.1
Summary: Polars data loading, multi-dataset column mapping, transformation tracking, and readable reports
Author: DataFramework Contributors
License: MIT License
        
        Copyright (c) 2026
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://example.com/dataframework-column-tools
Project-URL: Repository, https://example.com/dataframework-column-tools
Project-URL: Issues, https://example.com/dataframework-column-tools/issues
Keywords: polars,dataframe,data-cleaning,data-mapping,column-mapping,jupyter,tkinter
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries
Requires-Python: <3.15,>=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: polars<2,>=1.44
Requires-Dist: numpy<3,>=2
Requires-Dist: reportlab<5,>=4.2
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: twine>=6; extra == "dev"
Requires-Dist: pytest>=8; extra == "dev"
Dynamic: license-file

# DataFramework Column Tools

`dataframework-column-tools` is a modular Python package for loading multiple datasets into Polars, interactively mapping semantically similar columns across datasets, applying one Polars transformation function to the final selection, tracking exactly what changed, and exporting one readable HTML/PDF report.

The package is designed as a **library/framework component**, not as a standalone application. It does not create user accounts, dashboards, or a database.

## Python support

- Python 3.10–3.14
- Tkinter is used only for the optional interactive mapping window and is part of the normal Python installation on Windows. Some Linux distributions require the system `python3-tk` package.

## Install

After publishing to PyPI:

```bash
python -m pip install dataframework-column-tools
```

For local development:

```powershell
python -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt
python -m pip install -e .
```

## Load multiple datasets

```python
from dataframework import load_files

loaded = load_files(
    "customers.csv",
    "crm.csv",
    "archive.xlsx",
    "events.json",
    namespace=globals(),
)
```

The loader automatically creates names such as:

```text
df_file_1_csv
df_file_2_csv
df_file_1_excel
df_file_1_json
```

Counters are independent by logical file type. Each source file is read once. The returned dictionary and optional notebook namespace point to the same Polars DataFrame objects; the loader does not call `.clone()` or convert the data to NumPy during ingestion.

Supported formats:

- CSV
- TSV
- Excel (`.xlsx`, `.xls`)
- JSON
- NDJSON / JSONL
- Parquet

Excel support uses Polars' configured Excel engine. Depending on your file/environment, an additional Excel engine package may be required by the Polars installation.

## Interactive column mapping

A transformation receives a Polars expression and the target column metadata:

```python
import polars as pl
from dataframework import column


def normalize_phone(expr: pl.Expr, column_name: str, dataset_name: str) -> pl.Expr:
    return expr.cast(pl.String).str.replace_all(r"[^0-9+]", "")


datasets = {
    "customers": df_file_1_csv,
    "crm": df_file_2_csv,
    "archive": df_file_1_excel,
}

column(normalize_phone, "customers", "phone", datasets)
```

The Tkinter window keeps the target column in the center and can show matching columns from any number of other datasets. The user can:

- accept automatic semantic suggestions,
- add another column manually,
- remove any selected connection,
- drag nodes,
- click a node to inspect dtype, semantic type, confidence, null count, unique count, and up to 20 random non-null values,
- press **Done** to run the function on every final selected column.

The mapper uses semantic type, column-name similarity, and physical dtype compatibility rather than requiring exact column names.

## Tracking and reports

Every applied transformation prints a status line immediately:

```text
[dataframework] DONE    df_file_1_csv.phone                 function=normalize_phone changed_elements=8421 rows=10000->10000 columns=12->12
```

The execution report records, per operation:

- UTC timestamp
- function name
- dataset and column
- semantic type
- rows before/after
- columns before/after
- column element count before/after
- exact changed-cell count
- changed-row count
- null counts before/after
- unique counts before/after
- non-null counts before/after
- notes

The framework does **not** retain a second complete DataFrame solely to calculate the report.

Export one consolidated report for the whole session:

```python
from dataframework import DataSession

session = DataSession()
session.load("customers.csv", "crm.csv", namespace=globals())
session.column(normalize_phone, "df_file_1_csv", "phone")

session.export_report(
    "dataframework_report.pdf",
    "dataframework_report.html",
)
```

## Recommended notebook usage

Because the API accepts `namespace=globals()`, it can be used naturally in a Jupyter notebook:

```python
from dataframework import load_files

load_files("customers.csv", "crm.csv", namespace=globals())
```

After that, the generated names are directly available in the notebook session.

## Build the package

Install build tooling:

```powershell
python -m pip install -r dev-requirements.txt
```

Clean old build output first:

```powershell
Remove-Item -Recurse -Force build, dist -ErrorAction SilentlyContinue
Get-ChildItem -Recurse -Directory -Filter __pycache__ | Remove-Item -Recurse -Force -ErrorAction SilentlyContinue
Get-ChildItem -Recurse -File -Filter *.pyc | Remove-Item -Force -ErrorAction SilentlyContinue
```

Build:

```powershell
python -m build
```

Check the artifacts:

```powershell
python -m twine check dist/*
```

## Test on TestPyPI first

Upload to TestPyPI:

```powershell
python -m twine upload --repository testpypi dist/*
```

Then install from TestPyPI in a **fresh virtual environment** and test the public package import before using the real PyPI repository.

## Publish to PyPI

After the TestPyPI package is verified:

```powershell
python -m twine upload dist/*
```

Use a PyPI API token rather than a password. Keep the token outside source control.

## Versioning

Every published version must have a new version number. Recommended sequence:

```text
1.0.0
1.0.1
1.1.0
1.2.0
```

Do not reuse an already-published version number.

## Package layout

```text
dataframework-column-tools/
├── src/
│   └── dataframework/
│       ├── __init__.py
│       ├── api.py
│       ├── loader.py
│       ├── mapper.py
│       ├── semantic.py
│       ├── tracking.py
│       ├── report.py
│       └── py.typed
├── tests/
├── examples/
├── pyproject.toml
├── requirements.txt
├── dev-requirements.txt
├── MANIFEST.in
├── README.md
└── LICENSE
```
