Metadata-Version: 2.5
Name: damicore-normalizer
Version: 0.2.0
Summary: Object materialization stage for the DAMICORE pipeline: delimited text, spreadsheets, and file corpora.
Project-URL: Homepage, https://github.com/Delbem-Research-and-Innovation/damicore
Project-URL: Repository, https://github.com/Delbem-Research-and-Innovation/damicore
Project-URL: Issues, https://github.com/Delbem-Research-and-Innovation/damicore/issues
Project-URL: Documentation, https://github.com/Delbem-Research-and-Innovation/damicore/blob/main/docs/quickstart.md
Project-URL: Changelog, https://github.com/Delbem-Research-and-Innovation/damicore/blob/main/CHANGELOG.md
Author-email: Ennio Politi Lopes <enniolopes@users.noreply.github.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: canonical-serialization,csv,damicore,normalization,xlsx
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Typing :: Typed
Requires-Python: <3.15,>=3.11
Requires-Dist: openpyxl<4,>=3.1
Requires-Dist: pandas<4,>=2.2
Requires-Dist: pydantic<3,>=2.10
Description-Content-Type: text/markdown

# damicore-normalizer

damicore-normalizer turns an input source into canonical objects, recording the
size and SHA-256 of every one. Objects come from one of two sources: a dataset
split by column or by row -- delimited text (`.csv`, `.tsv`, `.txt`) or an
`.xlsx`/`.xlsm` worksheet -- or a set of files that already are the objects. It
is the first stage of the DAMICORE pipeline; most users install the aggregate
`damicore` distribution, which runs all four stages end to end. Install this
package alone to materialize objects without the rest of the pipeline.

```bash
pip install damicore-normalizer
```

## Python

```python
from damicore_normalizer import (
    DelimitedSource,
    FileCorpusSource,
    NormalizationConfig,
    SpreadsheetSource,
    materialize_objects,
)

# Split a delimited file. Any single character is a delimiter, so a tab-separated
# `.txt` is this same source with `delimiter="\t"`.
result = materialize_objects(
    "dataset.csv",
    "normalization",
    config=NormalizationConfig(source=DelimitedSource(split="columns"), chunk_rows=50_000),
)

# Split a worksheet. `sheet` is required when the workbook holds more than one.
spreadsheet = materialize_objects(
    "dataset.xlsx",
    "normalization-xlsx",
    config=NormalizationConfig(source=SpreadsheetSource(split="columns")),
)

# Adopt files that already are the objects. No split, delimiter, or encoding.
corpus = materialize_objects(
    "corpus",
    "normalization-files",
    config=NormalizationConfig(source=FileCorpusSource(recursive=True)),
)
```

A delimited source streams through `pandas.read_csv` and a worksheet streams
through `openpyxl` in read-only mode; both bound open column files with an LRU
pool. A file corpus is copied in and hashed, so the run directory stays
self-contained. Every source writes the objects plus a `manifest.json` into the
output directory, and that manifest is the input the sibling `damicore-distance`
distribution consumes to compute the NCD matrix.

The manifest records `object_encoding` -- `json-lines/1` for a split dataset,
`raw-bytes/1` for adopted files -- because an NCD value is only meaningful
relative to the bytes it measured.

## Links

- Repository: <https://github.com/Delbem-Research-and-Innovation/damicore>
- Issues: <https://github.com/Delbem-Research-and-Innovation/damicore/issues>
- Documentation:
  <https://github.com/Delbem-Research-and-Innovation/damicore/blob/main/docs/quickstart.md>

Licensed under Apache-2.0.
