Metadata-Version: 2.4
Name: tlf-data-cleaning
Version: 0.1.1
Summary: Data extraction, transformation, normalization, deduplication, and quality-check pipelines for the TLF ecosystem.
Author-email: Sanchita Karki <karkisanchu06@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/ctpl-git/TLF-Data-Manager
Project-URL: Repository, https://github.com/ctpl-git/TLF-Data-Manager
Project-URL: Issues, https://github.com/ctpl-git/TLF-Data-Manager/issues
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: pandas>=2.0.0
Requires-Dist: pdfplumber>=0.10.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: reportlab>=4.0.0; extra == "dev"

# tlf-data-cleaning

Data extraction, transformation, normalization, deduplication, and quality-check pipelines for the TLF ecosystem — the `tlf-data-cleaning` package described in the [TLF-Data-Manager](https://github.com/ctpl-git/TLF-Data-Manager) README, built out as a real, installable, tested package.

It solves a concrete problem: government census reports (Bangladesh BBS, Pakistan PBS, etc.) are published as PDFs, not clean CSVs. This package extracts the raw tables out of those PDFs and runs them through a composable set of cleaning rules to produce the canonical schema that `tlf-census-stats` expects — without hardcoding any of that cleaning logic inside the stats package itself.

## Install

```bash
pip install -e .
# or, for running the test suite too:
pip install -e ".[dev]"
```

## Modules

| Module | Responsibility |
|---|---|
| `extract.py` | `PDFTableExtractor` — pulls raw tables out of a PDF, no cleaning applied. |
| `rules.py` | Composable `CleaningRule` steps: rename, strip whitespace, coerce numeric, dedupe, drop/fill missing, drop-by-predicate. |
| `pipeline.py` | `CleaningPipeline` — chains rules together, optionally starting from a PDF, and can export straight to CSV. |
| `quality.py` | `QualityReport` — missing values, duplicate rows, negative-value checks, summary snapshot. |

## Quick example: PDF → clean CSV

```python
from tlf_data_cleaning import (
    CleaningPipeline, RenameColumns, StripWhitespace, CoerceNumeric, DropDuplicates,
)

pipeline = CleaningPipeline([
    RenameColumns({"Division": "region", "District": "subregion",
                   "Total Population": "total_population", "Male": "male",
                   "Female": "female", "Households": "households",
                   "Literacy Rate": "literacy_rate"}),
    StripWhitespace(columns=["region", "subregion"]),
    CoerceNumeric(["total_population", "male", "female", "households", "literacy_rate"]),
    DropDuplicates(subset=["region", "subregion"]),
])

clean_df = pipeline.run_from_pdf("bbs_district_report.pdf", page=12)
print(pipeline.quality_report().summary())

pipeline.run_and_export(clean_df, "data/sample/bangladesh_census_2022.csv")
```

## Canonical schema

`tlf-census-stats`'s `CensusLoader` expects data in this shape (see its `country_profiles.py` for exact per-country column aliases):

| Column | Required? | Notes |
|---|---|---|
| `region` | Yes | e.g. Division/Province/State |
| `subregion` | Yes | e.g. District |
| `total_population` | Yes | |
| `male` | Yes | |
| `female` | Yes | |
| `households` | Yes | |
| `urban_population` | Optional | |
| `rural_population` | Optional | |
| `literacy_rate` | Optional | |
| `avg_household_size` | Optional | |
| `third_gender` | Optional | A country's non-binary census category (e.g. Bangladesh's "Hijra"). Not every country publishes this. |

This package is the upstream step for two of the three ways data reaches `tlf-census-stats` — it turns a raw government PDF into a CSV in this same shape.

## Three ways to get data into tlf-census-stats

**1. Already-clean CSV/Excel/JSON — skip this package entirely**

If your source is already a clean spreadsheet/CSV/JSON in (or close to) the canonical shape above, just hand it straight to `CensusLoader` — this package isn't needed at all:

```python
from tlf_census_stats import CensusLoader
df = CensusLoader("already_clean_bangladesh_census.csv", country="bangladesh").load()
```

**2. Flat-country PDF (e.g. Bangladesh, Pakistan) — this package handles the whole thing**

Most countries' census PDFs are a flat table per page — extract, clean, and hand off directly:

```python
from tlf_data_cleaning import (
    CleaningPipeline, RenameColumns, StripWhitespace, CoerceNumeric, DropDuplicates,
)
from tlf_census_stats import CensusLoader, StatsReporter

# 1. PDF -> clean CSV (this package)
pipeline = CleaningPipeline([
    RenameColumns({"Division": "region", "District": "subregion",
                   "Total Population": "total_population", "Male": "male",
                   "Female": "female", "Households": "households",
                   "Literacy Rate": "literacy_rate"}),
    StripWhitespace(columns=["region", "subregion"]),
    CoerceNumeric(["total_population", "male", "female", "households", "literacy_rate"]),
    DropDuplicates(subset=["region", "subregion"]),
])
pipeline.run_from_pdf_and_export(
    "bbs_district_report.pdf", page=12,
    output_path="data/sample/bangladesh_census_2022.csv",
)

# 2. clean CSV -> stats report (tlf-census-stats)
reporter = StatsReporter("data/sample/bangladesh_census_2022.csv", country="bangladesh")
reporter.run()
```

**3. India's hierarchical PDF layout — extract here, transform in tlf-census-stats**

India's census PDF has a nested INDIA → STATE → DISTRICT → SUB-DISTRICT structure that a generic `CleaningPipeline` of rename/strip/coerce rules can't reshape — that's what `tlf-census-stats`'s `IndiaCensusTransformer` is specifically for. This package only does the raw extraction step; the reshaping happens on the other side:

```python
from tlf_data_cleaning import PDFTableExtractor
from tlf_census_stats import IndiaCensusTransformer, CensusLoader, StatsReporter

# 1. Extract raw tables (this package) — no cleaning/reshaping applied
extractor = PDFTableExtractor("india_2011.pdf")
tables = extractor.extract_pages(range(1, 2226), known_header=IndiaCensusTransformer.EXPECTED_HEADER)

# 2. Reshape the hierarchical rows into canonical region/subregion rows (tlf-census-stats)
df = IndiaCensusTransformer().transform(tables)
df.to_csv("data/sample/india_from_pdf.csv", index=False)

# 3. Analyze (tlf-census-stats)
reporter = StatsReporter("data/sample/india_from_pdf.csv", country="india")
reporter.run()
```

Column names produced by workflow 2's pipeline just need to match the canonical schema table above for a given country's profile — workflow 3 doesn't need column renaming at all, since `IndiaCensusTransformer` already outputs canonical column names directly.

## Known Limitations

- **Rotated tables without ruling lines are not yet supported.** Some government PDFs embed a wide table as 90°-rotated text so it fits a portrait page (the table visually reads sideways). `PDFTableExtractor` correctly reconstructs the *text* of rotated characters (as of 0.1.1), but it still relies on `pdfplumber`'s built-in row/column detection to find the table's *shape* — and that detection only works reliably when the table has ruling lines around its cells. A rotated table that instead relies on consistent spacing (no visible per-row/per-column borders) will extract with the wrong number of rows/columns even though the text within each cell it does find is now correct. Nepal's census "Table 17: Population by single year of age and sex" is a known example — it's both rotated and unruled, and currently needs to be extracted manually rather than through this package. Properly supporting this layout would require reconstructing table geometry directly from character positions (clustering by x/y position) instead of using `pdfplumber`'s line- or text-based strategies.

## Notes on real government PDFs

- `pdfplumber`'s table detection works well on ruled/gridded tables (like the BBS district reports) but can miss tables with no visible borders — pass `table_settings` to `extract_page`/`extract_all` to tune detection (see pdfplumber's docs for `vertical_strategy` / `horizontal_strategy` options) if a specific report doesn't extract cleanly.
- Multi-line headers or footnote rows sometimes get pulled in as data rows — inspect `extractor.extract_page(n)` output before wiring it into a `CleaningPipeline`, and add a `DropRowsWhere` rule to filter out any non-data rows if needed.

## Tests

```bash
python -m pytest tests/ -v
```

`tests/fixtures/generate_fixture.py` builds the sample PDF used by the test suite — a small table with thousands separators, stray whitespace, and a duplicate row, mimicking real BBS/PBS-style reports.
Regenerate it with:

```bash
python tests/fixtures/generate_fixture.py
```
