Metadata-Version: 2.4
Name: tlf-geo-auto
Version: 0.1.0
Summary: Auto-detect and resolve geographic columns in CSV/XLSX/JSON datasets using tlf-geo
Author: Pujan Pandey
License-Expression: MIT
Project-URL: Homepage, https://github.com/PujanPandey07/tlf-geo-auto
Project-URL: Repository, https://github.com/PujanPandey07/tlf-geo-auto
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: tlf-geo>=0.1.0
Requires-Dist: tlf-core>=0.1.1
Requires-Dist: pandas>=1.5
Requires-Dist: openpyxl>=3.1
Requires-Dist: click>=8.0
Requires-Dist: rich>=13.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"

# tlf-geo-auto

Auto-detect and resolve geographic columns (province, district, local level) in messy CSV, XLSX, or JSON datasets — no manual column configuration required.

Part of [TLF (The Living Fact)](../tlf-project), Corpola Tech's civic-data interoperability toolkit for Nepal. Built on top of [`tlf-geo`](../tlf-geo)'s official NSO administrative registry and [`tlf-core`](../tlf-core)'s header-resolution engine.

## The problem

`tlf-geo` can resolve a Nepali place name to its official administrative code — but you have to tell it exactly which column holds the district, which holds the local level, and so on. That's fine for a dataset you built yourself, but real-world survey exports rarely cooperate:

- Column headers vary wildly (`जिल्ला`, `dist`, `District_Name`, or nothing recognizable at all)
- Place names have typos, inconsistent spelling, or missing suffixes
- The same local-level name can exist in multiple districts
- A row can have the _right_ place name paired with the _wrong_ district

`tlf-geo-auto` handles all of this automatically, and — critically — **never silently guesses**. Anything it isn't confident about gets surfaced to you for a quick decision, never quietly miscategorized or dropped.

## Installation

```bash
pip install tlf-geo-auto
```

Requires `tlf-geo` and `tlf-core` (installed automatically as dependencies).

## Quick start

```bash
tlf-geo-auto resolve survey.csv --output survey_clean.csv
```

That's it. The tool will:

1. Load your file (CSV, XLSX, or JSON — detected automatically by extension)
2. Drop any fully-empty rows
3. Detect which columns contain province/district/local-level data
4. Resolve every clean match automatically
5. Walk you through anything ambiguous, misspelled, or inconsistent
6. Save a cleaned file with official NSO codes added

## Example session

```
$ tlf-geo-auto resolve survey.csv --output clean.csv

Loaded 300 rows from survey.csv
Dropped 9 fully-empty row(s).
Detected district: column 'जिल्ला' (header, confidence 0.95)
Detected local_level: column 'स्थानीय तह' (content, confidence 0.76)

Reviewing 12 issue(s) in 'स्थानीय तह'

The following local level values could not be matched with full confidence.
For each one, pick the correct place from the list, or enter 0 to leave it
unresolved for now.
──────────────────────────── Tikpaur (not_found) ─────────────────────────────
District on file: kailali
Affects 1 row(s): [8]
┏━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━┓
┃ # ┃ Matched On ┃ Official Name ┃ Level        ┃ District ┃ Code  ┃
┡━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━┩
│ 1 │ tikapur    │ Tikapur       │ municipality │ kailali  │ 77113 │
└───┴────────────┴───────────────┴──────────────┴──────────┴───────┘
Which one is correct for 'Tikpaur'? (0 = none of these / skip) [0/1]: 1

Saved cleaned dataset to clean.csv
```

## Output

`tlf-geo-auto` **augments your file, it doesn't replace it.** Every original column stays intact. For each detected geo level, it adds:

| New column             | Meaning                                                                           |
| ---------------------- | --------------------------------------------------------------------------------- |
| `{level}_code`         | The official NSO administrative code (e.g. `district_code`, `local_level_code`)   |
| `{level}_status`       | `resolved`, `resolved_by_review`, `ambiguous`, `not_found`, or `invalid_district` |
| `{level}_error_reason` | Human-readable explanation for anything not cleanly resolved                      |

**If you confirm an answer during review**, the tool also corrects the original text column to match — since at that point it's a human-confirmed fact, not a guess. Rows that auto-resolve without any review are never touched beyond having code columns added; their original text is left exactly as given.

## How detection works

For each column, `tlf-geo-auto` tries two signals, in order:

1. **Header match** (fast) — checks if the column name matches a known alias (via `tlf-core`'s `FieldResolver`) for `district`, `province`, or `municipality`.
2. **Content match** (fallback) — if the header is missing, misspelled, or unrecognized, samples the column's actual values and checks whether they resolve as real places (via `tlf-geo`'s `GeoResolver.search()`). Only counts as strong evidence at exact/near-exact match rates.

This means a column can be detected correctly even with a header `tlf-core` has never seen before, as long as its actual values are real place names.

## How review works

Every row that can't be cleanly auto-resolved falls into one of these categories:

- **`ambiguous`** — the name matches multiple real places even after considering district context; pick the correct one from a list.
- **`not_found`** — no exact match; shown up to 3 fuzzy-matched suggestions **only if genuinely similar** (a similarity threshold filters out coincidental near-matches that aren't real typos).
- **`invalid_district`** — the place name is real, but doesn't exist in the district your row claims; shown where it actually exists instead, so you can confirm and correct the district.

**Grouping:** identical problems are grouped and reviewed once, regardless of how many rows share them — e.g. 50 rows with the same typo become one prompt, not 50. Rows are only grouped together when the answer would genuinely be the same for all of them (a `not_found` name is grouped regardless of district, since district doesn't change that outcome; ambiguous/invalid-district cases are kept separate per district, since the correct answer can differ).

**Never a silent auto-approval:** no confidence threshold auto-accepts a fuzzy match, no matter how close. Every non-exact match is shown to a human before being written to the file. This is a deliberate design choice — "never guess, never drop" applies to every layer of this tool, not just the underlying registry.

## CLI options

```
tlf-geo-auto resolve INPUT_PATH --output OUTPUT_PATH [OPTIONS]

  --output, -o TEXT           Path to write the cleaned file. [required]
  --skip-empty-rows /
  --keep-empty-rows           Drop fully-blank rows before processing.
                               (default: skip)
  --no-review                 Skip interactive review entirely; ambiguous/
                               unresolved rows are left as-is with their
                               status columns intact for later inspection.
```

Supported input/output formats: `.csv`, `.xlsx`, `.xls`, `.json` (auto-detected by file extension).

## Design philosophy

`tlf-geo-auto` follows the same principle as every package in the TLF ecosystem:

> **Never guess, never drop.** Unresolved or uncertain data always surfaces with enough context to fix it by hand — it's never silently miscategorized, and it's never silently discarded.

Concretely, this means:

- No confidence threshold ever auto-approves a non-exact match
- A correction only ever happens after explicit human confirmation
- Everything left unresolved stays visibly flagged in the output, never dropped from the dataset

## Known limitations

- **No batch/non-interactive review mode yet** — `--no-review` skips review entirely rather than applying an automated policy; there's currently no middle ground like "auto-apply the top suggestion above score X."
- **No standalone audit report** — a JSON/HTML summary of everything reviewed and changed is planned but not yet built; for now, the `{level}_status` and `{level}_error_reason` columns in the output file are the audit trail.
- **Assumes NSO/CBS-standard administrative codes** — if your dataset already contains codes from a different registry (e.g. postal codes), this tool doesn't detect or account for that.

## Related packages

- [`tlf-core`](../tlf-core) — header/value normalization registry
- [`tlf-geo`](../tlf-geo) — the underlying Nepal administrative boundary resolver
- [`tlf-pgreconcile`](../tlf-pgreconcile) — normalization-aware Postgres database reconciliation

## License

MIT
