Metadata-Version: 2.4
Name: entiscope-ko
Version: 0.1.0
Summary: entiscope-ko — Korean PII detection & masking engine (ONNX, BIOES + Viterbi)
Author: entiscope Core Team
License: Apache-2.0
Project-URL: Homepage, https://github.com/zafrem/entiscope-ko
Project-URL: Documentation, https://github.com/zafrem/entiscope-ko/blob/main/README.md
Keywords: pii,ner,redaction,privacy,korean,onnx
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.22
Requires-Dist: PyYAML>=6.0
Requires-Dist: onnxruntime>=1.17
Requires-Dist: transformers>=4.38
Requires-Dist: huggingface_hub>=0.20
Provides-Extra: train
Requires-Dist: torch>=2.0; extra == "train"
Requires-Dist: datasets>=2.16; extra == "train"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Dynamic: license-file

# entiscope-ko

Korean **PII detection & masking** engine — part of the
[entiscope](https://github.com/zafrem) series. Detects and masks person names,
phone numbers, national IDs, emails, addresses, financial info, private dates,
and credentials in Korean text.

> ⚠️ entiscope is a redaction **aid**, not an anonymization or compliance
> guarantee. See [Limitations](#limitations).

📖 **[한국어 README](README.ko.md)**

## Install

```bash
pip install entiscope-ko
```

## 60-second quickstart

**Python**

```python
from entiscope_ko import Entiscope

engine = Entiscope.from_pretrained()                 # downloads ONNX weights on first run
result = engine.redact("홍길동의 전화번호는 010-1234-5678")

result.masked_text        # "<PER>의 전화번호는 <PHONE>"
result.detected_spans     # [DetectedSpan(label="PER", start=0, end=3, ...), ...]
result.summary            # {"span_count": 2, "by_label": {"PER": 1, "PHONE": 1}, ...}
```

**CLI**

```bash
entiscope redact "홍길동의 전화번호는 010-1234-5678"
cat notes.txt | entiscope redact --operating-point high_recall
```

## Documentation

Full guides live in [`docs/`](docs/) — organised by what you want to do:

| I want to… | Guide |
|---|---|
| Run it from the terminal | [CLI Reference](docs/CLI.md) |
| Call it from Python | [Python API Reference](docs/API.md) |
| Understand the JSON output | [Output Schemas](docs/OUTPUT_SCHEMAS.md) |
| Score it on my labelled data | [Evaluation & Output Modes](docs/EVAL_AND_OUTPUT_MODES.md) |
| Trade precision vs recall | [Operating Points](docs/OPERATING_POINTS.md) |
| Fine-tune on my own data | [Fine-tuning](docs/FINETUNING.md) |

## Entities

`PER` · `PHONE` · `ID_NUM` · `EMAIL` · `LOC` · `BANK` · `DATE` · `SECRET`.
Regex patterns live in [`entiscope/regex_rules.yaml`](entiscope/regex_rules.yaml)
(user-extensible, no code change); KO-specific extended entities in
[`entiscope/entity_config.yaml`](entiscope/entity_config.yaml).

## How it works

A **two-stage hybrid pipeline** (SRS §3.4), results merged via Union:

1. **Regex filter** — structurally obvious PII (phone, email, IDs, cards, secrets).
2. **ONNX NER** — a BIOES token classifier with a constrained Viterbi decoder for
   contextual PII (names, addresses, private dates).

Inference is **ONNX Runtime only — no PyTorch at runtime**. Recall-first, with
runtime [operating-point](docs/OPERATING_POINTS.md) tuning (no retraining).
PyTorch is needed only to [fine-tune](docs/FINETUNING.md).

## Model & performance

- **Architecture** — `klue/roberta-base` encoder → BIOES token-classification
  head → constrained Viterbi decoder.
- **Runtime artifact** — INT8-quantized ONNX, **~105 MB** (well under the
  ≤ 150 MB budget), max sequence length 256. Weights download from Hugging Face
  Hub on first use, with a SHA-256 `checksum.txt` for integrity verification.
- **Accuracy** — meets the entity-level **strict F1 ≥ 0.93** target on a
  held-out validation set whose examples are disjoint from the training data, so
  the score reflects generalization rather than memorization. See
  [Evaluation & Output Modes](docs/EVAL_AND_OUTPUT_MODES.md) to score it on your
  own labelled data.

## Limitations

- Not an anonymization/compliance guarantee; use as one layer of privacy-by-design.
- Known failure modes: under-detection of uncommon/regional names; over-redaction
  of public entities in ambiguous contexts; fragmented spans in heavily
  mixed-format text; missed `SECRET` for novel credential formats.
- Extra human review recommended for medical/legal/financial/government workflows.

## License

Apache-2.0. Weights are distributed on Hugging Face Hub under Apache-2.0 with a
`checksum.txt` (SHA-256) for integrity verification. Contributions welcome — see
[CONTRIBUTING.md](CONTRIBUTING.md).
