# thaiphon-data-volubilis

An exact-match Thai pronunciation lexicon for the
[thaiphon](https://pypi.org/project/thaiphon/) phonological engine,
derived from the VOLUBILIS *Mundo* Multilingual Thai Dictionary &
Database by Belisan. When installed alongside `thaiphon`, the
pipeline short-circuits on words it finds in this data and returns
the exact reading instead of running the rule-based derivation.

Installing it lifts `thaiphon`'s Wiktionary-IPA exact-match rate from
~57 % on the base engine to **~75 %**. That gain is the word-boundary
and variant coverage a rule-based engine can't infer from orthography
alone.

## Install

```bash
pip install thaiphon-data-volubilis
# or
uv add thaiphon-data-volubilis
```

Python 3.10+, no runtime dependencies. `thaiphon` picks the lexicon up
automatically on import; no configuration needed.

## What's in the wheel

A single `lexicon.db` file: a read-only SQLite database of 84 k Thai
words, each mapped to its pre-derived phonological word
(syllable-segmented, with onsets, vowels, codas, and tones resolved).
Keys are Thai strings; the primary-key index over `thai_word` gives
O(log n) lookup in tens of microseconds.

The package exposes one `Mapping[str, PhonologicalWord]` named
`ENTRIES` with the usual dict-like API (`__getitem__`, `__contains__`,
`get`, `keys`, `items`, `values`, `__len__`, `__iter__`). Callers
don't see the storage backend, so a future release can change it
without breaking anyone.

## Memory footprint

The lexicon is memory-mapped via SQLite's `mmap_size` pragma and
opened with `immutable=1`. Every process that imports the package
shares the same underlying pages through the operating-system page
cache. Numbers from this machine, against the previous in-memory
representation:

| Measurement                          | Before   | After    |
|--------------------------------------|---------:|---------:|
| Per-process RSS after first lookup   | ~340 MiB |   ~7 MiB |
| Per-process RSS after 1 k lookups    | ~340 MiB |  ~35 MiB |
| Wheel size                           |  1.6 MiB |  2.7 MiB |
| `.pyc` on Python 3.10 (import cost)  |  117 MiB |     0    |

The ~35 MiB after warmup is the per-process LRU of the last 10 k
inflated entries. The ~50 MiB or so of physical memory holding the
mmap'd database pages is counted once by the kernel and shared by
every Python worker that has imported the package.

### What this means in practice

For multi-worker web servers (gunicorn, uvicorn with workers, uWSGI)
total lexicon RSS is bounded by the size of `lexicon.db` plus a small
per-worker LRU. It does not multiply with worker count. Dozens of
workers on a single host stay practical.

SQLite forbids sharing a connection across threads, so each Python
thread that touches the lexicon lazily opens its own connection via
`threading.local()`. FastAPI / Starlette's threadpool, Django's async
views, and any other thread model work without extra plumbing.

Serverless cold starts (AWS Lambda, Cloud Run, Fargate, and Cloudflare
Workers when Python support allows) don't pay a `.pyc` unpack or a
lexicon-inflation cost. The file is only read from as entries are
requested.

Pre-forking servers like sync gunicorn and uWSGI inherit no SQLite
handle across `fork()`, since the connection lives in `threading.local`.
Each child opens its own on first use.

## Attribution

Source data: VOLUBILIS Mundo Multilingual Thai Dictionary & Database by
Belisan — <https://belisan-volubilis.blogspot.com/>.

## License

The data is distributed under
[CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/),
matching the upstream source license. Derivative works of this data
must also be licensed under CC BY-SA 4.0 or a compatible license. See
`LICENSE` and `NOTICE` for details.
