Metadata-Version: 2.5
Name: cloud-language-tools-wenlin
Version: 2.0.1
Summary: Wenlin Chinese dictionary data and parser for Cloud Language Tools
Project-URL: Homepage, https://github.com/Vocab-Apps/cloud-language-tools-wenlin
Project-URL: Repository, https://github.com/Vocab-Apps/cloud-language-tools-wenlin
Author-email: Luc <languagetools@mailc.net>
License-Expression: GPL-3.0-or-later
License-File: LICENSE
Keywords: cantonese,chinese,dictionary,linguistics,mandarin,wenlin
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: Chinese (Simplified)
Classifier: Natural Language :: Chinese (Traditional)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# cloud-language-tools-wenlin

Wenlin Chinese-English dictionary data and parser, packaged for
[Cloud Language Tools](https://github.com/Vocab-Apps/cloud-language-tools-core).

The dictionary database (202,256 entries, 235,594 definitions) is **bundled inside
the wheel**. There is no download step at install time and no external data
directory to provision.

```
pip install cloud-language-tools-wenlin
```

The import name is `clt_wenlin`.

## Usage

Look up a word by its simplified or traditional form:

```python
import json
import clt_wenlin

connection = clt_wenlin.open_connection()
for (entry,) in connection.execute(
        'SELECT entry FROM words WHERE simplified=?', ('你好',)):
    print(json.loads(entry))
connection.close()
```

```python
{'simplified': '你好',
 'traditional': '你好',
 'pinyin': 'nị̌',
 'parts_of_speech': [{'part_of_speech': 'intj.',
                      'definitions': [{'definition': 'How are you?; Hello.'}]}]}
```

Search the English definitions, which are indexed with sqlite FTS5:

```python
connection.execute(
    "SELECT definition, entry_id FROM definitions WHERE definitions MATCH 'confirmation'")
```

If you would rather manage the connection yourself, `clt_wenlin.get_wenlin_db_path()`
returns the filesystem path of the bundled database.

## Database schema

| table | columns | notes |
| --- | --- | --- |
| `words` | `simplified`, `traditional`, `entry`, `entry_id` | `entry` is the entry serialized as JSON; indexed on `simplified`, `traditional` and `entry_id` |
| `definitions` | `definition`, `entry_id` | FTS5 virtual table over every English definition |

Each `entry` document looks like:

```python
{
    'simplified': str,
    'traditional': str,
    'pinyin': str,
    'parts_of_speech': [{
        'part_of_speech': str | None,
        'definitions': [{
            'definition': str,
            'example_pinyin': str,       # optional
            'example_chinese': str,      # optional
            'example_translation': str,  # optional
            'measure_word': str,         # optional
        }],
    }],
}
```

## Parsing a cidian.u8 file

The parser that produced the database is part of the package:

```python
import clt_wenlin

entries = clt_wenlin.read_dictionary_file('cidian.u8')
clt_wenlin.create_sqlite_file('cidian.u8', 'wenlin.db')
```

Only the English side of the dictionary is kept; French definitions are dropped.

## Development

The project uses [uv](https://docs.astral.sh/uv/):

```bash
uv sync
uv run pytest
```

The database is committed to the repository xz-compressed as
`data/wenlin_revA.db.xz` (16MB). The build hook in `hatch_build.py` decompresses
it into `src/clt_wenlin/data/` so that it lands, uncompressed and ready to query,
in the wheel. To rebuild it from a Wenlin source file:

```bash
uv run python tools/build_db.py /path/to/cidian.u8
```

Bump `WENLIN_DB_REV` in `src/clt_wenlin/database.py` when the data changes.

### Releasing

```bash
./package.sh 2.0.1
```

## License

GPL-3.0-or-later. The Wenlin dictionary data is the property of Wenlin Institute.
