Metadata-Version: 2.4
Name: auto-i18n-lib
Version: 0.5.1
Summary: Post-render HTML and frontend UI dictionary translation for Python projects with OpenAI-backed caching
Author-email: Andrey Bondarenko <bona.plus2030@gmail.com>
License: MIT
Project-URL: Homepage, https://bona-plus.ru
Project-URL: Source, https://github.com/Aalam2000/autoi18n
Project-URL: Issues, https://github.com/Aalam2000/autoi18n/issues
Keywords: i18n,l10n,translation,html,fastapi,flask,jinja2,openai
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: openai>=1.0.0
Dynamic: license-file

# auto-i18n-lib

Lightweight file-based translation library for Python applications.

`auto-i18n-lib` translates:

- rendered HTML
- frontend UI dictionaries
- backend phrases by stable keys

The library works in two modes:

- **render mode** — only reads ready local files
- **background mode** — finds missing translations and fills caches

This allows pages, UI, PDFs, and backend bot/system messages to work without synchronous translation requests during runtime.

---

## Features

- post-render HTML translation
- frontend UI dictionary translation
- backend phrase translation by keys
- separate backend dictionaries for:
  - `bot`
  - `system`
- separate pending queues for backend dictionaries
- per-language JSON file caches
- no synchronous translation calls during render
- fallback to source text when translation is missing
- background translation worker
- nested `dict` / `list` support for UI dictionaries
- language normalization
- fallback language chain
- legacy page cache compatibility

---

## Installation

```bash
pip install auto-i18n-lib
````

---

## Requirements

* Python 3.9+
* OpenAI API key

---

## Environment

You can configure the library through constructor arguments or environment variables.

### Linux / macOS

```bash
export OPENAI_API_KEY="your_key"
export SOURCE_LANG="ru"
export AUTO_I18N_TARGET_LANGS="en,az,tr"
export AUTO_I18N_JS_SCAN_ENABLED="true"
export AUTO_I18N_JS_GLOBS='["web/static/js/**/*.js"]'
export AUTO_I18N_UI_NAMESPACE="ui"
export AUTO_I18N_DYNAMIC_DOM_ENABLED="true"
export AUTO_I18N_FALLBACK_LANG="ru"
```

### Windows PowerShell

```powershell
$env:OPENAI_API_KEY="your_key"
$env:SOURCE_LANG="ru"
$env:AUTO_I18N_TARGET_LANGS="en,az,tr"
$env:AUTO_I18N_JS_SCAN_ENABLED="true"
$env:AUTO_I18N_JS_GLOBS='["web/static/js/**/*.js"]'
$env:AUTO_I18N_UI_NAMESPACE="ui"
$env:AUTO_I18N_DYNAMIC_DOM_ENABLED="true"
$env:AUTO_I18N_FALLBACK_LANG="ru"
```

---

## Core idea

## Render mode

During normal application work the library:

* reads translations only from ready JSON files
* does not call OpenAI during render
* returns original text if translation is missing

This keeps rendering fast and stable.

## Background mode

In the background the library:

* scans registered pages
* finds missing HTML strings
* finds missing UI dictionary strings
* finds missing backend key phrases
* puts missing items into pending files
* translates them in batches
* writes ready translations into local JSON files

---

## Quick start

```python
from autoi18n import Translator

translator = Translator(
    api_key="YOUR_OPENAI_API_KEY",
    cache_dir="./translations",
    source_lang="ru",
    target_langs=["en", "az", "tr"],
)
```

---

## HTML translation

Use `translate_html()` for already rendered HTML.

```python
html = """
<h1>Главная</h1>
<p>Добро пожаловать в систему</p>
<button>Сохранить</button>
"""

translated_html = translator.translate_html(
    html=html,
    target_lang="en",
    page_name="home",
)

print(translated_html)
```

### Important

If translation is missing:

* original text is returned immediately
* missing text is added to pending queue
* background worker translates it later

---

## Frontend UI dictionary translation

Use `translate_dict()` for JS tables, forms, modals, filters, buttons, and other frontend dictionaries.

```python
ui = {
    "title": "Грузы",
    "filters": {
        "search": "Поиск",
        "reset": "Сбросить"
    },
    "modal": {
        "save": "Сохранить",
        "cancel": "Отмена"
    }
}

translated_ui = translator.translate_dict(
    page_name="cargo",
    dict_name="table",
    source_dict=ui,
    target_lang="en",
)
```

You can restrict translation to specific keys:

```python
translated_ui = translator.translate_dict(
    page_name="cargo",
    dict_name="table",
    source_dict=ui,
    target_lang="en",
    filter_keys=["title", "search", "save", "cancel"],
)
```

---

## Backend phrase translation by keys

This block is предназначен for bot messages, system messages, service replies, validation texts, and other backend phrases that must not be hardcoded by language in Python code.

### Main idea

You register source phrases once by keys:

```python
translator.register_keys(
    {
        "bot.welcome": "Добро пожаловать в систему",
        "bot.ask_email": "Введите email администратора",
    },
    dict_name="bot",
)
```

Then you read translations locally by key:

```python
text = translator.translate_key(
    key="bot.welcome",
    lang="en",
    default="Добро пожаловать в систему",
    dict_name="bot",
)
```

If translation is missing:

* `default` is returned immediately
* the key remains registered
* missing translation is added to backend pending file
* worker translates it later

### Supported backend dictionaries

* `bot`
* `system`

These dictionaries are stored separately and do not mix with HTML or UI translation storage.

---

## register_keys()

Registers source backend phrases by stable keys.

### Example

```python
translator.register_keys(
    {
        "bot.welcome": "Добро пожаловать в систему",
        "bot.ask_email": "Введите email администратора",
        "bot.done": "Готово",
    },
    dict_name="bot",
)
```

### For system phrases

```python
translator.register_keys(
    {
        "system.user_not_found": "Пользователь не найден",
        "system.access_denied": "У вас нет доступа",
        "system.validation_error": "Ошибка проверки данных",
    },
    dict_name="system",
)
```

### Behavior

* source phrases are saved into source-language backend dictionary
* missing target-language translations are added into backend pending file
* already translated keys are not re-added
* storage is separate for `bot` and `system`

---

## translate_key()

Returns local backend translation by key.

### Example

```python
text = translator.translate_key(
    key="system.access_denied",
    lang="az",
    default="У вас нет доступа",
    dict_name="system",
)
```

### Behavior

* reads translation only from local backend files
* checks fallback language chain
* if translation is missing, returns `default`
* also ensures the key is registered and queued for background translation

---

## Django example

## View

```python
import json
from django.shortcuts import render
from autoi18n import Translator

translator = Translator(
    api_key="YOUR_OPENAI_API_KEY",
    cache_dir="./translations",
    source_lang="ru",
    target_langs=["en", "az", "tr"],
)

translator.register_keys(
    {
        "bot.welcome": "Добро пожаловать в систему",
        "system.access_denied": "У вас нет доступа",
    },
    dict_name="bot",
)

translator.register_keys(
    {
        "system.access_denied": "У вас нет доступа",
    },
    dict_name="system",
)

def cargo_page(request):
    target_lang = translator.detect_browser_lang(
        request.headers.get("Accept-Language", "")
    )

    html = """
    <h1>Грузы</h1>
    <button>Сохранить</button>
    """

    translated_html = translator.translate_html(
        html=html,
        target_lang=target_lang,
        page_name="cargo_page",
    )

    ui = {
        "title": "Грузы",
        "search": "Поиск",
        "empty": "Нет данных",
        "save": "Сохранить",
        "cancel": "Отмена",
    }

    translated_ui = translator.translate_dict(
        page_name="cargo_page",
        dict_name="table_ui",
        source_dict=ui,
        target_lang=target_lang,
    )

    bot_welcome = translator.translate_key(
        key="bot.welcome",
        lang=target_lang,
        default="Добро пожаловать в систему",
        dict_name="bot",
    )

    return render(request, "cargo/page.html", {
        "translated_html": translated_html,
        "ui_json": json.dumps(translated_ui, ensure_ascii=False),
        "bot_welcome": bot_welcome,
    })
```

## Template

```html
<div>
    {{ translated_html|safe }}
</div>

<div>
    {{ bot_welcome }}
</div>

<script>
    window.CARGO_I18N = {{ ui_json|safe }};
</script>
```

---

## Background page registration

Register pages for automatic background HTML scanning.

```python
def render_dashboard():
    return """
    <h1>Панель управления</h1>
    <button>Продолжить</button>
    """

translator.register_page(
    page_name="dashboard",
    html_getter=render_dashboard,
    target_langs=["en", "az", "tr"],
)
```

---

## Background processing

## Process all translations once

```python
report = translator.process_all_translations(batch_size=50)
print(report)
```

This processes:

* registered HTML pages
* frontend UI pending queue
* backend key pending queues for `bot`
* backend key pending queues for `system`

## Run continuous background loop

```python
translator.run_translation_loop(interval=300, batch_size=50)
```

## Run in a separate thread

```python
import threading

threading.Thread(
    target=translator.run_translation_loop,
    kwargs={"interval": 300, "batch_size": 50},
    daemon=True,
).start()
```

---

## Public API

## `Translator(...)`

```python
Translator(
    cache_dir="./translations",
    api_key=None,
    source_lang=None,
    model="gpt-4o-mini",
    target_langs=None,
)
```

### Parameters

* `cache_dir` — root directory for translation files
* `api_key` — OpenAI API key
* `source_lang` — source language code
* `model` — OpenAI model name
* `target_langs` — list of target languages for background translation

If `target_langs` is not passed, the library tries to read them from:

```text
AUTO_I18N_TARGET_LANGS
```

---

## HTML / UI API

### `register_page(page_name, html_getter, target_langs, context=None)`

Registers a page for background scanning.

### `scan_page(page_name, target_lang)`

Scans one registered page and returns missing translatable HTML items.

### `scan_all_pages()`

Scans all registered pages and all configured target languages.

### `process_page_translations(page_name, target_lang, batch_size=50)`

Processes missing HTML translations for one page and one language.

### `process_all_translations(batch_size=50)`

Processes all registered HTML pages and backend pending queues.
When `AUTO_I18N_JS_SCAN_ENABLED=true`, this method also scans JS files and adds `_js_keys` statistics into the report.

### `extract_js_keys(js_globs=None, namespace=None)`

Extracts stable UI translation keys from JS sources and enqueues missing target-language translations.

Supports:

* explicit calls: `t("ui.key", "Текст")`
* explicit calls: `translateKey("ui.key", "Текст")`
* dictionary-like entries: `field: "Заголовок"`

### `run_translation_loop(interval=300, target_lang=None, page_name="page", stop_event=None, batch_size=50)`

Runs continuous background translation loop.

### `translate_text(text, target_lang, page_name="page", prompt_type="normal")`

Returns translated plain text from cache if available.
If translation is missing, returns original text and adds it to pending queue.

### `translate_html(html, target_lang, page_name="page")`

Translates rendered HTML using local cache.
Missing strings remain unchanged until background processing fills them.

### `translate_dict(page_name, dict_name, source_dict, target_lang=None, filter_keys=None)`

Translates frontend UI dictionaries.

Rules:

* supports nested `dict`
* supports nested `list`
* stores keys as `ui::page_name::dict_name::path`
* returns original values if translation is missing
* uses shared HTML/UI cache and pending queue

### `get_frontend_translations(lang, namespace=None)`

Returns frontend translation dictionary for runtime usage.

### `build_frontend_runtime(lang, namespace=None, dynamic_dom_enabled=None)`

Returns ready-to-inline JavaScript runtime with:

* `window.autoI18n.translateKey(key, fallback)`
* `window.autoI18n.translateDom(root?)`
* optional `MutationObserver` support when dynamic DOM mode is enabled

---

## Backend key API

### `register_keys(items, dict_name="bot", target_langs=None)`

Registers backend phrases by stable keys.

Parameters:

* `items` — dict like `{key: source_text}`
* `dict_name` — `"bot"` or `"system"`
* `target_langs` — optional override for target languages

### `translate_key(key, lang, default, dict_name="bot")`

Returns backend phrase translation from local files.

Parameters:

* `key` — stable backend phrase key
* `lang` — target language
* `default` — source/fallback text
* `dict_name` — `"bot"` or `"system"`

### `enqueue_backend_missing_keys(items, target_lang, dict_name="bot")`

Adds backend keys to backend pending file manually.

### `get_backend_pending_entries(dict_name="bot", target_lang=None)`

Returns backend pending items.

### `process_backend_key_translations(dict_name="bot", target_lang=None, batch_size=50)`

Processes backend pending queue for one backend dictionary.

### `process_all_backend_key_translations(batch_size=50)`

Processes both backend dictionaries:

* `bot`
* `system`

### `get_backend_translation_coverage(lang, dict_name="bot")`

Returns translation coverage for backend dictionary.

Example response:

```python
{
    "dict_name": "bot",
    "lang": "en",
    "translated": 120,
    "pending": 8,
    "total": 128,
    "percent": 93.75
}
```

---

## General helper API

### `enqueue_missing_texts(items, target_lang, page_name="page")`

Adds missing HTML/UI items to shared pending queue.

### `get_pending_entries(target_lang=None)`

Returns current shared HTML/UI pending items.

### `process_pending_translations(target_lang=None, page_name="page", batch_size=50)`

Processes shared HTML/UI pending queue.

### `get_translation_coverage(lang)`

Returns translation statistics for shared HTML/UI storage.

### `detect_browser_lang(accept_language)`

Extracts and normalizes browser language from `Accept-Language`.

### `get_alternative_lang(current_lang, browser_lang)`

Returns helper language for language switch logic.

---

## Language normalization

The library normalizes language codes before using them in cache files.

Examples:

* `EN` -> `en`
* `en_US` -> `en-us`
* `en-US` -> `en-us`

---

## Fallback chain

When translation is missing, the library checks fallback languages.

Example:

```text
en-us -> en -> source_lang
```

### For HTML / UI

If nothing is found:

* original text is returned

### For backend keys

If nothing is found:

* `default` is returned

---

## Cache structure

Translations are stored as JSON files inside the cache directory.

```text
translations/
  en.json
  az.json
  tr.json
  _pending.json
  backend/
    bot.ru.json
    bot.en.json
    bot.az.json
    bot.tr.json
    system.ru.json
    system.en.json
    system.az.json
    system.tr.json
    _pending_bot.json
    _pending_system.json
```

---

## Shared HTML / UI cache format

```json
{
  "Сохранить": "Save",
  "ui::cargo::table::title": "Cargo",
  "ui::cargo::table::filters.search": "Search"
}
```

---

## Backend cache format

### `backend/bot.ru.json`

```json
{
  "bk::bot.welcome": "Добро пожаловать в систему",
  "bk::bot.ask_email": "Введите email администратора"
}
```

### `backend/bot.en.json`

```json
{
  "bk::bot.welcome": "Welcome to the system",
  "bk::bot.ask_email": "Enter administrator email"
}
```

### `backend/system.en.json`

```json
{
  "bk::system.access_denied": "Access denied",
  "bk::system.user_not_found": "User not found"
}
```

---

## What is translated

The library translates:

### HTML

* visible HTML text nodes
* button labels
* selected HTML attributes:

  * `placeholder`
  * `title`
  * `alt`
  * `aria-label`
  * `value` for button-like inputs

### Frontend UI

* frontend dictionaries
* nested dict/list structures
* table labels
* modal texts
* form labels
* buttons
* filters
* empty-state texts

### Backend keys

* bot replies
* system replies
* validation texts
* service messages
* backend-generated phrases by stable keys

---

## What is skipped

The parser skips:

* `script`
* `style`
* `noscript`

It also skips:

* elements with `translate="no"`
* elements with `data-translate="no"`
* element with `id="langSwitch"`
* pure numbers
* number-like strings
* hashes / UUID-like strings
* many technical values

---

## PDF usage

Do not translate a generated PDF file directly.

Correct flow:

```text
render HTML -> translate_html -> generate PDF
```

If PDF contains backend phrases inserted by Python code:

```text
translate_key / translate_dict / translate_html -> build final translated HTML -> generate PDF
```

---

## Recommended integration pattern

Use the library in this order:

1. render HTML as usual
2. translate rendered HTML through `translate_html()`
3. pass JS/UI dictionaries through `translate_dict()`
4. register backend phrases through `register_keys()`
5. read backend phrase translations through `translate_key()`
6. run background worker to complete missing translations
7. generate PDF only from already translated HTML

---

## Recommended bot integration pattern

For bots and backend services:

1. define stable backend keys
2. register source phrases once during startup
3. request translated text only through `translate_key()`
4. never hardcode per-language backend phrase dictionaries in code
5. let background worker fill missing target translations

### Example

```python
translator.register_keys(
    {
        "bot.greeting": "Привет! Чем помочь?",
        "bot.ask_phone": "Введите номер телефона",
        "system.error_generic": "Произошла ошибка",
    },
    dict_name="bot",
)

translator.register_keys(
    {
        "system.error_generic": "Произошла ошибка",
        "system.not_allowed": "Действие недоступно",
    },
    dict_name="system",
)

reply = translator.translate_key(
    key="bot.greeting",
    lang=user_lang,
    default="Привет! Чем помочь?",
    dict_name="bot",
)
```

---

## Limitations

* the library works on rendered HTML, not templates
* translation quality depends on source text quality
* highly unstable HTML reduces cache reuse efficiency
* JS string extraction is manual: developer passes UI dictionaries explicitly
* backend phrase keys must be registered with stable source texts
* target languages must be configured explicitly if background translation should cover them automatically

---

## License

MIT
