bpe-continue
Copyright 2026 Malkhaz Suladze

Licensed under the Apache License, Version 2.0 (the "License"); you may not use
these files except in compliance with the License. You may obtain a copy of the
License in the accompanying LICENSE file, or at

    http://www.apache.org/licenses/LICENSE-2.0

------------------------------------------------------------------------------

This product includes software developed at Hugging Face.

It is built on a modified copy of the `tokenizers` Rust crate:

    https://github.com/huggingface/tokenizers
    Copyright 2020 The HuggingFace Team
    Licensed under the Apache License, Version 2.0

The modification adds continued BPE training: `BpeTrainer::continue_from_model`,
which seeds the vocabulary and merges from an already-trained model rather than
from a character alphabet, along with the `Trainer::reserve_tokens` hook that
lets a tokenizer keep its added tokens' ids across such a training run.

This is an independent project. It is not affiliated with, endorsed by, or
supported by Hugging Face.
