Metadata-Version: 2.4
Name: mintdim
Version: 0.1.21
Summary: Token unit pipeline for building low-padding binary token shards from JSONL datasets.
Author: MintDim contributors
License: MIT License
        
        Copyright (c) 2026 MintDim contributors
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: dataset,jsonl,shards,tokenizer,training
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.12
Requires-Dist: numpy>=1.24
Requires-Dist: sentencepiece>=0.1.99
Requires-Dist: tokenizers>=0.15
Provides-Extra: dev
Requires-Dist: pytest-cov>=4.1; extra == 'dev'
Requires-Dist: pytest>=7.4; extra == 'dev'
Description-Content-Type: text/markdown

# MintDim

MintDim is a token unit pipeline for building low-padding binary token shards from JSONL datasets.

The first public API focuses on one task:

> stream JSONL batches → render templates → tokenize exactly → assign by token length → write binary token unit shards

## Install

~~~bash
pip install mintdim
~~~

For local development:

~~~bash
pip install -e .
~~~

## Quick start

~~~python
from mintdim import pipeline


def build_instruction_units():
    job = (
        pipeline("unit-build")
        .source.jsonl(
            files=[
                "./data/alpaca.jsonl",
            ],
            fields=[
                ["instruction", "input", "output"],
            ],
            templates=[
                "{instruction}\n\n{input}\n\n{output}",
            ],
        )
        .tokenizer.sentencepiece(
            files=["./tokenizer/tokenizer.model"],
        )
        .units(
            sizes=[[128, 192, 256, 320]],
            build_batch=[4096],
        )
        .output.dir(
            "./artifacts/instruction_units",
            samples_per_shard=[10000],
        )
    )

    return job.run()
~~~

## API flow

~~~text
pipeline("unit-build")
→ source
→ tokenizer
→ units
→ output
→ run
~~~

MintDim uses a fluent pipeline API. Each step declares one part of the build job. Nothing is executed until `job.run()` is called.

Batch build flow:

~~~text
stream JSONL sequentially
→ collect source batch
→ render text batch from template
→ hash rendered text batch
→ tokenizer.encode_batch(texts)
→ assign unit by exact token_count inside the batch
→ group records by unit_size
→ writer[unit].write_batch(token_ids)
→ write_many sample/UNK/duplicate index rows
→ update stats and histogram in batch
~~~

## Path rules

All relative paths are resolved from the current working directory.

Example:

~~~python
.source.jsonl(
    files=["./data/train.jsonl"],
    fields=[["text"]],
    templates=["{text}"],
)
~~~

If the script is run from the project root, MintDim resolves the file as:

~~~text
./data/train.jsonl
~~~

MintDim does not scan the repository or search the filesystem automatically. Users must provide explicit paths.

## JSONL source

~~~python
.source.jsonl(
    files=["./data/train.jsonl"],
    fields=[["text"]],
    templates=["{text}"],
)
~~~

Three arguments are required:

~~~text
files      → list of JSONL paths
fields     → list-of-list; fields[i] is the JSONL keys MintDim reads from files[i]
templates  → list of user-defined templates; templates[i] formats samples of files[i]
~~~

Multiple files are supported:

~~~python
.source.jsonl(
    files=[
        "./data/alpaca.jsonl",
        "./data/sharegpt.jsonl",
    ],
    fields=[
        ["instruction", "output"],
        ["prompt", "response"],
    ],
    templates=[
        "{instruction}\n\n{output}",
        "{prompt}\n{response}",
    ],
)
~~~

Each JSONL sample MUST contain every key listed in `fields[i]`. Extra JSONL keys are allowed and ignored.

## Templates

Templates are **user-defined strings**. MintDim does not ship preset templates (no `chatml`, `alpaca`, etc.). The user controls every character.

Template syntax:

~~~text
- placeholders use {field_name}
- placeholder names must match fields[i] exactly as a set
- any other character is treated as a literal (\n, \t, custom separators, etc.)
- placeholders are substituted by the JSONL sample's field values
~~~

Example:

~~~python
templates=[
    "{instruction}\n\n{input}\n\n{output}",
]
~~~

Rules:

~~~text
1. set(placeholders in templates[i]) == set(fields[i])
2. JSONL samples must contain every declared field in fields[i]
3. Declared fields that do not appear in the template are invalid
4. Static parts of templates[i] (every character outside placeholders)
   must encode cleanly with the tokenizer (no UNK)
5. Template rendering is a pure substitution; no extra tokens are injected
~~~

A user may use any character their tokenizer supports. If the tokenizer encodes `\t`, then `"{a}\t{b}"` is valid. If not, MintDim fails pre-flight.

Invalid example:

~~~python
fields=[
    ["prompt", "answer", "separator", "source"],
]
templates=[
    "{prompt}{separator}{answer}",
]
~~~

`source` is declared but not consumed by the template, so MintDim raises
`TemplateFieldMismatchError`.

## Tokenizer

V1 supports:

~~~text
- SentencePiece tokenizer: .model
- Hugging Face tokenizer: .json
~~~

Each tokenizer step takes a `files` list of paths — one path shared across all input files, or one path per input file. The user declares only the tokenizer file path. MintDim loads `unk_id` and `pad_id` from the tokenizer library itself.

SentencePiece:

~~~python
.tokenizer.sentencepiece(
    files=["./tokenizer/tokenizer.model"],
)
~~~

Hugging Face tokenizer JSON:

~~~python
.tokenizer.hf_json(
    files=["./tokenizer/tokenizer.json"],
)
~~~

Multiple tokenizer files (one per input file):

~~~python
.tokenizer.sentencepiece(
    files=[
        "./tokenizer/alpaca.model",
        "./tokenizer/sharegpt.model",
    ],
)
~~~

Required argument:

~~~text
files → list of tokenizer file paths
~~~

Validation rules:

~~~text
- each file path is a string and exists on disk
- the tokenizer library exposes a valid UNK id
- the tokenizer library exposes a valid PAD id
~~~

Why UNK and PAD are required:

~~~text
UNK
→ used for tracking unknown tokens and dataset integrity statistics

PAD
→ used to pad samples to assigned unit size for fixed-width binary shard layout
~~~

The tokenizer is never modified by MintDim. UNK and PAD ids come from the loaded tokenizer object, not from user-declared values or MintDim-side vocab parsing.

## Tokenizer contract

MintDim treats the tokenizer file as the single source of truth for vocabulary,
normalization, encoding behavior, and special-token ids. The pipeline does not
maintain its own token whitelist, preset template, or hand-curated character set.

What MintDim **does**:

~~~text
- load the tokenizer file you point at (.model for SentencePiece, .json for HF)
- delegate encoding to that tokenizer's vocabulary and model as-is
- read unk_id and pad_id directly from the tokenizer library
- treat every character outside {placeholders} in a template as a literal
- accept any template literal whose tokenization does not produce UNK
- record samples whose field values produce UNK in unk_index.jsonl
~~~

What MintDim **does not do**:

~~~text
- prepend or append BOS / EOS / CLS / SEP or any other control symbols
- inject newlines, separators, role tokens, or chat wrappers around fields
- maintain a built-in allow-list of characters that templates may use
- ship preset templates such as chatml, alpaca, or llama-style wrappers
- shrink, truncate, remap, or reinterpret the tokenizer's vocabulary
- modify the tokenizer file or its in-memory state
~~~

What this means for templates:

~~~text
- "{a}\n\n{b}", "<s>{prompt}</s>{answer}", "{q}\t{a}",
  emoji separators, CJK punctuation, and custom delimiters are valid
  as long as your tokenizer encodes the literal characters without UNK
- literal characters between placeholders are tokenized by the same
  tokenizer that runs on field values; there is no separate encoder,
  normalizer, sanitizer, or escape layer
~~~

Tokenization is delegated entirely to the tokenizer library:

~~~text
SentencePiece: SentencePieceProcessor.encode(text, out_type=int)
Hugging Face : Tokenizer.encode_batch(texts, add_special_tokens=False)
~~~

If you want a `<bos>` marker at the start of every sample, put that marker
directly in your template. MintDim will pass it to the tokenizer as literal
text; whether it becomes a special token, a normal token sequence, or UNK
depends entirely on your tokenizer.

Tokenizer-side normalization still applies. SentencePiece and Hugging Face
tokenizers may apply NFKC, lowercasing, dummy prefix, byte-level remapping,
or other normalization declared inside the tokenizer file. That normalization
runs on every text MintDim hands to the tokenizer, including template literals
during preflight and field values at runtime. MintDim does not change this
behavior in either direction.

## Units

Shared unit config:

~~~python
.units(
    sizes=[[128, 192, 256, 320]],
    build_batch=[4096],
)
~~~

Mapped unit config:

~~~python
.units(
    sizes=[
        [128, 192, 256, 320],
        [96, 128, 160, 192, 224, 320],
    ],
    build_batch=[
        4096,
        2048,
    ],
)
~~~

Meaning:

~~~text
sizes
→ target shard unit sizes

build_batch
→ number of raw samples processed per tokenizer batch
~~~

MintDim always tokenizes samples exactly. Unit assignment is based on exact token count.

Example:

~~~text
token_count = 143
sizes = [128, 192, 256, 320]
target unit = 192
~~~

If a sample's `sum(segment token lengths)` exceeds `max(sizes)`, MintDim pauses the build at that sample and asks the user how to recover:

~~~text
[unit-build] Sample exceeds max unit_size
  source       <file>:<line>
  sample_id    <id>
  token_count  <n>
  max(sizes)   <m>

  EN: How should this sample be handled?
      [1] truncate  drop trailing tokens to fit max(sizes)
      [2] extend    declare a new unit_size that fits this sample
      [3] skip      drop this sample (recorded in overflow_index.jsonl)
  VN: Xử lý sample này thế nào?
      [1] truncate  cắt token cuối cho vừa max(sizes)
      [2] extend    khai báo thêm unit_size đủ chứa sample
      [3] skip      bỏ qua sample (ghi vào overflow_index.jsonl)

  [1/2/3] >
~~~

Choosing `extend` triggers a follow-up asking for an integer `new_unit_size >= token_count`. After picking an action, MintDim asks the scope:

~~~text
  EN: Apply this choice to:
      [1] only this sample
      [2] all subsequent overflow samples in this run
  VN: Áp dụng lựa chọn này cho:
      [1] chỉ sample này
      [2] mọi sample overflow tiếp theo trong lần build này
~~~

All three actions ask for scope. With `all`:

~~~text
truncate / all → every subsequent overflow is truncated to max(sizes)
skip / all     → every subsequent overflow is dropped
extend / all   → every subsequent overflow auto-adds a new unit_size = token_count
                 (no further prompts; sizes grow as needed to fit each oversized sample)
~~~

The prompt can be replaced with a fixed policy via `.run(on_overflow=...)`:

~~~text
"prompt"    (default) ask interactively per overflow sample; abort if stdin is not a TTY
"abort"     raise UnitOverflowError immediately, no prompt
"truncate"  drop trailing tokens to max(sizes) for every overflow sample
"skip"      drop every overflow sample
~~~

Every overflow event (truncate / extend / skip) writes one row to `overflow_index.jsonl` for audit.

MintDim V1 does not:
- continue partial builds
- cache build progress

## UNK handling

MintDim distinguishes two kinds of UNK and treats them differently.

### Tier 1 — template/tokenizer contract

Before any data is processed, every literal segment of every template is tokenized once. If any literal produces UNK, MintDim **prompts the user before building**:

~~~text
[unit-build] Template UNK detected
  file index <i>
  literal segments with UNK <n>

  EN: Static-text tokens will be UNK in every encoded sample.
      Continue building?
  VN: Token của static-text sẽ là UNK ở mọi sample.
      Tiếp tục build?

  [y/N] >
~~~

Default = `N`. The decision can be made non-interactive via `.run(on_template_unk=...)`:

~~~text
"prompt"   (default) prompt the user; abort if stdin is not a TTY
"abort"    raise TemplateTokenizerError immediately, no prompt
"continue" proceed silently — every sample carries the same template UNK at fixed positions
~~~

When the build continues with template UNK, the literal UNK positions are recorded in `unk_index.jsonl` like any other UNK — each entry carries `unk_field_indices` so readers can tell which segment the UNK came from.

### Tier 2 — data content UNK

After the template contract is resolved, samples are processed. If a sample's field values produce UNK tokens, MintDim records an entry in `unk_index.jsonl`. Each entry includes:
- `unk_positions` — positions within the encoded payload
- `unk_field_indices` — which segment of `sequence_template` each UNK came from (0-base)
- `unk_chars` — the original substring that produced the UNK

Tier 2 UNK does not fail the build — it is a dataset/tokenizer coverage signal for audit.

Summary:

~~~text
template/tokenizer mismatch
→ prompt user (default), abort, or continue per .run(on_template_unk=...)

data content produces UNK
→ record in unk_index.jsonl with segment/field index
~~~

## Token storage dtype

MintDim writes tokenized shards as binary token arrays. Every slot on disk — magic marker, segment lengths, and token payload — uses the same dtype.

V1 supports:

~~~text
uint16
uint32
~~~

Dtype selection:

~~~text
vocab_size <= 0xFFFF   (65535)
→ uint16, magic = 0xFFFF

vocab_size <= 0xFFFFFFFF
→ uint32, magic = 0xFFFFFFFF
~~~

The `magic` value is reserved for the per-record marker at offset 0; it must not collide with any real token id. If `vocab_size > magic` for the chosen dtype, MintDim raises `TokenDTypeCapacityError` at tokenizer load (the dtype selector promotes uint16 → uint32 automatically when needed).

Selected dtype and magic are written to `manifest.json` and `length_field.json`.

Example:

~~~json
{
  "token_dtype": "uint16",
  "magic": 65535
}
~~~

## Output

~~~python
.output.dir(
    "./artifacts/instruction_units",
    samples_per_shard=[10000],
)
~~~

Two arguments:

~~~text
path / paths         → output directory path(s)
samples_per_shard    → list of positive ints; samples_per_shard[i] applies to files[i]
~~~

The output directory must be new or empty. MintDim V1 does not support overwrite. Use a new output path for a new build.

### Sharding

Shards are rolled **purely by sample count**:

~~~text
shard_000000  →  samples_per_shard[i] samples
shard_000001  →  samples_per_shard[i] samples
shard_xxxxxx  →  remainder (may be smaller)
~~~

This guarantees deterministic shard cardinality for the train loader. Shard byte size depends on `unit_size × bytes_per_token × samples_per_shard[i]` and varies per `unit_xxx/` directory.

### Shard observability

MintDim emits one line per closed shard:

~~~text
[unit-build] shard=000012 samples=100000 bytes=241.8 MiB unit=512
~~~

If a closed shard exceeds an internal recommended size (512 MiB), MintDim prints a warning but does not stop the build:

~~~text
[unit-build:warning] shard=000012 size=684.3 MiB exceeds recommended 512 MiB.
Build continues because sharding is controlled by samples_per_shard.
~~~

The threshold is an internal heuristic, not a user-facing config. To produce smaller shards, lower `samples_per_shard[i]`.

## Multiple files, fields, templates, tokenizers, units, and outputs

MintDim resolves mapping per config block.

For `N` input files, each list-style config may be either:

~~~text
len == 1
→ shared by every input file

len == N
→ mapped by input file index

anything else
→ fail fast
~~~

This means `fields`, `templates`, `tokenizer`, `sizes`, `build_batch`,
`output.dir`, and `samples_per_shard` can each be shared or mapped
independently.

### Fully shared

~~~python
job = (
    pipeline("unit-build")
    .source.jsonl(
        files=[
            "./data/alpaca.jsonl",
            "./data/sharegpt.jsonl",
        ],
        fields=[
            ["instruction", "output"],
        ],
        templates=[
            "{instruction}\n\n{output}",
        ],
    )
    .tokenizer.sentencepiece(
        files=["./tokenizer/tokenizer.model"],
    )
    .units(
        sizes=[[128, 192, 256, 320]],
        build_batch=[4096],
    )
    .output.dir(
        "./artifacts/instruction_units",
        samples_per_shard=[10000],
    )
)
~~~

All files share one schema, one template, one tokenizer, one unit config, and
one output directory. The files are written into the same output dataset.

### Fully mapped

~~~python
job = (
    pipeline("unit-build")
    .source.jsonl(
        files=[
            "./data/alpaca.jsonl",
            "./data/sharegpt.jsonl",
        ],
        fields=[
            ["instruction", "input", "output"],
            ["prompt", "response"],
        ],
        templates=[
            "{instruction}\n\n{input}\n\n{output}",
            "{prompt}\n{response}",
        ],
    )
    .tokenizer.sentencepiece(
        files=[
            "./tokenizer/alpaca.model",
            "./tokenizer/sharegpt.model",
        ],
    )
    .units(
        sizes=[
            [128, 192, 256, 320],
            [96, 128, 160, 192, 224, 320],
        ],
        build_batch=[
            4096,
            2048,
        ],
    )
    .output.dir(
        [
            "./artifacts/alpaca_units",
            "./artifacts/sharegpt_units",
        ],
        samples_per_shard=[10000, 5000],
    )
)
~~~

Everything is mapped by input order:

~~~text
files[i]
↔ fields[i]
↔ templates[i]
↔ tokenizer[i]
↔ sizes[i]
↔ build_batch[i]
↔ output_dir[i]
↔ samples_per_shard[i]
~~~

### Mixed shared and mapped

You may map only the parts that need separate outputs while sharing the rest:

~~~python
tokenizer_path = "./tokenizer/fineweb_edu_32k.model"

job = (
    pipeline("unit-build")
    .source.jsonl(
        files=[
            "./data/train_a.jsonl",
            "./data/train_b.jsonl",
        ],
        fields=[
            ["prompt", "answer", "separator"],
        ],
        templates=[
            "{prompt}{separator}{answer}",
        ],
    )
    .tokenizer.sentencepiece(
        files=[tokenizer_path, tokenizer_path],
    )
    .units(
        sizes=[[32, 64, 128, 256]],
        build_batch=[64],
    )
    .output.dir(
        [
            "./output1",
            "./output2",
        ],
        samples_per_shard=[256],
    )
)
~~~

Resolved order for this example:

~~~text
file 0 → fields[0], templates[0], tokenizer[0], sizes[0], build_batch[0], output1, samples_per_shard[0]
file 1 → fields[0], templates[0], tokenizer[1], sizes[0], build_batch[0], output2, samples_per_shard[0]
~~~

Passing a single-entry `files` list shares one tokenizer across every input file:

~~~python
.tokenizer.sentencepiece(files=[tokenizer_path])
~~~

Passing N entries maps tokenizers by file index, even if the entries are identical:

~~~python
.tokenizer.sentencepiece(files=[tokenizer_path, tokenizer_path])
~~~

# message errors

1.
[english]
PipelineValidationError:
Pipeline configuration is incomplete.

Missing:
- source.files
- source.fields
- source.templates
- tokenizer
- units.sizes
- units.build_batch
- output.dir
- output.samples_per_shard

[vietnam]
PipelineValidationError:
Pipeline chưa được cấu hình đầy đủ.

Thiếu:
- source.files
- source.fields
- source.templates
- tokenizer
- units.sizes
- units.build_batch
- output.dir
- output.samples_per_shard

Thiếu cấu hình nào thì báo thiếu cấu hình đó.


2.
[english]
MintDimFileNotFoundError:
Required file was not found.

Context:
source

Missing file:
./data/train.jsonl

[vietnam]
MintDimFileNotFoundError:
Không tìm thấy file cần dùng.

Ngữ cảnh:
source

File thiếu:
./data/train.jsonl


3.
[english]
SourceParseError:
JSONL line could not be parsed.

File:
./data/alpaca.jsonl

Line:
18291

Reason:
invalid JSON syntax

[vietnam]
SourceParseError:
Không parse được dòng JSONL.

File:
./data/alpaca.jsonl

Dòng:
18291

Lý do:
invalid JSON syntax


4.
[english]
SampleSchemaError:
JSONL sample is missing declared fields.

File:
./data/alpaca.jsonl

Line:
18291

Declared fields:
instruction
input
output

Missing fields:
input

Available keys:
instruction
output
metadata

[vietnam]
SampleSchemaError:
Sample JSONL thiếu field đã khai báo.

File:
./data/alpaca.jsonl

Dòng:
18291

Field đã khai báo:
instruction
input
output

Field bị thiếu:
input

Key hiện có:
instruction
output
metadata


5.
TokenizerValidationError reports structural problems with the tokenizer step. Special-token ids are loaded from the tokenizer library; users do not declare `unk_id` or `pad_id`.

Components:

~~~text
files → structural problem in the tokenizer.files list: entry is not a path,
        path is not a string, missing path, or unknown keys leaked through
~~~

Full ruleset:

~~~text
tokenizer.files must be a list of tokenizer file paths
each tokenizer file path must be a string
unk_id and pad_id are loaded from the tokenizer library
~~~

Example — a non-string path was passed to `tokenizer.files`:

[english]
TokenizerValidationError:
Invalid tokenizer configuration.

Component:
files

Problem:
tokenizer file path must be str, got int

Rules:
- tokenizer.files must be a list of tokenizer file paths
- each tokenizer file path must be a string
- unk_id and pad_id are loaded from the tokenizer library

[vietnam]
TokenizerValidationError:
Cấu hình tokenizer không hợp lệ.

Thành phần:
files

Vấn đề:
tokenizer file path must be str, got int

Quy tắc:
- tokenizer.files phải là list các đường dẫn file tokenizer
- mỗi đường dẫn file tokenizer phải là string
- unk_id và pad_id được lấy từ thư viện tokenizer

Example — tokenizer library does not expose a required special token:

[english]
TokenizerSpecialTokenError:
Tokenizer library did not expose a required special token.

Tokenizer:
./tokenizer/tokenizer.model

Special token:
pad_id

Problem:
missing pad_id

Rule:
MintDim uses tokenizer-provided special token ids only; do not declare unk_id/pad_id manually.

[vietnam]
TokenizerSpecialTokenError:
Thư viện tokenizer không cung cấp special token bắt buộc.

Tokenizer:
./tokenizer/tokenizer.model

Special token:
pad_id

Vấn đề:
missing pad_id

Quy tắc:
MintDim chỉ dùng special token id do tokenizer cung cấp; không khai báo unk_id/pad_id thủ công.


6.
[english]
TokenizerValidationError:
Unsupported tokenizer format.

Supported formats:
- sentencepiece (.model)
- huggingface tokenizer (.json)

[vietnam]
TokenizerValidationError:
Định dạng tokenizer chưa được hỗ trợ.

Các định dạng hiện hỗ trợ:
- sentencepiece (.model)
- huggingface tokenizer (.json)


7.
[english]
TokenizerMappingError:
Received 3 input files but 2 tokenizers.

MintDim supports:
- 1 tokenizer shared across all files
- N tokenizers for N files

[vietnam]
TokenizerMappingError:
Nhận 3 file đầu vào nhưng có 2 tokenizer.

MintDim hỗ trợ:
- 1 tokenizer dùng chung cho tất cả file
- N tokenizer cho N file


8.
[english]
FieldMappingError:
Received 3 input files but 2 field configs.

MintDim supports:
- 1 field config shared across all files
- N field configs for N files

[vietnam]
FieldMappingError:
Nhận 3 file đầu vào nhưng có 2 cấu hình fields.

MintDim hỗ trợ:
- 1 cấu hình fields dùng chung cho tất cả file
- N cấu hình fields cho N file


9.
[english]
TemplateMappingError:
Received 3 input files but 2 templates.

MintDim supports:
- 1 template shared across all files
- N templates for N files

[vietnam]
TemplateMappingError:
Nhận 3 file đầu vào nhưng có 2 template.

MintDim hỗ trợ:
- 1 template dùng chung cho tất cả file
- N template cho N file


10.
Example — template uses a placeholder that is not declared in fields:

[english]
TemplateFieldMismatchError:
Source file:
./data/train.jsonl

API declaration for this file:
fields = ['instruction', 'input', 'output']
template = '{instruction}\n\n{prompt}\n\n{output}'

Field causing template mismatch:
prompt
Reason:
template uses {prompt}, but fields does not declare it

[vietnam]
TemplateFieldMismatchError:
File nguồn:
./data/train.jsonl

Khai báo API cho file này:
fields = ['instruction', 'input', 'output']
template = '{instruction}\n\n{prompt}\n\n{output}'

Field gây lệch template:
prompt
Lý do:
template dùng {prompt}, nhưng fields không khai báo field này

Example — fields declares a field that the template does not use:

[english]
TemplateFieldMismatchError:
Source file:
./data/train.jsonl

API declaration for this file:
fields = ['instruction', 'input', 'output']
template = '{instruction}\n\n{output}'

Field causing template mismatch:
input
Reason:
fields declares this field, but template does not use {input}

[vietnam]
TemplateFieldMismatchError:
File nguồn:
./data/train.jsonl

Khai báo API cho file này:
fields = ['instruction', 'input', 'output']
template = '{instruction}\n\n{output}'

Field gây lệch template:
input
Lý do:
fields khai báo field này, nhưng template không dùng {input}


11.
[english]
TemplateTokenizerError:
Template literal segments produce UNK when encoded by the tokenizer.

File index:
0

Template:
'{a}\n\n{b}'

Problem:
template literal characters are not fully covered by tokenizer vocabulary

Fix:
adjust template literal characters, use a tokenizer that covers them, or run with an explicit template-UNK policy.

[vietnam]
TemplateTokenizerError:
Các đoạn literal của template tạo UNK khi được tokenizer encode.

Vị trí file:
0

Template:
'{a}\n\n{b}'

Vấn đề:
ký tự literal trong template không được tokenizer vocabulary bao phủ đầy đủ

Cách sửa:
chỉnh ký tự literal trong template, dùng tokenizer bao phủ các ký tự đó, hoặc chạy với template-UNK policy rõ ràng.


12.
[english]
UnitMappingError:
Received 3 input files but 2 unit configs.

MintDim supports:
- 1 unit config shared across all files
- N unit configs for N input files

[vietnam]
UnitMappingError:
Nhận 3 file đầu vào nhưng có 2 cấu hình unit.

MintDim hỗ trợ:
- 1 cấu hình unit dùng chung cho tất cả file
- N cấu hình unit cho N file đầu vào


13.
[english]
OutputMappingError:
Received 3 input files but 2 output directories.

MintDim supports:
- 1 shared output directory
- N output directories for N input files

[vietnam]
OutputMappingError:
Nhận 3 file đầu vào nhưng có 2 thư mục output.

MintDim hỗ trợ:
- 1 thư mục output dùng chung
- N thư mục output cho N file đầu vào


14.
[english]
UnitValidationError:
Invalid unit config.

Rules:
- sizes must be positive integers
- sizes must be sorted ascending
- build_batch must be a positive integer

[vietnam]
UnitValidationError:
Cấu hình unit không hợp lệ.

Quy tắc:
- sizes phải là các số nguyên dương
- sizes phải được sắp xếp tăng dần
- build_batch phải là số nguyên dương


15.
[english]
UnitOverflowError:
Sample token length exceeds max unit size.

Sample:
./data/alpaca.jsonl:18291

token_count:
417

max_unit_size:
320

Fix:
- increase unit sizes
- or run with an overflow policy that truncates, skips, or prompts

[vietnam]
UnitOverflowError:
Độ dài token của sample vượt quá unit size lớn nhất.

Sample:
./data/alpaca.jsonl:18291

token_count:
417

max_unit_size:
320

Cách sửa:
- tăng unit sizes
- hoặc chạy với overflow policy để truncate, skip, hoặc prompt


16.
[english]
OutputValidationError:
Output directory already exists and is not empty.

Directory:
./artifacts/token_shards

Use a new output directory.

[vietnam]
OutputValidationError:
Thư mục output đã tồn tại và không rỗng.

Thư mục:
./artifacts/token_shards

Hãy dùng thư mục output mới.


17.
[english]
TokenDTypeCapacityError:
Tokenizer vocabulary size is too large for the token dtypes supported by MintDim.

Tokenizer vocab_size:
5,000,000,000

Supported token dtypes:
- uint16: max vocab_size 65,535
- uint32: max vocab_size 4,294,967,295

Fix:
- use a tokenizer with a smaller vocabulary
- or use a MintDim version that supports a wider token dtype

[vietnam]
TokenDTypeCapacityError:
Kích thước vocabulary của tokenizer quá lớn so với token dtype MintDim hỗ trợ.

Tokenizer vocab_size:
5.000.000.000

Token dtype được hỗ trợ:
- uint16: vocab_size tối đa 65.535
- uint32: vocab_size tối đa 4.294.967.295

Cách sửa:
- dùng tokenizer có vocabulary nhỏ hơn
- hoặc dùng phiên bản MintDim hỗ trợ token dtype rộng hơn


18.
SamplesPerShardError reports value-level and arity-level problems for `samples_per_shard`.

Rules:

~~~text
each samples_per_shard[i] is a positive int
len(samples_per_shard) == 1 or len(samples_per_shard) == file_count
~~~

Example — non-positive value:

[english]
SamplesPerShardError:
Invalid samples_per_shard.

Problem:
samples_per_shard[0] must be a positive int, got 0

Rules:
- each samples_per_shard[i] is a positive int
- len(samples_per_shard) == 1 or len(samples_per_shard) == file_count

[vietnam]
SamplesPerShardError:
samples_per_shard không hợp lệ.

Vấn đề:
samples_per_shard[0] phải là số nguyên dương, hiện tại là 0

Quy tắc:
- mỗi samples_per_shard[i] phải là số nguyên dương
- len(samples_per_shard) == 1 hoặc len(samples_per_shard) == số lượng file

Example — arity mismatch:

[english]
SamplesPerShardError:
Invalid samples_per_shard.

Problem:
Received 3 input files but 2 samples_per_shard entries

Rules:
- each samples_per_shard[i] is a positive int
- len(samples_per_shard) == 1 or len(samples_per_shard) == file_count

[vietnam]
SamplesPerShardError:
samples_per_shard không hợp lệ.

Vấn đề:
Nhận 3 file đầu vào nhưng có 2 entries samples_per_shard

Quy tắc:
- mỗi samples_per_shard[i] phải là số nguyên dương
- len(samples_per_shard) == 1 hoặc len(samples_per_shard) == số lượng file


## Repository layout

~~~text
mintdim/
├─ pyproject.toml
├─ README.md
├─ LICENSE
├─ .gitignore
│
├─ docs/
│  └─ apis/
│     ├─ unit_build.md
│     └─ unit_read.md
│
├─ src/mintdim/
│  ├─ __init__.py
│  ├─ pipeline.py
│  │
│  ├─ cli/
│  │  ├─ __init__.py
│  │  ├─ main.py
│  │  └─ commands/
│  │     ├─ __init__.py
│  │     ├─ unit_build.py
│  │     └─ unit_read.py
│  │
│  ├─ kernel/
│  │  ├─ __init__.py
│  │  ├─ encoding.py
│  │  ├─ exceptions.py
│  │  ├─ filesystem.py
│  │  └─ result.py
│  │
│  └─ apis/
│     ├─ __init__.py
│     ├─ registry.py
│     │
│     ├─ unit_build/
│     │  ├─ __init__.py
│     │  ├─ api.py
│     │  ├─ config.py
│     │  ├─ contracts.py
│     │  ├─ diagnostics.py
│     │  ├─ validation.py
│     │  ├─ runtime.py
│     │  ├─ errors.py
│     │  ├─ events.py
│     │  │
│     │  ├─ adapters/
│     │  │  ├─ __init__.py
│     │  │  │
│     │  │  ├─ sources/
│     │  │  │  ├─ __init__.py
│     │  │  │  ├─ jsonl.py
│     │  │  │  └─ template.py
│     │  │  │
│     │  │  ├─ tokenizers/
│     │  │  │  ├─ __init__.py
│     │  │  │  ├─ hf_json.py
│     │  │  │  └─ sentencepiece.py
│     │  │  │
│     │  │  └─ output/
│     │  │     ├─ __init__.py
│     │  │     ├─ directory.py
│     │  │     ├─ indices.py
│     │  │     ├─ metadata.py
│     │  │     └─ shard_writer.py
│     │  │
│     │  ├─ domain/
│     │  │  ├─ __init__.py
│     │  │  ├─ models.py
│     │  │  ├─ overflow.py
│     │  │  ├─ planner.py
│     │  │  └─ rules.py
│     │  │
│     │  └─ ui/
│     │     ├─ __init__.py
│     │     ├─ logger.py
│     │     └─ prompts.py
│     │
│     └─ unit_read/
│        ├─ __init__.py
│        ├─ api.py
│        ├─ config.py
│        ├─ contracts.py
│        ├─ validation.py
│        ├─ runtime.py
│        ├─ errors.py
│        ├─ events.py
│        │
│        ├─ adapters/
│        │  ├─ __init__.py
│        │  └─ input/
│        │     ├─ __init__.py
│        │     ├─ metadata.py
│        │     └─ shard_reader.py
│        │
│        └─ domain/
│           ├─ __init__.py
│           ├─ batch.py
│           ├─ masking.py
│           └─ queue.py
│
├─ tests/
│  ├─ conftest.py
│  ├─ __init__.py
│  ├─ fixtures/
│  │  └─ apis/
│  │     └─ unit_build/
│  │
│  ├─ unit/
│  │  ├─ apis/
│  │  │  ├─ unit_build/
│  │  │  │  ├─ test_indices.py
│  │  │  │  ├─ test_jsonl_source.py
│  │  │  │  ├─ test_loaders.py
│  │  │  │  ├─ test_output.py
│  │  │  │  ├─ test_shard_writer.py
│  │  │  │  ├─ test_template.py
│  │  │  │  ├─ test_tokenizer_validate.py
│  │  │  │  ├─ test_units.py
│  │  │  │  └─ test_validate.py
│  │  │  └─ unit_read/
│  │  │     ├─ test_api.py
│  │  │     ├─ test_batch.py
│  │  │     ├─ test_masking.py
│  │  │     ├─ test_queue.py
│  │  │     ├─ test_queue_path_resolution.py
│  │  │     ├─ test_shard_reader.py
│  │  │     └─ test_validation.py
│  │  ├─ cli/
│  │  │  └─ test_unit_read_command.py
│  │  └─ kernel/
│  │     └─ test_exceptions.py
│  │
│  └─ integration/
│     └─ apis/
│        ├─ unit_build/
│        │  ├─ test_api_chain.py
│        │  └─ test_smoke.py
│        └─ unit_read/
│           ├─ test_api_chain.py
│           └─ test_read_bin.py
~~~

## Output layout

### Shared output layout

~~~text
./artifacts/instruction_units/
├─ manifest.json
├─ length_field.json
├─ histogram.json
├─ stats.json
├─ sample_index.jsonl
├─ duplicate_index.jsonl
├─ unk_index.jsonl
├─ overflow_index.jsonl
│
├─ unit_128/
│  ├─ shard_000000.bin
│  ├─ shard_000001.bin
│  └─ ...
│
├─ unit_192/
│  ├─ shard_000000.bin
│  ├─ shard_000001.bin
│  └─ ...
│
├─ unit_256/
│  ├─ shard_000000.bin
│  ├─ shard_000001.bin
│  └─ ...
│
└─ unit_320/
   ├─ shard_000000.bin
   ├─ shard_000001.bin
   └─ ...
~~~

Meaning:

~~~text
manifest.json
→ pipeline metadata, build configuration, tokenizer metadata, token dtype, magic, sequence_template

length_field.json
→ per-record header schema readers use to decode shards (magic, sequence_template, dtype, pad_token_id)

histogram.json
→ sample distribution by assigned unit size

stats.json
→ summarized token, shard, duplicate, and UNK statistics

sample_index.jsonl
→ per-sample hash, token count, unit assignment, shard location, empty_fields

duplicate_index.jsonl
→ grouped duplicate samples by hash

unk_index.jsonl
→ samples containing UNK tokens with positions, field indices, and original chars

overflow_index.jsonl
→ samples whose token_count exceeded max(sizes) and how the policy resolved them
  (truncate / extend / skip)

unit_xxx/
→ binary token shards grouped by unit size

unit_xxx/EMPTY_UNIT.md
→ generated when a declared unit receives no assigned samples

unit_xxx/UNIT_BUILD_FAILED.md
→ generated when the build fails before the unit output is finalized
~~~

### manifest.json

Example:

~~~json
{
  "pipeline": "unit-build",
  "mintdim_version": "0.1.18",
  "tokenizer": {
    "type": "sentencepiece",
    "path": "./tokenizer/tokenizer.model",
    "vocab_size": 32000,
    "unk_id": 0,
    "pad_id": 1
  },
  "token_storage": {
    "dtype": "uint16",
    "bytes_per_token": 2,
    "magic": 65535
  },
  "sequence_template": [
    "token_template",
    "instruction",
    "token_template",
    "input",
    "token_template",
    "output"
  ],
  "length_field_metadata": "length_field.json",
  "units": {
    "sizes": [128, 192, 256, 320],
    "build_batch": 4096
  },
  "source": {
    "files": ["./data/alpaca.jsonl"],
    "fields": [["instruction", "input", "output"]],
    "templates": ["{instruction}\n\n{input}\n\n{output}"]
  },
  "output": {
    "samples_per_shard": 10000
  }
}
~~~

If the overflow policy extends `sizes` mid-build, the persisted list includes the new size — manifest reflects the layout that actually exists on disk.

### length_field.json

Example:

~~~json
{
  "magic": 65535,
  "sequence_template": [
    "token_template",
    "instruction",
    "token_template",
    "input",
    "token_template",
    "output"
  ],
  "token_dtype": "uint16",
  "pad_token_id": 0
}
~~~

Readers use this file to:
- verify the magic marker at slot 0 of every record
- decode the N length headers that follow (`N = len(sequence_template)`)
- know which segment (field or literal) each header refers to

`"token_template"` is the reserved name for a literal segment of the template. Field segments use their declared field name.

### histogram.json

Example:

~~~json
{
  "128": 30291,
  "192": 82191,
  "256": 120882,
  "320": 43121
}
~~~

Meaning:

~~~text
histogram.json counts how many samples were assigned to each unit size.
~~~

### stats.json

Example:

~~~json
{
  "total_samples": 1823811,
  "total_tokens": 482918221,
  "token_dtype": "uint16",
  "bytes_per_token": 2,
  "duplicate_groups": 821,
  "duplicate_samples": 1942,
  "samples_with_unk": 321,
  "unk_tokens": 1821,
  "unit_distribution": {
    "128": 582111,
    "192": 891221,
    "256": 301882,
    "320": 48291
  }
}
~~~

### sample_index.jsonl

Each line describes one source sample after tokenization and unit assignment.

Example:

~~~json
{
  "sample_id": 91822,
  "hash": "8f2c1b7d9a9d4a0d...",
  "token_count": 143,
  "unit_size": 192,
  "source_file": "./data/alpaca.jsonl",
  "source_line": 18291,
  "shard_path": "unit_192/shard_000001.bin",
  "empty_fields": []
}
~~~

`empty_fields` lists the field names whose values rendered to an empty string for this sample (the field stays in `sequence_template` with `len_field = 0`). It's a fast filter for finding samples missing actual content in a slot.

Purpose:

~~~text
sample_index.jsonl enables:
- dataset audit
- sample-level reproducibility
- deduplication
- corruption detection
- shard rebuild/debug
- token count verification
- missing-content detection (via empty_fields)
~~~

### duplicate_index.jsonl

Each line groups all duplicated samples sharing the same content hash.

Example:

~~~json
{
  "hash": "8f2c1b7d9a9d4a0d...",
  "count": 3,
  "samples": [
    {
      "sample_id": 12031,
      "source_file": "./data/alpaca.jsonl",
      "source_line": 991
    },
    {
      "sample_id": 91822,
      "source_file": "./data/alpaca.jsonl",
      "source_line": 18291
    },
    {
      "sample_id": 140992,
      "source_file": "./data/sharegpt.jsonl",
      "source_line": 2012
    }
  ]
}
~~~

Purpose:

~~~text
duplicate_index.jsonl enables:
- duplicate audit
- dataset cleanup
- repeated sample detection
- duplicate filtering
- corruption investigation
~~~

### unk_index.jsonl

Each line describes one sample whose tokenization produced UNK. This includes both UNK from field values (Tier 2) and, when the build was allowed to continue past Tier 1, UNK from template literals.

Example:

~~~json
{
  "sample_id": 91822,
  "hash": "8f2c1b7d9a9d4a0d...",
  "source_file": "./data/alpaca.jsonl",
  "source_line": 18291,
  "unk_count": 2,
  "unk_positions": [17, 89],
  "unk_field_indices": [1, 3],
  "unk_chars": ["𓀀", "🜲"]
}
~~~

Field meaning:

~~~text
unk_positions
→ positions of each UNK within the encoded payload (0-base, payload-relative)

unk_field_indices
→ segment index in sequence_template (0-base) where the UNK originated;
  aligned 1:1 with unk_positions

unk_chars
→ original substring from the rendered text that produced each UNK,
  aligned 1:1 with unk_positions
~~~

Purpose:

~~~text
unk_index.jsonl enables:
- tokenizer quality audit
- unknown token analysis (by source segment)
- tokenizer coverage inspection
- dataset cleanup
- encoding issue detection
~~~

### overflow_index.jsonl

Each line records one sample whose `sum(segment token lengths)` exceeded `max(sizes)` and how the policy resolved it.

Example:

~~~json
{
  "sample_id": 91822,
  "source_file": "./data/alpaca.jsonl",
  "source_line": 18291,
  "token_count": 412,
  "max_unit_size": 320,
  "action": "extend",
  "new_unit_size": 512
}
~~~

`action` is one of `"truncate"`, `"extend"`, or `"skip"`. `new_unit_size` appears only when `action == "extend"`.

Purpose:

~~~text
overflow_index.jsonl enables:
- visibility into samples that didn't fit the original sizes plan
- audit trail for the overflow policy resolution
- post-hoc analysis to decide whether to grow `sizes` in the next build
~~~

### Binary token shards

Each record in `unit_xxx/shard_*.bin` is laid out as:

~~~text
[magic] [len_seg_0] [len_seg_1] ... [len_seg_{n-1}] [payload_0 .. payload_{unit_size-1}]
└──1──┘ └──────────── n length headers ────────────┘ └──── payload of length unit_size ────┘
~~~

- All slots use `token_dtype` (uint16 → magic `0xFFFF`, uint32 → magic `0xFFFFFFFF`).
- `n = len(sequence_template)`. Segment names live in `length_field.json`.
- `unit_size` counts payload slots only. The first `1 + n` slots are header metadata, not training tokens.
- The payload concatenates each segment's tokens in `sequence_template` order, then pads with `pad_id` to `unit_size`.
- Total record size in slots = `1 + n + unit_size`; in bytes = that × `bytes_per_token`.
- The k-th record in a shard starts at byte `k * (1 + n + unit_size) * bytes_per_token`.

To slice a segment back out:

~~~text
payload_offset    = 1 + n
segment_i_start   = payload_offset + sum(len_seg_0..i-1)
segment_i_end     = segment_i_start + len_seg_i
~~~

`magic` is reserved — readers can verify it at slot 0 of every record to catch shard corruption.

### Multi-output layout

~~~text
./artifacts/
├─ alpaca_units/
│  ├─ manifest.json
│  ├─ histogram.json
│  ├─ stats.json
│  ├─ sample_index.jsonl
│  ├─ duplicate_index.jsonl
│  ├─ unk_index.jsonl
│  │
│  ├─ unit_128/
│  │  ├─ shard_000000.bin
│  │  └─ ...
│  ├─ unit_192/
│  │  ├─ shard_000000.bin
│  │  └─ ...
│  ├─ unit_256/
│  │  ├─ shard_000000.bin
│  │  └─ ...
│  └─ unit_320/
│     ├─ shard_000000.bin
│     └─ ...
│
└─ sharegpt_units/
   ├─ manifest.json
   ├─ histogram.json
   ├─ stats.json
   ├─ sample_index.jsonl
   ├─ duplicate_index.jsonl
   ├─ unk_index.jsonl
   │
   ├─ unit_96/
   │  ├─ shard_000000.bin
   │  └─ ...
   ├─ unit_128/
   │  ├─ shard_000000.bin
   │  └─ ...
   ├─ unit_160/
   │  ├─ shard_000000.bin
   │  └─ ...
   ├─ unit_192/
   │  ├─ shard_000000.bin
   │  └─ ...
   ├─ unit_224/
   │  ├─ shard_000000.bin
   │  └─ ...
   └─ unit_320/
      ├─ shard_000000.bin
      └─ ...
~~~

Each output directory is isolated and mapped by input order when
`output.dir` has one path per input file. Other blocks may still be shared:

~~~text
files[i]
↔ fields[0] or fields[i]
↔ templates[0] or templates[i]
↔ tokenizer[0] or tokenizer[i]
↔ sizes[0] or sizes[i]
↔ build_batch[0] or build_batch[i]
↔ output_dir[i]
↔ samples_per_shard[0] or samples_per_shard[i]
~~~

## Design principles

~~~text
small public API
explicit paths
explicit user-defined templates (no presets)
required-field sample validation (extra JSONL keys are ignored)
fail fast validation
two-tier UNK contract (template prompt vs data record)
interactive overflow resolution (truncate / extend / skip)
exact per-segment tokenization
direct unit assignment by token length
fixed-width binary shards with [magic][lengths][payload] framing
length_field.json describes per-record header schema for readers
sequence_template preserves field/literal boundaries on disk
automatic token dtype selection with magic-collision guard
no tokenizer mutation
special token ids loaded from tokenizer library
no automatic repo scanning
no overwrite in V1
no build resume/cache
pipeline-specific internals
sample-level reproducibility
dataset quality audit support (sample_index, unk_index, overflow_index, duplicate_index)
~~~

## Version

Initial public version:

~~~text
0.1.18
~~~
## Documentation

- `docs/apis/unit_build.md`
- `docs/apis/unit_read.md`
- `docs/apis/unit_read/queue.md`
- `docs/apis/unit_read/binary_contract.md`
- `docs/apis/unit_read/batch_contract.md`
- `docs/apis/unit_read/cli.md`
- `docs/apis/unit_read/errors.md`
- `docs/apis/unit_pipeline.md`