Metadata-Version: 2.4
Name: code-weaver-tokenizers
Version: 0.1.0
Summary: Ready-to-use tokenizers for CodeWeaver — tiktoken and HuggingFace tokenizers with a unified interface
Project-URL: Homepage, https://knitli.com
Project-URL: Repository, https://github.com/knitli/codeweaver
Project-URL: Issues, https://github.com/knitli/codeweaver/issues
Author: Knitli Inc.
Author-email: Adam Poulemanos <adam@knit.li>
License-Expression: MIT OR Apache-2.0
Keywords: codeweaver,embeddings,nlp,tiktoken,tokenizers
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: tiktoken>=0.12.0
Requires-Dist: tokenizers>=0.22.1
Requires-Dist: typing-extensions>=4.15.0
Provides-Extra: test
Requires-Dist: pytest; extra == 'test'
Requires-Dist: pytest-cov; extra == 'test'
Description-Content-Type: text/markdown

<!--
SPDX-FileCopyrightText: 2025 Knitli Inc.
SPDX-FileContributor: Adam Poulemanos <adam@knit.li>

SPDX-License-Identifier: MIT OR Apache-2.0
-->

# codeweaver-tokenizers

Independent tokenizer package for CodeWeaver.
