Metadata-Version: 2.1
Name: llama_tokens
Version: 0.0.1
Summary: A Quick Library with Llama 3.1 & 3.2 Tokenization
Author-email: Jeff Tang <jeffxtang@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/jeffxtang/llama-tokens
Keywords: llama 3,tokenization,tokens,llama-tokens
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tiktoken==0.8.0
Requires-Dist: blobfile==3.0.0

# llama-tokens
A Quick Library with Llama 3.1 &amp; 3.2 Tokenization
