Byte Tokenizer turns text into token IDs and back again. It splits text into deterministic subwords and packs each one into bijective base-256 chunk IDs, up to 2 UTF-8 bytes per ID, capped at 65792, so every text has exactly one encoding and every ID decodes cleanly. Optional special tokens start at 65793 and match longest-first, so <|im_start|> is never split into a shorter <|im|> prefix. There is no vocabulary file, no training, no lowercase folding, and no Unicode normalization, so case, combining marks, null bytes, and a leading BOM stay exactly as encoded. Duplicate special tokens are ignored, and invalid UTF-8 bytes decode as U+FFFD.
Deno:
deno add npm:@neabyte/byte-tokenizernpm:
npm install @neabyte/byte-tokenizerCDN (jsDelivr/esm.sh):
<script type="module">
import Tokenizer from 'https://cdn.jsdelivr.net/npm/@neabyte/byte-tokenizer/dist/index.mjs'
</script>Or via esm.sh:
<script type="module">
import Tokenizer from 'https://esm.sh/@neabyte/byte-tokenizer'
</script>import Tokenizer from '@neabyte/byte-tokenizer'
// Deterministic subword chunks
const tokenizer = new Tokenizer()
const rawIds = tokenizer.encode('Hello!')
// [18790, 28013, 112, 34]
const rawText = tokenizer.decode(rawIds)
// 'Hello!'
// Special tokens start at 65793
const customTokenizer = new Tokenizer(['<|bos|>', '<|eos|>'])
const customIds = customTokenizer.encode('<|bos|>Hello!<|eos|>')
// [65793, 18790, 28013, 112, 34, 65794]
const customText = customTokenizer.decode(customIds)
// '<|bos|>Hello!<|eos|>'new Tokenizer(specialToken?)specialToken<string[]>: (Optional) Special token strings, defaulting to[]. Empty strings are rejected, duplicates keep the first ID, and IDs start at65793in first-seen order.- Throws:
TypeErrorwhen the argument is not an array, or a token is empty or not a string.
tokenizer.encode(text)text<string>: Text to encode. Plain spans become subword chunk IDs, while special tokens collapse to a single ID, and the longest match wins.- Returns:
<number[]> - Throws:
TypeErrorwhentextis not a string or contains an unpaired surrogate.
tokenizer.decode(identifier)identifier<ArrayLike<number>>: Token IDs, including typed arrays. IDs1to65792unpack into subword bytes, IDs65793and above expand to their special-token text, and a leading BOM is kept.- Returns:
<string> - Throws:
TypeErrorwhen the value is not array-like, or an ID is not an integer. - Throws:
RangeErrorwhen an ID is negative,-0, or outside the vocabulary.
npm run builddeno task checkdeno task testThis project is licensed under the MIT License.