Skip to content

Latest commit

 

History

6 Commits

Folders and files

Repository files navigation

Byte Tokenizer

Encode text into deterministic byte IDs and special tokens

Deno Node Bun Browser License

What is Byte Tokenizer?

Byte Tokenizer turns text into token IDs and back again. It splits text into deterministic subwords and packs each one into bijective base-256 chunk IDs, up to 2 UTF-8 bytes per ID, capped at 65792, so every text has exactly one encoding and every ID decodes cleanly. Optional special tokens start at 65793 and match longest-first, so <|im_start|> is never split into a shorter <|im|> prefix. There is no vocabulary file, no training, no lowercase folding, and no Unicode normalization, so case, combining marks, null bytes, and a leading BOM stay exactly as encoded. Duplicate special tokens are ignored, and invalid UTF-8 bytes decode as U+FFFD.

Installation

Deno:

deno add npm:@neabyte/byte-tokenizer

npm:

npm install @neabyte/byte-tokenizer

CDN (jsDelivr/esm.sh):

<script type="module">
  import Tokenizer from 'https://cdn.jsdelivr.net/npm/@neabyte/byte-tokenizer/dist/index.mjs'
</script>

Or via esm.sh:

<script type="module">
  import Tokenizer from 'https://esm.sh/@neabyte/byte-tokenizer'
</script>

Usage

import Tokenizer from '@neabyte/byte-tokenizer'

// Deterministic subword chunks
const tokenizer = new Tokenizer()

const rawIds = tokenizer.encode('Hello!')
// [18790, 28013, 112, 34]

const rawText = tokenizer.decode(rawIds)
// 'Hello!'

// Special tokens start at 65793
const customTokenizer = new Tokenizer(['<|bos|>', '<|eos|>'])

const customIds = customTokenizer.encode('<|bos|>Hello!<|eos|>')
// [65793, 18790, 28013, 112, 34, 65794]

const customText = customTokenizer.decode(customIds)
// '<|bos|>Hello!<|eos|>'

API

constructor

new Tokenizer(specialToken?)
  • specialToken <string[]>: (Optional) Special token strings, defaulting to []. Empty strings are rejected, duplicates keep the first ID, and IDs start at 65793 in first-seen order.
  • Throws: TypeError when the argument is not an array, or a token is empty or not a string.

encode

tokenizer.encode(text)
  • text <string>: Text to encode. Plain spans become subword chunk IDs, while special tokens collapse to a single ID, and the longest match wins.
  • Returns: <number[]>
  • Throws: TypeError when text is not a string or contains an unpaired surrogate.

decode

tokenizer.decode(identifier)
  • identifier <ArrayLike<number>>: Token IDs, including typed arrays. IDs 1 to 65792 unpack into subword bytes, IDs 65793 and above expand to their special-token text, and a leading BOM is kept.
  • Returns: <string>
  • Throws: TypeError when the value is not array-like, or an ID is not an integer.
  • Throws: RangeError when an ID is negative, -0, or outside the vocabulary.

Build

npm run build

Testing

deno task check
deno task test

License

This project is licensed under the MIT License.