Advanced Tokenizer Visualization

Learn how the Artificial Intelligence sees the ~~world~~ your text

Select Tokenizers to Load:

GPT-2

BERT

Llama

Qwen2

DeepSeek

Select tokenizers and they will load automatically. First-time loading may take a moment. This entirely runs locally on your device.

Enter text to tokenize:

GPT-2 Tokenizer

Ready to load...

Select GPT-2 above to load tokenizer

BERT Tokenizer

Ready to load...

Select BERT above to load tokenizer

T5 Tokenizer

Ready to load...

Select T5 above to load tokenizer

Llama Tokenizer

Ready to load...

Select Llama above to load tokenizer

Qwen2 Tokenizer

Ready to load...

Select Qwen2 above to load tokenizer

DeepSeek Tokenizer

Ready to load...

Select DeepSeek above to load tokenizer

Select tokenizers above to see comparison