BPE Visualizer
?
In computing, byte-pair encoding (BPE), or digram coding, is an algorithm, first described in 1994 by Philip Gage, for encoding strings of text into smaller strings by creating and using a translation table. A slightly modified version of the algorithm is used in large language model tokenizers.
296
Characters
0
Merges
0
Tokens
⏮
⏴
▶
⏵
⏭
Step 0/-1
Max
Merges (0)
No merges yet
Tokens
Text
UTF-8
Enter text above to see tokens