BPE vs Tiktoken Comparison
Input Text
Your Custom BPE
Token IDs
Subword Tokens
Tiktoken (GPT-2)
Token IDs
Subword Tokens