Improve Unigram tokenization performance for long inputs#34

bepsvpt wants to merge 1 commit into

huggingface:mainhuggingface/tokenizers.js:mainfrom

bepsvpt:fix/unigram-quadratic-tokenizationbepsvpt/tokenizers.js:fix/unigram-quadratic-tokenizationCopy head branch name to clipboard