Multilingual Tokenizer Leaderboard: likes over time
eduagarcia/multilingual-tokenizer-leaderboard is a docker Space on the Hugging Face Hub. It has 15 likes, 0 in the last 7 days and 1 in the last 30, and ranks #6,613 among Spaces by likes.
Evaluating and comparing tokenizers of language models
What it uses
- sentence-transformers/all-MiniLM-L6-v2 (model)
- sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 (model)
- google/electra-base-discriminator (model)
- google-bert/bert-base-uncased (model)
- BAAI/bge-m3 (model)
- google-t5/t5-small (model)
- openai/clip-vit-base-patch32 (model)
- sentence-transformers/all-mpnet-base-v2 (model)
- BAAI/bge-reranker-v2-m3 (model)
- jonatasgrosman/wav2vec2-large-xlsr-53-japanese (model)
- FacebookAI/xlm-roberta-base (model)
- openai-community/gpt2 (model)
- nomic-ai/nomic-embed-text-v1.5 (model)
- google/gemma-4-31B-it (model)
- sentence-transformers/paraphrase-multilingual-mpnet-base-v2 (model)
- BAAI/bge-large-en-v1.5 (model)
- Qwen/Qwen3-Embedding-0.6B (model)
- openai/clip-vit-large-patch14 (model)
- distilbert/distilbert-base-uncased (model)
- FacebookAI/roberta-base (model)
- facebook/opt-125m (model)
- meta-llama/Llama-3.1-8B-Instruct (model)
- microsoft/mdeberta-v3-base (model)
- jonatasgrosman/wav2vec2-large-xlsr-53-portuguese (model)
- ProsusAI/finbert (model)
- openai/gpt-oss-120b (model)
- stabilityai/stable-diffusion-xl-base-1.0 (model)
- jonatasgrosman/wav2vec2-large-xlsr-53-russian (model)
- answerdotai/ModernBERT-base (model)
- openai/whisper-large-v3 (model)
- google-bert/bert-base-cased (model)
- facebook/esm2_t33_650M_UR50D (model)
- meta-llama/Prompt-Guard-86M (model)
- openai/whisper-small (model)
- w11wo/indonesian-roberta-base-posp-tagger (model)
- distilbert/distilbert-base-uncased-finetuned-sst-2-english (model)
- nomic-ai/nomic-embed-text-v1 (model)
- google-bert/bert-base-multilingual-uncased (model)
- facebook/bart-large-mnli (model)
- SamLowe/roberta-base-go_emotions (model)
Open eduagarcia/multilingual-tokenizer-leaderboard on Hugging Face