How to contribute

• If there is a model or benchmark you would like to see here, please submit it as a GitHub issue with the evaluation request form.
• If you would like to collaborate on LLM evaluation research, or can offer data, GPU time or compute credits, please contact me at seungtaek.choi@hufs.ac.kr.

Click a benchmark name to show its subtasks, a column title to sort, a model name for details.

• to be added not evaluated yet and not in the queue yet
• queued in the queue
• running being evaluated now
• 0.0 the model cannot take this benchmark, such as images for a text-only model; counted as 0 in averages, hover for the reason
• not supported yet this leaderboard does not run the benchmark with this setting yet
• 68.1* average over only some of the checked benchmarks; hover for which are missing. Rows with fewer than half of them come after the other rows when sorted by Average or Avg rank
• – no value

×

Score vs. cost

Each point is one model in one thinking setting, and a line joins the settings of the same model. Hollow points are thinking off. The dotted line is the Pareto front: no other point scores higher at a lower cost. Hover a point for details. The model filters, the benchmarks checked for Average and the column grouping of the table above apply here too.

How models are evaluated

• Each model is evaluated with lm-evaluation-harness on vLLM, in bfloat16 with greedy decoding and the model's own chat template.
• Decoding follows each model's own recommendation: the sampling values its card gives for that thinking mode, or its generation_config.json, with greedy decoding when it recommends none (seed 1234). Runs before 2026-11-05 used greedy decoding for every model and are kept as separate greedy rows, the way thinking off and on are separate rows. Hovering a cell shows the values used and how many items hit the generation limit of this evaluation.
• Thinking is the chat template's thinking mode. off and on are for models with an on/off switch, where on is the model's own default amount of thinking. low, medium and high are for models that take an effort level instead. With thinking on, the answer may use the output length the model recommends, or 30,720 tokens when it gives none, in a context of that length plus 8,192 tokens for the prompt (about 28,000 for LoCoMo, whose conversations are long), within the model's own context window. MIH, BFCL and ChemBench, which run on their official code, may use whatever the prompt leaves of that context. Only the text after the thinking section is scored. A response whose thinking does not finish within that limit counts as unanswered.
• With thinking off, benchmarks run in an 8,192-token context (AIME 2025, LoCoMo and image benchmarks in 32K). A model whose context window is shorter is run within its own window, with up to half of it for the answer; items that do not fit count as wrong. LoCoMo inputs alone run to about 26,000 tokens, so such a model shows it as one it cannot take.
• Benchmarks marked image come with images and are run only on models that take images, with every image scaled down to at most 2,048 pixels on the longer side.
• Average is the macro average of the official scores of the checked benchmarks, and Avg rank uses the same benchmarks. All benchmarks are checked by default, image benchmarks included. A benchmark a model cannot take (for example, images for a text-only model) counts as 0 unless you choose to leave it out, and a benchmark this leaderboard does not run with a setting yet is always left out. Use the button under Average to change either.
• Columns can be regrouped by benchmark properties such as language, task type, or domain with the tags above the Model column. A group shows the macro average of the columns under it (and the micro average, weighted by items, in its tooltip when every column is an accuracy); a value marked * is missing some columns.
• Eval time is the wall-clock time of one benchmark on the listed GPU, including model loading.
• Output tokens are the generated tokens per item, counted with the model's own tokenizer and including any thinking text.
• GPU memory is the model weights plus the KV cache for one 8,192-token request, as allocated by vLLM.
• Only models that load with standard transformers architectures (no remote code) are evaluated automatically. Others are reviewed first.

Flag icons are from Twemoji, licensed under CC-BY 4.0.