Comma separates patterns. * matches anything, /…/ is a regular expression, a leading ! excludes. Click a benchmark name to show its subtasks, a column title to sort, a model name for details.

×

Score vs. cost

Each point is one model in one reasoning setting, and a line joins the settings of the same model. Hollow points are thinking off. The dotted line is the Pareto front: no other point scores higher at a lower cost. Hover a point for details. The filters above apply here too.

How models are evaluated

• Each model is evaluated with lm-evaluation-harness on vLLM, in bfloat16 with greedy decoding and the model's own chat template.
• Reasoning records whether the chat template's thinking mode was on, and at which effort when the model supports it. With thinking on, every benchmark allows up to 30,720 generated tokens (LoCoMo 4,096) in a 32K context, and only the text after the thinking section is scored. A response whose thinking does not finish within that limit counts as unanswered.
• Benchmarks marked image come with images and are run only on models that take images, with every image scaled down to at most 2,048 pixels on the longer side. They are not part of the Average or Avg rank.
• Eval time is the wall-clock time of one benchmark on the listed GPU, including model loading.
• Output tokens are the generated tokens per item, counted with the model's own tokenizer and including any thinking text.
• GPU memory is the model weights plus the KV cache for one 8,192-token request, as allocated by vLLM.
• Only models that load with standard transformers architectures (no remote code) are evaluated automatically. Others are reviewed first.

Flag icons are from Twemoji, licensed under CC-BY 4.0.

Collaborate with us