Object Counting Benchmark

Each model answers the same counting prompts and is asked for a single number per image. An LLM judge grades every answer against the ground-truth count, and the score is the percentage graded correct. The strict-match column only credits an answer that contains exactly one integer equal to the count.

Counting requires finding every instance of the target, including partially hidden or overlapping ones, without double-counting. Cluttered scenes, small objects, and look-alike distractors all cause miscounts, which makes this one of the most failure-prone tasks for current models.

Strict accuracy counts an answer only when it matches the ground truth exactly after normalization. LLM-judged accuracy asks a second model acting as judge, Gemini 3.5 Flash at temperature zero, to compare the answer with the ground truth and decide whether it is correct, so a right answer still counts when it arrives with an explanation or in different words. The judge is the headline score because it measures whether the model got it right rather than whether it was terse; the strict score sits beside it on each task page, and the gap between the two shows how verbose a model is.

Every task is run at two levels of reasoning effort. The low row is the model answering with minimal deliberation, the high row is the same model on the same questions allowed to think longer. Both rows are ranked together so you can see whether the extra thinking is worth its cost and latency, which the Est. cost and Speed columns show, and the effort filter above the table shows one tier at a time. Models benchmarked before the current protocol have a high-effort row for Reasoning only until they are re-run. Every other table on this site, including the overall Average, uses the low-effort pass so all models are compared on the same amount of compute.

Under the current protocol each task is run three times at each effort tier, and the score shown is the mean over those runs. The ± is half the range between the lowest and highest run, so it shows how much the score moved between runs; it is a range, not a confidence interval. Hover the ± for the run count and range. A score without one comes from a single run; models benchmarked before the protocol are being re-run under it. OCR results are currently a single preliminary run per model, so they carry no ±.

Most models in this leaderboard link to their Playground page. Click the model name to open it, then upload your own image and run it. A few models are benchmarked for comparison only and do not have a Playground page yet.