Vision Evals (legacy)
Looking for rankings based on real user votes? See Arena Rankings
This is the legacy version of Vision Evals, our previous benchmark. It is preserved for reference and for models that predate the current benchmark, including deprecated models that can no longer be re-run. See the current Vision Evals
Open-vocabulary object detection benchmarks. Models draw bounding boxes from text queries across SaCo-Gold and COCO-100.
What is Detection?
Detection evals measure how well models localize objects given a text query. We combine results from SaCo-Gold (attributes/crowded/metaclip configs) and a 100-image COCO val2017 subset.
Methodology
For each (image, text-query) pair the model returns bounding-box predictions with confidence scores. We compute positive micro-F1 (pmF1) across IoU thresholds 0.5:0.05:0.95. The overall score is the query-count-weighted average pmF1 across datasets.
Token usage & cost. Where shown, “output tokens” is the median per-prompt output count measured directly from each provider’s API response, and includes reasoning / thinking tokens, normalized across providers so the figure is comparable (for example, Gemini reports reasoning separately, and we add it into the output count). Input tokens include image tokens, which dominate and differ by model. “Est. cost / task” is that measured token usage multiplied by the model’s published per-1M pricing at the time of our last price sync, so it is an estimate on this benchmark, not a universal model cost. Figures come from a single evaluation run at low temperature; output for reasoning models can vary run to run. Models we haven’t measured (or that don’t expose token usage) show no token or cost figure rather than a zero.
Last evaluated: July 9, 2026
Frequently Asked Questions
Positive micro-F1 is the F1 score computed over all predictions and ground-truth boxes on images where the queried class is present. We average it over IoU thresholds 0.5:0.05:0.95, matching the COCO convention.
SaCo-Gold (facebook/SACo-Gold) and a 100-image subset of COCO val2017. We weight each dataset by query count when computing the combined score.
