Vision Evals
Looking for rankings based on real user votes? See Arena Rankings
Open-vocabulary instance segmentation benchmarks. Models output pixel masks from text queries across SaCo-Gold and COCO-100.
What is Segmentation?
Segmentation evals measure pixel-level localization given a text query. Same datasets as Detection, but scored on mask IoU instead of bounding-box IoU.
Methodology
For each (image, text-query) pair the model returns mask predictions with confidence scores. We compute positive micro-F1 (pmF1) on mask IoU across thresholds 0.5:0.05:0.95. The overall score is the query-count-weighted average pmF1 across datasets.
Token usage & cost. Where shown, “output tokens” is the median per-prompt output count measured directly from each provider’s API response, and includes reasoning / thinking tokens, normalized across providers so the figure is comparable (for example, Gemini reports reasoning separately, and we add it into the output count). Input tokens include image tokens, which dominate and differ by model. “Est. cost / task” is that measured token usage multiplied by the model’s published per-1M pricing at the time of our last price sync, so it is an estimate on this benchmark, not a universal model cost. Figures come from a single evaluation run at low temperature; output for reasoning models can vary run to run. Models we haven’t measured (or that don’t expose token usage) show no token or cost figure rather than a zero.
Last evaluated: July 9, 2026
Frequently Asked Questions
Same datasets and queries, but scored on mask IoU (pixel-level agreement) instead of bounding-box IoU. Only models that produce masks appear in the Segmentation leaderboard.
SaCo-Gold (facebook/SACo-Gold) and a 100-image subset of COCO val2017, filtered to results that include segmentation masks.