IDEA Research: Grounded SAM

Released on November 19th, 2025, Segment Anything 3 (SAM 3) is a zero-shot image segmentation model that “detects, segments, and tracks objects in images and videos based on concept prompts.” This model was developed by Meta as the third model in the Segment Anything series.

Unlike its previous SAM models (Segment Anything and Segment Anything 2), you can provide SAM 3 with the prompt “shipping container” and it will generate precise segmentation masks for all shipping containers in an image. SAM 3 generates segmentation masks that correspond to the location of the objects found with a text prompt.

Grounding DINO is an open-vocabulary object detection model developed by IDEA Research, released in March 2023 under the Apache 2.0 license. It extends the DINO transformer-based detector with grounded pre-training, enabling it to detect arbitrary objects described by free-form text queries rather than a fixed set of predefined categories. The model integrates a text encoder with a visual backbone through a feature fusion module that aligns language and visual representations at multiple scales.Grounding DINO achieves strong zero-shot detection performance on COCO, LVIS, and ODinW benchmarks, and supports referring expression comprehension tasks. It is widely used as a foundation for open-vocabulary detection pipelines and as the detection backbone in systems such as Grounded-SAM. The model is particularly suited for applications requiring flexible, text-driven object localization across diverse domains.

YOLOE (YOLO with Everything) is an open-vocabulary object detection and segmentation model developed by THU-MIG at Tsinghua University, released in March 2025 under the AGPL-3.0 license. It extends the YOLO architecture to support open-vocabulary detection through text and visual prompts, enabling the model to detect arbitrary object categories beyond a fixed training set without retraining. The design integrates prompt encoding directly into the YOLO framework while preserving real-time inference speed.YOLOE is evaluated on COCO and LVIS benchmarks and supports both closed-set and open-vocabulary detection modes. It is built on the Ultralytics codebase and maintains compatibility with standard YOLO training and export workflows. YOLOE is suited for applications requiring flexible, prompt-driven object detection where the target object vocabulary may change at inference time.

YOLO-World v2 Small (YOLO-World-S-v2) is the smallest variant of Tencent AI Lab’s YOLO-World v2 family, released around February 2024 under GPL-v3. With ~13 million parameters, it adopts a prompt-then-detect paradigm using offline vocabularies and is pretrained on large-scale datasets such as Objects365 and GoldG. The model processes image inputs at 640×640 or 1280×1280 resolutions and supports zero-shot open-vocabulary object detection, enabling recognition of novel categories from text prompts without retraining.Evaluations show competitive results across benchmarks like LVIS and COCO, while maintaining real-time efficiency. On an NVIDIA V100, the small variant reaches ~74 FPS at standard resolutions. Together with larger YOLO-World v2 models, it provides a scalable framework for efficient, open-vocabulary detection across diverse deployment settings.

Florence-2, introduced by Microsoft Research at CVPR 2024, is an open-source vision-language foundation model designed to unify diverse computer vision tasks within a single sequence-to-sequence framework. Unlike traditional models that specialize in specific tasks, Florence-2 accepts both images and text prompts and outputs text for tasks such as captioning, object detection, segmentation, OCR, and region-based grounding. It comes in two sizes—Florence-2-base (~230M parameters) and Florence-2-large (~770M parameters)—and is trained on FLD-5B, a large dataset of ~126M images with ~5.4B annotations.The model demonstrates strong zero-shot and fine-tuned performance, often rivaling larger vision-language systems while remaining lightweight and efficient. Released under the MIT license, all weights are publicly available, making it accessible for fine-tuning and deployment in applications like VQA, content tagging, accessibility, and research. Florence-2’s compact design, versatility, and openness position it as a practical alternative to larger proprietary multimodal models.

SAM-CLIP is a unified vision foundation model introduced by researchers at Apple and the University of Illinois Urbana-Champaign in October 2023. It merges two popular vision foundation models — Meta's Segment Anything Model (SAM) and OpenAI's CLIP — into a single shared Vision Transformer backbone through a combination of multi-task learning, continual learning, and teacher-student distillation. The method requires only a small fraction of the original pretraining datasets and demonstrates that complementary capabilities from distinct foundation models can be consolidated without retraining from scratch, reducing the storage and compute cost of running both models in inference.The resulting model retains SAM's zero-shot segmentation ability and CLIP's zero-shot classification and image-text retrieval, while introducing new capabilities the individual models lacked. SAM-CLIP establishes state-of-the-art results on zero-shot semantic segmentation across five benchmarks, improving mean IoU by 6.8 points on Pascal VOC and 5.9 points on COCO-Stuff over prior specialized models. The paper was accepted at the UniReps Workshop at NeurIPS 2023 and the eLVM Workshop at CVPR 2024. Apple has published the research but has not released model weights or inference code publicly.