IDEA Research: Grounding DINO

Florence-2, introduced by Microsoft Research at CVPR 2024, is an open-source vision-language foundation model designed to unify diverse computer vision tasks within a single sequence-to-sequence framework. Unlike traditional models that specialize in specific tasks, Florence-2 accepts both images and text prompts and outputs text for tasks such as captioning, object detection, segmentation, OCR, and region-based grounding. It comes in two sizes—Florence-2-base (~230M parameters) and Florence-2-large (~770M parameters)—and is trained on FLD-5B, a large dataset of ~126M images with ~5.4B annotations.The model demonstrates strong zero-shot and fine-tuned performance, often rivaling larger vision-language systems while remaining lightweight and efficient. Released under the MIT license, all weights are publicly available, making it accessible for fine-tuning and deployment in applications like VQA, content tagging, accessibility, and research. Florence-2’s compact design, versatility, and openness position it as a practical alternative to larger proprietary multimodal models.

OWL-ViT (Open-World Localization with Vision Transformers) is an open-vocabulary object detection model released in May 2022 by Google Research. It adapts a pretrained CLIP-style image-text model by removing the final pooling layer and attaching lightweight classification and box prediction heads to each Transformer output token, producing a detector capable of localizing arbitrary objects described by free-form text at inference time. Rather than being restricted to a fixed taxonomy such as the 80 categories in Microsoft COCO, OWL-ViT can detect object classes specified by a user's text query, including categories the model was never explicitly trained on.OWL-ViT accepts an image and a list of text queries as input, and produces bounding boxes with class assignments drawn from the supplied queries. It also supports one-shot image-conditioned detection, where a cropped image region is used as the query instead of text, allowing the model to find visually similar instances within a target scene. The model is released in multiple Vision Transformer sizes (ViT-B/32, ViT-B/16, ViT-L/14) and CLIP-pretrained variants, distributed through the Google Research scenic repository and Hugging Face under the Apache 2.0 license. A successor model, OWLv2, was released in June 2023, introducing the OWL-ST self-training recipe that scales training to over one billion pseudo-annotated examples and substantially improves detection performance on rare and long-tail categories while preserving the open-vocabulary interface.

YOLO-World v2 Small (YOLO-World-S-v2) is the smallest variant of Tencent AI Lab’s YOLO-World v2 family, released around February 2024 under GPL-v3. With ~13 million parameters, it adopts a prompt-then-detect paradigm using offline vocabularies and is pretrained on large-scale datasets such as Objects365 and GoldG. The model processes image inputs at 640×640 or 1280×1280 resolutions and supports zero-shot open-vocabulary object detection, enabling recognition of novel categories from text prompts without retraining.Evaluations show competitive results across benchmarks like LVIS and COCO, while maintaining real-time efficiency. On an NVIDIA V100, the small variant reaches ~74 FPS at standard resolutions. Together with larger YOLO-World v2 models, it provides a scalable framework for efficient, open-vocabulary detection across diverse deployment settings.

Grounded SAM is an open-vocabulary image segmentation model developed by IDEA Research, released in January 2024 under the Apache 2.0 license. It combines Grounding DINO, a zero-shot open-vocabulary object detector, with the Segment Anything Model to produce precise segmentation masks for objects identified through free-form text prompts. The two models are used sequentially: Grounding DINO localizes objects from a text query, and SAM generates the corresponding segmentation masks.Grounded SAM enables zero-shot instance segmentation without task-specific training data, making it applicable to domains where labeled segmentation data is scarce. It supports arbitrary text queries and can segment objects not represented in standard training sets. The model is commonly used in automated labeling pipelines, robotic perception, and domain-specific vision applications requiring open-vocabulary segmentation.

YOLOE (YOLO with Everything) is an open-vocabulary object detection and segmentation model developed by THU-MIG at Tsinghua University, released in March 2025 under the AGPL-3.0 license. It extends the YOLO architecture to support open-vocabulary detection through text and visual prompts, enabling the model to detect arbitrary object categories beyond a fixed training set without retraining. The design integrates prompt encoding directly into the YOLO framework while preserving real-time inference speed.YOLOE is evaluated on COCO and LVIS benchmarks and supports both closed-set and open-vocabulary detection modes. It is built on the Ultralytics codebase and maintains compatibility with standard YOLO training and export workflows. YOLOE is suited for applications requiring flexible, prompt-driven object detection where the target object vocabulary may change at inference time.

DETR (Detection Transformer) is an end-to-end object detection model developed by Facebook Research (Meta), released in May 2020. It is one of the first models to eliminate hand-crafted components such as anchor generation and non-maximum suppression by framing object detection as a direct set prediction problem, solved with a transformer encoder-decoder architecture built on top of a CNN backbone.DETR achieves 42.0% AP on the COCO benchmark with a ResNet-50 backbone, performing comparably to a well-tuned Faster R-CNN at the time of release. Its attention-based design allows it to reason about global context and long-range dependencies within an image. DETR is primarily used as a research baseline and architectural reference, with subsequent works such as Deformable DETR and DINO building on its foundations to address its slower training convergence and limited small-object detection capability.