Meta: Segment Anything Model (SAM)

SAM 2 is a real-time image and video segmentation model developed by Meta AI, released in July 2024 under the Apache 2.0 license. It extends the original Segment Anything Model to support video inputs by introducing a streaming memory architecture that maintains object state across frames, enabling consistent segmentation of objects through occlusion, motion, and scene changes. For image inputs, SAM 2 operates similarly to its predecessor with improved mask quality and speed.SAM 2 accepts point, box, and mask prompts and produces object masks interactively or in a fully automated mode. Its memory architecture enables video segmentation at real-time speeds. SAM 2 is used in annotation pipelines, video analysis, robotic perception, and any application requiring high-quality promptable segmentation across both images and video.

Mask R-CNN is an instance segmentation model developed by Facebook AI Research (Meta), released in October 2017. It extends Faster R-CNN by adding a parallel branch that predicts binary segmentation masks for each detected object, independent of the classification and bounding box regression branches. A key contribution is RoIAlign, which replaces RoIPool with bilinear interpolation to preserve spatial correspondence between features and input pixels, significantly improving mask quality.Mask R-CNN achieves strong performance on the COCO instance segmentation benchmark and supports keypoint detection as an additional output head. It remains a foundational architecture in instance segmentation and is available through Meta's Detectron2 framework. The model is most appropriate for tasks requiring pixel-level object delineation, such as medical imaging, autonomous driving, and industrial inspection.

YOLOv8 Instance Segmentation is the segmentation variant of the YOLOv8 model developed by Ultralytics, released in January 2023 under the AGPL-3.0 license. It extends the standard YOLOv8 detection head with a mask prediction branch that generates pixel-level segmentation masks for each detected object using a prototype mask approach. This enables real-time instance segmentation within a single forward pass.YOLOv8 Instance Segmentation shares the same backbone and neck architecture as the base detection model and is available in the same size range. It is deployable through Roboflow Inference and supports fine-tuning on custom COCO-format segmentation datasets. It is suited for applications requiring both object localization and precise mask prediction at real-time speeds.

YOLO11 is an object detection and multi-task vision model developed by Ultralytics, released in September 2024 under the AGPL-3.0 license. It is the latest generation in the Ultralytics YOLO series and supports object detection, instance segmentation, image classification, pose estimation, and oriented bounding box detection within a single unified framework. YOLO11 introduces architectural refinements that improve accuracy while reducing parameter count compared to YOLOv8 at equivalent model sizes.YOLO11 is available in five model sizes from Nano to Extra Large and is deployable through the Ultralytics Python package, Roboflow Inference, and export formats including ONNX, TensorRT, and CoreML. It supports fine-tuning on custom datasets through the standard Ultralytics training API.

Grounded SAM is an open-vocabulary image segmentation model developed by IDEA Research, released in January 2024 under the Apache 2.0 license. It combines Grounding DINO, a zero-shot open-vocabulary object detector, with the Segment Anything Model to produce precise segmentation masks for objects identified through free-form text prompts. The two models are used sequentially: Grounding DINO localizes objects from a text query, and SAM generates the corresponding segmentation masks.Grounded SAM enables zero-shot instance segmentation without task-specific training data, making it applicable to domains where labeled segmentation data is scarce. It supports arbitrary text queries and can segment objects not represented in standard training sets. The model is commonly used in automated labeling pipelines, robotic perception, and domain-specific vision applications requiring open-vocabulary segmentation.

SAM-CLIP is a unified vision foundation model introduced by researchers at Apple and the University of Illinois Urbana-Champaign in October 2023. It merges two popular vision foundation models — Meta's Segment Anything Model (SAM) and OpenAI's CLIP — into a single shared Vision Transformer backbone through a combination of multi-task learning, continual learning, and teacher-student distillation. The method requires only a small fraction of the original pretraining datasets and demonstrates that complementary capabilities from distinct foundation models can be consolidated without retraining from scratch, reducing the storage and compute cost of running both models in inference.The resulting model retains SAM's zero-shot segmentation ability and CLIP's zero-shot classification and image-text retrieval, while introducing new capabilities the individual models lacked. SAM-CLIP establishes state-of-the-art results on zero-shot semantic segmentation across five benchmarks, improving mean IoU by 6.8 points on Pascal VOC and 5.9 points on COCO-Stuff over prior specialized models. The paper was accepted at the UniReps Workshop at NeurIPS 2023 and the eLVM Workshop at CVPR 2024. Apple has published the research but has not released model weights or inference code publicly.