SAM-CLIP vs YOLOv9

SAM-CLIP is a unified vision foundation model introduced by researchers at Apple and the University of Illinois Urbana-Champaign in October 2023. It merges two popular vision foundation models — Meta's Segment Anything Model (SAM) and OpenAI's CLIP — into a single shared Vision Transformer backbone through a combination of multi-task learning, continual learning, and teacher-student distillation. The method requires only a small fraction of the original pretraining datasets and demonstrates that complementary capabilities from distinct foundation models can be consolidated without retraining from scratch, reducing the storage and compute cost of running both models in inference.

The resulting model retains SAM's zero-shot segmentation ability and CLIP's zero-shot classification and image-text retrieval, while introducing new capabilities the individual models lacked. SAM-CLIP establishes state-of-the-art results on zero-shot semantic segmentation across five benchmarks, improving mean IoU by 6.8 points on Pascal VOC and 5.9 points on COCO-Stuff over prior specialized models. The paper was accepted at the UniReps Workshop at NeurIPS 2023 and the eLVM Workshop at CVPR 2024. Apple has published the research but has not released model weights or inference code publicly.

YOLOv9 is a real-time object detection model developed by Chien-Yao Wang and Hong-Yuan Mark Liao at Academia Sinica, released in February 2024 under the GPL-3.0 license. It introduces Programmable Gradient Information (PGI), a mechanism that preserves complete input information through auxiliary reversible branches during training to address information loss in deep network layers. It also introduces the Generalized Efficient Layer Aggregation Network (GELAN), which achieves better parameter utilization compared to prior CSP-based designs.

YOLOv9-C achieves 53.0% AP on COCO with 42% fewer parameters and 21% less computation than YOLOv8-C at comparable accuracy. YOLOv9-E achieves 55.6% AP. The model is deployable through Roboflow Inference and supports fine-tuning via the standard training pipeline in the official repository.