RF-DETR Segmentation vs YOLOv8 Instance Segmentation

RF-DETR Segmentation is a real-time instance segmentation model developed by Roboflow, with a preview base model released in October 2025 under the Apache 2.0 license and the full variant family — Nano through 2XL — released in January 2026. It extends the RF-DETR object detection architecture with a segmentation head inspired by MaskDINO, enabling pixel-level object delineation while maintaining the real-time performance characteristics of the base model. It is deployable through Roboflow Inference and the open-source rfdetr Python package.

RF-DETR Segmentation supports fine-tuning on custom COCO- or YOLO-format instance segmentation datasets and is benchmarked on Microsoft COCO. It is suited for applications requiring both precise object masks and real-time inference, such as robotic manipulation, quality control, and augmented reality overlays.

YOLOv8 Instance Segmentation is the segmentation variant of the YOLOv8 model developed by Ultralytics, released in January 2023 under the AGPL-3.0 license. It extends the standard YOLOv8 detection head with a mask prediction branch that generates pixel-level segmentation masks for each detected object using a prototype mask approach. This enables real-time instance segmentation within a single forward pass.

YOLOv8 Instance Segmentation shares the same backbone and neck architecture as the base detection model and is available in the same size range. It is deployable through Roboflow Inference and supports fine-tuning on custom COCO-format segmentation datasets. It is suited for applications requiring both object localization and precise mask prediction at real-time speeds.