Hugging Face: YOLOS

YOLOS uses a Vision Transformer (ViT) style backbone instead of a convolutional neural network. Images are split into patches, which are treated as a sequence of tokens, similar to text tokens in NLP transformers.

The model consists of:

• A transformer encoder that processes a sequence of image patch embeddings.
• A detector head that maps the resulting sequence of detection representations to class labels and bounding box predictions.

YOLOS is considered a YOLO model because it processes the sequence of image patches once to produce detections, in line with the “You Only Look Once” paradigm.

Before YOLOS, YOLO models used convolutional neural networks as their backbone to extract image features. Different YOLO variants changed the exact CNN design, but all built the detection neck on top of a CNN backbone.

YOLOS differs from these models in several ways:

• The backbone is a transformer encoder instead of a CNN.
• Image patches are treated as a sequence, similar to tokens in NLP models.
• Aside from the single-pass detection concept, the YOLOS network architecture does not share other structural components with previous YOLO models.

Transformers were first introduced for NLP in the “Attention is All You Need” paper and became a standard approach for sequence modeling. They are widely used to process sequences of text.

Later, transformers were adapted to computer vision through the Vision Transformer (ViT), which set new standards in image classification at the time. ViT treats an image as a sequence of patches, analogous to sequences of text tokens in models such as GPT and BERT.

YOLOS extends this idea from classification to object detection by using a ViT-style encoder as the backbone and adding a detection head.

YOLOS was not designed to be state of the art in accuracy. The original research positioned it as an exploration of transformers for object detection rather than a benchmark leader.

On the COCO dataset:

• The YOLOS-B variant, reported as the best performing YOLOS model in the referenced summary, achieved an Average Precision (AP) of 42.0.
• This result used 1000 pre-training epochs followed by 150 fine-tuning epochs.

Compared to other YOLO models on COCO:

• The most performant YOLOS variant achieved 42.0 AP.
• YOLOv7 E6E, which was a state-of-the-art model in November 2022, achieved 56.8 AP.
• YOLOS does not reach the AP of YOLOv4 and more recent YOLO variants when benchmarked on COCO.

These comparisons indicate that YOLOS is primarily a research model for transformer-based detection rather than a leading choice for maximum accuracy.

YOLOS can be trained on custom datasets using the approach described in its original resources and repository. The training process follows the transformer-based design, with long pre-training and fine-tuning schedules reported for the YOLOS-B result on COCO.

For users whose main goal is the highest available performance on a custom dataset, other YOLO variants such as YOLOv5 are typically recommended over YOLOS. YOLOS is more suitable when the objective is to study or apply transformer-based architectures for object detection.

YOLOS is most appropriate in the following scenarios:

• Research on transformer architectures for vision and detection.
• Experiments that compare CNN backbones and transformer backbones within the YOLO paradigm.
• Educational use to understand how ViT-style models can be adapted from classification to detection.

For production deployments that prioritize accuracy or mature ecosystem support, other YOLO models are generally preferred over YOLOS.