PaliGemma vs SigLIP

PaliGemma is a vision-language model released in May 2024 by Google, built by pairing the SigLIP-So400m vision encoder with the Gemma 2B language model. It is designed primarily as a compact, transfer-friendly base model for fine-tuning to downstream vision-language tasks, rather than as a chat-optimized assistant. PaliGemma draws architectural inspiration from the PaLI-3 model at Google Research, applying a similar encoder-decoder approach at a smaller and more accessible parameter scale.

PaliGemma accepts an image together with a text prompt and generates text output, supporting image captioning, visual question answering, optical character recognition, object detection, referring expression segmentation, and a range of related vision-language tasks when fine-tuned on task-specific data. The model is released at three input resolutions (224, 448, and 896 pixels), with higher resolutions providing stronger performance on tasks requiring fine visual detail such as OCR and document understanding. Google released pretrained (PT) checkpoints intended as fine-tuning bases, along with Mix variants that have been fine-tuned on a mixture of downstream tasks for direct use without additional training. PaliGemma is distributed under the Gemma license, a custom license from Google that permits commercial use subject to the terms of the Gemma Prohibited Use Policy. It was succeeded by PaliGemma 2 in December 2024, which extends the architecture to larger Gemma 2 language backbones at 3B, 10B, and 28B parameter sizes.

SigLIP is a vision-language model released in March 2023 by researchers at Google DeepMind. It adapts the CLIP image-text pretraining approach by replacing CLIP's softmax-based contrastive loss with a pairwise sigmoid loss, which operates independently on each image-text pair rather than requiring a global view of all pairs in a batch. This change decouples the loss from batch size, enabling more memory-efficient training and improved performance at smaller batch sizes, a regime where softmax contrastive learning typically struggles. Despite this simplification, SigLIP matches or exceeds CLIP-style models on zero-shot image classification and image-text retrieval benchmarks when trained on comparable data.

SigLIP is distributed as an image encoder plus aligned text encoder, supporting zero-shot classification with arbitrary class vocabularies, image-text retrieval, and use as a frozen backbone in downstream vision-language models. Pretrained models are available at multiple Vision Transformer sizes and input resolutions, including 224, 256, 384, and 512 pixel inputs. SigLIP is released under the Apache 2.0 license by Google and is used as the vision encoder in Google's PaliGemma and PaliGemma 2. A successor, SigLIP 2, was released in February 2025 with multilingual support across 109 languages, improvements to localization and dense prediction, and two resolution handling variants (FixRes for backward-compatible fixed resolutions and NaFlex for native aspect ratio with variable sequence length).