Microsoft: Florence-2
Florence-2 Overview
Florence-2, introduced by Microsoft Research at CVPR 2024, is an open-source vision-language foundation model designed to unify diverse computer vision tasks within a single sequence-to-sequence framework. Unlike traditional models that specialize in specific tasks, Florence-2 accepts both images and text prompts and outputs text for tasks such as captioning, object detection, segmentation, OCR, and region-based grounding. It comes in two sizes—Florence-2-base (~230M parameters) and Florence-2-large (~770M parameters)—and is trained on FLD-5B, a large dataset of ~126M images with ~5.4B annotations.
The model demonstrates strong zero-shot and fine-tuned performance, often rivaling larger vision-language systems while remaining lightweight and efficient. Released under the MIT license, all weights are publicly available, making it accessible for fine-tuning and deployment in applications like VQA, content tagging, accessibility, and research. Florence-2’s compact design, versatility, and openness position it as a practical alternative to larger proprietary multimodal models.
Florence-2 Interactive Demo
Upload an image
Drag and drop an image here, or click to browse
Results appear here. Add an image or pick an example to run Florence-2.
Alternatives to Florence-2
Other models worth comparing for similar use cases.
Unlike its previous SAM models (Segment Anything and Segment Anything 2), you can provide SAM 3 with the prompt “shipping container” and it will generate precise segmentation masks for all shipping containers in an image. SAM 3 generates segmentation masks that correspond to the location of the objects found with a text prompt.
Deploy Florence-2 with an API
Florence-2 runs as a hosted REST endpoint through Roboflow Workflows. Pick a task, then hand the prompt to your coding agent or copy the code. Deploying the workflow into a free Roboflow workspace replaces the your-workspace and YOUR_API_KEY placeholders with your own.
Add the Roboflow MCP server
claude mcp add --transport http roboflow https://mcp.roboflow.com/mcp
Run /mcp and authorize Roboflow in your browser when the OAuth flow opens.
Start a new Claude Code session so the MCP loads, then paste the prompt below (it works the same in any agent).
Deploy this workflow to your Roboflow workspace to use it.
Integrate the Roboflow "Florence-2" workflow into my app.
- Endpoint: POST https://serverless.roboflow.com/<your-workspace>/workflows/florence-2-captioning
- Auth: send my Roboflow API key as `api_key` in the request body, read from the ROBOFLOW_API_KEY env var (never hardcode).
- Body: { "api_key": ..., "inputs": { `image`: { type: "url" | "base64", value } } }.
With the Roboflow MCP connected, call `workflows_get` on "florence-2-captioning" to read the exact input schema (the source of truth), then `workflows_run` on a sample image to confirm the output shape before writing code (the MCP is authenticated, so this needs no key). Without the MCP, use the contract above.
Before running the app, set up these keys so it does not error at runtime:
- `ROBOFLOW_API_KEY` (sent as `api_key`) from https://app.roboflow.com/settings/api
Create a .gitignore'd .env with these variables, using placeholder values for any I haven't given you. Then pause and tell me directly, in your reply: the full path to the .env file, exactly which keys I need to paste in, and the link to get each one. Wait for me to confirm I've added them before you run anything. Do not run the app until I confirm.
Then add the integration to my codebase: match my project's language, framework, and conventions; read every key from environment variables (never hardcode); add basic error handling; and include a small runnable example. If you can't tell what language my project uses, ask me.pip install inference-sdkDeploy this workflow to your Roboflow workspace to use it.
# 1. Import the library
from inference_sdk import InferenceHTTPClient
# 2. Connect to your workflow
client = InferenceHTTPClient(
api_url="https://serverless.roboflow.com",
api_key="YOUR_API_KEY"
)
# 3. Run your workflow on an image
result = client.run_workflow(
workspace_name="your-workspace",
workflow_id="florence-2-captioning",
images={
"image": "YOUR_IMAGE.jpg" # Path to your image file
},
use_cache=True # cache workflow definition for 15 minutes
)
# 4. Get your results
print(result)Deploy this workflow to your Roboflow workspace to use it.
const response = await fetch('https://serverless.roboflow.com/your-workspace/workflows/florence-2-captioning', {
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify({
api_key: 'YOUR_API_KEY',
inputs: {
"image": {"type": "url", "value": "IMAGE_URL"}
}
})
});
const result = await response.json();
console.log(result);Deploy this workflow to your Roboflow workspace to use it.
curl --location 'https://serverless.roboflow.com/your-workspace/workflows/florence-2-captioning' \
--header 'Content-Type: application/json' \
--data '{
"api_key": "YOUR_API_KEY",
"inputs": {
"image": {"type": "url", "value": "IMAGE_URL"}
}
}'Florence-2 License
MIT · Permissive licenseFlorence-2 is released under MIT, a permissive license. The Florence-2 license lets you use, modify, and sell work built on the model, with the copyright notice as the only real obligation and no requirement to open-source related code changes.
- Commercial use
- Permitted with no separate commercial license. No usage caps, revenue thresholds, or field-of-use limits apply to Florence-2.
- Modification
- Permitted. You can fine-tune or rewrite Florence-2 and keep the result closed-source.
- Redistribution
- Permitted. Include the original copyright and permission notice in copies or substantial portions of the work.
MIT grants no explicit patent license and disclaims all warranties. If patent exposure is a concern for your deployment, review it with counsel before launch.
Read the full MIT license ↗Do I need a commercial license for Florence-2?
No commercial license is needed for Florence-2: permissive terms let you keep related code private while deploying commercially.
Do not hesitate to reach out with questions for your commercial project — our team will help you start solving business problems on the first call. See Roboflow commercial licensing for the models included in each plan.
Talk to salesThis model is released under the MIT License, a short and permissive open-source license that allows commercial use, modification, and redistribution.
Yes. Under the terms of the MIT license, you can freely use this model for commercial purposes. You must retain the copyright notice and license text when redistributing.
License information is provided as a guide and is not legal advice.