GPT 5.6 Sol: OpenAI's New Vision Powerhouse

Recently, OpenAI unveiled the GPT-5.6 family, consisting of three distinct models: Sol, Terra, and Luna. While the official launch stream emphasized computer use—specifically the ability to navigate desktop environments, manage UI agents, and generate 3D visualizations—these features are fundamentally built upon a foundation of superior visual comprehension.

To verify these claims, we put the new lineup through our upcoming VLM benchmark.

Benchmark Objectives

• Object Detection
• Object Counting
• Optical Character Recognition (OCR)
• Targeted Data Extraction

The Bottom Line: GPT-5.6 Sol is the most capable vision model OpenAI has ever produced. The improvement is most staggering in counting and detection, areas where GPT-5.5 previously struggled compared to top-tier VLMs.

---

🎯 Object Detection

The most dramatic leap in performance occurred in detection. We measured this using $\text{mAP@50}$ (mean Average Precision).

Key Findings in Detection

• Document Intelligence: Sol excels at identifying layout elements like signatures, tables, paragraphs, and titles.
• Dense Environments: The model was tested on images of eggs and pills (see below), which typically trip up VLMs due to the high density of similar objects.
• The Coordinate Challenge: Unlike standard detectors, VLMs output coordinates as text. As the number of objects increases, the risk of hallucinations or coordinate drift rises.

🛠️ Implementation Tip: Coordinate Formats

To maximize accuracy, you must use the correct coordinate system.

• GPT-5.6: Use absolute XYXY coordinates in pixels.
• Gemini 3.5 Flash: Use normalized YXYX coordinates (range $0 \dots 1000$).

Using the wrong format for GPT-5.6 resulted in a performance drop of $\approx 15$ mAP points.

# Example of the preferred GPT-5.6 coordinate output
# [xmin, ymin, xmax, ymax]
detection = [120, 450, 300, 600] 

Stability Issues

We noticed that Sol can become unstable with images $\ge 2000 \times 2000$ pixels. In these cases, it may generate "ghost boxes" in unnatural, evenly spaced grids.

• Fix A: Increase "reasoning effort" (increases cost/latency).
• Fix B: Resize or crop images before API submission (recommended).

---

🔢 Object Counting

Counting accuracy saw a universal lift across the 5.6 family.

• Sol: 73.0% (up from 64.9% in GPT-5.5)
• Terra: 67.6%
• Luna: 66.2%

Complex Scenarios Sol Handled:

1. Overlapping Objects: Successfully counted metal brackets that were partially obscured.
2. Conditional Counting: Counted bullet holes only within specific scoring zones.
3. State Differentiation: Distinguished between filled and empty slots in a pill package despite reflections.

Note: Sol did fail on an "abnormal candy" test, though it's unclear if this was a counting error or a category misunderstanding.

---

✍️ OCR and Data Extraction

Interestingly, OCR performance remained relatively flat compared to the previous generation.

Performance Nuances

We distinguished between Full Transcription (all text) and Targeted Extraction (specific data).

• Wins: Sol successfully read handwritten notes, extracted live hockey scores, and read tire size sequences from curved, dirty surfaces.
• Losses: It failed to read a blister pack expiration date due to low contrast, vertical orientation, and glare.

---

⚖️ Trade-offs: Cost, Speed, and Tokens

The increase in visual intelligence is not free. The GPT-5.6 models consume more tokens, which impacts the bottom line for high-volume users.

Latency Comparison:

• Luna: $\approx 5$ seconds (Best balance of speed/quality; similar to Gemini 3.5 Flash).
• Terra: $\approx 6$ seconds.
• Sol: Highest latency and cost.

Financials: In our tests, Sol cost approximately $0.025 per image.

You can experiment with these models via the Roboflow Playground to compare Sol, Terra, and Luna against competitors like Claude Fable 5.