Analysis of the SWE-rebench v2 Leaderboard

This report examines the performance of various Large Language Models (LLMs) and autonomous agents tasked with solving real-world software engineering (SWE) challenges across multiple programming languages, including Python, Java, Go, Rust, and TypeScript.

📊 Benchmark Overview

The SWE-rebench v2 evaluation provides a rigorous look at how AI handles complex codebase modifications. The current data slice is defined by the following parameters:

Temporal Window: 15/05/2026 $\rightarrow$ 01/07/2026
Scope: 111 distinct problems sourced from 65 different repositories.
Objective: Measure the ability of models to resolve issues autonomously.

Evaluation Workflow

🏆 Top Performing Models and Agents

The following table details the high-tier performers. Note that the "Resolved Rate" is expressed as $\mu \pm \sigma$ (mean $\pm$ standard deviation).

Rank	Model/Agent	Type	Resolved Rate (%)	Pass@5 (%)	Cost/Prob ($)	Tokens/Prob	Cache %
1	`Anthropic Fable 5`	Model [high]	$64.5\% \pm 1.41\%$	78.4%	$4.40	2,518,308	94.9%
2	`Grok 4.5`	Model [high]	$63.8\% \pm 0.60\%$	77.5%	$1.47	2,429,424	92.6%
3	`Anthropic Opus 5`	Model [high]	$63.4\% \pm 1.35\%$	74.8%	$3.47	4,322,143	95.7%
4	`Z.ai GLM-5.2`	Model [high]	$62.9\% \pm 1.19\%$	81.1%	$1.40	5,524,892	92.1%
5	`OpenAI GPT-5.6 Sol`	Model [med]	$62.3\% \pm 1.83\%$	79.3%	$0.85	605,340	84.7%
6	`Junie`	Agent	$61.8\% \pm 0.54\%$	73.9%	$0.81	1,684,501	85.9%
7	`Claude Code`	Agent	$60.4\% \pm 1.03\%$	75.7%	$3.39	3,341,581	93.4%
8	`OpenAI Codex`	Agent	$58.0\% \pm 1.29\%$	73.0%	$1.59	2,070,976	94.6%
9	`Anthropic Sonnet 5`	Model [high]	$56.8\% \pm 0.94\%$	74.8%	$1.43	4,645,617	96.4%
10	`Cursor`	Agent	$51.7\% \pm 0.84\%$	65.8%	$0.41	1,827,002	94.3%
11	`MiniMax M3`	Model	$47.2\% \pm 1.13\%$	69.4%	$0.95	13,869,459	97.0%
12	`MiMo V2.5 Pro`	Model	$46.5\% \pm 0.54\%$	65.8%	$0.10	4,687,987	95.7%
13	`OpenAI GPT-5.6 Luna`	Model [med]	$43.6\% \pm 1.47\%$	59.5%	$0.11	395,522	85.2%
14	`DeepSeek-V4 Pro`	Model [high]	$40.2\% \pm 1.29\%$	64.0%	$0.15	3,955,414	91.2%
15	`Qwen3.6-27B`	Model	$31.2\% \pm 1.68\%$	57.7%	$0.62	3,078,970	76.8%
16	`Qwen3.6-35B-A3B`	Model	$24.7\% \pm 0.79\%$	43.2%	$0.27	3,726,061	87.5%
17	`Qwen3.5-35B-A3B`	Model	$17.1\% \pm 1.07\%$	36.9%	$0.99	5,939,227	72.3%

🔍 Key Insights & Nuances

Efficiency vs. Performance

There is a stark contrast between raw power and cost-efficiency. For instance, while Anthropic Fable 5 leads in resolution, it costs $\approx \$ 4.40 $per problem. Conversely, `MiMo V2.5 Pro` offers a resolution rate of$ 46.5% $at a fraction of the cost ($ $0.10$).

Agentic Behavior

The leaderboard includes both standalone models and Agents. Agents typically wrap models in a loop of thought $\rightarrow$ action $\rightarrow$ observation.

Top Agent: Junie (61.8% Resolved)
Most Affordable Agent: Cursor ($0.41/problem)

Token Consumption

The token usage varies wildly. MiniMax M3 consumes a massive $\approx 13.8$ million tokens per problem, whereas GPT-5.6 Luna is highly concise, using only $\approx 395$ thousand.

🛠 Technical Implementation Note

To replicate these results, the system likely employs a loop similar to this pseudo-code:

def evaluate_swe_task(model, problem):
    context = load_repository(problem.repo)
    attempt_count = 0
    while attempt_count < 5:
        solution = model.generate_fix(context, problem.issue)
        if verify_solution(solution):
            return "RESOLVED"
        attempt_count += 1
    return "FAILED"

⏳ Pending Evaluations

A significant number of models have been registered but currently show N/A for their metrics. These include:

Anthropic Suite: ~~Claude Opus 4.1~~, Opus 4.5, 4.6-high, 4.7-high, 4.8-xhigh, Sonnet 3.5, 4, 4.5, 4.6.
DeepSeek Suite: R1-0528, V3, V3-0324, V3.1, V3.2, V4 Flash.
Mistral Suite: Devstral-2-123B, Devstral-Small-2-24B, Devstral-Small-2505.
Gemini Suite: 3 Flash/Pro, 3.1 Pro, 3.5 Flash, 2.0 Flash, 2.5 Flash/Pro, Gemma 4 31B, Gemma-3-27b.
Z.ai Suite: GLM-4.5 (Air), 4.6, 4.7 (Flash), 5, 5.1.
OpenAI Suite: GPT-4.1 (mini/nano), GPT-5 (high/med/min), GPT-5 Codex, GPT-5.1, GPT-5.2, GPT-5.3, GPT-5.4, GPT-5.5, GPT-OSS (120b/20b).
Others: Grok 4, Grok Code Fast 1, OpenRouter Horizon (Alpha/Beta), Kimi K2 (Instruct/Thinking/2.5/2.6), Llama-3.3-70B, Llama-4 (Maverick/Scout), MiniMax M2 series.