GPT-6 Astra Successfully Navigates Real-World Driving

DrivingBench is a rigorous evaluation framework developed by Aditya Ramabadran, Simon Mahns, and Tobias Gessler to answer a pivotal question regarding the current state of artificial intelligence:

"Can frontier models drive a real car?"

To test this, the researchers granted several high-end AI models direct control over a Toyota Corolla, specifically managing the steering, accelerator, and braking systems. The models were tasked with navigating a predefined course marked by cones.

🛠️ Experimental Methodology

The evaluation allows each model up to three attempts within a single, continuous chat session, enabling the model to reflect on previous failures.

Evaluation Criteria

The team tracked the following metrics to determine performance:

• Progress: The percentage of the centerline completed while remaining within a 4-meter threshold.
• Distance: The total length traveled.
• Finish Time: Duration from the first command to the final engagement.
• Commands: The number of set_motion and stop_now calls.
• Cost: Total token expenditure based on list prices.

The distance is calculated using the integral of GPS speed: $\text{Distance} = \int_{t_{start}}^{t_{end}} v_{gps}(t) \, dt$

📊 Leaderboard Results

The following table summarizes the performance of the tested models (all using "medium" settings):

---

🔍 Detailed Model Analysis

1. GPT-6 Astra Codex

The clear winner, this model demonstrated an ability to learn from its mistakes.

• Attempt 1: Reached 49% progress (67.3 m) before a DNF. Cost: $2.01 (1.2M tokens).
• Attempt 2: Achieved 100% completion in 5:22, covering 134.7 m. Cost: $7.74 (6.6M tokens).
• Total Commands: 32 calls to set_motion and stop_now.

2. Claude Fable 5.1 (Claude Code)

While it showed improvement across attempts, it failed to finish.

• Attempt 1: 9% (17.5 m) $\rightarrow$ DNF ($0.96)
• Attempt 2: 10% (27.3 m) $\rightarrow$ DNF ($1.35)
• Attempt 3: 45% (73.7 m) $\rightarrow$ DNF ($1.64)

3. Grok 4.6 Cursor

Grok struggled significantly with the course, showing very little variance between attempts.

• Attempt 1: 8% (14.4 m) $\rightarrow$ DNF ($0.18)
• Attempt 2: 11% (22.6 m) $\rightarrow$ DNF ($0.19)
• Attempt 3: 10% (22.2 m) $\rightarrow$ DNF ($0.29)

4. GPT-5.6 Sol Codex

The lowest performing model in this benchmark, failing to make meaningful progress.

• Attempt 1: 6% (15.0 m) $\rightarrow$ DNF ($0.35)
• Attempt 2: 6% (15.6 m) $\rightarrow$ DNF ($0.43)
• Attempt 3: 6% (17.1 m) $\rightarrow$ DNF ($0.27)

💻 Technical Implementation

The models interacted with the vehicle via a specific API. A typical command sequence might look like this:

# Example of a motion command sent by the model
vehicle.set_motion(
    steering_angle=15.5, 
    acceleration=0.2, 
    brake_pressure=0.0
)

# Emergency stop command
vehicle.stop_now()

The results indicate that while most frontier models struggle with the precision required for real-world driving, GPT-6 Astra Codex has successfully bridged the gap, completing the course on its second try.