Choosing an AI Model: One Prompt, 11 Models, Diverse Outcomes

More models, more choice: A comparative look at 11 different AI models, including DeepSeek, Qwen, Kimi, and hundreds of other open-source options now available.

---

Date: August 12, 2026
Author: Elad Rosenheim, Principal Product Manager

🚀 The Big Announcement

We have officially entered into a partnership with OpenRouter, which unlocks two major enhancements for our users:

1. The AI Gateway: Your projects can now integrate any model hosted on OpenRouter. This is a game-changer for developers whose web apps provide AI-driven features to end-users, offering a massive library of models to balance specific task requirements against budget constraints.
2. Expanded Agent Runners: We are broadening the array of frontier coding models available via our Agent Runners—the integrated chat interface within Netlify used for building new projects or refining existing ones.

New Model Availability

The lineup now includes highly anticipated open models such as:

• Kimi K3
• GLM 5.2
• DeepSeek V4

Note: We call these "Agent Runners" because they utilize a full coding agent rather than a stripped-down version.

Previously, we focused on Claude Agent, OpenAI Codex, and Gemini CLI. To better support the influx of new models, we have integrated the open-source OpenCode as a new agent option. These agents are equipped with specialized skills and project context, ensuring they know exactly how to implement Netlify-specific tools like:

• Netlify Database
• AI Gateway
• Netlify Identity

🛠️ How We Evaluate: The AXIS Framework

With so many options, users often wonder: Which model is the most cost-effective? Which one actually delivers "mind-blowing" results?

To find out, we use AXIS, an internal tool for automated model evaluation that we have recently open-sourced. The evaluation process follows this logic:

The AXIS Checklist

AXIS focuses on functionality over aesthetics. The goal is to ensure the site actually works.

• Does the site use a database when the prompt requires data persistence?
• Is the site free of over-engineering? (e.g., $\text{No Database}$ if a static site suffices).
• Are Netlify-specific skills applied correctly?
• Is the credit consumption optimized?

If a model fails these checks or consumes excessive credits for simple tasks, it is either excluded or the underlying "skill" is optimized.

🧪 The Experimental Setup

We wanted to move beyond internal metrics and see what happens when you build real-world projects with different models. We selected three distinct use-cases:

---

A Note on GPT 5.6 Sol

We currently run GPT 5.6 Sol on a low effort setting by default. This provides a budget-friendly alternative to Opus while maintaining high quality. While we set the default, the effort setting is now fully under your control.

☕ Scenario #1: The Local Coffee Shop

In this first scenario, we test a basic request.

The Prompt:

"Build a one-page site for a neighbourhood coffee shop: opening hours, the address, a short menu and a photo."

We included the mention of a "photo" to signal to the model that a complex Content Management System (CMS) was unnecessary. Our internal skills guide the model to avoid "purple AI slop" and instead create a visual identity that fits the brand.

Credit Consumption Analysis

The cost of generation varies wildly. We can represent the average cost $\bar{C}$ as: $\bar{C} = \frac{\sum (\text{Run}_1 + \text{Run}_2 + \text{Run}_3)}{3}$

Visual Results

Below are the visual outputs from the different models:

Claude Opus 5 (Left: | Right: )

Claude Sonnet 5 (Left: | Middle: | Right: )

GPT 5.6 Sol (Low Effort) (Left: | Middle: | Right: )

This concludes the first scenario. Stay tuned for follow-up posts where we explore more complex use-cases!