A two-line Python function exposed a striking point about coding-model claims: all eight tested models spotted the same subtle production bug and proposed the same core repair. Yet that result says almost nothing about which model is “best at coding” overall.
Disclosure: I work with OrcaRouter, the router used for this evaluation. It is a leading router in RouterArena and gives one API key access to 200+ models.
AI-generated illustration featuring official model logos; logos and model names are used descriptively and remain the property of their respective owners.
Want to try GPT-5.6 yourself? Explore it on OrcaRouter.
The function was this:
`python
def add_task(task, queue=[]):
queue.append(task)
return queue
`
In the test, each model identified queue=[] as a mutable default argument: Python creates that list once at function definition, rather than making a fresh list for every call. As a result, a first call such as add_task(“a”) can affect a later add_task(“b”). GPT-5.6 Sol illustrated the outcome as [‘a’, ‘b’] where a caller might expect [‘b’], then replaced the default with None and created a list inside the function.
That is a useful diagnosis. It is also one custom bug-diagnosis prompt per model—not a software-engineering leaderboard, and not a SWE-Bench score.
Eight answers, one clear pattern
The selected run contained eight calls: one call for code-001 to each of eight models. All eight calls had an observed ok status, and none requested or effectively used search.
Every response made the central point: the default list is shared by calls that omit queue. Every response also supplied the standard sentinel pattern:
`python
def add_task(task, queue=None):
if queue is None:
queue = []
queue.append(task)
return queue
`
The answers did more than name the issue. GPT-5.6 Terra connected it to leakage across requests, users, jobs, or tests in production processes. GPT-5.6 Luna described the risk in a long-lived process such as a web worker, where earlier tasks can persist into later requests.
Claude Fable 5 went further into the mechanics, saying the reused list can be seen in add_task.__defaults__, and warning that if queue is None is more precise than if not queue because an explicitly supplied empty list should remain usable. Claude Opus 4.8 similarly stressed that the failure is history-dependent: a quick one-call test can appear fine, while repeated calls expose accumulated state.
Grok 4.5, Gemini 3.5 Flash, and GLM-5.2 gave the same diagnosis and correction, with examples of repeated defaulted calls carrying earlier values forward.
What was observed—and what was scored
The observed result is straightforward: on this particular prompt, all eight selected API calls returned a plausible explanation, a concrete failure example, and corrected code.
A separate judge-version audit assigned an accuracy value of 5 to each selected successful answer under the v3 judge. But those are reference-guided DeepSeek v3 judgments, not human code review; human code review is pending. The scores are therefore an audit signal about how the answers matched the evaluation reference, not final ground truth about code quality.
That distinction matters even for a familiar bug. A model can describe a correct fix while adding irrelevant claims, choosing an unsuitable design for a larger codebase, or missing constraints that were never included in this minimal prompt. Conversely, a strong real-world coding evaluation would need repositories, tests, debugging context, implementation changes, and repeated samples—not just one short function.
Why this cannot settle cross-model coding claims
The test was deliberately narrow: one custom Python bug-diagnosis prompt per model, with one selected call per question-model cell. A single question-model call is an observation, not proof of stable behavior.
It cannot support an overall ranking among GPT-5.6 Sol, GPT-5.6 Terra, GPT-5.6 Luna, Claude Opus 4.8, Claude Fable 5, Grok 4.5, Gemini 3.5 Flash, and GLM-5.2. It also cannot validate a claim that one provider leads another on SWE-Bench or any other coding benchmark.
This is the central gap behind many broad coding comparisons: a vendor’s benchmark announcement may be useful evidence about that vendor’s reported evaluation, but it is not automatically a verified common comparison across providers. OpenAI’s GPT-5.6 announcement is an official vendor benchmark source, with an explicit caveat against using it for unverified cross-provider SWE-Bench claims. And Humanity’s Last Exam is a closed-ended academic benchmark, not a software-engineering benchmark. Neither source fills the missing common citation for a cross-provider coding leaderboard.
The absence of a verified common citation does not mean every comparative coding claim is false. It means this evidence pack does not establish one, so none should be repeated as fact here.
Reproducible data figure from this article’s selected API records; it is not a general ranking.
One-call API observations are not a speed chart
The run also recorded latency and some billing fields, but these are descriptive observations from one API or gateway call per model—not general measures of quality, value, or consumer-product experience. Gateway/API behavior should not be generalized to consumer subscription products.
| Model | Observed latency for this call | Observed billed USD |
|---|---|---|
| Gemini 3.5 Flash | 3.99 seconds | — |
| GPT-5.6 Sol | 8.07 seconds | 0.0093 |
| GPT-5.6 Terra | 8.32 seconds | 0.0046 |
| Claude Opus 4.8 | 10.82 seconds | 0.0183 |
| GPT-5.6 Luna | 13.03 seconds | 0.0029 |
| Grok 4.5 | 13.6 seconds | 0.0174 |
| Claude Fable 5 | 13.72 seconds | 0.04 |
| GLM-5.2 | 17.76 seconds | — |
These numbers are recorded for the selected calls only. They do not establish a general latency or cost winner. The prompt-token counts also differed substantially across the calls, from 48 observed prompt tokens for GPT-5.6 Sol and Terra to 5,040 for Grok 4.5, another reason not to compress the data into a simplistic ranking. Vendor effort labels, where they exist, should not be read as equivalent compute budgets across providers.
Practical takeaway
For a developer facing this exact Python pattern, the useful answer is consistent across the eight transcripts: use None as the default, create a fresh list inside the function, and preserve explicitly supplied lists. That is a concrete, testable repair.
For someone choosing a coding assistant, the more important lesson is methodological: ask what was tested, how many times, under what API settings, and whether the comparison uses a shared, verifiable benchmark. A clean answer to one familiar bug can demonstrate basic competence. It cannot, by itself, demonstrate broad software-engineering superiority.
Limitations
This case study covers one custom bug-diagnosis prompt, `code-001`, across eight selected API calls, with one call per model. No search was used in the selected calls. The evidence does not include repeated trials, repository-level tasks, test execution, patch validation, or human code review. Judge scores are reference-guided DeepSeek v3 judgments, and human code review remains pending. No unverified SWE-Bench comparison is treated here as fact.
Editorial illustration; it frames careful code diagnosis and is not test evidence.
Explore the Models
Explore the current catalog on OrcaRouter Models.
This evaluation was run through OrcaRouter. The author works with OrcaRouter; model access does not imply affiliation with, endorsement by, or sponsorship from model providers.
Model names and logos are used descriptively. All trademarks belong to their respective owners.
Sources
- GPT-5.6 announcement — OpenAI; retrieved 2026-07-17.
- Humanity’s Last Exam — Center for AI Safety and Scale AI collaborators; retrieved 2026-07-17.