Gemini 4 Carbon vs. Fable 5.2, Opus 5.5 and GPT-6.1 Sol on the same coding tasks
Updated November 10, 2026 · first published November 10, 2026
Claude Opus 5.5 and GPT-6.1 Sol each passed all six coding tasks. Opus had the lower cost per pass at $0.0267 versus Sol’s $0.0383. Fable 5.2 passed five. Gemini 4 Carbon passed three, and its other three attempts hit the 1,100-token completion cap on the Flex route.
Cost per pass includes spend on failed attempts. Carbon uses reported Flex cost. Claude and Sol use API-equivalent costs calculated from recorded CLI token usage.
Which models passed each coding task?
Each cell shows the outcome, elapsed time and cost for one attempt. A pass required every hidden check to match the expected output with input preserved. Capped means the endpoint hit its completion limit without producing a passing submission.
| Task | Carbon | Fable 5.2 | Opus 5.5 | GPT-6.1 Sol |
|---|---|---|---|---|
| Retry scheduler | Pass 6.5 s $0.0021 | Pass 15.3 s $0.0870 | Pass 16.2 s $0.0429 | Pass 12.1 s $0.0436 |
| Temporal access rules | Capped 18.0 s $0.0057 | Pass 8.7 s $0.0390 | Pass 7.8 s $0.0161 | Pass 10.0 s $0.0430 |
| Deployment overlays | Capped 19.6 s $0.0057 | Pass 9.0 s $0.0376 | Pass 9.1 s $0.0190 | Pass 10.6 s $0.0434 |
| Incident grouping | Capped 15.4 s $0.0058 | Pass 8.9 s $0.0383 | Pass 11.0 s $0.0237 | Pass 11.8 s $0.0442 |
| Order dependent migrations | Pass 7.1 s $0.0017 | Pass 8.0 s $0.0319 | Pass 13.7 s $0.0333 | Pass 15.2 s $0.0441 |
| Apply nested configuration overrides | Pass 14.0 s $0.0033 | Fail 9.7 s $0.0434 | Pass 11.6 s $0.0250 | Pass 12.3 s $0.0115 |
Carbon reached its cap on temporal access rules, deployment overlays and incident grouping. Two of those runs returned no code. The third stopped mid-function. On nested config overrides, Fable’s forbidden-key lookup missed __proto__ and returned a key the spec required it to ignore. The other three models passed that task.
What did a successful task cost?
| Model and route | Passed | Median time, all runs | Median time, passes | Total cost | Cost per pass |
|---|---|---|---|---|---|
| Gemini 4 Carbon OpenRouter Flex | 3/6 | 14.7 s | 7.1 s | $0.0244 | $0.0081 |
| Claude Fable 5.2 Claude Code | 5/6 | 9.0 s | 8.9 s | $0.2772 | $0.0554 |
| Claude Opus 5.5 Claude Code | 6/6 | 11.3 s | 11.3 s | $0.1600 | $0.0267 |
| GPT-6.1 Sol Codex CLI | 6/6 | 12.0 s | 12.0 s | $0.2297 | $0.0383 |
Cost per pass is total inference cost across all six attempts divided by the number of accepted submissions. The per-task table shows the cost of the individual attempt. The totals table reports both median wall time across all attempts and median wall time for passes, because quick failed attempts can change the first number.
Opus and Sol passed the same set of six tasks, so their medians cover the same workload. Opus finished in a median 11.3 seconds versus 12.0 for Sol and cost about 30% less per pass at the stated rates. Fable’s five passes took a median 8.9 seconds, but the missed config task raised its cost per pass to $0.0554. Carbon’s median was 7.1 seconds among its three passes and 14.7 seconds across all six attempts.
Sol’s config attempt cost $0.0115, down from $0.0430–$0.0442 on the other five tasks. That attempt logged 24,192 cached input tokens, versus 7,168 on each of the others. Caching is part of the observed run cost and can change the result on a fresh session.
How do these models compare on full web apps?
Vals AI’s Vibe Code Bench v1.1 asked all four models to build web apps from scratch through the same OpenHands agent harness. Its November 9, 2026 snapshot gives Fable 5.2 the highest score, Carbon the shortest duration per test among these four, and Sol the lowest cost per test. Vals AI leaderboard and methodology.
| Model | Published score | Cost per test | Duration per test |
|---|---|---|---|
| Claude Fable 5.2 | 93.77% | $22.13 | 40m 37s |
| Claude Opus 5.5 | 90.29% | $35.77 | 1h 36m |
| Gemini 4 Carbon | 89.06% | $8.25 | 36m 12s |
| GPT-6.1 Sol | 88.93% | $6.23 | 1h 15m |
These are Vals’ aggregate test scores and per-test figures. An app can pass some checks and miss others, so the score cannot be used as an app acceptance rate to calculate cost per successful app. Our six-task run required every check to pass for each submission. Sol cost less per test than Opus in Vals’ app builds, while Opus cost less per pass in our one-function runs.
CursorBench 4.0 gives another view on ambiguous, multi-file coding work. It reports 60.1% at $8.49 per task for Carbon Medium and 60.0% at $32.79 for Fable 5.2 Max. Those rows show what the two configurations achieved on Cursor’s tasks. Carbon’s three capped attempts here describe its behavior under our 1,100-token limit. CursorBench leaderboard.
How were the models and submissions tested?
We ran 24 first attempts on November 10, 2026. Every model received the same user prompt for each of six original JavaScript tasks, each in a fresh context with medium reasoning requested. They returned one synchronous function. The benchmark gave them no hidden tests or grader feedback, and we made no repair attempts. None of the successful submissions used an external tool. The grader ran 42 hidden input cases across the six tasks and compared outputs against fixed expected values.
Carbon ran via OpenRouter on the Google AI Studio Flex endpoint with a 1,100 completion-token limit. Thinking counts toward that limit. Claude Fable 5.2 and Opus 5.5 ran with tools disabled in Claude Code 2.1.318. GPT-6.1 Sol ran in Codex CLI 0.166.0 and used no tools. The CLIs supplied their own system context and output limits. Elapsed time is from request or CLI process start to receipt of the complete response. It includes route, network and CLI startup time. The local grading time is excluded.
For Carbon, we use OpenRouter’s reported inference cost. For Claude and Sol, we apply published Standard API rates to CLI-reported token categories, including cached input, cache writes when present, and output including thinking. The CLI values estimate API-equivalent cost for the observed sessions. They are separate from subscription charges. The sample measures these setups on these six functions. It has no repeated trials and does not measure repository navigation or tool-driven repair. OpenRouter metering · Gemini Flex prices · Claude Standard prices · Sol Standard prices.
Retry scheduler prompt
Write JavaScript source defining a synchronous function named solve. The task statement describes what solve returns when invoked. Your reply must contain only the complete source, with no explanation, markdown, imports or external dependencies. Do not invoke solve yourself.
Implement solve(jobs, capacity, window, maxAttempts). Each job has {id, ready, outcomes}, with unique id and nonnegative integer ready time. outcomes is an array of booleans (true = success); attempts after its end fail. Each attempt is instantaneous. Globally at most capacity attempts may start within (t-window,t], counting attempts already started at the same timestamp. Process the waiting job with earliest ready time, breaking ties by original input index; when capacity delays an attempt, preserve this ordering by scheduled ready time, even when several jobs are already overdue. On failed attempt number k (starting at 1), schedule its retry at actual start time + 2**(k-1) if k < maxAttempts. A successful or exhausted job leaves the queue. Return the attempt log as [[id,time,success], ...] in execution order. capacity, window and maxAttempts are positive integers; at most 50 jobs and 10 attempts each. Example: solve([{id:"x",ready:0,outcomes:[false,true]}],1,3,3) returns [["x",0,false],["x",3,true]].
Temporal access rules prompt
Write JavaScript source defining a synchronous function named solve. The task statement describes what solve returns when invoked. Your reply must contain only the complete source, with no explanation, markdown, imports or external dependencies. Do not invoke solve yourself.
Implement solve(rules, queries) for a permissions audit. A rule is {prefix,start,end,action,priority}, where prefix is an array of path segments, start/end are integer timestamps or end=null for infinity, action is "allow" or "deny". Query is {path,at} with path also an array. A rule applies when start <= at < end (or open end) and prefix matches whole segments from the beginning of path. Pick longest prefix, then highest priority, then deny over allow, then earlier original rule index. Return one [action, ruleIndex] per query in input order, or ["deny",-1] if nothing applies. Arrays can be empty; path segments are case-sensitive strings and priorities may be negative. At most 500 rules and 500 queries.
Deployment overlays prompt
Write JavaScript source defining a synchronous function named solve. The task statement describes what solve returns when invoked. Your reply must contain only the complete source, with no explanation, markdown, imports or external dependencies. Do not invoke solve yourself.
Implement solve(releases,start,end). Each release {from,to,version,weight} covers [from,to), where to=null means infinity. For each point in the integer-bounded report range [start,end), the visible version is the active release with highest weight, breaking ties by later input index. When none is active use null. Return maximal adjacent segments as [start,end,version] triples, ordered by start, clipping to the report. Merge neighbors when their versions are strictly equal even if the winning release differs. All coordinates and weights are safe integers, from < to if finite, versions are strings. start <= end. Inputs may be empty; coordinate span can exceed 10**12 so do not iterate one tick at a time. Up to 150 releases.
Incident grouping prompt
Write JavaScript source defining a synchronous function named solve. The task statement describes what solve returns when invoked. Your reply must contain only the complete source, with no explanation, markdown, imports or external dependencies. Do not invoke solve yourself.
Implement solve(events,gap) for a monitoring pipeline. Event = {id,service,at,kind}. First deduplicate id across the whole input, keeping its last occurrence (even if service or at differs). Then sort surviving events by service ascending (JS ordinary string comparison), at ascending, and original input index ascending. An incident starts at the first event for a service or when the difference from the previous event for that service is strictly greater than gap. For each incident return {service,start,end,ids,counts} with event ids in this sorted order, counts as an array of [kind,count] pairs sorted by kind ascending (ordinary JS string comparison), start and end equal to its first and last event timestamp. Return incidents ordered by service then start. All times are safe integers within +/-10**12, gap is nonnegative. Kinds and ids may be arbitrary strings, including "__proto__". Up to 500 events.
Order dependent migrations prompt
Write JavaScript source defining a synchronous function named solve. The task statement describes what solve returns when invoked. Your reply must contain only the complete source, with no explanation, markdown, imports or external dependencies. Do not invoke solve yourself.
Implement a synchronous pure JavaScript function named solve. It receives one argument: an array of migration objects {id, dependsOn}, where id is a unique string and dependsOn is an array of migration IDs (it may be omitted, meaning []). Return {order, unresolved}. Build a deterministic topological order by repeatedly choosing the lexicographically smallest ID among migrations whose dependencies are all already ordered. If no remaining migration is ready, stop. unresolved must contain every migration not placed in order, sorted lexicographically. Thus a migration with a missing dependency, and every migration downstream of it, remains unresolved. Do not mutate the input. Return only the object, with arrays of IDs.
Apply nested configuration overrides prompt
Write JavaScript source defining a synchronous function named solve. The task statement describes what solve returns when invoked. Your reply must contain only the complete source, with no explanation, markdown, imports or external dependencies. Do not invoke solve yourself.
Implement a synchronous pure JavaScript function named solve. It receives (base, overrides), where base is a JSON-like object and overrides is an array of {path, value}; path is a nonempty dot-separated sequence of object keys. Return a new configuration with overrides applied in array order. Create missing intermediate objects. If an intermediate value is null, a primitive, or an array, replace it with an object before continuing. Replace the leaf value entirely, including when it is null, false, 0, an empty string, or an array. Do not mutate base or override values. Ignore any override whose path contains the exact key __proto__, prototype, or constructor. Return only the resulting object.
Frequently asked questions
Which models passed the same coding tasks?
Gemini 4 Carbon passed 3 of 6, Claude Fable 5.2 passed 5 of 6, Claude Opus 5.5 passed 6 of 6, GPT-6.1 Sol passed 6 of 6. These are first attempts on six original JavaScript tasks.
Why did Carbon hit the completion limit?
The Flex request allowed 1,100 completion tokens, including reasoning. 3 of Carbon’s attempts hit that limit without producing a passing answer. The Claude Code and Codex runs used their CLI output limits.
How did you calculate cost per successful task?
Divide total model cost for all attempts by the number of passes. Carbon uses the gateway’s reported inference cost. Claude and Sol use recorded CLI token usage priced at their published Standard API rates.
Related
Want this applied to your stack? Bring the provider bills, gateway logs, and top workflows; we will map the cost drivers and savings path. Book a free audit →