Frontier coding models, measured over time

The top DeepSWE scores are close enough that cost changes the decision. GPT-6 Astra, Gemini 3.8 Flash, and Claude Opus 5 sit within half a percentage point in this snapshot. Their average task costs range from $2.36 to $11.84.

I’m keeping this page as a running comparison of frontier coding models. I want to see what a model solves, what the run costs, and whether the result holds up in the agent I would use.

Last source check September 22, 2026. The frozen Datacurve rows were checked September 11, 2026, and its leaderboard is dated September 3, 2026. This is a curated snapshot.

DeepSWE at a glance

DeepSWE v1.1 has 113 tasks across 91 repositories and five languages. Datacurve writes original engineering tasks with functional and regression tests. The leaderboard uses mini-swe-agent across models, which makes it useful for comparing models under a shared harness.

Pass@1 measures success on one attempt per task, averaged over the evaluation. Average cost per task includes unsuccessful attempts. A model that spends more time on a failing task can still leave you with a larger bill.

The comparison includes all 70 configurations across the 28 models in Datacurve’s current v1.1 dataset, plus the GPT-6 Sol, GPT-6 Luna, Muse Spark 1.3, Fable 5.1, and DeepSeek V4.1 Flash claims. The Benchmark selector offers DeepSWE v1.1, Arena Code WebDev’s Frontend category, the FrontendBench paper, the historical Terminal-Bench 2.1 snapshot, and the current Terminal-Bench 4.0 frontier. Versions and benchmarks stay separate. Muse has separate Standard and Contributor entries, each with its own model filter. Use the searchable model dropdown to choose a comparison, and the Options dropdown beside it for the scale, frontier, uncertainty, and supplemental-result toggles. Fable 5 and GPT-5.6 Luna are included at every published reasoning level. DeepSeek V4.1 Flash connects low, high, and max with the standard reasoning-series line. Its hollow markers identify the self-reported points. The GPT-6 models are max-effort points because OpenAI publishes exact DeepSWE scores only for those settings. Fable 5.1 appears in the score-only panel and configuration table because its DeepSWE cost is unknown.

The upper-right is the useful direction on this chart, as on Datacurve. Cost decreases toward the right. The default logarithmic cost scale keeps the roughly $0.01 to $26.40 range readable. Switch it off for a linear scale. The axes rescale to the visible models, so a narrow comparison uses the full plot area. Each solid curve connects the published reasoning levels of one model. Hover or select any point for that configuration’s score and cost; hovering dims the other models so the active curve is easier to trace. Small comparisons label every model. Larger comparisons label the frontier models, self-reported results, and whichever model you inspect.

A configuration is on the Pareto frontier when no other included configuration is at least as good on both cost and score and strictly better on one. Turn on “Show frontier” to see it as a dashed line. It is computed from all visible configurations with cost data and updates with the model and supplemental-result filters. Supplemental points join the displayed frontier only when their source reports a cost; hide them to see the official-only frontier. DeepSWE claims use editorial cost estimates, while Anthropic’s Terminal-Bench sweep uses Anthropic’s reported mean cost per task. A configuration below this line could still be useful for a different workload.

What I take from the current results

Gemini 3.8 Flash at high effort is the cheapest of the three leading configurations here. Astra at xhigh buys a small increase in the point estimate for a higher task cost. Opus 5 at max costs more than either and has a slightly lower score in this run.

GPT-5.6 Luna at max reaches 67.2% for $0.61 per task at Datacurve’s corrected prices. GLM-5.3 Flash reaches 63.4% for $0.24. Both extend the official frontier at lower budgets. Fable 5 peaks at 69.9% at xhigh for $13.41, so it sits below the cost/score frontier in this run.

The reported uncertainty ranges overlap. A 0.3-point gap between Astra and Gemini is too small to treat as an established quality advantage from these numbers alone. The chart uses the point estimates to calculate dominance. It does not test statistical significance.

Astra also uses far fewer output tokens and agent steps in this snapshot. Gemini’s lower bill comes with a longer trajectory. Steps are useful context, but wall-clock latency also depends on token throughput, tool execution, and provider delays. I would measure latency separately before choosing an interactive default.

Where GPT-6 Sol and Luna fit

OpenAI reports 68.8% for GPT-6 Sol at max effort and 66.6% for GPT-6 Luna at max effort on DeepSWE v1.1. OpenAI does not publish the exact harness, token usage, step count, uncertainty, or absolute task bill for either run, so both remain self-reported hollow markers rather than official Datacurve rows.

The cost coordinates are derived from OpenAI’s relative comparisons against rows already in this snapshot. Sol is described as approximately 80% cheaper per task than Fable 5 at xhigh; 20% of Fable’s displayed $13.41 bill is $2.68. Luna is described as 93% cheaper than Opus 5 at medium and 96% cheaper than Fable 5 at medium. Those comparisons imply roughly $0.23–$0.24, so the chart uses $0.24. These are rounded editorial estimates, not measured bills.

At those coordinates, Luna sits between DeepSeek V4.1 Flash low and Muse Spark 1.3 Contributor on the displayed all-results frontier. Sol does not extend this DeepSWE frontier: GPT-5.6 Sol at high is slightly better and cheaper in the frozen Datacurve snapshot, while the unusually aggressive Muse Contributor scenario also dominates the new point when all claims are enabled.

Where DeepSeek V4.1 Flash fits

DeepSeek’s V4.1 Flash model card reports a continuously controllable reasoning effort from 1 to 100. For consistency with the rest of this chart, I label the three supplied evaluation points at 25, 75, and 100 as low, high, and max. This is an editorial chart mapping, not DeepSeek’s API terminology: its technical report maps the public API presets low, high, and max to 50, 75, and 100. The report does not tabulate an exact DeepSWE coordinate for effort 50, so I have not inferred a separate medium point from the figure. The displayed sweep rises from 66.0% at low to 72.0% at high and 74.2% at max. These remain self-reported rather than official results, but they participate in the displayed frontier while self-reported results are enabled.

At max, the reported 74.2% reaches the same band as Claude Opus 5 and Gemini 3.8 Flash. DeepSeek’s own comparison reports Opus 5 at 74.0%; the Datacurve snapshot plotted here has Opus at 73.6% and Gemini at 73.8%. The estimated V4.1 Flash bill is about $0.48 per task, roughly 4% of Opus 5’s measured $11.84 bill in this snapshot.

The low setting is the more consequential result for routine work. Its reported 66.0% already exceeds DeepSeek V4 Pro at max in the Datacurve snapshot, 62.8%, while the estimated 38k output tokens are about 64% below Pro’s measured 106k. The estimate is about $0.19 per task. High provides a middle point at 72.0%, about 60k output tokens, and roughly $0.30.

DeepSeek’s launch documentation lists output at $0.60 per million tokens and uncached input at $0.155 per million, with lower cache-hit rates that vary by peak window. The chart’s cost, token, and step values are estimates based on the release report rather than measured Datacurve bills, so the points use hollow markers.

Where Fable 5.1 fits

Anthropic’s system card reports 67.4% on DeepSWE v1.1, averaged over five trials. Section 8.3 is on page 168. The standard configuration is adaptive thinking at max effort. The DeepSWE section does not specify the harness or provide cost, token usage, steps, or uncertainty.

Fable 5.1 is selectable alongside Fable 5. The chart keeps Anthropic’s result as a score-only entry because that run has no published cost. The result is self-reported and remains outside the official frontier.

Artificial Analysis provides a separate five-level effort sweep on its Intelligence Index v4.1.1. That index combines nine evaluations, including Terminal-Bench v2.1, SciCode, HLE, and GPQA Diamond. The scores and weighted task costs below belong to that suite.

AA’s September 1 writeup puts max at 66 and xhigh at 65, with xhigh costing $1.04 less per Index task in that launch snapshot. High scores 62, matching Fable 5 at max. AA evaluated Fable 5.1 with default server-side fallback, which routed some requests to Opus 4.8 or Opus 5 and accounted for roughly 4% of output tokens across the Index. Its five effort levels used 13.1M to 143.7M output tokens across the suite, an elevenfold range.

AA’s September 3 coding-agent comparison reports 70 for Fable 5.1 in Claude Code. Astra in Codex scores 67, approximately level with Opus 5, Fable 5, and Muse Spark 1.3. Those are composite Coding Agent Index scores from that dated comparison.

The live leaderboard has since moved to v1.5. On September 10, its independent DeepSWE v1.1 result for Fable 5.1 at max effort is 64.3% in Claude Code. AA also reports $12.39 per task for that configuration across the full three-benchmark Coding Agent Index. Its public per-benchmark data omits a DeepSWE-only cost. That suite-wide bill cannot supply the missing chart coordinate, and Claude Code is a different harness from Datacurve’s mini-swe-agent.

Where Muse Spark 1.3 fits

Meta reports 75.4% on DeepSWE v1.1. Its evaluation methodology specifies Muse Spark 1.3 at max effort with a mini-swe agent. The comparison with Muse Spark 1.2 also changes the effort setting from xhigh to max.

I have kept 1.3 as self-reported and pending independent review because it was absent from the Datacurve leaderboard at this source check. Both hollow diamonds are overlays of the same reported score. They receive no official rank, but their estimated costs participate in the displayed frontier while self-reported results are enabled.

Both pricing entries are visible by default. Standard (non-contributor) uses an estimated $2.75 per task, and Contributor uses $0.25. Each can be selected or hidden through the model filters. These estimates came from the initial comparison notes. They are not measured DeepSWE bills, and token prices alone cannot establish them.

Meta’s pricing page says contributor usage is used to improve its products. Standard usage is not. That makes the discounted tier a separate purchasing decision even for the same model.

Meta also reports roughly 25% fewer tokens and 20% fewer tool calls in comparisons by its engineers. Applying those reductions to the 1.2 leaderboard row gives roughly 74k output tokens and 80 steps. Those are projections from general engineering workflows, not measured 1.3 DeepSWE statistics. Tool calls and agent steps may also be counted differently.

The missing inputs are the actual input-token volume, cache mix, and billed usage for that evaluation. A reproducible cost calculation needs uncached input tokens times their rate, cached input tokens times their rate, and output tokens times their rate, summed across the run and divided by the number of tasks. Until those numbers are available, I use the two scenarios to explore where the claim might land.

Check a second kind of work

Terminal-Bench adds a different view of agent work in terminal environments. I keep its version and agent name next to each score because both change what a result means.

Arena’s Code WebDev Frontend leaderboard is a separate evaluation from FrontendBench. It uses blind community votes and a live Bradley-Terry rating. The August 4 snapshot has Claude Opus 5 at 1,710 and Kimi K3 at 1,692. Kimi ranks second with a 1–4 rank spread. The chart includes 20 frontier configurations from the 110-model Frontend category, with uncertainty, rank spread, votes, API token prices, and context retained. These are not pass rates and do not enter the cost frontier.

FrontendBench tests 148 front-end tasks across five categories. The paper’s automatic evaluator runs generated HTML with Puppeteer and Jest. Its Table 3 puts o3-mini first at 83.11%, followed by DeepSeek R1 at 75.00%, Gemini 2.5 Pro at 70.27%, and DeepSeek V3 at 66.89%. The selector also includes the paper’s human pass rates and automatic-human consistency rates. The paper does not report task costs, so these results use score bars instead of the cost axis.

The historical 2.1 selector is a frozen mirror of the 22 official rows displayed on Harbor Hub across 89 tasks, normally five trials per task. Its point-estimate frontier starts with Grok 4.5 in Cursor CLI at $0.3013 per task, then Terminus 2 with Fable 5, followed by GPT-6 Astra at low, medium, and high effort in Codex. Harbor associates 445 trials with every row, but the Opus 4.7 / Claude Code metric aggregate reports 447; its derived metric averages use that published metric denominator. No vendor or independent supplemental rows are included in this versioned view.

The current 4.0 chart view contains 18 official configurations and 33 supplemental configurations across vendor and independent sources. Anthropic supplies 15 complete score-and-cost points for Fable 5.1, Mythos 5.1, and Mythos 5 across low, medium, high, xhigh, and max effort. DeepSeek supplies seven max-effort scores under DeepSeek Harness Minimal, and Artificial Analysis supplies eight independent scores. Supplemental runs without published cost remain in the score-only panel.

The official-only Pareto frontier runs from Luna at max through Astra’s low, medium, high, and max settings. With supplemental results visible, Anthropic’s Mythos 5.1 xhigh and max points extend the high-capability end. The lines connect reasoning settings only when the model, agent, evidence status, and source match.

Gemini’s strong DeepSWE result does not carry over to the same position in this Terminal-Bench 4.0 selection. Astra and Opus run through their native coding agents here, while Gemini runs through mini-SWE-agent. Task mix and harness both differ, so this is evidence about those evaluated systems. It cannot isolate how much of the gap comes from the model itself.

Meta’s 88.8% Terminal-Bench claim is for version 2.1, under its native-harness evaluation setup. It is not an official displayed Harbor Hub row, so it does not appear in either chart dataset. A future Muse 1.3 result needs the same benchmark version and official source before I add it.

For a purchasing decision, I would add a small set of tasks from my own repositories. I care about regressions, time to a patch I can accept, and the total bill including retries. The public benchmarks help me choose which configurations to try first.

Sources

• Datacurve DeepSWE v1.1 leaderboard
• DeepSWE v1.1 configuration data (before website pricing corrections)
• DeepSWE methodology and evaluation harness
• Datacurve reporting corrections
• Meta Muse Spark 1.3 scores and token pricing
• Meta Muse Spark 1.3 evaluation methodology
• Meta Muse Spark 1.3 announcement and efficiency claims
• Terminal-Bench 4.0 leaderboard
• Terminal-Bench 2.1 official Harbor Hub leaderboard
• FrontendBench paper and reported model scores
• Arena Code WebDev Frontend leaderboard
• Anthropic Fable 5.1 and Mythos 5.1 Terminal-Bench 4.0 results
• Artificial Analysis Intelligence Index v4.3 Terminal-Bench results
• Flowtivity summary of Artificial Analysis Terminal-Bench 4.0 runs
• Anthropic Fable 5.1 system card, §8.3 DeepSWE v1.1 (p. 168)
• Artificial Analysis Fable 5.1 pre-release evaluation, September 1
• Artificial Analysis Fable 5.1, max with fallback
• Artificial Analysis Fable 5.1, xhigh with fallback
• Artificial Analysis Fable 5.1, high with fallback
• Artificial Analysis Fable 5.1, medium with fallback
• Artificial Analysis Fable 5.1, low with fallback
• Artificial Analysis coding-agent comparison, September 3
• Artificial Analysis live Coding Agent Index
• DeepSeek V4.1 Flash model card and technical report
• DeepSeek V4.1 Flash launch and API pricing
• OpenAI GPT-6 Sol and Luna launch and DeepSWE v1.1 results