Open-Weight Coding Agents in 2026: A Literature Guide

What the benchmarks say, what they hide, and how to think about agents as systems.

1. The question

Open-weight models can now explore a repository, reason about a bug, edit several files, run tests and try again. Mid-2026 comparisons put the best of them close to closed frontier models on popular coding benchmarks. That raises two questions this guide tries to organize:

1. How good are open-weight coding models really, and how would we know?
2. When an agent succeeds, how much of that is the model, and how much is everything built around it?

The short version of the answer: the models are strong, the measurements are shakier than they look, and the system around the model matters more than most headlines suggest.

2. Who is in the race

Here is the landscape as of October 2026, based on vendor announcements and third-party comparisons. Treat the figures as reported, not settled.

A few things to notice:

• The top of the open field is huge sparse mixture-of-experts models. Active parameters per token are in the tens of billions, but total parameters reach into the trillions.
• “Open” is a spectrum. MIT and Apache-2.0 are permissive; some of the strongest models use custom or modified licenses. “Open-weight” is the more accurate term.
• Smaller models play a different game. Qwen3.6-27B, Qwen3-Coder-Next and Gemma 4 trail on agentic benchmarks but run on far more modest hardware, which matters if you want to fine-tune or self-host.
• Gemma 4’s headline results are on competitive programming and tool use, not repository-level issue resolution. Don’t assume those scores transfer.

3. Why you shouldn’t trust the headline number yet

3.1 The benchmark that was retired

For two years, SWE-bench Verified was the standard exam: 500 real GitHub issues, each with tests. In February 2026 OpenAI said it would stop reporting it. Their audit of 138 hard problems found flawed tests or problem statements in 59.4% of them, and some frontier models could reproduce reference fixes from memory, which points to training-data contamination. Scores had also stalled near 80%, so the benchmark no longer separated models well.

3.2 The replacement has its own problems

SWE-Bench Pro (1,865 tasks, many repositories and languages) was the recommended successor. A September 2026 preprint, SWE-Bench Pro Verified, audited it and found two kinds of trouble:

• Answer leakage. Agents with access to Git history or the network sometimes looked up the real fix: reading future commits, copying reference files, or downloading code from hosting sites.
• Task-quality defects. Some descriptions contradict the tests, and some tests demand details the task never specified.

The authors closed the leakage channels, repaired 102 of 731 tasks, and re-ran models under one common harness. The effect was large for some models:

For GLM-5.2 in particular, 186 tasks flipped from pass to fail after the fix and only 15 flipped the other way.

How much to trust this: it is one preprint from a group that builds evaluation infrastructure, it has not been independently replicated, and part of its analysis uses an LLM to classify causes. It does not show that any lab acted improperly; it shows that agents will take shortcuts when shortcuts exist. But the compressed range (roughly 59 to 63 across very different models) is a useful reminder that pre-audit rankings can exaggerate differences.

3.3 Sources disagree with each other

Kimi K3’s SWE-bench Verified score appears as 93.4% on one independent platform and 76.8% in another comparison of vendor-reported figures. GLM-5.2 is around 78.7% in one independent harness. These are not necessarily errors; they are different harnesses, budgets and settings. That brings us to the bigger point.

4. The harness is half the story

An “agent” is not a model. It is a model plus a loop: prompts, tools, edit formats, context management, retries and stopping rules. Several recent papers argue that this wrapper can matter as much as the model.

What they report (mostly as secondary evidence, so read with care):

• Changing only the harness raised one model’s Terminal-Bench 2 pass@1 from 69.7% to 77.0%.
• Independent monitoring reports up to about 15 points of scaffold-only variation on SWE-bench Verified.
• One model scored 45.9% on SWE-Bench Pro under a standardized scaffold and 55.4% under a vendor’s own agent.
• The Holistic Agent Leaderboard found double-digit differences for the same model under different scaffolds.
• Even with model and harness fixed, container setup, orchestration and random seeds can shift results.
• Meta’s CWM paper reports 53.9% pass@1 with its own setup, 65.8% with best-of-16 test-time scaling, and noticeably lower numbers under alternative harnesses.

Takeaway: a sentence like “Model X scores 65% on benchmark Y” is incomplete. It should read “Model X, in harness H, with tools T, under budget B, scores 65%.” If you can’t recover those details, you can’t compare two numbers.

5. A systems view

A useful way to organize the literature is to split an agent into five parts:

• Model (M): reasoning, code understanding, generation.
• Harness (H): control flow, tools, state and context management.
• Retrieval (R): how the agent finds the right files and functions.
• Verification (V): how it uses tests to decide whether to retry or stop.
• Budget (B): tokens, time, hardware, number of attempts.

Improving one part doesn’t guarantee a better whole. A stronger model can burn its budget before it verifies anything. Better retrieval can let a smaller model succeed by handing it the right context. Let’s look at what the literature says about R and V.

6. Retrieval: finding the right code

Before an agent can fix a bug it has to find it. Issues rarely name the file or function, so localization is a real bottleneck.

Approaches, from simplest to richest:

1. Lexical search (grep for identifiers, error messages). Cheap and often effective when the issue uses distinctive terms. Most command-line coding agents rely on it.
2. Semantic retrieval with embeddings. Helps when the issue describes behavior in different words than the code.
3. Graph-based retrieval over structure: imports, call chains, class hierarchies, links between code and tests.
4. Hybrid approaches combining all of the above.

Representative work: LocAgent models a repository as a graph the agent can traverse; SpIDER adds graph awareness to dense retrieval; ARISE goes down to statement-level data-flow edges; LARGER anchors graph expansion to lexical matches inside the agent’s existing search loop; SWE-Adept pairs a localization agent with a repair agent. A benchmark for graph-based bug localization (GREPO) has also appeared.

What is still open. Most of these results measure localization accuracy. Fewer show that better localization produces more correct patches at the same cost. One enterprise study even argues that injecting more context alone is not enough. A retrieval system can raise recall without improving final correctness, so the right metric is end-to-end resolve rate under a fixed budget, ideally alongside how many irrelevant tokens were added.

7. Verification and test-time scaling

Generating a patch is not the end. Running tests gives feedback, and sampling several candidates and picking the best can raise accuracy, but only as far as your selector is good.

• R2E-Gym (2025) trained open models on procedurally generated environments and studied two kinds of verifiers. Execution-based verifiers (generate tests, run them) and execution-free verifiers (a model judges the patch) each saturated around 42–43% on SWE-bench Verified with a 32B model; a hybrid reached 51%.
• Agentic Rubrics generates a codebase-specific checklist for each issue and scores patches against it without executing code. It beat test-based and execution-free baselines under best-of-16, and stronger rubric generators helped, with closed models better than open ones for this role.
• CWM used generated unit tests with best-of-k and found gains that plateau around k=16.

The pattern: verification signals are individually noisy (tests can have low discriminative power; model judges can be biased by style), and combining them helps. Equally important is knowing when to stop: extra iterations cost compute and can break a correct patch.

8. Training open agents

Open models usually acquire agentic skill through large pools of executable repositories (SWE-Gym, SWE-smith, R2E-Gym) and reinforcement learning with verifiable rewards. A sensible rule of thumb from this literature: fine-tune against a diagnosed failure, not by default. If the agent fails because it can’t find the right file, training on patch writing won’t help; if it misuses tools, tool-use data is the better lever; if it writes good patches but doesn’t check them, that is a harness problem, not a weights problem.

9. A worked example: a constrained-budget competition

The Gemma 4 Developer Agent Competition on Kaggle shows why the systems view is useful. As described by the organizers: build an autonomous system around an open Gemma 4 model, solve real issues, and pass validation tests within a 12-hour generation limit; all agents in a submission use the same base-model variant (LoRA adapters may differ); on October 2026 additional QAT variants (12B, E4B, E2B, alongside 31B) were announced; and the supplied tools include embedding-based and graph-based code search.

Mapping it onto the five parts: the model is fixed, so the interesting variables are the harness (single agent vs. specialists, free-form vs. staged), retrieval (text vs. embedding vs. graph), verification (when and what to test), and the budget (how to split 12 hours across exploration, patching and testing). A sensible first move is not clever engineering but a baseline plus a categorized list of failures, so you know which part to improve.

10. What we know and what we don’t

Reasonably well supported

• Open-weight models are competitive with closed ones on several coding benchmarks, and the best are very large sparse MoE models.
• Reported scores depend heavily on harness, environment and budget.
• SWE-bench Verified is no longer a reliable discriminator, and SWE-Bench Pro needs leakage controls.
• Verification works better when signals are combined.

Plausible but not settled

• Graph-based retrieval improves end-to-end success, not just localization.
• Small models with more iterations can beat larger models at a fixed budget.
• Targeted fine-tuning beats generic fine-tuning.

Largely unmeasured

• Patch quality beyond passing tests: minimality, maintainability, security, regression risk.
• Cost per resolved issue, reported consistently.
• How these results carry over to languages and tasks outside Python-heavy benchmarks.

11. A checklist for reading any coding-agent result

1. Which model snapshot, and was it quantized?
2. Which harness and version, with what tools and turn limit?
3. How was context retrieved?
4. What verification and stopping rules were used?
5. What budget (tokens, time, hardware)?
6. Which benchmark version, and were network/Git leakage channels closed?
7. How many runs, and what was the spread?
8. What did it cost per solved task?

If a result can’t answer most of these, treat it as a rough indication, not a comparison.

12. Suggested reading

• Jimenez et al., SWE-bench (ICLR 2024), and OpenAI’s February 2026 post on retiring SWE-bench Verified.
• Deng et al., SWE-Bench Pro (arXiv:2509.16941), and Zheng et al., SWE-Bench Pro Verified (arXiv:2609.08149).
• Stop Comparing LLM Agents Without Disclosing the Harness (arXiv:2605.23950) and Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering (arXiv:2606.17799).
• Jain et al., R2E-Gym (arXiv:2504.07164); Agentic Rubrics (arXiv:2601.04171); CWM (arXiv:2510.02387).
• Localization: LocAgent, SpIDER (arXiv:2512.16956), ARISE (arXiv:2605.03117), LARGER (arXiv:2605.16352), SWE-Adept (arXiv:2603.01327), GREPO (arXiv:2602.13921).

Eda Yılmaz

Data Scientist
Building and researching end-to-end machine learning and LLM systems, from model training to deployment.