METR presents a useful check on benchmark interpretation, but the limited repository sample and one-shot agent setup make the measured gap evidence about evaluation conditions rather than a definitive ceiling on coding-agent capability.
Limits of the evidence: The findings cover 95 of 500 SWE-bench Verified issues across three of twelve repositories, and agents were not allowed to iterate in response to maintainer feedback. The reported difference in improvement rates is described by the source as suggestive.
What to watch
Taken from the first report; the whole story has not been re-assessed yet.
Many SWE-bench-Passing PRs Would Not Be Merged into Main
METR BlogMar 10, 2026First report
No revisions yet.
No maturity ladder applies to this desk.
Sens.ai aggregates and assesses published reporting. The assessment above is machine generated; the original sources are authoritative.