# FrontierCode Diamond: 13.4% Mergeable Patches on 50 Hard Tasks (June 2026)
> The June 2026 FrontierCode results: 13.4% on Diamond’s 50 hard tasks versus 51.8% on Extended, and how to inspect an AI-generated patch.
Source: https://ailoot.ai/blog/en/frontiercode-benchmark-mergeable-ai-code/
Canonical: https://ailoot.ai/blog/en/frontiercode-benchmark-mergeable-ai-code/
Language: en
Publisher: Ailoot (https://ailoot.ai/)
Translation: guide-173
Revision: sha256-76afef3cddd524f291abb34df3c4bebaab1cccaec14f0ceb6fb4bbf58f5d2164
Key takeaway
On June 8, 2026, Cognition released FrontierCode, a benchmark that measures whether a real open-source repository maintainer would agree to merge AI-generated code into their project. The headline result: even Claude Opus 4.8, the industry's best model, passes only 13.4% of the 50 hardest tasks. Older benchmarks such as HumanEval and SWE-bench measured “does the code work”; the new one measures “would the maintainer take this PR?” This changes the frame for everyone pushing AI-generated code to production.
Additional materials on configuring tools are in the [Ailoot documentation](https://ailoot.ai/docs/).
[Ailoot guides](https://ailoot.ai/docs/)
## What is the FrontierCode benchmark?
Key takeaway
FrontierCode is an AI-code quality benchmark from Cognition, published on June 8, 2026. It evaluates whether a patch satisfies the benchmark’s maintainer-oriented merge criteria. That is a defined benchmark outcome, not a general production-readiness rate for all AI code. More than 20 maintainers of major open-source projects spent over 40 hours on each task to assemble scenarios that require not just correctness, but also style, security, and tests.
Cognition is the company behind Devin (the first autonomous AI engineer, March 2024) and Windsurf (acquired in July 2025). They are also developing their own SWE-1.x model line. The benchmark was published on the public blog at `cognition.com/blog/frontier-code`.
The key shift is not about new numbers. It's about **the scale used to measure things now**.
The benchmark has three difficulty levels: **Extended** (150 tasks, the full set), **Main** (the 100 hardest, including Diamond), and **Diamond** (50 hardcore tasks). Cognition deliberately left Diamond “largely unsaturated”—even the best model solves fewer than 15%. This is intended to give the benchmark a year's headroom and keep it from being saturated in a month, as HumanEval was.
## What FrontierCode Measures: “Would the Maintainer Merge This PR?”
Key takeaway
FrontierCode evaluates each solution along 6 axes: whether the patch solves the task, avoids breaking other code, passes the build and lint checks, has valid tests, stays within scope, and has readable code. For a patch to count as mergeable, it must pass all 6. Fail even one, and the PR won't be merged. This shifts the measure from automated tests to a real maintainer's decision.
The six evaluation axes (from Cognition's official blog):
1. **Behavioral correctness** — does the patch solve the task?
2. **Regression safety** — does it avoid breaking existing code?
3. **Mechanical cleanliness** — does it pass the build, lint, and style checks?
4. **Test correctness** — are the tests written by the agent valid (tests that pass on any code do not count)?
5. **Scope** — does the patch avoid touching files that were not requested?
6. **Code quality** — does the code follow the project's conventions, and is it readable?
For a patch to count as mergeable, it must pass **all six axes**. Fail even one, and it won't be merged.
That's the main shift in perspective. “Correctness is now table stakes,” as Cognition puts it. Before, benchmarks asked, “Did the code pass the tests?” Now they ask, “Would a real tech lead take this into the project?”
Cognition introduced three new grading techniques:
- **Reverse-Classical**: a test written by an agent must fail on a broken version of the code. Otherwise, the test is a sham.
- **Code Scope**: an automated checker verifies that a patch stays within the files and line count agreed for the task.
- **Adaptive Classical Grading**: the LLM tool “mutagent” adapts strict tests to valid solution variants.
## Why Don't the Old Benchmarks Work Anymore?
Key takeaway
HumanEval (Codex, 2021) and SWE-bench check whether code passes automated tests. They don't catch cases where the test passes but nobody will accept the PR: the code is ugly, breaks neighboring modules, or goes beyond the task's scope. In March 2026, an independent study evaluated 296 AI-generated PRs from SWE-bench Verified with 4 real maintainers: the maintainers' decisions averaged 24 percentage points below the automated evaluator. That means roughly half of the “successful” PRs won't be merged into a real repository.
AI agents had already aced HumanEval by 2024. SWE-bench Verified is much the same: in August 2025, OpenAI publicly acknowledged the contamination problem (models had “seen” the dataset during training). Benchmarks lost their ability to differentiate between top models.
The key study that undermined the old yardstick came out in March 2026.
What they did:
- Selected 4 active maintainers from 3 repositories (scikit-learn, Sphinx, pytest).
- Had them evaluate 296 AI-generated PRs from SWE-bench Verified—all of these PRs passed automated tests.
- Compared the maintainers' decisions with the automated evaluator's.
In other words, the benchmark says “passed,” while the maintainer says “won't merge”—and that happens in an average of a quarter of cases. The top 3 reasons for rejection, from least to most serious:
1. **Code quality** — doesn't match the repository's style.
2. **Breaks other code** — breaks neighboring code.
3. **Core functionality failure** — doesn't fully solve the task, even though the test is green.
FrontierCode formalizes exactly this problem. Cognition took this idea and expanded it into a full benchmark with more than 20 maintainers instead of 4.
Another nail in the coffin is a December 2025 study on the security of AI-generated patches.
In it, 61% of SWE-Agent solutions with Claude 4 Sonnet were functionally correct, but only 10.5% of them were safe.
The code passes the test, and yet every other patch introduces a security hole. You can't see this at all on SWE-bench benchmarks—they don't measure security. FrontierCode at least tries to catch it through the “mechanical cleanliness” and code quality axes.
## Where the Models Rank: Opus 4.8 at 13.4%, GPT-5.5 at 6.3%
Key takeaway
Cognition published a leaderboard of 12 models across three subsets. Claude Opus 4.8 from Anthropic leads on Diamond at 13.4%. Next are GPT-5.5 at 6.3%, Claude Opus 4.7 at 5.2%, and Google's Gemini 3.1 Pro at 4.7%. The best open-code model is Moonshot's Kimi K2.6 at 3.8%. Cognition's own SWE-1.6 model sits in the lower part of the table at 2.5%. This is an important argument against accusations of self-promotion.
| Model | Diamond (50) | Main (100) | Extended (150) |
|---------------------------------|--------------|------------|----------------|
| **Claude Opus 4.8** (Anthropic) | **13.4%** | 34.3% | 51.8% |
| GPT-5.5 (OpenAI) | 6.3% | \- | \- |
| Claude Opus 4.7 (Anthropic) | 5.2% | \- | \- |
| Gemini 3.1 Pro (Google) | 4.7% | \- | \- |
| GPT-5.4-mini (OpenAI) | 4.6% | \- | \- |
| Kimi K2.6 (best open-code model) | 3.8% | \- | \- |
| Claude Sonnet 4.6 | 3.5% | \- | \- |
| **SWE-1.6** (Cognition) | **2.5%** | \- | \- |
| MiniMax M2.7 | 2.4% | \- | \- |
| MiniMax M2.5 | 1.1% | \- | \- |
| Kimi K2.5 | 1.0% | \- | \- |
| Gemini 3.1 Flash Lite | 0.7% | \- | \- |
What stands out from the table:
- **One clear leader.** Anthropic's Opus 4.8 is roughly twice as far ahead as its nearest competitor. The gap that was hard to notice on SWE-bench is plainly visible on FrontierCode.
- **Open models haven't caught up yet.** Kimi K2.6—the best open-code model—scores 3.8% on Diamond. That's 3–4 times lower than Opus 4.8. Cognition didn't publish a complete Kimi K2.6 grid for Main and Extended, so there is no comparison on the full set yet.
- **Cognition honestly put its own model** in the lower part of the table.
The Diamond subset was deliberately left “unsaturated.” Cognition built in a safety margin for years to come. If previous benchmarks saturated in 12–18 months, FrontierCode is designed to have a higher ceiling.
## What Does “13% Mergeable” Mean for a Vibe Coder Using Claude Code?
Key takeaway
For the reported Opus 4.8 run on Diamond’s 50 selected hard tasks, 13.4% of patches met the benchmark’s mergeability criteria and 86.6% did not. That does not establish that the rejected patches all looked correct, or that the same rate applies to ordinary project work. In vibe-coding practice, this means you can't just look at a green test and push when making difficult changes. Simple changes in a fresh project are a different story; AI handles those well. The boundary between “when to push and when to refine it by hand” has shifted.
First, an important reframing: FrontierCode's **Diamond** set is deliberately hardcore. On ordinary tasks in a fresh startup project without forty dependent modules, an AI agent does noticeably better. The same Opus 4.8 scores 51.8% mergeable on Extended (150 tasks, the full set). That's a completely different picture.
The 51.8% result applies to the benchmark’s 150-task Extended set. It is higher than the Diamond result, but neither number estimates the mergeability of every model’s patches on your own tasks. Missing Main or Extended scores for other models do not justify filling in their performance.
What this means for a vibe coder pushing to production:
- **A green test no longer means “ready to merge.”** Tests written by the AI itself may be duds—they pass on any code. FrontierCode measures this directly through Reverse-Classical: if a test doesn't fail on a broken version, it doesn't count.
- **Repository style and conventions matter.** If a project has an established pattern and the AI doesn't follow it, a real collaborator won't accept the change. In solo projects with no rules, this isn't a big deal. On a team, it's problem number one.
- **Tests don't solve security.** Remember the figure above: only 10.5% of patches are safe despite 61% being correct. This needs to be checked separately, using tools that catch tokens, `.env` files, and default passwords.
- **Going beyond the task is a typical AI mistake.** You ask it to fix an authorization function and get a patch that also rewrote the internal layer and logging. This is one of FrontierCode's 6 axes.
[Ailoot guides](https://ailoot.ai/docs/)
## What 6 Categories of Errors Does AI Make on Difficult Tasks?
Key takeaway
The 6 FrontierCode axes correspond to 6 types of errors AI agents make regularly. These are predictable patterns of weak results. If you take a random patch from Claude or Codex and assess it on these 6 axes, a real tech lead won't accept it if even one item fails. To push to production with confidence, you need to check these 6 axes yourself before merging.
Let's go through each axis in plain language:
**1. Behavioral correctness — does the patch solve the task?**
The most basic axis. AI often writes code that **looks** like a solution but solves the wrong problem. I ask it to “add a read receipt to the message” and get a `markAsRead` function, but it hasn't been connected to the chat component itself. The function test passes. The business task isn't solved.
**2. Regression safety — does it avoid breaking neighboring code?**
AI changes one module, and now another one doesn't work. The most common case: an AI agent changes a function signature, forgetting that it's called from 17 other places. You can't see this on SWE-bench—they run tests only for one file. On FrontierCode, they run **the repository's full test suite**.
**3. Mechanical cleanliness — does it pass the build, lint, and style checks?**
AI can produce code that compiles locally but fails in CI: a missing import, tabs instead of spaces, two trailing spaces, or a slip in `eslint.config.js`. In a live project, style isn't a whim; it's a team rule.
**4. Test correctness — are the tests written by AI valid?**
The trickiest axis. AI writes a test that passes. But it **passes on any code, including broken code**. I've seen this many times: the test checks “did the script crash?” but not “did it return the right result?” FrontierCode's Reverse-Classical catches exactly this case.
**5. Scope — does the patch avoid touching unrelated files?**
I ask it to fix one function and get a patch for 12 files, including files unrelated to the task. That's the very “initiative” from AI that breaks a production project by changing things I didn't ask it to change. This axis is a hard blocker in FrontierCode.
**6. Code quality — does the code follow the repository's conventions?**
AI writes its own function “from scratch” instead of using existing helpers in `lib/`. Or it names variables in a way that doesn't match the project's style. Or it puts logic in the wrong architectural layer. From the test's perspective, everything works. From the tech lead's perspective, it won't be merged.
## How to Check AI Code Before Pushing: 5 Steps
Key takeaway
A 5-step checklist that covers the 6 FrontierCode axes and a security check. Run through it for every patch from Claude or Codex before release. There are few steps and each takes minutes. The payoff is good: spend 10 minutes on the checklist rather than 4 hours rolling back a production project with git rewind and restoring the database from a backup.
1. ### Run the full test suite, not just tests related to the change
AI often changes code beyond the task. The problem will show up in neighboring tests. I run the complete test suite (with the single command `npm test`, without a path), not just tests for a specific file. On large projects, it's convenient to do this in parallel using a git worktree (see [Git worktree in Claude Code](https://ailoot.ai/blog/en/claude-code-git-worktree/)).
2. ### Run lint, style, and type checks
Three commands in order: `eslint .`, `prettier --check .`, `tsc --noEmit`. An AI agent may miss both tabs and an extra trailing space. If the project has strict checks, run them locally before release.
3. ### Test the test itself
The most important of the 5 checks. I make a small “break” in the code (change `return true` to `return false`). The test should fail. If the test **doesn't fail**, AI wrote it just for show, and it's useless. I delete the test and ask AI to rewrite it with a mandatory condition: the test must fail on the broken version.
4. ### Check which files AI touched
The `git diff --stat` command shows which files changed and how many lines. If AI touched 12 files in response to “fix the contact form,” it went beyond the task's scope. I revert the extras with a single command for the file.
5. ### Run a security check using a ready-made skill
Tests don't catch SQL injections or hardcoded passwords. I use a ready-made security skill for Claude Code (see 754 ready-made security skills). At a minimum, search the project by hand for the words “password,” “secret,” and “key” in the code, and check `.env.example` for completeness.
All 5 steps cover 5 of FrontierCode's 6 axes (1–5). The sixth axis—code quality—is checked automatically through linting plus a human review.
## What Actually Improves the Chances of Mergeable Code?
Key takeaway
FrontierCode describes the problem. The solution lies in the methodology for working with AI; switching the model itself makes practically no difference here. Three practices make the difference: project rules, a project map, and context engineering. Without them, even Opus 4.8 produces plausible junk. They give the agent clearer constraints, but difficult changes still need the repository’s checks and review before merging.
An AI agent in an unprepared project sees **only the file you've opened for it**. It has none of your decision history, none of your conventions, and no map of your modules. That's not a big deal for simple changes. For difficult changes, missing project context is a risk worth addressing. The benchmark’s 13.4% figure does not isolate context quality as its cause.
What you need to do to turn that 13% into something you can actually push to production:
**1. Project rules — so AI reproduces your reasoning instead of the internet's average.**
This is a set of your ready-made solutions, habits, and learned rules in a format the AI agent reads with every request. Once there are 30+ rules, AI stops inventing the “industry average” and starts making suggestions in your style. Going beyond the task's scope can be addressed with one line in the rules: “don't touch files outside those explicitly named in the request.”
**2. A project map — a folder that describes the project.**
This is where you put products, audience, economics, metrics, and setups. Before each request, the AI agent reads the table of contents and understands **which part of the business it's working on**. This automatically addresses two of the reasons for rejection listed above: style and breaking other code.
**3. Context Engineering — managing what AI sees before the request.**
It's not “how to phrase a prompt,” but “what to put in the context window BEFORE the prompt.” CLAUDE.md, prerequisites, and targeted links to files. This addresses behavioral correctness and code quality failures.
Use the combination as preparation for the checks, not as evidence that five axes already pass. Verify behavior, regressions, tests, scope, and code quality on the actual patch before accepting it.
An additional layer is review agents:
- **Cursor Bugbot** — runs a review on every PR. According to Cursor, the tool processes around 2 million PRs a month.
- **Nested subagents in Claude Code v2.1.172** — one agent writes the code, a second reviews it, and a third runs tests. It's the same idea as FrontierCode: separate the roles of “writer” and “tech lead.”
See also: [Vibe Coding Security](https://ailoot.ai/blog/en/vibe-coding-security/), [How to Configure Claude Code So It Doesn't Break Your Project at Night](https://ailoot.ai/blog/en/claude-code-safe-overnight-setup/), and Context Engineering as a concept.
### Sources
- [Cognition - Introducing FrontierCode (June 8, 2026)](https://cognition.com/blog/frontier-code)
- [Cursor Bugbot - Code Review Agent](https://cursor.com/bugbot)
For additional materials on working with tools, see the [Ailoot documentation](https://ailoot.ai/docs/).
[Ailoot guides](https://ailoot.ai/docs/)
[Go to Ailoot](https://ailoot.ai/)
## Questions about FrontierCode Diamond and mergeable AI patches
### Does 13.4% mean only that share of all AI code is production-ready?
No. It is the reported mergeability result for Opus 4.8 on FrontierCode Diamond’s 50 selected hard tasks. The same table reports 51.8% on the 150-task Extended set; neither rate measures all AI-generated code.
### Why is a passing test insufficient for merging a patch?
The benchmark also checks regressions, build and style, whether the tests detect a broken implementation, scope, and code quality. Inspect the actual diff and run the repository’s acceptance checks; a green test alone does not establish all those conditions.
## Ailoot · The company and its project portfolio
Ailoot combines established operating experience, a large client base, stable service and ongoing KPI control.
The company and its project portfolio: millions of customers; monthly sales in the billions.
Ailoot brings that operating experience to API access for AI features in customer products, agents, applications and automations.