Real-SWE Tests AI Coding on Private Codebases

An AI coding agent that aces a public benchmark may just have a very good memory. Show it code it has never seen, and the illusion tends to slip.

The Real-SWE coding benchmark and its release built around private codebase tasks are aimed squarely at that illusion. Instead of testing AI agents on the public code that floods the internet, this class of benchmark uses tasks drawn from real, private company codebases the models were never trained on. The result is unglamorous and important: scores that looked dazzling on public tests fall noticeably when the code is genuinely new. That gap is the whole point.

Short Answer

• Real-SWE and similar benchmarks evaluate AI coding agents on private, previously unseen codebases rather than public ones.
• They exist because public benchmarks can leak into training data, letting models memorize answers instead of reasoning through problems.
• On private code, top models drop several points, which gives buyers a more honest read on real-world coding ability.

What Real-SWE Is

To understand Real-SWE, start with the benchmark it improves on. The widely cited SWE-bench asks an AI agent to resolve real software issues from open-source projects, which is a genuinely useful test. Its weakness is that those projects are public, so their code and fixes can end up in a model’s training data.

Real-SWE-style benchmarks close that door. They use tasks built from private codebases, often contributed by real startups under agreements that keep the code off the public internet. An agent facing these tasks cannot have seen them before, which turns the test from a memory check into a reasoning check.

Why Private Codebases Change Everything

The problem they solve has a plain name: contamination. When a benchmark is public, its questions and answers can seep into the next model’s training set, and a model that has effectively seen the answer key will look smarter than it is. This is the quiet reason so many benchmark numbers feel too good to be true.

Private code breaks the cycle. A held-out, commercial codebase that never gets published cannot be memorized, so a high score has to come from actually working through the problem. Some efforts go further and split tasks into tiers, including public repositories, held-out private ones, and commercial code that is never released, which makes gaming the test far harder.

The Scores Drop, and That Is the Point

Here is where it gets concrete, and a little humbling for the hype. On private, unseen codebases, leading models slip. In one closely watched private evaluation, a top model’s resolution rate fell from roughly 22.7% to 17.8%, and another dropped from about 23.1% to 14.9%. Those are not rounding errors. They are the difference between a demo and a dependable tool.

Worth sitting with that second number for a second. A model solving fewer than one in six real, unfamiliar tasks is a very different product from one the marketing suggests. The private benchmark is not being harsh. It is being realistic, which public benchmarks often are not.

What This Means for AI Coding Claims

For anyone choosing an AI coding tool, this changes the questions worth asking. A leaderboard number means little without knowing what it was measured on. A short checklist helps:

• Ask which benchmark a claim is based on, and whether the test code was public or held out.
• Treat public-benchmark records as a ceiling, not a promise, since real work looks more like the private tests.
• Run a trial on your own codebase, which is the ultimate private benchmark and the only one that reflects your stack.
• Watch the gap between a vendor’s public and private scores, because a big drop is a signal about generalization.

Developers reached this skepticism first. On engineering forums, a common refrain is that a tool’s benchmark glory rarely survives contact with a real, messy codebase, and some studies have even found experienced programmers working slower with AI on code they know well, despite feeling faster. Private benchmarks are catching up to a gut instinct working developers already had: the number on the leaderboard and the help you feel at your desk are two different measurements.

Main Takeaways

• Real-SWE-style benchmarks test AI coding agents on private, unseen codebases.
• They fight contamination, where public test data leaks into training and inflates scores.
• Top models score meaningfully lower on private code, revealing a reasoning-versus-memory gap.
• Buyers should weight private-benchmark and own-codebase results over public leaderboards.

Frequently Asked Questions

What is Real-SWE?

It is a coding benchmark that evaluates AI agents on tasks drawn from private, real-world codebases the models have not seen before. It builds on the public SWE-bench by removing code that could have leaked into training data.

Why do private codebases matter for benchmarks?

Because public benchmarks can contaminate training data, letting a model memorize answers. Private, held-out code cannot be memorized, so a high score must reflect genuine problem-solving rather than recall.

Why do AI models score lower on private tests?

On unfamiliar code, a model cannot lean on anything it may have seen during training. It has to reason through the actual problem, which is harder, so resolution rates fall compared with public benchmarks.

Does a lower private score mean AI coding tools are bad?

No. It means the honest number is lower than the marketing number. These tools remain useful, but private benchmarks give a more realistic picture of where they succeed and where they still struggle.

How should I evaluate an AI coding tool myself?

Ask what benchmark a claim uses and whether the code was held out, treat public scores as a ceiling, and run the tool on your own codebase. Your private repository is the most relevant test you have.

What This Means

Benchmarks shape what companies build and buy, so a benchmark that resists gaming quietly raises the standard for everyone. Real-SWE and its private-codebase cousins will not make headlines the way a shiny new model does, but they change the conversation from who claims the highest score to who earns it on code nobody has seen. For more developer coverage, browse Wayodd’s Computing and Software sections. The next time an AI coding tool posts a jaw-dropping number, the useful reflex is a single question: on whose code?