24 Percent and Still Failed: What Opus 5 Really Shows in SlopCodeBench

The usual benchmarks for coding models follow a fairly straightforward pattern. The model receives a bug report, part of a repository and the task of producing a patch. In the end, the result is judged by whether the tests pass.

That is useful, but it covers only a small part of software development. Real projects do not end after the first successful patch. Requirements change, more features are added and earlier decisions have to survive. The code written today becomes tomorrow’s legacy code.

That is exactly where SlopCodeBench starts. The benchmark does not merely test whether a coding agent can solve one task. It measures how a codebase develops across several successive changes. New requirements build on the state produced by earlier checkpoints. Defects, unnecessary abstractions and poor architectural decisions do not disappear with the next prompt; they become part of the next problem.

Shortly after Claude Opus 5 was released, the first independent SlopCodeBench run appeared. The headline result sounds impressive: Opus 5 achieved a Strict pass rate of roughly 24 percent. Opus 4.8 and Sonnet 5 both reached only about 6 percent. (Anthropic)

The number is interesting, but considerably less definitive than the percentages suggest.

A benchmark for the second, third and eighth prompt

The current SlopCodeBench contains 36 problems with 196 checkpoints in total. Every problem describes software that is developed incrementally. A problem may begin with a simple parser, a database migration or a small analytics engine. Later checkpoints add requirements, edge cases and changes to existing behaviour.

At every checkpoint, the agent receives a fresh task description and the codebase it left behind at the previous checkpoint. It does not receive the previous chat history. Architectural decisions therefore remain in the repository while their rationale disappears. That is very close to the situation where a coding agent starts a new session and takes over an existing ticket. (arXiv)

The tests are hidden. The agent knows the specification, but neither the expected internal architecture nor the test cases. It can choose its own implementation while being unable to optimise directly for visible tests.

SlopCodeBench distinguishes several success metrics.

Core Pass checks whether the explicitly described core behaviour works.

Isolated Pass runs all tests for the current checkpoint but ignores regressions from earlier checkpoints.

Strict Pass requires both the current change and every previous requirement to continue working.

That distinction is essential. An agent can implement a new feature correctly and still lose Strict Pass because it broke something from an earlier step. Strict therefore measures more than the ability to generate code. It measures whether an agent can evolve a growing codebase without losing control of it. (arXiv)

Each checkpoint runs in a fresh Docker container under an unprivileged user. Agents may work for up to two hours. There is no artificial limit on tool calls or cost. The prompt is deliberately plain and simply asks the agent to solve the task. The benchmark therefore resembles an autonomous agent more than a carefully moderated pair-programming session. (arXiv)

The Opus 5 run

The published Opus 5 run does not cover the complete benchmark. It tests three selected problems with 17 checkpoints in total: one easy, one medium and one difficult problem. Opus 5, Opus 4.8 and Sonnet 5 were compared using the same prompts and the same Claude Code harness. Each model completed one full run per problem. (selinyilmaz.dev)

Model	Strict checkpoints passed	Strict pass rate	Problems solved completely
Claude Opus 5	4 of 17	23.5%	0 of 3
Claude Opus 4.8	1 of 17	5.9%	0 of 3
Claude Sonnet 5	1 of 17	5.9%	0 of 3

Opus 5 is clearly ahead in this run. It passes four times as many checkpoints as the other two models. At the same time, the most informative number in the table is not 23.5 percent but zero: none of the three models takes even one problem from beginning to end without intermediate failures and regressions. (selinyilmaz.dev)

Three of Opus 5’s four successful checkpoints come from the first three steps of the same problem. The fourth is the first checkpoint of a database-migration problem. Once trajectories become longer and the consequences of earlier decisions accumulate, the Strict streak ends for Opus 5 as well.

With only 17 checkpoints, a single pass changes the result by 5.9 percentage points. The difference between 24 and 6 percent therefore consists of exactly three additional successful checkpoints. The checkpoints are not statistically independent either. A poor architectural decision or defect introduced early leaves the next checkpoint starting at a disadvantage.

The run is therefore strong evidence that Opus 5 handles a growing codebase better than Opus 4.8 and Sonnet 5 in this scenario. It is too small, however, to support the broader claim that Opus 5 generally achieves a 24 percent SlopCodeBench score.

The 24 percent is not a new official high score

Discussion around the Opus 5 run often compares it with a previous top score of roughly 17 percent. That value comes from the first version of the SlopCodeBench paper. At the time, the benchmark contained 20 problems with 93 checkpoints and Opus 4.6 achieved 17.2 percent Strict Pass.

The revised May 2026 version expanded the benchmark to 36 problems and 196 checkpoints. The published model scores were recalculated accordingly. Opus 4.6 now reaches 9.7 rather than 17.2 percent. The best official score currently belongs to GPT-5.5 at 14.8 percent. (arXiv)

That is not a contradiction. It demonstrates how strongly a benchmark result depends on the benchmark version and the selected tasks.

The 23.5 percent Opus 5 score comes from an independent run over 17 checkpoints. The official scores come from 196 checkpoints. They can be discussed side by side, but not treated as one common leaderboard.

Official results from other models

The following table shows a selection of results from the current second version of the paper. Opus 5, Opus 4.8 and Sonnet 5 are not included there yet.

Model	Strict	Isolated	Core	Cost per checkpoint	Erosion	Verbosity
GPT-5.5	14.8%	28.1%	66.8%	US$1.51	0.49	0.32
GPT-5.3 Codex	11.2%	26.0%	60.7%	US$0.66	0.64	0.46
GPT-5.4	10.7%	23.5%	62.8%	US$0.72	0.51	0.33
Kimi K2.6	10.7%	18.9%	51.0%	US$0.74	0.76	0.51
GPT-5.2 Codex	9.7%	21.9%	56.1%	US$0.85	0.73	0.50
Claude Opus 4.6	9.7%	20.9%	67.3%	US$3.17	0.75	0.44
Claude Opus 4.7	8.2%	20.9%	65.8%	US$2.17	0.76	0.48
Claude Sonnet 4.6	7.1%	16.8%	57.7%	US$1.96	0.75	0.44
Composer 2	6.1%	16.3%	52.6%	US$0.44	0.72	0.45
MiniMax M2.7	2.0%	4.1%	28.1%	US$0.34	0.73	0.47

Lower is better for Erosion and Verbosity. The cost figures are the published average model cost per checkpoint. Models ran through their respective native agent harnesses. (arXiv)

Even the best model fails to solve any of the 36 problems completely. GPT-5.5 reaches 14.8 percent Strict Pass but 66.8 percent Core. The model can therefore implement a large share of the immediately described behaviour. It fails much more often once edge cases, hidden requirements and regressions are included. (arXiv)

For me, that gap is one of the most interesting results in the benchmark. Coding agents are now quite good at making a feature look finished. They are considerably worse at making sure the entire application is still correct afterwards.

Why Strict is so low

Strict Pass is deliberately difficult. A defect from checkpoint two can cause checkpoints three through eight to fail Strict as well even if the newly added features work in isolation.

That may look unfair. For autonomous software maintenance, however, it is exactly the right kind of harshness. A regression in a real repository does not disappear because the agent later implements other features correctly.

Isolated Pass helps separate the two effects. GPT-5.5, for example, reaches 28.1 percent Isolated Pass but only 14.8 percent Strict. GPT-5.3 Codex reaches 26.0 and 11.2 percent respectively. A substantial share of the current changes therefore works in isolation but fails across the complete trajectory.

SlopCodeBench measures two different things as a result.

One is the model’s immediate implementation ability. Core and Isolated are useful for that.

The other is the stability of a development trajectory. Strict is the important metric there.

Anyone asking whether an agent can produce useful code from a clearly written task should not look exclusively at Strict. Anyone planning to let an agent process several tickets in sequence without supervision should take that metric very seriously.

What “slop” actually means

SlopCodeBench does not only run tests. It also tracks how the generated code changes.

Structural Erosion measures how much structural complexity becomes concentrated in particularly complex functions. It combines measures including cyclomatic complexity and function size.

Verbosity flags suspicious lines through 137 static AST rules or duplication detection. Examples include unnecessary wrappers, redundant checks, repeated logic and other patterns that can inflate a codebase. (arXiv)

Across all official runs, structural erosion increases in 77 percent of development trajectories. Verbosity increases in 75.5 percent. The average number of particularly complex functions grows from 3.6 at the beginning to 23.7 at the end. Code duplication increases by an average of 96 percent over a trajectory. (arXiv)

For comparison, the authors analysed 473 public Python repositories. According to their metrics, agent-generated code was on average 2.3 times as verbose and 2.0 times as eroded as the comparison projects. The authors explicitly note that this is not a controlled human-versus-machine comparison. Public repositories arise under very different conditions, with code review, refactoring, changing requirements and sometimes many years of development. The figures are better understood as calibration for the metrics. (arXiv)

Opus 5 writes considerably more code

One striking observation in the small Opus 5 run is that the model creates substantially more functions than Opus 4.8 and Sonnet 5. Across all analysed artefacts, it produced roughly five times as many functions. Looking only at production code and excluding additional tests, the volume was about 1.8 times that of Opus 4.8. (selinyilmaz.dev)

That cannot be interpreted unambiguously as either a strength or a weakness.

Many small functions can reflect clean decomposition. They can just as easily come from over-abstraction, unnecessary helpers and defensive code. Opus 5 had the lowest average complexity per function in the run and kept duplication largely stable. Opus 4.8 produced fewer functions but allowed both complexity and duplication to grow much more strongly. (selinyilmaz.dev)

Saying “Opus 5 produces five times more slop” would therefore be just as unserious as saying “Opus 5 writes code that is five times better structured”. Without qualitative architectural review, the raw function count means little.

The same applies to the Verbosity metric. In the Opus 5 run, roughly 65 percent of lines were sometimes flagged at the first checkpoint and as much as 80 percent later. Values that high suggest that the rules fire very aggressively and that not every flagged line is genuinely poor code. The run’s authors themselves note that these metrics provide signals rather than a robust substitute for maintainability assessment. (selinyilmaz.dev)

The quality metrics are interesting, not laws of nature

Cyclomatic complexity, line count, duplication and AST rules have one major advantage: they are deterministic. Unlike LLM-as-a-Judge evaluation, no additional language model is deciding whether the code looks “clean” or “maintainable”.

That does not make the metrics automatically objective.

A long function can be problematic. It can also contain a linear and readable sequence of processing steps. Many small functions can form a clear architecture or spread a simple flow across several files. Additional error checks can be redundant or provide exactly the robustness missing from the specification.

The paper itself examines how well its Erosion metric predicts later results. Correlation with passing the next checkpoint is almost zero. Total code size and the maximum complexity of individual functions correlate more strongly with later cost than the composite Erosion score. (arXiv)

For me, that does not make quality measurement useless. It should be treated as a warning signal rather than an automatic judgement of architecture. The strongest indicator remains whether the software satisfies its requirements, preserves existing behaviour and can be extended at a reasonable cost.

A better prompt does not solve the problem

The authors also tested whether stricter prompts could prevent degradation, including an Anti-Slop instruction and a Plan-first prompt.

Both variants improved the measured code-quality metrics. Erosion and Verbosity went down. At the same time, however, the average Strict pass rate fell. With the Anti-Slop prompt it dropped by 2.4 percentage points; with Plan-first by 3.6. Costs increased by an average of 12.1 percent. For GPT-5.5, Strict fell from 14.8 percent with the simple prompt to 9.2 and 8.2 percent respectively. (arXiv)

That is a familiar problem when working with coding agents. Additional rules compete for attention. Forcing the model to think constantly about abstractions, duplication, function size and architecture may produce cleaner-looking code, but not necessarily more working software.

A long prompt with twenty quality rules does not replace an engineering process.

What SlopCodeBench does not measure

SlopCodeBench is not a universal judgement of which model is the best programmer.

It measures a relatively specific but important use case: an agent receives requirements in sequence, works largely autonomously, cannot see the hidden tests and has to continue from its own earlier code.

Many real coding sessions look different. A well-run project includes clarifying questions, architectural decisions, code reviews, CI feedback and a person who checks after each step whether the agent is still moving in the right direction. An agent will also rarely be given eight major changes in a row without anyone examining the state in between.

The agent software is part of the measured result as well. Models ran with their native harnesses, whose tool use, context management and default behaviour differ. The benchmark therefore compares practically relevant complete systems, not only the underlying language models. So far, only the Python track has been evaluated. (arXiv)

None of that makes SlopCodeBench worse. It simply defines what conclusions can reasonably be drawn from it.

What I take from the results

Opus 5 produces a good result in the small comparison. Four passed checkpoints instead of one is not statistical noise, and the trajectory suggests that the model can carry existing structure forward better than Opus 4.8 and Sonnet 5. A reliable placement against GPT-5.5, GPT-5.4 or GPT-5.3 Codex requires a complete run on the same benchmark version, however.

The common weakness of all models matters more. Not one agent solves even one complete problem in the current benchmark. That is true for the official 36 problems and for the three tasks in the Opus 5 run.

That does not make coding agents useless. Quite the opposite: Core scores above 60 percent show that models can implement many individual requirements correctly. The problematic leap is assuming that this means an agent can maintain growing software unsupervised over a long period.

For real projects, the result suggests a fairly sober workflow. Tasks should remain small enough to review. Regression tests need to run after every step. Architecture and code growth need their own checkpoints. Refactorings should not hide inside feature tickets. And by the third time an agent extends the same part of the system, somebody should check whether the original structure still holds.

The human is not needed because he types faster. He is needed because he decides when a locally plausible solution starts damaging the system as a whole.

Conclusion

The headline “Opus 5 reaches 24 percent in SlopCodeBench” is technically correct and still misleading.

It is true that Opus 5 passed four of 17 checkpoints in one independent run and clearly outperformed Opus 4.8 and Sonnet 5.

It is equally true that the run covers only three selected problems, each checkpoint is worth almost six percentage points, three of the four successes occur at the beginning of the same problem, and no model completes a single problem.

There is therefore no robust evidence yet that Opus 5 leads the complete SlopCodeBench. There is an initial indication that it can evolve its own codebase better than its immediate Claude predecessors.

The more important result concerns no single model. The gap between a convincing first implementation and reliably maintainable software remains large. SlopCodeBench makes that gap visible.

That is probably more valuable than another benchmark in which every model supposedly scores above 90 percent.