AI Engineering

AI Coding Agents Have a Scope Problem, Not Just a Code Quality Problem

By GitMe Team •

The code works. But did the agent build what you actually needed?

Imagine asking a coding agent to add CSV export to an existing reporting screen.

The agent returns a working implementation. The tests pass. The code is reasonably structured. The pull request is ready.

There is just one problem.

Instead of adding a simple export function, the agent has introduced a generalized export framework, a background processing service, additional configuration, and abstractions for formats nobody requested.

The implementation may be technically correct.

But the team now owns a system it never intended to build.

This is a hypothetical scenario, yet it captures an important distinction in AI-assisted software development:

A technically correct solution can still be an engineering mistake.

The problem is not necessarily code quality.

It is scope.

Code Quality and Scope Quality Are Different Problems

Traditional code review asks familiar questions.

Does the implementation work? Is it readable? Is it secure? Does it follow the project's conventions? Are the tests sufficient?

These questions remain essential when AI agents write the code.

But they leave another question unanswered:

Did the agent implement the right amount of the right thing?

A coding agent can produce clean, well-tested code that exceeds the requested scope.

It can introduce a new abstraction when an existing function would have been sufficient.

It can expand a narrow bug fix into a larger refactoring.

It can add defensive behavior for hypothetical requirements while overlooking the actual operating constraints.

It can even build an entirely new feature that appears useful but was never requested.

These are not all the same failure. Some are overengineering. Others involve unintended side effects, misunderstood requirements, or changes outside the intended boundary.

What they share is a gap between the requested outcome and the delivered implementation.

That gap matters because every additional component creates obligations for future development, testing, security, deployment, and maintenance.

And AI makes it unusually easy to create those obligations.

What Research on Reverted Agent Code Reveals

A 2026 paper presented at the International Conference on Mining Software Repositories, When AI Code Doesn't Stick: An Empirical Study on Reverted Changes Introduced by AI Coding Agents, examined why some contributions from coding agents are later reversed.

The researchers analyzed 33,580 agentic pull requests containing 86,315 commits associated with five coding agents: Claude, GitHub Copilot, Cursor, Devin, and OpenAI Codex.

They found that 2.66% of the examined pull requests contained at least one reverting commit.

To investigate why those reversions occurred, the researchers manually analyzed 500 reverting commits and developed a taxonomy covering eight categories and 25 themes.

The findings highlight several types of problems, including unintended side effects, overengineering, functional incorrectness, code quality issues, and dependency management failures.

An important conclusion is that agent-generated changes can fail because of scope management and contextual understanding, not only because their code is functionally wrong.

This does not mean scope errors are more common than correctness errors across all AI-generated code.

The qualitative analysis examines reverting commits, not a representative sample of every successful and unsuccessful agent contribution.

And a reverting commit is not a complete measure of software quality. Changes may be rejected before merge, silently rewritten afterward, or legitimately replaced because requirements have evolved.

Still, the research identifies a failure mode that engineering organizations cannot address through functional testing alone.

The code may do what the agent intended while doing more than the organization needed.

Rejection Research Points to the Same Boundary

A separate MSR 2026 study, Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub, examined approximately 33,000 agent-authored pull requests and manually analyzed 600 PRs to investigate rejection patterns.

Alongside CI failures and review-related issues, the researchers identified unwanted feature implementations and agent misalignment.

These observations add another perspective.

Not every rejected pull request is rejected because the implementation is broken.

Some contributions fail to match what maintainers expected or wanted.

For engineering leaders, that distinction is consequential.

A change can pass unit tests but still require substantial human effort to narrow, restructure, or discard it.

If an organization measures agent success primarily through generated code, completed tasks, or passing tests, that cost may remain invisible.

Passing Tests Does Not Establish Scope Alignment

Tests usually verify expected behavior against a defined set of conditions.

They rarely establish that every implemented capability was necessary.

Suppose an agent adds a new service to solve a request that could have been handled within an existing module.

The tests may verify that the new service works.

They may not ask whether the service should exist.

The same problem appears with unnecessary abstractions, redundant dependencies, excessive configuration, and unrelated refactoring.

A technically successful change can increase the surface area that future engineers must understand and maintain.

This creates two different review obligations.

Correctness review: Does the implementation satisfy the required behavior?

Scope review: Does the implementation stay within the intended problem, constraints, and architectural boundaries?

Neither can replace the other.

A narrow change can still be defective. A large change can be entirely justified.

The objective is not to minimize code at any cost.

It is to make the scope of the implementation proportional to the problem being solved.

The Hidden Cost of Doing More Than Necessary

Overengineering is often discussed as a design preference.

In an AI-assisted workflow, it can become an operational cost.

Consider the lifecycle of an unnecessary abstraction.

First, the coding agent creates it.

Then reviewers must understand it.

Tests may need to cover it.

Other developers may begin using it.

Later, the team discovers that the abstraction is not needed or does not fit the architecture.

Removing or simplifying it creates another round of engineering work.

The initial implementation might have taken the agent minutes.

But that does not mean the organization received value equal to everything the agent produced.

The relevant question is what happened after that initial delivery.

Was the implementation preserved?

Was it simplified?

Did it require additional fixes?

Did it make subsequent changes easier or harder?

Was the extra complexity justified by real requirements?

Some revisions are healthy engineering. Removing unnecessary code is often a valuable improvement, not evidence of failure by itself.

The concern is the repeated creation of work that never needed to exist.

Cheap code generation can make unnecessary engineering surprisingly expensive.

Scope-Aware Code Review

Engineering teams do not need a separate review bureaucracy for every AI-generated change.

They need a more explicit definition of completion.

Before implementation, a coding task should clarify four things.

1. What is actually required?

Specify the observable outcome.

Prefer concrete acceptance criteria over broad instructions such as "improve the reporting system" or "make this more scalable."

2. What is outside the scope?

State important exclusions.

If the task does not require a new service, a database migration, a dependency change, or a general-purpose framework, make that boundary explicit.

This does not prohibit justified architectural changes. It ensures that they are proposed deliberately.

3. What must remain unchanged?

Identify existing behavior, interfaces, security assumptions, compatibility requirements, and system boundaries.

A successful feature should not silently change unrelated functionality.

4. How will we verify the result?

Tests should cover required behavior, but review should also compare the final change with the original request.

Inspect unexpected files, dependencies, configuration changes, new interfaces, and unrelated refactoring.

Ask the agent to explain any expansion of scope.

And where a change affects important architecture or business behavior, keep the final decision with an accountable engineer.

These practices do not guarantee perfect results.

They make the intended boundary more visible before unnecessary work reaches the codebase.

Measure What Happens After the Pull Request

Scope alignment is fundamentally a requirements and engineering judgment problem.

A repository analytics tool cannot determine whether a product manager truly needed a CSV export, a new service, or a different architecture simply by inspecting the resulting commits.

But repository history can help answer a related question:

What happened to the engineering work after it was delivered?

This is where GitMe contributes another measurement perspective.

Real Effort Value (REV) estimates the baseline engineering effort represented by a contribution, rather than treating the number of lines or commits as a measure of productivity.

AI Effort Share helps identify how much of the observed work appears meaningfully AI-assisted.

AI Leverage adds a separate view of the relationship between modeled engineering output and modeled human effort.

Rework and refactoring signals help teams examine how contributions are corrected, revised, replaced, or restructured over time.

Effort Survival Cohort follows how much modeled engineering effort from earlier periods remains effective as the codebase evolves.

These measures are not a scope-compliance detector.

GitMe cannot determine from repository changes alone whether an additional abstraction was unnecessary, whether a feature matched its original requirements, or why an engineer later rewrote the code.

But the combination of modeled effort, AI participation, rework, and durability can help teams identify outcomes worth investigating.

Imagine a team reports increasing AI Leverage while also seeing more subsequent correction work and weaker effort survival.

That pattern would not prove that its agents are overengineering solutions.

It would justify asking whether task definitions, architectural boundaries, review practices, or validation processes need attention.

Conversely, strong AI Leverage accompanied by limited avoidable rework and durable output may indicate a workflow worth studying and sharing.

The important distinction is between measuring engineering outcomes and claiming to know their causes.

The Right Goal Is Not the Smallest Change

There is a danger in turning scope discipline into another simplistic metric.

A smaller pull request is not automatically better.

Sometimes the correct solution requires substantial refactoring, a new abstraction, or a broader architectural change.

And sometimes deleting existing code represents more meaningful engineering work than adding new functionality.

Engineering leaders should not reward coding agents merely for minimizing changed lines.

They should evaluate whether the implementation was appropriately scoped for the requirement and whether the resulting work remains useful.

That means comparing the planned change, the delivered behavior, the engineering effort involved, and the downstream consequences.

It also means resisting the temptation to convert any single signal into a universal productivity score.

AI can reduce implementation effort.

It cannot remove the need to decide what is worth implementing.

The Next Question for AI Engineering

As coding agents improve, the challenge is no longer limited to getting them to produce working code.

Engineering organizations also need to ensure that agents solve the intended problem without creating unnecessary work around it.

That requires better task boundaries, stronger review, explicit architectural constraints, and evidence from what happens after delivery.

The winning team will not necessarily be the one whose agents generate the most code or close the most tasks.

It will be the one that consistently turns AI assistance into necessary, maintainable, and durable engineering outcomes.

The best AI-generated code is not the most code, or even the cleanest code. It is the code that solves the right problem without creating a larger one.

GitMe helps engineering organizations examine the effort, AI Leverage, rework patterns, and durability of their contributions—making it easier to look beyond code generation and investigate the value that lasts.

Explore GitMe.

See What Lasts After AI Delivery

Explore modeled effort, AI participation, rework, and durability together.

Get Started

Related GitMe Reading

Sources

  1. Oukhay, I., Begoug, M., Chouchen, M., & Ouni, A. (2026). When AI Code Doesn't Stick: An Empirical Study on Reverted Changes Introduced by AI Coding Agents. Proceedings of the 23rd International Conference on Mining Software Repositories, 847–851. Conference abstract.
  2. Ehsani, R., Pathak, S., Rawal, S., Al Mujahid, A., Imran, M. M., & Chatterjee, P. (2026). Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub. Proceedings of MSR 2026.