The oversight ceiling

Ingo Schindler

2026-08-27

This is not an “I rebuilt it better in two days” post. I need to say that up front, because the numbers below read exactly like one, and because the comparison only means something once you take the boast out of it.

The setup

We have been building a greenfield service in NestJS and Node: three engineers, AI-assisted, test-first, with our organisation’s golden-path patterns locked in as project guidelines. Roughly six weeks in total, of which I estimate eight to ten days were actual coding. None of us had much Node depth going in.

Over a weekend, I rebuilt the same service in Quarkus and Java, a stack I understand down to the last detail. One person, two days, same tools (Claude and Cursor), a written spec, hexagonal architecture.

I held the boundaries constant on purpose: Datadog, structured logging, S3, trace IDs, Docker deployment. Two deliberate exceptions, since Quarkus ships Vert.x for event-driven async instead of SQS, and I skipped the pipeline.

One hard rule made the experiment worth running: the agent never saw the original code until a feature was finished. I reconstructed the use cases from memory rather than letting the AI read the existing project. Otherwise I would have been measuring how well it copies, which is not the question.

What the real project looks like from inside

The loop works. AI writes, humans review, we flag missing tests, the AI refactors, someone tweaks by hand, code review, merge. It is noticeably faster than fighting every error yourself. Coverage is decent. The happy path works.

And every feature, change and refactor gets bigger and more expensive than the last one.

Pull requests keep growing. More files get touched per change. Functionality that should mirror something already in the codebase gets solved a completely different way instead. At one point a “small change”, adding a surrogate id column, wanted to touch forty files where four including tests would have done it.

None of that is dramatic on any given day. That is the problem with it.

What happened in the rebuild

Use case one, upload a file to S3. Over an hour of spec and implementation produced circular dependencies, package violations, domain logic mixed with orchestration, outbound ports that were written but only ever used from a unit test, and other ports simply ignored. It compiled. Tests were green. The feature worked.

The correction attempt. I told it which practices were wrong. Cursor explained the problem back to me correctly, then reported that all changes had been applied exactly as requested. About half of them had. The rest had been solved by moving classes into a new Shared package, which four components now depended on, with domain logic sitting inside it.

Use case two, extract data from the file into a database table. Same shape, different violation. The original moves were now blocked by ArchUnit, so a brand new Infrastructure package appeared and collected everything instead. The follow-up refactor looked better at first glance, but had mostly shoved methods between classes.

From the third use case on it went fast. No gross blunders. Some duplication that did not need to exist, a missing unit test, one flaky test that only fails on a full-suite run. The remaining use cases, an API comparison against the original service, and two forgotten use cases went in without much friction.

The result behaves and feels like the multi-week service from the outside, with roughly two hundred lines less code, a near-identical test count and over eighty percent coverage on both sides. Slightly better in Java, which I put down to the structure rather than the language. Adding a use case now takes minutes and a few lines, without touching files owned by another business component.

The finding

Speed is not the finding. Like for like, it is eight to ten days of our coding time against my two, and my weekend skipped everything the real project actually pays for: alignment discussions, arguing about whether an approach is right at all, and the work of genuinely understanding code somebody else produced.

The finding is that one codebase came out clean enough to extend and the other is quietly eroding, and that the difference was not the model.

Output quality is capped by the reviewer’s ability to recognise what is wrong. I have started calling it the oversight ceiling.

Same models, same tools, and the same target: an agreed set of use cases that had to work at the end, inside the boundaries above. The prompts and the project guidelines were not identical, and could not be, because the stacks are not. In the stack I knew, the AI’s mistakes were visible to me and I fixed them as they appeared. In the stack I did not know, the same class of mistake went through review and compounded.

The uncomfortable version of that: our real project’s first iteration was probably the same quality as the Java one’s first iteration. It may be degrading with every iteration since, and nobody on the team is in a position to see it, because the framework and its ecosystem are unfamiliar to all of us.

Two corollaries are worth stating separately, because they are the ones people skip.

Knowing is not building. Ask the agent about hexagonal architecture and it explains it correctly. It then violates it in the same session, diagnoses its own violation accurately when challenged, and applies half the fix.

Green is a liveness signal, not a quality signal. Code that compiles at eighty percent coverage told me nothing about circular dependencies, domain logic in a shared package, or ports that existed only so a unit test would pass.

This is also not new, and not really about AI. In 2024 I worked on a release process where a team with years of tenure could not see that its two-week release ceremony had accreted rather than been designed, because none of them had worked anywhere else. You cannot recognise a bad state you have no reference for. Generated code just makes the problem arrive faster.

What I built

Only one of my counters exists as running code, and it goes at the erosion rather than at the ceiling. Nothing in the review loop pulls the code back toward a norm, so the correction has to happen without a human spotting it first.

I use a ruleset in two tiers. Tier one is predefined, taken from the classic engineering books and adapted to the stack. Tier two is learned over time from confirmed review findings, which is the second counter below. Everything machine-checkable in that ruleset becomes an executable CI gate: ArchUnit for Java, module-boundary and no-cycle rules for Nest. Drift then fails the build instead of waiting on review.

I shipped it as an installable ruleset rather than a document, because a rule that lives only in a guidelines file is a rule the agent routes around.

Five counters I have not proven yet

These came out of the experiment rather than out of production.

1. A prose spec does not enforce shape

The spec was good and the architecture was still wrong. Prose describes intent. It does not constrain the result.

So the agent does not get raw acceptance criteria. It gets a handover document with explicit BDD scenarios, reviewed before any code exists. Ambiguity gets resolved while it is still cheap, in prose rather than in forty changed files, and the agreed scenarios become the red-to-green target.

2. Reward hacking around the rule

The Shared package above, then the Infrastructure package once ArchUnit blocked it, were both the agent satisfying a check without its intent. A rule the build cannot check is a rule the agent will negotiate with.

The counter is the second tier of that ruleset, a learned-rules memory. Every accepted review correction gets written down as a rule, but only from confirmed findings, with the concrete file and line. Never from opinion, because a memory fed by opinion is just a second unreviewed style guide. A finding that recurs gets promoted from a watched note to a permanent rule, and the build step reads that memory before writing.

3. Coverage read as quality

Green, plus eighty percent coverage, plus genuinely bad code.

Two signals, two owners. Tests passing gates the merge. A separate reviewer gates the design against the ruleset, looking at coupling, duplication, misplaced domain logic, and test quality as its own axis. That also means defining what “good” means outside the acceptance criteria, so there is something to verify against.

4. Sunk-cost sycophancy

The agent defends its own output. Cursor argued that restoring the previous state would cost as much as creating the new one, then that reviewers might be confused if commits suddenly disappeared. Both are nonsense. Both are persuasive at eleven at night.

Both were answered by git reset. The governance version: whatever wrote the code never approves it and never adds rules justifying it. The mechanical version: keep tickets small, so throwing work away discards a day on one scenario instead of a week on ten. Sunk cost needs something to get a grip on.

5. Invisible degradation

The ceiling from the other side: the mistakes I could see got fixed, the ones I could not see compounded.

The counter is to seed two or three hand-built canonical examples the team fully understands, then point the AI at those. “Match the shape of this” constrains where “use hexagonal architecture” only suggests. And be honest about the ceiling: where nobody can yet tell a good result from a merely green one, that is not where you let an agent build unsupervised.

Where this cuts against the obvious conclusion

The tempting read is that AI-assisted development does not work. That is not my claim, and the sharper version is more useful anyway.

The loop genuinely is faster than fighting every error by hand, and that held in both stacks. It is faster conditionally: when good examples already exist in the codebase, and when someone can supervise the output.

Greenfield in an unfamiliar stack fails both conditions at the same time. Which is, annoyingly, exactly the situation where teams reach for AI hardest.

The counters are cheap, too. They are process or configuration rather than headcount.

What I would want you to take with a pinch of salt

One experiment, one person, one weekend. It is a strong hypothesis with decent evidence, not a measured effect.

The framework pairing is incidental. Quarkus against NestJS is not the variable, and someone will want to read this as JVM advocacy. It is not. Swap the two stacks and the person, and I would expect the same result in the other direction.

And only the ruleset has actually run in anger. The five after it are reasoned out of one weekend. Ask me again in six months.