Why 'It Compiles' Is Not Verification
Last week, I spent a solid six hours hunting a bug that, technically, didn’t exist. The task was a classic piece of technical debt cleanup: refactor an ancient, gnarled JavaScript function called legacy_billing.js. It was a monster, full of nested if statements and weird edge cases for our tiered enterprise pricing. A perfect job for an AI coding agent. I fed it the file, the product spec for our pricing tiers, and asked it to rewrite it as a clean, modern, testable calculateTieredPrice function.
In less than five minutes, it handed back a masterpiece. The code was elegant, readable, and used modern syntax. It was beautiful. Even better, it came with its own Jest test file, calculateTieredPrice.test.js. I opened the test file. It created a sample user, assigned them to our “Starter” pricing tier, ran the calculation, and asserted that the result was correct. I ran the test. PASS. Green checkmarks everywhere.
It compiled. It passed its test. The code looked a thousand times better. It was late on a Thursday, so I did what any sane engineer would do: I approved the PR and deployed it. The next morning, our finance lead sent me a one-line Slack message: “Hey, is something broken? The new enterprise invoices are all way too low.” My stomach dropped. I dove into the logs and, sure enough, he was right. The agent had perfectly implemented the logic for our simple Tier 1 “Starter” plan. But it had completely misunderstood the complex, compounding logic for our Tier 2 “Growth” and Tier 3 “Enterprise” plans. It was just applying the base rate to everyone. The code was perfect, clean, and tested. And it was silently vaporizing thousands of dollars of revenue.
The Self-Grading Student Problem
This is the new, insidious failure mode of AI-assisted development. For the last couple of years, the big complaint was AI generating code that just didn’t work. As one developer on Hacker News, v_CodeSentinal, put it, “I built Autonoma because I was tired of Copilot suggesting code that didn’t compile.” We got mad, the models got better, and now, more often than not, the code does compile. But we haven’t solved the problem; we’ve just moved it upstream to a place where it’s far more dangerous.
We’ve moved from obvious syntax errors to subtle logical flaws. The new problem is that the agent now helpfully provides the proof that its own flawed logic is correct. It writes the code, and then it writes the tests to validate that code. It’s a perfectly circular, self-referential system of delusion. As one commenter on a recent HN thread pointed out, the core issue is “Testing functions that have been created by the LLM for the test to make it pass.”
The agent isn’t just the student doing the homework anymore; it’s also the one grading it. I saw this captured perfectly in a draft for a social media post on Reddit: “The agent writes both the code AND the tests - it’s like a student writing the exam and grading it.” They’re going to give themselves an A+ every single time, even if the answer is fundamentally wrong.
Why This Happens
This isn’t because the AI is lazy or trying to deceive you. It’s a structural problem rooted in what a large language model actually is. An LLM’s entire worldview is based on statistical probability. Its goal is not to understand your business logic; its goal is to generate a sequence of text that is the most plausible continuation of the text it has already seen.
When you ask it to write a function and then write a test for it, it doesn’t think, “How can I adversarially probe this code for weaknesses based on the user’s original intent?” It thinks, “What does a test file that typically accompanies a code file like this look like?” The most statistically common type of test is a simple, happy-path unit test. So that’s what it gives you.
The code and the test are born from the same flawed understanding of the requirements. The test isn’t an independent verification of the code against the spec; it’s just a confirmation that the code is self-consistent with its own internal, possibly wrong, interpretation. The link back to the ground truth - the original requirement - is completely gone. Compilation verifies syntax. AI-written tests often just verify the AI’s own interpretation. Neither verifies that the code actually solves the original business problem.
Why Prompting Won’t Fix It
I know what you’re thinking. “Julian, you just needed a better prompt! You should have told it: ‘And write a comprehensive test suite with separate tests for each pricing tier, including edge cases for user upgrades and downgrades!’”
And yes, that would have been better. It might have even caught this specific bug. But relying on prompts to enforce correctness is like trying to build a skyscraper with sticky notes. You are using a conversational request to solve a structural problem. You’re fighting gravity.
First, you can’t prompt for what you don’t know. The whole point of a rigorous testing process is to uncover the unknown unknowns. If I have to anticipate every single failure mode in advance to write the perfect prompt, I’m doing all the hard cognitive work myself.
Second, when you ask an agent to “be more comprehensive,” it will just generate more text that looks like comprehensive tests. It might create three test cases instead of one, but they could all be based on the same flawed interpretation of the logic. You’re asking it to check its own work, but it lacks the external perspective to do so. It’s trapped inside its own context window. You can’t prompt your way out of a fundamentally circular process.
Tired: “Prompting the agent to write better tests for its own code.” Wired: “Forcing the agent’s code to pass tests derived independently from the original spec.”
The Fix
The solution isn’t a more clever prompt or a bigger model. It’s an embarrassingly simple principle that we, as an industry, have used for decades, but seem to have forgotten in our excitement over AI.
The fix is external, independent verification enforced by the system.
You have to break the circle. The code generation process and the test validation process must be separate, and both must be anchored to a single source of truth: the specification. The agent’s claim that it wrote working code is irrelevant. The agent’s claim that it wrote passing tests is irrelevant. The only thing that matters is when an independent process validates that the code fulfills the requirements of the spec, as proven by a set of tests that were also derived from the spec. You need a chain of evidence, not a circle of trust.
What This Looks Like in Practice
This is the entire philosophy we built Ceetrix on. We assume the agent will get the logic wrong. We assume its tests will be self-serving. So we built a system of automated gates that enforces traceability from requirement to code to test.
Let’s replay my billing engine disaster, but run it through Ceetrix. The work would start with a PRD in our Document Editor. That PRD would have three distinct, trackable requirements:
- REQ-TIER-1: Calculate correct pricing for “Starter” plan.
- REQ-TIER-2: Calculate correct compounding pricing for “Growth” plan.
- REQ-TIER-3: Calculate correct compounding pricing for “Enterprise” plan.
Thanks to Spec Chain Enforcement, these requirements are the non-negotiable contract for the job. The agent takes the task and, as before, generates the beautiful code that only works for REQ-TIER-1. It also generates a unit test for that first tier. Then, it attempts to mark the story as complete.
It fails instantly. The submission is blocked by our automated Gate System (G0-12). Specifically, the Coverage Checking gate fires. It looks at the contract (the three requirements) and the submitted work. It sees code and a test linked to REQ-TIER-1. But it sees a glaring hole for REQ-TIER-2 and REQ-TIER-3. There are no tests to prove those requirements have been met. The gate slams shut.
The agent is blocked. It cannot proceed. It is now structurally forced to generate tests for the other two tiers. It writes a test for the “Growth” plan, but because its code logic is wrong, the test fails. The gate system runs the tests as part of its validation, sees the failure, and rejects the submission again. The agent is now in a loop it can only escape by actually fixing the implementation to match the requirements defined in the spec.
This is the key. We use different Test Task Types (unit, integration, etc.) to ensure that tests are treated as first-class citizens derived from the spec, not as an afterthought to the code. The system’s Task Completion Evidence requirements demand proof of this linkage. We didn’t need a better prompt. We needed a better system. We needed a system that demands evidence, not just artifacts.
Have your say: What’s the most expensive bug you’ve seen that came from code that compiled perfectly and passed its own basic tests? I want to hear the war stories. And when you’re ready to stop hoping and start verifying, try Ceetrix.