Sriram Krishnan posted recently about his personal coding arc over 20 years: from writing a lot of code every day, to writing almost none, to now generating significant amounts through LLMs — building on Andrej Karpathy’s analysis of coding models. It’s a clean description of where enterprise development is heading. More code produced per developer, with a larger share of that code originating from AI tools rather than from human hands. The follow-on question is how testing has to evolve alongside that shift.
The answer is not “test more.” It’s “test differently, in the specific areas where AI tools fail differently than humans do.”
stateDiagram-v2 direction TB state "Assume AI output is correct" as Assume state "Unit tests pass, boundary cases untested" as Unit state "Integration gaps at component boundaries" as IntGap state "Production failure or security incident" as Fail state "Explicit interface definitions before generation" as Def state "Boundary tests added manually post-generation" as Bound state "Integration tests at AI-component boundaries" as Int state "Regression protocol for AI-modified existing code" as Reg state "Production-grade AI-assisted codebase" as Prod [*] --> Assume Assume --> Unit Unit --> IntGap IntGap --> Fail [*] --> Def Def --> Bound Bound --> Int Int --> Reg Reg --> Prod
The assumptions that change when AI writes the code
Traditional testing is built on a model of where bugs come from. Developers make logical errors when implementing complex requirements. They misunderstand edge cases in the specification. They introduce regressions when modifying existing code. They make precision failures.
AI-generated code fails differently. It fails with confident implementation of the wrong requirement. It fails by handling specified inputs correctly and unspecified inputs incorrectly. It fails by implementing plausible-looking patterns that work in isolation and break in combination. And it fails at the boundaries — between the AI-generated component and the system it connects to — because the AI’s understanding of the surrounding system is limited to whatever was in the context window.
Testing practices designed for human failure modes catch the failures humans make. They need to be extended for the failures AI makes — which are predictable enough to address systematically.
What traditional test coverage misses in AI-generated code
Boundary and edge case coverage. Human developers often write unit tests that cover the cases they thought of while writing the code. AI tools generate code for the cases described in the prompt. Both leave the same gap: the cases nobody thought to specify. AI-generated code has this gap structurally, because the prompt defines the test scope as surely as it defines the implementation scope. If the prompt didn’t describe the edge case, the implementation likely doesn’t handle it, and the test likely doesn’t cover it.
Integration boundary testing. AI-generated components are usually developed and tested in isolation — the prompt describes the component’s behavior and the AI generates tests that verify that behavior in isolation. What the tests don’t verify is whether the component integrates correctly with the rest of the system: whether its input format matches what calling code produces, whether its output format matches what consuming code expects, whether its error behavior is compatible with the system’s error handling conventions.
Regression coverage for AI-modified existing code. When a human engineer modifies a component, they have context about what else might break. An AI tool modifying a component has only the context provided in the prompt. Regression testing for AI-assisted modifications needs to be more explicit — the AI may not know what it didn’t know to check.
What to add to an enterprise testing strategy
Three additions close most of the gap:
Explicit boundary testing. For every AI-generated function or component, the test suite should include cases for: null or empty inputs, inputs at the maximum expected size, inputs with unexpected character encoding, and inputs designed to be invalid in an adjacent system. These are the categories AI-generated code most often handles incorrectly, and they’re the categories most likely to surface in production without appearing in development.
Integration test coverage at AI component boundaries. Any interface between an AI-generated component and the existing system should have integration tests that verify the format, content, and error behavior of that interface — not just the behavior of each component independently. This is where the system-level failures accumulate.
A review gate for AI-modified existing code. When AI tools modify code that already exists — as opposed to generating new code — the diff should be reviewed specifically for regressions: which existing tests pass or fail after the change, and does the behavior change match the stated intent? This is a targeted check, not a full code review.
The Carvana case for architecture-driven testing
At Carvana — where a five-person team parsed millions of vehicle records daily through event-driven architecture — the testing strategy was architecture-driven. The event-driven pattern meant each component’s input and output were explicitly defined. The test suite verified those contracts, not just internal component behavior. When a parsing error slipped through, the event boundary testing caught it before it propagated downstream.
That architecture-driven approach maps well to AI-generated code. When each AI-generated component has an explicit interface definition — what it accepts, what it produces, what it does with malformed inputs — testing that interface is both straightforward and reliable. The AI’s implementation correctness can be verified through the interface. The AI’s edge case handling can be tested explicitly by the human who writes the test suite.
The investment in defining component interfaces before AI generation, rather than inferring them afterward, is what makes this work. It is architectural discipline applied to testing strategy, and the testing improvement is one of its secondary benefits.
The workflow that closes the gap
An enterprise testing workflow for AI-generated code:
- Before generation: define the component interface explicitly — inputs, outputs, error behavior, integration contracts.
- Generate the code and implementation tests via AI.
- Add boundary and edge case tests manually — the cases not described in the prompt.
- Run integration tests at the component boundary.
- For any AI-assisted modification to existing code: run regression tests and review the diff for unintended behavior changes.
Steps 3 and 5 are human steps. The AI doesn’t add them automatically. But they are predictable enough to be standardized and consistently applied — and they close the specific failure modes that AI-generated code most reliably introduces.
