The first version reviewed like an intern who had read every style guide and understood none of the code. It commented on everything — a variable name, a missing trailing comma, a formatting nit a linter already owned — with exactly the same confidence it brought to a real bug. Engineers muted it within a week. That failure handed me the whole thesis: at scale, code review quality is a filtering problem, not a generation problem. Anyone can make a model say something about a diff. The hard part is making it stay quiet.
Today that system runs automated review across 266 repositories for an organization of more than 500 engineers. It costs roughly a cent per review and surfaces three to five genuinely critical findings per hundred pull requests. None of those numbers came for free, and most of the engineering went into deciding what not to say.
Why the ready-made tools didn't survive contact
Before building anything, we tried existing AI review tools. On a small team they demo well. Inside a 500-engineer org they quietly died, and the reason had almost nothing to do with the model.
The failure was a process problem. Adoption was uneven, so the signal was inconsistent. Feedback never made it back to whoever owned the tool, so nobody could tell whether it was helping. Without that loop there was no way to prove value, and a tool nobody can prove is working is a tool that gets switched off at the first budget review. The model was never the bottleneck. The bottleneck was the absence of a system around it — measurement, feedback, and the discipline to act on both.
So the real product was never "an AI that reviews code." It was the loop that keeps such a thing honest.
Signal design: earning the right to be read
The default failure mode of an LLM reviewer is verbosity weighted by nothing. It comments in proportion to how much it has to say, not how much the comment is worth. And the economics of that are brutal: one bad comment costs ten good ones. Engineers mute fast and they never un-mute. So the first design decision was a severity gate — comment only above a confidence and severity threshold, and stay silent otherwise. Silence is a feature. A reviewer that usually says nothing earns the right to be read when it finally speaks.
What deserves a comment is the blocking class of problem: breaking changes, risky migrations, security issues, and the review blind spots humans skim past. In practice the highest-value catches are unglamorous — a typing error that would surface at runtime, a connection or dependency that isn't wired up, a class that won't resolve. The kind of thing a tired reviewer approves at 6pm and an incident finds at 2am.
None of that works on the diff alone. A diff tells you what changed, not what it means. So the model sees more than the patch:
- Product and system documentation for the repo.
- A semantically indexed knowledge base over the entire codebase, so a change can be judged against code it never touches directly.
- The organization's CI rules and conventions.
- Past incident reports — so a pattern that once caused an outage is recognized the next time it shows up.
- Jira acceptance-criteria definitions, so "does this actually do what was asked" is a question the reviewer can hold.
With that context and a strict gate, a typical pull request draws a handful of comments, not a wall of them — roughly one to five critical, a few medium, a few low. The point of the gate isn't to comment less for its own sake. It's that every comment left standing has already survived a filter, which is the only reason anyone reads the next one.
Cost is an architecture constraint, not an afterthought
At 266 repositories, per-review cost multiplied by pull-request volume is real money every month. Cost had to be a first-class design constraint, not something to optimize once the bill arrived.
The architecture got there by specializing. It began simple — a frontier model as the primary reviewer with a second as fallback — then moved to running reviews in parallel, and settled on six specialized sub-agents, each responsible for a slice of the problem rather than one generalist trying to hold everything at once. Specialization is what makes a penny-per-review number defensible: each agent does a narrower job on a tighter context, and the expensive reasoning is spent only where it changes the outcome.
Every review is traced. Which PR, which agent, how much context, what it cost — all of it lands in CloudWatch, so cost is observable per pull request rather than a lump sum discovered at the end of the month. You cannot hold a cost target you cannot see, and "about a cent per review" is a number we can point at because every review is individually accounted for.
Rolling out to 266 repositories
An org-wide tool lives or dies on setup friction. Anything that requires per-team effort dies in the long tail, because the long tail never gets around to it. So zero-per-repo setup was a design goal: the default configuration had to be good enough that a repository got useful review the day it was switched on, with no work from its team.
Customization, where it exists, is deliberate and centralized. Each repository can carry a review.md under a prompts/ directory holding repo- and product-specific direction; beyond that, there's no bespoke per-repo tuning to maintain. It's enough to specialize the review without creating 266 little configurations that rot.
The rollout itself was staged, not announced-and-abandoned. A first proof of concept ran about a month across ten repositories and three teams. A second, roughly a month and a half, widened it. Only then did it go to full coverage — and from that point it ran on a weekly cadence of feedback, fixes, and improvements rather than a launch followed by silence.
Measuring whether it actually works
The headline metric is three to five critical findings per hundred pull requests — deliberately "per hundred PRs," not "per PR," because at review volume you care about the rate at which the system earns its place, not what any single review produced.
A finding only counts once a human confirms it. For high-severity, blocking findings, that validation is explicit: an engineer looks and agrees it was real. On the softer end, resolution behavior is the signal — comments that get acted on versus dismissed. Between a quarter and a half of surfaced comments turn out to be genuinely actionable, which sounds modest until you remember the alternative is a bot everyone muted.
I'll be honest about the ceiling: the most valuable thing this system does is invisible. A prevented incident leaves no trace. We can measure resolution rates and validated findings; we cannot directly measure the outage that never happened. Some of the value is real and permanently unquantifiable, and it's worth saying so out loud rather than inventing a number for it.
False positives are the real product surface
The model was never the ongoing work. False-positive management is. A reviewer that cries wolf is worse than no reviewer, so the tuning loop is the actual product, and for a long time it was relentless — several times a day at first, as patterns surfaced and got corrected. It has since settled into a weekly recurring rhythm, which is its own kind of success: the system stopped being an emergency.
The loop is boring in exactly the way durable systems are boring. Usage produces feedback — through a Slack helpline channel and a feedback sheet. A responsible owner triages it. A fix goes out as a pull request, gets reviewed and tested like anything else, and ships to every user through versioned plugins. Nothing about that is clever. It's just closed. And "closed" is the entire difference between the tool we built and the ready-made ones that didn't survive their first month.
What it comes down to
Useful AI code review isn't a better prompt. It's a filter with a feedback loop wrapped around it: decide what deserves a comment, make silence the default, put real context in front of the model, price every review so cost stays honest, and never stop closing the loop on what it gets wrong. The generation was always the easy part. Everything that made it work was the restraint around it.
Salimcan Venedik is a Staff Software Engineer and Technical Lead at Insider, where he works on messaging infrastructure and AI product engineering. — About / Contact