AI code review: Every pull request checked before a person reads it
- Platform
- Agentic workflows
- 2026
- User value
- A reviewer’s time spent on the questions only a person can answer.
- Business value
- Code review that keeps pace with AI-written code.
- Stack
- Claude Code
- Codex
- GitHub Actions
- CODEOWNERS
- Related case studies
Problem
Most of our code is now written by an AI coding agent, and it ships faster than people can review it. Every PR arrived in a different shape, so reviewers spent their time on routine checks. Our goal was a review where a person decides only what a person has to: is this the right change, and is it safe to ship?
Design challenge: deciding what a machine can judge and what needs a person. If merging past a failed check is never right, the check is required. If it depends on the change, the agent flags it and a person decides.
What I did
- Started with an audit. I mapped how PRs and deploys moved, then designed the ideal flow and worked back to what we’d need to build. The team agreed on it in one review meeting.
- Put the rules where agents read them. Each repository’s AGENTS.md, the machine-readable file agents read first, holds one set of review rules for design, engineering, and QA. Every rule reads “when a PR changes X, verify Y,” has a priority from P0 to P3, and names a past PR it should have flagged, so it can be tested.
- Gave every PR the same shape. Intent, changes, related PRs, test plan, findings, validation, risk, and how to revert. The agentic PR workflow fills it in.
- Gave every PR a preview. Once its checks pass, a new PR claims one of our testbeds in dev, and its preview link can go to anyone in the company. Leadership and other teams try a feature before it merges, so testing reaches well past engineering and QA. Testbeds unlink after four or five days, and a full set can be replaced by hand.
- Capped agent review at three rounds. After that, a teammate is asked whether the approach is right. That ends fix loops and catches a wrong approach early.
| Priority | Finding | Outcome |
|---|---|---|
| P1 | Changing the ID in the URL loads another organization’s records | Fixed in 4e1c9a2 |
| P2 | Chart labels fall below 3:1 contrast in dark mode | Fixed in 7b20d5f |
| P2 | A failed save shows no error | Fixed in 7b20d5f |
| P3 | A new helper repeats one that already exists | Deferred to a follow-up ticket |
| P3 | The retry can run after the request is cancelled | Won’t fix: the request is safe to repeat |
Outcome
The pilot is running on two repositories. It’s measured on review effort (human comments, wait time, and rounds per PR), with reverts and hotfixes as the guardrail: if those increase, the work moved instead of shrinking.
Who did what: I designed the workflow, wrote the design rules and the PR template, and merged the team’s review guides into one set of rules per repository. I came up with the testbeds and how they work, and an engineer built them.
What’s next: the review loop is the first of three phases. Next is a release pipeline: each release candidate is frozen on stage while QA tests it, and production takes only the build QA approved. After that, we measure review effort and escaped bugs and tune the rules.