# Review the Review: Across Both Loops, Methods as Team Consensus

The Bottleneck Moved

AI made writing code faster. Google Cloud’s Office of the CTO put it bluntly: once the production bottleneck disappeared, the constraint moved to review and integration—oversized PRs, tangled dependency chains, and approval fatigue that turns judgment into reflexive clicking.

Faster models will not fix this. The constraint is no longer “can we generate?” It is “who proves a change is safe to merge—and by what method?”

For teams, hiring more human reviewers will not close the gap. Humans have a ceiling on reading diffs; AI output volume punches through that ceiling. Keeping the old assumption—“a human inspects every line of every artifact”—burns both speed and quality.

Thesis: Review the Review, Not Every Artifact

We already covered HITL / HOTL / HOOL: oversight should route by risk, not sit in front of every step. Addy Osmani pushes the same line further—the inner loop (investigate, implement, parts of verify) can be hosted by scripts and agents; humans should own the outer loop: constraints, sampling, audit, and accountability.

Easy mistake: review is not the same thing as the outer loop. Review is end-to-end—the inner loop can and should run automated review too. After an Apply, a file write, or a self-test round, immediately run lint, tests, and an independent AI evaluator so gates fire inside the loop, without promoting every check into a human approval.

A clearer split:

Layer	What review does	Who does it
Inner loop	Post-action automated verification: script gates, AI spec/risk checks, fail-and-retry	Scripts + AI (hostable)
Outer loop	Agree how we review: routing, thresholds, sampling, audit, accountability; audit whether review methods drift and correct them	Team consensus (titles may blur; the team must stay clear)

In AI4SE, job titles will keep blurring; the team stays clear. The outer loop is not a private duty attached to one title. It is the team’s shared contract for how we prove a change is safe to merge—not a claim that every review action may only happen outside the loop.

Applied to review:

Push repeatable, decidable review down into the inner loop—scripts and AI. In the outer loop, what the team primarily reviews is whether those review methods themselves are sound. When they are not, fix gates, rules, and evaluators together—do not rely on one person to re-read every large diff.

This does not conflict with accountability stays with humans. Final sign-off and incident ownership remain human. What changes is which layer human attention buys. Osmani’s warning still holds: same-family models rubber-stamping each other is often borrowed confidence—a closed loop can be very sure and still wrong.

Objects Widened: From Code to Artifacts

In classical team practice, “review” usually meant code review. Under AI4SE / SDD, the review surface widens.

In OpenSpec-style flows, a Change ships as proposal, specs, design, and tasks—mostly AI-drafted. The lean move is to review before Spec Ready, not reconstruct intent after thousands of lines land. You still review before merge; and the end-to-end User Harness (rules, skills, directory conventions, permission boundaries) drifts too, so it also needs review.

This matches spec as the source of truth: when agents generate most code, the human-readable anchor moves up to the spec. Bryan Finster sharpens it—if review is the first time you discover whether the implementation is correct, you never had a real quality process; AI only made the collapse impossible to ignore.

Review object	What the team should care about together
Proposal / Spec / Design / Tasks	Is intent, scope, and non-goals clear enough to serve as an acceptance anchor?
Code diff / PR	Checked against the spec? Still reviewable in size? Evidence (tests / risk) present?
User Harness	Are Guides/Sensors stale, conflicting, or bypassed by agents?

See Harness Engineering.

Timing Multiplied: Review Across the E2E Path

Review is not a single-phase job for a single role. It is a set of gates across delivery:

1. After Spec Ready — agree on what to build before agents Apply
2. After Apply, before merge — spec check + quality scan + human judgment (see Review Pipeline)
3. Entropy management in flight — context drift, rule bloat, stale skills, directory decay also need review and harness write-back

More moments do not mean more human appearances. The opposite: the more gates you need, the more deterministic checks must sink into inner-loop automation, so outer-loop human attention stays on high blast-radius decisions and method correction.

Methods Layered: Scripts → AI → Humans

Automation plus humans remains true; what changed is a dedicated AI review layer—and the order should be intentional:

Deterministic rules / scripts  →  Independent AI evaluator  →  High-leverage human judgment
         ↑ hostable in inner loop ↑                    ↑ outer sample / sign-off ↑

• Scripts first (often inner): naming, formatting, dependency policy, forbidden calls, green tests, PRs must link a spec—these are crystallized human experience and should be scripted. Coding agents can help write the scripts; the team still defines rules and thresholds in the outer loop.
• AI next (also often inner): architectural smell, spec-alignment summaries, risk-surface scans, missing-test hints—checks that are reason-able but hard to hard-code, and that can run automatically after actions without waiting for a human click.
• Humans last (leans outer): Is this even the right change? Are the trade-offs acceptable? What about the gap nobody put in the spec? Can we sign accountability for merge? Plus sampling audits of what the inner automated review concluded.

An independent Evaluator is not optional. The same Generator with a swapped prompt is not a second perspective—see Planner / Generator / Evaluator.

The Review Pipeline post covers the three steps on a PR. This piece is the team view: audit together whether those layers are misconfigured or silently failing—especially whether inner automated review is still trustworthy.

What the Outer Loop Owns: What to Audit, How to Correct

What to audit (the review system itself)

Signal	Likely drift
AI review stays “all green” and never correlates with incidents	Under-reporting; prompts/rules too loose or missing real failure modes
Humans still leave style nits and the same comments	Deterministic experience still stuck in the human queue
Many gates, yet high-risk items get instant approve	Approval fatigue; routing not risk-layered
Generator and Evaluator share origin / session	Self-grade bias; borrowed confidence
Spec gates are theater; huge diffs go straight to PR	Object/timing design failed; humans forced back to line-by-line reading
Harness only grows; agent behavior gets noisier	Entropy unmanaged; Sensors never feed Guides

What to do when it is wrong (corrective moves)

1. Re-route — low risk → HOOL/HOTL; irreversible / compliance → HITL; cut meaningless micro-approvals.
2. Strengthen specs / scenarios — write recurring misses into specs or acceptance scenarios so scripts and AI have anchors.
3. Shrink size and demand evidence — enforce small batches; PRs carry spec links, risk notes, and test evidence—not naked diffs.
4. Separate the Evaluator — different role, model family, or at least different system constraints; ban self-LGTM.
5. Codify every catch — each deterministic pattern a human catches becomes a script, CI rule, or skill so humans stop reviewing the same class.
6. Sample and audit — the team does not read every artifact, but shares a sampling contract: full check on high-risk paths, spot-check inner automated review conclusions on low-risk work, and post-mortems on “automation should have caught this.”

In one line: the inner loop can review automatically; the outer loop reviews whether those automated methods are sound. When the method is wrong, fix the system—do not add more meatware.

Anti-Patterns and Minimal Actions

Anti-patterns

• The same agent generates and reviews, then calls it an automated pass
• Treating review as “outer-only”—no automated verification in the inner loop, every check piled onto humans
• Pride in reading every large diff line by line—pushing method governance back into human scanning of the inner loop
• Bolting on an AI review bot without changing gates, PR contracts, or Evaluator independence
• Never reviewing the User Harness; rules and skills only accumulate
• Treating “tests are green” as a substitute for “this is the right thing to build”
• Defaulting the outer loop to one title while everyone else only clicks Approve

Three things this week

1. Inventory the review gates that actually fire: which run automatically in the inner loop, which need human sign-off in the outer—and mark which layer is carrying weight it should not
2. Name an Evaluator independent of the Generator (human, agent, or both) and write what it checks against—as team consensus, not personal preference
3. Install a short feedback loop: escaped defect or missed review → classify → fix script/rule/spec (write back into the inner loop) → sample next week for recurrence

Generation will keep getting faster. A team’s leverage is not finishing every AI artifact—it is making the end-to-end review system itself more accurate over time: the inner loop runs often; the outer loop decides rightly; and both stay shared practice rather than one role’s solo act.