Field notes — the blog

Notes from inside the system.

Specific write-ups on decisions that moved the needle: architecture trade-offs, delivery pipelines, and leading engineers who are better than you at their part of it.

              {{ totalPosts }} notes
              {{ totalMinutes }} min total
              Monthly
            

/

                Latest
                {{ featured.date }}
                {{ featured.cat }}
                {{ featured.read }}
              

{{ featured.title }}

{{ featured.excerpt }}

Read the note →

// before → after
{{ row.text }}

{{ listHeading }}

{{ resultCount }}

                {{ post.cat }}
                {{ post.date }}
              

{{ post.title }}

{{ post.excerpt }}

{{ post.read }}

Nothing matches that.

Try a different term, or clear the filters.

The architect's dispatch

One note a month. Nothing else.

A single deep write-up on a system I worked on: the trade-off, the numbers, what I'd do differently.

            {{ article.cat }}
            {{ article.date }}
            {{ article.read }}
          

{{ article.title }}

{{ article.excerpt }}

{{ part.text }}{{ part.text }}{{ part.text }}

{{ b.alt }} — {{ b.caption }}

{{ b.label }}

{{ h.text }}

{{ cell.text }}

{{ b.label }}

{{ link.label }} {{ link.note }} {{ link.host }}

{{ b.caption }}

{{ b.text }}

{{ b.text }}

{{ b.label }}

{{ line.text }}

{{ item.no }}

{{ part.text }}

{{ part.text }}

{{ part.text }}

Takeaways

▖ {{ t }}

Tags

{{ tag }}

← Newer

{{ prevPost.title }}

Older →

{{ nextPost.title }}

Related notes

{{ post.cat }}

{{ post.title }}

{{ post.read }}

Wire an AI reviewer into CI without it becoming noise

AI · Aug 2026 · 15 min read

The naive version comments on everything and gets muted in a fortnight. Scoping to the diff, structured verdicts, a confidence gate, and measuring the signal ratio.

The naive version of this takes an afternoon: post the diff, ask for a review, write the response as a pull request comment. It works for about two weeks. Then people stop reading it, because eight of every ten comments are style opinions or confident nonsense, and a reviewer nobody reads is worse than no reviewer — it is a reviewer that makes the real comments harder to find.

This is the version that survived. The core idea: constrain what it is allowed to say, make it justify each finding in a structure you can filter, and measure whether the output is worth reading.

Step 1: give it the diff and the context around it

Sending the raw diff produces comments about code that is fine, because a diff hides the thing it changed. Send the changed hunks plus enough surrounding file for the change to make sense, and skip the files where review has no value.

git diff origin/main...HEAD --unified=25 \
  -- ':!*.lock' ':!*.min.*' ':!public/build/**' \
     ':!**/__snapshots__/**' ':!*.svg' > diff.txt

# 25 lines of context, not 3. the model cannot review a change
# it cannot see the function signature for.

# bail out on the changes where this is a waste of money:
LINES=$(wc -l < diff.txt)
[ "$LINES" -gt 3000 ] && { echo 'too large, skipping'; exit 0; }
[ "$LINES" -lt 10 ]   && { echo 'trivial, skipping';   exit 0; }

Add the repository conventions file to the prompt as well. If the model does not know you use repositories rather than facades in domain code, it will suggest facades, and every such comment costs you a little more of the team's willingness to read the next one.

Step 2: tell it what it may not comment on

This is the highest-leverage part of the whole exercise. A general request for a review returns a general review: naming, formatting, and suggestions to add comments. Your linter already owns those, and the overlap is exactly what trains people to skim.

You are reviewing a diff. Report ONLY:
  - logic that is wrong for a stated input
  - unhandled error or null cases on a reachable path
  - concurrency: races, non-idempotent handlers, lost updates
  - security: injection, authz gaps, secrets, unsafe deserialisation
  - N+1 queries and unbounded loops over external calls

Do NOT report: naming, formatting, comment density, test style,
missing docblocks, or anything a linter enforces.

For each finding you MUST give a concrete failure scenario:
specific inputs -> the wrong output or the crash.
If you cannot construct one, do not report the finding.

That last instruction does more work than everything above it. Requiring a concrete failure scenario is a filter the model has to apply to itself, and a large share of plausible-sounding findings simply cannot survive it. It is the same discipline you would want from a human reviewer.

Step 3: demand structure, then filter it

Free-form prose cannot be filtered, ranked, or measured. Ask for structured output and you can enforce a confidence gate mechanically, which is what keeps the comment count down without you having to trust the model's own sense of proportion.

{
  "findings": [{
    "file": "src/Billing/Internal/LedgerPoster.php",
    "line": 41,
    "category": "concurrency",
    "severity": "high",
    "confidence": 0.9,
    "summary": "balance read outside the transaction",
    "failure_scenario": "two concurrent posts both read 100,", 
       "both write 90; one debit is lost"
  }]
}

# then, in the job:
jq '[.findings[] | select(.confidence >= .8)] | .[0:5]'
# max five comments per PR. a wall of comments is not read.

Step 4: advisory, never blocking

Do not fail the build on this. A false positive that blocks a merge produces an override habit within a week, and once people are overriding by reflex they are overriding your real gates too. Post the findings, let a human decide, and keep the mechanical gates for things that are mechanically true.

- name: AI review (advisory)
  continue-on-error: true          # never blocks the merge
  run: |
    ./scripts/collect-diff.sh
    ./scripts/review.sh diff.txt > findings.json
    ./scripts/post-comments.sh findings.json

# post as review comments on the exact line, not one summary
# blob. and prefix every one so nobody mistakes it for a human:
#   **[ai · concurrency · 0.9]** balance read outside the ...

# and re-run on force-push by deleting the previous bot
# comments first, or the thread doubles every push.

Step 5: measure whether it is worth keeping

This is the step everybody skips, and it is the only one that tells you if the thing works. Add two reactions to every bot comment and count them monthly. You now have a signal ratio, and a signal ratio makes the decision for you.

month     comments   acted on   dismissed   ratio
month 1        84         11         73      13%   <- bad
month 2        31         14         17      45%
month 3        22         15          7      68%   <- keep

# month 1 -> 2 was one change: the do-not-report list.
# month 2 -> 3 was the confidence gate and the cap of five.

# below ~40%, people stop reading. that is the number that
# decides whether you tune it or turn it off.

Fewer comments, each of which is worth reading, beats complete coverage nobody looks at. Precision is the only metric here.

What it is genuinely good at, and what it is not

Good: spotting the error case nobody handled on a path the tests do not cover, and noticing the second write that makes a handler non-idempotent.
Good: catching N+1 patterns in code that reads perfectly well line by line.
Not good: knowing whether a trade-off is right for your team. It does not know your on-call rotation is one person.
Not good: anything requiring history. It cannot know this module is being deleted next sprint, so weigh its findings accordingly.

One thing worth being clear about internally: this does not replace review, and saying it does is how you get a team that stops reading diffs. It is a cheap second pass that is unusually good at a narrow band of mistakes — the ones a tired human skims past at the end of a long pull request. Position it that way and it earns its place. Position it as a reviewer and it will not.

Takeaways

Send changed hunks with generous context plus your conventions file, and skip trivial or enormous diffs entirely.
Enumerate what it may not report; the do-not-report list matters more than the prompt.
Require a concrete failure scenario per finding, demand structured output, gate on confidence, and cap at five comments.
Keep it advisory, and track the acted-on ratio monthly — below about 40% people stop reading it.

All notes · Shehzad Aslam