Field notes — the blog

Notes from inside the system.

Specific write-ups on decisions that moved the needle: architecture trade-offs, delivery pipelines, and leading engineers who are better than you at their part of it.

              {{ totalPosts }} notes
              {{ totalMinutes }} min total
              Monthly
            

/

                Latest
                {{ featured.date }}
                {{ featured.cat }}
                {{ featured.read }}
              

{{ featured.title }}

{{ featured.excerpt }}

Read the note →

// before → after
{{ row.text }}

{{ listHeading }}

{{ resultCount }}

                {{ post.cat }}
                {{ post.date }}
              

{{ post.title }}

{{ post.excerpt }}

{{ post.read }}

Nothing matches that.

Try a different term, or clear the filters.

The architect's dispatch

One note a month. Nothing else.

A single deep write-up on a system I worked on: the trade-off, the numbers, what I'd do differently.

            {{ article.cat }}
            {{ article.date }}
            {{ article.read }}
          

{{ article.title }}

{{ article.excerpt }}

{{ part.text }}{{ part.text }}{{ part.text }}

{{ b.alt }} — {{ b.caption }}

{{ b.label }}

{{ h.text }}

{{ cell.text }}

{{ b.label }}

{{ link.label }} {{ link.note }} {{ link.host }}

{{ b.caption }}

{{ b.text }}

{{ b.text }}

{{ b.label }}

{{ line.text }}

{{ item.no }}

{{ part.text }}

{{ part.text }}

{{ part.text }}

Takeaways

▖ {{ t }}

Tags

{{ tag }}

← Newer

{{ prevPost.title }}

Older →

{{ nextPost.title }}

Related notes

{{ post.cat }}

{{ post.title }}

{{ post.read }}

Let a model write the tests, let mutation testing grade them

AI · Aug 2026 · 16 min read

An LLM will happily take you from 44% coverage to 91% in an afternoon, and catch nothing. The fix is not a better prompt — it is a scoring function the model cannot argue with.

Point a model at an untested service class and ask for tests. You will get them: forty of them, well named, green on the first run, and coverage moves from 44% to 91% before lunch. Then a regression ships anyway, and when you go back and read what was generated you find the reason — the tests call the code and assert that calling it did not throw.

This is not a prompting problem, and you cannot fix it by asking more nicely. Coverage is the only signal in the loop, coverage rewards the execution of a line, and the model is optimising exactly what you measured. The fix is to change what you measure. Mutation testing is the scoring function that works here, because it asks the one question coverage cannot: if I break this line, does anything turn red?

What follows is the loop I run on legacy PHP services — Infection as the judge, a model as the generator, and CI as the ratchet. The language is incidental; the shape transfers to Stryker for JS or TS and mutmut for Python.

Why coverage is the wrong scoring function

Coverage measures which lines the test suite executed. It says nothing about whether the suite would notice if those lines changed. That gap is not academic — it is the entire failure mode of machine-generated tests, because a generator that wants a green bar and a high percentage discovers very quickly that assertions are the risky part.

public function test_discount_is_applied(): void
{
    $cart = new Cart([new Item('SKU-1', 1000)]);
    $result = $this->pricer->apply($cart, new Coupon('SAVE10', 10));

    $this->assertNotNull($result);          // true for every return value
    $this->assertInstanceOf(Cart::class, $result);   // true by signature
    $this->assertIsInt($result->total());   // true for 900, 1000, and -50
}

Every line of apply() is covered. Change $total * (1 - $rate) to $total * (1 + $rate) and the test still passes. It is a coverage instrument wearing a test's clothes, and a suite full of them is worse than no suite: it produces a number that makes people stop looking.

What mutation testing measures instead

A mutation testing tool makes small, deliberate edits to your source — flips a > to >=, turns + into -, replaces a return value with null, deletes a method call — and reruns the tests that cover the mutated line. Each edit is a mutant. If a test fails, the mutant is killed. If everything stays green, it escaped.

the two numbers, and which one to gate on
	What it counts	Use it for
Coverage	Lines the suite executed	Finding code nobody tests at all
MSI	Mutants killed ÷ mutants generated	Whether the tests assert anything
Covered MSI	Mutants killed ÷ mutants on covered lines	Test quality, with the untested code excluded

Covered MSI is the honest quality metric — it asks how good the tests that already exist are, without dragging in code nobody has touched yet. MSI is the one to ratchet over time. Gate on both, at different thresholds, and the difference between them tells you whether your problem is missing tests or weak ones.

An escaped mutant is also the most useful artefact in this whole pipeline, because it is not a complaint — it is a concrete counterexample. "Line 42 changed from >= to > and your suite was fine with it" is a failing test waiting to be written, and it is specific enough to hand to a model without any further explanation.

Step 1: get a baseline on one class

Do not run this repo-wide on the first day. Mutation testing is expensive — every mutant is a test run — and a full-codebase first run tells you only that the codebase is large. Pick the class that scares you most.

composer require --dev infection/infection

# a single file is a positional argument (--filter is deprecated since 0.34)
vendor/bin/infection src/Pricing/Pricer.php \
  --threads=max \
  --show-mutations=max \
  --log-verbosity=all

# Metrics:
#   Mutation Score Indicator (MSI): 31%
#   Mutation Code Coverage:         88%
#   Covered Code MSI:               35%

That shape — coverage in the high eighties, MSI in the low thirties — is the signature of a suite written for the coverage number. It is also the most common starting point on any codebase where someone has previously run a "raise coverage" initiative.

Step 2: feed the escaped mutants back, not the source file

This is the step everyone gets wrong, and it is the whole trick. The instinct is to hand the model the source file and ask for better tests. Do not. You did that already and got the assertion-free suite. Hand it the escaped mutants.

The difference is that "write tests for this class" is an open-ended request with no failure condition, while "here are eleven specific edits that your suite did not notice, write a test that fails for each" is a closed task with a checkable answer. The model stops inventing scenarios and starts closing gaps.

{
  "source": { "directories": ["src"] },
  "logs": {
    "json": "build/infection.json",       // every mutant, with diffs
    "github": true                        // PR annotations on escapes
  },
  "mutators": {
    "@default": true,
    "global-ignoreSourceCodeByRegex": [
      "Log::.*"                           // logging is not behaviour
    ]
  }
}

The JSON log carries each escaped mutant with its file, line, mutator name, and a unified diff of the edit. That diff is the payload. Extract just the escapes and pipe them in:

#!/usr/bin/env bash
set -euo pipefail

TARGET="${1:?usage: close-the-gaps.sh <source-file>}"

vendor/bin/infection "$TARGET" --threads=max --log-verbosity=all

ESCAPED=$(jq -r '.escaped[] | "\(.mutator.mutatorName) at \(.mutator.originalFilePath):\(.mutator.originalStartLine)\n\(.diff)\n"' \
  build/infection.json)

[ -z "$ESCAPED" ] && { echo "nothing escaped"; exit 0; }

claude --bare -p "$(cat <<PROMPT
Each block below is a mutation applied to ${TARGET} that the existing test
suite did not detect. For each one, add a test that FAILS when that mutation
is applied and PASSES against the current source.

Rules:
- Assert on values, never on not-null, not-empty, or instance-of alone.
- Do not modify ${TARGET}. Tests only.
- Follow the conventions in the neighbouring test file.
- If a mutation is equivalent (no observable behaviour change), skip it
  and say so rather than writing a test that asserts nothing.

${ESCAPED}
PROMPT
)" --allowedTools "Read,Edit,Write,Bash(vendor/bin/phpunit *)"

vendor/bin/infection "$TARGET" --threads=max   # did the score move?

Step 3: the equivalent-mutant escape hatch

Some mutants cannot be killed, because the edit does not change observable behaviour. Swapping < for <= on a loop bound that is never hit at the boundary is the classic case. These are equivalent mutants, they are undecidable in general, and they are the reason 100% MSI is not a goal.

Without an explicit way out, a model asked to kill every mutant will write a test for the untestable one anyway — usually by reaching into a private via reflection, or asserting on a log line. Both are worse than the escape. Giving it permission to say "this one is equivalent" is what keeps the generated suite honest, and the ones it flags are worth reading: an equivalent mutant is often dead code with better manners.

/**
 * @infection-ignore-all
 */
private function clampToNonNegative(int $n): int
{
    return $n < 0 ? 0 : $n;   // callers already guarantee $n >= -1
}

Step 4: gate CI on the diff, not the codebase

A repo-wide MSI gate on an existing codebase fails on day one and gets disabled on day two. Gate the lines the pull request touched. --git-diff-lines mutates only changed lines, which makes the run fast enough for every PR and turns the metric into a ratchet: the code you write today has to clear the bar, and the rest of the codebase improves whenever someone happens to touch it.

- name: Mutation test the diff
  run: |
    vendor/bin/infection \
      --git-diff-lines \
      --git-diff-base=origin/main \
      --min-covered-msi=80 \
      --min-msi=65 \
      --ignore-msi-with-no-mutations \
      --logger-github \
      --threads=max
  env:
    XDEBUG_MODE: coverage

Start the thresholds below where you already are, not where you want to be. Ratchet by five points a month in a one-line PR that everybody sees. A gate people raise deliberately survives; a gate that blocks an unrelated hotfix at 2am gets continue-on-error: true and is never removed.

What actually changed

Three services, one quarter, same team. The first column is what the model produced from "write tests for this class"; the second is the same model in the escaped-mutant loop.

before and after the scoring function
	Prompt-only	Mutant-driven loop
Line coverage	91%	88%
Covered MSI	34%	81%
Tests written	127	63
Assertions per test	1.4	3.1
Regressions caught pre-merge, Q3	2	11

Coverage went down by three points and the suite got dramatically better, which is the entire argument in one row. Half as many tests, each of them load-bearing. The generated tests also became reviewable — a test that exists to kill a named mutant has an obvious reason to exist, and reviewers stopped rubber-stamping the test file.

What this is bad at

Cost. Every mutant is a test run. On a suite with slow integration tests this is hours, not minutes. --git-diff-lines and reusing existing coverage reports are not optimisations here, they are the difference between viable and not.
Integration and end-to-end tests. Mutation testing rewards fast, isolated, deterministic tests. Point it at a suite that hits a database and it will time out mutants and report them as escapes, which is noise dressed as signal.
Design. A killed mutant says the test noticed a change. It does not say the behaviour is correct, that the API is sensible, or that the abstraction earns its keep. MSI is a floor, not a ceiling, and a suite can score 90% while testing the wrong thing thoroughly.
Anything with a mocked boundary. Mutants inside code you have mocked out are unkillable by construction. Exclude those paths explicitly rather than letting them drag the score down and train everyone to ignore it.

Pitfalls

Running mutation testing before the suite is green. Infection needs a passing baseline; a pre-existing failure makes every mutant in that file look killed.
Letting the model edit source. Give it Write on the test directory only. Otherwise the fastest route to a killed mutant is deleting the branch that produced it.
Chasing 100% MSI. Equivalent mutants make it unreachable, and the last twenty points are bought with reflection and brittleness. Somewhere in the eighties is where the curve flattens.
Trusting the first loop. Rerun Infection yourself after the model finishes — a suite that passes phpunit has not been graded, and the model claiming it closed eleven gaps is not evidence that it did.
Forgetting XDEBUG_MODE: coverage in CI. Without a coverage driver the run produces nothing and, depending on flags, can exit zero.

The general lesson is bigger than testing. A model given a metric will move the metric, and it will find the cheapest path there — so the leverage is almost never in the prompt, it is in choosing a metric whose cheapest path is the thing you actually wanted. Coverage has a cheap path that is worthless. Mutation score does not.

References

Infection command-line options — Every flag used above, with current deprecations.
Infection mutators — What edits it makes, and how to switch profiles off.
Stryker Mutator — The same loop for JavaScript, TypeScript, C#, and Scala.
Claude Code headless mode — `-p`, `--bare`, `--allowedTools`, and structured output.
Wire an AI reviewer into CI without it becoming noise — The same constrain-and-measure idea, applied to review.
A Laravel CI pipeline that actually catches things — Where this stage fits in the wider pipeline budget.

Takeaways

Coverage rewards executing a line; a model optimising coverage writes tests that execute lines and assert nothing. Change the metric, not the prompt.
Feed the model escaped mutants, not the source file — a specific counterexample is a closed task, "write better tests" is not.
Gate CI on --git-diff-lines against the merge base, so the metric ratchets on new code instead of failing the whole repo on day one.
Give the model an explicit way to declare a mutant equivalent, or it will kill the unkillable ones with reflection and log assertions.
Rerun the judge yourself. A suite that passes phpunit has not been graded, and the generator is not a reliable witness to its own score.

All notes · Shehzad Aslam