Measuring the walk
Stéan Bouwer · May 2026
I ran 27 controlled trials to find out what an AI agent review phase costs, and what it buys. The cost came back clean. The second question turned out to be unanswerable by this design — and establishing that took more work than the cost model did.
Here is how I know my blind failed.
I had built a corpus of code bundles, stripped of anything identifying, to be scored by an independent reviewer who would not know which experimental condition produced which bundle. Before running it, I tested whether the blind held. It did not — a probe kept identifying pipeline output by quoting review language it found in the files. Phrases like "found by adversarial review", "the walker", "caller-ID census".
That looked like a leak. Then I counted the phrases across all 24 bundles and
found them in every single one — including every bundle produced by the control
condition, which I had launched with --disallowed-tools Agent Task and which
is therefore structurally incapable of spawning a review agent at all.
The tell that appears in the arm that cannot produce the behaviour is proof the tell is baseline.
The vocabulary was not coming from the review phase. It was coming from the repository's own always-loaded instruction file, which every condition reads before it does anything. The control arm was the instrument: a condition unable to exhibit the thing being detected converts every hit into a free false-positive rate.
That was the first of three structural problems. Two of them I fixed. The third I could not, and it is the reason this piece is about measurement rather than about the workflow.
What I was trying to measure
I maintain a codebase with a heavily instrumented agent workflow — five purpose-built subagents, handoff contracts between them, and a prescribed sequence for building and reviewing changes. I wanted to know whether that machinery was worth its price. Not whether AI agents help in general. Whether this pipeline, on this codebase, returned more than it cost.
Three conditions, defined before anything ran, laddered so each step isolates one variable:
- Arm A — control. Main session only, enforced by launching with
--disallowed-tools Agent Task. Zero subagents possible. The one condition where finding none is correct. - Arm B — choice. Free to delegate or not. Same tree, same tools, the same twelve agents advertised. No prescribed sequence.
- Arm C — prescribed. Free to spawn none, one or many, but when it spawns,
through my own five agents and their handoff contracts, in the
/build→/walk→/wrapsequence.
A→B asks whether delegation helps at all. B→C asks whether prescribing the sequence helps, given the same agents. Agent tuning is held constant across B and C, so it is not measured here — but it is not confounding either. It is a third question needing its own experiment.
Three tasks, three replicates each, 27 scored cells. Every task was real outstanding work from my own backlog, not a synthetic benchmark.
Two corrections I owe the reader before the numbers.
Arm B was originally documented as stripped down to stock agents. It never was. B and C are bit-identical in tree and tool configuration and differ only in the prompt — my twelve agents are advertised to B exactly as to C. That is stronger for the comparison and weaker as a claim about out-of-the-box behaviour. Anyone who read the original wording would take a B result as evidence about stock tooling, which it is not.
And the run order was a fixed rotation — r1 A B C, r2 B C A, r3 C A B — not randomisation. Every arm occupies each position exactly once, which removes the position confound exactly rather than in expectation. But it is not a random draw, and a reader who wanted randomisation should discount accordingly.
One operational detail that matters more than it sounds: every worktree shares
one node_modules junction, and the test runner writes its count file into it.
Two conditions running at once score each other's trees. Cells ran strictly
sequentially. I wrote that down before I started, classified it as a rule with
no mechanism behind it, and then broke it once anyway — the driver launched one
cell on top of a live one. It did no damage, because the intruding cell happened
to be read-only for its whole 83-second life. A rule you have correctly
identified as unenforceable is still unenforced.
The cost model worked
This is the half that succeeded, and it is not what the piece is about, so it gets one section.
Arm C separates cleanly. The two solo arms do not separate from each other.
Every figure in this section is in US dollars. On the third task, C's cheapest cell cost 23.56 — more than the most expensive cell from any other condition, at 15.60, by a factor of 1.5. Medians: A 8.65, B 14.67, C 32.46. C over A is 3.75×.
But arm B's own spread across its three cells is 3.04× — larger than the 1.70× ratio between the A and B medians that any A-vs-B claim would have to beat. So the A-vs-B comparison is not resolvable at this sample size, and I record no reading of it. That is a statement about resolving power, not about which condition is better. It is also the negative control doing its job: two conditions that behaved similarly should not separate, and they don't.
The one robust positive finding: the prescribed arm does more work and pays less per unit of it.
Sixty cents per model-minute of work, against 72 cents for the solo arm and 66 for the free-choice arm — with C's full range [0.58, 0.60] sitting entirely below both. On an earlier task the same measure read 0.54 against 0.58 and 0.62. Two tasks, two different shapes, same direction, control cleared both times. The premium is volume purchased at a discount, not waste.
And my registered prediction was falsified. I had predicted the cost gap would close on tasks where delegation actually pays. On the first task it widened instead — C over A on cost went to 2.24×, against 1.98× on the task where delegation was not supposed to help. I record it as falsified on the numbers and unevaluable on its premise, because the premise assumed arm B would delegate by judgment, and B turned out to be under a standing prohibition at the time. Those are different failures, and collapsing them would flatter the design.
Failure one: the blind is defeated by house vocabulary
Return to the opening. Every bundle in both of the first two tasks carried at least one review phrase, control arm included.
This is not fixable by stripping harder. The vocabulary is inherited from a file every condition loads before it starts work — the same instruction file that makes the codebase legible to an agent in the first place. Any repository with a documented review culture will have this property. The more mature your written practice, the more thoroughly it defeats your own blind.
I also had to exclude the handoff artefacts entirely — the contracts, the agent reports, the intermediate documents — because they are identifying on sight and only one condition produces them. That exclusion means the handoff contract, the thing the whole workflow is built around, is invisible to the quality measurement by construction. It was never going to be scorable by this design, and I should have noticed that before building the corpus rather than after.
For contrast, here is what a validated blind looks like. On the third task, bundle file counts came out A 23, B 24, C-post 26, C-pre 26, with every top tell-carrying file present in all twelve bundles. Balanced tells mean inherited vocabulary. Tells concentrated in one condition would mean a leak. They were balanced. That is a measured blind rather than an asserted one, and the measurement is cheap.
Failure two: you cannot rewind a shell
To ask whether the review phase improved anything, you need the tree as it stood before the review and after it. I had the after — it is on disk. I tried to reconstruct the before by rewinding the transcript, undoing each recorded edit.
That does not work, and the reason is structural. A transcript records tool
calls. An agent that writes with a heredoc, or edits in place with sed -i, or
copies a file, or deletes one, leaves no invertible record — the operation
happened inside a shell and the shell does not report what it overwrote. I
measured the mutation shapes first and found 10 to 21 shell redirects per cell,
plus in-place edits, copies, deletions and checkouts. A faithful forward replay
of that is a shell interpreter.
Five of six pre-review trees on the first two tasks carried post-review content. And the bias runs one way: contamination makes the before-state look better than it was, which flatters exactly the intervention being measured.
The third task fixed it by abandoning reconstruction entirely. Instead of replaying, I recorded — sampling the live git tree every ten seconds during the run, and cutting the pre-review state out of the recording at the instant the review began. All three cuts verified exact: identical tree hashes either side of the cut instant, which is a proof rather than a proximity. Zero files fell back to post-review content, against eight of 124 on the earlier tasks.
When the forward replay needs an interpreter, stop replaying and start recording.
Failure three: the comparison is a tautology, and this one cannot be fixed
This is the finding. Not the counts that follow it.
The only quality comparison this design offers is C-pre against C-post — the same session's tree before and after its own review phase. But C-post is C-pre with the review's own findings fixed. The review necessarily removes the defects it detected. Any drop it produces is guaranteed by construction and is not evidence of anything.
I saw this coming and pre-registered a guard before the corpus existed. For each finding the independent reviewer raised, I asked: had this cell's own review agent already named that file? Only findings the review agent never named can carry signal. I matched by file path — deliberately coarse, which over-counts "already named" and therefore errs against detecting an effect. That is the safe direction.
Every major finding raised in a prescribed-arm cell sits in a file that cell's own review agent had already named. Six of six. The "review agent never mentioned it" bucket — the only bucket that could evidence a quality effect — is empty. Across all findings, not just majors, it is one of seven for C-post against one of six for C-pre. Identical, and essentially nothing.
And an all-yes return is itself a finding about the guard. At file-path granularity it cannot separate the review agent named this defect from the review agent touched this file. So the correct phrasing is no effect detectable, not no effect — and that distinction is not a hedge. It is the difference between a measurement and a null.
There are two readings of the convergence and neither is established. Either the review looks in the right places and does not finish the job there — reviewer and review agent were briefed differently and run independently, and found the same regions interesting. Or it is a shared blind spot: reviewer and review agent are the same underlying model, and agreement between two passes of one model is weak evidence about the files and strong evidence only about the model. Distinguishing them needs a reviewer that is not the same model, or defect-level rather than file-level matching. I have neither. Naming both beats picking one.
The instrument: one major finding in eight was wrong
This is the most portable number here, and it is about the reviewer rather than about any condition.
Every major finding was re-checked against its own tree — the full pre-registered sample, no discretion over which got checked. Seven confirmed. One refuted.
The refuted one is worth quoting in full, because its shape is the point. The finding claimed two unsuppressed lint violations in a script would fail the build gate. Running the linter against that tree returns exit 0 and zero bytes of output. The rule had left that path days earlier — and the configuration file recording the removal was inside the reviewer's own bundle, carrying a comment naming the date and the reason.
The reasoning was correct in every step. The premise was false.
Nothing about a well-argued finding announces that its first sentence is wrong.
Where it landed matters as much as the rate. The refuted finding belonged to C-post, so confirming it moved C-post from two majors to one. Had I banked the null without a confirmation pass, my published counts would have been wrong in the direction that flattered the intervention — which is the error that never gets caught later, because nobody re-opens a result that came out the way they expected. The confirmation pass changed the numbers it was run to protect.
Two properties make this generalise well past one experiment. The check is asymmetric: refuting a premise costs one grep, while acting on it costs a change to code that was already correct. And the error rate compounds against small samples: at the handful-of-findings-per-diff counts real review produces, one in eight is enough to reverse an ordering on its own.
One confirmation reproduced verbatim — a finding predicted a specific failure line, and executing that tree's check printed exactly that string. Two others were confirmed by mechanism rather than execution, because their trigger is a host privilege state and a shallow clone. I state that boundary rather than blurring it.
Overriding a pre-registration, with both halves stated
My registered rule: report a between-condition difference only if it exceeds the observed spread between repeat reviews of the same bundle. Measured spread across both repeat pairs: zero. By the letter, that licenses reporting every difference I found.
I did not accept it, because the two pairs behaved differently in a way a count threshold cannot see.
- Pair one reproduced the same defect — same file, same symbol, same severity, both passes.
- Pair two reproduced the same count — one and one — while sharing zero findings. A minor issue at one line against a major issue at another.
Counts reproduced two of two. Content reproduced one of two. A floor of zero on the count, with content unmeasured, is not a floor of zero. Agreement on volume without agreement on substance is agreement by coincidence.
Overriding a pre-registered threshold on evidence discovered afterwards is normally the cardinal sin of experimental work. I think this is the exception, on two conditions that both hold and both have to be stated:
- The evidence showed the rule measured the wrong quantity — counts, where the thing that mattered was content.
- The override moves the verdict toward less claimed, not more.
State only the first and it reads as rationalisation. State only the second and it reads as modesty. Both together are what make it the exception rather than the sin.
The power limitation is a property of the design
One to three findings per bundle. One review pass per bundle. Three bundles per condition.
At this defect density, nothing could have separated the conditions. The design had almost no dynamic range, and that is true independently of how the conditions actually performed.
This must not be read as the conditions are equal. It is this instrument could not have told them apart. A reader who supplies "so the workflow makes no difference" has drawn a conclusion the data cannot support in either direction.
Six instruments that produced plausible numbers
The most transferable output of the whole programme may be this list. Every item is a case where the measuring apparatus, not the subject, was producing the number — and every one was published or nearly published before it was caught.
The 19-millisecond review agent. A concurrency sweep timed the interval between dispatching a tool and receiving its result — which, for an agent running in the background, times the dispatch and not the run. One agent was recorded at 19 milliseconds against a real 11.6 minutes. The published claim that "the prescribed arm never ran two agents at once" was withdrawn: another cell ran three search agents concurrently, and the free-choice arm fanned out too. Per-cell agent-minutes are left unmeasured, because two defensible rules for when an asynchronous agent finishes disagree by 84 agent-minutes.
The check that could not fail. A completeness probe compared reports against
a copy it often failed to find, and in JavaScript text.includes("") is always
true — so three reports passed a check that had no failing case. This is my own
documented vacuous-probe failure mode, occurring inside the instrument built to
apply it. I discarded it and recorded it rather than quietly replacing it.
The lossy extraction. Review reports were pulled from the last assistant message; every report actually spans between 3 and 22 blocks. Between 127 and 5,582 bytes per report were being dropped — 47% for one cell, 58% for another — and one cell declared six findings while the retained text enumerated three.
The non-greedy fence. The runner extracted structured output with a non-greedy match that terminates at the first closing code fence, so any finding containing an embedded code block was truncated mid-string and declared unparseable. This one is severity-correlated: concrete reproductions are exactly what a good finding contains, so the instrument was preferentially losing its best material.
328, then 29, then 21. An earlier pattern-matching check reported three
different violation counts across three rebuilds, misclassifying a known-good
file every time. Root cause: a \s degrading to s inside a template literal.
I abandoned it unmeasured rather than publish a fourth number.
An unexplained intermittent, still open. A total test-collection failure — 126 of 126 files, zero import time — has now fired four times. Both halves of my published trigger hypothesis are falsified: it fired with no worktree activity at all, and it did not self-repair on re-run. There is a working remedy and no diagnosis, which is its own hazard, because "delete the cache and re-run" and "route around a real failure" are the same motion. Flagged, not chased.
The pattern across all six: in every case the broken instrument produced a plausible number rather than an error. A dispatch time looks like a duration. A vacuous probe looks like a pass. A truncated report looks like a short one. And partial fixes are the worst of all, because a plausible middle number is more believable than an implausible extreme.
What this licenses
Licensed. The cost model: three conditions, three tasks, the prescribed arm separating cleanly at 3.75× on the third task's medians, and the price-per-unit-of-work result replicating in the same direction across two tasks with the control cleared both times. A search-strategy finding that turned out to be the largest single cost lever I measured anywhere — which I have written up separately, because it deserves better than a paragraph here. And the instrument's error rate, which anyone proposing to run this design at scale needs, and which I have not seen published elsewhere.
Not licensed. Any claim that the workflow produces better work, in either direction. The quality axis returns a null with a documented account of why the wider comparison is intrinsically unblindable, and the design lacked the range to separate the conditions even had the blind held.
The cost axis compares conditions. The quality axis compares one condition to itself. This design can never join them.
And the framing that matters. This is not a failed experiment. Here is what it costs, and here is why the obvious way to measure what it buys cannot work is a publishable finding. The failure to measure quality is a result — provided it is reported as one rather than as a gap.
Establishing it required building the blind, the live-tree cut, the tautology guard, the repeat pairs and the confirmation pass. A programme that skipped all five would have published a clean, confident, wrong number.
Open questions this design leaves standing
- Does a structured review phase help at a defect density where an instrument has range? Nothing here answers it.
- Is the convergence between reviewer and review agent competence, or a shared blind spot? Needs a reviewer that is not the same model.
- Does agent tuning matter, holding sequence constant? Held constant here by design — a third experiment, not a second variable in this one.
- Per-cell agent-minutes, blocked on a definitional choice about when an asynchronous agent finishes that changes the answer by 84 minutes.
Three lessons filed against my cross-project register: house vocabulary defeats a blind · a finding can be sound and rest on a false premise · an intervention that repairs cannot be scored by its own detector.