PBC for Claude Code - Rebuild44 vs Rebuild45 Case Studies
- Context
- What We Observed
- Where The Time Went
- Mapping to the Research
- The Claim: Wide Range Means Assignable Cause, Not Noise
- Findings by Variable
- Open Questions From These Cases
Context
This page is a worked comparison of the methodology in PBC for Claude Code: Research. Two Darmok scenarios ran on consecutive nights against sheep-dog-grammar, on branches Rebuild44 (May 8) and Rebuild45 (May 9). Both pair-ranges were wide enough to inspect, and both diagnoses came back to the same assignable cause in the prompt template — even though the two scenarios have different surface signatures (one with two valid fix locations, one with only one).
| Case 1 | Case 2 | |
|---|---|---|
| Scenario | This object step definition text parameter exists with other parameters validation |
This object step definition parameter set doesn't exist validation |
| Rebuild44 commit | 3c9a7a2 |
9138e195 |
| Rebuild45 commit | b4bf1adf |
667219ab |
| Pair-range (total) | 5.6 min | 4.2 min |
| Files patched | different between branches | same between branches |
| Diagnosed cause | prompt — JaCoCo cross-reference skipped | prompt — JaCoCo cross-reference skipped |
Case 1 leaves room to argue “the spec was ambiguous, the prompt is incidental.” Case 2 removes that argument: the spec does not admit two file locations, both branches converged on the same set of file additions, and the pair-range still fired at 4.2 min on the same prompt-step variance.
What We Observed
Case 1 — text parameter exists with other parameters validation
| Rebuild44 (May 8) | Rebuild45 (May 9) | |
|---|---|---|
| Commit | 3c9a7a2 |
b4bf1adf |
| Total wall-clock (start → commit) | 13.2 min | 7.6 min |
| Green phase only | 9.5 min | 3.3 min |
| Refactor phase | 2.7 min | 3.5 min |
| Production file patched | TextIssueDetector.java |
TestStepIssueDetector.java |
| Test passes after fix | yes | yes |
Pair-range = 5.6 min. Both runs satisfied the test, but they patched different production files. The other validator’s bug remained latent in each branch. This is the ambiguity signature from Part 1 of pbc-research: wide pair-range, both runs pass, functional difference between work-trees.
Case 2 — parameter set doesn't exist validation
| Rebuild44 (May 8) | Rebuild45 (May 9) | |
|---|---|---|
| Commit | 9138e195 |
667219ab |
| Total wall-clock (start → commit) | 11.8 min | 7.6 min |
| Green phase only | 6.9 min | 4.8 min |
| Refactor phase | 3.3 min | 1.8 min |
| Files added/modified | RowIssueDetector (new), RowIssueTypes (new), ValidateActionImpl (wire) |
same three files |
| Test passes after fix | yes | yes |
Pair-range = 4.2 min. Both runs added the same files; ValidateActionImpl wiring is byte-identical; RowIssueTypes differs only by a trailing newline. The only meaningful divergence is implementation style of the new validateCellListWorkspace method (Rebuild44 used nested for-loops with a mutable found boolean; Rebuild45 used a single Stream.flatMap…anyMatch pipeline). Same external behavior on both sides.
This case removes the “spec ambiguity” interpretation that case 1 leaves open. Both runs landed on the same files; there was nothing about the spec’s choice of fix-site that differed between them.
Where The Time Went
The two sessions’ JSONL transcripts and the runner log gave minute-by-minute timelines for each. In both cases the variance lived in the green phase’s first 3–7 minutes, and both times the proximate cause was identical: Rebuild44 mined JaCoCo via a coverage cross-reference; Rebuild45 ran ls on the JaCoCo dir and pivoted away.
Case 1 timeline
Rebuild44, 22:43–22:50 EDT (~7 minutes). Spawned an Agent subagent with description “Explore coverage and UML spec for failing test”. Inside, the subagent ran:
find .../jacoco-with-tests -type f -name "*.html" | head -20
find .../jacoco-with-tests -name "*.html" \
| xargs grep -l "Validation for Workspace Issues|object step definition text parameter exists with other parameters"
find .../jacoco-with-tests -type f -name "*IssueDetector*"
The xargs grep -l is the cross-reference: it maps the failing scenario name to the specific class HTMLs (TestStepIssueDetector.html, TextIssueDetector.html) by searching inside coverage. The subagent then opened TextIssueDetector.java.html and read the lines highlighted as hit by the failing test. That read produced the file:line for the patch.
Rebuild45, 07:48–07:50 UTC (~2 minutes). Inlined the same green prompt; ran ls .../jacoco-with-tests/ and ls .../jacoco-with-tests/org.farhan.dsl.issues/, got back package directories with no test-name context, and pivoted in parallel to source-tree greps:
grep -ri "text parameter exists with other" .../site/uml/ # no matches
grep -r "text parameter exists with other" .../src/ # empty
find ... -name "*.feature" | xargs grep -l "text parameter" # empty
None of those returned anything useful. By 07:49:05 the run was reading uml-communication.md and never returned to coverage. It eventually reached a fix candidate via direct grep on TestStepIssueDetector rather than via coverage — different file, same shape of bug.
Both validators have the same logic defect (comparing a single string against something that is actually a collection of cells). Either fix passes the assertion. The test does not say which validator is the correct site.
Case 2 timeline
Rebuild44, 22:19–22:23 EDT (~4 minutes). Spawned three Agent subagents in succession:
- “Explore JaCoCo coverage for failing test path” — same
find+xargs grep -lpattern as case 1, this time mapping the scenario toTestStepIssueDetector.html,CellIssueDetector.html, andXtextValidateAnnotationSteps.java.html. - “Read UML spec and existing validator code”
- “Read validator, interfaces and test impl for missing RowIssueDetector”
By 22:24 the run had the file shape and was writing RowIssueDetector.java and RowIssueTypes.java.
Rebuild45, 07:08–07:11 UTC (~3 minutes). Inlined; same ls .../jacoco-with-tests/ + ls .../org.farhan.dsl.issues/ then no further coverage reads. Pivoted to source-tree greps and a sequence of UML reads:
grep -nE "step definition parameter set" .../sheep-dog-grammar/src/
find ... -name "*.feature" | xargs grep -l "parameter set"
By 07:11 it had the file shape and was writing the same two new files.
The implementation-style difference (nested loops vs streams) is downstream of how each run got there: Rebuild44 read CellIssueDetector.java first as the precedent and copied its imperative shape; Rebuild45 read TestStepIssueDetector.java first as the precedent and copied its stream-pipeline shape. Same prompt, different first-precedent, different code style.
Mapping to the Research
The signature lines up with pbc-research Part 1 in both cases:
| Predicted | Case 1 observed | Case 2 observed |
|---|---|---|
| Wide pair-range | 5.6 min vs ~1–2 min floor | 4.2 min vs ~1–2 min floor |
| Each run individually within in-control band | yes | yes |
| Both runs pass the test | yes | yes |
| Two work-trees differ | functionally (different files patched) | stylistically (same files, different code shape) |
The cause is in the system, not the worker. Neither Rebuild45 run was worse; both were steered by the same prompt down a path the prompt didn’t constrain.
The Claim: Wide Range Means Assignable Cause, Not Noise
The unifying claim is in Wheeler/Deming vocabulary: a wide pair-range is assignable cause — diagnosable, attributable to a specific element of the system, removable. It is not common-cause variation, the irreducible run-to-run noise you have to live with until you change the system itself.
The research framing locates the assignable cause in the specification — a test that admits two valid implementations. These two cases add a second site: the prompt template that drives the worker. Both are “in the system.” Both are influenceable. Neither is the worker getting unlucky. Case 2 in particular rules out spec ambiguity as the necessary explanation, since both runs converged on the same files.
The current green.md says, in step 2, “Examine the failing code path — ${jacocoPath}. Use coverage to see what’s already implemented along the path the failing test exercises.” It does not spell out how. A directory listing is one way to “examine” coverage; the cross-reference grep that maps test name → class HTMLs is another. Without the cross-reference, the model has no path from “scenario name” to “specific file” via coverage, and it falls back to less effective discovery.
When we replaced the under-specified instruction with the explicit cross-reference steps and split green into a problem-identification phase that resumes into a fix phase (issue #384), the assignable cause is removed from the system. We expect the pair-range distribution on subsequent runs to tighten. If it doesn’t, the next assignable cause becomes visible — which is the iterative-limits discipline from Wheeler that the research already calls out.
The chart’s job is the same regardless of where the cause sits. The wide-range signal flags that something in the system is influenceable here — spec, prompt, harness, or fixture data. Diagnosis comes from inspecting the runs; the chart itself only tells you to look. The remediation differs by site:
- Cause in the spec: clarify the test (the original framing in pbc-for-claude-code)
- Cause in the prompt: tighten the instructions so the path is forced (this case)
- Cause in the harness: change the platform (Wheeler’s “tampering with common cause” warning still applies — only act on assignable evidence)
The opposite of the claim is the one to refute deliberately: that wide-range runs are just LLM noise — temperature, sampling, model-of-the-day — and not worth investigating. Every wide-range case we have inspected so far has surfaced an assignable cause on the first or second pass. None has reduced to “the model felt different that night.”
Findings by Variable
Each subsection records this run’s findings about one Wheeler variable. Read the same heading across the run sequence to see how our understanding of that variable evolved.
green time pair range
Both cases fired a wide pair-range (Case 1: 5.6 min, Case 2: 4.2 min) on a ~1–2 min floor, and both traced to the same assignable cause in the prompt template, not the worker. The cheap “same file vs different file” disambiguation was not sufficient: Case 1 diverged functionally (different production files patched, the spec-ambiguity signature) while Case 2 converged on the same files and diverged only in code style (nested loops vs streams), yet both were prompt-cause — so same-file-vs-different-file tells you about the spec, not the prompt. Caveat: this run compares two runs taken on different nights (Rebuild44 May 8, Rebuild45 May 9), so the pair-range carries a time-of-day / server-decode confound that was not understood yet here; the doc’s primary diagnosis remained the prompt’s under-specified “examine the failing code path” step.
green time pair range moving range
No finding this run.
green time
No finding this run.
green time moving range
No finding this run.
scale & green tokens
The two halves of each pair were collected on different nights (one per night), so any decode-rate or server/time-of-day noise between the two evenings is folded into the measured pair-range; at the time this was not separated out (the series only later addressed it with token-scaling and then concurrent same-commit pairing). The doc’s actual diagnosis stayed on the prompt-template cause rather than server timing.
warm-up position
No finding this run.
Open Questions From These Cases
- We treated the pair-range as a rank-order signal here. The research’s Part 1 (pair-range chart with UCL) is still to-be-written; once it exists, both cases become a check on whether the formal chart would have flagged the pair without the manual inspection.
- The cheap-disambiguation signal proposed in pbc-research Open Questions (“same file vs different file”) needs at least one more dimension. Candidate: exploration depth in the JSONL — count of
Agentsubagents spawned, count of distinct files read, count ofxargs grepcross-reference commands. A pair where one side has zero of these and the other has several is the prompt-cause signature regardless of where the patch lands. - The same scenarios across
Rebuild40–Rebuild43would tell us whether the prompt-driven cause was new in this window or has been latent. If it has been, prior wide-range scenarios on this harness may also have prompt-template causes mis-attributed to spec ambiguity. - After #384 lands, the natural follow-up is to re-run a known-clean scenario six times on each branch and look at whether the pair-range distribution narrows.