Context

This page is a worked comparison in the tradition of pbc-4445 and pbc-4849. The same Darmok scenario ran twice against sheep-dog-grammar on May 29, on branches Rebuild65 (14:06) and Rebuild66 (18:18). Unlike the earlier cases — new-feature paths with no main-code patch in green — this scenario patches main code, and the two runs produced architecturally different implementations of the same behavior. One puts the business logic in production code; the other inlines it into the test layer. Both pass.

The headline findings:

  1. This is an assignable cause, not common-cause noise. The green-phase pair-range was 154,752 ms (2.6 min) — the widest in the validation tab and far outside the run-to-run spread of the other twelve scenarios (most ≤ 50,000 ms). On a Wheeler/Deming reading that is a signal, not noise: a specific, reproducible thing in the system produced it, and it is something we can act on — not variance to be tolerated.
  2. A smarter model did not infer the architecture. Both runs used --model opus. The slower run still concluded that domain logic (Character.isUpperCase(...), the literal issue strings) belonged in the test glue rather than in production code. Model capability did not supply the architectural rule “business logic lives in src/main.” That rule has to be enforced by the harness, not assumed from the model. (The Co-Authored-By: Claude Sonnet 4.5 trailer on both commits is a hardcoded default — #424 — not the model that ran.)
  Case
Scenario No component in the first step triggers an error validation
Rebuild65 commit 562105288bc360d0101b825a2456c196127e378b
Rebuild66 commit 3e9f512a1b012ef4253471e76c671b8ac7d19e24
Green-phase range 2.6 min (154,752 ms) — assignable cause
Files patched Different — Rebuild65: detector + types + impl (3 files, 2 in src/main); Rebuild66: impl only (1 file, 0 in src/main)
Model opus on both runs
Diagnosed cause path-dependence — the branch’s first validation scenario inlined into the test layer; every later scenario copied the local precedent
Remediation #425 (green-verify conformance check), backstopped by #423

Charts

Scenarios are numbered in run order (shortest→longest); see the tables below for which scenario each index is.

Rebuild 65 vs 66 green time pair range with XmR upper control limit

Rebuild 65 vs 66 green a time vs green b time per scenario

What We Observed

The study scopes to the green phase. Red and refactor are shown for completeness.

  Rebuild65 562105 Rebuild66 3e9f512
Phase total 8:11 (491,330 ms) 9:31 (571,217 ms)
Red phase 1:29 (89,856 ms) 1:13 (73,127 ms)
Green phase 3:38 (218,945 ms) 6:13 (373,697 ms)
Refactor phase 2:00 (120,618 ms) 0:52 (52,490 ms)
Green session UUID df53fbe4-a15c-4e26-b7fe-4ceb8278cb58 f6da0132-a46e-4073-9a28-235229dd6080

The slower green run also produced the architecturally worse result — green wall-clock here is not just slower, it is slower and wrong.

What each run committed

Rebuild65 — correct, follows the interaction-test.md Pattern 4 seam. Logic in main code; impl delegates:

  • TestStepContainerIssueDetector.java (+19) — new validateTestStepListOnly(...), sibling to the existing validateNameOnly
  • TestStepContainerIssueTypes.java (+1) — new enum constant TEST_STEP_CONTAINER_TEST_STEP_LIST_ONLY("The first step must have a component")
  • ValidateActionImpl.java (+3) — one delegating call

Rebuild66 — inlined into the test layer. The full rule body and the literal "The first step must have a component" are hardcoded in ValidateActionImpl (+11). TestStepContainerIssueDetector / IssueTypes untouched — zero src/main files changed.

Root Cause: a fork at scenario 1, then cascade by example

The divergence is not in the prompt (identical), not in the spec (identical red commit and RGRTest), and not in the model tier (both Opus). It is path-dependence across the branch’s scenario sequence.

Both branches start from the same base (a0ed9da12), where neither TestStepContainerIssueDetector nor the validation impl exist. The first validation scenario on each branch created them — and created them differently:

First validation scenario Rebuild65 6373869c Rebuild66 f0c5e09a
Files created TestSuiteIssueDetector (+21), TestSuiteIssueTypes (+12), ValidateActionImpl (+45) ValidateActionImpl only (+43)
src/main files touched 2 0
Impl body delegates: TestSuiteIssueDetector.validateNameOnly(testSuite) inlines: Character.isUpperCase(name.charAt(0)) + literal "Name should start with a capital"

From there it cascaded by example. Each subsequent scenario’s green phase reads the existing ValidateActionImpl (tool-call 10 in the run under study) and follows whatever pattern is already there:

Rebuild66 commit Scenario src/main touched
f0c5e09a Test suite name capital (scenario 1) 0
1400b986 Test case name capital (scenario 2) 0
3e9f512 No component in first step (scenario 3, under study) 0

Every commit on the Rebuild66 branch implemented validation entirely in test code. By the time the run under study reads ValidateActionImpl, inlining is the established local convention — so it adds a third inline block. Rebuild65, whose scenario 1 delegated, shows delegation in the same file, so its scenario 3 adds a detector method and delegates.

The genuinely nondeterministic event is scenario 1 on a clean base. With no detector to copy and no deterministic guard, the first scenario is a coin flip between “invent the detector and delegate” (correct) and “inline into the impl” (also green). That single outcome then determines every downstream scenario on the branch by example-following. The wide pair-range we caught at scenario 3 is the visible end of a divergence that actually happened two commits earlier.

The JaCoCo shortlist is a symptom, not the cause

An earlier draft of this case fingered the jacoco-shortlist (#404, the very mechanism introduced to fix pbc-4849). The shortlists did differ — recovered verbatim from each session’s Read tool-result:

Shortlist property Rebuild65 df53fbe4 Rebuild66 f6da0132
Total rows 90 86
TestStepContainerIssueDetector / IssueTypes rows present (rank 41–44, validateNameOnly shown) absent

But that absence is downstream of the inline cascade: because Rebuild66’s impl never calls the detector, the detector is never covered, so it never enters the coverage-derived shortlist. The bad prior implementation poisoned both signals the run had — the code it read and the shortlist derived from that code. JaCoCo can only report on the previous test’s implementation; if that implementation was wrong, JaCoCo faithfully propagates the wrong shape forward. The shortlist nondeterminism is real but secondary; fixing it would not stop scenario 1 from inlining in the first place.

Where The Time Went

Green is a two-call phase: green-compile (no-op exit when nothing fails to compile) then green-verify --resume (identify-and-fix). The divergence is entirely in the identify segment.

Segment Rebuild65 df53fbe4 Rebuild66 f6da0132
green-compile ~26 s, no-op ~32 s, no-op
green-verify identify → fix landed ~101 s (17:59:58 → 3 edits done 18:01:39) ~231 s (22:10:56 → 2 edits done 22:14:47)
Reads (session) 19 13
Edits (session) 6 2
Globs (session) 1 5
Output tokens 9,832 11,063
Cache-read tokens 3,675,596 3,451,150
Silent stalls / timeouts none none

Neither run stalled. Rebuild66 spent ~130 s more in identify because, with the detector absent from both its shortlist and its impl precedent, it globbed for *IssueDetector* twice (never opening a result) and detoured through the construction path (SheepDogBuilder.createTestStep, the TestStepNameParser fake, addTestStepWithFullName) before inlining.

The 4-read vs 0-read injection-point divergence

Calls 1–7 are identical (TodoWrite seed → grep log.txt → read shortlist → read RGRTest). They split at call 8:

# Rebuild65 (correct) Rebuild66 (inlined)
8 Read TestStepContainerIssueDetector.java Read log.txt
9–11 reads IssueTypes, TestSuiteIssueDetector, TestSuiteIssueTypes globs the feature, reads ValidateActionImpl, ValidateAnnotationImpl
19–22 Glob **/dsl/issues/*.java (1×, confirms files already read) Glob *IssueDetector* / *IssueTypes* twice, never opens a result
23–26 — construction-path detour
edits detector + types + impl (3 files) impl, impl (1 file, twice)

At call 8 Rebuild65 reads the detector and sees the existing validateNameOnly exemplar; the fix writes itself. Rebuild66 never reads it — the construction path it studies has no bearing on where validation belongs, and it falls back to the inline precedent in ValidateActionImpl.

The Fix

Sited where the inline first becomes possible — the green-verify prompt — and backstopped at refactor.

  1. #425 — green-verify conformance check (primary). green-verify.md already reads uml-interaction-test.md (line 11) but its TodoWrite process (lines 25–29) only says read → fix RGRTest → fix regression; nothing checks the impl diff against the patterns. Add a step: after editing, confirm every added/modified impl method matches a pattern in uml-interaction-test.md; if a change carries domain logic or string literals that fit no pattern, move that logic into src/main and delegate. Also tighten line 3 — it says “modify production code under the allowlist,” but the allowlist includes src/test/java/org/farhan/impl/, which is glue, not production. This prevents the inline at scenario 1, before it can cascade.
  2. #423 — deterministic regex backstop (refactor phase). Even with a perfect prompt a model can still inline. A regex check that fails an impl method carrying logic/literals catches it at refactor time. Producer-side prevention (#425) + consumer-side guard (#423).

Rejected alternative — allowlist rule “green must touch src/main.” False-positives on legitimate negative-path scenarios: the positive path already exists, red fails only for missing glue (a new impl method or branch that delegates to existing main code), and green correctly resolves it with a test-only change. The conformance check (#425) keys off what the impl change looks like rather than whether main changed, so it passes pure-delegation glue and flags inlined logic.

Mapping to the Research

Predicted symptom from pbc-research Observed here
Pair-range flags an assignable cause, not noise 154,752 ms — widest in the tab, far outside the others’ spread; a concrete, reproducible cause
Same-file vs different-file output is not the disambiguator Files differ (3 vs 1) AND both pass red→green; pair-range still the right trigger
Fix is in the producer, not the consumer green-verify prompt (#425), not the impl
JSONL recovers the cause in one pass Decisive evidence (the two shortlists, the per-call read order) lived only in the transcripts

Findings by Variable

Each subsection records this run’s findings about one Wheeler variable. Read the same heading across the run sequence to see how our understanding of that variable evolved.

green time pair range

This was a two-run comparison (Rebuild65 vs Rebuild66) rather than an in-run _a/_b pair, but the run-to-run green range of 154,752 ms (2.6 min) was the widest in the validation tab and far outside the spread of the other twelve scenarios (most ≤ 50,000 ms). The two runs patched architecturally different files — Rebuild65 touched the detector + types + impl (3 files, 2 in src/main), Rebuild66 touched the impl only (1 file, 0 in src/main) — so the same-file-vs-different-file shape did not disambiguate; the range itself was the right trigger and flagged an assignable cause, not noise.

green time pair range moving range

No finding this run.

green time

The slower run’s absolute green time was 6:13 (373,697 ms) against the faster run’s 3:38 (218,945 ms), and the slower run also produced the architecturally worse result — slower and wrong. There was no contradiction or timeout; neither run stalled. The extra ~130 s in the slow run came from globbing *IssueDetector* twice without opening a result and detouring through the construction path before inlining.

green time moving range

No finding this run.

scale & green tokens

The two runs were on the same model (opus) but ran at different times of day (14:06 vs 18:18 on May 29) on separate Darmok runs, a server/time-of-day confound that this two-run comparison cannot rule out. Output tokens were comparable (9,832 vs 11,063) and cache-read tokens similar (3.68M vs 3.45M), so the divergence was not explained by token scale.

warm-up position

The genuinely nondeterministic event was scenario 1 on a clean base: the branch’s first validation scenario forked between inventing the detector and delegating (correct) versus inlining the rule body and literal into the test layer (also green). That single first-scenario outcome founded the local convention every later scenario inherited by example-following, so the wide range caught at scenario 3 was the visible end of a divergence that happened two commits earlier. This is a warm-up/position effect — the first scenario seeds the pattern the rest copy.

Open Questions From This Case

  1. Why did scenario 1 fork the way it did? Both branches started identically. Pinning down whether anything beyond model sampling nudged f0c5e09a toward inlining would tell us whether #425’s prompt fix is sufficient or whether the base also needs a worked delegation example to anchor scenario 1.
  2. Should the first scenario of a new validation family seed a worked example? If scenario 1 is the coin flip, giving green an explicit “here is the detector-delegation shape for the first scenario” reference may matter more than any later guard.
  3. False-negative risk in the #423 regex on impls whose legitimate delegation is more complex than a single detector.validateX(...) call — the deterministic-vs-prescriptive tradeoff noted in #423.