Context

This page continues the pbc-4445 / pbc-4849 series. The same Darmok scenario ran on consecutive cycles against sheep-dog-grammar, both on 2026-05-25, on branches Rebuild52 and Rebuild53. The pair-range was flagged outside the upper control limit on the gsheet (flag: out) — the only such row on this tab — so this is the cleanest assignable-cause signal in the recent series. The diagnosis traces to a different prompt-template site than pbc-4849: not a missing JaCoCo annotation, but a missing sibling-scenario guardrail in fix.md.

  Case
Scenario This object step definition text parameter doesn't exist validation
Rebuild52 commit 9cb99b9b
Rebuild53 commit 95ba4ac9
Green-phase range 3.0 min (178,631 ms) — outside UCL (170,071 ms)
Files patched (final, post-refactor) R52: 5 files, R53: 4 files (R52 added an enum value; R53 reused an existing one)
Diagnosed cause prompt — fix.md does not direct Claude to check the negation-paired sibling Test-Cases that share the runner’s @RGR tag

The sibling list is already intrinsic to the runner: the runner class is tag-driven (@RGR), every Test-Case in the runner carries that tag, and the matching .feature file (which both runs read) lists all nine siblings in one block. The remediation is therefore a behavioral instruction in fix.md, not a new precompute file.

What We Observed

  Rebuild52 Rebuild53
Commit 9cb99b9b 95ba4ac9
Phase total 885,425 ms 761,306 ms
Red phase 75,468 ms 81,039 ms
Green phase 499,384 ms 320,753 ms
Refactor phase 238,644 ms 289,699 ms

Green-phase pair-range = 178,631 ms (3.0 min) against a range_UCL of 170,071 — the gsheet flags this row as out-of-control. Both runs eventually passed and committed; the variance is entirely in green-phase wall-clock.

The two runs ran the same Darmok flow: identify.md then resumed-session fix.md, then /compact, then /rgr-refactor forward sheep-dog-grammar. Single --session-id per scenario throughout.

  R52 session R53 session
Green session UUID 8d20e22d-d6a5-4f92-b85f-3f1019840e14 ddfa5604-a95d-4c53-a941-01572d3a15b2
identify.md issued 23:48:16Z 03:02:19Z
fix.md resumed 23:49:22Z (+1m 06s) 03:02:59Z (+0m 40s)
/compact issued 23:56:35Z 03:07:41Z
identify wall-clock ~66 s ~40 s
fix wall-clock ~409 s ~257 s

The +26 s identify gap is small-N latency noise on a ~50 s baseline. The +152 s fix-phase gap is the assignable signal.

Where The Time Went

The two JSONLs were parsed for the green-phase span (identify start through /compact).

  R52 green (slow) R53 green (fast)
Wall-clock 7m 57s 4m 59s
Assistant turns 81 60
Thinking blocks 17 13
Output tokens 14,422 12,826
Cache-read tokens 3.92 M 2.67 M
Bash calls 3 6
Read calls 12 9
Glob calls 10 3
Grep calls 7 7
Edit calls 6 4
TodoWrite 7 7
ToolSearch 1 1
mvn build cycles inside fix 2 (first FAILED) 1 (passed first try)
Time-to-first-Edit 5m 25s 4m 01s
Time from first Edit to passing mvn 2m 19s 0m 45s

Two contributors stack:

  1. Exploration phase (identify→first edit): R52 ran 10 Glob calls hunting one file at a time (**/ValidateAction*.java, **/*IssueDetector.java, **/*IssueTypes.java, **/ITestStep.java, **/IText.java, **/IStepParameters.java, **/IStepDefinition.java, **/SheepDogUtility.java, plus two earlier). R53 ran 3 Glob calls and substituted content-based Grep plus find via Bash to enumerate interfaces in one pass. Net: +1m 24s on R52.
  2. Build-retry cycle (post-edits): R52’s first attempt compiled but the test suite reported Tests run: 22, Failures: 1 — and the failing test was the sibling scenario text parameter exists validation, not the TC under fix. R52 then took 41 s of inspection (Grep+Read of the sibling .feature) and one corrective Edit to add a Content-parameter guard, then re-ran mvn. Net: +1m 34s on R52.

The two contributors approximately sum to the 2m 58s green-phase gap (3.0 min pair-range).

The 3 vs 10 Glob exploration calls

Both sessions opened with the same shape — calls 1–8 are functionally identical (ToolSearch:TodoWrite, TodoWrite, Grep log.txt, TodoWrite, Bash ls jacoco.xml, TodoWrite, TodoWrite, Read jacoco-shortlist.md). The divergence point is call 9:

# R52 (slow) R53 (fast)
1–8 (identical: ToolSearch, TodoWrite ×4, Grep log.txt, Bash ls jacoco.xml, Read jacoco-shortlist.md) same
9 Read log.txt offset=1520 limit=60 (re-read failure context) Glob target/darmok/* (survey darmok artifacts)
10 TodoWrite Grep "object step definition text parameter doesn't exist" .
11 Glob **/Validation for Workspace Issues.feature Grep "..." src-gen/test/resources
12 Glob **/ValidateAnnotationImpl.java Grep "..." src-gen/test/resources
13 Read .../Validation for Workspace Issues.feature Read .../Validation for Workspace Issues.feature (found via Grep)
14 Read .../ValidateAnnotationImpl.java TodoWrite
15 Glob **/ValidateAction*.java Grep "validateAnnotation\|getAnnotation\|validate annotation\|step parameters don't exist" src
16 Read .../ValidateActionImpl.java Read .../ValidateAnnotationImpl.java
17 Glob **/*IssueDetector.java Read .../ValidateActionImpl.java
18 Read .../TestStepIssueDetector.java Grep "navigateToNode\|navigateToDocument" src/test/java/org/farhan/impl
19 Glob **/*IssueTypes.java Grep "IText\|getText\|interface.*Text" src/main/java/org/farhan/dsl/grammar
20 Read .../TestStepIssueTypes.java Glob src/main/java/org/farhan/dsl/grammar/I*.java
21 Grep navigateToNode src/test/java/org/farhan/common Glob src/main/java/org/farhan/dsl/**/*.java
22 Grep navigateToNode\|cursor .../EMFTestObject.java Bash find .../dsl -name "*.java" \| head -40
23 Glob **/ITestStep.java Bash find .../dsl/grammar -name "I*.java"; find .../dsl/grammar -name "ITest*"
24 Read .../ITestStep.java Bash find .../src-gen/main/java/org/farhan/dsl/grammar -name "*.java" \| head -20
25 Glob **/IText.java Bash find .../src-gen/main/java/.../grammar -name "IText.java" -o -name "ITest..."
26 Glob **/IStepParameters.java Read .../IText.java
27 Read .../IText.java Read .../TestStepIssueDetector.java
28 Read .../IStepParameters.java Read .../TestStepIssueTypes.java
29 Read .../RowIssueDetector.java Read .../SheepDogUtility.java
30 Glob **/IStepDefinition.java Read .../ITestStep.java
31 Grep getStepObjectFullNameForTestStep src-gen/main/java/... TodoWrite
32 Glob **/SheepDogUtility.java Edit TestStepIssueDetector.java — first edit

R52’s exploration strategy is filename-pattern-first: assume a filename and Glob for it, then Read the unique match. When the next file name is uncertain, repeat. R53’s strategy is content-first: Grep for the scenario text or method names that should appear in the relevant code, then Read the matches; resort to Bash find only when filename enumeration is the cheapest way to list a directory.

R53 reached the first Edit at 3m 51s into the session vs R52’s 5m 25s — a 1m 34s lead from search strategy alone.

The build-retry cycle

R52’s first batch of 5 edits compiled. The first mvn clean verify came back with:

[ERROR] Validation for Workspace Issues.Validation for Workspace Issues -
        This object step definition text parameter exists validation -- Time elapsed: 0.026 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <Empty> but was: <Present>

The failing test is the sibling scenario text parameter exists validation, not the TC the green phase is implementing. R52’s new code path returned a Present (issue-detected) result for any IText cursor, regressing the case where the step parameter Content does exist.

R52 then ran two Greps (expected:|but was:|AssertionFailedError in log.txt; text parameter exists|text parameter doesn't exist in the .feature), Read the .feature for the sibling-row Step-Parameters table, and made one corrective Edit to TestStepIssueDetector.java adding the Content-parameter guard. Second mvn passed.

R53’s first batch of 4 edits already included the guard: it located the existing validateStepParameterListWorkspace(IRow) method (which has the canonical “if Content parameter exists, return empty” pattern) and factored a shared helper getStepDefinitionForTestStep(ITestStep) out, then wrote the new validateTextWorkspace(IText) against the same helper, applying the same Content guard.

The two final implementations differ:

  R52 (final, post-refactor) R53
New method validateTextWorkspace(IText) — copy of the lookup logic inline validateTextWorkspace(IText) — calls extracted getStepDefinitionForTestStep helper
Helper extracted no yes — also rewrites existing validateStepParameterListWorkspace to use it
Enum value adds TEST_STEP_TEXT_WORKSPACE with description "The step parameters don't exist for the step definition" reuses existing TEST_STEP_STEP_PARAMETER_LIST_WORKSPACE (same description text)
Files in commit 5 (TestStepIssueTypes.java extra) 4

Both passed the test once the guard was in place; the enum-duplication in R52 is a code-quality difference the refactor phase did not collapse.

Root Cause

The sibling-shortlist already exists implicitly. The runner class is tag-driven — RGRTest runs every Test-Case carrying @RGR, and the matching .feature file lists all nine siblings in a single block. Verified by grep -nE "^ Scenario:" "src-gen/test/resources/cucumber/specs/Language Definition/Issues/Validation for Workspace Issues.feature":

 10:  Scenario: This object doesn't exist validation
 26:  Scenario: This object step definition doesn't exist validation
 44:  Scenario: This object step definition parameter set doesn't exist validation
 68:  Scenario: This object step definition text parameter exists validation
 89:  Scenario: This object step definition text parameter exists with other parameters validation
111:  Scenario: This object exists validation
126:  Scenario: This object step definition exists validation
141:  Scenario: This object step definition parameter set exists validation
164:  Scenario: This object step definition text parameter doesn't exist validation

The negation pair sits 96 lines apart in the same file. Both worker sessions read this file (call 13 in each timeline). The data is in context.

The miss is therefore not data availability — it is the prompt failing to direct Claude to scan against the negation pairs before editing. fix.md directs Claude to use ${shortlistPath} and the UML interaction specs to find the classes to edit; it says nothing about scanning the scenarios in the runner’s own .feature to spot pairs whose pass-state the edit must preserve. R52 read the file, focused on the failing TC, edited a code path that satisfies it, and submitted — without re-scanning the sibling list. R53 wrote a guarded implementation that happened to preserve the sibling, but largely by aligning on the existing validateStepParameterListWorkspace pattern, not by an explicit sibling-check pass.

The Phase Verification check then runs the whole runner — every @RGR Test-Case — so R52’s edit fails on the sibling and the build retries.

Place: prompt (fix.md behavioral instruction; no new data file is needed because the @RGR tag already scopes the sibling set).

The Fix

Proposed scope (no issue filed yet — propose filing one):

  1. fix.md behavioral instruction. Add one paragraph telling Claude that the runner is tag-driven and every sibling Test-Case sharing the runner’s tag is part of the Verify check. Recommend the worker scan the runner’s .feature for negation pairs (X exists / X doesn't exist, is Y / isn't Y, with / without) before submitting an edit. The runner’s tag is already known to GreenPhase (used to build the runner-class name) and can be templated in as ${runnerTag}.
  2. No new precompute file. The @RGR tag scopes the sibling set; the .feature file already lists the siblings in one block and is already in Claude’s read history. Adding a precomputed sibling-scenarios.md would re-materialize data that is intrinsic to the tag-filtered runner.
  3. No code-side enforcement. The Verify check already catches the regression; the goal is to shave the retry cost (1m 34s on this run) by getting it right on the first attempt.

The fix is in the prompt’s behavioral framing, not in any precompute or data shape. R52 had the sibling visible in its own context window and still missed it — that is the gap to close. Tightening the directive turns a passive read into an active check.

This is orthogonal to #404. #404 addresses search-strategy variance via the JaCoCo coverage shortlist; this addresses sibling-regression variance via prompt phrasing. The two fixes stack independently — #404 narrows the where-to-edit search, this narrows the will-the-edit-regress-a-sibling check.

Mapping to the Research

Predicted (pbc-research) Observed
Wide pair-range green-phase +3.0 min, outside UCL (flag: out)
Each run individually within in-control band no — this pair is the only out-of-control row on the gsheet
Both runs pass the test yes
Two work-trees differ yes — R52 adds a TestStepIssueTypes enum value, R53 reuses an existing one; both pass

The signature differs from pbc-4445 (different files patched, both valid) and from pbc-4849 (same files patched, identical artifacts, pure-path variance). Here the variance is two-layer:

  1. Search-strategy variance (1m 24s) — same shape as pbc-4849’s bash-call variance, different prompt step.
  2. Sibling-test regression cost (1m 34s) — new shape, not present in pbc-4445 or pbc-4849.

A study that only counted bash-call variance would attribute everything to layer 1 and recommend tightening exploration. The actual dominant cost on this pair is layer 2 — the broken-sibling retry — and the recommended fix sits at a different prompt step.

Findings by Variable

Each subsection records this run’s findings about one Wheeler variable. Read the same heading across the run sequence to see how our understanding of that variable evolved.

green time pair range

This is the out-of-UCL case on its tab — the only row flagged flag: out, making it the cleanest assignable signal in the recent series. The green-phase pair-range was 178,631 ms (3.0 min) against a range_UCL of 170,071 ms. Pair-range outside UCL proved a stronger signal than pair-range alone: the flagged row had the cleanest divergence, while other rows with positive pair-range stayed inside the band and would have been less informative to investigate. The wide pair traced to two stacked layers: search-strategy variance (filename-pattern-first Glob exploration in the slow run vs content-first Grep in the fast run, ~1m 24s) and a sibling-test regression that forced a build-retry (~1m 34s). A study counting only search-strategy variance would undercount this pair’s cost; the dominant layer here is the broken-sibling retry.

green time pair range moving range

No finding this run.

green time

The slow run’s absolute green time was inflated by a build-retry: its first mvn clean verify failed because a sibling negation-paired Test-Case (text parameter exists validation) regressed — the new code path returned Present for any IText cursor, breaking the case where the step-parameter Content does exist. Recovery cost ~1m 34s of whack-a-mole (two Greps, a sibling .feature Read, one corrective Edit, second mvn). Root cause is a prompt gap: fix.md does not direct Claude to scan the runner’s .feature for negation pairs whose pass-state the edit must preserve, even though the runner is tag-driven (@RGR) and that sibling list is already in context. The Verify check runs the whole runner class, so the regression surfaces only after the retry — making this extra build cycle pure additive work the first attempt should have avoided.

green time moving range

No finding this run.

scale & green tokens

The two runs ran at different times of day (Rebuild52 at 23:48Z, Rebuild53 at 03:02Z), so part of the green-phase wall-clock gap may carry an unquantified server/time-of-day confound layered on top of the diagnosed search-strategy and sibling-retry costs. Output tokens were comparable (14,422 slow vs 12,826 fast), so the variance is not token-volume driven.

warm-up position

No finding this run.

Open Questions From This Case

  • The 1m 24s search-strategy variance is independent of the 1m 34s retry variance. Does a fix.md negation-pair instruction help on the retry only, or does framing the runner as tag-driven also shorten exploration (Claude may grep less if it knows the .feature covers all siblings)? Re-running the same scenario after the prompt edit would tell.
  • R52’s commit ships a TestStepIssueTypes enum value with description text that duplicates an existing enum’s description. The refactor phase did not collapse the duplicate. A separate study could check how often green-phase additive edits leave duplicates that refactor doesn’t fold — that may be a third remediation site (refactor-prompt awareness of enum-value collisions).
  • The two-prompt split (identify.md then fix.md) is unchanged from pbc-4849. On this case, identify ran short (40–66 s) and added no signal; fix carried 100% of the variance. If a future case shows identify-side variance again, collapsing back into one prompt may be re-litigated.