Context

This page is a worked comparison in the same tradition as pbc-4445. One Darmok scenario ran on consecutive days against sheep-dog-grammar, on branches Rebuild48 (May 23) and Rebuild49 (May 24). The pair-range was wide enough to inspect; the diagnosis traces to the same prompt-template site as the pbc-4445 cases but with a sharper proximate cause — the prompt asks Claude to mine a JaCoCo annotation (title="<test-name>") that the project’s pom does not produce.

  Case
Scenario This object step definition parameter set doesn't exist validation
Rebuild48 commit 965a81ab
Rebuild49 commit dfa011f1
Green-phase range 2.0 min (121,761 ms)
Files patched (same — new feature path, no main-code patch in green)
Diagnosed cause prompt — JaCoCo cross-reference asks for data the report does not contain

The remediation is filed as issue #404: precompute the JaCoCo coverage shortlist inside GreenPhase and have identify.md reference the resulting file rather than scrape per-line title= attributes that don’t exist.

What We Observed

  Rebuild48 (May 23) Rebuild49 (May 24)
Commit 965a81ab dfa011f1
Phase total 793,422 ms 1,084,929 ms
Red phase 70,515 ms 78,639 ms
Green phase 452,852 ms 574,613 ms
Refactor phase 177,260 ms 350,078 ms

Green-phase pair-range = 121,761 ms (2.0 min) on a green baseline that has been running 4–9 min on this scenario across the rebuild series. This study focuses on the green phase only; refactor variance is real but is treated separately.

The two runs ran the same Darmok flow (identify.md → fix.md, one --session-id per phase, auto-compact between them as configured). Both runs succeeded; the test passed in both. Total output-token volume in identify is essentially identical (16,488 vs 16,328) — the variance is not “Claude wrote more text,” it is “Claude ran more tool calls.”

Where The Time Went

The session JSONLs were parsed to extract per-segment counts.

  R48 identify R49 identify R48 fix R49 fix
Wall-clock 342 s 436 s (+94 s) 201 s 217 s (+16 s)
Assistant turns 75 83 14 21
Thinking blocks 16 17 1 2
Output tokens 16,488 16,328 4,074 4,281
Cache-read tokens 3.55 M 3.95 M 0.94 M 1.51 M
Bash calls 23 37 (+14) 1 1
Read 13 9 0 2
Edit / Write 0 0 4 4
mvn verify duration — — 20.4 s 20.7 s

Essentially all of the gap is in identify, and identify is bash-bound. The mvn verify step in fix was wall-clock-identical (20.4 vs 20.7 s); the +16 s in fix is the 7-turn difference at ~2 s per Claude turn round-trip.

The 10 vs 15 JaCoCo bash calls

Both runs started with the same shape: grep -rl "<scenario>" against target/site/jacoco-with-tests/. Both received (Bash completed with no output) because the full scenario name does not appear in those HTMLs. From there each run made progressively narrower queries to extract test-name annotations.

# R48 (10) R49 (15)
1 grep -rl "<full scenario name>" same
2 grep -rl "parameter set" grep -rl "parameter set doesn"
3 ls jacoco-with-tests/ grep -rl "step definition parameter"
4 grep -rl "step definition parameter"; grep -rl "doesn't exist" grep -rl "doesn.t exist"
5 grep -o 'title="..."' TestStepIssueTypes.java.html same
6 grep -oP 'title="\K..."' TestStepIssueTypes.java.html grep -oP 'Tests: [^<]+' TestStepIssueTypes.java.html
7 grep -n "doesn't exist\|..." TestStepIssueTypes.java.html grep -oP '(?<=<span...>)...validation...'
8 grep -n "...step parameters..." TestObject.java.html grep -c "." && head -5 TestStepIssueTypes.java.html
9 grep -n "...PARAMETER_SET..." TestStepIssueTypes.java.html; ls org.farhan.dsl.issues/ grep -n "doesn.t exist\|..." TestObject.java.html
10 grep -rl "...stepDefinitionParameter..." jacoco-with-tests/ grep -rl "step definition parameter\|..." jacoco-with-tests/
11 — grep -n "..." SheepDogBuilder.java.html
12 — grep -n "..." SheepDogUtility.java.html
13 — ls org.farhan.dsl.issues/
14 — grep -n "..." TestStepIssueDetector.java.html
15 — ls org.farhan.dsl.grammar/ \| grep -i "row\|step.*param\|IStep"

The divergence is at command #6. R48 ran grep -oP 'title="\K[^"]+' (extract title= attribute values), got empty, took the empty result as terminal, and pivoted to keyword grep -n on specific HTMLs. R49 ran grep -oP 'Tests: [^<]+' (a different JaCoCo annotation convention), got empty, tried grep -oP '(?<=<span...>)...' which failed with lookbehind assertion is not fixed length, then ran grep -c "." && head -5 to inspect the file format, then widened to three more .java.html files (SheepDogBuilder, SheepDogUtility, TestStepIssueDetector) before bailing to source-tree search.

R49 spent ~50 s on commands #6–#15 chasing the missing annotation. R48 reached the same fallback in ~12 s.

Root Cause

The title="<test-name>" annotation the prompt instructs Claude to extract does not exist in this project’s JaCoCo reports. Verified by re-running mvn clean verify on sheep-dog-grammar and inspecting a fresh *.java.html:

attribute names present: charset, class, content, encoding, equiv, href,
                         id, lang, onload, rel, src, type, version, xmlns

Only class="fc" / class="nc" (full/no-coverage span markers) and id="Lxx" (line numbers). No per-line test attribution.

jacoco-sessions.html confirms why: exactly one session is recorded —

Session: ubuntu-client-f6c99eb5
Start:   May 24, 2026, 6:22:19 PM
Dump:    May 24, 2026, 6:22:25 PM

The pom configures the standard prepare-agent execution + an Ant ReportTask over a single jacoco.exec. Per-test attribution requires multiple sessions (one execution-data file per test, aggregated). The directory is named jacoco-with-tests (aspirational) but the config produces aggregate-only coverage.

The aggregate coverage data is still useful for the prompt’s stated goal — narrowing Claude’s search to the classes the failing test’s family actually touches, rather than reading the whole tree. The available signal is: production classes with non-zero coverage, sorted by which packages the runner exercises. That signal lives in jacoco.xml (line-level <line nr="N" mi=".." ci=".."/> entries inside <sourcefile> blocks) and jacoco.csv (class-level totals).

The Fix

Issue #404 — precompute a coverage shortlist in GreenPhase and reference it from identify.md.

Concretely:

  1. GreenPhase parses target/site/jacoco-with-tests/jacoco.xml once before invoking Claude.
  2. Writes target/darmok/jacoco-shortlist.md containing:
    • Production classes in the runner-class’s package family, sorted by covered/missed instruction ratio, with per-class covered line ranges
    • Sibling test classes that heavily cover those production classes
  3. identify.md step 2 changes from “grep title= attributes in *.java.html” to “read ${darmokDir}/jacoco-shortlist.md and use the listed classes as the search shortlist.”
  4. The UML/source-tree fallback paths stay in the prompt for the compile-error case where jacoco.xml may be absent.

The change is in the system, not in the worker. The expectation is that the +60–90 s of identify-phase variance attributable to the missing-annotation chase disappears across runs.

Collapsing identify.md + fix.md into one green prompt is not in scope of #404. That decision is deferred until the shortlist’s wall-clock impact has been measured — the two-phase split also serves as a “diagnose-before-edit” guardrail independent of search cost.

Mapping to the Research

Predicted (pbc-research) Observed
Wide pair-range green-phase +2.0 min on a 4–9 min baseline
Each run individually within in-control band yes — both within typical green-phase range for this scenario
Both runs pass the test yes
Two work-trees differ not in this case — green produced the same artifacts; the variance is entirely in how long it took to get there

The signature differs from pbc-4445 in one important way: the pbc-4445 cases had observable artifact differences (different files patched in case 1, different code styles in case 2). The Rebuild48–49 case has no work-tree difference at all — same files, same code. The variance is purely in the path the worker took through the prompt. The signal still fired at 2.0 min on the same prompt step.

This strengthens the pbc-4445 claim rather than weakening it. A range chart that only flagged artifact-differing pairs would miss pure-path variance like this one; the pair-range signal catches it the same way.

Findings by Variable

Each subsection records this run’s findings about one Wheeler variable. Read the same heading across the run sequence to see how our understanding of that variable evolved.

green time pair range

This was a two-run comparison (Rebuild48 vs Rebuild49) with a wide green-phase pair-range of 2.0 min (121,761 ms) on a 4–9 min baseline. The diagnosis was a prompt/producer cause: identify.md instructs Claude to mine a title="<test-name>" JaCoCo annotation that the project’s pom never produces, so each run made progressively narrower bash queries chasing a missing attribute and diverged at command #6 — one run bailed to a source-tree fallback in ~12 s, the other spent ~50 s widening across more .java.html files. Pair-range caught this even though both runs produced identical artifacts (same files, same code); the variance was purely in the path the worker took through the under-specified prompt step. The fix is in the producer (#404): precompute a coverage shortlist from jacoco.xml and have identify.md read it rather than scrape attributes that don’t exist.

green time pair range moving range

No finding this run.

green time

No finding this run.

green time moving range

No finding this run.

scale & green tokens

Total output-token volume in identify was essentially identical between the two runs (16,488 vs 16,328), so the variance was tool-call count, not text generation. The two runs ran on consecutive days at different times, so a server/decode-rate confound was folded into the pair-range here; that confound was only later separated out by token-scaling and concurrent pairing. The prompt/producer cause remains primary for this case.

warm-up position

No finding this run.

Open Questions From This Case

  • The pbc-4445 open question about exploration-depth signatures (Agent subagent count, distinct files read, xargs grep cross-reference count) holds here too. Rebuild48 and Rebuild49 used no subagents; the divergence is in bash variation depth. A “bash retries on the same target file” counter might be the cheap signal for this prompt-shape case.
  • After #404 lands, re-running this scenario 2–3 times on the same branch will show whether identify-phase variance narrows. If it does not, the next assignable cause becomes visible (likely: fix-phase Claude-turn-count variance, currently the +16 s minor signal).
  • The remaining refactor-phase variance (+173 s in this pair) is out of scope here but worth a parallel study. The post-compact session for R49 ran a full Phase 2 behavioral analysis on a new file (RowIssueDetector.java); R48 short-circuited at Phase 1 on a structural rename. Different cause site, different prompt — separate investigation.