Context

This page continues the pbc-4445 / pbc-4849 / pbc-5253 series. The same Darmok scenario ran on consecutive cycles against sheep-dog-grammar, both on 2026-05-26, on branches Rebuild54 and Rebuild55. The pair-range topped the gsheet tab (gid=1837142131) for the run window.

The investigation led to a verified harness bug: the JaCoCo coverage shortlist producer added in #404 (commit abc5e3f, 2026-05-25) has never written a single byte. The Java parser fails on every call with FileNotFoundException: report.dtd, caught silently and returned as null. The narrowing artifact that fix.md step 1 directs Claude to read does not exist on disk, on any run, since the feature shipped.

  Case
Scenario No component in the first step triggers an error validation
Rebuild54 commit 41cf7898
Rebuild55 commit e7955df9
Green-phase range 3.2 min (194,411 ms) — top of gid=1837142131
Files patched (final, post-refactor) R54 and R55: identical — same two files, same enum name, same detector logic
Diagnosed cause harness — DocumentBuilderFactory default loads external DTDs, JaCoCo’s report.dtd not shipped in report dir, parse throws, silent catch returns null

Both runs arrived at the same final fix: an additional enum value TEST_STEP_CONTAINER_STEP_OBJECT_NAME_ONLY in TestStepContainerIssueTypes and a guard in TestStepContainerIssueDetector.validateNameOnly that returns the new description when the first step’s component is empty. There is no spec ambiguity here and no sibling-test retry — pure path variance, attributable to the missing shortlist.

A meta-observation from this case: a buggy harness shows up as a noisy baseline, not as an outright run failure. Both runs passed, both committed, both produced equivalent code. The methodology’s signal is the pair-range itself — once it widens past the in-control band, the cause is somewhere in the system the runs share. Here that “somewhere” was an unfixed JAXP default in the harness’s coverage-shortlist producer, masking a feature that shipped but never actually executed. The case-study series doesn’t just catch model variance; it catches infrastructure variance that the build green-light wouldn’t.

What We Observed

  Rebuild54 Rebuild55
Commit 41cf7898 e7955df9
Phase total 717,341 ms 485,600 ms
Red phase 67,876 ms 75,911 ms
Green phase 419,599 ms 225,188 ms
Refactor phase 149,718 ms 130,788 ms

Green-phase pair-range = 194,411 ms (3:14), the widest on gid=1837142131 for the Rebuild54/Rebuild55 column pair. The local metrics.csv agrees with the sheet to the millisecond — no override divergence.

Both runs ran the same Darmok flow: identify.md then resumed-session fix.md, then /compact, then /rgr-refactor forward sheep-dog-grammar. Single --session-id per scenario throughout.

  R54 session R55 session
Green session UUID a2286b94-26f7-4193-8178-65a7c8ce22a4 b6fe96d2-6066-4999-8a4b-66c234e696dd
identify.md issued 11:35:10 EDT (15:35:10Z) 16:21:49 EDT (20:21:49Z)
fix.md resumed 11:35:39 EDT (+0m 28s) 16:22:20 EDT (+0m 31s)
Green: Completed (mojo) 11:41:49 EDT 16:25:14 EDT
identify wall-clock ~28 s ~31 s
fix wall-clock (identify-end → Green: Completed) ~370 s ~174 s

The +3 s identify gap is noise on a ~30 s baseline. The +196 s fix-phase gap is the assignable signal and accounts for the entire 194,411 ms pair-range.

Each JSONL ends end_turn cleanly with stop_reason: end_turn on the final assistant message — no timeouts, no recoveries. Both runs passed every gate and committed.

Where The Time Went

The two JSONLs were parsed for the full session span. The green-phase window is from session start through the green mvn-verify pass; the rgr-refactor block sits inside the same JSONL afterward and is included below for completeness, but the pair-range is driven by the green-phase portion.

  R54 (slow) R55 (fast)
Wall-clock (full session) 10m 26s 6m 27s
Assistant turns 97 72
Thinking blocks 25 17
Output tokens 13,911 8,440
Cache-read tokens 4.81 M 2.97 M
Total tool calls 58 43
Bash calls 6 6
Read calls 20 13
Glob calls 13 7
Grep calls 5 3
Edit calls 5 5
TodoWrite 7 7
ToolSearch 1 1
Skill 1 1
mvn build cycles 3 (all green-pass + post-refactor) 3 (all green-pass + post-refactor)
Time-to-first-Edit (from fix.md prompt) 4m 31s 1m 46s

The mvn build cycles are identical in count and shape. No retries on either side. The variance lives in pre-edit exploration: R54 made 28 tool calls between the fix.md prompt and its first Edit (≈4:31); R55 made 14 (≈1:46). The +2:45 time-to-first-Edit delta accounts for ~85% of the green-phase pair-range; the remaining ~30s is small-jitter spread across the validate/refactor cycle.

The 14 vs 28 fix-phase tool calls

Both sessions opened with the same shape — calls 1–10 are functionally identical (ToolSearch:TodoWrite, TodoWrite seed, Grep log.txt for errors, TodoWrite, Read jacoco-shortlist.md → file does not exist on both sides, TodoWrite, Read uml-interaction-main.md, Read uml-interaction-test.md, TodoWrite). The divergence point is call 11:

# R54 (slow) R55 (fast)
1–10 (identical: identify phase + fix.md preamble — same 10 calls) same
11 Glob **/*Validation*File*Issues* Grep "No component in the first step" sheep-dog-grammar
12 Read .../Validation for File Issues.feature (whole file) Grep "No component in the first step" .../Validation for File Issues.feature
13 Glob **/ValidateActionImpl.java Glob **/ValidateActionImpl.java
14 Read .../ValidateActionImpl.java Read .../ValidateActionImpl.java
15 Glob **/TestStepContainerIssueDetector.java Glob **/TestStepContainerIssueDetector.java
16 Read .../TestStepContainerIssueDetector.java Read .../TestStepContainerIssueDetector.java
17 Glob **/TestStepContainerIssueTypes.java Glob **/TestStepContainerIssueTypes.java
18 Read .../TestStepContainerIssueTypes.java Read .../TestStepContainerIssueTypes.java
19 Glob **/ITestStepContainer.java Glob **/StepObjectRefFragments.java
20 Read .../ITestStepContainer.java Read .../StepObjectRefFragments.java
21 Glob **/StepObjectRefFragments.java Glob **/ITestStepContainer.java
22 Read .../StepObjectRefFragments.java Read .../ITestStepContainer.java
23 Glob **/ITestStep.java Glob **/ITestStep.java
24 Read .../ITestStep.java Read .../ITestStep.java (limit=60)
25 Glob **/StepObjectRefComponentTypes.java Edit TestStepContainerIssueTypes.java — first edit
26 Read .../StepObjectRefComponentTypes.java Edit TestStepContainerIssueDetector.java (imports)
27 Glob **/RGRTest.java Edit TestStepContainerIssueDetector.java (guard logic)
28 Read .../RGRTest.java Bash mvn verify -Dtest=RGRTest
29 Grep "@RGR" src-gen/test/resources/cucumber (continues to mvn-pass + skill)
30 Read .../Validation for Only Issues.feature (peer file) —
31 Glob **/EMFTestObject.java —
32 Read .../EMFTestObject.java —
33 Glob **/*ProcessIssues*Impl.java —
34 Grep "setTestStepListNode|setCreated|TestStepFullName|testStepList" ProcessIssuesAsciidocFile... —
35 Glob **/TestStepNameParser.java —
36 Read .../TestStepNameParser.java —
37 Glob **/StepObjectRefObjectVertexTypes.java —
38 Read .../StepObjectRefObjectVertexTypes.java —
39 Edit TestStepContainerIssueTypes.java — first edit —

R55 reached the first Edit after 14 tool calls, going straight from Read ITestStep.java to Edit TestStepContainerIssueTypes. R54 took 28 tool calls, spending its second-half exploration on artifacts that turned out to be irrelevant to the fix:

  • StepObjectRefComponentTypes.java — sibling enum, not referenced by the final edit
  • RGRTest.java and the @RGR grep — runner harness, not modified
  • Validation for Only Issues.feature — the peer spec file (different scenario, different observable), opened and read in full
  • EMFTestObject.java — base test-object class
  • *ProcessIssues*Impl.java and the stepdef grep — impl-side machinery
  • TestStepNameParser.java — fake/parser helper used by impls
  • StepObjectRefObjectVertexTypes.java — another enum, also not referenced

None of these files were modified by either run’s final fix. R55’s strategy was content-first (Grep for the scenario text, follow the matches); R54’s strategy was filename-pattern-first plus speculative neighbourhood expansion — Glob the next plausible filename, Read it, repeat until the model felt informed.

The first Edits in both runs are nearly identical (modulo enum name, which the post-mvn refactor step then normalizes to the same final value — see below).

The post-mvn refactor cycle

Both runs reach a green mvn-verify pass on the first Edit batch. Both then proceed: Skill rgr-refactor forward sheep-dog-grammar → validate_main.py → finds one violation: the enum name does not match the {TYPE}_{ASSIGNMENT}_{SCOPE} convention required by the UML uml-class-TypeIssueTypes.md regex. Both runs rename to the same compliant name:

  R54 first-Edit enum R55 first-Edit enum Final enum (after refactor cycle)
Name TEST_STEP_CONTAINER_FIRST_STEP_COMPONENT_ONLY TEST_STEP_CONTAINER_FIRST_STEP_COMPONENT TEST_STEP_CONTAINER_STEP_OBJECT_NAME_ONLY
Description "The first step must have a component" "The first step must have a component" "The first step must have a component"

R54 wrote it once with the _ONLY suffix (which still failed the assignment-segment regex because FIRST_STEP_COMPONENT is not in the valid assignment set); R55 wrote it without _ONLY (also failed for the same regex reason). Both converged on STEP_OBJECT_NAME_ONLY after the refactor pass — STEP_OBJECT_NAME is a valid assignment, ONLY is a valid scope, and the description text describes the same observable.

The final commit diffs are equivalent: same two files (TestStepContainerIssueTypes.java, TestStepContainerIssueDetector.java), same enum name, same detector logic shape. The pair-range is therefore not driven by spec ambiguity, search strategy producing different fixes, or sibling-test regression — it is pure exploration overhead before the first byte of the same edit.

Root Cause

Both runs called Read /home/farhan/git/sheep-dog-main/sheep-dog-core/sheep-dog-grammar/target/darmok/jacoco-shortlist.md and both received:

File does not exist. Note: your current working directory is /home/farhan/git/sheep-dog-main.

The shortlist is the artifact fix.md step 1 directs Claude to read first — “if populated it narrows your search to the classes other passing tests in this family already exercise plus the production classes those tests cover.” It is produced by GreenPhase.computeJacocoShortlist(targetDir), which parses target/site/jacoco-with-tests/jacoco.xml and writes a single-table list of classes with covered line ranges. The producer was added in commit abc5e3f on 2026-05-25 — before both Rebuild54 and Rebuild55 (2026-05-26), so the code was present in the build.

The mojo log on 2026-05-26 has zero occurrences of Green: Writing jacoco-shortlist.md:

$ grep -c "Green: Writing jacoco-shortlist" /logs/darmok.mojo.2026-05-26.log
0

GreenPhase.computeJacocoShortlist returned null for every scenario that day. The producer has three null exits:

  1. if (!Files.exists(jacocoXml)) return null;
  2. catch (Exception e) { return null; } — silently swallowed parse failure
  3. if (classCoveredLines.isEmpty()) return null;

jacoco.xml is present on disk (517 KB on the case-study run’s checkout; produced by the red phase’s mvn verify -Dtest=<RunnerClass>, since the pom binds jacoco-ant’s report-with-tests execution to the verify phase). The third path can’t fire either — sheep-dog-grammar has 129 sourcefiles and many covered lines after a green test run, not zero.

The actual failing branch is the silent catch, reproduced directly against the on-disk file:

$ cat TestJacocoParse.java
import javax.xml.parsers.DocumentBuilderFactory;
import java.io.File;
public class TestJacocoParse {
    public static void main(String[] args) throws Exception {
        var doc = DocumentBuilderFactory.newInstance().newDocumentBuilder()
            .parse(new File(args[0]));
        System.out.println("OK");
    }
}

$ java TestJacocoParse .../target/site/jacoco-with-tests/jacoco.xml
PARSE FAILED: java.io.FileNotFoundException:
  .../target/site/jacoco-with-tests/report.dtd (No such file or directory)

JaCoCo’s XML report opens with a DOCTYPE:

<!DOCTYPE report PUBLIC "-//JACOCO//DTD Report 1.1//EN" "report.dtd">

report.dtd is a relative system identifier. Java’s default DocumentBuilderFactory resolves it next to the XML file. JaCoCo doesn’t ship report.dtd in the report directory (ls target/site/jacoco-with-tests/ confirms — index.html, jacoco.csv, jacoco.xml, per-package directories, and nothing else). The parse throws FileNotFoundException on every call. The default JAXP parser would do this on every machine — Linux, Mac, Windows, CI, local. It has been throwing on every Darmok green-phase since #404 landed.

The exception was swallowed by catch (Exception e) { return null; } at the bottom of the method, and the surrounding code branches on if (shortlist != null) — so null silently means “no shortlist this round, no log entry, fall back to UML guidance.” That fallback is what fix.md is told to lean on when the shortlist is empty; the prompt has no way to distinguish “shortlist absent because nothing’s covered” from “shortlist absent because the producer threw.”

Why every scenario, not just one

There is no per-scenario filtering in computeJacocoShortlist — it parses whatever jacoco.xml is at target/site/jacoco-with-tests/jacoco.xml and emits a coverage table for the whole run. The DOCTYPE is identical across jacoco’s output, so the parse fails the same way on every call. The producer is a per-run no-op disguised as a per-scenario no-op.

Why this wasn’t caught by tests

The producer’s unit-test coverage doesn’t exercise the actual JaCoCo XML output — it would have needed an end-to-end repro reading a real jacoco.xml from a green-phase mvn verify, or a fixture file containing the DOCTYPE. The producer ships, runs, returns null, and nothing fails. The only downstream consumer is Claude, which sees “file doesn’t exist” and falls back to UML guidance — also not a failure mode that bubbles up.

The way this surfaced was the pair-range methodology: a feature shipped to narrow Claude’s search across runs, yet the inter-run variance on shared scenarios stayed in the same band. That’s the signal — the fix shipped but the variance it was supposed to reduce didn’t move.

Place: harness — producer. The fix lives in GreenPhase.computeJacocoShortlist.

The Fix

Patch applied (two edits to GreenPhase.java):

// Disable external DTD loading on the parser.
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
factory.setFeature("http://apache.org/xml/features/nonvalidating/load-external-dtd", false);
factory.setAttribute(XMLConstants.ACCESS_EXTERNAL_DTD, "");
Document doc = factory.newDocumentBuilder().parse(jacocoXml.toFile());

And remove the silent catch — let any producer failure bubble up with an ERROR log at the call site:

String shortlist;
try {
    shortlist = computeJacocoShortlist(targetDir);
} catch (Exception e) {
    mojoLog.error("Green: jacoco-shortlist.md producer failed: "
        + e.getClass().getName() + ": " + e.getMessage());
    throw e;
}

Reproduced the fix against the on-disk jacoco.xml: PARSED OK, root: report — packages: 16, sourcefiles: 129. mvn compile on darmok-maven-plugin succeeds.

The Files.exists(jacocoXml) path is also tightened: a missing jacoco.xml after the red phase’s mvn verify ran is a fatal upstream invariant violation (the jacoco-ant report-with-tests execution failed to bind, the target dir was wiped between phases, etc.), not a state to silently degrade through. Throws IllegalStateException with the resolved path. The classCoveredLines.isEmpty() null-return is kept — that represents a legitimate “the runner exercised nothing” state, which is data, not a bug. The catch-and-swallow path is removed entirely: “I couldn’t parse the file you handed me” is never a state worth pressing on through.

Place: harness — producer. No spec change, no prompt change, no test-side change required. The producer now does what it was claimed to do in #404’s commit message.

Mapping to the Research

Predicted (pbc-research) Observed
Wide pair-range green-phase +3:14, top of gid=1837142131
Each run individually within in-control band not flagged out-of-control by the gsheet on this tab, but the widest on the pair
Both runs pass the test yes
Two work-trees differ no — final commits are equivalent (same two files, same enum name, same logic). Pure path variance, pbc-4849-style.

The signature matches pbc-4849 (same files patched, identical final artifacts, pure path variance) and not pbc-4445 (different files patched, both valid) or pbc-5253 (two-layer: search + sibling-regression). The cost site is upstream of where pbc-4849 sat — pbc-4849 diagnosed which command Claude typed given a narrowing prompt; this case diagnosed why Claude was free to type anything at all because the narrowing artifact never existed.

Findings by Variable

Each subsection records this run’s findings about one Wheeler variable. Read the same heading across the run sequence to see how our understanding of that variable evolved.

green time pair range

The green-phase pair-range topped gid=1837142131 at 194,411 ms (3:14) — yet both runs produced identical final commits (same two files, same enum name, same detector logic). A wide pair with identical output is the headline of this case: it exposed a harness bug, not model variance. The JaCoCo coverage-shortlist producer added in #404 never wrote a byte — a report.dtd FileNotFoundException was swallowed silently by catch (Exception e) { return null; }, so the narrowing artifact fix.md step 1 reads never existed on any run since the feature shipped. With no shortlist to narrow the search, both runs reverted to wide pre-edit exploration (R54 took 28 tool calls to first Edit, R55 took 14), and the pair-range stayed in the same band the feature was meant to shrink. The pair-range methodology catching a buggy harness — a feature that shipped but never executed, invisible to CI because both runs still passed — is what makes this case distinct.

green time pair range moving range

No finding this run.

green time

The slow half (R54) ran a full session of 10m 26s vs R55’s 6m 27s; the assignable signal was the +196 s fix-phase gap (R54 ~370 s, R55 ~174 s), accounting for the entire pair-range. The variance was pure exploration overhead before the first byte of the same edit — no timeouts, no recoveries, no contradiction; both JSONLs ended end_turn cleanly and both committed. R54’s +2:45 time-to-first-Edit delta accounted for ~85% of the green-phase pair-range, the rest small-jitter across the validate/refactor cycle.

green time moving range

No finding this run.

scale & green tokens

R54 emitted 13,911 output tokens vs R55’s 8,440, tracking the heavier exploration (20 vs 13 Read calls, 13 vs 7 Glob calls, 4.81 M vs 2.97 M cache-read tokens). The two runs ran at different times of day (R54 issued identify.md at 11:35 EDT, R55 at 16:21 EDT), so a server/time-of-day decode-rate confound is present, but the dominant delta here is the tool-call count driven by the missing shortlist, not throughput noise.

warm-up position

No finding this run.

Open Questions From This Case

  • The patched producer will start logging at INFO (“Green: Writing jacoco-shortlist.md”) on the next Darmok run. The follow-up question is whether the resulting shortlist actually shrinks the pair-range on this scenario. Re-running Rebuild56/Rebuild57 after the patch lands would tell — same Test-Case, same model, but fix.md step 1 is no longer a no-op.
  • A grep for DocumentBuilderFactory.newInstance() across the broader sheep-dog tree would surface other JAXP-default-DTD-loading sites. The asciidoc/UML pipeline parses XML in several places; whether any of those hit the same trap is worth a 5-minute audit.
  • The patch also makes a missing jacoco.xml fatal — an upstream invariant violation rather than a silently-skipped state. Worth a separate scan of red-phase / pom configuration to confirm jacoco-ant’s report-with-tests execution always fires when red’s mvn verify runs, so the new throw never surfaces in normal operation.
  • The two-prompt split (identify.md then fix.md) is unchanged. On this case, identify ran short (28–31 s) and added no signal; fix carried 100% of the variance. If a future case shows identify-side variance again, collapsing back into one prompt may be re-litigated.