Context

This page continues the pbc-4445 / pbc-4849 / pbc-5051 / pbc-5253 / pbc-5455 series. The same Darmok scenario ran on two cycles against sheep-dog-grammar: Rebuild56 (2026-05-26 late evening) and Rebuild58 (2026-05-27 evening). Both runs are post the harness fix that pbc-5455 diagnosed (commit abc5e3f, Precompute JaCoCo coverage shortlist in GreenPhase, reference from identify.md) — the shortlist producer now writes its artifact on every scenario, observable as 12 occurrences of Green: Writing jacoco-shortlist.md in the 2026-05-26 mojo log and 17 in 2026-05-27. That regression case is closed.

The pair-range topped the gsheet tab (gid=886919682) for the run window with a different scenario — This object step definition text parameter exists with other parameters validation, green delta 229,852 ms (~3:50). Inspection isolates two findings:

  1. A new identify-phase variance (152 s, ~66% of the gap) driven by a Claude Code Glob tool quirk: when given pattern=<absolute path>, the tool finds files inside gitignored directories; when given pattern=<relative> + path=<projectRoot>, the gitignore filter trips and the same file returns “No files found.” identify.md has no explicit no-error path, so each run improvises a different verification — one form found the file, the other didn’t, and the slower run spent 196 s reasoning about a missing-but-present jacoco.xml and even launched a mvn verify to “repopulate” it.
  2. A 78 s residual fix-phase variance of the same shape as pbc-4849 / pbc-5455: search-strategy drift before the same single edit lands in TextIssueDetector.java. Even with the shortlist now populated, claude’s pre-edit exploration still spans different breadths.
  Case
Scenario This object step definition text parameter exists with other parameters validation
Rebuild56 commit 6cadef46
Rebuild58 commit 17e5b05f
Green-phase range 229,852 ms (~3:50) — top of gid=886919682
Files patched (green phase) R56 and R58: same file (TextIssueDetector.java), structurally different patch site
Diagnosed cause prompt + tool — identify.md has no explicit no-COMPILATION-ERROR exit; both runs improvise, one form of the improvised Glob trips Claude Code’s gitignore-aware path= mode

Unlike pbc-5455, where both final commits were identical, run-56 and run-58 patched different lines of the same file with structurally different patches — both valid for the failing assertion. The fix-phase signature here is pbc-4445-shaped (different patches, both valid) overlaid on the pbc-4849 / pbc-5455 pre-edit-exploration cost. The identify-phase signature is new.

What We Observed

  Rebuild56 Rebuild58
Commit 6cadef46 17e5b05f
Phase total 666,463 ms 707,786 ms
Red phase 76,130 ms 78,481 ms
Green phase 195,030 ms 424,882 ms
Refactor phase 321,334 ms 127,537 ms

Green-phase pair-range = 229,852 ms (~3:50), the widest on gid=886919682 for the Rebuild56/Rebuild58 column pair. The local metrics.csv agrees with the sheet to the millisecond. (Refactor went the other way — R56 refactor is 194 s longer than R58 — but refactor variance is not what gid=886919682 flagged and is out of scope for this case study; refactor in this scenario for R56 also did three additional Edit calls that R58 did not. Worth a future pair-range tab.)

Both runs ran the same Darmok flow: identify.md then resumed-session fix.md, then /compact, then /rgr-refactor forward sheep-dog-grammar. Single --session-id per scenario throughout. Both reached end_turn cleanly; no timeouts, no retries.

  R56 session R58 session
Green session UUID 1acd2421-1bae-4419-ae94-9fd0f4735210 9658e7fb-ad75-49e0-b51a-1e5a68076c19
identify.md issued 23:17:00 EDT (03:17:00Z) 17:51:43 EDT (21:51:43Z)
Green: Writing jacoco-shortlist.md (mojo) 23:17:44 EDT (+0m 44s) 17:54:59 EDT (+3m 16s)
fix.md resumed 23:17:44 EDT 17:54:59 EDT
Green: Completed (mojo) 23:19:55 EDT 17:58:27 EDT
identify wall-clock ~44 s ~196 s
fix wall-clock (identify-end → Green: Completed) ~131 s ~208 s

The +152 s identify gap accounts for ~66% of the green-phase pair-range; the +78 s fix gap accounts for the rest. Per-segment breakdown lives in the next section.

Green: Writing jacoco-shortlist.md is emitted by the mojo between identify.md ending and fix.md resuming — the precompute step gates the resume. So the identify-phase wall-clock equals the time claude spent on identify.md plus the time mojo spent computing the shortlist. Both runs’ shortlist-write time is sub-second (the JaCoCo XML is the same shape both days), so the identify-phase delta is entirely claude’s wall-clock.

Where The Time Went

The two JSONLs were parsed for the full session span. Refactor-phase events are included for completeness but the pair-range is driven by the green-phase portion.

  R56 (fast) R58 (slow)
Wall-clock (full session) 9m 28s 10m 6s
Assistant turns (tool_use) 30 43
Stop reasons end_turn×1, stop_sequence×1, tool_use×30 end_turn×4, stop_sequence×1, tool_use×43
Output tokens 13,270 19,642
Cache-read tokens 4.56 M 5.96 M
Total tool calls 51 60
Bash calls 11 12
Read calls 18 19
Glob calls 3 6
Grep calls 5 12
Edit calls 4 1
TodoWrite 7 8
ToolSearch 1 1
Skill 1 1
Write 2 0
identify-phase tool calls 6 23
fix-phase tool calls 14 20

The deltas line up with the segment timing: 6→23 identify-phase tool calls (delta 17, average ~9 s/call) explains the +152 s identify gap, and 14→20 fix-phase tool calls (delta 6) plus a wider grep set explains the +78 s fix gap.

The 6 vs 23 identify-phase tool calls

Both sessions opened with the same shape — calls 1–4 are functionally identical:

# R56 (fast) R58 (slow)
1 ToolSearch select:TodoWrite ToolSearch select:TodoWrite
2 TodoWrite (seed list per identify.md) TodoWrite (seed list per identify.md)
3 Grep log.txt for COMPILATION ERROR\|FAILED\|ERROR\|Tests run.*Failures\|BUILD FAILURE Grep log.txt (same regex, same path)
4 TodoWrite (mark task 1 completed) TodoWrite (mark task 1 completed)

Both runs reach the same conclusion at the end of call 4: “No COMPILATION ERROR found. The log shows only test failures, not compilation errors.” identify.md’s tasks 2a/2b/2c (read UML specs, stub classes, loop mvn verify) are explicitly conditional on COMPILATION ERROR being present — so when it isn’t, the prompt is silent on what to do next. Both runs improvise a “let me at least confirm jacoco.xml is there” check. Call 5 is where they diverge:

# R56 (fast) R58 (slow)
5 Glob pattern="/home/farhan/.../target/site/jacoco-with-tests/jacoco.xml" (absolute, no path) → numFiles: 1 Glob pattern="target/site/jacoco-with-tests/jacoco.xml" path="/home/farhan/.../sheep-dog-grammar" → numFiles: 0, "No files found"
6 TodoWrite (mark tasks complete, identify exits) Grep "jacoco" pom.xml (start of a 18-call detour)

R56’s identify ends here at call 6 — claude saw numFiles: 1, concluded the precondition is met, and returned. The fix.md resume fires 44 s after identify.md was issued.

R58’s identify continues for 18 more tool calls:

# R58 call Purpose
6 Grep "jacoco" pom.xml Looking for how jacoco is configured
7 Read pom.xml (lines 285–334) Reading jacoco-ant report-with-tests execution config
8 Glob target/jacoco.exec path=<projectRoot> Check the agent’s binary output — also returns numFiles: 0 (target/ filtered)
9 Read pom.xml (lines 250–289) Re-read pom for the prepare-agent execution
10 Read log.txt (offset 1620, limit 50) Look for jacoco lifecycle in the existing log
11 Read log.txt (offset 1, limit 20) Look at log header
12 Bash mvn verify -Dtest=RGRTest -Dmaven.test.failure.ignore=true > log.txt 2>&1 & Launch a fresh mvn verify in background to “repopulate” jacoco.xml
13 Glob target/site/jacoco-with-tests/jacoco.xml path=<projectRoot> Re-check — still numFiles: 0
14 Grep "jacoco\|COMPILATION ERROR\|BUILD" log.txt Look at the new log
15 Grep "jacoco\|antrun\|BUILD\|verify" log.txt offset=1640 head=50 Look deeper
16 Bash wc -l log.txt Check log length
17 Read log.txt (offset 240, limit 30) Read mid-log
18 Bash tail -5 log.txt && wc -l log.txt Tail it
19 Glob target/site/jacoco-with-tests/jacoco.xml path=<projectRoot> Re-check (third time) — still numFiles: 0
20 Grep "antrun\|jacoco-report\|verify\|jacoco.xml" log.txt offset=1880 Look for the antrun execution
21 Read log.txt (offset 1880, limit 41) Read antrun output
22 Glob target/site/jacoco-with-tests/jacoco.xml path=<projectRoot> Re-check (fourth time) — still numFiles: 0
23 Bash ls target/site/ 2>&1 && ls target/site/jacoco-with-tests/ Finally fall back to ls — and the ls shows jacoco.xml is right there
(24) TodoWrite (mark identify done) identify exits

Call 23 is the moment R58 sees the file. By then 196 s have elapsed.

R58 ran mvn verify in the background (call 12) — but & returned immediately with PID=$!, so claude wasn’t blocked on the build. Claude was blocked on its own retry loop on a Glob that kept returning empty. The mvn verify did re-run jacoco-ant in the background, which is why the eventual ls (call 23) showed the file freshly present at the same path the Globs were querying — but that fresh write only confirmed what the file was already showing pre-run-58.

The Glob argument forms in question:

Form R56 call 5 R58 call 5
pattern "/home/farhan/git/sheep-dog-main/sheep-dog-core/sheep-dog-grammar/target/site/jacoco-with-tests/jacoco.xml" (absolute) "target/site/jacoco-with-tests/jacoco.xml" (relative)
path (omitted) "/home/farhan/git/sheep-dog-main/sheep-dog-core/sheep-dog-grammar"
Result {"filenames": [".../jacoco.xml"], "numFiles": 1, "durationMs": 27} {"filenames": [], "numFiles": 0, "durationMs": 20}

Same file. Same project. Same checkout state. Different Glob behavior. **/target is in .gitignore at the repo root — the relative-pattern + path= form respects gitignore and skips target/; the absolute-pattern form bypasses gitignore filtering because the explicit absolute path is treated as user-asserted scope.

The 14 vs 20 fix-phase tool calls

Both sessions read jacoco-shortlist.md (now populated, thanks to pbc-5455’s fix), both read uml-interaction-main.md and uml-interaction-test.md, both grep for the scenario text, both read TextIssueDetector.java. The first 9 calls are functionally identical. Divergence is at calls 10+:

# R56 (fast) R58 (slow)
1–9 (identical: jacoco-shortlist + UML reads + scenario grep + TextIssueDetector.java read) same
10 Grep "getCellListAsString" SheepDog… Grep "interface IStepParameters" src-gen
11 Edit TextIssueDetector.java — first edit at the paramCells.equals("Content") block Read IStepParameters.java
12 Bash mvn verify -Dtest=RGRTest Grep "stepParameters\|StepParameters\|setName" test/impl
13 Bash mvn verify (full regression) Grep "Step Parameters Name\|StepParameters\|getStepParametersList" test
14 TodoWrite (mark complete, fix exits) Grep "StepParameters\|stepParametersList\|Content" main
15 — Grep "getRowList\|getCellList" src-gen
16 — Grep "String getName" ICell.java
17 — Edit TextIssueDetector.java — first edit at the anyMatch(sp -> "Content".equals(sp.getName())) block
18 — Bash mvn verify -Dtest=RGRTest
19 — Bash mvn verify (full regression)
20 — TodoWrite (mark complete)

R56 reaches the first Edit after 10 fix-phase tool calls, going from Grep getCellListAsString straight to Edit. R58 takes 16 fix-phase tool calls, spending an extra 6 Grep/Read calls walking the grammar interface hierarchy (IStepParameters, ICell, getRowList, getCellList, getName) before deciding where to patch.

Same logical bug. Same file. Different patch sites. Both passed mvn verify.

  R56 edit R58 edit
Patch site Inside the nested for (IStepParameters sp ...) { ... paramCells.equals("Content") ... } block (legacy code path) Inside the anyMatch(sp -> "Content".equals(sp.getName())) block (newer stream code path)
old_string shape Replaces the SheepDogUtility.getCellListAsString(...).equals("Content") joined-string check Replaces the anyMatch(sp -> "Content".equals(sp.getName())) name-only check
new_string shape Adds a nested for (ICell cell : ...) { if (cell.getName().equals("Content")) ... } loop Adds a nested anyMatch(cell -> "Content".equals(cell.getName())) stream
Logic Walk cells of row 0, look for one named "Content" Walk cells of row 0, look for one named "Content"

Same final behavior, two structurally distinct edits to the same file. R56 then made three more Edits during the refactor phase (additional import, replace org.farhan.dsl.grammar.ICell with ICell, restructure if (!found)); R58 made none during refactor. The post-refactor commits are not byte-identical, but functionally equivalent.

Root Cause

Two contributors, in priority order.

(1) identify.md has no explicit no-COMPILATION-ERROR exit

identify.md lines 17–23:

Before doing any other work, call TodoWrite once with this exact list,
then mark each item completed as you finish it:

1. Grep ${logPath} for COMPILATION ERROR|FAILED|ERROR|Tests run.*Failures|BUILD FAILURE; do NOT read the full file.
2. If COMPILATION ERROR is present:
   a. Read both ${umlDir}/uml-interaction-main.md and ${umlDir}/uml-interaction-test.md.
   b. Create stub classes under ${projectPath}/src/test/java/org/farhan/impl/...
   c. Loop mvn verify -Dtest=${runnerClassName} until only non-compile failures remain (jacoco.xml populated).

When there is no COMPILATION ERROR, the prompt provides no step 3 — no “exit, you’re done” instruction. Both runs improvise an extra check (“let me at least confirm jacoco.xml is there”). The improvisation is free to use any tool form claude prefers. R56 picked an absolute-pattern Glob; R58 picked a relative-pattern + path= Glob; the two forms return different things on a gitignored target — see (2).

The prompt also mentions in its opening paragraph: “The baseline check before this run and the Phase Verification check after each fix guarantee that ${runnerClassName} is the only failing test.” That invariant should let identify exit cleanly on the no-error path — but it’s a parenthetical, not a step. Claude reads the steps and improvises around them.

(2) Claude Code Glob tool: absolute-pattern bypass vs path=-arg gitignore filter

Verified directly against the file currently on disk (same path the runs were querying):

$ stat /home/farhan/.../target/site/jacoco-with-tests/jacoco.xml
size: 522280, mtime: 2026-05-27 18:02:12 -0400
$ grep -E "^target|/target" /home/farhan/git/sheep-dog-main/.gitignore
**/target

The file exists. Repo-root .gitignore has **/target.

Glob form Behavior Why
pattern="/abs/path/to/file/in/target/foo.xml" (no path) Finds the file (R56 form) Absolute pattern is user-asserted scope; gitignore not applied.
pattern="target/foo.xml" path="/abs/projectRoot" Returns "No files found" (R58 form) path + relative pattern engages the project-scoped glob mode that respects .gitignore; the **/target rule excludes the match.

This is a discoverable, reproducible quirk in how Claude Code’s Glob tool treats the path argument. The tool documentation does not call out the gitignore behavior on the path= form. It is a real foot-gun whenever the prompt is asking claude about a build-output file under a gitignored directory.

R58 spent 196 s in identify; R56 spent 44 s. The 152 s delta is purely the cost of this Glob-form quirk meeting the silent no-error path in identify.md.

Why two contributors and not one

Either fix on its own would close the case-study gap:

  • If identify.md had an explicit exit immediately on no COMPILATION ERROR, both runs would skip call 5 onward. No Glob, no quirk, no detour.
  • If the Glob tool didn’t filter on path=, both runs would have got numFiles: 1 and exited at call 6 the same way.

The harness can’t fix the Glob tool (it’s a Claude Code internal behavior). The harness can fix the prompt, so that’s where the remediation sits.

Why this wasn’t caught by pbc-5455’s fix

pbc-5455 fixed the shortlist producer so fix.md’s precondition is met. Both runs in this case study confirm the producer is working: each shows Green: Writing jacoco-shortlist.md in the mojo log and a successful Read jacoco-shortlist.md returning content in the JSONL. That fix is in place.

What pbc-5455 did not touch is identify.md’s no-error path. The identify-phase fix-up was out of scope of the producer bug, but it’s the next layer down — the same kind of “prompt under-specification” the pbc-4849 / pbc-5455 series has been peeling off.

Place: prompt (with a tool-quirk amplifier). The fix lives in identify.md.

The Fix

Two prompt patches, tracked as #414 (identify.md) and #415 (fix.md). Both apply the same pattern: the prompt currently describes what the system provides without issuing the corresponding imperative, leaving claude to improvise — and improvisations differ across runs.

(1) identify.md — explicit no-COMPILATION-ERROR exit (#414)

3. Otherwise (no COMPILATION ERROR), identify is complete — exit immediately.
   Do not Glob, Read, or Bash to confirm jacoco.xml or other preconditions;
   the mojo's baseline check before this prompt and the precompute hook after
   it are the contracts.

This closes the silent no-error branch and names the disallowed actions explicitly. “Do not Glob, Read, or Bash to confirm preconditions” is the load-bearing clause — without it, a thorough claude might still improvise a precondition check (“just to be sure”) before exiting. Naming the disallowed actions makes exit mean exit now.

The identify-phase fix doesn’t help every scenario in the case-study window — This object step definition text parameter doesn't exist validation had a clean identify phase on both runs and 100% fix-phase variance. The identify fix tackles the cases where claude improvised; the fix-phase variance is the next layer down. (Confirmed on three sibling scenarios scanned during case-study follow-up: identify improvisation appears sporadically, not universally.)

(2) fix.md — explicit “read every listed class before any Grep/Glob” (#415)

The follow-up scan that surfaced the fix-phase shape (sibling scenario text parameter doesn't exist validation, +122s, all fix-side) traced to a parallel under-specification in fix.md step 1:

Read ${shortlistPath} first: if populated it narrows your search to the classes other passing tests in this family already exercise plus the production classes those tests cover. The sibling test classes show you the shape the failing test’s code path should take; the production classes show you the code path the passing tests exercise.

The literal imperative is read the shortlist file. The prose describes what the listed classes contain. There’s no instruction to read those classes — claude is left to infer “and now go read them” and picks its own search method. R56 read 7 sibling files end-to-end; R58 grep/glob’d the interface hierarchy symbol-by-symbol. The pattern the shortlist exists to surface lives in file structure, not in indexable symbols — whole-file reads converge faster than symbol grep on this codebase.

Proposed step 1 rewrite:

1. Read ${shortlistPath} first. Then read every production class listed in
   the shortlist end-to-end before any other Read, Grep, or Glob. The sibling
   test classes show you the shape the failing test's code path should take;
   the production classes show you the code path the passing tests exercise.
   Do not Grep or Glob for symbols, interfaces, or method names until after
   the listed classes are read — the pattern lives in file structure, not
   in indexable symbols.

Same anti-improvisation shape as the identify.md patch. Related: #411 proposes expanding the TodoWrite list to enumerate the shortlist files; the two changes compose (TodoWrite enumerates the files; step-1 wording enforces the order).

(3) Defense-in-depth (out of scope for this case)

The mojo could write the prompt with absolute paths interpolated for ${jacocoPath} already, removing the temptation to invent the relative-pattern + path= Glob form. Candidate for a separate sweep across all darmok prompt templates.

The Glob tool’s path=-arg gitignore filtering is silently inconsistent with the absolute-pattern form. Not Anthropic’s bug to fix from this repo, but worth a pbc-tool-quirks.md entry so future case studies don’t re-discover it. The tool returns “No files found” with no indication of why; the runner-log + JSONL pair is the only way to spot it externally.

Place for both prompt fixes: prompt — producer (mojo distributes the prompt resource at target/darmok/identify.md and target/darmok/fix.md). Single file each. No spec change, no test change, no tool-side change required.

Mapping to the Research

Predicted (pbc-research) Observed
Wide pair-range green-phase +3:50, top of gid=886919682
Each run individually within in-control band both run independently fine, both passed all gates
Both runs pass the test yes
Two work-trees differ yes — same file (TextIssueDetector.java), different patch site, different old_string/new_string — both functionally equivalent. pbc-4445-shape on the patch + pbc-4849/pbc-5455-shape on the exploration overhead.

This case stacks two signatures previously seen separately:

  • The identify-phase variance is a new pattern not seen in pbc-4445/4849/5051/5253/5455: a tool-form-dependent divergence on what should be a deterministic check, triggered by an under-specified prompt step.
  • The fix-phase variance is a familiar pattern: search-strategy drift before the same final fix.

Whereas pbc-5455 could attribute all variance to one producer bug, this case has the two contributors stacked. The methodology handled the stack — the pair-range fired, the JSONL walk separated them, and the fix sites are independently actionable.

Findings by Variable

Each subsection records this run’s findings about one Wheeler variable. Read the same heading across the run sequence to see how our understanding of that variable evolved.

green time pair range

This is a two-run comparison, and the green-phase pair-range topped gid=886919682 at 229,852 ms (~3:50) for the Rebuild56/Rebuild58 column pair. Both runs patched the same file (TextIssueDetector.java) at different patch sites — R56 inside the legacy paramCells.equals("Content") block, R58 inside the newer anyMatch(sp -> "Content".equals(sp.getName())) block — both valid for the failing assertion, a pbc-4445-shaped same-file-different-patch divergence. The pair was wide because two contributors stacked: a fix-phase search-strategy variance that landed the same file at different patch sites, sitting on top of an identify-phase variance where identify.md’s silent no-COMPILATION-ERROR branch let each run improvise a different jacoco.xml verification.

green time pair range moving range

No finding this run.

green time

R56 green was 195,030 ms, R58 green was 424,882 ms. The green-phase delta decomposes into a +152 s identify gap (~66% of the pair-range) and a +78 s fix gap. The identify gap traces to an under-specified prompt step: identify.md has no explicit exit on the no-COMPILATION-ERROR path, so both runs improvised a precondition check, and R58’s improvised Glob form spent 196 s in a retry loop against a present-but-gitignored jacoco.xml before falling back to ls. No contradiction or timeout — both runs reached end_turn cleanly with no retries.

green time moving range

No finding this run.

scale & green tokens

R58 emitted more output tokens (19,642 vs 13,270) and made more tool calls (60 vs 51), consistent with its longer identify and fix detours rather than a scale difference in the work itself. The two runs ran at different times of day (R56 at 23:17 EDT, R58 at 17:51 EDT) and on different dates (2026-05-26 vs 2026-05-27), so a server/time-of-day confound on decode rate cannot be ruled out, though the dominant cost here is clearly the extra tool calls and the Glob retry loop, not raw token throughput.

warm-up position

No finding this run.

Open Questions From This Case

  • After #414 and #415 land, the next Rebuild pair on this scenario should show identify wall-clock collapse to ~30–50 s on both sides (one TodoWrite + one Grep + UML reads + done — UML reads are now unconditional per #411’s direction) and fix-phase pair-range compress as the symbol-grep improvisation gives way to ordered whole-file reads. If either doesn’t move, the prompt changes weren’t the right knob.
  • The refactor delta on this scenario (R56 was 194 s slower than R58) is itself worth a follow-up case study on a refactor-phase-targeted pair-range tab. R56’s refactor made three extra Edits (import normalization, identifier shortening, control-flow restructure) that R58 did not — possibly because R58’s first-Edit shape didn’t need those follow-ups. Whether that’s a spec interpretation gap or a model-choice difference is unknown without inspecting the /rgr-refactor JSONL portions.
  • A pbc-tool-quirks.md entry on Glob’s path= + gitignore interaction would prevent re-discovery. The same quirk presumably affects any darmok prompt that asks claude to verify target/ artifacts; identify.md is just the first one to surface it.
  • The Green: Writing jacoco-shortlist.md line appears between identify and fix in the mojo log, but the mojo computes the shortlist itself, not via claude. That means identify’s claude session ends, the mojo writes the shortlist, then the fix session resumes — which is right. But the gating means identify wall-clock includes any mojo work after claude exits identify. On these two runs the mojo’s shortlist write is sub-second, so the wall-clock attribution is clean. If a future case shows otherwise, the segment boundary will need a second timestamp from the runner log.