- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper argues that diagnosing a failed agent workflow should name the persisted artifact write to change, together with the descendants to replay, rather than a responsible agent or step. The proposed framework has three parts.
First, it record a read-time provenance DAG over artifact writes. The nodes are tool command writes (except the root node indicating the initial state), and the edges represents the target node reads the outcome from the source node.
Second, the framework construct a directional local "fracture" score per write. It uses two fixed anchor set: A+ contains required content missing from the final artifact and A- contains erroneous content present in it. Then a "carrier" measures anchor support in the write's text and in its confidence-pooled parents (row-capped unbalanced OT with MiniLM costs, cosine, or exact lexical/typed key–value inclusion). The score is defined in Eq. (7). The score tries to measure the introduction of wrong content plus loss of required content for the write.
Third, it have a replay mechanism to replay the selected write and its descendants: It track a output-ancestor scope (A). And it uses a generator-supplied "field contracts" (K) that gate the loss term to separate legitimate projection from deletion.
The paper evaluate the proposed framework on synthetic workflows.
Critical strengths
-
The problem reframing is useful. Naming an editable write plus its replay closure is more actionable than naming an agent and step.
-
Contracts separate projection from deletion. Gating loss by declared obligations, and showing that gating by "current keys" erases true deletions, is a clean mechanism argument.
-
The provided code is extensive and reproducibility of the paper is excellent.
Critical weaknesses
-
There is no comparison to any published attribution method. The experiments only show that each part of the framework does what it was designed to do on the authors’ synthetic workflows, where every dependency is recorded. They don’t show an advantage over published methods or over simple position baselines on real agent traces.
-
The primary benchmark contains no agents and no natural-language reasoning. In real agent workflows detecting and tracking reads and writes an agent make along can be challenging, especially considering many "read" and "write" may not be a simple file action (e.g. arbitrary shell commands or scripts, interacting with websites and other external systems). A synthetic benchmark alone cannot prove the framework will work in real-world, or in controlled environment with real agents.
-
Natural-trace results are low relative to simple baselines, and chance level is not reported.
a. The main configuration is below the paper’s own position baselines. On the 127 TracerTraj coding traces, no-reference UOT with provenance reaches 46.4% aligned hit@1 (Table 4b, §4.8). The paper reports first-write at 61.8% and last-write at 53.6% (Table 4a, Table 9).
b. Reference anchors and provenance don’t close the gap. With reference anchors, UOT-provenance (58.2%) is still below first-write. Provenance does not improve on the chain variant in either setting (46.4 vs. 48.2 without a reference; 58.2 vs. 62.7 with one).
c. The core directional configuration fails on the reviewed cases. On the five write-level targets of the review pilot, it obtains 0/5 hit@1 (Table 16).
d. Chance level is reported elsewhere but not here. The paper gives a uniform baseline for the controlled cohorts (Table 11) and the 12-case pilot (Table 16), but not for the 127 traces. It is therefore unclear whether the method beats chance on the only real agent data.
Questions for authors
- The paper cites AgenTracer, Who&When, TRAIL, AgentTrace, and a few more previous works, however none of these is run as a baseline. Could you please compare your framework with at least one of them, or explain the reason why you can't?
- Could the authors evaluate on a benchmark where faults are made by, or propagated through, LLM agents, not a deterministic generator? (Who&When, an injected-fault variant of real OpenHands trajectories in which downstream steps are actually re-run by the agent, or anything else that involves real LLM agents)
- Could the authors report the expected hit@1, top-3 and MRR of a uniform random selection over the same candidate writes and target sets (visible-25, aligned-110, full-127), analogous to the “Uniform” rows of Table 11 and “All-uniform (analytic)” in Table 16?
- How's the framework handle reads and writes performed through shell commands (sed -i, cat >, scripts)? I think OpenHands have a lot of execute_bash tool calls and many reads and writes are done through commands. It appears no trivial to parse the commands since they're quite flexible and sometime agent combines both read and write in one command (like search and replace style commands).