acceptodds
Withdrawn from ICLR 2027

Zero-Training Fracture Attribution via Directional Provenance Differencing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

An actionable agent diagnosis must identify which write to change, the evidence for changing it, and the downstream work to replay. We introduce a zero-attribution-training framework that connects these objects through versioned artifact provenance. A bounded, directional local carrier compares fixed defect anchors against a write and its pooled parents, separating wrong-content introduction from required-content loss; confidence-aware pooling preserves anchor budgets at merges. Output ancestry determines relevance, while optional field contracts distinguish legitimate projection from deletion. On executable failures with unbalanced optimal transport (UOT) as the carrier, restricting scoring to output ancestors raises exact-write hit\@1 from to , and replacing unsigned with directional local change adds a further points (paired workflow-cluster interval ). With supplied contracts, an exact lexical/typed carrier reaches from , and typed UOT gains likewise. Executed replay separates localization, repair, and regeneration cost; fixed missing-information experiments measure non-target selection, abstention, and replay-closure errors. Observation-equivalence ceilings separate what dependency structure alone can identify from what frozen defect anchors add. The result is an inspectable interface from defect evidence to testable interventions, with explicit observation requirements and no trained attribution predictor.

Primary area:
interpretability and explainable AI
Keywords:
LLM Agents, Agent Diagnosis, Fault Localization, Provenance, Attribution

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_b16KOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that diagnosing a failed agent workflow should name the persisted artifact write to change, together with the descendants to replay, rather than a responsible agent or step. The proposed framework has three parts.

First, it record a read-time provenance DAG over artifact writes. The nodes are tool command writes (except the root node indicating the initial state), and the edges represents the target node reads the outcome from the source node.

Second, the framework construct a directional local "fracture" score per write. It uses two fixed anchor set: A+ contains required content missing from the final artifact and A- contains erroneous content present in it. Then a "carrier" measures anchor support in the write's text and in its confidence-pooled parents (row-capped unbalanced OT with MiniLM costs, cosine, or exact lexical/typed key–value inclusion). The score is defined in Eq. (7). The score tries to measure the introduction of wrong content plus loss of required content for the write.

Third, it have a replay mechanism to replay the selected write and its descendants: It track a output-ancestor scope (A). And it uses a generator-supplied "field contracts" (K) that gate the loss term to separate legitimate projection from deletion.

The paper evaluate the proposed framework on synthetic workflows.

Critical strengths
  1. The problem reframing is useful. Naming an editable write plus its replay closure is more actionable than naming an agent and step.

  2. Contracts separate projection from deletion. Gating loss by declared obligations, and showing that gating by "current keys" erases true deletions, is a clean mechanism argument.

  3. The provided code is extensive and reproducibility of the paper is excellent.

Critical weaknesses
  1. There is no comparison to any published attribution method. The experiments only show that each part of the framework does what it was designed to do on the authors’ synthetic workflows, where every dependency is recorded. They don’t show an advantage over published methods or over simple position baselines on real agent traces.

  2. The primary benchmark contains no agents and no natural-language reasoning. In real agent workflows detecting and tracking reads and writes an agent make along can be challenging, especially considering many "read" and "write" may not be a simple file action (e.g. arbitrary shell commands or scripts, interacting with websites and other external systems). A synthetic benchmark alone cannot prove the framework will work in real-world, or in controlled environment with real agents.

  3. Natural-trace results are low relative to simple baselines, and chance level is not reported.

    a. The main configuration is below the paper’s own position baselines. On the 127 TracerTraj coding traces, no-reference UOT with provenance reaches 46.4% aligned hit@1 (Table 4b, §4.8). The paper reports first-write at 61.8% and last-write at 53.6% (Table 4a, Table 9).

    b. Reference anchors and provenance don’t close the gap. With reference anchors, UOT-provenance (58.2%) is still below first-write. Provenance does not improve on the chain variant in either setting (46.4 vs. 48.2 without a reference; 58.2 vs. 62.7 with one).

    c. The core directional configuration fails on the reviewed cases. On the five write-level targets of the review pilot, it obtains 0/5 hit@1 (Table 16).

    d. Chance level is reported elsewhere but not here. The paper gives a uniform baseline for the controlled cohorts (Table 11) and the 12-case pilot (Table 16), but not for the 127 traces. It is therefore unclear whether the method beats chance on the only real agent data.

Questions for authors
  1. The paper cites AgenTracer, Who&When, TRAIL, AgentTrace, and a few more previous works, however none of these is run as a baseline. Could you please compare your framework with at least one of them, or explain the reason why you can't?
  2. Could the authors evaluate on a benchmark where faults are made by, or propagated through, LLM agents, not a deterministic generator? (Who&When, an injected-fault variant of real OpenHands trajectories in which downstream steps are actually re-run by the agent, or anything else that involves real LLM agents)
  3. Could the authors report the expected hit@1, top-3 and MRR of a uniform random selection over the same candidate writes and target sets (visible-25, aligned-110, full-127), analogous to the “Uniform” rows of Table 11 and “All-uniform (analytic)” in Table 16?
  4. How's the framework handle reads and writes performed through shell commands (sed -i, cat >, scripts)? I think OpenHands have a lot of execute_bash tool calls and many reads and writes are done through commands. It appears no trivial to parse the commands since they're quite flexible and sometime agent combines both read and write in one command (like search and replace style commands).
Reviewer_L6zrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

When a multi-agent coding workflow produces a wrong result, this paper tries to find the specific file edit that introduced the problem, without training a model to do it. It records which file versions each edit read. Then, given a description of what is wrong with the final output, it scores each edit by whether it added wrong content or dropped required content compared with what it read. Edits the final output doesn’t depend on are ignored. On 226 faults injected into synthetic workflows, the best version finds the faulty edit 100% of the time, but only when it is told which fields each edit is responsible for; without that information it reaches 76%. On real agent traces, it does no better than simply guessing the first edit.

Critical strengths
  1. Failure attribution is an important area.
  2. The evaluation is reasonably done.
Critical weaknesses
  1. The biggest issue with this paper is the writing. The abstract requires that you've read the whole paper to understand. Core terms are used in the abstract and introduction before they are defined, and some are never defined. Every sentence has qualifications.
  2. The technical novelty is limited. The core idea is very simple: Track which file versions each agent edit read. Keep only edits the final output depends on. Score each edit by whether it added wrong content or dropped required content relative to its inputs. Rerun from the chosen edit to check the fix.
  3. The method only works well on the synthetic setting. On real traces, it doesn't do any better than guessing the first edit.
Questions for authors
  1. Have you tested the method on workflows you didn’t write yourselves, such as real agent runs with injected faults, or a second, independently built generator? If so, how does accuracy compare with the synthetic results?
  2. All three fault types change a single field in a four-field policy. How does the method perform on harder faults, such as errors spread across several fields or logic errors that don’t appear as a changed value?
  3. The 100% result depends on field contracts exported by your generator. In a real pipeline, where would those contracts come from, and how accurate is the method with contracts that are inferred automatically or partly wrong?
  4. Is there any setting in your experiments where optimal transport beats exact or lexical matching? If not, what does the method contribute beyond dependency slicing plus a diff?
  5. Is there any setting in your experiments where optimal transport beats exact or lexical matching? If not, what does the method contribute beyond dependency slicing plus a diff?
Reviewer_jxAUOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies fault attribution in agent workflows that create and modify persistent artifacts. The authors construct a write-level provenance graph and compare fixed defect anchors with each write and its parent states to locate where erroneous content was introduced or required content was lost. The framework uses output-ancestry filtering, directional scoring, and optional field obligations, followed by downstream replay. Experiments on executable synthetic workflows and released agent traces examine the contributions of provenance, directionality, and field responsibility to fault localization.

Critical strengths

S1. A useful write-level attribution interface. The method identifies specific artifact operations for inspection and replay, offering finer intervention granularity than conventional agent- or step-level fault attribution. This is a practical problem worth investigating. S2. Controlled analysis of the main components. The experiments separately examine provenance scope, directional scoring, and field responsibility. For raw UOT, directionality improves Hit@1 from 29.6% to 50.9%. S3. Clear separation of localization and repair outcomes. The paper reports localization accuracy, repair success, and replay cost separately. It also includes less favorable results for certain carriers and natural traces, making the empirical findings easier to interpret.

Critical weaknesses

Critical Weaknesses W1. The methodological novelty could be more clearly established. Related work has explored agent workflow provenance (PROV-AGENT), intervention-driven debugging and replay (DoVer), and provenance-based pipeline diagnosis (BugDoc). These approaches address different settings, but several underlying ideas overlap with the proposed framework. The paper would benefit from a more precise explanation of the methodological advance introduced by write-level directional differencing. W2. The strongest results depend on favorable experimental conditions. The 100% Hit@1 result relies on reference-derived defect anchors and generator-supplied field contracts, while typed exact without contracts achieves 76.1%. Although the contracts contain no expected values or fault labels, their availability remains an important assumption. The synthetic evaluation also covers a relatively narrow set of four-field policies and three fault types, leaving broader applicability uncertain. W3. Limited evidence from natural agent traces. On 110 aligned TracerTraj cases, no-reference UOT achieves 46.4% Hit@1, below the 61.8% first-write baseline. The aligned targets are nearest-write proxies rather than verified faulty writes. The separate 12 case agent-assisted review identifies only five supported targets, without independent human ground truth or executed repairs. These results provide limited support for the method's practical attribution accuracy. W4. The advantages of directionality and soft matching are inconsistent. The 21.2% directional gain is specific to raw UOT, with no aggregate improvement for several other carriers. Raw lexical matching also outperforms directional UOT on the synthetic validation set (64.2% versus 50.9% Hit@1). In natural traces, semantic anchor extraction excludes operators and many numeric literals, potentially discarding information relevant to code defects. It remains unclear when soft UOT matching offers an advantage over simpler lexical or value-aware methods. W5. The replay evaluation provides limited evidence of practical intervention effectiveness. The synthetic replay regenerates descendants without reinjecting the original fault. As a result, slice-first repairs 100% of failures despite achieving 0% Hit@1, although it requires more regenerated writes than typed exact with contracts (14.51 versus 7.16). The authors appropriately distinguish localization from repair success. The results establish a replay-cost advantage under the controlled protocol, while the effectiveness of the identified interventions in natural workflows remains untested.

Questions for authors

Q1. What specific failure scenarios or methodological properties distinguish the proposed write-level directional differencing from existing provenance tracing and intervention-driven debugging? A concrete example would help clarify the contribution. Q2. How would sufficiently accurate defect anchors and field obligations be obtained in practical workflows without a reference artifact? Which of these supplied signals are essential to the strongest reported configuration? Q3. How should the actionability of the identified writes be interpreted given the proxy labels and lack of executed repairs on natural traces? A few examples from the existing reviewed cases could help clarify this point.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.