acceptodds
Desk rejected at ICLR 2027

WHEN ATTRIBUTION IS NOT REPAIR: RESPONSIBILITY AND INTERVENTION IN LLM MULTI- AGENT SYSTEMS

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Debugging tools for LLM multi-agent systems usually assume that finding where an error entered a trace tells us where to fix it. We test this assumption directly. A failed trace exhibits source–repair mismatch (SRM) when some logged action would repair the final answer but none of those actions lies at an annotated error source. On 116 repairable failures, two independent blinded Source annotations give SRM rates of 0.543 and 0.733, and 0.388 of traces stay mismatched even when every event marked by either annotator counts as a source. As a consequence, a policy restricted to source events can recover at most 61.2% of candidate-oracle logged repairs (71/116 repairable traces), corresponding to 31.7% absolute Repair@1 on the 224-failure cohort, however accurate its localization. We release a benchmark that pairs six-role event responsibility labels, a 1,600-item human-adjudicated test set for verifier and contract roles, and checker-scored single-intervention repair. We also introduce RAISER, which ranks an agent system’s own logged answers with standardized responsibility evidence and passes the same evidence to a frozen verifier over a three-item shortlist. On 196 failures held out from development, aligned graph and role channels improve ranking NDCG over a matched local ranker (Holm p=0.007), permuting them within each trace erases the gain, and the full system raises Repair@1 from 65.3% to 70.9% of failures (p=0.027). On the development and validation cohorts, the same final system recovers 86.8% and 85.6% of candidate-oracle repairs; its margins over the matched verifier controls are +1.5 and +4.4 oracle points, respectively. A preregistered decomposition on 1,200 further failures locates the remaining headroom in recognizing a correct answer that is already shortlisted. Code: https://anonymous.4open.science/r/Trace-R. Data: https://anonymous-hf.com/a/psa97qn014rw/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-agent system, Failure Attribution, Responsibility Modeling, Intervention and Repair, Trajectories

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ivnEOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the distinction between failure attribution and effective repair in LLM-based multi-agent systems, introducing the concept of source–repair mismatch (SRM) to characterize cases where the error source differs from the location of an available correction. The authors formalize this phenomenon and construct a benchmark with event-level responsibility annotations and checker-based repair evaluation. They further propose RAISER, a responsibility-aware repair selection framework that combines trace evidence graphs, learned role estimation, candidate ranking, and LLM-based verification to select corrective answers from failed execution traces. Experiments across multiple datasets evaluate the prevalence of SRM and the effectiveness of responsibility-guided repair selection.

Critical strengths

S1. Important Distinction Between Failure Attribution and Effective Repair

The paper identifies an important limitation of attribution-first agent debugging: accurately locating where an error originated does not necessarily identify where an effective repair should be applied. This distinction is particularly relevant to multi-agent systems, where downstream agents may correct or compensate for upstream errors, making the optimal repair location different from the original error source. Also, such a source-repair mismatch may lead to shallow intervention.

S2. Formalization and Empirical Investigation of Source–Repair Mismatch

I appreciate that the paper goes beyond qualitative observations by formally defining source–repair mismatch and systematically investigating its prevalence and implications (although the current formulation seems to have flaws). The authors construct a benchmark with responsibility annotations and checker-based repair evaluation, and conduct controlled experiments and sensitivity analyses to support their findings.

Critical weaknesses

W1. Limited Generalizability Beyond Final-Answer Replacement

The paper primarily evaluates intervention-based repair by selecting an already-logged correct answer to replace the failed final decision. However, this is a highly constrained setting that differs substantially from realistic agent debugging, where interventions typically modify earlier actions and require re-executing subsequent steps.

W2. Limited Reliability and Cross-Dataset Generalizability of Responsibility Estimation

RAISER relies heavily on learned responsibility scores to guide repair selection, yet the Role Model achieves relatively modest classification performance, and the authors acknowledge weak cross-dataset transfer. Although the paper demonstrates improvements over several baselines, it remains unclear whether the logistic regression-based responsibility estimation captures generalizable failure patterns or dataset-specific characteristics. The lack of comprehensive cross-dataset evaluation, particularly regarding downstream repair performance, raises concerns about the robustness and broader applicability of the proposed framework.

Questions for authors

Q1. Generalizability Beyond Final-Answer Replacement

The paper primarily evaluates intervention-based repair by selecting an already-logged answer to replace the failed final decision. However, this setting is substantially different from realistic agent debugging, where interventions may modify earlier actions and require re-executing subsequent steps. To what extent do the source–repair mismatch findings and RAISER's improvements generalize to such settings?

Q2. Reliability and Cross-Dataset Generalization of Responsibility Estimation

RAISER relies on learned responsibility scores from logistic regression models, but their absolute classification performance is relatively modest, and Appendix C acknowledges weak cross-dataset transfer. How reliably can these models identify responsibility roles in unseen agent frameworks or benchmark distributions?

Q3. Availability and Reliability of Contracts and Obligations

The framework assumes access to agent contracts and cross-agent obligations to construct responsibility evidence, particularly for identifying missed verification duties. How are these obligations obtained in practice, and how reliably can they be extracted from ordinary multi-agent execution traces?

Reviewer_rXj8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that the event where an error enters a multi-agent trace (Source) is often not the event whose logged answer can repair the task, and calls this source–repair mismatch (SRM). On 116 repairable development failures, SRM is 0.543 and 0.733 under two blinded Source annotations, and 0.388 under their union. Proposition 1 then bounds any policy that intervenes only at Source events at 61.2% of the candidate oracle. The paper also releases a six-role responsibility benchmark with a 1,600-item human MV/CV test set. Finally, it proposes RAISER, which builds a typed evidence graph, scores event roles with sparse linear heads, ranks logged answers by how much responsibility they absorb, and passes the Top-3 together with this evidence to a frozen Qwen3.8-27B verifier. On 196 held-out failures, RAISER raises Repair@1 from 65.3% to 70.9% over a matched local system.

Critical strengths
  1. The permutation control in Table 3A is well designed. Rotating the responsibility block within each trace drops NDCG below the local-only ranker, which shows the ranker relies on candidate-aligned evidence rather than extra dimensions.

  2. Separating where an error originates from where it can be repaired is a reasonable question for MAS debugging, and the released role labels could be useful for attribution work.

Critical weaknesses
  1. The main finding depends on the chosen intervention operator and the breadth of the Source rubric. The operator replaces the final answer with an answer that some event already concluded. A strict Source event introduces the harmful state, so its own answer is usually wrong by construction; Figure 1 shows exactly this, where the source-first repair returns 0.5. Consistent with this, SRM is 0.914 under the A∩B definition and falls as more events are labeled Source: 0.388 when the union marks 654 of 1,267 events, and 0.190 at agent level (Table 6). The SRM number therefore mostly tracks how many events are labeled, not a stable property of traces. More importantly, attribution-based debuggers such as DoVer intervene by editing or regenerating at the source, not by copying the source's answer, so Proposition 1 bounds a policy nobody actually uses. Where the paper does test regeneration (Section E), both policies repair only 3/150 in LangGraph/AutoGen, and on native AgentVerse RAISER (19/50) is below majority voting (23/50).

  2. Responsibility evidence adds little beyond the verifier. On development, RAISER is only +1.5 points over the query-only verifier (Holm p=1.0p=1.0). On validation it is +4.4 (p=0.053p=0.053 and 0.0870.087). In the preregistered 1,200-failure test, the boundary-flow ranker loses 2.3 Rank@1 points (p=0.007p=0.007), the responsibility context adds +0.8 (p=0.51p=0.51), and the residual candidate brings no gain. The ranker was then revised after inspecting this cohort, and the only significant end-to-end gain comes from 196 traces (16 wins vs. 5 losses). Since the verifier solves the task itself and maps its own answer onto the shortlist (Algorithm 1, line 10), most of the repair likely comes from the 27B verifier rather than the trace. On MATH, RAISER (0.500) is even below the query-only control (0.536) and DeepSeek-V4-Pro (0.589) (Table 11). Repair is also evaluated only on short-answer QA (GSM8K, MATH, MMLU, SciBench), while the agentic settings where attribution matters most (Who&When, TRAIL) are used only for role labels.

Questions for authors
  1. What Repair@1 does Qwen3.8-27B reach if it answers the task directly, without the trace, on the same failures? How often does the MAPEXACT branch fire, i.e., the verifier's own answer exactly matches a shortlisted candidate? If the verifier alone comes close to RAISER, the gain should be attributed to the verifier rather than to the trace or the responsibility evidence.

  2. How often does a strict Source event conclude a correct answer at all? If this is rare, SRM is largely implied by the operator. Can you evaluate the bound under a source-edit operator, which is how attribution-based debuggers actually intervene? Also, in Section E the frozen candidate oracle is 0.067 but both policies reach only 0.020. What prevents regeneration from recovering the corrections that exist in the log?

  3. The SRM audit and the 61.2% bound use 116 repairable traces under the earlier parser, while Table 2 reports RAISER against 136 repairable traces under the current parser. Were the 20 additional traces Source-annotated? If not, the comparison between the 61.2% ceiling and RAISER's 86.8% is made on different sets, and the gap between them is not directly comparable.

Reviewer_kGbcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper separates where an error entered a failed multi-agent trace from where the failure can be repaired. Repair means replacing the final decision with an answer already in the log, scored by a checker the policy never sees. Source–repair mismatch (SRM) is the share of repairable traces in which no such answer sits at an annotated Source event; on 116 repairable failures it is 0.543 and 0.733 under two blinded annotations and 0.388 under their union, which caps source-restricted repair at 61.2% of the candidate oracle. The paper releases responsibility labels and a 1,600-item human test set, and proposes RAISER, a responsibility-aware ranker whose Top-3 shortlist and evidence go to a frozen verifier. On 196 held-out failures it improves ranking NDCG (Holm p = 0.007) and raises Repair@1 from 65.3% to 70.9% (p = 0.027); on development and validation the gains over matched controls are +1.5 and +4.4 oracle points.

Critical strengths

(1) Separating "where the error entered" from "which logged answer fixes the task" is a useful reframing, and the paper supports it with a practical observation: 136 of 224 development failures already contain a correct logged answer (l.303). Table 9 shows how much existing attribution front-ends leave on the table. Their Top-1 recovers 60.3–66.4% of oracle repairs, while re-ranking their own Top-3 recovers 80.2–82.8%. For people building debuggers, that gap is actionable.

(2) The evaluation practice is careful. Held-out manifests and candidate scores are hashed before checker labels are joined (l.905–909), preregistered tests are reported even when they fail, the main comparison families (Tables 2 and 3A) are Holm-corrected, and permutation controls check that the ranking gain depends on aligned evidence. The 1,600-item human test set will be useful beyond this paper. This care is also why I am comfortable drawing the conclusions below from the paper's own tables.

Critical weaknesses

(1) The repair gain, which is the paper's method contribution, is not shown to come from the verifier stage. The ranking result itself holds up (NDCG +0.0115, Holm p = 0.007, removed by the permutation control; l.416–420). For repair, Rank@1 is the share of failures whose top candidate is a correction (l.307), the same denominator as Repair@1 (l.437), and RAISER's own fallback returns exactly the ranker's Top-1 (l.261–262). In Table 3B, returning the responsibility ranker's Top-1 without any verifier therefore gives 73.0% (143/196), and full RAISER gives 70.9% (139/196). The verifier overrides a correct Top-1 on 20 traces (l.429–430), which leaves at most 16 versus 20 discordant traces (exact binomial p ≈ 0.62), so on this cohort the verifier does not improve on the ranker. The headline +5.6 is measured against local ranker + query-only verifier (65.3%), which is below RAISER's own ranker alone.

The other comparisons do not close this gap. Against the query-only verifier on the same shortlist (+4.6, p = 0.064), the evidence-aware arm adds both the ranker's rank prior and six responsibility summaries (l.938–941), so the two contributions cannot be separated. This end-to-end protocol was also frozen after the ranking analysis on the same 196 traces, whose checker labels had already been joined (l.908–909, l.917); the authors note that a new cohort would further test the gain (l.469–470). On development the gain over the matched control is +1.5 (Holm p = 1.000), on validation p = 0.053, and on the 1,200-failure cohort Table 13 row (d) shows Repair@1 2.4 points above Rank@1 with earlier components, without a test.

(2) The SRM finding, the paper's first contribution, is not yet separated from the repair operator, and its size depends on noisy labels. Under logged-answer replacement, a Source event can be a repair only if it concludes an answer and that answer is correct. The paper reports how often some Source event carries a correct answer (1 − SRM), but not how many Source events conclude any answer at all (a_t ≠ ⊥). Without that number one cannot tell whether mismatch mostly reflects sources with a wrong answer, which would say something about multi-agent failures, or sources with no answer, which the operator cannot use by construction.

On labels: the Source annotations are blinded LLM annotations with κ = 0.33. The abstract and §5.1 describe them as blinded annotations next to the human Gold set, and their LLM nature is stated in the Limitations and the AI-use statement (l.467, l.497–498). SRM is 0.543–0.733 across the individual annotators and 0.388–0.914 between their union and intersection (Table 6), which the authors acknowledge (l.322–324). On scope: the Limitations say the bound is specific to logged-answer repair (l.469), but l.49–51 and l.204–205 speak of editing or intervening at the source in general, and no experiment compares re-running from the Source event with re-running from the repair event. I find the qualitative point plausible, since SRM stays at or above 0.388 under every event-level definition, but it should be framed as a result about this operator.

Questions for authors

(1) Could you add ranker-only rows, and a verifier that receives only the rank prior, for the development, validation and held-out cohorts, with paired tests against full RAISER? On which cohort, if any, does the evidence-aware verifier do significantly better than returning the ranker's Top-1? This is the answer most likely to change my assessment.

(2) For each Source label set, what fraction of Source events conclude an answer, and what fraction of those answers are correct? The 116 repairable traces used for SRM match the earlier-parser count (Table 9, l.865–866), whereas the current parser gives 136 (l.303); could you also recompute SRM and the 61.2% bound under the current parser?

(3) (Clarification) App. E re-executes from the selected event, but it does not compare Source events with repair events. Is there any experiment that does? If not, I would suggest narrowing l.49–51 and l.204–205 to the logged-answer setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.