acceptodds
Withdrawn from ICLR 2027

Where to Repair? When the Root Cause Is Not the Best Target in Multi-Agent Workflows

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recovering failed multi-agent workflows is essential for reliable Artificial Intelligence (AI) automation: an error passed between agents can prevent an entire task from succeeding. Existing work diagnoses causes and tests repairs, yet diagnosis alone does not determine which permitted change best restores task success. One challenge is to identify feasible recovery choices beyond the error source under permission and budget constraints. Another is to compare interacting repairs under matched conditions, separating the effect of repair locations from search quality. We propose a Cross-Surface Constrained Recovery Model that represents source repairs, communication changes, and downstream checks in one constrained action space. We implement the model with Feasibility-Gated Joint Replay Search through three modules: candidate representation specifies actions, gated joint replay evaluates compatible combinations, and selection and loss audit selects repairs and measures value excluded by location restrictions. On our controlled recovery suite (study identifier: CRPA-R1), full candidate access improves mean recovery-rate Lower Confidence Bound (LCB) by over exhaustive root-only repair across 96 repairable-root traces; on 27 AgentRx retail calibration cases with three seeds, changing only the recovery rule improves executable recovery by percentage points over root-following continuation. These findings support designing recovery systems that use diagnosis to guide a broader search for feasible changes that restore task success.

Primary area:
causal reasoning
Keywords:
multi-agent LLMs, causal inference, network interference, causal credit assignment, communication topology, counterfactual evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_RnaaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes the Cross-Surface Constrained Recovery Model, a framework for evaluating recovery strategies in failed multi-agent workflows. The authors evaluate the framework primarily on CRPA-R1, a synthetic benchmark consisting of 144 template-generated traces, and investigate how constraints on intervention locations, action spaces, and budgets affect recovery performance. The results suggest that allowing interventions beyond the diagnosed root cause can improve recovery rates.

Critical strengths

None

Critical weaknesses

The paper appears insufficiently developed in its current form. While the research question is potentially interesting, the limited experimental validation, unclear methodological novelty, and presentation issues significantly reduce the contribution of this work.

  1. Limited alignment between motivation and evaluation. The paper motivates recovery from failures in multi-agent systems, yet its primary experiments rely on synthetic scenarios with predefined root causes and intervention outcomes. It remains unclear whether the findings generalize to realistic multi-agent failures.

  2. Insufficient experimental validation. CRPA-R1 contains only 144 template-generated traces, with limited discussion of the underlying agent design, failure-generation process, and representativeness. The AgentRx evaluation covers only 27 calibration cases and compares continuation rules rather than directly validating the proposed cross-surface recovery framework.

  3. Limited methodological novelty. Feasibility-Gated Joint Replay Search primarily enumerates feasible intervention combinations and selects the one maximizing recovery. The methodological contribution beyond expanding the intervention space and applying constrained exhaustive search is unclear.

  4. Limited practical significance of oracle-based comparisons. The primary results compare an offline oracle against restricted intervention sets. Since the full candidate set subsumes the restricted sets, its optimal recovery value is guaranteed to be no worse. These comparisons quantify the cost of restricting intervention locations but do not demonstrate improvements achievable by a practical recovery method under comparable computational budgets.

  5. Unclear experimental presentation. The results introduce multiple settings (e.g., Full, Strict-Root, RCF-H, all traces, and root-feasible) without clearly explaining their distinctions. For example, Table 1 reports comparisons across subsets of 144, 96, and 48 traces, making the results difficult to interpret without consulting other sections.

Minor weakness

Overly complex conceptual figure. Figure 2 combines the framework, algorithm, theoretical analysis, and experimental findings into an overly dense illustration. Several components are difficult to identify; for example, the downstream check discussed in Lines 161–189 is not clearly indicated. A simplified figure focusing on the core workflow would improve clarity.

Questions for authors

See weakness

Reviewer_bpg7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper pushes back on a default assumption in multi-agent failure recovery: that once you have correctly diagnosed the responsible agent or step (the "root"), the right move is to repair that root. The authors argue that a correct diagnosis does not determine the best feasible recovery. Under real permission and budget constraints the root may not be editable, or editing it may recover less task success than a downstream intervention — for example, validating propagated state before final acceptance rather than fixing the agent that produced it. They name the mismatch between responsibility-derived locations and the locations needed for the best recovery "Responsibility–Recourse Separation" (RRS).

There are two parts. First, a Cross-Surface Constrained Recovery Model that puts root repairs, communication or routing changes, and downstream checks into a single action space over "surfaces" (the locations an action modifies), with a feasibility gate for permission, safety, budget, and compatibility, plus an abstention option. Recovery value is a one-sided Clopper–Pearson lower confidence bound on replay success, and a loss audit compares the best admitted bundle under full access against the best under root-only (Strict-Root) and responsibility-conditioned (RCF-H) location restrictions, which defines the location-restriction loss. A short theory section decomposes recovery loss into location, information, and selection components and shows each is nonnegative. Second, a Feasibility-Gated Joint Replay Search that enumerates compatible action bundles, jointly replays each one under shared seeds to estimate recovery under matched conditions, and then selects the highest-value admitted bundle, repeating the selection under each location restriction to read off the excluded value.

Evaluation uses two testbeds. CRPA-R1 is a controlled, fully synthetic suite of 144 constructed traces with no external model calls, built to include cases where a downstream guard beats root repair; the headline is a mean recovery-LCB gain of about +0.135 for full access over exhaustive root-only repair on 96 repairable roots, driven by 48 dominance cases where downstream validation recovers three more of twelve replays than root restore. A candidate-family ablation shows the validation guard is necessary and sufficient for the gap, and a selection-rule comparison shows that common candidate access alone does not determine recovery quality. AgentRx is a smaller non-synthetic continuation test on 27 retail calibration cases from the Tool-Agent-User benchmark across three seeds, where a feasibility-aware continuation cut improves executable recovery over root-following by 19.75 percentage points.

Critical strengths

A clean conceptual separation that the attribution literature mostly conflates. Most prior work on multi-agent failure stops at "find the responsible agent or step." This paper argues, and formalizes, that responsibility and the best place to repair can come apart once permissions, budgets, and feasibility enter the picture, and it gives a concrete way to measure the value excluded by restricting recovery to the diagnosed root. The reframing is simple to state but genuinely under-served.

An experimental design that isolates the thing being claimed. The central comparison varies only the allowed intervention surfaces while holding diagnosis, candidate access, budget, seeds, scorer, and ties fixed, and the paper backs this with a candidate-family ablation showing the guard family is both necessary and sufficient for the gap, plus a separate selection-rule comparison showing that common candidate access alone does not determine recovery quality. Even granting that the suite is synthetic, this is the right way to separate location restriction from search quality, and it makes the mechanism claim credible rather than confounded.

Critical weaknesses

The central quantitative evidence is a self-constructed synthetic suite, so it demonstrates the mechanism but not that it matters in practice. CRPA-R1 is built to contain the dominance cases it then reports, with no model calls anywhere in construction or replay, and the dominance gap is template-imposed (a zero-width interval). The paper is admirably honest that this "establishes the mechanism, not its prevalence," but that honesty also marks where the contribution stops: it is closer to a well-argued existence proof than to a demonstrated empirical phenomenon, and for an empirical venue that leaves the "how often does this actually bite, and is the search worth its cost?" question open. End-to-end deployment efficiency is explicitly not measured, and the downstream-interception evidence rests on a single validation-guard family.

The one non-synthetic cohort is thin and partly off-target for the headline framing. The AgentRx test covers 27 retail calibration cases over three seeds, and the result is 6 wins, 21 ties, and no losses — most cases are indistinguishable. More importantly, exact-step attribution accuracy there is about 7%, so the comparison is largely testing how a feasibility-aware continuation behaves under wrong attribution, not the paper's sharper claim that a correctly diagnosed and editable root is still not the best target. The direction is consistent and the authors report this plainly, but a lot of the external-validity weight rests on a modest cohort that does not cleanly match the framing it is meant to support.

Questions for authors
  1. On AgentRx: given roughly 7% exact-step attribution accuracy, can you split the 19.75-point gain by whether attribution was actually correct on each case (using the data you already have)? That would tell us whether the real-data evidence speaks to Responsibility–Recourse Separation proper — root correctly identified, yet not the best repair — or mainly to robustness against bad attribution. This distinction materially affects how I read the paper's central claim.
  2. The dominance-case numbers are fixed by construction. Could you state in the main text, not just the appendix, what a reader should and should not take from values like +0.135 and +0.270 — i.e., that they are existence magnitudes under your suite rather than expected gains? A sentence would prevent an easy misreading.
  3. The downstream-interception result comes from one validation-guard family. Is there reason to expect the location-versus-selection separation to behave similarly for reroute, replan, or orchestration changes, or is the guard special because it can inspect propagated state that a root edit cannot reach? A short discussion would help scope how general the effect is.
Reviewer_54aCOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper argues that the best place to fix a failed multi-agent workflow isn't always the agent that caused the error. Sometimes a check added later in the workflow works better. The paper's method treats every possible fix as a candidate (a root repair, a message edit or a downstream check), keeps only the ones that are allowed and affordable, replays each one, and picks whichever recovers the task most reliably. Experiment wise, on 144 synthetic traces, allowing fixes anywhere beats fixing only the root by about 0.135 in recovery score, and nearly all of that gain comes from downstream checks. On 27 τ\tau-bench retail cases, a different rule for where to restart improves recovery by about 20 points.

Critical strengths
  • Framing that the diagnosis does not necessarily mean it's the best place to repair is an interesting way or attributing errors, especially when prior work primarily focuses on the start of the errors being the best place to repair.
  • The paper is candid about its limits, including that the Holm correction was preregistered but missed and is reported afterwards.
Critical weaknesses
  • The system is very complex. In fact, much of the crucial details are delegated to the appendix and some terms are not clearly defined such as "primitive actions" and "location sufficiency" to name a couple.
  • The contribution of the work is not clear to me. The paper proposes a model (CSCR), but there are no comparisons to prior methods. There is also the Feasibility-Gated Joint Replay Search that is proposed but this isn't compared either. CRPA-R1 looks to be a constructed dataset from the paper; if so, is this something that is proposed? There is also no example of a CRPA-R1 trace.
  • The paper uses AgentRx's data, but never compares against the AgentRx method. The paper calls it the "AgentRx comparison" throughout, which makes it sound like a comparison against AgentRx.

Minor/Nit weaknesses

  • 6 decimal places seems excessive.
Questions for authors
  • Could you include a complete example CRPA-R1 trace in the main paper or appendix: the task, each agent's messages, the failure, the candidate fixes, and how a replay determines success without model calls?
  • Why not use AgentRx's diagnosis as the "root" in the root-following arm and then the CSCR recovery to show the value of recovering from non root can lead to better results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.