acceptodds
Withdrawn from ICLR 2027

Budgeted Error Correction in LLM-Based Multi-Agent Systems from Observable Intervention Responses

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Similar failures in LLM-based multi-agent systems can require different repairs, but locating a failure does not by itself determine which repair will help. Under a shared budget, selection must further weigh uncertain repair gains against costs across systems. We study whether already acquired intervention responses can guide this selection before repair success is known. To investigate this question, we introduce EvidenceGraph, which compares intervention responses and repair outcomes across failure mechanisms in systems matched on task and topology. We further characterize theoretically when these observations improve repair selection and bound the gap to full-information control, separating repair coverage and information limits from prediction and allocation errors. To turn observable intervention responses into repair decisions, we introduce Observable Action-Effect Prediction (OAE), which uses observed answers and evidence margins to predict sequence-specific replay agreement as a proxy for repair selection. Its greedy allocator prioritizes predicted gains over no intervention per unit cost under the shared budget. In independent evaluation, OAE improves task success by 8.18 percentage points over a prespecified learned comparator at equal realized terminal spend.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM-based multi-agent systems, Error propagation, Budgeted repair

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_F9VZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to choose repairs for failing LLM multi-agent systems when a total repair budget is shared across task instances, using intervention responses that have already been collected.

  • Benchmark: It introduces EvidenceGraph, a synthetic benchmark of 492 matched blocks drawn from three strata. In each block, one task is instantiated with six planted failure mechanisms in a four-role pipeline that ends in a deterministic ACCEPT/REJECT reducer. Every system is replayed under nine ordered repair sequences with fixed unit costs, giving 26,568 outcomes.

  • Theory: The paper defines best-fixed, observable-adaptive, and full-response values (Thm. 1). It bounds the regret of an observable shared-budget policy against a full-information oracle by four terms: coverage, information, twice the prediction error, and allocation error (Thm. 2).

  • Method: OAE computes 11 features from baseline, edge-mask, and conditional source-swap replays. It fits nine logistic heads that predict whether each sequence's replay will match its expected answer, then allocates greedily by predicted gain per unit cost.

  • Evaluation: Development, Primary, and Fresh cohorts are disjoint, with a prespecified comparator (PARE), matched-block bootstrap intervals, and max-T adjustment for multiple comparisons. On Fresh, OAE-Greedy reaches 42.44% success versus 34.26% for PARE at 2,592 units, an 8.18-point gain with 95% interval [6.02, 10.34].

Critical strengths
  1. Careful, preregistered evaluation on a matched design. The cohorts are disjoint, and scoring keys stay hidden until policies are locked. Resampling is done at the block level, with max-T adjustment applied separately to Primary and Fresh (Sec. 3.1, 5.1). Matched six-mechanism blocks with clean-fork replays (Fig. 2) let repair outcomes be compared from a common snapshot. This makes the Fresh contrast a genuine held-out test, which is more rigorous than most work on multi-agent failure analysis.

  2. The response landscape shows that adaptive selection matters. Outcomes depend strongly on the failure mechanism (Table 2, Fig. 3). At least one repair sequence succeeds for 99.77% of Fresh systems, versus 66.90% for the best single sequence (Sec. 5.2). This motivates the problem well.

Critical weaknesses
  1. The headline gain is measured against the authors' own variant, and a trivial equal-spend baseline is missing. The 8.18-point result in the abstract is relative to PARE, which is OAE plus residual corrections, not an external method.
  • The only fixed baseline spends half the budget: "Best fixed" spends 1,296 units (Table 4: "best fixed 1,296, others 2,592"). Yet applying the cost-2 sequence MS (edge mask, then source swap) to every system spends exactly 2,592 units. From Table 2, MS averages about 37.4% on Fresh. That would beat every equal-spend baseline in Table 3, including PARE, and leave OAE only about 5 points ahead rather than 8. It also beats the Exact, Lagrangian, and LocalSearch allocators (33.9–36.4%, Table 5).
  • Probe cost is excluded: The budget "covers terminal repairs, excluding replay-bank acquisition" (Sec. 4.3). OAE's observations are themselves edge-mask and source-swap replays on every system, costing about 1–2 units each, which is comparable to the whole terminal budget. Fixed and random baselines don't need these probes.
  • Binary answers make the probes informative: Answers are ACCEPT/REJECT and the baseline is wrong 94% of the time. Seeing that a probe flips the answer therefore nearly reveals success, and only OAE gets this signal. Together these mean it is not yet established that observable responses guide budgeted repair better than simple alternatives. Report fixed MS at 2,592 units, with block-bootstrap intervals for OAE minus MS. Then either charge probe cost to the budget, or compare against a simple rule that uses the same probes.
  1. The learned predictor is near chance, so it is unclear what OAE learns, and the setting is narrow relative to the framing.
  • The predictor is weak. On Fresh, its AUROC is 0.533 and its calibration error is 0.428 (Sec. 5.5).
  • Better optimization of its scores hurts. Allocating exactly on the predicted gains loses 8.56 points to the greedy rule (Table 5). The win may come from the greedy rule's structure rather than from the predictions. -The gain is concentrated. It comes from two mechanisms: terminal corruption (+25.5 points) and shared retriever (+31.9). It is partly offset on source mismatch (−6.4), where OAE gets 26.9% while fixed MS gets 94.0% (Fig. 6a). The ranking also reverses at budgets of 324 and 1,296 units (Sec. 5.4), which the abstract does not mention.
  • The theory is never instantiated. None of the Thm. 2 terms are estimated. With a calibration error near 0.43, the prediction-error term alone is likely near 0.86, which makes the bound vacuous here.
  • The benchmark is narrow. It is fully synthetic, and the same model family generated and reviewed all records with no human check. Replays use one LLM. The "multi-agent system" ends in a deterministic binary reducer, and mechanisms map almost one-to-one to repairs.
Questions for authors
  1. Please report fixed MS on all systems at 2,592 units on Primary and Fresh, with matched-block intervals for OAE minus MS. Why was "Best fixed" limited to a cost-1 sequence? If OAE's advantage stays clearly positive after max-T adjustment, I would weigh Weakness 1 much less.
  2. What is the average unit cost of acquiring the observations per system? How do the results change if it is charged against the budget? How does OAE compare with a simple rule that uses the same probes: choose the cheapest probed sequence whose answer differs from baseline; otherwise, a default sequence?
  3. Is the 0.533 AUROC measured against task success or against replay agreement? What does OAE-Greedy achieve with permuted or random scores under the same allocator? Why does exact maximization of the predicted gains lose 8.56 points?
  4. What are the full-information budgeted value and the Thm. 2 terms on Fresh at 2,592 units? Knowing where OAE sits relative to them would show whether the theory says anything about the method actually run.
  5. Are replays deterministic, and at what temperature? If observing a probe's answer equals its final outcome, the "hidden outcome" setup is weaker than described. Do the conclusions hold with a second behavior model?
  6. The cost-matched random baseline's intervention rate equals OAE's on Fresh (61.96%) but PARE's on Primary (87.58%). Which policy is it matched to?
Reviewer_oV2UOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies repair selection for LLM-based multi-agent systems under a shared terminal-action budget. It introduces EvidenceGraph, a controlled benchmark with three source strata, a fixed four-role system, six planted failure mechanisms, and nine predefined repair sequences. OAE uses 11 observable response features to train nine separate logistic predictors of sequence-specific replay agreement. A fixed greedy allocator selects repairs using predicted gain over no intervention per unit cost. On the independent Fresh cohort, the paper reports an 8.18 percentage-point success improvement over PARE at equal terminal spend. The evaluated scope is offline selection on an already acquired replay bank.

Critical strengths
  1. The motivation is important: identifying an error does not establish which intervention will repair it, and repair opportunities must compete for limited resources. The matched construction provides a useful setting for examining this distinction.

  2. The experimental organization is a positive aspect. Fitting, configuration selection, and Fresh evaluation are separated, and the paper reports matched-block uncertainty, budget sensitivity, and cases in which OAE does not outperform its comparator. This makes the reported operating regime easier to assess.

Critical weaknesses
  1. The evidence for general applicability is limited. All three source strata are converted into a common evidence-support decision task within a fixed four-role framework, and the main replays use one execution-model family. Chain/star variation does not establish transfer to different orchestrators, task operations, or heterogeneous model combinations. I therefore remain uncertain how much of the observed advantage reflects general repair-selection principles rather than the particular benchmark construction.

  2. The scientific justification connecting the fixed design to effective, resource-efficient repair remains incomplete. The choice of five actions, nine sequences (all nonempty sequences start with edge masking), and their relative costs needs clearer motivation. The training procedure is specified, but it is less clear why replay agreement should reliably predict the task-success gains needed for allocation, particularly given the poorer realized performance of Exact allocation and the budget-dependent ranking reversal. The theoretical decomposition does not by itself establish that OAE has small task-success prediction error. Moreover, the budget uses assigned terminal-action units and excludes response acquisition, so the results do not establish an end-to-end token, latency, or computation advantage. These issues leave the broader insight beyond the substantial benchmark and pipeline construction insufficiently established for me.

Questions for authors
  1. Please provide a representative Controlled EvidenceGraph task and clarify the operational injection rules for the six failure mechanisms. What motivated the five actions and nine sequences, especially requiring edge masking as the first step of every nonempty sequence?

  2. Please strengthen the evidence for generalizability with targeted additional validation across (i) different execution-model families, including heterogeneous LLM combinations where feasible; (ii) benchmarks with distinct task requirements, beyond converting different sources into the same evidence-support decision format; and (iii) alternative MAS orchestrators or role configurations, beyond chain/star variants of the fixed four-role framework. A representative extension would help clarify which findings transfer; a full cross-product of all settings is not requested. Please also explain why DeepSeek V4 Flash was selected and which features, action definitions, or fitted parameters require adaptation in a new setting. If additional validation is not feasible during rebuttal, please clearly delimit the supported scope and explain the remaining uncertainty.

  3. Why is sequence-specific replay agreement an appropriate surrogate for repair success? How does its prediction error connect to the task-success error in Theorem 2, and how should readers interpret Exact allocation performing worse despite optimizing the surrogate more accurately?

  4. What motivates the 2,592-unit operating budget and the relative action costs? Given the low-budget ranking reversal, please clarify the intended operating regime and whether including observation acquisition and actual token/latency/computation costs would change the conclusion. Please distinguish evidence already available from proposed future validation.

Reviewer_xwy7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to select repairs for LLM-based multi-agent systems under a shared budget. The authors introduce EvidenceGraph, a benchmark with different failure mechanisms and repair sequences. They also propose OAE, which predicts repair outcomes from observed intervention responses, and use a greedy algorithm to allocate the repair budget across systems.

Critical strengths

1.The paper addresses a useful problem. Identifying where a failure occurs does not necessarily tell us how to fix it, especially when repair costs are limited. 2. The benchmark uses matched tasks with different failure mechanisms, which helps compare the effects of different repair operations.

Critical weaknesses

1.The proposed method mainly combines manually designed intervention features, logistic regression models, and a greedy budget allocation strategy. While the overall problem setting is interesting, the individual components are fairly standard. It is not clear what new methodological insight the approach provides beyond applying these techniques to repair selection.

2.A major concern is the quality of the predicted repair scores. The reported AUROC is only 0.533, and the task-success ECE is 0.428. Moreover, exact allocation using the same predicted scores performs worse than the greedy approach. These results raise questions about how well the predictions reflect actual repair benefits.

Questions for authors
  1. Why is replay agreement used as the prediction target rather than task success? Could the authors compare these two objectives more directly?
  2. How would the performance and cost change if the intervention probes were included in the total budget?
  3. Can the authors evaluate the approach on naturally occurring multi-agent failures or different agent architectures?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.