acceptodds
Withdrawn from ICLR 2027

Same Message, Same Credit? Diagnosing Credit Stability under Rerouting in Multi-Agent LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In multi-agent question answering, determining whether a message's contribution remains valid after communication routes change is essential for reliable system redesign, because outdated scores can misidentify the evidence exchanges on which answers depend. Existing collaboration and attribution methods do not by themselves quantify the error of reusing contribution scores across communication graphs. Alternative routes can make an indispensable message redundant without changing its content. Structural summaries can also merge cases with different effects, preventing predictors from distinguishing their contributions. We propose a three-module diagnostic framework: paired measurement compares retained and disrupted messages on matched tasks, and cross-graph stability computes the optimal fixed credit and its exact worst-case error over evaluated graphs. Exposure sufficiency converts equal-summary effect differences into empirical worst-case error bounds and identifies the responsible states. Measured controlled credits give optimal fixed-score error in all 12 settings, factorial equal-summary bounds are –, and 160 questions from the 2WikiMultiHopQA multi-hop question answering dataset yield a statistically uncertain graph difference in Token-Overlap F-Measure (F1) (95% interval ). These diagnostics distinguish measuring a message's contribution from establishing its reuse, helping developers assess existing scores when communication patterns change.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent systems, failure attribution, decision sufficiency, post-failure recourse, causal diagnosis, decision-focused evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nut2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether message-level causal credit remains valid when the communication topology of a multi-agent system changes. It proposes paired message interventions, a cross-graph stability measure, and an exposure sufficiency diagnostic. The framework quantifies the error of reusing fixed credit scores and identifies cases where structural summaries fail to distinguish different message effects. Experiments on controlled communication tasks and 2WikiMultiHopQA show substantial credit variation in constructed settings, but weaker evidence on natural tasks.

Critical strengths
  1. The paper addresses a meaningful problem. Message importance can change when alternative communication paths become available, which matters for attribution and topology optimization in multi-agent systems.
  2. The paired intervention design is reasonably controlled. Comparing deletion and conflicting-message replacement is useful, and the authors are transparent about the limitations of their natural-task results.
Critical weaknesses
  1. The main stability bound follows directly from the midpoint and half-range of observed credits, while the exposure sufficiency bound applies the same idea within groups. These results are mathematically correct but rather elementary.

  2. Also, the four-agent tasks are designed with saturated outcomes, leading to the same 0.5 error bound across all 12 settings. This makes it difficult to assess whether the diagnostics capture nontrivial behavior in realistic multi-agent workflows.

  3. On 2WikiMultiHopQA, the graph-level F1 difference is not statistically significant, and most message interventions have no measurable effect. The natural-task results therefore provide limited support for the practical importance of cross-graph credit instability.

  4. The exposure-conditioned reconstruction succeeds in only one of four settings, and structural predictors barely outperform a constant baseline. The framework mainly diagnoses already measured effects rather than enabling reliable credit estimation on unseen communication graphs.

Questions for authors
  1. Beyond the midpoint and half-range identities, what is the main technical contribution relative to existing causal attribution and effect-transport methods?
  2. Can the authors evaluate the diagnostics on larger, more realistic communication graphs with multiple interacting messages?
  3. Given the limited held-out reconstruction results, what evidence supports the practical value of exposure-conditioned credit transfer?
Reviewer_VKyAOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether message-level credit in multi-agent LLM systems remains stable when communication topologies change. The authors propose a diagnostic framework consisting of paired intervention measurements, cross-graph stability diagnostic, and exposure sufficiency diagnostic. The framework quantifies the worst-case approximation error of reusing a fixed credit score and identifies cases where structural summaries fail to distinguish different message effects. Experiments demonstrate credit variations across topologies. However, the natural-task results show limited statistical evidence, and the predictive benefits of structural information remain modest.

Critical strengths
  1. The paper investigates an interesting and relatively underexplored question regarding the stability and reusability of message-level credit across different multi-agent communication topologies.
Critical weaknesses
  1. Limited methodological novelty. The main theoretical results are relatively straightforward applications of midpoint-based minimax approximation. In particular, Proposition 2 in Section 3.4 essentially extends Proposition 1 to states sharing the same exposure summary, without introducing substantial additional technical novelty. Moreover, as acknowledged in Section 4.1, the saturated endpoints in the controlled experiments largely follow from the experimental construction, making the identical minimax error of 0.5 across all 12 settings less informative.
  2. Insufficient empirical evidence of practical utility and generalizability. As acknowledged in Section 4.6, the natural-task evaluation is limited to one dataset, one model setting, and two communication graphs. The aggregate credit drift in Table 4 is not statistically significant, with only 18 of 320 paired F1 effects being nonzero. Furthermore, the held-out reconstruction in Table 3 succeeds in only one of four settings, while the structural predictors in Table 6 yield only marginal improvements over a pooled scalar baseline. These results provide limited evidence that the proposed diagnostics offer practical benefits or generalize to more complex multi-agent systems.
Questions for authors
  1. Given that Propositions 1 and 2 are both based on standard midpoint-based minimax approximation, what are the key technical contributions beyond applying these existing mathematical principles to message-level credit? How do the proposed diagnostics provide actionable guidance for multi-agent communication design?
  2. Given the limited improvements in held-out credit reconstruction (Table 3) and structural prediction (Table 6), how can the proposed diagnostics be reliably applied to unseen or more complex communication topologies? Could the authors provide stronger empirical evidence demonstrating their practical benefits in realistic multi-agent scenarios?
Reviewer_ggatOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines whether a message’s intervention effect remains valid after communication routes change in multi-agent LLM workflows. It proposes diagnostics to measure cross-graph variation and determine whether structural summaries distinguish different effects. Experiments cover controlled routing tasks and 160 2WikiMultiHopQA questions. The paper distinguishes finite-sample bounds from population inference, noting that aggregate drift on natural questions is statistically uncertain and exposure-based reconstruction succeeds in only one o f four tested settings.

Critical strengths
  • Clear separation of measurement and generalization. The paper carefully defines its interventions and diagnostics, explains what the empirical bounds establish, and acknowledges the limitations of its bootstrap intervals. This helps avoid treating attribution from one execution as a reusable score.

  • Transparent reporting of negative results and limitations. The paper reports statistically uncertain aggregate drift on natural tasks and successful reconstruction in only one of four tests. It also acknowledges the limits of its structural descriptors and distinguishes local relay credit from final-answer credit without claiming to identify mediation.

Critical weaknesses
  • The practical value of the diagnostics needs stronger evidence. The core mathematical results are simple, and the controlled tasks largely produce expected routing effects. The natural-task study covers only two small graphs and one model setting, leaving it unclear how useful the diagnostics would be in more complex systems.

  • The prediction comparison lacks a basic baseline. Only 18 of 320 measured F1 effects are nonzero, so always predicting zero would achieve an MAE of at most 0.05625—lower than all fitted predictors in Table 6. This baseline is needed to assess whether structural information provides useful predictive gains.

Questions for authors
  • Baseline comparison: Does topology conditioning outperform a constant-zero predictor in MAE and RMSE?

  • Relay consistency: Why does Relay A’s credit change when only a downstream connection is added? Are its inputs and execution settings unchanged?

  • Diagnostic usefulness: Can you show an example where the diagnostic identifies missing structural information and adding it improves prediction on held-out data?

  • Broader validation: Do the findings hold in larger communication graphs or additional natural tasks where the system answers more questions correctly before intervention?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.