- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines whether message-level causal credit remains valid when the communication topology of a multi-agent system changes. It proposes paired message interventions, a cross-graph stability measure, and an exposure sufficiency diagnostic. The framework quantifies the error of reusing fixed credit scores and identifies cases where structural summaries fail to distinguish different message effects. Experiments on controlled communication tasks and 2WikiMultiHopQA show substantial credit variation in constructed settings, but weaker evidence on natural tasks.
Critical strengths
- The paper addresses a meaningful problem. Message importance can change when alternative communication paths become available, which matters for attribution and topology optimization in multi-agent systems.
- The paired intervention design is reasonably controlled. Comparing deletion and conflicting-message replacement is useful, and the authors are transparent about the limitations of their natural-task results.
Critical weaknesses
-
The main stability bound follows directly from the midpoint and half-range of observed credits, while the exposure sufficiency bound applies the same idea within groups. These results are mathematically correct but rather elementary.
-
Also, the four-agent tasks are designed with saturated outcomes, leading to the same 0.5 error bound across all 12 settings. This makes it difficult to assess whether the diagnostics capture nontrivial behavior in realistic multi-agent workflows.
-
On 2WikiMultiHopQA, the graph-level F1 difference is not statistically significant, and most message interventions have no measurable effect. The natural-task results therefore provide limited support for the practical importance of cross-graph credit instability.
-
The exposure-conditioned reconstruction succeeds in only one of four settings, and structural predictors barely outperform a constant baseline. The framework mainly diagnoses already measured effects rather than enabling reliable credit estimation on unseen communication graphs.
Questions for authors
- Beyond the midpoint and half-range identities, what is the main technical contribution relative to existing causal attribution and effect-transport methods?
- Can the authors evaluate the diagnostics on larger, more realistic communication graphs with multiple interacting messages?
- Given the limited held-out reconstruction results, what evidence supports the practical value of exposure-conditioned credit transfer?