- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how shared plans affect the accuracy and agreement of multi-agent LLM systems. It introduces a controlled planner–executor framework and two statistics, correctness ICC and excess answer agreement, to examine different failure patterns. Across seven models and three reasoning domains, strong coordination reduces accuracy in 19 of 21 settings. The authors use statistical tests, compute-budget controls, and selected failure cases to support their findings and argue that executor agreement alone is not a reliable sign of correctness.
Critical strengths
The paper studies an important practical question: whether a shared plan improves multi-agent reasoning or spreads errors. The seven-model, three-domain comparison reports accuracy drops in 19 of 21 settings, providing useful evidence that this design choice needs careful testing.
The evaluation uses question-level statistical tests and includes compute-budget controls. These checks strengthen the empirical accuracy comparison, although they do not resolve my concerns about the proposed failure diagnosis.
Critical weaknesses
-
The main diagnostic does not separate marginal answer concentration from correlated failure. In Section 3.2, κ compares within-question agreement with answer frequencies pooled across questions, so it can increase even when executors remain independent within each question. This undermines the claimed distinction between the two failure mechanisms, which is a central contribution.
-
The analysis does not directly measure increased agreement on wrong answers. Section 5.4 infers this from lower accuracy and higher overall agreement, but these changes could occur on different questions. Since a wrong-only analysis is left for future work, the evidence does not establish the proposed correlated-failure explanation.
Questions for authors
-
Clarification: How does the pooled chance-agreement term in Equation 4 distinguish within-question dependence from changes in individual answer distributions? Can you show that the model grouping remains after accounting for question-specific answer distributions?
-
Evidence needed for the current claim: Does agreement on identical wrong answers increase in the models labeled as cascade-prone? Since overall agreement and accuracy can change on different questions, a wrong-answer-specific analysis would materially affect my assessment of the proposed mechanism split.
2 comments
Reply to review continued:
We thank the reviewer — both weaknesses were correct and drove the main changes in this revision.
W1/Q1. With near-unique answers per question, pooled p_e is small and κ tracks raw agreement (Spearman 1.00 across the 21 Strong cells, Appendix G). We now say this explicitly in §3.2 and Appendix G, and we no longer read Δκ ≈ 0 as "benign narrowing". We did not apply a question-specific chance correction; instead we measure wrong-answer agreement directly.
W2/Q2. Conditional Wrong Agreement (CWA, Table N1) is computed within question and conditions on both executors being wrong: among wrong pairs, the fraction with identical normalized output. Under Strong it rises in all 9 cascade-group cells (per-model means +0.197/+0.185/+0.198) and in 9 of the 12 others (+0.071/+0.041/+0.083/+0.036). The groups separate at model level and overlap at cell level — four non-cascade cells have ΔCWA ≥ +0.10 — so non-cascade models are not free of wrong-answer convergence; they show less of it. We removed the sentence that inferred wrong-answer concentration from aggregate agreement and accuracy.
Two limits we state. CWA is a concentration measure, not a dependence measure: executors are separate calls given (question, plan), so conditional independence holds by construction and concentration onto a shared wrong answer is the mechanism. And CWA alone does not attribute the effect to sharing; the independent-plan control (N2) addresses that, with partial accuracy recovery only.
Reply to review.
We sincerely thank the reviewer for their careful reading and for identifying the core methodological limitation regarding pooled pe and wrong-answer convergence. Your feedback directly prompted the most substantial new experiments in our revision. We have addressed your concerns with new metrics and extensive new data in Appendix N.
W1: The main diagnostic (κ) does not separate marginal answer concentration from correlated failure.
Response: We completely agree. In the original submission, pooled pe could conflate per-question narrowing with genuine within-question coupling. Prompted by your critique, we verified (Appendix G) that in our domains, κ is essentially a monotone transform of raw pairwise agreement (Spearman rs=1.00). We have rewritten Section 3.2 to explicitly state this limitation. We no longer claim that κ isolates correlated failure; instead, we use it purely as a chance-corrected summary of raw agreement.
W2: The analysis does not directly measure increased agreement on wrong answers.
Response: We agree that inferring wrong-answer convergence from overall agreement and accuracy drops was insufficient. To resolve this, we introduced a new, direct metric: Conditional Wrong Agreement (CWA). CWA strictly conditions on trials where ≥2 executors are wrong and measures the probability that they output the identical wrong string, completely bypassing pooled pe. As detailed in Appendix N, Table N1, CWA surges massively under Strong coordination specifically in the cascade-prone models (e.g., Mistral-7B/GSM8K rises from 0.126→0.434; Qwen2.5-1.5B/GSM8K from 0.101→0.474). The cross-domain mean ΔCWA is +0.19 for the cascade group versus +0.06 for the degradation group. All wrong-answer claims in the revised manuscript now rest strictly on CWA and qualitative trace evidence, fully addressing your concern.
We hope these substantial additions and the softened, more precise claims in the revised text address your soundness concerns and warrant a reconsideration of your score.