- Rating
- 1
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This work studies whether multi-agent debate can correct shared mistakes beyond what additional independent sampling achieves, using matched experiments with misleading premises and corrupted evidence.
Critical strengths
- The problem setup is important, and the paper's motivation is important.
Critical weaknesses
- The paper, to me, was a bit difficult to follow. It feels like some phrases or terms are just thrown into the paragraphs without proper definitions or explanations.
- It is not clear what is actually being measured. The authors keep noting "recovery", but the meaning of that value is not clearly stated.
- What is the intuition behind the selection of the fixed committee of five chat models: DeepSeek-V3-0324, Qwen3-Coder, Kimi-K2, GPT-OSS-120B, UnslopNemo-12B, and the verifier Qwen3-32B?
- Also, I am curious about how the datasets are chosen. To me, the chosen benchmarks, GSM8K and TruthfulQA, are both way too easy for the 5 chat models selected. If I understand correctly, "recovery" measures the "correct final decision conditional on an initially incorrect committee majority". If this value is large, that means that the initial round answers were mostly wrong, which I am not so convinced of, given the powerful set of chat models. Could the authors provide some real examples of the task question, how each model responded initially, and how that changed through debate?
- Other experimental details are not provided in detail, raising concerns about reproducibility. Could the authors add an implementation detail section organizing precisely how the experimental setups were decided?
Questions for authors
N/A