- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents TrueMAD, a framework that creates different channels between a targeted agent and its peer. Different from existing multi-agent debate protocols, which often broadcast the conversation and reasoning to every agent, TrueMAD aims to address conformity (suppressed by the majority answer) and unaddressed reasoning in existing MAD.
Critical strengths
-
Separating the channel and avoiding broadcasting everything to every agent is a sound and intuitive design.
-
Multi-agent debate has been well explored in the community, making it hard to achieve consistent improvement, especially with less hand-designed or post-hoc heuristics. However, the evaluation on AIME, GPQA and TruthfulQA shows that TrueMAD is effective across models and datasets.
Critical weaknesses
-
The ordered source-target pair, noted in line 225, makes the framework expensive. If I understand correctly, assuming 3 agents in a debate, this will form 6 ordered pair, i.e., (1,2), (1,3), (2,3), (3,2), (3,1), (2,1) and this can go for multiple rounds. So scaling this framework is harder than existing work (for example, if we want 10 agents to participate in a debate).
-
Several representative baselines are missing. For example, Reconcile [1] and Mixture-of-Agent [2] should be compared. Most importantly, it is not clear whether all the baselines are compute-matched. That is, if TrueMAD uses 3 model instances with 5 rounds, the baselines should also get up to 15 LLM calls. Or the token usage should roughly match.
-
The authors should also report compute-matched self-consistency [3] and Best-of-N [4]. I would expect these to be strong but easy-to-implement baselines, and that would justify whether we need a much more complex pipeline.
-
In section 4.4, there is an analysis about "auxiliary-call budget", but I didn't fully get it. What does "auxiliary-call budget is allocated to feedback" mean? My understanding is that this is different from the LLM call or token budget I mentioned in points 2 and 3. But if not, please clarify. Also in this budget-matched comparison, the margin is smaller than the main result table, suggesting that the main result table is not budget-matched.
References:
Questions for authors
-
In lines 175-179, the authors mention: "Increasing the number of agents or debate rounds expands answer exposure, but does not ensure that any agent examines a specific peer claim. Strengthening a generic revision prompt may elicit more elaborate responses while leaving the source and target of feedback implicit. Imposing fixed adversarial roles guarantees opposition rather than evidence-sensitive engagement, and may unnecessarily challenge valid reasoning." I am curious whether there are any empirical results or prior work that can support these hypotheses.
-
Can we use different model backbones in the debate, instead of using the same model multiple times?
4 comments
Response to Reviewer fFLT (2/2)
3. Token-matched Self-Consistency and Best-of-N (W3)
We agree that these are important controls. We added Self-Consistency [5] and Best-of-N [6], selecting candidate counts K to match TrueMAD’s total-token budget through independent sampling and candidate selection. Post-experiment token accounting confirmed comparable consumption. Both baselines use Qwen3-30B-A3B and three independent seeds, consistent with the main experiments.
| Method | AIME Final-answer accuracy (%) | GPQA-Diamond Final-answer accuracy (%) |
|---|---|---|
| Self-Consistency | 29.26 ± 0.64 | 54.71 ± 1.05 |
| Best-of-N | 32.22 ± 1.92 | 53.87 ± 0.58 |
| TrueMAD | 34.81 ± 1.70 | 56.73 ± 0.29 |
Under comparable token budgets, TrueMAD exceeds the stronger baseline on AIME and GPQA-Diamond by 2.59 and 2.02 percentage points, respectively. This supports targeted peer engagement as an effective use of inference compute relative to additional independent sampling or candidate selection.
4. Clarifying the auxiliary-call budget and supporting the motivation (W4, Q1)
For W4, “auxiliary calls” are additional LLM calls that generate feedback before answer revision. Section 4.4 matches their number across TrueMAD, ComputeMAD, and Self-AdviceMAD while varying feedback generation. This matches calls, and token consumption can differ. The main table matches agent counts and response rounds, while the new SC/BoN comparisons match total-token budgets. We clarified these distinctions in the revised Section 4.4.
For Q1, we refined the motivation and added supporting literature to distinguish protocol-level observations from empirical findings. See the revised manuscript's motivation discussion (lines 181–189).
We hope these responses address your concerns and look forward to your further feedback.
[1] Chen J, Saha S, Bansal M. ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs. ACL 2024.
[2] Yoffe L, Amayuelas A, Wang W Y. DebUnc: Improving Large Language Model Agent Communication with Uncertainty Metrics. Findings of EMNLP 2025.
[3] Cui Y, Fu H, Zhang H, et al. Free-MAD: Consensus-Free Multi-Agent Debate. Findings of ACL 2026.
[4] Wang J, Wang J, Athiwaratkun B, et al. Mixture-of-Agents Enhances Large Language Model Capabilities. ICLR 2025.
[5] Wang X, Wei J, Schuurmans D, et al. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.
[6] Lightman H, Kosaraju V, Burda Y, et al. Let's Verify Step by Step. ICLR 2024.
Dear Reviewer,
We have carefully addressed all your comments and provided point-by-point responses. As the rebuttal period is nearing its end, we would be grateful if you could take a moment to review our replies.
Please let us know if you have any further concerns or suggestions.
Thank you very much.
Thank the authors for their detailed responses. Whilte they have mostly addressed my questions, the scalability issue remains. Therefore, after careful thought, I decided to keep my original score.
Response to Reviewer fFLT (1/2)
We thank the reviewer for the constructive feedback. We added the requested baselines and token-budget comparisons and clarified TrueMAD's computational overhead and motivation.
1. Scalability and compatibility with different backbones (W1, Q2)
Your interpretation is correct. The fully connected advice stage requires N(N−1) calls per debate update. Our four-agent setting is consistent with recent MAD studies. ReConcile [1] and DebUnc [2] use three agents, and Free-MAD [3] evaluates three- and four-agent configurations.
Our objective is a simple, effective plug-in that makes peer engagement explicit while preserving the underlying debate protocol. The all-pairs implementation has quadratic advice-call overhead. Extending it to larger groups is an important direction. Pruning redundant interactions or routing advice to a bounded number k of peers per agent would reduce advice calls to O(Nk) per update while retaining directed feedback and avoiding exhaustive pairwise communication.
The text-based advice interface can accommodate mixed-backbone teams. Our TrueDMAD experiments demonstrate compatibility with heterogeneous reasoning strategies.
2. Additional representative MAD baselines (W2)
Following your suggestion, we added ReConcile, MoA [4], and Free-MAD. Comparisons use Qwen3-30B-A3B, the main experiments' evaluation data and debate-round setting, and three independent seeds. ReConcile and MoA use same-backbone implementations. Metrics follow the main table, with values reported as mean ± standard deviation (%).
TrueMAD outperforms all three added baselines on both metrics and datasets. TrueMAD exceeds the strongest added baseline in Vote ACC by 4.07 points on AIME and 2.14 points on GPQA-Diamond. These comparisons cover confidence-based consensus, layered aggregation, and consensus-free debate. The token-matched comparisons below address the budget question.
Rebuttal time constraints limit the current comparisons to AIME and GPQA-Diamond. We will include TruthfulQA results in the updated manuscript once completed.