- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper addresses the limitations of relying on a single reasoning trace and examines the vulnerabilities of unconstrained interaction within multi-agent debate frameworks. The authors observe that existing MAD approaches generally lack reliability constraints on both the content and the propagation of exchanged information, allowing flawed reasoning to spread across agents alongside valid arguments. To mitigate this issue, the authors introduce StageMAD, a three-stage framework consisting of local self-correction, restricted peer guidance, and mutual verification under close votes. By integrating these three stages, the authors suggest that the proposed method produces more reliable final answers. The authors validate their approach using Qwen2.5-7B-Instruct, Qwen3.5-9B, and DeepSeek-Math-7B across multiple benchmarks.
Critical strengths
- The authors address an important failure mode in MAD frameworks where correct minority answers are overwritten by peer influence.
- The authors characterize the theoretical vulnerabilities of multi-agent debate and present a well-motivated framework as a potential remedy.
- Rather than limiting the evaluation to final accuracy, the authors incorporate diverse analytical metrics, including question-level transitions(w->c / c->w) as well as a frozen-pool continuation setup.
Critical weaknesses
- The problem setting targeted by the authors appears overly specific. I believe the proposed method should be evaluated against both general MAD frameworks and simple non-debate inference baselines. When compared against these standard baselines, the performance gains achieved through the proposed multi-stage pipeline appear rather marginal.
- The performance improvement relative to the computational cost is quite limited. For instance, with DeepSeek-Math-7B-Instruct, StageMAD consumes over ten times the token budget of standard CoT while occasionally yielding lower accuracy.
- Could the authors clarify the exact performance margin of StageMAD over a simple single-pass inference baseline?
- When proposing a multi-stage framework, conducting a thorough ablation study is essential to justify the necessity of each individual stage. However, the current manuscript seems not provide sufficient ablation experiments to support this design.
- While Figure 4 presents an ablation analysis, the reported results do not sufficiently justify the inclusion of each stage. The authors need to rigorously verify both the necessity and the reliability of every individual stage.
- Furthermore, the overall framework appears to rely heavily on Stage 3. Even if arbitrary responses were placed in Stages 1 and 2 and only Stage 3 were executed, the final performance would likely show minimal difference.
- According to Figure 2, the correction effect of the framework is substantial on relatively straightforward benchmarks such as MATH and GSM8K, whereas the gains on more challenging benchmarks like MMLU-Pro and AIME24 are negligible (or even worse). This disparity suggests that the effectiveness of the method fluctuates considerably depending on task difficulty.
- In addition, the main experimental results fail to isolate the contribution of controlled communication, which is the central claim of this paper. The non-communicative PRM self-revision baseline performs comparably to or even outperforms StageMAD across several settings. A direct comparison against communication-based baselines equipped with the same PRM is necessary for a fair evaluation.
- Debate frameworks are primarily needed when solving complex problems. Exhibiting weaknesses on difficult benchmarks undermines the practical utility of StageMAD. Given the increasing complexity of tasks expected from modern LLMs, the authors should more clearly demonstrate whether StageMAD remains effective on demanding tasks.
- The experiments are restricted to models with relatively small capacities of 9B parameters or fewer. The authors need to demonstrate scalability by verifying whether their findings generalize to more capable models. In particular, the performance gains appear smallest on Qwen3.5, which is the strongest among the tested models. Consequently, it remains uncertain whether this framework provides meaningful benefits for larger models or heterogeneous agent configurations commonly adopted in recent MAD literature.
- Several baseline results raise questions regarding their validity. With Qwen3.5, CoT-SC@15 achieves only 53.2 on GSM8K, which is lower than the accuracy of the same model on MATH (65.0) and more than 30 points below all other methods in Table 1 (ranging from 85.8 to 95.3), despite all methods starting from independent samples generated by the same task model.
- As a researcher working extensively with LLMs, I have observed in my own experiments that Qwen3.5 achieves substantially higher performance (than the scores reported by the authors) on these benchmarks even without repeating CoT 15 times. These unexpectedly low figures suggest a potential flaw in the experimental implementation. The authors should inspect their pipeline for issues such as output truncation or answer-parsing failures.
- The main manuscript is not sufficiently self-contained. Essential details required to understand and reproduce the proposed method are either relegated to the appendix or omitted entirely. Information that is indispensable for comprehending the core methodology must be included in the main text rather than the appendix, as deferring such details merely serves to circumvent the page limit. For example,
- The number of agents used in the debate
- Details of the PRM backbone, generation hyperparameters, and evaluation dataset sizes
- The specific values and selection criteria for the core thresholds α2, δU, and δv in Equations 3 to 5
- Clear explanations of how group representatives are selected in Stage 3 (Line 280) and what the phrase "with the help of the PRM" specifically entails during source guidance generation (Line 266)
Questions for authors
The terminology used for the stages is inconsistent across the manuscript. The abstract (Lines 22 to 23) refers to the three stages as "local self-correction, directed logical guidance based on process evidence, and mutual critique of residual disagreements," whereas Sections 1 and 4 describe them as "local self-correction, restricted peer guidance, and mutual verification under close votes." Similarly, the messages exchanged in Stage 2 are variously termed "directed logical hints" (Line 105), "constraint-level guidance," "answer-free guidance," and "restricted guidance." Adopting a unified set of terms throughout the paper would improve readability for the reader.
7 comments
Q2: When proposing a multi-stage framework, conducting a thorough ablation study is essential to justify the necessity of each individual stage. However, the current manuscript seems not provide sufficient ablation experiments to support this design.
- While Figure 4 presents an ablation analysis, the reported results do not sufficiently justify the inclusion of each stage. The authors need to rigorously verify both the necessity and the reliability of every individual stage.
- Furthermore, the overall framework appears to rely heavily on Stage 3. Even if arbitrary responses were placed in Stages 1 and 2 and only Stage 3 were executed, the final performance would likely show minimal difference.
A2: The stage-wise results do not support the interpretation that Stage 3 accounts for most of the improvement. Table 13 shows that Stage 1 contributes 4.4 and 6.8 percentage points on MATH with Qwen2.5-7B-Instruct and Qwen3.5-9B, respectively, whereas Stages 2 and 3 together contribute 1.8 and 1.1 points. To directly examine your suggestion that the first two stages could be skipped, we additionally evaluated Stage 3 alone from the initial candidate pool.
| Method | MATH | GSM8K | MMLU-Pro | AIME24 |
|---|---|---|---|---|
| Stage 3 only | 66.7% | 92.1% | 57.1% | 13.3% (4/30) |
| StageMAD (Table 1, three-run mean) | 71.7% | 93.8% | 58.3% | 15.6% |
Stage 3 alone scored below the Table 1 averages on all four datasets. It also fell short of the full pipeline on all four datasets in the single-seed comparison from the same R0 candidates. This shows that the preceding stages contribute to the final performance. Figure 4 addresses a complementary question by fixing the R1 or R2 candidate pool and comparing the subsequent communication policies.
Q3: According to Figure 2, the correction effect of the framework is substantial on relatively straightforward benchmarks such as MATH and GSM8K, whereas the gains on more challenging benchmarks like MMLU-Pro and AIME24 are negligible (or even worse). This disparity suggests that the effectiveness of the method fluctuates considerably depending on task difficulty.
- In addition, the main experimental results fail to isolate the contribution of controlled communication, which is the central claim of this paper. The non-communicative PRM self-revision baseline performs comparably to or even outperforms StageMAD across several settings. A direct comparison against communication-based baselines equipped with the same PRM is necessary for a fair evaluation.
- Debate frameworks are primarily needed when solving complex problems. Exhibiting weaknesses on difficult benchmarks undermines the practical utility of StageMAD. Given the increasing complexity of tasks expected from modern LLMs, the authors should more clearly demonstrate whether StageMAD remains effective on demanding tasks.
- The experiments are restricted to models with relatively small capacities of 9B parameters or fewer. The authors need to demonstrate scalability by verifying whether their findings generalize to more capable models. In particular, the performance gains appear smallest on Qwen3.5, which is the strongest among the tested models. Consequently, it remains uncertain whether this framework provides meaningful benefits for larger models or heterogeneous agent configurations commonly adopted in recent MAD literature.
A3: Figure 2 presents the framework overview. If you are referring to the correction–corruption counts in Figure 3, these are raw counts across datasets with different sizes and task characteristics. AIME24, for example, contains only 30 questions. These counts therefore do not isolate task difficulty as the cause of the performance differences. We agree that the gains vary across models and tasks, but the results do not establish a general relationship in which StageMAD becomes less effective as task difficulty increases.
The contribution of controlled communication is examined separately from the full-system comparison. Table 1 compares complete inference methods, whereas Figure 4 fixes the Round-1 or Round-2 candidate pools before comparing subsequent communication policies. These comparisons separate the effect of the communication policy from the quality of the preceding candidates. With the Round-1 pool fixed, StageMAD achieves the highest or tied-highest accuracy among the MAD policies compared in 11 of the 12 settings.
To address your request for a communication baseline equipped with the same PRM, we additionally evaluated MAD-M² (S) with the same GenPRM used by StageMAD on Qwen2.5-7B-Instruct. This variant uses GenPRM to filter the peer reasoning trajectories available to agents. The table reports one run for MAD-M² (S) + GenPRM and the three-run means from Table 1 for the other methods.
| Dataset | MAD-M² (S) | MAD-M² (O) | MAD-M² (S) + GenPRM | StageMAD |
|---|---|---|---|---|
| MATH | 67.4% | 65.7% | 66.6% | 71.7% |
| GSM8K | 91.7% | 90.7% | 91.3% | 93.8% |
| MMLU-Pro | 58.6% | 56.9% | 60.0% | 58.3% |
| AIME24 | 13.3% | 10.0% | 13.3% | 15.6% |
Access to the same GenPRM does not consistently close the performance gap. StageMAD has higher reported accuracy on MATH, GSM8K, and AIME24, while MAD-M² (S) + GenPRM performs better on MMLU-Pro. Together with the matched-pool experiments, this provides evidence that how process judgments are used to organize communication matters, beyond simply making a PRM available.
To examine whether the findings extend beyond homogeneous teams, we compared StageMAD with vanilla MAD using three Qwen2.5-7B-Instruct agents and two Qwen3.5-9B agents. Both methods used the same team and initial candidates on each dataset. We conducted one run per dataset.
| Dataset | Heterogeneous vanilla MAD | Heterogeneous StageMAD |
|---|---|---|
| MATH | 69.4% | 74.4% |
| GSM8K | 94.3% | 95.6% |
| MMLU-Pro | 71.2% | 70.2% |
| AIME24 | 13.3% (4/30) | 20.0% (6/30) |
StageMAD improves over vanilla MAD on three of the four datasets with this heterogeneous team, including AIME24, while performing worse on MMLU-Pro. These results provide initial evidence beyond homogeneous teams. They address heterogeneous composition rather than larger model scale; whether the gains extend to substantially larger models remains open. We will distinguish these two questions explicitly in the revised manuscript.
Dear Reviewer 87JT:
Thank you for examining our experimental comparisons and the clarity of the manuscript. Your comments prompted us to recheck the CoT baseline more carefully, extend the evaluation, and clarify the implementation details. We address each point below and would appreciate any further questions.
A1: Table 1 already compares StageMAD with vanilla MAD, MAD-M² (S), MAD-M² (O), and the non-communicative CoT-SC@15 and PRM self-revision baselines. To directly answer your question about the advantage over single-pass inference, we additionally report CoT with one reasoning trajectory per question.
The comparison with single-pass CoT differs from the comparison with CoT self-consistency. On DeepSeek-Math/MATH, StageMAD achieves 55.3% accuracy compared with 41.1% for single-pass CoT, an improvement of 14.2 percentage points. The baseline that achieves higher accuracy at lower token cost in this setting is CoT-SC@15, which reaches 57.7%.
The complete single-pass comparison is shown below. All accuracies are averages over three runs, and the StageMAD results are from Table 1.
StageMAD achieves higher accuracy than single-pass CoT in all twelve reported settings. These differences measure the benefit of the complete inference pipeline, including multiple candidates, local correction, and communication. Figure 4 separately examines the contribution of the communication policy by holding the preceding candidate pools fixed.
We agree that StageMAD is not the best use of additional computation in some cases. On DeepSeek-Math/MATH, CoT-SC@15 exceeds StageMAD by 2.4 percentage points while using about one tenth as many tokens. On Qwen2.5-7B-Instruct/MATH, CoT-SC@15 also achieves higher accuracy, 73.8% versus 71.7%, at lower token cost. These are settings where independent sampling provides a better accuracy–cost trade-off.
Our contribution concerns improving communication within MAD, where StageMAD achieves the highest accuracy among the compared MAD methods in 11 of the 12 main settings. The non-communicative baselines establish when this form of interaction is useful relative to simpler inference strategies. We will add the single-pass results and distinguish these comparisons explicitly in the revised manuscript.