- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes GCL-MAD, a multi-agent reasoning framework that identifies conflicts between structured agent outputs and selectively verifies them before adjudication. Solver, Skeptic, and Edge-case agents generate typed components, which are connected through reviewed conflict relations. Evaluation covers six reasoning benchmarks and two backbone models, supplemented by ablations, answer-transition analyses, and controlled routing comparisons. The results suggest that selective verification can improve answer correction, although the evidence for a distinct graph contribution and broader generalization remains limited.
Critical strengths
The controlled routing comparison is the strongest part of the evaluation. Reusing initial agent outputs and matching verifier-call counts helps distinguish target selection from the benefit of additional inference. The answer-transition analysis also measures both successful corrections and harmful revisions, providing a more informative assessment of reliability than final accuracy alone. These experiments support selective conflict verification as a useful design choice.
Critical weaknesses
The evaluation is too narrow to substantiate the framework’s generality. Only two backbones are tested, leaving unclear whether conflict identification, confidence-based filtering, and conservative adjudication remain effective across models with different reasoning capabilities. Although six benchmarks are included, they concentrate on mathematical reasoning and knowledge or commonsense questions with short, objectively scored answers. They provide limited coverage of tasks requiring sustained reasoning, external evidence, or execution-based verification. Several datasets are also subsampled to 300 questions, and small gains lack paired uncertainty estimates. These limitations weaken the evidence that the observed benefits extend beyond the selected models and task formats.
The graph-specific contribution is insufficiently isolated. Nodes represent complete output fields, and the graph routes pairwise verification without propagation or path-based reasoning. The closest alternative, Flat Conflict Ranking, performs within 0.67 percentage points on MATH and 0.89 on GPQA. Without uncertainty estimates and separate controls for consolidation and structural filtering, the results do not establish a substantial advantage over structured conflict review. I therefore lean toward rejection despite the promising routing results.
Questions for authors
For clarification of generalization, what evidence indicates that the routing benefit is stable across both backbones, rather than specific to GPT-5-mini in the controlled comparison? Were model choices, benchmark subsets, and the confidence threshold fixed before examining evaluation outcomes?
For clarification of the central contribution, do the differences from Flat Conflict Ranking remain under paired uncertainty estimates, and which graph-specific operation accounts for them? Broader model and task coverage would be valuable future work, but the immediate assessment depends on the robustness and attribution of the gains already reported.
5 comments
Response part 2 [W2&Q1&Q2]
We thank the reviewer for the constructive comments. We address the concerns regarding graph-specific contributions, generalization, and experimental reliability below.
-
Graph-Specific Contributions (W2&Q2).
As shown in Table R3, removing conflict consolidation reduces accuracy by 2.00 percentage points, primarily due to fewer successful corrections. Removing edge rationales decreases accuracy by 1.11 points, with fewer successful corrections and more erroneous revisions. Together with Table 3, these results demonstrate that conflict consolidation improves verification-target selection, while explicit relational information enhances verification reliability, providing more direct evidence for the effectiveness of ECG's structured routing mechanism.
Table R3. Ablation of conflict consolidation and edge rationales on DeepSeek-V4-Flash (GPQA).
Method Acc. (%) W→C (%) C→W (%) Full ECG 61.60 3.57 0.23 w/o Conflict Consolidation (matched verifier calls) 59.60 1.56 0.22 w/o Edge Rationale (fixed verification targets) 60.49 2.68 0.45 -
Cross-Backbone Generalization (Q1&Q2). To examine whether ECG's benefits depend on the backbone model, we extend the controlled comparison with Flat Conflict Ranking to DeepSeek-V4-Flash.
As shown in Table R4, ECG achieves larger improvements over Flat Ranking on DeepSeek-V4-Flash. The consistent improvements across both backbones support the effectiveness of ECG-guided verification routing.
Table R4. Cross-backbone comparison of ECG Routing and Flat Conflict Ranking (%).
Backbone Method GPQA Acc. GPQA W→C GPQA C→W MATH Acc. MATH W→C MATH C→W GPT-5-mini Full ECG 75.22 5.80 0.00 88.67 1.33 0.67 Flat Conflict Ranking 74.33 5.13 0.22 88.00 1.33 1.33 DeepSeek-V4-Flash Full ECG 61.60 3.57 0.23 83.89 2.67 0.00 Flat Conflict Ranking 59.82 2.23 0.67 81.55 0.33 0.00 -
Paired Uncertainty Estimates (Q2).
To assess the statistical reliability of ECG's improvements over Flat Conflict Ranking, we compute paired bootstrap 95% confidence intervals using matched question-level predictions.
As shown in Table R5, ECG shows positive accuracy gains across both backbones and datasets. Three paired confidence intervals exclude zero, while the remaining interval reaches zero at the reported precision, providing additional statistical support for ECG-guided routing.
Table R5. Paired bootstrap confidence intervals for ECG versus Flat Conflict Ranking.
Backbone Dataset ECG − Flat (Acc.) 95% CI GPT-5-mini MATH +0.67 [0.00, 1.67] GPT-5-mini GPQA +0.89 [0.22, 1.79] DeepSeek-V4-Flash MATH +2.34 [0.67, 4.00] DeepSeek-V4-Flash GPQA +1.78 [0.67, 3.12] -
Experimental Settings (Q1&Q2).
- We selected intermediate-capability backbone models to investigate the benefits of structured multi-agent debate.
- Benchmark subsets were randomly sampled using fixed seeds shared across methods, without performance-based selection.
- The confidence threshold of τ = 3 was chosen based on preliminary sensitivity analysis to filter weakly supported conflicts that could cause unnecessary answer revisions, and was kept fixed in the main comparisons.
We welcome your further feedback and would be happy to address any remaining questions.
As the rebuttal period is drawing to a close, we would appreciate your reviewing our latest response. If these address your concerns, we would be grateful if you could consider raising your score.
Response part 1 [W1]
We thank the reviewer for the constructive feedback. We address the concerns regarding evaluation scope, dataset subsampling, and statistical reliability through additional experiments and clarifications, as detailed below.
Extended Evaluation Across Backbones and Challenging Benchmarks.
GCL-MAD is built on a novel idea: localizing epistemic conflicts in multi-agent debate and addressing them through targeted verification, with its effectiveness demonstrated on established reasoning benchmarks such as MATH. We further evaluate on two challenging benchmarks, HMMT (Feb. 2025, 2026) and AIME (2024 I/II, 2025, 2026), with a third backbone, Qwen-Flash. Across all six settings in Table R1, GCL-MAD outperforms all baselines and ablated variants by up to 12.70 percentage points, proving effective across different difficulties and models.
Dataset Subsampling and Paired Uncertainty Estimates.
Full-dataset evaluation is computationally expensive. For example, evaluating a single baseline on the complete MATH test set requires nearly 10 days in our setup. Given the multiple baselines, backbones, ablations, and random seeds, we therefore randomly sample 300 questions per run on certain datasets, using fixed seeds and identical question subsets across compared methods. Subset-based evaluation is also adopted in related work:
[1] Wu et al., ACL 2026 Main, Beyond Examples: Towards Automated Thought-level In-Context Reasoning for Large Language Models.
[2] Guo et al., ICLR 2025, Temporal Consistency for LLM Reasoning Process Error Identification
[3] Choi et al., NeurIPS 2025, Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?
To quantify the reliability of the observed improvements, we additionally compute paired bootstrap 95% confidence intervals using matched question-level predictions. As shown in Table R2, all confidence intervals exclude zero, providing statistical evidence that GCL-MAD outperforms the strongest competing MAD baseline on the evaluated subsets.