- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper investigates reliable dissent preservation in LLM-based multi-agent reasoning, aiming to retain useful minority evidence while filtering unsupported disagreements under limited communication budgets. It proposes DASH-Agent, a claim-level coordination framework that combines graph-based dissent discovery, reliability-aware prioritization, and selective local verification.
Critical strengths
1.The paper addresses an important problem in multi-agent reasoning: how to preserve potentially corrective minority evidence without propagating unreliable disagreements. Although related failure phenomena have been studied, focusing on the preservation and verification of individual claims is a meaningful direction with the potential to support more reliable and interpretable multi-agent coordination. 2.The paper evaluates DASH-Agent across mathematical reasoning, multi-hop question answering, and tool-use benchmarks, reporting consistent improvements over the included baselines.
Critical weaknesses
1.The paper motivates DASH-Agent through dissent collapse, but does not sufficiently investigate how and why corrective minority evidence is lost in existing multi-agent systems. Although the proposed discovery, prioritization, and verification stages provide a reasonable decomposition, this structure largely follows the architecture of DASH-Agent rather than being derived from an independent analysis of failure mechanisms. 2.In DASH-Agent, the three main components largely build on established techniques, including graph-based disagreement modeling, reliability-aware scoring, budgeted selection, and selective verification. The paper does not sufficiently clarify which specific limitations of existing methods necessitate the proposed framework or why its particular design choices offer distinct advantages over simpler alternatives. A more precise comparison with prior approaches and stronger justification of the methodological contributions are needed. 3.The primary experiments construct all eight agents using the same Qwen2.5-7B-Instruct backbone. Although homogeneous configurations help isolate coordination effects, agents sharing the same foundation model may exhibit correlated reasoning errors, knowledge limitations, and disagreement patterns. Heterogeneous multi-agent teams could produce different distributions of minority claims and evidence reliability, potentially affecting both graph-based discovery and DST prioritization. Evaluating teams composed of different model families would provide stronger evidence that the proposed framework generalizes beyond homogeneous multi-agent settings. 4.The paper shows that DASH-Agent retains more useful minority claims, but does not sufficiently demonstrate whether these retained claims actually help correct wrong majority decisions. Preserving a correct claim does not necessarily mean that the final answer uses it or becomes correct. The authors should directly report how often initially incorrect majority answers are corrected after coordination and whether these corrections result from the preserved minority evidence.
Questions for authors
1.What specific limitations of existing approaches, particularly those addressing majority convergence, minority correction, and evidence-weighted debate, require the proposed claim-level framework? Can the authors provide more direct evidence that the specific design choices in DASH-Agent offer advantages over simpler claim-level selection and verification strategies under matched conditions? 2.What empirical evidence supports the proposed decomposition of dissent collapse into discovery, prioritization, and verification failures? Can the authors quantify where and why corrective minority evidence is lost in existing multi-agent methods, rather than analyzing these failures only through the stages of DASH-Agent? 3.Does DASH-Agent maintain its advantages when agents are constructed from different foundation models rather than sharing the same backbone? If heterogeneous-agent experiments are available, how do the results compare in terms of task performance and useful-dissent preservation? 4.Among instances where the initial majority answer is incorrect but corrective minority evidence is available, how often does DASH-Agent produce the correct final answer? Can the authors show whether these corrections actually result from the retained minority claims?
2 comments
Thank you again for your careful review and consideration of our work. We would like to briefly follow up and ask whether our response has adequately addressed your main concerns. If any points remain unclear or require further clarification, we would be more than happy to continue the discussion. We sincerely appreciate your time and any further feedback.
We thank the reviewer for recognizing the importance of reliable minority evidence and DASH-Agent's gains across benchmarks. The concerns about design necessity, failure mechanisms, and actual correction are especially valuable.
1. Why claim-level coordination, rather than simpler existing approaches?
Original Secs. 2–3 distinguish the emphases of related work. ReConcile and CONSENSAGENT focus on consensus, SELENE uses selective evidence-weighted debate, and Graph-of-Agents adapts interaction structure. DASH-Agent more explicitly separates which corrective claims are exposed, which deserve verification, and which have evidential support. These decisions are not interchangeable. Original Secs. 4.4–4.5 evaluated their roles through preservation metrics and ablations.
To clarify the value of prioritization beyond candidate exposure, revised Table 5 compares strategies using the same candidate pool, initial responses, verifier, verification budget, instances, and seeds:
Reliability-only Matched 65.4 ± 0.8 → Full DST 68.0 ± 0.4 (+2.6 points)
DST w/o Novelty 66.1 ± 1.2 → Full DST 68.0 ± 0.4 (+1.9 points)
Under matched exposure and capacity, these gains support DST's combined signals over simpler ranking alternatives. Appendix B also explains why claims missed at discovery cannot be recovered by downstream ranking and why claim-level decisions distinguish evidence within the same response.
2. What does the evidence reveal about corrective-claim loss?
The request for evidence beyond a framework-based failure taxonomy is important. Original Secs. 4.4 and 4.8 examined retention, suppression, and coordination failures. Revised Sec. 4.8 and Appendix G.4 (Table G.4) now compare the observable outcome Not Retained + Final Incorrect on the same 512 eligible wrong-majority instances:
Thus, unretained corrective evidence accompanying an incorrect final answer occurs across methods, and DASH-Agent has the lowest rate in this comparison. Appendix G.3 (Table G.3) further partitions DASH-Agent's Not Retained + Final Incorrect cases by their earliest observable loss: Not Discovered 28.0 | Discovered but Not Selected 23.8 | Selected but Not Accepted 19.6 instances on average. These categories are disjoint. Combined with Fig. 6 and the matched controls, these findings connect the observed outcomes to discovery, prioritization, and adjudication decisions.
3. Does the approach transfer to heterogeneous agents?
Original Appendix D.4 evaluated a Llama-3.1-8B-Instruct backbone and additional CommonsenseQA/StrategyQA tasks. Given the concern about correlated errors in homogeneous teams, revised Sec. 4.7 and Table 6 compare six Qwen Proposers with three Qwen + three Llama Proposers, keeping two Qwen Verifiers and other conditions fixed.
With mixed Proposers, MATH 47.6 → 48.0, HotpotQA 72.9 → 73.5, and subset UDS 81.7 → 82.4. API-Bank changes from 63.0 → 62.7. These task-dependent results support effectiveness in the tested mixed-family setting, alongside the original backbone transfer.
4. Does retained evidence actually contribute to correction?
The distinction between claim retention and final correctness is central to our evaluation. Original MCR/UDS/IDS metrics examined preservation and suppression. Revised Sec. 4.4, Table 4, Fig. 5, and Appendices G.3–G.4 examine final outcomes on the same 512 eligible cases. DASH-Agent reaches MCR 84.7 ± 0.9% and FCR 77.2 ± 1.3%. Joint outcomes show 75.9% Retained + Final Correct and 89.6 ± 1.5% Correct Given Retained, without equating retention with correctness.
More directly, Sec. 4.7 and Appendix G.2 (Table G.2) mask evidence immediately before verification, holding initial responses, priorities, verification capacity, and instances fixed:
FCR 77.2% (full) → 57.9% (corrective evidence masked), a paired drop of 19.3 ± 2.2 points.
With matched evidence length and intervention budget, non-corrective masking yields 74.9 ± 1.5% FCR. This contrast supports the contribution of corrective content to correct final answers, beyond an association between evidence retention and success.
We sincerely appreciate the reviewer's careful assessment and hope these targeted results provide a stronger basis for reassessing DASH-Agent's methodological contribution and practical value.