acceptodds
Under review as a conference paper at ICLR 2027

DASH-Agent: Reliable Dissent Preservation for LLM Multi-Agent Reasoning

Abstract

Large language model (LLM)-based multi-agent systems benefit from diverse reasoning and evidence, yet coordination can erase the information that makes this diversity useful. When several agents share an error, consensus may suppress a minority claim carrying the evidence needed to correct the final answer. Preserving every disagreement instead propagates noise and consumes verification budget. Rather than treating consensus or sparse communication as the endpoint, we study reliable dissent preservation: retaining evidence-bearing minority claims while filtering unsupported deviations under bounded communication. DASH-Agent implements this objective with a claim-level discovery–validation–verification pipeline. A task-conditioned interaction graph exposes candidate dissent, Dissent Signal Triage (DST) ranks candidates by novelty, task utility, reliability, and cost, and sparse local groups verify selected claims before synthesis. Extensive experiments across reasoning, multi-hop question answering, and tool-use benchmarks show improved task quality with lower communication, while claim-level analyses show stronger useful-dissent retention and incorrect-dissent suppression.

Primary area:
causal reasoning
Keywords:
Multi-Agent Reasoning, Large Language Model, Claim-level Verification, Dissent Preservation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4ZwYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates reliable dissent preservation in LLM-based multi-agent reasoning, aiming to retain useful minority evidence while filtering unsupported disagreements under limited communication budgets. It proposes DASH-Agent, a claim-level coordination framework that combines graph-based dissent discovery, reliability-aware prioritization, and selective local verification.

Critical strengths

1.The paper addresses an important problem in multi-agent reasoning: how to preserve potentially corrective minority evidence without propagating unreliable disagreements. Although related failure phenomena have been studied, focusing on the preservation and verification of individual claims is a meaningful direction with the potential to support more reliable and interpretable multi-agent coordination. 2.The paper evaluates DASH-Agent across mathematical reasoning, multi-hop question answering, and tool-use benchmarks, reporting consistent improvements over the included baselines.

Critical weaknesses

1.The paper motivates DASH-Agent through dissent collapse, but does not sufficiently investigate how and why corrective minority evidence is lost in existing multi-agent systems. Although the proposed discovery, prioritization, and verification stages provide a reasonable decomposition, this structure largely follows the architecture of DASH-Agent rather than being derived from an independent analysis of failure mechanisms. 2.In DASH-Agent, the three main components largely build on established techniques, including graph-based disagreement modeling, reliability-aware scoring, budgeted selection, and selective verification. The paper does not sufficiently clarify which specific limitations of existing methods necessitate the proposed framework or why its particular design choices offer distinct advantages over simpler alternatives. A more precise comparison with prior approaches and stronger justification of the methodological contributions are needed. 3.The primary experiments construct all eight agents using the same Qwen2.5-7B-Instruct backbone. Although homogeneous configurations help isolate coordination effects, agents sharing the same foundation model may exhibit correlated reasoning errors, knowledge limitations, and disagreement patterns. Heterogeneous multi-agent teams could produce different distributions of minority claims and evidence reliability, potentially affecting both graph-based discovery and DST prioritization. Evaluating teams composed of different model families would provide stronger evidence that the proposed framework generalizes beyond homogeneous multi-agent settings. 4.The paper shows that DASH-Agent retains more useful minority claims, but does not sufficiently demonstrate whether these retained claims actually help correct wrong majority decisions. Preserving a correct claim does not necessarily mean that the final answer uses it or becomes correct. The authors should directly report how often initially incorrect majority answers are corrected after coordination and whether these corrections result from the preserved minority evidence.

Questions for authors

1.What specific limitations of existing approaches, particularly those addressing majority convergence, minority correction, and evidence-weighted debate, require the proposed claim-level framework? Can the authors provide more direct evidence that the specific design choices in DASH-Agent offer advantages over simpler claim-level selection and verification strategies under matched conditions? 2.What empirical evidence supports the proposed decomposition of dissent collapse into discovery, prioritization, and verification failures? Can the authors quantify where and why corrective minority evidence is lost in existing multi-agent methods, rather than analyzing these failures only through the stages of DASH-Agent? 3.Does DASH-Agent maintain its advantages when agents are constructed from different foundation models rather than sharing the same backbone? If heterogeneous-agent experiments are available, how do the results compare in terms of task performance and useful-dissent preservation? 4.Among instances where the initial majority answer is incorrect but corrective minority evidence is available, how often does DASH-Agent produce the correct final answer? Can the authors show whether these corrections actually result from the retained minority claims?

2 comments

We thank the reviewer for recognizing the importance of reliable minority evidence and DASH-Agent's gains across benchmarks. The concerns about design necessity, failure mechanisms, and actual correction are especially valuable.

1. Why claim-level coordination, rather than simpler existing approaches?

Original Secs. 2–3 distinguish the emphases of related work. ReConcile and CONSENSAGENT focus on consensus, SELENE uses selective evidence-weighted debate, and Graph-of-Agents adapts interaction structure. DASH-Agent more explicitly separates which corrective claims are exposed, which deserve verification, and which have evidential support. These decisions are not interchangeable. Original Secs. 4.4–4.5 evaluated their roles through preservation metrics and ablations.

To clarify the value of prioritization beyond candidate exposure, revised Table 5 compares strategies using the same candidate pool, initial responses, verifier, verification budget, instances, and seeds:

Reliability-only Matched 65.4 ± 0.8 → Full DST 68.0 ± 0.4 (+2.6 points)

DST w/o Novelty 66.1 ± 1.2 → Full DST 68.0 ± 0.4 (+1.9 points)

Under matched exposure and capacity, these gains support DST's combined signals over simpler ranking alternatives. Appendix B also explains why claims missed at discovery cannot be recovered by downstream ranking and why claim-level decisions distinguish evidence within the same response.

2. What does the evidence reveal about corrective-claim loss?

The request for evidence beyond a framework-based failure taxonomy is important. Original Secs. 4.4 and 4.8 examined retention, suppression, and coordination failures. Revised Sec. 4.8 and Appendix G.4 (Table G.4) now compare the observable outcome Not Retained + Final Incorrect on the same 512 eligible wrong-majority instances:

MethodNot retained + final incorrect
Debate28.5%
Graph-of-Agents21.0%
ReConcile19.0%
SELENE16.0%
DASH-Agent13.9%

Thus, unretained corrective evidence accompanying an incorrect final answer occurs across methods, and DASH-Agent has the lowest rate in this comparison. Appendix G.3 (Table G.3) further partitions DASH-Agent's Not Retained + Final Incorrect cases by their earliest observable loss: Not Discovered 28.0 | Discovered but Not Selected 23.8 | Selected but Not Accepted 19.6 instances on average. These categories are disjoint. Combined with Fig. 6 and the matched controls, these findings connect the observed outcomes to discovery, prioritization, and adjudication decisions.

3. Does the approach transfer to heterogeneous agents?

Original Appendix D.4 evaluated a Llama-3.1-8B-Instruct backbone and additional CommonsenseQA/StrategyQA tasks. Given the concern about correlated errors in homogeneous teams, revised Sec. 4.7 and Table 6 compare six Qwen Proposers with three Qwen + three Llama Proposers, keeping two Qwen Verifiers and other conditions fixed.

With mixed Proposers, MATH 47.6 → 48.0, HotpotQA 72.9 → 73.5, and subset UDS 81.7 → 82.4. API-Bank changes from 63.0 → 62.7. These task-dependent results support effectiveness in the tested mixed-family setting, alongside the original backbone transfer.

4. Does retained evidence actually contribute to correction?

The distinction between claim retention and final correctness is central to our evaluation. Original MCR/UDS/IDS metrics examined preservation and suppression. Revised Sec. 4.4, Table 4, Fig. 5, and Appendices G.3–G.4 examine final outcomes on the same 512 eligible cases. DASH-Agent reaches MCR 84.7 ± 0.9% and FCR 77.2 ± 1.3%. Joint outcomes show 75.9% Retained + Final Correct and 89.6 ± 1.5% Correct Given Retained, without equating retention with correctness.

More directly, Sec. 4.7 and Appendix G.2 (Table G.2) mask evidence immediately before verification, holding initial responses, priorities, verification capacity, and instances fixed:

FCR 77.2% (full) → 57.9% (corrective evidence masked), a paired drop of 19.3 ± 2.2 points.

With matched evidence length and intervention budget, non-corrective masking yields 74.9 ± 1.5% FCR. This contrast supports the contribution of corrective content to correct final answers, beyond an association between evidence retention and success.

We sincerely appreciate the reviewer's careful assessment and hope these targeted results provide a stronger basis for reassessing DASH-Agent's methodological contribution and practical value.

Thank you again for your careful review and consideration of our work. We would like to briefly follow up and ask whether our response has adequately addressed your main concerns. If any points remain unclear or require further clarification, we would be more than happy to continue the discussion. We sincerely appreciate your time and any further feedback.

Reviewer_iiUjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to solve the failure in multi-agent setups where the group votes for a wrong answer, and the one agent that actually has the right evidence gets outvoted. It works in three handcrafted agentic steps — searching, ranking, and checking. This keeps genuinely useful minority evidence while dropping unsupported noise, all within a fixed communication budget

Critical strengths

The motivation is clear

Critical weaknesses

1 This paper is more like an auto-research pipeline output than a carefully developed human work.

2 The verifier is never evaluated, yet all headline mechanism metrics depend on it. The verifiers are instantiated from the same 7B backbone, but it still does not measure verifier accuracy against human labels.

3 The problem definition shifts between the abstract and method. The abstract and intro claim the problem as "minority claim vs. wrong majority answer," but the method claims any claim with sufficient claim-level conflict, including claims that agree with the majority answer but rest on divergent evidence. The scope is quietly broadened from "minority answer preservation" to "general evidence inconsistency."

Questions for authors

Please refer above

2 comments

We sincerely thank the reviewer for recognizing DASH-Agent's motivation and raising these important concerns. The feedback is highly valued and each point has been carefully considered.

1. On the “auto-research pipeline” impression

We were genuinely disheartened by this characterization. DASH-Agent grew from sustained efforts to understand a concrete failure in multi-agent reasoning: a wrong majority may suppress the minority evidence needed to correct it. Considerable work went into studying this failure, designing the mechanisms, and testing whether evidence preservation improves final answers. We hope this research effort can be more fully appreciated. Generative AI use is documented in the AI Usage Disclosure.

The framework addresses three distinct questions. Discovery determines which claims become visible, DST which deserve limited verification capacity, and Verification which are supported by evidence. Original Sec. 3 explains this rationale, while Secs. 4.4–4.6 evaluate preservation, suppression, ablations, and communication costs. Ranking cannot recover claims missed during discovery, and high priority alone cannot establish correctness.

Nevertheless, the three-stage structure might initially resemble a familiar pipeline. To avoid further misunderstanding, revised Table 5 compares strategies with the same candidates, initial responses, verifier, budget, instances, and seeds:

Matched strategyThree-task mean ± SDGap to Full
Reliability-only Matched65.4 ± 0.8+2.6
DST w/o Novelty66.1 ± 1.2+1.9
Full DST68.0 ± 0.4—

65.4 → 68.0 under matched exposure and verification capacity shows that the gain is not simply due to more resources. Appendix B further analyzes graph variation, missed exposure, and the expressiveness of claim-level decisions. These results reinforce the design's empirical and theoretical basis.

2. On evaluating the Verifier

The reviewer's attention to this component is appreciated, as adjudication is central to reliable dissent preservation. Original Sec. 3.4 specified evidence-based decisions of supported, contradicted, and unresolved. The original mechanism experiments also examined verification's downstream effects on retention and suppression.

To make these judgments directly assessable, revised Sec. 4.3, Table 3, and Fig. 3 add a dedicated evaluation, with protocols in Appendices C.2 and C.4. It covers 3,828 claims from 1,200 audited instances, using independent human annotations with third-annotator adjudication:

  • Accuracy 91.7% | Macro-F1 91.2%, with three-class outcomes and error flows in Fig. 3.
  • Top-label ECE 0.061 ± 0.008 across five runs, evaluating post-verification confidence separately.
  • Human agreement 88.6% (κ = 0.81), measuring annotation consistency rather than Verifier accuracy.

Each claim contributes one modal prediction across five runs. These results directly characterize Verifier quality and calibration, complementing the original task- and mechanism-level analyses.

3. On the apparent shift in problem definition

The motivation emphasizes minority correction, while the method examines broader claim-level disagreement. The key distinction is between the outcome sought and the evidence that must be inspected.

This distinction was already present in the original formulation. Original Sec. 3.1 states: “Claim-level dissent is broader because agents can agree on the final answer while relying on conflicting evidence.” Original Sec. 4.4 also explains how answer-dissent-only routing may miss evidence conflicts. An agent's final answer need not represent every claim within its response.

These levels serve distinct purposes:

  • Objective → Preserve supported minority evidence capable of correcting an initially wrong majority.
  • Discovery → Examine individual claims, including conflicts within otherwise agreeing answers, instead of excluding evidence based only on speaker-level agreement.
  • Evaluation → Restrict MCR and FCR to wrong-majority instances with independently supported, materially corrective evidence from an answer-level minority Proposer.

Thus, broader discovery ≠ broader research objective. Discovery prevents early loss of potentially corrective evidence, while minority correction remains the central goal. Revised Sec. 3.1 and Appendix C.2 clarify this relationship and the MCR/FCR eligibility conditions without changing the original objective.

We sincerely appreciate the reviewer's careful consideration and hope the matched comparisons, direct Verifier evaluation, and clarifications provide a stronger basis for reassessing DASH-Agent's contribution.

Thank you again for your careful review and consideration of our work. We would like to briefly follow up and ask whether our response has adequately addressed your main concerns. If any points remain unclear or require further clarification, we would be more than happy to continue the discussion. We sincerely appreciate your time and any further feedback.

Reviewer_eGWtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DASH-Agent separates claim-level dissent discovery, reliability-aware prioritization, and selective verification before synthesis. Five reasoning, question-answering, and tool-use benchmarks evaluate task quality, dissent preservation, and communication with a shared Qwen2.5-7B backbone.

Critical strengths
  • The distinction between disagreement, verification priority, and evidential support is useful. The 1,200-instance blinded claim audit and explicit retention/suppression metrics provide evidence beyond final-answer accuracy.

  • DASH-Agent improves on SELENE by 1.3–3.3 points across the five primary tasks. Five-seed results, component removals, and communication accounting support practical benefits within the tested configuration.

Critical weaknesses
  • The specific value of the discovery and ranking machinery remains incompletely isolated. Reliability-only routing changes both the candidate pool and ranking rule; random routing shares the graph-exposed pool but is a weak prioritization control. These comparisons do not establish the advantage over simpler evidence-aware ranking on identical claims.

  • Preservation measures availability to final synthesis, not correction of the final answer. The audit samples dissent-containing instances, so its 512 eligible wrong-majority cases do not establish failure prevalence in the full test population. Transfer is limited to another small backbone and additional tasks, leaving broader practical significance uncertain.

Overall assessment: Weak rejection. The results support the pipeline, but stronger matched controls and a clearer link from dissent preservation to answer correction are needed to establish its distinct contribution.

Questions for authors
  • Evidence: How do reliability-only or simpler evidence-aware scores perform on the same candidate pool, with the same verifier and verification budget?

  • Clarification and evidence: What fraction of all test instances exhibits an initially wrong majority with corrective minority evidence, and how often does retaining that evidence correct the final answer?

2 comments

We sincerely thank the reviewer for the thoughtful assessment of DASH-Agent and for recognizing the value of our claim-level audit, preservation metrics, and gains across five tasks. The questions about fair comparisons and whether preserved evidence actually corrects answers are particularly important to the paper.

1. Separating discovery from prioritization

The concern about candidate exposure and ranking quality is well taken. In the original submission, Random Routing used graph-exposed candidates, whereas Reliability-only Routing ranked explicit answer-dissent claims without graph-conditioned discovery. The component ablations also examined discovery, prioritization, and verification individually. These controls served different purposes, making a comparison on identical candidates a useful further test.

Accordingly, Table 5 in the revision compares strategies with the same candidate pool, initial responses, verifier, verification budget, instances, and seeds.

StrategyThree-task mean ± SD
Random Routing62.0 ± 1.0
Reliability-only Matched65.4 ± 0.8
DST w/o Novelty66.1 ± 1.2
Full DST68.0 ± 0.4

Reliability-only Matched 65.4 → Full DST 68.0 (+2.6 points) and DST w/o Novelty 66.1 → Full DST 68.0 (+1.9 points). Because candidate exposure and verification capacity are held fixed, these gains support the value of DST's combined prioritization signals beyond simpler evidence-aware ranking. Appendix C.4 details the matched rules, while Appendix B explains why missed discovery and downstream ranking concern distinct decisions.

2. How frequent is corrective minority evidence, and does it lead to correct answers?

Evidence reaching final synthesis is not the same as the final answer being correct. The original MCR, UDS, and IDS measures distinguished useful-claim preservation from incorrect-claim suppression. The reviewer's questions also call for measuring the prevalence of this failure in unfiltered data and examining the link to final correction.

Prevalence → Appendix G.1 (Table G.1) now reports five independent runs, each sampling 240 instances per benchmark without conditioning on dissent. Across 6,000 instance-run observations from the full labeled evaluation splits, an initially wrong majority with materially corrective minority evidence occurs in 20.0 ± 1.1%. This is distinct from the dissent-enriched audit's 512 eligible instances used for mechanism comparisons.

Retention → correction → Sec. 4.4, Table 4, Fig. 5, and Appendices G.3–G.4 assess both outcomes on those same 512 cases. DASH-Agent achieves MCR 84.7 ± 0.9% and FCR 77.2 ± 1.3%, compared with SELENE's 76.1 ± 1.4% and 66.5 ± 1.9%. The joint-outcome analysis shows 75.9% of eligible cases are Retained + Final Correct. Among cases retaining corrective evidence, 89.6 ± 1.5% yield a correct final answer. Four mutually exclusive outcomes ensure that preservation is not treated as equivalent to correction.

Evidence intervention → Sec. 4.7 and Appendix G.2 (Table G.2) further test the contribution of corrective content by masking evidence before verification, with initial responses, candidate priorities, verification capacity, and instances fixed.

FCR: 77.2% (full) → 57.9% (corrective evidence masked), a paired decrease of 19.3 ± 2.2 points.

With evidence length and intervention budget matched, masking non-corrective evidence instead yields 74.9 ± 1.5% FCR. The contrast offers more direct evidence that corrective content contributes to final-answer accuracy, beyond being available to synthesis.

3. Generalization beyond homogeneous agents

We also appreciate the question about broader applicability. Appendix D.4 already tested Llama-3.1-8B-Instruct and additional CommonsenseQA and StrategyQA tasks. Sec. 4.7 and Table 6 in the revision extend this to a heterogeneous team, replacing six Qwen Proposers with three Qwen and three Llama Proposers, while keeping two Qwen Verifiers and the evaluation setup fixed.

With mixed Proposers, DASH-Agent changes from 47.6 → 48.0 on MATH, 72.9 → 73.5 on HotpotQA, and 81.7 → 82.4 in subset UDS. API-Bank changes slightly from 63.0 → 62.7, reflecting task-dependent variation. Together with the backbone-transfer results, these findings support the method's effectiveness in the tested heterogeneous setting.

We greatly appreciate the reviewer's constructive suggestions and careful consideration. We hope these matched comparisons and direct analyses of final correction provide a stronger basis for reassessing DASH-Agent's contribution.

Thank you again for your careful review and consideration of our work. We would like to briefly follow up and ask whether our response has adequately addressed your main concerns. If any points remain unclear or require further clarification, we would be more than happy to continue the discussion. We sincerely appreciate your time and any further feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.