acceptodds
Under review as a conference paper at ICLR 2027

When Consensus Masks Risk: Energy-Guided Control of Diagnostic Discussion

Abstract

Multi-agent deliberation can improve high-stakes classification, yet consensus alone does not establish safety. In diagnostic-style decisions, agents with asymmetric expertise may reach agreement while high-cost alternatives remain unresolved, a failure mode we call premature diagnostic closure. We cast this process as a protocol-control problem without online ground truth: using agents' reports, optional justifications, calibration confusion matrices, and an asymmetric loss, a mediator must decide whether to continue, challenge, certify, or escalate. We propose Diagnostic Consensus Energy Minimization (D-CEM), a loss-aware controller that interprets each agent's confusion matrix as a map of plausible differential diagnoses. Given current reports, D-CEM forms confusion-induced posteriors, identifies the most dangerous plausible miss under the loss, and computes a diagnostic consensus energy combining posterior disagreement, expected harm, and loss-aware margin. The policy continues while risk decreases, issues targeted differential challenges, certifies only low-energy large-margin decisions, and escalates when high-risk deliberation stagnates. We derive loss-sensitive risk bounds, calibrated decision guarantees, and cost-aware stopping bounds. Experiments on synthetic and clinical tasks show that D-CEM reduces high-risk misses and harmful consensus while improving safety–cost trade-offs over aggregation baselines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multi-Agent Deliberation, Loss-Aware Energy Control, Selective Prediction and Deferral

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_iJEZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Diagnostic Consensus Energy Minimization (D-CEM), a framework for controlling discussions between two agents in high-stakes classification tasks. It uses historical confusion matrices and asymmetric error costs to identify dangerous alternative diagnoses and decide when agents should continue discussing, reconsider their predictions, make a decision, or escalate. Experiments on synthetic and clinical datasets show that D-CEM can reduce harmful consensus and high-risk errors while improving the trade-off between decision quality and deliberation cost.

Critical strengths

Risk-aware approach to multi-agent deliberation: The paper addresses an important problem where agents can agree on an incorrect decision despite appearing confident. Using historical confusion patterns and asymmetric losses to guide targeted challenges, stopping, and escalation is a meaningful improvement over relying on consensus alone.

Strong experimental and theoretical support: The paper combines theoretical risk analysis with experiments on synthetic and clinical datasets, including comparisons against aggregation and selective-prediction baselines. The additional studies of steering compliance, confusion-matrix quality, and escalation help explain which components contribute to the observed improvements.

Critical weaknesses

Reliance on historical confusion matrices: D-CEM uses fixed confusion matrices to estimate risk throughout deliberation, even though agent behavior may change after targeted challenges or repeated interaction. While the paper includes robustness experiments, it remains unclear how reliably these estimates reflect actual risk when model behavior or error patterns shift substantially.

Sensitivity to escalation costs and limited clinical generalization: Some safety improvements depend on escalating uncertain cases, and the assumed cost of escalation affects whether this is beneficial. For example, Appendix I.3 shows that on DDXPlus, the full method has higher total cost than its no-escalation variant at the paper's chosen unit escalation cost, while the clinical experiments themselves are limited to controlled tasks and two LLM backbones

Questions for authors

How sensitive is D-CEM to changes in agents' confusion patterns during multi-round discussions, particularly when targeted challenges change their predictions? Could the authors evaluate whether updating or recalibrating these estimates improves the reliability of the controller?

How should escalation costs be selected in realistic applications, and under what conditions does escalation improve overall decision quality enough to justify its cost? Additional analysis would help clarify when the full protocol is preferable to simpler stopping or rejection strategies.

3 comments
AuthorsOpenReview ↗

Response to Reviewer iJEZ (1/2)

Thank you very much for your careful review and for recognizing the importance of the problem, our risk-aware deliberation approach, and our theoretical and experimental support. We directly address each of your concerns and questions below.

W1 & Q1. How sensitive is D-CEM to shifting confusion patterns, and does recalibration help?

Fixed confusion matrices (CMs) remain reliable across discussion rounds, even after targeted challenges change the agents' predictions, and recalibrating them brings only a small efficiency gain.

We hold the CMs fixed because the mediator cannot access the true label of the current case, so a pre-deployment offline estimate from historical data (CAL, 70–80 labeled cases) is a practical approach. Our work focuses on short-term deliberation; for longer-term use, periodic recalibration between deployment windows can be applied. Within this scope, we add two new experiments.

Experiment 1: do fixed CMs still match agent behavior after steering? Using DDXPlus ground-truth labels, we check fixed-CM posteriors and predicted risks against observed outcomes, both across rounds and for the same agent on the same case just before vs. after DIFFERENTIAL_STEER (nn is the number of agent-round reports; all metrics are group means).

GroupnnPosterior TV ↓Brier ↓Pred. report risk RL(y^a;Pa)R_L(\hat{y}_a;P_a)Obs. report loss L(y^a,y)L(\hat{y}_a,y)
Matched pre-steer (round tt)2,4600.1220.8022.9302.728
Matched post-steer (round t+1t+1)2,4600.1200.7882.8782.620
Round 1 (all initial cases)3,8400.1630.4471.6031.200
Rounds 2–3 (continuing cases)3,7920.1610.6502.1772.043
Rounds 4–6 (unstopped hard tail)4,4100.2030.7472.1652.337
Certified stops (rounds 2–6)2740.2450.4281.1340.880

Takeaway: steering does not widen the gap between fixed-CM and empirical posteriors (TV 0.122 → 0.120). Moreover, predicted risk exceeds observed loss in rounds 1–3 and at certified stops, i.e., the estimates are conservative where decisions are made; slight underestimation appears only in unresolved hard cases (rounds 4–6), which escalation covers. Targeted challenges therefore do not break the fixed-CM risk estimates.

Experiment 2: does recalibrating the CMs make the controller more reliable? We then run the complete DDXPlus controller with three CM choices, all fitted on CAL: fixed CaC_a, separate pre-/post-steer CMs, and round-specific CMs.

CM policyR^Lpool\widehat{R}_L^{\mathrm{pool}} ↓R^Lsys\widehat{R}_L^{\mathrm{sys}} ↓J^esc(1)\widehat{J}_{\mathrm{esc}}(1) ↓H.C. (%) ↓Esc. (%)Avg. rounds
Fixed CaC_a0.799 ± 0.0450.132 ± 0.0310.574 ± 0.0141.20 ± 0.2444.2 ± 4.53.14
Pre-/post-steer Capre/CapostC_a^{\mathrm{pre}}/C_a^{\mathrm{post}}0.809 ± 0.0740.136 ± 0.0310.554 ± 0.0161.09 ± 0.2741.8 ± 1.53.03
Round-specific Ca(t)C_a^{(t)}0.660 ± 0.0430.123 ± 0.0360.541 ± 0.0031.15 ± 0.0941.8 ± 3.33.13

Takeaway: safety is similar across all three (H.C. 1.20% vs. 1.09%/1.15%), while recalibrated CMs only modestly lower escalation (44.2% → 41.8%) and total cost. However, they need multi-round deliberation on CAL and split its 70–80 cases across rounds or intervention states, which is a substantial offline overhead. Fixed CaC_a is therefore a lightweight default with similar observed safety.

Moreover, safety is preserved even under much larger CM errors: when each CM is mixed with a uniform matrix, Cη=(1−η)C+ηUC_\eta=(1-\eta)C+\eta U, H.C. stays at 0.4% even at η=0.5\eta=0.5, and only the total cost rises (main DDXPlus setting: 0.341 → 0.445; Appendix I.5, Table 20).

In our revised manuscript, Section 2.2 clarifies how the CMs are estimated and states the short-term scope, and Appendix I.2 reports both analyses (Tables 13 and 14).

AuthorsOpenReview ↗

Response to Reviewer iJEZ (2/2)

W2-1 & Q2. How should the escalation cost be chosen, and when is escalation worth it?

The escalation cost is deployment-specific: each hospital can set it according to its own workflow and safety priorities, and whether escalation pays off depends on this choice. Specifically:

  1. How to choose the cost.

We use cesc=1c_{\mathrm{esc}}=1 only as a reference point: under the DDXPlus loss (0 correct, 1 ordinary error, 5 high-risk miss), it counts one escalation as one ordinary error. In deployment, hospitals can set their own value from the cost of a specialist review relative to the harm of a misdiagnosis [1]. If this is hard to state, they can instead pick the operating point on the CAL risk–coverage curve that meets their referral budget or safety ceiling [2].

  1. When escalation is worth it.

Given this cost, escalation lowers risk but pays for each referral. As a result, full D-CEM has lower total cost than D-CEM without escalation whenever cescc_{\mathrm{esc}} is below the break-even value c∗=(R0−Rf)/qc^* = (R_0 - R_f)/q, where RfR_f and R0R_0 are the system risks with and without escalation and qq is the escalation rate (Appendix I.3, Figure 6). For instance:

  • MedQA-Tri (c∗≈1.05c^* \approx 1.05): escalation already pays off at cesc=1c_{\mathrm{esc}}=1 (0.640 vs. 0.648).
  • DDXPlus (c∗≈0.79c^* \approx 0.79), the example you mention: at cesc=1c_{\mathrm{esc}}=1, full D-CEM costs slightly more (0.341 vs. 0.286) but cuts autonomous system risk by about 72%; at cesc=0.5c_{\mathrm{esc}}=0.5, it is also cheaper (0.211 vs. 0.286).

In short, escalation is worth it when specialist review is cheap relative to misdiagnosis harm (as on MedQA-Tri, or on DDXPlus at cesc=0.5c_{\mathrm{esc}}=0.5), or when a hospital prioritizes safety over a small cost increase (as on DDXPlus at cesc=1c_{\mathrm{esc}}=1). Moreover, whenever a hospital does escalate, D-CEM is cheaper than a simpler rejection strategy: at the same 26% escalation rate on DDXPlus, it costs 0.048 less than Static-CM-ER for every cesc≥0c_{\mathrm{esc}}\ge 0 (Appendix I.3).

In our revised manuscript, Appendix E explains how to choose cescc_{\mathrm{esc}} in practice, complementing the break-even analysis in Appendix I.3 (Figure 6).

[1] Mozannar & Sontag, "Consistent Estimators for Learning to Defer to an Expert," ICML 2020.

[2] Geifman & El-Yaniv, "SelectiveNet: A Deep Neural Network with an Integrated Reject Option," ICML 2019.

W2-2. Generalization beyond controlled tasks and two LLM backbones

The controlled scenarios (S1–S3b) are a test design, not a requirement of D-CEM: we use them to isolate premature diagnostic closure under asymmetric expertise, complementary errors, and shared bias. D-CEM itself treats agents as black boxes and supports O(NK)O(NK) multi-agent pooling (Appendix B.7), so new backbones or larger panels keep the same control logic and only require recalibrating CaC_a and the thresholds on CAL (Appendix K).

Empirically, two sets of results in Appendix I.4 support this. First, Table 16 (original submission) removes all role anchors on DDXPlus under neutral prompts and expands the label space to K=8,16,24K=8, 16, 24; D-CEM achieves the lowest J^esc(1)\widehat{J}_{\mathrm{esc}}(1) and the lowest or tied-lowest H.C. at every KK. Second, as a new experiment, we add a third backbone (Llama-3.2-3B paired with gpt-4o-mini) in the same setting with enlarged test splits (Table 17); D-CEM again achieves the lowest H.C. and J^esc(1)\widehat{J}_{\mathrm{esc}}(1) at every KK, both at 0% escalation and at matched escalation.

In our revised manuscript, Appendix I.4 reports the neutral-prompt results (Table 16) and the new third-backbone results (Table 17).

AuthorsOpenReview ↗

Looking forward to further discussion

Dear Reviewer iJEZ,

We sincerely thank you again for your thoughtful review. As the discussion deadline approaches, we wanted to gently follow up. If time permits, we would be very grateful to know whether our responses have addressed your concerns or whether you have any remaining questions. Thank you again for your time and thoughtful review.

Best regards,

Authors

Reviewer_JPYkOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to address premature consensus in agentic deliberation. Specifically, this paper proposes Diagnostic Consensus Energy Minimization (D-CEM), an ordered policy for intervening on LLM agents during diagnostic deliberation. The policy is based on (1) diagnostic-risk energy, which is a weighted sum of posterior disagreement based on KL divergence, expected cost-aware risk, and the margin between the lowest and second lowest risk decisions; (2) the risk margin itself; and (3) the expected cost-aware risk itself. Theoretical results provide justification for individual components of the policy. Compared to baseline policies, D-CEM performs better on synthetic and LLM systems with 2 agents, in terms of accuracy, cost-aware population risk, miss rate, and the rate of harmful consensus.

Critical strengths
  • The motivation of the paper is clear. The problem and experiment setups are clearly scoped. A range of evidences were used to support the claim, including theoretical guarantees, synthetic experiments, and experiments with LLMs.
  • The proposed approach is simple to implement and can be adopted to any multi-agent deliberation.
Critical weaknesses
  • The theoretical guarantees seem to hold individually for each component of the mediator policy. That is, Proposition 1 supports DIFFERENTIAL_STEER, Theorem 1 supports STOP_AND_DECIDE, Theorem 2 supports CONTINUE, and Theorem 3 supports STOP_AND_ESCALATE. However, it is unclear whether these theoretical guarantees still hold when these components are put together in the final mediator policy, as implemented in the paper.
  • It seems unfair to compare the baseline policies against D-CEM in terms of the rate of harmful consensus when the baseline policies don't have escalation in their action space.
Questions for authors
  • Do the theoretical guarantees hold for the ordered policy of D-CEM in its entirety?
  • How do the baseline policies compare to D-CEM in terms of the rate of harmful consensus, after excluding the escalated cases by D-CEM?
  • What is the motivation for including all three terms in the diagnostic-risk energy? For example, ΔL(t)\Delta_L^{(t)} is also used for STOP_AND_DECIDE, so why is ΔL(t)\Delta_L^{(t)} needed in the diagnostic-risk energy?
5 comments
AuthorsOpenReview ↗

Response to Reviewer JPYk (1/2)

We sincerely thank you for your time, constructive feedback, and recognition of our clear motivation, simple control design, and theoretical and empirical results. We address each point below.

W1 & Q1. Do the guarantees hold for the complete ordered policy?

Yes, the guarantees hold for the complete policy, because none of them depends on the order in which the actions are taken. Every run ends at a single stopping round τ\tau, where the mediator either decides or escalates, and each guarantee concerns that round or the final outcome. Specifically, the complete policy has three properties (Theorem 4 in the revision):

(1) Every autonomous decision passes the safety gates (this extends Theorem 1). D-CEM decides only when Eτ≤εsafeE_\tau\le\varepsilon_{\mathrm{safe}} and ΔL(τ)≥msafe\Delta_L^{(\tau)}\ge m_{\mathrm{safe}}; otherwise, the case is escalated. Theorem 1 bounds the risk of any such decision using only the posteriors of that round, RL(d1(τ);Q(τ))≤BL(εsafe,msafe)R_L\big(d_1^{(\tau)};Q^{(\tau)}\big)\le B_L(\varepsilon_{\mathrm{safe}},m_{\mathrm{safe}}). As a result, the bound holds even if earlier steering raised the energy or changed an agent's diagnosis. Likewise, Proposition 1 applies to every challenge, since each target is recomputed from the current posterior.

(2) The actual risk is measured on the complete policy. The bound above relies on the confusion matrices, so we also bound the actual diagnostic loss. Under Assumption 5, we run the whole frozen controller θ\theta on independent cases and record only the final result of each case, which already includes every earlier action and how agents actually responded to challenges. This gives Rdec(θ)≤UθR_{\mathrm{dec}}(\theta)\le U_\theta with probability at least 1−ζ1-\zeta (Eq. 23). Empirically, the 95% upper bound is 0.329 and the observed test risk is 0.100 (Appendix I.6).

(3) Every case ends, and high-risk cases are escalated (this extends Theorems 2 and 3). Theorem 2 bounds the energy-plus-communication cost Jt=Et+δtJ_t=E_t+\delta t at the stopping round, and the extension adds explicit terms for the delays that steering causes. Moreover, challenges are finite; once they run out, Theorem 3 forces escalation if the energy stays above εlow\varepsilon_{\mathrm{low}}, and the round budget Tmax⁡T_{\max} ends every case.

In short, D-CEM is analyzed as one process, and its guarantees hold however the four actions are combined, without assuming that steering always helps.

In our revised manuscript, Section 4 states these properties as Theorem 4, with the proof in Appendix B.5.8, which also adds a new bound on the total system cost (diagnostic loss, escalation, and deliberation), Jp(θ)≤J^p(θ)+hJ\mathcal{J}_p(\theta)\le\widehat{\mathcal{J}}_p(\theta)+h_J.

W2 & Q2. Does D-CEM's lower harmful consensus rate come from escalation?

We ran exactly this comparison. For each scenario, we remove the cases escalated by D-CEM and evaluate all methods on the same retained cases (DDXPlus, H.C. in %, mean ± std over 10 seeds).

ScenarioRetainedSingle bestFree disc.Fixed poolD-CEM
S1 Ideal94.94%0.46 ± 0.538.16 ± 8.3910.73 ± 7.810.46 ± 0.53
S2 Asymmetric67.06%3.38 ± 1.5437.25 ± 8.8543.42 ± 7.530.27 ± 0.61
S3a Noisy-comp.53.94%41.86 ± 3.4458.17 ± 3.0458.17 ± 3.040.86 ± 1.01
S3b Noisy-non-comp.80.06%10.70 ± 2.2827.34 ± 6.8033.29 ± 4.690.72 ± 0.85
Macro-avg.74.00%14.10 ± 1.0332.73 ± 4.0536.40 ± 3.390.58 ± 0.47

On the cases that D-CEM keeps, the two discussion baselines still reach harmful consensus in 32.73% and 36.40% of cases on average, in contrast to 0.58% for D-CEM. This indicates that the gap remains when all methods are compared on the same non-escalated cases.

In addition, two comparisons already in the paper address the same concern from the other directions:

  • Escalation turned off in D-CEM (Appendix I.3, Tables 5 and 15). Its H.C. on DDXPlus is 1–3% across S1–S3b, against 9–38% for Free discussion and 12–38% for Fixed pool; on MedQA-Tri, it also has lower system risk and H.C. than both.
  • Escalation given to the baselines (Appendix G, Table 8). D-CEM has the lowest escalation-inclusive cost and the lowest or tied-lowest H.C. on all three datasets. For instance, at the same 26% escalation rate on DDXPlus, D-CEM has lower cost (0.341 vs. 0.389) and H.C. (0.4% vs. 0.7%) than Static-CM-ER.

In our revised manuscript, Section 5.3 summarizes the retained-case comparison and points to the matched-escalation results (Appendix G) and the no-escalation results (Appendix I.3). Appendix F.4 gives the full retained-case table and the evaluation protocol.

AuthorsOpenReview ↗

Response to Reviewer JPYk (2/2)

Q3. Why does the energy need all three terms, and why is the margin used twice?

The three terms come directly from the risk bound: they are the three quantities that control the bound in Theorem 1 (Appendix B.5.4), and the energy puts them into one score:

εdiag(t)=αDsym(P1(t),P2(t))+β∑aRL(y^a(t);Pa(t))+γexp⁡(−ΔL(t)).\varepsilon_{\mathrm{diag}}^{(t)}=\alpha D_{\mathrm{sym}}\big(P_1^{(t)},P_2^{(t)}\big)+\beta\sum_a R_L\big(\hat{y}_a^{(t)};P_a^{(t)}\big)+\gamma\exp\big(-\Delta_L^{(t)}\big).

Each term catches a different failure:

  • The disagreement term is large when the agents have not resolved their conflict.
  • The report-risk term is large when the agents agree, but the agreed diagnosis may hide a high-cost miss. This is the "consensus masks risk" case.
  • The margin term exp⁡(−ΔL(t))\exp(-\Delta_L^{(t)}) is large when the best and second-best decisions are almost equally good, i.e., the decision is still ambiguous.

As for the margin, it appears twice because it does two different jobs. Inside the energy, it lets the mediator track during deliberation whether the decision is becoming clearer; in STOP_AND_DECIDE, it is a hard requirement at the moment of decision. This separate gate is needed because the energy is a sum: low disagreement and low report risk can make it small even when the margin is still small.

Empirically, replacing the full energy by a risk-only score in the DDXPlus steering study raises H.C. from 1.2% to 8.0% and autonomous system risk from 0.132 to 0.564 (Appendix I.1, Table 10).

In our revised manuscript, Section 3.1 introduces each energy term from the risk bound and says which failure it captures, and Section 3.2 explains the two roles of the margin: tracking inside the energy, and a hard requirement in STOP_AND_DECIDE.

Dear Reviewer JPYk,

Thank you for your detailed and constructive review. We genuinely appreciate your positive feedback. As the discussion period is nearing its end, we wanted to gently follow up and ask whether you have any remaining questions or whether our responses have adequately addressed your comments. Thank you again for your time and thoughtful review.

Best regards,

Authors

Comments on the paper

5 comments
AuthorsOpenReview ↗

Response to Reviewer 21dZ (1/2)

We sincerely appreciate your insightful review and constructive feedback. We are encouraged that you recognize our principled protocol-level formulation, the use of off-diagonal confusion-matrix structure to expose costly alternatives, and our comprehensive validation. We address each question and suggestion below.

W1 & Q1. How are the confusion matrices estimated, and do they stay reliable under intervention?

Each CaC_a is estimated before deployment from one-shot (round-1) independent reports on a small labeled set (CAL, 70–80 cases). We then hold the CMs fixed during deliberation because the mediator cannot access the true label of the current case, so a pre-deployment offline estimate from historical data is a practical approach. Our work focuses on short-term deliberation; for longer-term use, periodic recalibration between deployment windows can be applied.

Regarding the shift in error patterns after DIFFERENTIAL_STEER, we add the two experiments you suggested: per-round and post-steering calibration of the fixed-CM posteriors, and a comparison with round- and intervention-conditioned CMs. Both show that fixed CMs remain reliable across discussion rounds, even after steering.

Experiment 1: do fixed CMs still match agent behavior after steering? Using DDXPlus ground-truth labels, we check fixed-CM posteriors and predicted risks against observed outcomes, both across rounds and for the same agent on the same case just before vs. after DIFFERENTIAL_STEER (nn is the number of agent-round reports; all metrics are group means).

GroupnnPosterior TV ↓Brier ↓Pred. report risk RL(y^a;Pa)R_L(\hat{y}_a;P_a)Obs. report loss L(y^a,y)L(\hat{y}_a,y)
Matched pre-steer (round tt)2,4600.1220.8022.9302.728
Matched post-steer (round t+1t+1)2,4600.1200.7882.8782.620
Round 1 (all initial cases)3,8400.1630.4471.6031.200
Rounds 2–3 (continuing cases)3,7920.1610.6502.1772.043
Rounds 4–6 (unstopped hard tail)4,4100.2030.7472.1652.337
Certified stops (rounds 2–6)2740.2450.4281.1340.880

Takeaway: steering does not widen the gap between fixed-CM and empirical posteriors (TV 0.122 → 0.120). Moreover, predicted risk exceeds observed loss in rounds 1–3 and at certified stops, i.e., the estimates are conservative where decisions are made; slight underestimation appears only in unresolved hard cases (rounds 4–6), which escalation covers. Targeted challenges therefore do not break the fixed-CM risk estimates.

Experiment 2: does recalibrating the CMs make the controller more reliable? We then run the complete DDXPlus controller with three CM choices, all fitted on CAL: fixed CaC_a, separate pre-/post-steer CMs (intervention-conditioned), and round-specific CMs (round-conditioned).

CM policyR^Lpool\widehat{R}_L^{\mathrm{pool}} ↓R^Lsys\widehat{R}_L^{\mathrm{sys}} ↓J^esc(1)\widehat{J}_{\mathrm{esc}}(1) ↓H.C. (%) ↓Esc. (%)Avg. rounds
Fixed CaC_a0.799 ± 0.0450.132 ± 0.0310.574 ± 0.0141.20 ± 0.2444.2 ± 4.53.14
Pre-/post-steer Capre/CapostC_a^{\mathrm{pre}}/C_a^{\mathrm{post}}0.809 ± 0.0740.136 ± 0.0310.554 ± 0.0161.09 ± 0.2741.8 ± 1.53.03
Round-specific Ca(t)C_a^{(t)}0.660 ± 0.0430.123 ± 0.0360.541 ± 0.0031.15 ± 0.0941.8 ± 3.33.13

Takeaway: safety is similar across all three (H.C. 1.20% vs. 1.09%/1.15%), while recalibrated CMs only modestly lower escalation (44.2% → 41.8%) and total cost. However, they need multi-round deliberation on CAL and split its 70–80 cases across rounds or intervention states, which is a substantial offline overhead. Fixed CaC_a is therefore a lightweight default with similar observed safety.

Moreover, safety is preserved even under much larger CM errors: when each CM is mixed with a uniform matrix, Cη=(1−η)C+ηUC_\eta=(1-\eta)C+\eta U, H.C. stays at 0.4% even at η=0.5\eta=0.5, and only the total cost rises (main DDXPlus setting: 0.341 → 0.445; Appendix I.5, Table 20).

In our revised manuscript, Section 2.2 clarifies how the CMs are estimated and states the short-term scope, and Appendix I.2 reports both analyses (Tables 13 and 14).

AuthorsOpenReview ↗

Response to Reviewer 21dZ (2/2)

W2-1 & Q2. Larger neutral-prompt evaluation with naturally occurring harmful consensus

D-CEM reduces naturally occurring harmful consensus at every label-space size, both at full coverage and at matched escalation. This does not rely on role anchors: D-CEM is a protocol-level controller that only takes agent predictions and Ca,CbC_a, C_b as input, and the controlled scenarios (S1–S3b) only isolate specific failure modes.

In the original submission (Appendix I.4, Table 16), the gain at neutral K=8K=8 is small because round-1 H.C. is only 2.5%. In contrast, at K=16K=16 and 2424, where pre-intervention H.C. rises to 8.6% and 8.3%, the gain is much larger: D-CEM reduces it to 1.6% and 0.2%.

To confirm this at a larger scale, we add a new 10-seed neutral-prompt evaluation (Appendix I.4, Table 17) with 320/384/576 TEST cases per seed at K=8,16,24K=8, 16, 24 (2× to 3.6× the original split). We pair gpt-4o-mini with Llama-3.2-3B, so harmful consensus arises naturally from the capability gap (Llama-3.2-3B alone has round-1 H.C. of 25.9–32.2%). Each cell reports H.C. ↓ / J^esc(1)\widehat{J}_{\mathrm{esc}}(1) ↓.

MethodEscalationK=8K=8K=16K=16K=24K=24
Single best (before intervention)0%4.8% ± 0.7% / 0.31411.1% ± 0.9% / 0.58715.1% ± 0.8% / 0.862
Free discussion0%12.2% ± 1.0% / 0.81615.3% ± 0.9% / 0.91221.3% ± 0.7% / 1.276
D-CEM w/o escalation0%4.6% ± 0.9% / 0.3017.8% ± 0.8% / 0.42810.2% ± 0.8% / 0.669
Static-CM-ER3.2% / 11.7% / 11.5%4.1% ± 0.9% / 0.3405.8% ± 1.1% / 0.4338.2% ± 1.9% / 0.654
D-CEM (full)3.2% / 11.7% / 11.5%3.8% ± 0.8% / 0.2874.2% ± 1.4% / 0.3605.1% ± 1.1% / 0.517

Takeaway: before intervention, harmful consensus is frequent: Single best ends in it in 4.8–15.1% of cases, and Free discussion drifts further (12.2–21.3%). At matched escalation, D-CEM has lower H.C. than Static-CM-ER at every KK (e.g., 5.1% vs. 8.2% at K=24K=24), at lower total cost. Even at 0% escalation (100% coverage), D-CEM cuts H.C. relative to Free discussion by 49.0% and 52.1% at K=16K=16 and 2424, with the lowest cost among methods without escalation.

In our revised manuscript, Appendix I.4 reports this larger neutral-prompt evaluation with a third backbone and the matched-escalation comparison (Table 17).

W2-2 & Q3. Comparison with anti-consensus multi-agent baselines

As you suggested, we add two recent multi-agent baselines explicitly designed to disrupt premature consensus and promote dissent/divergence:

  • Free-MAD [1]: consensus-free debate that combines anti-conformity prompting with full-trajectory answer scoring, so early minority diagnoses are not overwritten by final-round majority voting.
  • Catfish-Agent [2]: clinical dissent injection through a dedicated challenger agent that disrupts agreement bias and triggers re-evaluation.

Each cell reports (J^esc(1)\widehat{J}_{\mathrm{esc}}(1) ↓ / H.C. ↓ / Accpool\mathrm{Acc}_{\mathrm{pool}} ↑).

MethodSyntheticDDXPlusMedQA-Tri
Free-MAD [1]0.305 ± 0.017 / 0.036 ± 0.008 / 0.736 ± 0.0140.930 ± 0.072 / 0.165 ± 0.016 / 0.731 ± 0.0180.681 ± 0.052 / 0.159 ± 0.016 / 0.637 ± 0.025
Catfish-Agent [2]0.337 ± 0.018 / 0.056 ± 0.011 / 0.722 ± 0.0131.189 ± 0.095 / 0.224 ± 0.021 / 0.707 ± 0.0190.690 ± 0.043 / 0.160 ± 0.014 / 0.630 ± 0.020
D-CEM0.043 ± 0.050 / 0.000 ± 0.000 / 0.989 ± 0.0140.341 ± 0.036 / 0.004 ± 0.003 / 0.803 ± 0.0270.640 ± 0.034 / 0.122 ± 0.012 / 0.652 ± 0.008

Across Synthetic, DDXPlus, and MedQA-Tri, D-CEM consistently outperforms Free-MAD and Catfish-Agent on J^esc(1)\widehat{J}_{\mathrm{esc}}(1), H.C., and Accpool\mathrm{Acc}_{\mathrm{pool}}. Moreover, even at 0% escalation, D-CEM w/o escalation already lowers H.C. to 0.000, 0.022, and 0.155 on the three datasets. This indicates that D-CEM's protocol controller provides benefits beyond anti-conformity prompting or dissent injection alone.

In our revised manuscript, Appendix G describes how both baselines are implemented and reports the full comparison (Table 8).

[1] Cui et al., "Free-MAD: Consensus-Free Multi-Agent Debate," arXiv:2509.11035, 2025.

[2] Wang et al., "Silence is Not Consensus: Disrupting Agreement Bias in Multi-Agent LLMs via Catfish Agent for Clinical Decision Making," arXiv:2505.21503, 2025.

Thank you for the detailed response; I have no further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.