- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies an information exchange method among multiple agents for medical datasets, which can slightly improve the system's performance under a limited budget.
Critical strengths
The paper's structure is complete, multiple medical datasets are used, and the proposed method can also improve the baselines to some extent.
Critical weaknesses
Compared with majority voting, the improvement of the proposed method is too small, even less than 1%. The paper lacks confidence intervals for the different methods. Adding confidence intervals is necessary to determine whether the proposed method is statistically significantly better.
Introducing different personas for different agents may lead to better performance, and additional experiments are needed to investigate this possibility.
Furthermore, the paper excludes all samples for which every agent initially provides an incorrect answer. However, these samples may still yield correct answers after interaction. The reviewer would like to see the results on this subset.
Questions for authors
The paper focuses on medical datasets, but the proposed pipeline seems applicable to all types of datasets. Has the method been specifically adapted for medical datasets?
On PubMedQA, the proposed method seems to perform worse than the baseline. What are the possible explanations for this?
Recently, several second-order information aggregation methods have been proposed. Would these methods outperform majority voting in your setting?
3 comments
Reply to Reviewer Q6Z7 (2/2)
W4. PubMedQA performance decrease
We examined group-level correctness changes during communication on 1,000 PubMedQA questions with Qwen3.8-27B. There were 10 wrong-to-correct (W→C) transitions but 15 correct-to-wrong (C→W) transitions, a net loss of five correct answers (0.5 percentage points). This directly motivates modeling both correction and harm: peer messages can repair mistakes but can also overturn correct group decisions.
Downstream aggregation also shows task dependence. In Table 4, Consensus Refinement changes PubMedQA accuracy from 79.00% to 77.90% for Dynamic and 78.80% to 77.60% for Static, relative to Majority Vote. Thus, additional communication or answer revision is not uniformly beneficial. We will clarify these observations alongside the component discussion in Appendix F.1 (Table F1).
W5. Alternative aggregation and medical applicability
Table 4 compares Majority Vote, Weighted Vote, Final Judge, and Consensus Refinement while holding MedDAC communication policies fixed. Consensus Refinement raises overall macro-average accuracy from 73.13% to 74.57% for Static and 72.90% to 74.56% for Dynamic. These are aggregation benefits, not effects attributable solely to graph selection.
To address the suggestion about additional aggregation, we evaluated ISP and OW-I alongside Majority Vote and Weighted Vote:
Table R2. Additional Aggregation Strategy Evaluation
| Aggregation | Dynamic Acc. (%) | Static Acc. (%) |
|---|---|---|
| Majority Vote | 73.02 | 73.16 |
| Weighted Vote | 73.23 | 73.36 |
| ISP | 73.09 | 73.33 |
| OW-I | 73.10 | 73.32 |
Weighted Vote performs best in this supplementary evaluation, improving upon Majority Vote by 0.21 percentage points for Dynamic and 0.20 for Static. ISP and OW-I also exceed Majority Vote but do not outperform Weighted Vote. This indicates that added aggregation complexity need not improve accuracy. We will include the new analysis in Appendix G.2 (Table G2).
Finally, MedDAC uses no handcrafted medical ontology or diagnosis-specific rules: it learns the corrective and harmful consequences of receiver-specific peer interventions. The general intervention and graph-selection framework could be adapted to other tasks with suitable utility calibration, but five medical QA benchmarks do not establish clinical safety or cross-domain transfer. We will clarify that distinction.
We thank the reviewer again for helping us more clearly separate communication selection, harmful peer influence, and downstream aggregation.
Dear Reviewer Q6Z7,
Thank you again for your thoughtful review. As the discussion deadline is approaching, could you please let us know whether our responses address your concerns? We would be happy to provide further clarification before the discussion closes if any issues remain.
Thank you for your time and consideration.
Warm regards, The Authors
Reply to Reviewer Q6Z7 (1/2)
We thank the reviewer for the careful assessment and valuable questions concerning modest gains, agent diversity, and calibration coverage.
W1. Performance improvements and statistical reliability
Table 1 evaluates the complete MedDAC pipeline with Consensus Refinement (Static: 74.57%; Vanilla: 69.77% across five benchmarks). Table 2 instead fixes Majority Vote to focus on communication selection: Static reaches 73.13%, Random-K6 72.52%, and Full Communication 72.99%. These comparisons answer different questions; the complete-pipeline gain should not be attributed solely to communication selection.
We additionally conducted paired bootstrap analyses (10,000 resamples) and stratified paired permutation tests.
Table R1. Paired Statistical Analysis of Communication Policies
Static shows a positive paired gain over Random-K6 (unadjusted p=0.0165). Dynamic's improvement is positive but not statistically significant under the permutation test. Both MedDAC policies are constrained to six directed messages, compared with twelve for Full Communication. The contribution is learned selective communication under a limited budget, rather than maximizing communication. Statistical results will be included in Appendix E.1 (Table E1).
W2. Sampling diversity versus specialist agents
We deliberately keep the same backbone and task instructions across four agents, varying sampling temperatures (0.1, 0.4, 0.7, 1.0). This isolates directional communication effects from heterogeneous backbones and specialist prompting; assigning a specialist persona does not itself establish verified clinical expertise. Each receiver has seven nonempty peer subsets, giving 28 candidate receiver-specific actions whose joint effects are estimated directly. The formulation can accommodate heterogeneous agents after appropriate recalibration, but we do not claim to have validated that extension.
W3. Initially all-wrong cases
Mixed-correctness selection is restricted to labeled calibration, where having both correct and incorrect initial answers helps identify helpful corrections and harmful revisions relative to matched self-review. No ground-truth correctness labels or all-correct/mixed/all-wrong classification are used at test time; all test questions remain in evaluation. An all-wrong group might recover through new reasoning induced by communication, but such recovery is not guaranteed, and the current calibration subset does not directly validate utilities in that regime. We will clarify this scope.