- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper propose a closed-loop discussion control framework QUORUM to address the issues of insufficient information sharing and premature consensus in hidden-profile multi-agent reasoning. The system first uses six metrics to represent the current discussion status, and then uses a controller to dynamically select discussion actions. The authors computed the One-Step Regret of candidate actions using Exhaustive Counterfactual Rollouts, filtered and replaced controller proposals using Round-Wise Conformal Calibration, and selected the speaking agent using Action-Conditioned Routing. The authors conducted experiments on HiddenBench and a synthetic travel decision dataset, and the results showed that QUORUM significantly outperforms fixed Round-Robin Discussion and surpasses Full-Information Baseline in some settings.
Critical strengths
- This paper proposes a systematic closed-loop framework to solve the problem of information sharing failure in multi-agent reasoning tasks. By combining discussion status monitoring, adaptive intervention selection, and a speaker guidance mechanism based on action conditions, this paper provides a structured approach that can effectively encourage parties to show private evidence and mitigate the tendency to reach consensus prematurely.
- This paper evaluates several different controllers and provides several ablation studies. The results show that these designs are a big improvement over traditional polling-based discussions. And the ablation study also provides useful insights into the contributions of evidence-seeking behavior, speaker scheduling, and controller state characteristics.
Critical weaknesses
- The conformal certification mechanism proposed in this paper lacks a sufficient theoretical and empirical support. First, the mechanism concerns only a single-round surrogate regret rather than the correctness of the final decision or the optimality of the global policy. Furthermore, the validity of the mechanism relies on an assumption of round-specific exchangeability. However, if early conformal replacements change the state distribution in subsequent rounds, this exchangeability assumption may be no longer valid. More importantly, the actual contribution of conformal acceptance remains unclear. Since the framework exhaustively evaluates all candidate actions during the inference phase, the action with the minimum regret could have been easily seen. The paper lacks a counterfactual greedy baseline, a comparative experiment based on matched-split (i.e., a comparison with and without conformal acceptance), and empirical analysis on coverage and action rejection rate. Therefore, the necessity and effectiveness of this core authentication mechanism have not been convincingly demonstrated.
- There is limited empirical evidence regarding fairness and generalization capabilities. QUORUM requires extensive counterfactual reasoning for candidate actions, which introduces a significant amount of additional inference computation. However, this expense was not fully reflected in its communication cost metric, nor was it effectively controlled in baseline comparisons. Therefore, a stronger baseline model with comparable computational cost is needed to distinguish the benefits of adaptive coordination from those of additional inference and structured hints. Furthermore, the evaluation of this study focuses on the structured hidden-profile task, and the standard MLP controller it uses is trained on training data generated from all benchmark tasks. Although the paper includes a held-out evaluation, the fact that the test set used for this evaluation is different from the test set used by the full-task controller weakens the persuasiveness of its conclusions regarding generalization ability.
Questions for authors
- Since QUORUM exhaustively evaluates all candidate actions during inference, so what is the practical advantage of conformal acceptance over directly selecting the minimum-regret action?
- How is the round-wise exchangeability assumption maintained when conformal action replacements alter subsequent discussion trajectories?
- What is the actual computational overhead of exhaustive counterfactual rollouts, including additional LLM calls and token consumption?
3 comments
Thank you for your detailed response. Your reponse has largely solve my problems, but two remain. Firstly, current experiment results do not strongly support the independent benefits and robustness of conformal acceptance. More importantly, acknowledging the assumptions of round-wise exchangeability and independence between calibration data and the score-generating procedure does not establish that they hold under the deployed sequential policy, particularly with data-dependent calibration thresholds. I will keep my original score, although my overall assessment is closer to 2.5. I would have no objection if the AC and PC decide to accept the paper.
We sincerely thank the reviewer for their thoughtful feedback, encouraging reassessment, and openness toward acceptance.
Regarding independent benefits, we clarify that the previously reported results come from independent end-to-end runs with conformal screening enabled versus disabled on the same tasks, with all other components fixed. These experiments show improvements for Rules (91.1% → 93.3%), MLP (75.6% → 84.4%), and LLM+Vote (82.2% → 84.4%), providing evidence of independent benefits of conformal acceptance. We acknowledge that broader robustness need further evaluation.
Regarding exchangeability, we appreciate the reviewer's important theoretical observation. Our calibration procedure incorporates conformal acceptance during calibration itself, allowing trajectories to account for preceding interventions and more closely reflect the deployed sequential policy, thereby reducing calibration–deployment mismatch. While establishing exchangeability rigorously in this adaptive setting is challenging, our theoretical guarantee is explicitly conditional on the stated assumptions.
We greatly appreciate the reviewer's constructive engagement and valuable consideration.
We sincerely thank the reviewer for the constructive feedback and recognition of QUORUM's closed-loop framework and ablation studies. We have strengthened the manuscript with additional experiments and clarified the theoretical assumptions, computational costs, and generalization evidence.
1. Conformal Acceptance and Greedy Selection
We agree that exhaustive evaluation identifies the minimum-regret action. However, QUORUM preserves the controller's proposed action when its regret is within the calibrated threshold, rather than always minimizing immediate surrogate loss.
We added a controlled Qwen3-8B comparison (Appendix D, Table 10):
These results support the value of controller-guided selection over greedy and fixed policies, although they do not independently isolate conformal screening.
We additionally report before/after conformal accuracy on 45 held-out tasks: Rules: 91.1%→93.3%; MLP: 75.6%→84.4%; LLM: 86.7%→86.7%; LLM+Vote: 82.2%→84.4%.
New diagnostics (Table 13) report 97.2% proposal acceptance and 2.8% rejection across 248 decision states. We clarify that conformal coverage concerns one-step surrogate regret, not final-answer correctness or global policy optimality.
2. Round-Wise Exchangeability
We appreciate this important observation. We clarify that the calibration procedure incorporates conformal action selection during calibration trajectories, so earlier interventions influence subsequent states as they do during evaluation (Section 3.2).
Nevertheless, identical procedures do not automatically establish exchangeability, particularly with data-dependent thresholds. Proposition 1 explicitly assumes round-wise exchangeability of calibration and evaluation scores, and we make no unconditional sequential or trajectory-level guarantee. Appendix A clarifies these conditions and distinguishes statistical proposal coverage from the evaluated regret of executed actions.
3. Evaluation Fairness and Generalization
Fairness: Hidden-Post and QUORUM share a maximum of 10 communication rounds (Section 4.1). The controlled-policy experiments retain the Qwen3-8B backbone and communication templates, while action and routing ablations (Tables 7–10) further examine individual components. We acknowledge that differences in realized discussion length and prompting limit complete causal isolation.
Generalization: We added a 13-split held-out evaluation (Section 5.3, Appendix B.3). Each MLP is trained on 5 of 65 HiddenBench tasks and evaluated on the other 60. The held-out controller achieves 0.73 accuracy versus 0.77 for the all-task reference, supporting transfer to unseen tasks while acknowledging the minimal gap. Heterogeneous-agent and generated-benchmark evaluations provide additional evidence across configurations.
4. Computational Overhead
We agree that exhaustive counterfactual evaluation is computationally expensive. New Table 11 reports LLM calls, tokens, counterfactual evaluations, and execution time.
Full QUORUM averages 142.8 LLM calls (101.3 counterfactual), 123.0k input tokens, 13.8k output tokens, and 81.4 seconds per task. Fixed evidence seeking requires 65.5 calls and 35.3 seconds, while greedy minimum-J requires 160.1 calls and 90.8 seconds.
These measurements expose the accuracy–computation trade-off. Although greedy selection incurs greater measured overhead with lower accuracy in this experiment, we acknowledge that a strictly compute-matched comparison with stronger baselines remains open.
We thank the reviewer again for these valuable suggestions. The revisions provide stronger controlled comparisons, held-out generalization evidence, explicit computational measurements, and a more precise statement of conformal guarantees. We believe they strengthen the central finding that adaptive, state-aware communication control improves evidence integration in hidden-profile multi-agent reasoning, while clearly delineating the method's assumptions and limitations.