- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes ARMAD, a training-free framework that adaptively controls the initiation and duration of heterogeneous multi-agent debate (MAD) based on agreement among agents' final answers. Specifically, Pre-debate Agreement Routing (PAR) determines whether debate is necessary based on the agreement among independent initial responses, while the Early Agreement Stopping Evaluator (EASE) terminates ongoing debates once a predefined agreement threshold is reached. The framework employs a fixed pool of three heterogeneous LLMs and uses deterministic majority voting to aggregate their final predictions. Experiments on five reasoning benchmarks report an average accuracy of 82.7% with 3.96 model calls per example, representing a 67% reduction in calls compared with fixed three-round heterogeneous debate. The authors further investigate routing effectiveness, threshold sensitivity, and failure cases involving incorrect consensus.
Critical strengths
ARMAD requires neither additional training nor access to model internals, making it compatible with black-box LLM APIs. Its agreement-based routing and stopping mechanisms are straightforward to implement and introduce little computational overhead beyond the underlying model calls.
Critical weaknesses
Limited methodological novelty and missing comparison with closely related prior work.
The core mechanisms of ARMAD substantially overlap with existing multi-agent communication frameworks. In particular, Exchange-of-Thought (Yin et al., EMNLP 2023) already employs agreement-based termination, including skipping further communication when agents initially agree and stopping subsequent interactions once consensus is reached. These mechanisms closely resemble ARMAD's PAR and EASE components, with the primary differences lying in the specific agreement thresholds and experimental configurations. The paper neither discusses this closely related work nor sufficiently demonstrates what methodological contribution distinguishes ARMAD from such existing approaches.
Questions for authors
N/A