- Rating
- 4(+2)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this work, the authors propose AceMAD to mitigate the Martingale Curse, which is a known problem for multi-agent debate. The AceMAD protocol attempts to increase the influence of agents whose predictions for their peers' beliefs are more accurate. Empirical results on a targeted subset of datasets show clear advantage of AceMAD over baselines.
Critical strengths
- The paper is well-written and well explains the complicated theoretical intuition behind the proposed method.
- The motivation and the way the authors try to solve the Martingale's Curse are highly relevant to the core problem commonly observed in multi-agent debate.
- The authors provided a broad range of experiments, and the "challenging" subset is well targeted for the problem setup.
- The experimental results are strong and well support the work's method.
Critical weaknesses
-
My main concern is related to the soundness of the theorems that support the method. Specifically, it feels like the assumptions stated in Theorem 4.6 can be a bit strong. In particular, the persistent ACE Gap assumption requires the exploratory agent to maintain a positive expected peer-prediction score advantage over the anchored majority, conditional on the debate history at every round. This seems potentially restrictive, especially early in debate when little interaction history is available. Could the authors report the exploratory and anchored agents' peer-prediction errors and score differences throughout the debate, including how frequently the exploratory agent lacks this advantage?
-
Another concern about the theory is that it seems the theorems don't fully account for the agents' changing beliefs throughout the debate. I've checked the derivations for Theorem 4.6, and to my understanding, the change in aggregate belief is only expressed by the changes in influence weights, and omits the effect of the changes in the agents' own beliefs. Since agents revise their beliefs during debate, these additional terms could offset the benefit of re-weighting even when the exploratory agent retains a positive signal gap. Could the authors account for these terms in the proof or explicitly state the additional assumptions needed? Also, it is unclear whether the calibration advantage persists as agents revise their beliefs and the initially minority answer becomes more widely supported. While Remark 4.7 discusses the minority gaining majority influence, it does not establish the subsequent evolution of beliefs and ACE scores. Clarification and empirical trajectories would help assess the mechanism's behavior through this transition.
-
The nature of this work and its underlying theorems work on expected values. However, the paper mainly reports a single-run result, if I am not wrong. Could the authors clarify the number of independent runs and report means and uncertainty estimates across repeated runs? It would also be helpful to include a small empirical analysis of the quantities underlying Theorem 4.6: e.g., the exploratory agent’s ACE score advantage over the anchored majority, the corresponding signal gap in probability assigned to the correct answer, the evolution of influence shares, and the change in aggregate belief in the correct answer across rounds. For example, estimating the score-gap and signal-gap parameters (gamma) and (beta), and reporting how often these gaps diminish or reverse, would help assess whether the theorem's assumptions describe the observed debate dynamics.
-
The results in Table 1 and Table 2 are at odds. To my understanding, both tables report the baselines' accuracy on the same subset of data with the same backbone LLM, GPT4o-mini. Could the authors clarify why the values are different in the two tables?
Questions for authors
See Weaknesses
Thank you for the authors' effort in conducting the extensive amount of additional experiments.The additional details are also helpful. It is also promising that the authors have updated the manuscript accordingly. I believe all my concerns have been sufficiently addressed, and would like to strongly advocate for acceptance. I am increasing my score to 4.