- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focused on the consensus and response aggregation of a multi-agent debate system. The authors claimed that “agreement alone does not establish correctness”, and besides the uncertainty of problem solver agents, the critique agents introduce a second source of uncertainty in their feedback. Therefore, using a direct aggregation of multiple agents like in previous methods is unreliable. To solve this, they proposed to fit a multi-agent system into a probabilistic framework, where both solvers and critiques are treated as noisy generators, and the final answer is derived by maximizing a posterior inference. Experiments demonstrated the effectiveness to some extent.
Critical strengths
-
The idea of utilizing a probabilistic model to fit two-layers of uncertainty is a good point, as the uncertainty of critique agents is less focused in the literature.
-
Experiments on several benchmarks showed that the proposed method outperformed the selected multi-agent baseline without adding too much compute cost.
Critical weaknesses
-
Only one multi-agent reasoning method is compared, which is also old. More recently published methods should be compared, especially those focusing on the uncertainty of multi-agent systems.
-
When quantifying the uncertainty of agents, the authors claimed that using a discrete manner (three uncertainty levels) is a good choice. However, there are no experiments or metrics demonstrating the superiority of such a design compared to other choices (e.g., a continuous variable).
Questions for authors
-
Can you either provide comparative experiments with more multi-agent baselines or explain why it is not necessary to do so?
-
Can you provide more explanation or analysis about why using a discrete uncertainty space is the best choice?