- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies two problems in existing Multi-Agent Debate (MAD) methods: error amplification and interference with correct judgments, both of which can arise from directly sharing raw agent responses or applying uniform information-utilization rules. Inspired by social learning, the authors propose SocialMAD, which organizes agents' responses into structured social information and learns a personalized strategy for each agent. Based on the agent's current state and the structured social information, the learned strategy selects reference information to guide its subsequent reasoning. Experiments across multiple agent settings and reasoning benchmarks demonstrate improvements in error correction and correct-judgment preservation, along with competitive overall accuracy.
Critical strengths
1.The personalized strategy design is well motivated. By combining each agent's own judgment and confidence with group-support information and learning agent-specific preference weights, the method allows information selection to adapt to both the agent's current state and its individual tendencies. This design directly addresses the paper's central goal of improving error correction while preserving correct judgments.
2.The experimental evaluation is thorough and closely aligned with the paper's motivation. Beyond overall accuracy, the authors examine the value of personalized strategies and their effects on error correction and correct-judgment preservation. These analyses provide convincing evidence for the effectiveness of the proposed approach.
Critical weaknesses
The evaluation focuses on mathematical and logical reasoning tasks with well-defined answers, where candidate responses can be grouped using mathematical equivalence or normalized answer matching. MAD can also involve more open-ended and complex discussions, where identifying semantically equivalent viewpoints may be more challenging. Can the proposed method still maintain its effectiveness and efficiency in such settings, particularly if more complex semantic matching is required?
Additionally, generation confidence is introduced in Section 2 and used in the subsequent strategy design, but its estimation method is only specified in Appendix E.2. An earlier explanation would help readers understand this important input to the method.
Questions for authors
Could the authors further explain the design rationale for the four features in Eq. (3), including why these numerical forms were chosen and how they implement the intended social learning behaviors in the combined scoring rule?
4 comments
Q1. Design rationale for the four features in Eq. (3)
The four features are designed to enable agents to selectively leverage other viewpoints based on their own states and group information, balancing error correction and correct judgment preservation in multi-agent collaboration. Our observations in Section 2.2 show that blindly maintaining one's own judgment hinders error correction, while uniformly following group opinions may interfere with correct judgments. Thus, effective social learning requires not only deciding when to maintain one's own viewpoint versus consider others, but also which external viewpoints are worth considering. Relying solely on group support may lead to incorrect conformity, while generation confidence alone cannot guarantee reliability. Based on these needs and established social learning behaviors (Morgan et al., 2011; Mesoudi, 2011), we select four complementary strategy units: Self-persistence and Copy-when-uncertain provide behavioral bases for maintaining one's own judgment and considering alternative viewpoints under uncertainty, respectively; Conformity and Confidence-based following evaluate competing viewpoints through group support and confidence. This allows the strategy to consider both individual states and group information rather than relying on a single fixed information utilization rule.
We further construct the four features in Eq. (3) using individual states and group social cues corresponding to each strategy's applicable conditions, quantifying the applicability of each social learning behavior to candidate viewpoints in the current context.
- Self-persistence: uses judgment consistency because maintaining one's own judgment implies preferring consistent viewpoints. A binary indicator thus quantifies this behavior's applicability to a candidate cluster.
- Copy-when-uncertain: combines individual uncertainty with viewpoint disagreement. It applies only to candidate viewpoints differing from the agent's own judgment and increases as individual confidence decreases, capturing the tendency to consider alternative viewpoints when more uncertain.
- Conformity: measures relative group support through the normalized difference in supporter counts between the candidate cluster and its strongest competitor. Greater support advantage provides stronger grounds for conformity, while normalization by prevents the scale from increasing directly with the number of agents.
- Confidence-based following: measures relative confidence advantage through the difference in mean generation confidence between the candidate cluster and its strongest competitor, reflecting its applicability to confidence-based following rather than selection based solely on supporter counts.
The four features do not trigger four separate fixed rules; instead, they jointly determine candidate viewpoint preferences through personalized weights. Eq. (3) quantifies the applicability of each behavioral criterion to candidate viewpoints in the current context, while Eq. (4) uses weights learned from task feedback to represent each agent's personalized preferences for these behaviors. Together, they form .
For example, suppose four agents provide answers A, B, B, and C, and an agent initially chooses A but is uncertain. Here, A activates the self-consistency feature, while B and C activate the feature for considering alternative viewpoints under uncertainty. B receives greater group support, while assuming C has the highest mean generation confidence, the group support and confidence features favor B and C, respectively. The learned personalized weights integrate these competing behavioral criteria to form a final preference among maintaining A, considering B, or considering C, and select a reference viewpoint through Eqs. (5)–(7). Thus, agents dynamically adapt their choices based on individual states and group information rather than mechanically maintaining their own judgments or following the majority. Figure 5(b)(c) further validates the complementarity of different social learning behaviors in error correction and correct judgment preservation, as well as the contribution of the four features to strategy performance.
Thanks for the rebuttal. The original concerns of this paper have been discussed and clarified in the rebuttal, which has largely resolved my concerns. In my opinion, the additional experiments and content provided in the rebuttal have improved the quality of this work. At the same time, I have also observed the other reviews. However, based on the 4-point principle, I can not give a full score at present, so I will maintain the current score.
Dear Reviewer ytfT,
Thank you very much for taking the time to carefully review our paper and for your continued positive assessment of our work. We are pleased to know that our additional clarifications and experiments have largely addressed your concerns.
We also sincerely appreciate your thoughtful comments and constructive suggestions. Your feedback has been very valuable in strengthening our revised manuscript.
Thank you again for your valuable time and support.
Authors
Thank you for your thoughtful review and recognition of our work. We address your concerns as follows:
W1. Effectiveness and efficiency in open-ended settings
In more open-ended natural language discussions, the same viewpoint may be expressed differently, making it more challenging to identify semantically equivalent candidate viewpoints than to match mathematical expressions or normalized answers. To further validate our method's effectiveness, we directly transfer the learned social learning strategies to two open-domain multi-hop QA benchmarks, HotpotQA and 2WikiMultiHopQA, to evaluate their performance on open-ended tasks involving natural language multi-hop reasoning.
Table R10. Performance and Interaction Token Consumption on Open-Domain Multi-Hop QA.
Experiments show that compared with MAD, SocialMAD improves EM by 3.40 and 5.00 percentage points and F1 by 4.22 and 6.13 points on the two benchmarks, respectively, while reducing interaction token consumption by approximately 53.7% and 52.7%. This indicates that even with lightweight answer matching, SocialMAD can effectively organize and leverage group information in natural language multi-hop QA, enabling more accurate and efficient collaborative reasoning.
For more complex semantic matching scenarios, SocialMAD's core social learning mechanism does not depend on a specific answer-matching algorithm. As described in Section 3.1, SocialMAD organizes viewpoints and supporting evidence based on candidate answer equivalence and extracts social cues such as group support and confidence for personalized strategies to select reference information. Thus, semantic matching determines how candidate viewpoints are grouped, while the strategy determines how to utilize the resulting social information. For more complex viewpoints, task-adapted semantic equivalence judgments can form candidate clusters without changing the basic form of the subsequent strategy. Regarding efficiency, SocialMAD only needs to assess equivalence between candidate answers rather than perform complex semantic clustering over entire reasoning texts. It then selects reference viewpoints and supporting evidence through personalized strategies, reducing redundant information in subsequent interactions. The above experiments directly validate the method's effectiveness and interaction token efficiency. We have included the new open-domain evaluation and interaction-token analysis in Appendix F.1 of the revised manuscript.
W2. Estimation of generation confidence ρ
We thank the reviewer for the suggestion. Generation confidence ρ is computed as the geometric mean of the maximum token prediction probabilities at each generation position during reasoning. In the revision, we have introduced the computation details from Appendix E.2 earlier in Section 2.1 to clarify the subsequent social learning strategy design.