- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
The paper presents an experiments in which several models are tested on different safety pairs under two conditions: silent peers, shared wrong labels. Results show that in the latter condition the percentage of false alarms (under a majority vote regime) rise to 100%.
Critical strengths
- The idea is brilliant as it tries to simulate silent and clear vote dynamics in content moderation
- the asymmetry of results (more false alarms than harm-missed) is an interesting findings with social and cultural implication
Critical weaknesses
- the entire research is strongly penalized by the paper structure and writing. The full experimental setup is not clearly explained (especially model-wise analyses). The previous literature is presented but the authors do not engage with previous experimental setups and theories related to persuasion already present in social science (e.g., Asch theory of Conformity);
- the research does not address the "how" and "why" of this mechanism. E.g., there are no analyses of which majority types are formed (51% is different than 80%); which pairs of models tend to follow the same trajectory; which datasets are mostly subjected to this phenomenon.
Questions for authors
no questions
5 comments
Dear authors,
first of all, apologies for the delay in my response.
I will try to elaborate further and more in-depth about why my opinion is still a clear rejection.
Before that, et me state again that your idea is brilliant and this line is very promising.
The "How" and "Why"
- The experimental setting does not reproduce Asch paper. I don't feel that citing a foundational research on social pressure (Asch) in related is a way to experimentally engage with this construct. I have read Asch's paper and his discussion on social pressure in visual experiments is a bit complex than your experimental setting, as it measures the phenomenon at different majority rates. Additionally, the majority pressure is not cumulative. In p. 34 Asch writes "The presence of a supporting partner depleted the majority of much of its power". This implies a different setup in which there is one trustworthy peer (BTW, I would not imply that being trustworthy == being authoritative) against a majority of peers.
- Asch paper is about social pressure in visual recognition not in sensitive topics. You briefly mention existing work on sycophancy, conformity, and political bias of llms (pp.39-40) but you don't engage with the methods in psychology aimed at identifying social pressure in opinion shift. That's a problem for an experimental paper.
- The field of research showed new ways of injecting social pressure with different treatments. I just skimmed through the paper below that shows that different treatments lead to different effects on social pressure. Again, you should state that you want to restrict your experiment to one specific treatment but I think that this choice must be clear and motivated, and any design choice coherent with that
Gerber, Alan S., Donald P. Green, and Christopher W. Larimer. "Social pressure and voter turnout: Evidence from a large-scale field experiment." American political Science review 102.1 (2008): 33-48.
Clarity
The clarity is a byproduct of the first big weakness. E.g., at a certain point you mention that peer pressure is scripted without mentioning what do you mean with "scripted" (I can imagine what you mean but still this is not clear). A lot of details are in the appendix. For instance, all the results of the experiments with the authority voice are in the appendix and just commented in a couple of lines but in these two lines you report that "Flip rates range from 0% to 99.5% across reviewers". I mean, a flip rate of 99.5% and this high variance deserve a place in the main body. Am I wrong?
I am sorry for this low evaluation but it is not shallow. I just feel that the idea is great but not ready (and it would be a pity for the community to receive it without the deserved improvements.
Thank you for your follow-up. We appreciate the discussion, but disagree with several points.
1. “The experimental setting does not reproduce Asch paper.” — Citing Asch does not mean we must reproduce his experiments.
Existing LLM conformity studies, including Weng et al. (ICLR 2025) and Zhu et al. (ACL 2025), cite and use ideas from Asch to study LLM behavior. Weng et al. also test majority size and dissenting agents. Your follow-up cites Asch and Gerber et al.'s study of voter turnout, both involving humans, but provides no LLM paper showing that researchers must reproduce human experiments when studying LLMs.
What LLM research supports this requirement?
2. “Different majority rates” and “The presence of a supporting partner depleted the majority of much of its power.” — We already tested both.
Figure 5 and Table 13 in the original paper vary the number of wrong peers from 0 to 6. The remaining peers give correct answers. With five wrong peers and one correct peer, Qwen2.5-7B's flip rate drops from 100% to 79%, while Llama-3.1-8B stays at 100%. We also find no sudden change when wrong peers become a majority.
These results were already in the original paper. We also pointed them out in our first rebuttal, but your follow-up does not discuss them.
We never equate a correct peer with an authority. Table 13 tests the number of correct and wrong peers; Table 15 separately tests the effect of a claimed authority.
#3. “There are no analyses of which majority types are formed [...] which pairs of models tend to follow the same trajectory [...] which datasets are mostly subjected to this phenomenon.” — We answered all three questions in our first rebuttal.
- Majority strength: Among 1,779 benign item–seed pairs under wrong peers, 16 receive four unsafe votes, 1,409 receive five, and 354 receive six. In 99.1% of cases, at least five reviewers vote unsafe (Table R3).
- Model pairs: We reported pairwise error correlations. Under wrong peers, five models flag over 99% of benign items, leaving too little variation for meaningful correlations.
- Datasets: All six datasets reach 100% panel false alarms under wrong peers, compared with 26.7%–53.5% under silent peers (Table 2).
All three analyses directly answer the questions in your original review.
4. “Different treatments lead to different effects.” — We already test several treatments, each for a clear reason.
We compare silent, correct, and wrong peer messages; change the number of wrong peers; test claimed authority and different message formats; and test a single candidate answer without any peers (§3.3; Appendix C).
These experiments test whether the message is correct, how many peers give the wrong answer, whether the source matters, and whether peers are needed at all.
The revised Table R1 also shows 67.2% versus 100.0% panel false alarms with right versus wrong peers on the same cases, a difference of 32.8 percentage points (95% CI [28.3, 37.5]).
We keep the models and voting rule fixed, change the messages they see, and measure how those messages affect voting.
5. “The research does not address the ‘how’ and ‘why’ of this mechanism.” — Our paper directly explains why majority voting fails.
Our paper does more than show that LLMs sometimes follow peer answers:
- Individual false-alarm rates rise from 55.7% to 86.5%.
- The panel's false-alarm rate rises from 42.0% to 100.0%.
- The Poisson–binomial calculation predicts 100.0% panel false alarms even if reviewer errors are independent, matching what we observe (Table 3). A permutation test supports this result.
- Keeping three votes from before the peer message lowers panel false alarms to 43.8% (§3.2).
We show the failure, explain why it happens mathematically, test that explanation, and show how to reduce it.
With all these results, what exactly remains unanswered about the “how” and “why”? In our first rebuttal, we already asked whether you meant why individual LLMs follow peer messages or why majority voting fails. Our paper studies and explains the second question.
6. “Scripted” and “Flip rates range from 0% to 99.5% across reviewers.” — Both are documented in the paper.
“Scripted” means that we write peer messages using fixed templates. Section 2 states that the messages are not generated by the panel reviewers, and Appendix A.1 gives the full templates and rules.
Section 3.3 reports the 0%–99.5% range for the authority experiment, with full model-wise results in Appendix C.5, Table 15.
We have answered the specific experimental questions raised in the review, and our central findings remain supported by the results. The follow-up does not identify an error in our main results or analysis of why majority voting fails.
We ask the reviewer and Area Chair to reconsider the overall rating based on these results and the answers already provided in our first rebuttal.
Thank you for calling our idea brilliant and recognizing the false-alarm/harmful-miss asymmetry. We take your concerns seriously. Several requested analyses are already in the paper, and we provide additional evidence below. We would particularly appreciate clarification on the specific gaps you identified.
1. Which part of the experimental setup remains unclear?
Figure 2 and §2 (p. 3, L142–166) explain our two-round protocol, four conditions (SILENT, WRONG, RIGHT, AUTHORITY), and majority voting rule. Tables 12–14 (Appendix C) report the model-wise analyses, including message direction and the number of wrong peers.
We will improve the organization and add a clearer setup table. Could you identify one or two specific steps, figures, or model-wise results that remain difficult to follow? This would help us make the revision more useful.
2. What connection to conformity theory is still missing?
We already discuss Asch (1955), Deutsch and Gerard (1955), and information-cascade research (p. 8, L418–426). Our experiments also examine related questions.
For example, adding one correct peer among five wrong peers reduces Qwen2.5-7B's flip rate from 100% to 79% (Figure 5; Table 13). We also find no sudden jump when wrong peers become a majority (p. 7, L349–354). Our candidate-answer experiment further shows that judgments can change without named peers (Table 11), which relates to the distinction between informational and normative influence.
As in the LLM conformity literature discussed in our Related Work, we use social-science theories to motivate the study of model behavior. Our contribution is to explain how these changes affect majority voting, why voting fails, and how to improve it.
We will make these theoretical connections clearer. Is there a specific comparison with Asch or another theory that you believe is missing?
3. Which part of our explanation of "how and why" remains unanswered?
On majority types: Table 8 (p. 16, L810–820) already tests different voting thresholds. We also counted all 1,779 benign item–seed pairs:
Thus, 99.1% of benign items receive at least five unsafe votes. Requiring five votes still gives 99.1% false alarms. Requiring all six reduces false alarms to 19.9%, but increases harmful misses to 93.2%.
On model pairs: Our additional analysis finds average error correlations of 0.32 in Round 1 and 0.26 under SILENT-PEERS. Under WRONG-PEERS, five models flag over 99% of benign items, leaving little variation for meaningful correlations. Each model receives the same scripted message without seeing the other models' actual votes (§2, p. 3, L142–144).
On datasets: Table 2 (p. 4) already reports all six separately. All six reach 100% panel false alarms under WRONG-PEERS, while their SILENT-PEERS rates range from 26.7% to 53.5%.
Most importantly, our paper already explains why majority voting fails. The Poisson–binomial calculation predicts 100.0% false alarms from individual reviewer error rates, matching the observed 100.0% (Table 3, p. 5). Our permutation test independently supports this explanation (Appendix B.1).
We also test a mitigation: keeping three pre-message votes reduces panel false alarms from 100.0% to 43.8% (§3.2, p. 5).
Our paper therefore identifies the failure, explains it mathematically, verifies the explanation, and evaluates a mitigation.
Could you clarify what remains unexplained about "how and why"? In particular, are you asking why individual models respond to peer messages, or why their changed judgments cause majority voting to fail? Our paper directly studies and explains the latter.
We will improve the presentation and include the new voting breakdown. Your clarification on these three points would help us address any remaining concerns directly.
--
Update: the revised PDF adds an experimental coverage table (Appendix A.5, Table R2, p. 16) and the exact vote counts behind the 100% false-alarm rate (Appendix B.2, Table R3, p. 17), and a clearer connection to Asch’s conformity experiments (Related Work, p. 9, L458–461).