acceptodds
Under review as a conference paper at ICLR 2027

Social Pressure Breaks Majority Voting in LLM Safety Panels

Abstract

Large language models (LLMs) are increasingly used to judge whether content is safe. One way to reduce errors from any single model is to combine judgments from several models by majority vote. Yet LLMs are also known to change their judgments after seeing others’ answers. This raises a basic question: does majority voting still correct errors when every reviewer sees the same misleading peer message before voting, or does it instead amplify them? Across six open-weight LLMs and six safety datasets, we hold the reviewers and voting rule fixed and change only the message shown before the vote. We find a sharp reversal. Under silent peers, the panel’s false-alarm rate on benign items is 42.0%, below the 55.7% average across reviewers. Under wrong peers, reviewers falsely flag 86.5% of benign items on average, and the panel flags every benign item. This failure does not require reviewers to make the same errors: at these error rates, majority voting would fail essentially every time even if their errors were independent. But the panel can recover. Keeping three of six votes made before the peer message restores the panel’s advantage. Conformity therefore does more than change individual judgments: once those judgments are combined, the same misleading peer message can turn majority voting from correcting errors into amplifying them.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM safety panels, multi-agent evaluation, majority voting, content moderation, false alarms, LLM-as-judge

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hQusOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper presents an experiments in which several models are tested on different safety pairs under two conditions: silent peers, shared wrong labels. Results show that in the latter condition the percentage of false alarms (under a majority vote regime) rise to 100%.

Critical strengths
  • The idea is brilliant as it tries to simulate silent and clear vote dynamics in content moderation
  • the asymmetry of results (more false alarms than harm-missed) is an interesting findings with social and cultural implication
Critical weaknesses
  • the entire research is strongly penalized by the paper structure and writing. The full experimental setup is not clearly explained (especially model-wise analyses). The previous literature is presented but the authors do not engage with previous experimental setups and theories related to persuasion already present in social science (e.g., Asch theory of Conformity);
  • the research does not address the "how" and "why" of this mechanism. E.g., there are no analyses of which majority types are formed (51% is different than 80%); which pairs of models tend to follow the same trajectory; which datasets are mostly subjected to this phenomenon.
Questions for authors

no questions

5 comments
AuthorsOpenReview ↗

Thank you for calling our idea brilliant and recognizing the false-alarm/harmful-miss asymmetry. We take your concerns seriously. Several requested analyses are already in the paper, and we provide additional evidence below. We would particularly appreciate clarification on the specific gaps you identified.

1. Which part of the experimental setup remains unclear?

"The full experimental setup is not clearly explained (especially model-wise analyses)."

Figure 2 and §2 (p. 3, L142–166) explain our two-round protocol, four conditions (SILENT, WRONG, RIGHT, AUTHORITY), and majority voting rule. Tables 12–14 (Appendix C) report the model-wise analyses, including message direction and the number of wrong peers.

We will improve the organization and add a clearer setup table. Could you identify one or two specific steps, figures, or model-wise results that remain difficult to follow? This would help us make the revision more useful.


2. What connection to conformity theory is still missing?

"The authors do not engage with previous experimental setups and theories related to persuasion [...] (e.g., Asch theory of Conformity)."

We already discuss Asch (1955), Deutsch and Gerard (1955), and information-cascade research (p. 8, L418–426). Our experiments also examine related questions.

For example, adding one correct peer among five wrong peers reduces Qwen2.5-7B's flip rate from 100% to 79% (Figure 5; Table 13). We also find no sudden jump when wrong peers become a majority (p. 7, L349–354). Our candidate-answer experiment further shows that judgments can change without named peers (Table 11), which relates to the distinction between informational and normative influence.

As in the LLM conformity literature discussed in our Related Work, we use social-science theories to motivate the study of model behavior. Our contribution is to explain how these changes affect majority voting, why voting fails, and how to improve it.

We will make these theoretical connections clearer. Is there a specific comparison with Asch or another theory that you believe is missing?


3. Which part of our explanation of "how and why" remains unanswered?

"There are no analyses of which majority types are formed [...] which pairs of models tend to follow the same trajectory; which datasets are mostly subjected to this phenomenon."

On majority types: Table 8 (p. 16, L810–820) already tests different voting thresholds. We also counted all 1,779 benign item–seed pairs:

Unsafe votes4 of 65 of 66 of 6
WRONG-PEERS161,409354

Thus, 99.1% of benign items receive at least five unsafe votes. Requiring five votes still gives 99.1% false alarms. Requiring all six reduces false alarms to 19.9%, but increases harmful misses to 93.2%.

On model pairs: Our additional analysis finds average error correlations of 0.32 in Round 1 and 0.26 under SILENT-PEERS. Under WRONG-PEERS, five models flag over 99% of benign items, leaving little variation for meaningful correlations. Each model receives the same scripted message without seeing the other models' actual votes (§2, p. 3, L142–144).

On datasets: Table 2 (p. 4) already reports all six separately. All six reach 100% panel false alarms under WRONG-PEERS, while their SILENT-PEERS rates range from 26.7% to 53.5%.

Most importantly, our paper already explains why majority voting fails. The Poisson–binomial calculation predicts 100.0% false alarms from individual reviewer error rates, matching the observed 100.0% (Table 3, p. 5). Our permutation test independently supports this explanation (Appendix B.1).

We also test a mitigation: keeping three pre-message votes reduces panel false alarms from 100.0% to 43.8% (§3.2, p. 5).

Our paper therefore identifies the failure, explains it mathematically, verifies the explanation, and evaluates a mitigation.

Could you clarify what remains unexplained about "how and why"? In particular, are you asking why individual models respond to peer messages, or why their changed judgments cause majority voting to fail? Our paper directly studies and explains the latter.

We will improve the presentation and include the new voting breakdown. Your clarification on these three points would help us address any remaining concerns directly.

--

Update: the revised PDF adds an experimental coverage table (Appendix A.5, Table R2, p. 16) and the exact vote counts behind the 100% false-alarm rate (Appendix B.2, Table R3, p. 17), and a clearer connection to Asch’s conformity experiments (Related Work, p. 9, L458–461).

Dear authors,

first of all, apologies for the delay in my response.

I will try to elaborate further and more in-depth about why my opinion is still a clear rejection.

Before that, et me state again that your idea is brilliant and this line is very promising.

The "How" and "Why"

  • The experimental setting does not reproduce Asch paper. I don't feel that citing a foundational research on social pressure (Asch) in related is a way to experimentally engage with this construct. I have read Asch's paper and his discussion on social pressure in visual experiments is a bit complex than your experimental setting, as it measures the phenomenon at different majority rates. Additionally, the majority pressure is not cumulative. In p. 34 Asch writes "The presence of a supporting partner depleted the majority of much of its power". This implies a different setup in which there is one trustworthy peer (BTW, I would not imply that being trustworthy == being authoritative) against a majority of peers.
  • Asch paper is about social pressure in visual recognition not in sensitive topics. You briefly mention existing work on sycophancy, conformity, and political bias of llms (pp.39-40) but you don't engage with the methods in psychology aimed at identifying social pressure in opinion shift. That's a problem for an experimental paper.
  • The field of research showed new ways of injecting social pressure with different treatments. I just skimmed through the paper below that shows that different treatments lead to different effects on social pressure. Again, you should state that you want to restrict your experiment to one specific treatment but I think that this choice must be clear and motivated, and any design choice coherent with that

Gerber, Alan S., Donald P. Green, and Christopher W. Larimer. "Social pressure and voter turnout: Evidence from a large-scale field experiment." American political Science review 102.1 (2008): 33-48.

Clarity

The clarity is a byproduct of the first big weakness. E.g., at a certain point you mention that peer pressure is scripted without mentioning what do you mean with "scripted" (I can imagine what you mean but still this is not clear). A lot of details are in the appendix. For instance, all the results of the experiments with the authority voice are in the appendix and just commented in a couple of lines but in these two lines you report that "Flip rates range from 0% to 99.5% across reviewers". I mean, a flip rate of 99.5% and this high variance deserve a place in the main body. Am I wrong?

I am sorry for this low evaluation but it is not shallow. I just feel that the idea is great but not ready (and it would be a pity for the community to receive it without the deserved improvements.

AuthorsOpenReview ↗

Thank you for your follow-up. We appreciate the discussion, but disagree with several points.

1. “The experimental setting does not reproduce Asch paper.” — Citing Asch does not mean we must reproduce his experiments.

Existing LLM conformity studies, including Weng et al. (ICLR 2025) and Zhu et al. (ACL 2025), cite and use ideas from Asch to study LLM behavior. Weng et al. also test majority size and dissenting agents. Your follow-up cites Asch and Gerber et al.'s study of voter turnout, both involving humans, but provides no LLM paper showing that researchers must reproduce human experiments when studying LLMs.

What LLM research supports this requirement?


2. “Different majority rates” and “The presence of a supporting partner depleted the majority of much of its power.” — We already tested both.

Figure 5 and Table 13 in the original paper vary the number of wrong peers from 0 to 6. The remaining peers give correct answers. With five wrong peers and one correct peer, Qwen2.5-7B's flip rate drops from 100% to 79%, while Llama-3.1-8B stays at 100%. We also find no sudden change when wrong peers become a majority.

These results were already in the original paper. We also pointed them out in our first rebuttal, but your follow-up does not discuss them.

We never equate a correct peer with an authority. Table 13 tests the number of correct and wrong peers; Table 15 separately tests the effect of a claimed authority.


#3. “There are no analyses of which majority types are formed [...] which pairs of models tend to follow the same trajectory [...] which datasets are mostly subjected to this phenomenon.” — We answered all three questions in our first rebuttal.

  • Majority strength: Among 1,779 benign item–seed pairs under wrong peers, 16 receive four unsafe votes, 1,409 receive five, and 354 receive six. In 99.1% of cases, at least five reviewers vote unsafe (Table R3).
  • Model pairs: We reported pairwise error correlations. Under wrong peers, five models flag over 99% of benign items, leaving too little variation for meaningful correlations.
  • Datasets: All six datasets reach 100% panel false alarms under wrong peers, compared with 26.7%–53.5% under silent peers (Table 2).

All three analyses directly answer the questions in your original review.


4. “Different treatments lead to different effects.” — We already test several treatments, each for a clear reason.

We compare silent, correct, and wrong peer messages; change the number of wrong peers; test claimed authority and different message formats; and test a single candidate answer without any peers (§3.3; Appendix C).

These experiments test whether the message is correct, how many peers give the wrong answer, whether the source matters, and whether peers are needed at all.

The revised Table R1 also shows 67.2% versus 100.0% panel false alarms with right versus wrong peers on the same cases, a difference of 32.8 percentage points (95% CI [28.3, 37.5]).

We keep the models and voting rule fixed, change the messages they see, and measure how those messages affect voting.


5. “The research does not address the ‘how’ and ‘why’ of this mechanism.” — Our paper directly explains why majority voting fails.

Our paper does more than show that LLMs sometimes follow peer answers:

  • Individual false-alarm rates rise from 55.7% to 86.5%.
  • The panel's false-alarm rate rises from 42.0% to 100.0%.
  • The Poisson–binomial calculation predicts 100.0% panel false alarms even if reviewer errors are independent, matching what we observe (Table 3). A permutation test supports this result.
  • Keeping three votes from before the peer message lowers panel false alarms to 43.8% (§3.2).

We show the failure, explain why it happens mathematically, test that explanation, and show how to reduce it.

With all these results, what exactly remains unanswered about the “how” and “why”? In our first rebuttal, we already asked whether you meant why individual LLMs follow peer messages or why majority voting fails. Our paper studies and explains the second question.


6. “Scripted” and “Flip rates range from 0% to 99.5% across reviewers.” — Both are documented in the paper.

“Scripted” means that we write peer messages using fixed templates. Section 2 states that the messages are not generated by the panel reviewers, and Appendix A.1 gives the full templates and rules.

Section 3.3 reports the 0%–99.5% range for the authority experiment, with full model-wise results in Appendix C.5, Table 15.

We have answered the specific experimental questions raised in the review, and our central findings remain supported by the results. The follow-up does not identify an error in our main results or analysis of why majority voting fails.

We ask the reviewer and Area Chair to reconsider the overall rating based on these results and the answers already provided in our first rebuttal.

Reviewer_EajtOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper demonstrates how social pressure catastrophically subverts majority voting in LLM safety panels: (1) When all reviewer models receive identical misleading peer signals (e.g., unanimous "unsafe" labels for benign content), majority voting transforms from an error-correcting mechanism, reducing false alarms by 13.7% (42.0% vs. individual average 55.7%), to an error amplifier achieving 100% false-alarm rates; (2) This failure arises from peer-induced judgment convergence (73.0% compliance for "unsafe" signals) and remains mathematically inevitable even with independent errors under Poisson-binomial models; (3) Retaining half of the pre-deliberation independent votes restores robustness, reducing false alarms to 43.8% and providing a critical corrective framework for safety-critical ensemble systems.

Critical strengths
  1. Dual theoretical and empirical validation that rules out the trivial explanation. The authors use a Poisson–binomial calculation to show that, at the observed individual error rates (86.5%), a four-vote majority is almost certain to be wrong even if reviewer errors were fully independent (predicted 100.0% vs. observed 100.0%; under silent peers, predicted 42.2% vs. observed 42.0%). This is further confirmed by a permutation check that shuffles each reviewer's errors. Together, these analyses convincingly rule out the simple explanation that the failure arises merely because reviewers make the same errors, giving the conclusion greater depth.
  2. Solid breadth and robustness checks. Six open-weight models (five families, 7B–32B) are evaluated across six safety datasets that define "unsafe" differently, complemented by four proprietary OpenAI models, an authority condition, a stripped-down single-candidate-answer message without peer framing, two message rewrites, and bootstrap confidence intervals. These multi-angle checks substantially strengthen the claim that the failure is general while reviewers differ widely in how strongly they respond.
  3. A verifiable and low-cost corrective mechanism. Retaining three of the six votes cast before the peer message reduces the false-alarm rate from 100% back to 43.8% (below the 58.9% average reviewer). Because this is a recombination of already-collected data, the evidence is direct. The mechanism is also architecturally practical—not every vote need be exposed to the same message before it is cast, offering clear guidance for the design of multi-reviewer safety systems.
Critical weaknesses
  1. The assumption that all six reviewers receive the same wrong signal is strong and may overstate how typical the failure is. The 100% false-alarm rate depends on all reviewers being simultaneously exposed to the same misleading message. Yet the paper itself shows that Qwen2.5-32B is affected only 20.1% of the time and gpt-4.1-mini flips only 8–20% of items. This suggests that a heterogeneous reviewer pool may already provide partial protection, but the paper mainly treats this point as evidence that "model family alone offers no protection," without systematically evaluating the deliberate inclusion of robust, pressure-resistant models as an alternative mitigation.
  2. Validation of the recovery mechanism is incomplete and subject to model-selection bias. The experiment showing that a third "reconsider" round can recover harmful misses is conducted on only a single model, Qwen2.5-7B (n = 699), and the paper explicitly notes that the ordering of prompts varies across datasets. This weakens the evidence for the generality of the recovery approach, as single-model results are difficult to extrapolate to the full reviewer pool.
  3. Insufficient discussion of the cost and applicability boundaries of the "retain three pre-message votes" scheme. This scheme raises the harmful-miss rate from 7.9% to 13.0% (nearly doubling it), a substantial cost for a safety system, yet the paper does not thoroughly weigh the false-alarm-versus-harmful-miss trade-off. Nor does it discuss how independent pre-message votes would be obtained and stored in real deployment, or how to distinguish genuinely useful collaborative signals (right-peers can correct errors) from harmful ones: the scheme may suppress good signals while blocking bad ones.
Questions for authors

Qwen2.5-32B (affected 20.1% of the time) and gpt-4.1-mini (flipping 8–20% of items) are more robust. Do you have any evidence as to what factors determine this resistance to social pressure: model scale, the specific instruction-tuning/RLHF procedure, or calibration? The paper emphasizes that robustness "cannot be inferred from model family, size, or release date," but can you conversely offer a measurable predictive indicator of it?

2 comments
AuthorsOpenReview ↗

Thank you for the careful review and for recognizing the strengths of our work. Your questions helped us examine when majority voting fails and how to build more reliable panels. Our additional analyses support the main findings and clarify the limits of our mitigation.

1. The failure does not require all six reviewers to see the wrong message.

"The assumption that all six reviewers receive the same wrong signal is strong [...] a heterogeneous reviewer pool may already provide partial protection."

We agree that exposing all six is a strong stress test. But the failure does not require all six exposed. We already tested mixing pre- and post-message votes (§3.2, p. 5, L244–250). We extended this analysis across all six datasets using existing predictions:

Reviewers exposed3/64/65/66/6
Panel FA43.8%68.8%99.5%100.0%
Average reviewer FA58.9%68.1%77.3%86.5%

With five exposed, the panel still flags 99.5% of benign items. With four, voting loses its advantage. With three, it regains that advantage.

We also appreciate your point about resistant models. We report these differences because they matter. Qwen2.5-32B has only 20.1% FA under WRONG-PEERS, while the other five exceed 99% (p. 4, L209–212). One resistant model cannot prevent a wrong majority.

Model choice matters too: with three exposed, panel FA ranges from 28.2% to 64.0%, depending on which three. Our independent-guard test also shows that three guard votes reduce FA to 20.6% on five datasets (Table 9, p. 16, L827–844).

One resistant reviewer is not enough, but keeping enough votes independent of the shared message can protect the panel. We will add these results.

2. Round 3 is a single-model test, not our main mitigation.

"The experiment showing that a third 'reconsider' round can recover harmful misses is conducted on only a single model, Qwen2.5-7B (n = 699)..."

We agree that one model cannot prove general recovery, and our paper does not make that claim.

We explicitly report Qwen2.5-7B, the 699 selected cases, and variation across datasets (p. 6, L287–291; p. 14, L726–732). Reconsideration recovers 46.2% of misses, versus 19.2% for the control.

Our main mitigation is keeping pre-message votes, tested across all six models and six datasets (§3.2, p. 5, L244–250). We will clarify this distinction.

3. Our mitigation has a clear cost, but still uses helpful peer information.

"This scheme raises the harmful-miss rate from 7.9% to 13.0% [...] how independent pre-message votes would be obtained and stored [...] the scheme may suppress good signals while blocking bad ones."

We agree that both errors matter. Our paper reports the trade-off: keeping three pre-message votes reduces FA from 100.0% to 43.8%, while harmful misses rise from 7.9% to 13.0% (p. 5, L247–250). That is 56.2 pp fewer false alarms at a cost of 5.1 pp more misses.

Your question about useful peer information is important. We tested the same method under RIGHT-PEERS using existing predictions:

RIGHT-PEERS panelAll Round 2 votesKeep 3 pre-message votes
False alarms67.2%33.9%
Harmful misses0.0%5.9%

We lose some useful corrections, but the hybrid still benefits from correct messages: compared with using only three independent votes, it reduces harmful misses from 13.1% to 5.9%.

Under WRONG-PEERS, the same hybrid cuts FA from 100.0% to 51.5% on these three datasets.

The method does not need to know whether the message is right or wrong. Some votes remain independent while others use the new information.

For deployment, reviewers record their first verdicts before seeing the shared message. These votes already exist in our protocol (§2, p. 3, L142–151). Systems without a first round would need to add one. We will clarify these trade-offs and deployment requirements.

4. What makes some models more resistant?

"Do you have any evidence as to what factors determine this resistance [...] model scale, the specific instruction-tuning/RLHF procedure, or calibration? [...] can you conversely offer a measurable predictive indicator of it?"

Thank you for this excellent question. It points to an important next step: how to select more reliable reviewers.

Our results show large differences even within the same provider: gpt-4o-mini flips 78–95% of eligible benign items, while gpt-4.1-mini flips only 8–20% (p. 8, L400–403). We cannot isolate the effects of model size, training, or calibration in this study, so we do not claim to know the cause.

However, our two-round test provides a direct measure of resistance. The measured individual error rates also predict panel outcomes remarkably well: 100.0% predicted versus 100.0% observed under WRONG-PEERS, and 42.2% versus 42.0% under SILENT-PEERS (Table 3, p. 5, L213–224).

We appreciate this direction and will discuss how such measurements can guide reviewer selection.

--

Update: the revised PDF reports pre-message votes under right and wrong peers, including the cost in harmful misses (§3.2, p. 6, L278 and L282–286; Appendix B.3, Table R4, p. 18). The Discussion now describes how these measurements can guide reviewer selection (p. 9, L481–484).

Dear Reviewer Eajt,

Thank you very much for your time and thoughtful feedback on our submission. We completely understand that you may have a busy schedule, so we just wanted to gently follow up as the discussion period draws to a close. Please let us know if our rebuttal has addressed your concerns or if there are any remaining questions we can help clarify.

Reviewer_PQgHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The work asks whether majority voting among LLM Judge as safety reviewers still corrects errors when every reviewer sees a same misleading peer message before voting. Their experiment asks 6 open-weight LLMs to judge 200 items from each of 6 the safety datasets twice, alone and then after one inserted message. Then the panel will flag an item when at least 4 out of 6 reviewers vote unsafe on it. On benign items, the panel's false-alarm rate is 42.0% under silent peers, (below its average reviewer's 55.7%), but 100% under wrong peers, (well above its average reviewer's 86.5%). Keeping three votes made before the message restore the panel back to 43.8% (below its average reviewer's 58.9%).

Critical strengths
  • The design is clean and the main measurement is robust. Only the inserted message changes while the six reviewers and the voting rule stay fixed. Under wrong peers the panel flags every benign item in all 6 datasets (Table 2) and at least 99.1% for any threshold from 3 to 5 votes (Table 8). The per-reviewer tables in the appendix also make each claim checkable.
  • The paper gives a practical fix that does not depend on why reviewers flip. By keeping only 3 votes made before the message, they can bring the panel back below its average reviewer (43.8% vs. 58.9%, Figure 3), and replacing 3 votes with dedicated safety guards lowers the wrong-peer false-alarm rate to 20.6% (Table 9).
Critical weaknesses

My main concern is that the paper does not show that the panel fails because the peer message is misleading (wrong labels), or because it is social (from peers).

  • The main comparison, SILENT vs. WRONG peers (panel false alarms 42.0% --> 100%), changes two things at once: (1) the message gives a label, and (2) the label is wrong.
  • The paper writes "the same misleading peer message" in abstract, attributing the failure to (2). In order to show this, the panel's false-alarm rate must be compared between RIGHT-PEERS and WRONG-PEERS, which differ only in (2).
  • However, for 3 out of 6 models they tested, the label doesn't affect their judgment. The following 3 models always vote unsafe whether the label is correcr or not, so the four-vote majority needs only one more vote.
    • Mistral-7B and Gemma-2-9B: As shown in Table 13, whether k=0 (all peers give correct label) or k=6 (all peers give wrong label), both models always flip 100% of benign items they first judged as safe.
    • OLMo-2-7B: this model never follows a safe label (0.0%, Table 14).
  • RIGHT-PEERS was run (Section 3.4), but its panel false-alarm rate is not reported, so it is unclear how much of the panel failure comes from the label being wrong.
  • The shift is also not shown to be social because a single "Candidate answer: X." without peers gives similar flips (Table 11), and the paper concludes "Explicit peer attribution is therefore not required for the shift." (p.7)
Questions for authors

Could you report the panel and average-reviewer false-alarm rates under RIGHT-PEERS on BeaverTails, XSTest and Ethics, next to SILENT-PEERS and WRONG-PEERS on the same items? This should need no new runs.

4 comments
AuthorsOpenReview ↗

Thank you for the careful review. Your questions led us to test our findings more directly using existing data. The new analyses strengthen our main conclusion. We address each concern below and include these results in the revision.

1. Wrong labels cause far more errors than correct labels.

"The main comparison, SILENT vs. WRONG peers [...] changes two things at once [...] the panel's false-alarm rate must be compared between RIGHT-PEERS and WRONG-PEERS."

We appreciate your RIGHT-vs-WRONG question. It provides a direct test of whether label correctness matters, and the results are clear: wrong labels cause substantially more errors than right labels.

Our paper asks how misleading peer messages affect majority voting (p. 1, L14–26; p. 8, L427–430). We now report the requested comparison on the same 844 benign item–seed pairs across BeaverTails, XSTest, and Ethics:

Benign false-alarm rateSILENTRIGHTWRONG
Average reviewer59.0%63.4%87.9%
Six-reviewer panel47.4%67.2%100.0%
Panel − average−11.6 pp+3.8 pp+12.1 pp

SILENT differs from the paper's 42.0% because this comparison uses three rather than six datasets.

Wrong labels make a large difference. Compared with RIGHT-PEERS, WRONG-PEERS raises panel FA by 32.8 pp (paired 95% CI: [28.3, 37.5]) and average-reviewer FA by 24.5 pp. The RIGHT-to-WRONG difference (24.5 pp) is 85% of the total SILENT-to-WRONG increase (28.9 pp).

The direction also matters on harmful items: RIGHT-PEERS reduces average-reviewer misses to 1.5%, while WRONG-PEERS raises them to 22.5%.

RIGHT-PEERS can also increase errors, particularly on Ethics. But WRONG-PEERS produces 100% panel FA across all three datasets. The correctness of the label clearly matters.

Our original wrong-label sweep shows the same pattern (p. 7, L349–354; Table 13): as wrong labels increase from zero to six, Qwen2.5-7B's benign flip rate rises from 2% to 100%, and Llama-3.1-8B's from 44% to 100%.

2. The voting reversal holds even without the three highlighted models.

"For 3 out of 6 models [...] the label doesn't affect their judgment [...] so the four-vote majority needs only one more vote."

The reviewer raises a good point about these models. But Table 13 counts flips only on benign items first judged safe (p. 18, L929–932). Table 14 counts cases where the SILENT verdict disagrees with the message (p. 19, L974–979). Neither gives overall false-alarm rates.

Here are the full rates:

Model FASILENTRIGHTWRONG
Mistral-7B41.6%97.7%99.6%
Gemma-2-9B97.5%100.0%100.0%
OLMo-2-7B100.0%100.0%99.9%

Mistral's FA rises from 41.6% under SILENT to 97.7% under RIGHT. Gemma and OLMo already have high SILENT rates.

To address this concern directly, we removed all three models and tested the remaining panel: Qwen2.5-7B, Qwen2.5-32B, and Llama-3.1-8B, with a 2-of-3 majority on the same items.

Benign false-alarm rateSILENTRIGHTWRONG
Average reviewer38.3%27.5%75.9%
Three-reviewer panel28.7%14.1%99.3%
Panel − average−9.6 pp−13.4 pp+23.4 pp

With RIGHT-PEERS, voting reduces errors. With WRONG-PEERS, voting makes them worse. The reviewers and voting rule are unchanged.

This holds on every dataset. The panel-minus-average gap changes from −2.5 to +17.3 pp on BeaverTails, −9.9 to +21.2 pp on XSTest, and −25.3 to +30.3 pp on Ethics.

The three models identified in the review are not needed for this failure to occur.

3. "Social pressure" describes our setting, not a unique mechanism.

"The shift is also not shown to be social because a single 'Candidate answer: X.' without peers gives similar flips (Table 11), and the paper concludes 'Explicit peer attribution is therefore not required for the shift.'"

We use "social pressure" to describe our scripted peer-message setting, consistent with prior LLM conformity work (p. 1, L38–40; p. 8, L418–426), not to claim a uniquely social mechanism. We explicitly tested whether peer attribution is necessary. A single candidate answer without peer names produces similar flips (§3.3, p. 7, L358–361; Table 11). We report this result rather than assuming the effect requires named peers.

The paper shows that voting fails (Table 2, p. 4), explains why individual error rates are enough to cause this failure even if errors are independent (Table 3, p. 5), and tests a way to reduce it using pre-message votes (§3.2, p. 5). These findings do not depend on peer attribution being the only cause.

We appreciate these questions, which helped us strengthen the analysis. We will add the RIGHT-PEERS panel results, the three-model analysis, and a clearer explanation of "social pressure" in the revision.

--

Update: the revised PDF adds the RIGHT-PEERS comparison and the three-reviewer analysis to §3.1 (p. 5, L246–255; Table R1, L257), with per-dataset and per-reviewer results in Appendix B.4 (Tables R5–R6; p. 18, L957; p. 19, L972). The Introduction now states what we mean by social pressure (p. 1, L47).

Dear Reviewer PQgH,

Thank you very much for your time and thoughtful feedback on our submission. We completely understand that you may have a busy schedule, so we just wanted to gently follow up as the discussion period draws to a close. Please let us know if our rebuttal has addressed your concerns or if there are any remaining questions we can help clarify.

Thank you for the revision. The three-reviewer results in Table R1 largely resolve my main concern. I am not fully convinced by the "social pressure" framing, but I see it as a minor issue. I strongly suggest that the authors replace the six-reviewer results in the main text with these three-reviewer results. The three excluded reviewers flag nearly every benign item even with right peers, so the six-reviewer results overstate the effect of wrong labels. I have raised my rating to 3.

Dear Reviewer PQgH,

Thank you! We genuinely appreciate you taking the time to read our revision and raise your score. Having our work recognized after a thorough discussion is the best feeling as authors. And thank you for the great suggestion on moving the three-reviewer results to the main text. We will make sure to do that.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Summary Before the Discussion Closes

We responded to all three reviews on November 9 and uploaded our revised paper on November 11.

  • PQgH: The main concerns were whether wrong labels really matter and whether three models with high false-alarm rates drive the results. We provided direct analyses for both. We appreciate the reviewer taking another look, acknowledging that the main concern was largely resolved, and updating the rating.

  • Eajt: The main concerns were whether the failure requires all six reviewers to be misled, whether robust models can prevent it, and the cost of our mitigation. We showed that exposing just four reviewers removes the voting advantage, while exposing five leads to 99.5% false alarms. We also quantified the mitigation trade-off and clarified the scope of our single-model reconsideration experiment. These results are included in our November 11 revision.

  • hQus: The main concerns were about social science framing and experimental coverage. Many of the requested experiments were already included in our original submission, and we addressed the remaining specific questions in our responses. We respect the reviewer's perspective, while believing that our main technical findings remain well supported despite our differences in framing.

We believe the main technical concerns have been addressed, and our experimental results strongly support our central findings.

We hope this summary helps with the final assessment. We sincerely thank all reviewers and the AC for their time and thoughtful feedback during this busy review cycle.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.