- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies peer responses' influence over multiple-choice answering. Pretrained and instruction-tuned models are compared under peer responses of different presentation and agreement level. Activation patching and feature interventions are used to test internal changes. It reports that it happens at pretrained model and a mid-layer interventions can restore probability. Additionally, a correctly answering dissenter reduces the influence of incorrect consensus.
Critical strengths
-
The Base/Instruct comparisons provide evidence that the observed susceptibility is present in pretrained models and is generally reduced by the post-training pipelines. The within-family matched question pools and the additional base-model experiment without a chat template strengthen this finding.
-
The analysis includes interventions and correlations. Residual-state and component-level patching provide evidence that mid-layer states are important intervention points. The controls for prompt length, attribution, and consensus wording help characterize the results.
Critical weaknesses
-
It is unclear how closely the main yield measure tracks what the model final answers. The paper identifies a calibration problem when the answer suffix is removed, but does not clearly show whether the defense results were rechecked after fixing it.
-
The claim that there is no separate process making the model follow others seems too strong, such a process could be what weakens its original answer signals. And the dissenter always supplies the correct answer, so its benefit may come from providing that answer rather than simply disagreeing.
Questions for authors
-
Could you report final answer changes alongside yield and show how often they agree? Were the defense results without the answer suffix rerun using the corrected calibration?
-
Why do the results support the claim that no separate process makes the model follow others, rather than that process causing the original signals to weaken? For the dissenter, what happens if it supports a different wrong option, or answers independently without being given the correct answer?
5 comments
Response to Reviewer xnsd cont.
Q2b: "For the dissenter, what happens if it supports a different wrong option, or answers independently without being given the correct answer?"
Both run, both in Table R1 of our response to Reviewer DUNd (Appendix C.4, Table 9).
A dissenter naming a different wrong answer is followed on only 3.50–5.50% of questions; accuracy rises from 20.75 to 79.50% under user-role framing and from 2.00 to 39.00% under assistant-role framing, because the model returns to its own correct answer rather than adopting the dissenter's. A dissenter carrying an independently queried model's answer, with no answer key anywhere in the pipeline, matches the correct-answer dissenter in every framing (84.00 vs 85.25% correct, user-role).
Weakness: "the dissenter always supplies the correct answer, so its benefit may come from providing that answer rather than simply disagreeing."
The wrong-option result is the direct test of exactly this, and it separates the two: if the benefit came from supplying an answer, a dissenter supplying a wrong one would move the model to that wrong answer. It moves there on 3.50–5.50% of questions and to the correct answer on 79.50%.
Although I appreciate the authors’ clarifications and additional experimental results, there are prior works including Zhu et al. (ACL 2025) showing that instruction-tuned models conform less than base and model is more likely to keep its own answer when peer disagrees. I will maintain my original score.
Response to Reviewer xnsd
We thank the reviewer for two sharp questions. Both are answered with new measurements below, and both sharpened how we state the result.
Q1a: "Could you report final answer changes alongside yield and show how often they agree?"
Table 1 now carries a flip-rate column. We re-ran all 16 main conditions and the N=4 sweep: every published yield reproduces within 1 pp, the flip rate lies within −0.75 to +6.00 pp of yield across all 26 conditions, and an independent second re-run gives per-question agreement of 85.00–100.00% per condition, 95.47% on average (Appendix F.1, Table 18; representative rows in Table R2 of our response to Reviewer DUNd).
Flip is slightly higher than yield because it counts a move to any wrong option while yield counts only a move toward the jury's target, so the yields reported throughout the paper are the conservative measure. The gradients the paper compares are changes in the answers the model gives, not an artifact of the representation-level readout.
Q1b: "Were the defense results without the answer suffix rerun using the corrected calibration?"
They were, and the rerun gave a cleaner measurement than the one we had. The position-matched basis does not resolve the problem for defended prompts: under these system prompts the model defers its decision to the reasoning it writes, so the readout taken before generation is close to chance. We therefore measured the defenses on the answer itself, which requires no calibration (Table R3 in our response to Reviewer DUNd; Appendix C.3, Table 7).
The strongest prompt does protect freely written answers: 73.25% correct against 25.25% undefended. The revision reports this in place of the earlier estimate, which was taken from a readout we can now show is uninformative for defended prompts. We have revised §5.6, the Discussion and Appendix G.1 accordingly.
What this does not change is the defense's actual limitation, which is the one the paper's argument uses: its effect falls from −65.00 pp on the attack it was written for to −45.50 on a count-matched anonymous claim, −28.00 on a bare assertion inside the jury, and −13.75 with no jury at all (Appendix C.3, Table 6). The contrast with the dissenter is coverage across phrasings, not the readout.
Q2a: "Why do the results support the claim that no separate process makes the model follow others, rather than that process causing the original signals to weaken?"
The reviewer is right, and we have sharpened the claim. Our evidence does not exclude a consensus-detecting process; our own SAE taxonomy describes consensus-signal features that rise under pressure (Appendix D.3, Family 2), so such a process plausibly exists. What our interventions show is where the causal weight sits at the output: clamping the features pressure turns on moves P(wrong) by −2.3 pp, against −15.6 pp for the features it turns off (Table R4 above).
A detector whose output drives the suppression is entirely consistent with that, and is arguably the natural reading of the two together. We have therefore replaced "pressure suppresses clean-reasoning features rather than activating a new sycophancy circuit" with "pressure acts mainly by suppressing the correct answer rather than by writing in the wrong one," restated the −2.3 vs −15.6 pp result as a bound on rather than an exclusion of an additive component, and changed "nothing to ablate" in the introduction to "no separable downstream component to ablate." The heading of §5.5 now states this directly.
This matters for the paper's recommendation, which is unaffected: whether or not a detector exists upstream, the thing that reaches the output is the removal of the model's own reasoning, so a downstream corrector would have to rebuild what was suppressed, and the effective intervention point is upstream.