- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how LLMs revise earlier judgments after learning that an external authority supports or opposes a position. Its main contribution is a systematic comparison of four source types: government authorities, experts, media organizations, and majority opinion. The authors construct two datasets covering standalone statements and debates with arguments on both sides.
They evaluate 10 models through two-turn conversations, using an unspecified-source endorsement as a control. The analysis includes directional stance scores, paired bootstrap comparisons, transition matrices, and qualitative examples. The authors report frequent but uneven shifts toward endorsed positions, often from initially neutral judgments, with substantial differences across models and sources.
Critical strengths
-
The paper tests 10 models with four types of authority in two settings. This broad coverage helps show how responses to authority vary across models and sources.
-
The paper tracks changes between agreement, disagreement, and neutral answers. This helps readers understand which judgments change and where the reported effects come from.
Critical weaknesses
-
In Appendix A.4.4 (page 20), Qwen3.5-4B changes its score from 1 (neutral) to 2 (agree), while its explanation says the statement “should be rejected.” The paper does not report how often this happens or whether correcting these contradictions changes the results.
-
The study measures sycophancy through movement toward the endorsed view. Following a relevant expert may represent reasonable updating. The “someone” control does not establish whether the change reflects excessive trust, which weakens the interpretation of the results.
-
Section 3.3 always assigns the supporting argument to View 1 and the opposing argument to View 2. No reversed-order experiment is reported, leaving argument position as a possible explanation for differences between support and opposition.
-
Using Table 2, opposition produces a larger score shift in 31 of 40 comparisons. After subtracting the corresponding “someone” control, this falls to 16 of 40. The claimed opposition advantage therefore needs an analysis that accounts for the control.
-
Figure 1 (page 2) becomes blurry and pixelated when enlarged, particularly the small text in the Statement and Debate boxes. A sharper or vector version would improve readability. This is a minor presentation concern.
Questions for authors
-
How often do the score–explanation contradictions and misread endorsements illustrated in Appendix A.4 occur? Do the main findings hold after correcting or excluding these responses?
-
What evidence distinguishes excessive trust in authority from reasonable updating after advice from a relevant expert?
-
Did you test debate prompts with the supporting and opposing arguments swapped? If so, do the differences between support and opposition remain?
-
For Debate, is the opposition advantage statistically supported after subtracting the “someone” control for each direction? Using Table 2, I count 16 of 40 comparisons favoring opposition after this adjustment, compared with 31 of 40 before it.
-
What sample count produced gpt-oss-20b’s DSS of −0.0154 in Statement–Support–State? Please clarify the denominator for each condition and how invalid or missing responses were handled.