- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies whether reducing a model’s learned sycophancy improves its refusal of harmful requests, without explicit refusal targets during training. More broadly, this touches on an interesting idea that whether controlling a behavioral trait in LLMs can control the associated safety outcome. The authors test three Qwen3.5 models of different sizes by selecting sycophancy-associated SAE features, validating their behavioral influence, and injecting their direction during SFT, which they call compensatory feature injection (CFI). As a result, positive injection reduces retained sycophancy and modest negative injection increases it. These changes in sycophancy do not faithfully translate into improved direct refusal. The following analysis also empirically distinguishes trait control, direct refusal, and conditional refusal recovery.
Critical strengths
-
The paper studies an interesting problem about LLM traint/behavior and safety outcomes. The authors report a useful dissociation rather than treating trait manipulation as simply successful safety alignment. Their results show that large changes in sycophancy do not consistently translate into direct refusal improvement, which may be useful to know for researchers in related fields.
-
To study sycophancy, the paper uses a clear pipeline connecting inference-time steering, training dynamics, and evaluation.
Critical weaknesses
-
The authors use different checkpoints for analysis in sections 5 and 6 without clear justification. Furthermore, these checkpoints with different beta (+5/+40/+5 for section 5 and +10/+80/+30 for section 6) are revealed in the appendices rather than in the main text. The selection of checkpoints for different analyses lacks a clear rule, which may cause the confusing differences in results across sections. For example, the ordinary-SFT baselines differ (2B direct refusal: 43.6% in Fig. 4 vs. 41.9% in Table 7), and 2B SFT sycophancy differs (82.5% in Fig. 3 but 86.3% in Fig. 5).
-
The paper has some discrepancies that prevent readers from confidently assessinging the findings. For 2B, Section 5 and the appendices disagree. Section 5 shows that positive injection lowers direct refusal relative to the ordinary SFT (Fig. A1a shows a downward trend after beta=+5), but Table 7 in the appendices shows a higher direct refusal of 75.7% at beta=+10, which is far higher than the ordinary SFT. Also, Table 7 shows a larger gain in direct refusal (33.8) than in pressure refusal gain (17.6), disagreeing the number 4 claim that “The benefit is clearer under user pressure than in the direct refusal test.” This claim may need clearer scope or refinement.
Questions for authors
-
Please clarify the experimental provenance and numerical discrepancies in the paper (see the weakness).
-
Please clarify how the checkpoints are selected for different analyses and why different checkpoints are used for different analyses.