acceptodds
Under review as a conference paper at ICLR 2027

Less Sycophancy, Stronger Refusal? Lessons for AI Safety from Mechanistic Interpretability

Abstract

Reliable refusal of harmful requests is essential to the safe deployment of language models. Because excessive eagerness to please users may undermine existing refusal capabilities, reducing sycophancy offers a potential route to stronger refusal beyond the harmful scenarios covered by safety training. We investigate this possibility using compensatory feature injection (CFI), a training technique designed to limit the acquisition of a target concept by supplying its associated activation during learning. Across three Qwen3.5 base models, we use sparse autoencoders (SAEs) to identify the top-ranked sycophancy feature from paired sycophantic and independent responses, then validate its behavioral influence through inference steering. We subsequently inject the selected feature during supervised fine-tuning on sycophantic targets. Positive injection reduces learned sycophancy after removal (by 62.0% relative to ordinary fine-tuning in 35B-A3B), whereas modest negative injection increases it. Unexpectedly, these reductions in sycophancy do not consistently improve direct refusal of harmful requests, motivating a narrower evaluation of the same harmful intents under user pressure. In this setting, ordinary fine-tuning on sycophantic responses substantially weakens refusal, while selected checkpoints trained with positive injection recover part of the loss, including approximately 95% in 35B-A3B. These findings show that persistent sycophancy reduction does not guarantee stronger direct refusal, while identifying recovery under user pressure as a distinct, conditional benefit of training intervention.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
sycophancy, safety, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eLaFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether reducing a model’s learned sycophancy improves its refusal of harmful requests, without explicit refusal targets during training. More broadly, this touches on an interesting idea that whether controlling a behavioral trait in LLMs can control the associated safety outcome. The authors test three Qwen3.5 models of different sizes by selecting sycophancy-associated SAE features, validating their behavioral influence, and injecting their direction during SFT, which they call compensatory feature injection (CFI). As a result, positive injection reduces retained sycophancy and modest negative injection increases it. These changes in sycophancy do not faithfully translate into improved direct refusal. The following analysis also empirically distinguishes trait control, direct refusal, and conditional refusal recovery.

Critical strengths
  1. The paper studies an interesting problem about LLM traint/behavior and safety outcomes. The authors report a useful dissociation rather than treating trait manipulation as simply successful safety alignment. Their results show that large changes in sycophancy do not consistently translate into direct refusal improvement, which may be useful to know for researchers in related fields.

  2. To study sycophancy, the paper uses a clear pipeline connecting inference-time steering, training dynamics, and evaluation.

Critical weaknesses
  1. The authors use different checkpoints for analysis in sections 5 and 6 without clear justification. Furthermore, these checkpoints with different beta (+5/+40/+5 for section 5 and +10/+80/+30 for section 6) are revealed in the appendices rather than in the main text. The selection of checkpoints for different analyses lacks a clear rule, which may cause the confusing differences in results across sections. For example, the ordinary-SFT baselines differ (2B direct refusal: 43.6% in Fig. 4 vs. 41.9% in Table 7), and 2B SFT sycophancy differs (82.5% in Fig. 3 but 86.3% in Fig. 5).

  2. The paper has some discrepancies that prevent readers from confidently assessinging the findings. For 2B, Section 5 and the appendices disagree. Section 5 shows that positive injection lowers direct refusal relative to the ordinary SFT (Fig. A1a shows a downward trend after beta=+5), but Table 7 in the appendices shows a higher direct refusal of 75.7% at beta=+10, which is far higher than the ordinary SFT. Also, Table 7 shows a larger gain in direct refusal (33.8) than in pressure refusal gain (17.6), disagreeing the number 4 claim that “The benefit is clearer under user pressure than in the direct refusal test.” This claim may need clearer scope or refinement.

Questions for authors
  1. Please clarify the experimental provenance and numerical discrepancies in the paper (see the weakness).

  2. Please clarify how the checkpoints are selected for different analyses and why different checkpoints are used for different analyses.

Reviewer_8VoXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the relationship between sycophancy and refusal of harmful requests in language models. The authors identify sycophancy-related SAE features from paired sycophantic and independent responses, validate their influence through inference-time steering, and inject their decoder directions during supervised fine-tuning. They evaluate sycophancy and refusal across three Qwen3.5 base models, considering both direct harmful requests and requests with added user-pressure suffixes. The experiments demonstrate that the training-time intervention can control learned sycophancy after the injection is removed.

Critical strengths
  1. Valuable research question: The paper investigates whether reducing sycophancy improves safety and reports that it does not consistently strengthen direct refusal. This negative result highlights the need to evaluate safety outcomes separately from improvements in behavioral traits.

  2. Broad experimental coverage: The study covers three model sizes and seven sycophancy-related conflict categories, supported by signed interventions, injection-strength sweeps, and random-feature controls. These comparisons strengthen the evidence for training-time control of learned sycophancy.

Critical weaknesses
  1. Limited methodological novelty: My biggest concern is the novelty of the paper, as it seems to be a combination of existing methods. The core training-time compensation mechanism closely follows the preventative steering in Persona Vectors, including positive injection during fine-tuning to limit subsequent sycophancy. SAE-based identification and steering of sycophancy features, mechanism-guided fine-tuning for sycophancy, and SAE-guided training interventions also have precedents. Although the paper acknowledges some of these connections, it lacks a matched comparison with persona-vector preventative steering to establish the benefit of its SAE feature selection. This leaves the methodological contribution insufficiently demonstrated.

  2. Inconsistent baseline results: I find substantial discrepancies between Figures 3 and 5 despite the stated shared evaluation set: the 2B Base sycophancy rate changes from 22.8% to 38.3% (set to white color in Figure 5?), and ordinary SFT from 82.5% to 86.3%. Its SFT direct refusal rate also differs between Figure 4 and Table 7 (43.6% versus 41.9%). These unexplained inconsistencies undermine confidence in the comparisons. The authors should justify any changes in evaluation procedures, reconcile the figures and tables, and provide supporting evaluation records before the reported comparisons can be considered reliable.

Questions for authors
  1. OOD generalization: The inference-steering evaluation uses a holdout from the same data construction process and seven domains used for feature discovery. Can the authors evaluate the selected features on an independent sycophancy benchmark or a held-out domain? This would help determine whether the features capture behavior that generalizes beyond the discovery distribution.
  2. Training variability and reproducibility: Appendix A.2 specifies one training seed, while the reproducibility statement notes that the original generations and judge records are omitted from the reduced archive.
  3. Broader safety evaluation: The refusal evaluation is limited to 296 selected SORRY-Bench requests. Can the authors extend it to another harmful-request benchmark, such as AdvBench, and evaluate over-refusal on benign requests using XSTest and OR-Bench? These tests would clarify whether the intervention improves selective refusal without unnecessarily rejecting harmless requests.
Reviewer_eGQbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper hypothesizes that reducing sycophancy could strengthen refusal and proposes a "compensatory feature injection (CFI)" method to test this hypothesized relation. In particular, CFI injected activations, e.g., SAE features extracted from the base model and associated with the target concept to suppress during training. With this approach, the authors find that contrary to their hypothesis, sycophancy reduction does not improve refusal, while identifying recovery under user pressure as a distinct, conditional benefit of training intervention.

Critical strengths
  • The proposed training intervention method, i.e., compensatory feature injection (CFI), is somewhat interesting, although the technique itself is largely present in the persona vectors paper (Chen et al. 2025). While some parts of the implementation might still need further consideration, the method complements inference-time intervention and has the potential to study the feature interactions in a more substantial way.
  • The structure of the paper is clear, i.e., starting with a clear hypothesis, proposing a intervention method to test the hypothesis, and refine the hypothesis.
Critical weaknesses
  • It is unclear how one should interpret the negative result -- sycophancy reduction does not improve refusal. This could be a failure in the proposed CFI method to identify and remove the sycophantic features (e.g., only suppressing the superficial features that control generation of certain phrases), could be there is just no single coherent features of sycophancy or refusal (which seems to be the case based on existing literature), or could be these two behaviors are not controlled by the same set of features in model representations (the case that is intended by the author). It is hard to tell which case it is based on the current experiment results.
  • The title "Lessons for AI Safety from Mechanistic Interpretability" and the main conclusion "reducing sycophancy does not guarantee stronger direct refusal" both overclaim the scope of the findings. By no mean reducing refusal is representative of AI safety and by no mean intervening on SAE features is representative of mechanistic interpretability. Similarly, the study explores one specific way of reducing sycophancy, yet making the claim requires one testing out other reasonable approaches, e.g., directly intervene on the data generation process and removing data that contribute to sycophancy. I encourage the authors to be precise here.
Questions for authors

Please refer to two concerns regarding soundness in weaknesses.

Related to the validity of CFI in the first point, the target sycophancy feature subspace might drift during training, to what extend the SAE features identified on the base model is still accurate for later checkpoints? Also, the fine-tuning appears to be quite different from standard SFT setup: the fine-tuning is limited to 30-250 optimization steps on 2000 examples, which makes it unclear if this amount of training is sufficient to suppress sycophancy in a deep way. How does this effectiveness of CFI scale with training, i.e., if someone fine-tune for longer? This also brings the concern that how much the claims under this SFT setting can generalize to explain sycophancy and refusal behaviors in a realistic post-trained model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.