acceptodds
Under review as a conference paper at ICLR 2027

Unlearning Sycophancy Improves Machine Creativity in Large Language Models

Abstract

Large Language Models (LLMs) tend to echo users' preferred answers, leading to the sycophancy issue. This conversational habit inherently stifles divergent thinking and restricts genuine machine creativity. Directly supervising models on creativity-oriented data often causes out-of-distribution degradation, while current sycophancy interventions are effective exogenously or compromise general model capabilities. Instead of teaching models to imitate creativity, we formulate sycophancy mitigation as a behavioral unlearning problem and explore whether unlearning undesirable sycophantic behaviors can provide an alternative pathway for improving LLM creativity. We construct sycophancy-specific forget sets through self-inoculation and introduce a lightweight positional refinement of negative preference optimization. Our method outperforms standard fine-tuning and unlearning baselines in sycophancy mitigation, creativity enhancement, and utility preservation, establishing sycophancy unlearning as an effective, cost-efficient approach to enhance LLMs' creativity without direct supervision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Machine Unlearning, AI Sycophancy, AI Creativity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2goaOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on using unlearning to mitigate sycophancy in LLMs and encourage their creativity. They introduce position-aware Negative Preference Optimization (p-NPO), which re-weights prefix tokens based on existing methods. The empirical results suggest the method mitigates sycophancy, improves creativity, and maintains general capabilities. Additional analysis is presented on the method and the performance correlation.

Critical strengths
  • The idea of using unlearning to indirectly encourage creativity is novel and reasonable to me.
  • The experiment design considered maintaining general capability while mitigating sycophancy and improving creativity.
  • The experiment scale is comprehensive.
  • The ablation study is solid.
Critical weaknesses
  • I think the main idea of the connection between sycophancy and creativity is not solid. Though the authors explained, I don't fully agree with the anti-side of sycophancy being creativity. A model can be both sycophantic and creative, e.g., agreeing with the user's given belief but generating a much more detailed idea the user hasn't thought of. On the other hand, from my perspective, what should be unlearned is the answer/idea that the user feels is not novel, which seems to be a different target from mitigating sycophancy.
  • Given this, I see the empirical improvement in creativity more like a side effect of these methods. I am not convinced that mitigating sycophancy and improving creativity have a causal relationship. The checkpoints analysis is on the same group of training trajectories where many common causes may exist, leading to a spurious correlation. I see the authors mention this in the limitations, but this point does negatively impact the novelty and excitement I feel about this paper.
  • The experiments don't compare other existing creativity-promoting methods (prompting-based, training-based, etc). So we actually don't have a reference to understand, for example, how significant the improvement of +0.85 is on DAT.
  • If viewing this paper mainly as a sycophancy mitigation method, it seems slightly incremental based on existing work (SimNPO), and the token weights are heuristics.
Questions for authors
  • I encourage the authors to include creativity-improving methods in the comparison.
  • Also, looking into creativity-improving methods can provide a more solid analysis in 6.2 (show correlation) and in 6.3 (identify the creativity gain vector direction).
  • I encourage the authors to think and discuss more about whether this is a causal relationship or not. If this is not the case, they should consider down-weighting creativity as motivation in this project (e.g., in the introduction), and think of what the potential gaps should be; if this is the case, more rigorous statistical analysis to eliminate the possibility of a causal relationship should be done.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 2goa (Part 1)

We sincerely thank the reviewer for the positive assessment of our work, particularly regarding the novelty of using unlearning to improve creativity. We also appreciate the thoughtful questions, which help us further clarify the contributions and implications of our work. We address the concerns point by point below.


1. Creativity Improvement and Comparisons with Existing Creativity-Enhancing Methods (Weakness and Question regarding creativity-improving baselines)

We appreciate the suggestion to compare our method with existing creativity-enhancing approaches. We would like to clarify that comparisons with creativity-enhancing baselines are already included in our original submission (see Figure 2, lines 262–265 and lines 365–372), including the prompt-based method (OPRO), inference-time intervention (Steering), as well as training-based methods (SFT and DPO). The results show that p-NPO outperforms existing creativity-enhancing baselines.

To provide a clearer comparison with existing creativity-enhancing methods, we further rank all five methods across the three creativity benchmarks based on their respective scores. Rank 1 indicates the best performance:

BackboneBenchmarkRank 1Rank 2Rank 3Rank 4Rank 5
Llama-3-8BDATp-NPO (82.46)OPRO (81.79)SFT (78.56)DPO (78.31)Steering (77.66)
Creat-v3p-NPO (41.99)OPRO (41.34)DPO (25.62)SFT (25.43)Steering (19.23)
LitBenchp-NPO (88.20)OPRO (65.00)DPO (48.55)SFT (39.75)Steering (33.75)
Qwen2.5-7BDATDPO (73.34)SFT (73.25)p-NPO (73.21)Steering (72.39)OPRO (71.43)
Creat-v3p-NPO (34.23)OPRO (31.22)Steering (31.12)DPO (20.70)SFT (17.97)
LitBenchp-NPO (70.45)OPRO (57.50)Steering (51.30)DPO (50.50)SFT (33.15)

The ranking results show that p-NPO achieves the strongest overall performance among the five methods, ranking first in five out of six comparisons. In particular, p-NPO ranks first on both Creative Writing v3 and LitBench across both backbones, demonstrating consistent advantages. On DAT, p-NPO also achieves the highest score on Llama, while remaining highly competitive on Qwen, with its score closely matching those of the top-ranked DPO and SFT baselines. These results further support the effectiveness of p-NPO across different dimensions of creativity.

2. The Relationship Between Sycophancy Mitigation and Creativity Improvement (Weakness and Question regarding causal relationship between creativity and sycophancy)

We thank the reviewer's thoughtful concern regarding the causal relationship between sycophancy mitigation and creativity improvement. As acknowledged in the Limitations section, we agree that our current experiments do not establish a definitive causal relationship between the two model properties.

Establishing a strict causal relationship between two high-dimensional behavioral properties of an LLM is challenging. Training inevitably modifies model parameters and potentially affects multiple models capabilities, such as the reasoning ability. Consequently, isolating the effect of sycophancy reduction from all other changes in the model is difficult. We therefore avoid claiming a universal causal law that reducing sycophancy must increase creativity. Instead, our study investigates an intervention: what happens to creativity when we selectively unlearn sycophantic behaviors. In the meanwhile, Discussion of Section 6.3 and the related experiments provide further evidence to help distinguish the effects of sycophancy-targeted unlearning from generic parameter updates induced by unlearning. More broadly, we believe this investigation has value even without establishing a mathematically definitive causal relationship. Our findings suggest that selectively unlearning undesirable behavioral tendencies offer an alternative pathway for improving existing model capabilities, without directly supervising the model on the target capability.

We acknowledge that the presentation in our original Introduction could have been clearer. In particular, the emphasis on the “sycophancy → creativity” relationship may have inadvertently suggested that we claim a direct causal relationship between these two properties. This is not our intended claim. Rather, our research focuses on “unlearning sycophancy → creativity improvement”, investigating whether selectively unlearning sycophantic behaviors can serve as a method for improving creativity. We will revise the Introduction to better reflect this perspective.

AuthorsOpenReview ↗

Response to Reviewer 2goa (Part 2)

3. Novelty of Unlearning Sycophancy and the Relationship to Existing Unlearning Methods (Weakness regarding contribution)

We appreciate the reviewer's observation regarding the relationship between our method and existing approaches such as NPO and SimNPO. We would like to clarify that our contribution is not merely an algorithmic improvement over existing unlearning methods. More fundamentally, we formulate sycophancy mitigation as a behavioral unlearning problem, extending the scope of machine unlearning from explicit knowledge removal to implicit conversational behaviors (to the best of our knowledge, our work is the first to explore machine unlearning as a dedicated approach to mitigating sycophantic behaviors in LLMs). To achieve the target of applying unlearning to mitigate sycophancy, we introduce a self-inoculating strategy to construct sycophancy-specific forget sets from the target model's own generations, together with a position-aware refinement of negative preference optimization to mitigate superficial prefix dominance during unlearning.

We would also like to clarify a point regarding the baselines: NPO and SimNPO are general-purpose unlearning methods, rather than methods originally developed for sycophancy mitigation. In our experiments, we adapt these existing algorithms to the sycophancy-specific forget and retain sets constructed using our framework.


We sincerely thank the reviewer again for the constructive feedback, helpful suggestions, and the time and effort devoted to evaluating our work. The review has helped us identify several areas for improvement, and we will carefully incorporate these improvements into the revised manuscript.

Reviewer_2goaOpenReview ↗

Thank you

Thank you for the response! I appreciate the additional information and maintain my positive assessment (3). I encourage the AC/SAC to consider accepting this paper.

AuthorsOpenReview ↗

Thank You for the Encouraging Feedback

We are glad that our responses and additional information were helpful. We truly appreciate your continued support and positive assessment!

Reviewer_smxSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether reducing sycophancy through behavioral unlearning can improve LLM creativity without creativity-specific training data. Starting from 400 prompts, the authors construct a forget set by eliciting sycophantic responses with explicit triggers and pairing these responses with the original, untriggered prompts. They introduce position-aware negative preference optimization (p-NPO), which down-weights early response tokens to reduce prefix-dominated unlearning. The paper reports reduced sycophancy and improved creativity scores, including LitBench win rates of 88.20% and 70.45% against the respective original models, together with positional ablations, additional factual-answer evaluations, and controlled unlearning targets.

Critical strengths
  1. The paper explores improving creative generation by modifying an undesirable conversational behavior rather than directly optimizing on creative-writing examples. The framework uses only 400 source prompts, and p-NPO adds a simple fixed positional weighting to a reference-free objective. Table 1 reports improvements over the original models on all three creativity evaluations for both backbones, while general utility is largely maintained.
  2. The paper includes several analyses that strengthen the behavioral interpretation. The positional ablations compare soft attenuation, zero weighting, inverse weighting, and attenuation at later positions. More importantly, the factual answer-sycophancy evaluation distinguishes inappropriate agreement from appropriate agreement: wrong-answer following decreases, while correct-answer following and clean accuracy are largely preserved.
Critical weaknesses
  1. Appendix B.2 combines the forgetting objective with a supervised loss on non-sycophantic self-generated responses. In contrast, the SFT and DPO baselines in Appendix B.3 use GPT-4o-generated targets, so comparisons with these baselines change both the optimization objective and the response-generation source. The Naive, Reversal, and Safety controls are useful, but they do not replace a retain-only SFT control or DPO trained on the same self-generated positive/negative responses. Consequently, it remains unclear how much of the improvement requires negative unlearning rather than positive counter-training on the retain set.
  2. The main algorithmic change from SimNPO is replacing uniform token averaging with fixed position-dependent weights. While this is a potentially useful practical refinement, the incremental methodological contribution appears limited, and the average sycophancy improvement over SimNPO is only 0.79 and 1.70 percentage points on the two backbones. Table 1 includes t-test markers, but the comparison and sampling unit are unclear, and the reliability of these incremental gains across independent training runs is not established. These issues directly affect confidence in the claimed advantage of p-NPO.
  3. Insufficient justification for the fixed prefix window. The method assumes that the first 15 tokens correspond to superficial sycophantic preambles, but the semantic validity of this boundary is not sufficiently established. Although Appendix F compares 10-, 15-, and 20-token windows, the analysis reports only sycophancy scores, leaving the sensitivity of creativity gains and utility preservation unclear.
Questions for authors
  1. Can you report retain-only SFT and DPO using the same self-generated retain/forget responses as p-NPO, with comparable tuning and training budgets? How much of the creativity and sycophancy improvement remains attributable to the negative unlearning term?
  2. What comparisons and sampling units underlie the significance markers in Table 1? Please provide paired confidence intervals and independent-training-run variability for p-NPO versus SimNPO. For creative writing, a direct paired comparison between these two methods would be more informative than comparing their separate win rates against the original model.
Reviewer_ksoDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Instead of teaching models to imitate creativity, we formulate sycophancy mitigation as a behavioral unlearning problem and explore whether unlearning undesirable sycophantic behaviors can provide an alternative pathway for improving LLM creativity. We construct sycophancy-specific forget sets through self-inoculation and introduce a lightweight positional refinement of negative preference optimization.

Critical strengths

NA

Critical weaknesses

Poor Motivation and presentation Poor Evaluation. Chosen creativity benchmarks are noisy and can be hillclimbed. No human eval

Questions for authors

NA

6 comments
AuthorsOpenReview ↗

Response to Reviewer ksoD (Part 1)

We thank the reviewer for the feedback. However, we respectfully believe that some aspects of the review do not fully reflect our benchmark selection, training protocol, and the existing validation of the evaluation framework. We would like to clarify these points below.

1. Concerns Regarding Creativity Benchmarks and Hill-Climbing

We would like to address two distinct aspects of this concern: the rationale behind our benchmark selection and why our experimental improvements should not be attributed to benchmark hill-climbing.

(1) Our creativity benchmarks are carefully selected based on established evaluation paradigms, rather than arbitrarily chosen.

Creativity is inherently multidimensional, and no single metric can fully capture it. We therefore deliberately employ three complementary evaluation paradigms to assess different aspects of creativity, rather than relying on a single potentially noisy benchmark.

  • Divergent Association Task (DAT) [1]. DAT is a well-established measure of divergent thinking, a fundamental component of creativity. Notably, semantic distance correlated at least as strongly with established creativity measures as those measures did with each other. This provides empirical and psychometric support for DAT as a brief, objective measure of divergent thinking. Furthermore, DAT has also been adopted in recent large-scale studies comparing divergent creativity between humans and LLMs, demonstrating its relevance to evaluating divergent creativity in language models [2].
  • Creative Writing v3 [3]. DAT primarily measures divergent thinking and does not assess the quality of extended creative writing. We therefore additionally adopt Creative Writing v3, a dedicated benchmark specifically designed to evaluate open-ended creative writing.
  • LitBench [4]. To further evaluate open-ended creative generation from the perspective of human writing preferences, we adopt LitBench. LitBench provides a large-scale pairwise story preference dataset consisting of 43,827 training pairs and a held-out test set of 2,480 pairs, curated from human-generated stories on Reddit. Its preference construction includes filtering for sufficient user engagement, meaningful preference margins, temporal exposure bias, and length bias.

These three evaluations therefore capture complementary aspects of creativity: semantic divergence, rubric-based creative writing quality, and preference-based evaluation. The use of complementary and independently developed evaluation paradigms is intended to reduce dependence on any single metric or evaluator. We believe that dismissing all three evaluations collectively as noisy does not adequately account for their distinct methodological foundations and validation procedures.

(2) Our creativity improvements are not obtained through benchmark hill-climbing.

More importantly, we would like to clarify a critical factual aspect of our experimental protocol: the proposed method is never optimized against any creativity benchmark or creativity evaluator. Our training procedure is exclusively designed for sycophancy-targeted behavioral unlearning:

  • We use only 400 sycophancy-related prompts from Perez et al.[5], consisting of 200 NLP and 200 POLI samples, to construct the forget set and retain set. Specifically, we query each target model with these prompts under explicit sycophantic triggers to generate responses for the forget set, and under corresponding non-sycophantic triggers to generate responses for the retain set, rather than from creativity-oriented training examples.
  • Neither DAT scores, Creative Writing v3 scores, nor LitBench reward-model predictions are used as training signals, optimization objectives, or feedback for updating the target models.

Therefore, our experimental procedure does not involve any benchmark-driven optimization, reward hacking, or hill-climbing against creativity evaluators. The improvements emerge from an independently motivated sycophancy-unlearning objective and are subsequently evaluated using multiple complementary creativity benchmarks.

AuthorsOpenReview ↗

Response to Reviewer ksoD (Part 2)

2. Clarification Regarding Human Evaluation

We would also like to clarify the reviewer's statement regarding the absence of human evaluation. The evaluation framework we adopt is not without human validation. Specifically, LitBench does not rely on an arbitrary zero-shot LLM judge. Its specialized reward models were trained on 43,827 human-preference story pairs and achieved approximately 78% agreement with human preferences on a held-out set of 2,480 preference comparisons, compared with 73% for the strongest evaluated off-the-shelf judge. LitBench also conducted an online human study to validate whether the reward-model rankings transfer to newly generated stories. We use the human-aligned evaluator following the original LitBench evaluation protocol.

Therefore, although we did not conduct a separate human-annotation study specifically on our generated pairs during the initial submission, the reported LitBench results are based on an evaluator that was trained and independently validated against human creative-writing preferences.

3. Clarification of the Research Motivation

Our motivation is based on the hypothesis that excessive sycophancy may bias models toward responses that conform to users' preferences, limiting the diversity and semantic associations in their outputs. We therefore investigate whether selectively unlearning this undesirable behavior can potentially improve creativity.

References

[1] Jay A. Olson, Johnny Nahas, Denis Chmoulevitch, Simon J. Cropper, and Margaret E. Webb. (2021). Naming unrelated words predicts creativity. Proceedings of the National Academy of Sciences, 118(25), e2022340118. https://doi.org/10.1073/pnas.2022340118

[2] Dawei Wang, Difang Huang, Haipeng Shen, and Brian Uzzi. (2026). A large-scale comparison of divergent creativity in humans and large language models. Nature Human Behaviour, 10, 531–540. https://doi.org/10.1038/s41562-025-02331-1

[3] Samuel J. Paech. (2025). EQ-Bench Creative Writing Benchmark v3. https://github.com/EQ-bench/creative-writing-bench

[4] Daniel Fein, Sebastian Russo, Violet Xiang, Kabir Jolly, Rafael Rafailov, and Nick Haber. (2026). LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7740–7755, Rabat, Morocco, March 2026. Association for Computational Linguistics. https://doi.org/10.18653/v1/2026.eacl-long.362

[5] Ethan Perez, Sam Ringer, Kamile Lukosiute, et al. (2023). Discovering Language Model Behaviors with Model-Written Evaluations. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 13387–13434. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.findings-acl.847

Reviewer_ksoDOpenReview ↗

Not Convinced

Please don't use AI to write rebuttals. I am interested in hearing your human opinion https://www.pangram.com/history/a207c8ed-2225-4f67-b707-b8eb370f9493?ucc=wfdQXjrFdnr&utm_source=pangram_share&utm_medium=copy_link

I think none of these benchmarks are good hallmark metrics of creativity. I am not convinced that DAT is relevant to LLM creativity and a benchmark created from reddit stories cannot be meaningful for creativity eval.

I said "It almost feels like hill climbing." not that its hill climbing.

Please do not justify bad evaluation. LITBench is created through distant supervision on reddit. It is not the kind of human evaluation I am asking. I am asking to evaluate your model outputs by expert humans who understand creativity

Creativity also isnt a single metric. It needs to be fine grained.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.