- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on using unlearning to mitigate sycophancy in LLMs and encourage their creativity. They introduce position-aware Negative Preference Optimization (p-NPO), which re-weights prefix tokens based on existing methods. The empirical results suggest the method mitigates sycophancy, improves creativity, and maintains general capabilities. Additional analysis is presented on the method and the performance correlation.
Critical strengths
- The idea of using unlearning to indirectly encourage creativity is novel and reasonable to me.
- The experiment design considered maintaining general capability while mitigating sycophancy and improving creativity.
- The experiment scale is comprehensive.
- The ablation study is solid.
Critical weaknesses
- I think the main idea of the connection between sycophancy and creativity is not solid. Though the authors explained, I don't fully agree with the anti-side of sycophancy being creativity. A model can be both sycophantic and creative, e.g., agreeing with the user's given belief but generating a much more detailed idea the user hasn't thought of. On the other hand, from my perspective, what should be unlearned is the answer/idea that the user feels is not novel, which seems to be a different target from mitigating sycophancy.
- Given this, I see the empirical improvement in creativity more like a side effect of these methods. I am not convinced that mitigating sycophancy and improving creativity have a causal relationship. The checkpoints analysis is on the same group of training trajectories where many common causes may exist, leading to a spurious correlation. I see the authors mention this in the limitations, but this point does negatively impact the novelty and excitement I feel about this paper.
- The experiments don't compare other existing creativity-promoting methods (prompting-based, training-based, etc). So we actually don't have a reference to understand, for example, how significant the improvement of +0.85 is on DAT.
- If viewing this paper mainly as a sycophancy mitigation method, it seems slightly incremental based on existing work (SimNPO), and the token weights are heuristics.
Questions for authors
- I encourage the authors to include creativity-improving methods in the comparison.
- Also, looking into creativity-improving methods can provide a more solid analysis in 6.2 (show correlation) and in 6.3 (identify the creativity gain vector direction).
- I encourage the authors to think and discuss more about whether this is a causal relationship or not. If this is not the case, they should consider down-weighting creativity as motivation in this project (e.g., in the introduction), and think of what the potential gaps should be; if this is the case, more rigorous statistical analysis to eliminate the possibility of a causal relationship should be done.
4 comments
Response to Reviewer 2goa (Part 2)
3. Novelty of Unlearning Sycophancy and the Relationship to Existing Unlearning Methods (Weakness regarding contribution)
We appreciate the reviewer's observation regarding the relationship between our method and existing approaches such as NPO and SimNPO. We would like to clarify that our contribution is not merely an algorithmic improvement over existing unlearning methods. More fundamentally, we formulate sycophancy mitigation as a behavioral unlearning problem, extending the scope of machine unlearning from explicit knowledge removal to implicit conversational behaviors (to the best of our knowledge, our work is the first to explore machine unlearning as a dedicated approach to mitigating sycophantic behaviors in LLMs). To achieve the target of applying unlearning to mitigate sycophancy, we introduce a self-inoculating strategy to construct sycophancy-specific forget sets from the target model's own generations, together with a position-aware refinement of negative preference optimization to mitigate superficial prefix dominance during unlearning.
We would also like to clarify a point regarding the baselines: NPO and SimNPO are general-purpose unlearning methods, rather than methods originally developed for sycophancy mitigation. In our experiments, we adapt these existing algorithms to the sycophancy-specific forget and retain sets constructed using our framework.
We sincerely thank the reviewer again for the constructive feedback, helpful suggestions, and the time and effort devoted to evaluating our work. The review has helped us identify several areas for improvement, and we will carefully incorporate these improvements into the revised manuscript.
Thank you
Thank you for the response! I appreciate the additional information and maintain my positive assessment (3). I encourage the AC/SAC to consider accepting this paper.
Thank You for the Encouraging Feedback
We are glad that our responses and additional information were helpful. We truly appreciate your continued support and positive assessment!
Response to Reviewer 2goa (Part 1)
We sincerely thank the reviewer for the positive assessment of our work, particularly regarding the novelty of using unlearning to improve creativity. We also appreciate the thoughtful questions, which help us further clarify the contributions and implications of our work. We address the concerns point by point below.
1. Creativity Improvement and Comparisons with Existing Creativity-Enhancing Methods (Weakness and Question regarding creativity-improving baselines)
We appreciate the suggestion to compare our method with existing creativity-enhancing approaches. We would like to clarify that comparisons with creativity-enhancing baselines are already included in our original submission (see Figure 2, lines 262–265 and lines 365–372), including the prompt-based method (OPRO), inference-time intervention (Steering), as well as training-based methods (SFT and DPO). The results show that p-NPO outperforms existing creativity-enhancing baselines.
To provide a clearer comparison with existing creativity-enhancing methods, we further rank all five methods across the three creativity benchmarks based on their respective scores. Rank 1 indicates the best performance:
The ranking results show that p-NPO achieves the strongest overall performance among the five methods, ranking first in five out of six comparisons. In particular, p-NPO ranks first on both Creative Writing v3 and LitBench across both backbones, demonstrating consistent advantages. On DAT, p-NPO also achieves the highest score on Llama, while remaining highly competitive on Qwen, with its score closely matching those of the top-ranked DPO and SFT baselines. These results further support the effectiveness of p-NPO across different dimensions of creativity.
2. The Relationship Between Sycophancy Mitigation and Creativity Improvement (Weakness and Question regarding causal relationship between creativity and sycophancy)
We thank the reviewer's thoughtful concern regarding the causal relationship between sycophancy mitigation and creativity improvement. As acknowledged in the Limitations section, we agree that our current experiments do not establish a definitive causal relationship between the two model properties.
Establishing a strict causal relationship between two high-dimensional behavioral properties of an LLM is challenging. Training inevitably modifies model parameters and potentially affects multiple models capabilities, such as the reasoning ability. Consequently, isolating the effect of sycophancy reduction from all other changes in the model is difficult. We therefore avoid claiming a universal causal law that reducing sycophancy must increase creativity. Instead, our study investigates an intervention: what happens to creativity when we selectively unlearn sycophantic behaviors. In the meanwhile, Discussion of Section 6.3 and the related experiments provide further evidence to help distinguish the effects of sycophancy-targeted unlearning from generic parameter updates induced by unlearning. More broadly, we believe this investigation has value even without establishing a mathematically definitive causal relationship. Our findings suggest that selectively unlearning undesirable behavioral tendencies offer an alternative pathway for improving existing model capabilities, without directly supervising the model on the target capability.
We acknowledge that the presentation in our original Introduction could have been clearer. In particular, the emphasis on the “sycophancy → creativity” relationship may have inadvertently suggested that we claim a direct causal relationship between these two properties. This is not our intended claim. Rather, our research focuses on “unlearning sycophancy → creativity improvement”, investigating whether selectively unlearning sycophantic behaviors can serve as a method for improving creativity. We will revise the Introduction to better reflect this perspective.