- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This study investigates the role of negative learning term during LLM finetuning dynamics. The tool for investigation is to check the alignment between gradients during tuning, which is retrospectively computed on the set of generation samples of the final checkpoints. The effects from positive and negative learnings could play different roles depending on the training phase, task distribution gaps, finetuning methods (and possibly models). In particular, the negative learning terms may work not only for boosting exploration but also for clear convergence to a target. An experiment shows that a modification of weighting terms could improve task performances.
Critical strengths
- Some analysis is interesting. The negative learning term can work as an exploration regularizer and a helper for convergence to a specifc target, and can switch the roles even in a short training trajectory.
Critical weaknesses
- Some logic and explanation are not clear. Especially, the details about the experiments (and their designs) are too brief to convince the generalization of the findings.
- The design and results of controlled optimization methods may not be well-connected from the experimental discussions while it somehow shows improvements.
Questions for authors
Although the variation of the experiments is great, it is also confusing for clear conclusions because more than one factors (e.g., model, task, finetuning method, ...) are changed simultaneously. In general, I'm not yet confident about the existence of clear and general tendency of findings. Each finding could be a special case depending on a special (and unidentified) combination. For example, when extending discussions from PPO to GRPO, the model is also switched to Gemma from Qwen or Pythia in Figure 5. Qwen measures OOD but does not ID in Figure 3.
It seems that the number of training steps could change some observations. For example, even if the paper claims that the negative effect (delta) is "always" negative with a config, it could be increasing after more training steps. Can we avoid such a suspect about the conclusions?
The explanation and discussion of Section 5 seem very fast and does not provide reasonable explanations about the motivation of the design and the (large) improvement by it. Without the deep understanding, we could suspect that the difference is just due to the lack of sufficient hyperparameter tuning of baselines.
For example,
Motivated by this finding, we introduce a family of controlled optimization methods that adjust the relative strength of negative learning throughout training. ...
This role exchange suggests placing greater relative emphasis on the component driving target shaping at each stage, while adjusting the regularization-like influence of the other component.
Positive learning moves the model toward the final endpoints, suggesting that positive-advantage responses provide useful exploitation targets, while negative learning counteracts this direction.
Could you clarify these logics? The connection between/among the preceding results, the logic, and the method design seems not so obvious and logical.
It would be nice to show the trajectory of task performances. In the current manuscript, except for Figure 4, we cannot know even whether the endpoint model is better than the initial one.
15 comments
Response to Reviewer WDvw (2/3)
Q4 (Q2). It seems that the number of training steps could change some observations. For example, even if the paper claims that the negative effect (delta) is "always" negative with a config, it could be increasing after more training steps. Can we avoid such a suspect about the conclusions?
A4. Thank you for this question. We would first note that, when describing our observations, we do not use absolute terms such as "always" (e.g., we say "remains negative" in Line 375), and all our observations and conclusions are stated under our experimental settings. However, this does not affect our central claim that the role of negative learning is far more complex than commonly assumed: rather than uniformly acting as target shaping, it plays different roles across methods and training phases.
Moreover, all our experiments follow commonly used training settings, e.g., two epochs for DPO following Tunstall et al. (2023), ten epochs for NPO following OpenUnlearning, and the full PPO and GRPO schedules in Appendix B, with endpoints generated by the final checkpoints of these runs. Training much longer or shorter than these settings would instead make the results less reliable, e.g., due to over-optimization or undertraining.
This is because our goal is to understand how negative learning behaves in the most common practical settings, which, to our knowledge, has not been carefully examined before. Relatively uncommon or extreme settings, such as training for an excessively long time or with poorly chosen hyperparameters, are beyond the scope of this work. We will clarify this scope in the revision.
Nevertheless, we also provide results under longer training for your reference. For example, we train Qwen3-1.7B DPO with the same configuration but twice as many epochs (480 steps), and probe it with endpoints generated by the step-480 checkpoint. Raw endpoint-probing scores:
| Step | NEG | POS |
|---|---|---|
| 100 | −9,818 | 21,263 |
| 200 | 9,493 | −8,757 |
| 300 | 12,664 | −21,784 |
| 400 | 13,419 | −27,341 |
The role exchange between positive and negative learning is preserved (between steps 100 and 200), consistent with Figure 3c. Positive learning drives endpoint fitting early while negative learning opposes it, and the two exchange roles later, even under twice the training length. This suggests that our conclusions hold to a certain extent beyond the common training length.
Q5 (W2 & Q3). The design and results of controlled optimization methods may not be well-connected from the experimental discussions while it somehow shows improvements. The explanation and discussion of Section 5 seem very fast and do not provide reasonable explanations about the motivation of the design and the (large) improvement by it. Could you clarify these logics? The connection between/among the preceding results, the logic, and the method design seems not so obvious and logical.
A5. Thank you for this question. The controlled methods follow directly from our analysis. Since the roles of positive and negative learning change across methods and training phases, our key insight is that coordinating their interaction more appropriately can further improve performance, rather than keeping their balance fixed as in standard objectives.
Specifically, in DPO, positive and negative learning exchange their roles of target shaping and regularization during training (Figures 3–4), while standard DPO weights them equally throughout, so cDPO gradually shifts the emphasis from positive to negative learning. In PPO and GRPO, negative learning consistently counteracts endpoint-aligned positive learning, acting as exploration (Figure 5), so cPPO and cGRPO adjust its strength to balance exploration and exploitation.
More broadly, explicitly coordinating positive and negative learning is itself a contribution of our work, and the proposed schedules are one way to implement it. We expect our findings to motivate better coordination strategies in future work. We will expand Section 5 to make this logic explicit in the revision.
Response to Reviewer WDvw (3/3)
Q6 (Q3). Without the deep understanding, we could suspect that the difference is just due to the lack of sufficient hyperparameter tuning of baselines.
A6. We believe that this suspicion is not supported by our description and experimental results.
First, our baselines follow commonly used settings, which we further tune when needed (e.g., DPO; Section 4.2), since our understanding is meaningful only in realistic, well-tuned settings. The resulting baselines are strong, e.g., DPO achieves 76.71% and 72.14% win rates against the supervised fine-tuning models on Pythia-2.8B and Qwen3-1.7B.
Second, the comparison is fair: the controlled methods keep all hyperparameters of the corresponding baselines unchanged, and only tune the newly introduced scheduling parameters. Moreover, the improvements are robust to these scheduling parameters. To verify that our conclusions are general, we conduct a robustness analysis over a wide range of schedules (Figures 19–21), most of which outperform the baselines, e.g., cDPO achieves 83.56–87.91% win rates on Qwen3-1.7B, compared with 72.14% for DPO.
Q7 (Q4). It would be nice to show the trajectory of task performances. In the current manuscript, except for Figure 4, we cannot know even whether the endpoint model is better than the initial one.
A7. Thank you for this suggestion. Following your suggestion, we report the AlpacaEval win rates (%) against the initial supervised fine-tuning models along the DPO and PPO trajectories analyzed in the main text (Figures 3 and 5), averaged over three seeds, where 50% means no change from the initial model:
| DPO, Pythia-2.8B | 1000 | 2000 | 3000 | 4000 | 5000 | 6000 | 7000 | 7644 |
|---|---|---|---|---|---|---|---|---|
| Win rate | 66.83 | 71.86 | 73.35 | 73.79 | 75.12 | 76.71 | 75.87 | 75.22 |
| DPO, Qwen3-1.7B | 30 | 60 | 90 | 120 | 150 | 180 | 210 | 240 |
|---|---|---|---|---|---|---|---|---|
| Win rate | 68.27 | 62.12 | 62.45 | 63.70 | 67.35 | 69.38 | 71.13 | 72.14 |
| PPO, Pythia-2.8B | 400 | 800 | 1200 | 1600 | 2000 | 2400 |
|---|---|---|---|---|---|---|
| Win rate | 53.60 | 54.60 | 54.53 | 55.96 | 55.34 | 54.97 |
| PPO, Qwen3-1.7B | 40 | 80 | 120 | 160 | 200 | 240 | 280 | 320 | 360 | 400 |
|---|---|---|---|---|---|---|---|---|---|---|
| Win rate | 53.11 | 56.58 | 56.65 | 57.08 | 55.40 | 56.71 | 55.16 | 57.10 | 54.10 | 56.48 |
The win rates remain above 50% at all checkpoints, confirming that the models along these trajectories, including the endpoint models, improve over their initial models. The gains of PPO are more modest than those of DPO. Since our goal is to understand the mechanism of negative learning rather than to maximize the performance of PPO, we follow its commonly used settings without additional engineering. We will add these trajectories to the revision.
Response to Reviewer WDvw (1/3)
We sincerely thank the reviewer for the thoughtful feedback and for recognizing our analysis of negative learning, especially its dual roles as an exploration regularizer and a helper for convergence to specific targets, as well as its role switching within a short training trajectory. We fully understand your concerns and are excited to discuss them further with you. We hope the responses below provide helpful clarification, and we look forward to continuing the discussion.
A1. Thank you for this comment. Due to space limitations and the large number of experiments, we consolidate the experimental details in the appendix. Most settings follow standard configurations in the literature, e.g., the recipe of Tunstall et al. (2023) for DPO, TRL for PPO, and verl for GRPO. Specifically, Appendix A describes how each method is decomposed into positive and negative learning, Appendix B reports the full experimental configurations, Appendix C provides additional experiments, and Appendix D analyzes the probing approximation. We are happy to provide any further details.
A2. Thank you for raising this concern. We do not claim a single pattern that holds across all settings. Our central claim is that the role of negative learning is far more complex than commonly assumed: rather than uniformly acting as target shaping, it plays different roles across methods and training phases.
We carefully scope our conclusions in the paper, e.g., "for the out-of-distribution endpoints studied" (Line 412) and "in the on-policy PPO and GRPO settings studied" (Line 424). Within these scopes, our findings are consistent across models and tasks: DPO shows the same phase-dependent role exchange on Pythia-2.8B and Qwen3-1.7B (Figures 3, 9–10), PPO shows opposing alignments on both models (Figures 5, 11–12), and GRPO shows opposing negative learning across three model–task combinations (Figures 13–15). Moreover, these findings motivate controlled methods that consistently improve over baselines (Figure 7), suggesting that they are meaningful beyond the probed settings.
The models differ across methods because we use mature, widely adopted models for each method and task (e.g., function calling requires models with tool-calling capability). These settings come with established configurations, which provide reliable starting points for hyperparameter selection. We use the default configurations where they are available (e.g., NPO follows the OpenUnlearning defaults), and otherwise tune from these starting points for stable training (e.g., DPO; see Appendix B). This choice is a practical necessity rather than a selection for favorable results.
A3. Thank you for pointing this out. Following your suggestion, we additionally probe Qwen3-1.7B DPO with in-distribution endpoints, constructed from 500 held-out UltraFeedback prompts paired with greedy responses from the final checkpoint, using the same setup as Figure 3c. We report raw endpoint-probing scores, as in Figure 3c.
Consistent with Pythia-2.8B (Figure 3a), negative learning on Qwen3-1.7B remains opposed to in-distribution endpoints throughout training, while it becomes aligned with out-of-distribution endpoints in the late phase (Figure 3c). The in- vs. out-of-distribution contrast therefore holds across both models. We will add these results to the revision.