acceptodds
Under review as a conference paper at ICLR 2027

Understanding Negative Learning in Large Language Model Fine-tuning

Abstract

Negative learning has never been more important than it is today, particularly in large language model (LLM) fine-tuning, where it underpins machine unlearning, preference- and reward-based optimization, and exploratory data generation. Yet its functional role remains a subject of ongoing debate, with existing work variously characterizing it as forgetting, regularization, or target shaping. However, analytical support focuses primarily on the single-update scale, while empirical results at the trajectory scale are typically obtained under negative-only training and may not extend to composite objectives in which it interacts with other components. To develop a deeper understanding, we introduce a gradient-based endpoint probing framework that can examine negative updates separately throughout training. Our analysis reveals that negative learning plays different roles across algorithms and training phases. For example, in off-policy DPO, it acts more like a regularizer early in training and shows evidence of target shaping only toward out-of-distribution endpoints and only in later phases. In on-policy PPO and GRPO, by contrast, negative updates offset endpoint-aligned positive learning without stable target shaping, consistent with a role in sustaining exploration. Guided by these findings, we develop a family of stage-aware methods that modulate the strength of negative updates throughout training, improving fine-tuning performance across tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Unlearning, Negative Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WDvwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study investigates the role of negative learning term during LLM finetuning dynamics. The tool for investigation is to check the alignment between gradients during tuning, which is retrospectively computed on the set of generation samples of the final checkpoints. The effects from positive and negative learnings could play different roles depending on the training phase, task distribution gaps, finetuning methods (and possibly models). In particular, the negative learning terms may work not only for boosting exploration but also for clear convergence to a target. An experiment shows that a modification of weighting terms could improve task performances.

Critical strengths
  • Some analysis is interesting. The negative learning term can work as an exploration regularizer and a helper for convergence to a specifc target, and can switch the roles even in a short training trajectory.
Critical weaknesses
  • Some logic and explanation are not clear. Especially, the details about the experiments (and their designs) are too brief to convince the generalization of the findings.
  • The design and results of controlled optimization methods may not be well-connected from the experimental discussions while it somehow shows improvements.
Questions for authors

Although the variation of the experiments is great, it is also confusing for clear conclusions because more than one factors (e.g., model, task, finetuning method, ...) are changed simultaneously. In general, I'm not yet confident about the existence of clear and general tendency of findings. Each finding could be a special case depending on a special (and unidentified) combination. For example, when extending discussions from PPO to GRPO, the model is also switched to Gemma from Qwen or Pythia in Figure 5. Qwen measures OOD but does not ID in Figure 3.


It seems that the number of training steps could change some observations. For example, even if the paper claims that the negative effect (delta) is "always" negative with a config, it could be increasing after more training steps. Can we avoid such a suspect about the conclusions?


The explanation and discussion of Section 5 seem very fast and does not provide reasonable explanations about the motivation of the design and the (large) improvement by it. Without the deep understanding, we could suspect that the difference is just due to the lack of sufficient hyperparameter tuning of baselines.

For example,

Motivated by this finding, we introduce a family of controlled optimization methods that adjust the relative strength of negative learning throughout training. ...
This role exchange suggests placing greater relative emphasis on the component driving target shaping at each stage, while adjusting the regularization-like influence of the other component.
Positive learning moves the model toward the final endpoints, suggesting that positive-advantage responses provide useful exploitation targets, while negative learning counteracts this direction.

Could you clarify these logics? The connection between/among the preceding results, the logic, and the method design seems not so obvious and logical.


It would be nice to show the trajectory of task performances. In the current manuscript, except for Figure 4, we cannot know even whether the endpoint model is better than the initial one.

15 comments
AuthorsOpenReview ↗

Response to Reviewer WDvw (1/3)

We sincerely thank the reviewer for the thoughtful feedback and for recognizing our analysis of negative learning, especially its dual roles as an exploration regularizer and a helper for convergence to specific targets, as well as its role switching within a short training trajectory. We fully understand your concerns and are excited to discuss them further with you. We hope the responses below provide helpful clarification, and we look forward to continuing the discussion.

Q1 (W1). Some logic and explanation are not clear. Especially, the details about the experiments (and their designs) are too brief to convince the generalization of the findings.

A1. Thank you for this comment. Due to space limitations and the large number of experiments, we consolidate the experimental details in the appendix. Most settings follow standard configurations in the literature, e.g., the recipe of Tunstall et al. (2023) for DPO, TRL for PPO, and verl for GRPO. Specifically, Appendix A describes how each method is decomposed into positive and negative learning, Appendix B reports the full experimental configurations, Appendix C provides additional experiments, and Appendix D analyzes the probing approximation. We are happy to provide any further details.

Q2 (Q1). Although the variation of the experiments is great, it is also confusing for clear conclusions because more than one factor (e.g., model, task, finetuning method, ...) are changed simultaneously. In general, I'm not yet confident about the existence of clear and general tendency of findings. Each finding could be a special case depending on a special (and unidentified) combination. For example, when extending discussions from PPO to GRPO, the model is also switched to Gemma from Qwen or Pythia in Figure 5.

A2. Thank you for raising this concern. We do not claim a single pattern that holds across all settings. Our central claim is that the role of negative learning is far more complex than commonly assumed: rather than uniformly acting as target shaping, it plays different roles across methods and training phases.

We carefully scope our conclusions in the paper, e.g., "for the out-of-distribution endpoints studied" (Line 412) and "in the on-policy PPO and GRPO settings studied" (Line 424). Within these scopes, our findings are consistent across models and tasks: DPO shows the same phase-dependent role exchange on Pythia-2.8B and Qwen3-1.7B (Figures 3, 9–10), PPO shows opposing alignments on both models (Figures 5, 11–12), and GRPO shows opposing negative learning across three model–task combinations (Figures 13–15). Moreover, these findings motivate controlled methods that consistently improve over baselines (Figure 7), suggesting that they are meaningful beyond the probed settings.

The models differ across methods because we use mature, widely adopted models for each method and task (e.g., function calling requires models with tool-calling capability). These settings come with established configurations, which provide reliable starting points for hyperparameter selection. We use the default configurations where they are available (e.g., NPO follows the OpenUnlearning defaults), and otherwise tune from these starting points for stable training (e.g., DPO; see Appendix B). This choice is a practical necessity rather than a selection for favorable results.

Q3 (Q1). Qwen measures OOD but does not ID in Figure 3.

A3. Thank you for pointing this out. Following your suggestion, we additionally probe Qwen3-1.7B DPO with in-distribution endpoints, constructed from 500 held-out UltraFeedback prompts paired with greedy responses from the final checkpoint, using the same setup as Figure 3c. We report raw endpoint-probing scores, as in Figure 3c.

StepNEGPOSTOT
30−22,50629,1736,667
60−9,01313,7454,732
90−4,9238,4783,555
120−4,3948,5584,164
150−3,7525,1581,406
180−3,7125,7632,051
210−4,6406,1251,486
240−4,1675,2331,066

Consistent with Pythia-2.8B (Figure 3a), negative learning on Qwen3-1.7B remains opposed to in-distribution endpoints throughout training, while it becomes aligned with out-of-distribution endpoints in the late phase (Figure 3c). The in- vs. out-of-distribution contrast therefore holds across both models. We will add these results to the revision.

AuthorsOpenReview ↗

Response to Reviewer WDvw (2/3)

Q4 (Q2). It seems that the number of training steps could change some observations. For example, even if the paper claims that the negative effect (delta) is "always" negative with a config, it could be increasing after more training steps. Can we avoid such a suspect about the conclusions?

A4. Thank you for this question. We would first note that, when describing our observations, we do not use absolute terms such as "always" (e.g., we say "remains negative" in Line 375), and all our observations and conclusions are stated under our experimental settings. However, this does not affect our central claim that the role of negative learning is far more complex than commonly assumed: rather than uniformly acting as target shaping, it plays different roles across methods and training phases.

Moreover, all our experiments follow commonly used training settings, e.g., two epochs for DPO following Tunstall et al. (2023), ten epochs for NPO following OpenUnlearning, and the full PPO and GRPO schedules in Appendix B, with endpoints generated by the final checkpoints of these runs. Training much longer or shorter than these settings would instead make the results less reliable, e.g., due to over-optimization or undertraining.

This is because our goal is to understand how negative learning behaves in the most common practical settings, which, to our knowledge, has not been carefully examined before. Relatively uncommon or extreme settings, such as training for an excessively long time or with poorly chosen hyperparameters, are beyond the scope of this work. We will clarify this scope in the revision.

Nevertheless, we also provide results under longer training for your reference. For example, we train Qwen3-1.7B DPO with the same configuration but twice as many epochs (480 steps), and probe it with endpoints generated by the step-480 checkpoint. Raw endpoint-probing scores:

StepNEGPOS
100−9,81821,263
2009,493−8,757
30012,664−21,784
40013,419−27,341

The role exchange between positive and negative learning is preserved (between steps 100 and 200), consistent with Figure 3c. Positive learning drives endpoint fitting early while negative learning opposes it, and the two exchange roles later, even under twice the training length. This suggests that our conclusions hold to a certain extent beyond the common training length.

Q5 (W2 & Q3). The design and results of controlled optimization methods may not be well-connected from the experimental discussions while it somehow shows improvements. The explanation and discussion of Section 5 seem very fast and do not provide reasonable explanations about the motivation of the design and the (large) improvement by it. Could you clarify these logics? The connection between/among the preceding results, the logic, and the method design seems not so obvious and logical.

A5. Thank you for this question. The controlled methods follow directly from our analysis. Since the roles of positive and negative learning change across methods and training phases, our key insight is that coordinating their interaction more appropriately can further improve performance, rather than keeping their balance fixed as in standard objectives.

Specifically, in DPO, positive and negative learning exchange their roles of target shaping and regularization during training (Figures 3–4), while standard DPO weights them equally throughout, so cDPO gradually shifts the emphasis from positive to negative learning. In PPO and GRPO, negative learning consistently counteracts endpoint-aligned positive learning, acting as exploration (Figure 5), so cPPO and cGRPO adjust its strength to balance exploration and exploitation.

More broadly, explicitly coordinating positive and negative learning is itself a contribution of our work, and the proposed schedules are one way to implement it. We expect our findings to motivate better coordination strategies in future work. We will expand Section 5 to make this logic explicit in the revision.

AuthorsOpenReview ↗

Response to Reviewer WDvw (3/3)

Q6 (Q3). Without the deep understanding, we could suspect that the difference is just due to the lack of sufficient hyperparameter tuning of baselines.

A6. We believe that this suspicion is not supported by our description and experimental results.

First, our baselines follow commonly used settings, which we further tune when needed (e.g., DPO; Section 4.2), since our understanding is meaningful only in realistic, well-tuned settings. The resulting baselines are strong, e.g., DPO achieves 76.71% and 72.14% win rates against the supervised fine-tuning models on Pythia-2.8B and Qwen3-1.7B.

Second, the comparison is fair: the controlled methods keep all hyperparameters of the corresponding baselines unchanged, and only tune the newly introduced scheduling parameters. Moreover, the improvements are robust to these scheduling parameters. To verify that our conclusions are general, we conduct a robustness analysis over a wide range of schedules (Figures 19–21), most of which outperform the baselines, e.g., cDPO achieves 83.56–87.91% win rates on Qwen3-1.7B, compared with 72.14% for DPO.

Q7 (Q4). It would be nice to show the trajectory of task performances. In the current manuscript, except for Figure 4, we cannot know even whether the endpoint model is better than the initial one.

A7. Thank you for this suggestion. Following your suggestion, we report the AlpacaEval win rates (%) against the initial supervised fine-tuning models along the DPO and PPO trajectories analyzed in the main text (Figures 3 and 5), averaged over three seeds, where 50% means no change from the initial model:

DPO, Pythia-2.8B10002000300040005000600070007644
Win rate66.8371.8673.3573.7975.1276.7175.8775.22
DPO, Qwen3-1.7B306090120150180210240
Win rate68.2762.1262.4563.7067.3569.3871.1372.14
PPO, Pythia-2.8B4008001200160020002400
Win rate53.6054.6054.5355.9655.3454.97
PPO, Qwen3-1.7B4080120160200240280320360400
Win rate53.1156.5856.6557.0855.4056.7155.1657.1054.1056.48

The win rates remain above 50% at all checkpoints, confirming that the models along these trajectories, including the endpoint models, improve over their initial models. The gains of PPO are more modest than those of DPO. Since our goal is to understand the mechanism of negative learning rather than to maximize the performance of PPO, we follow its commonly used settings without additional engineering. We will add these trajectories to the revision.

Reviewer_e2gwOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the role of negative learning in large language model fine-tuning. The authors introduce endpoint probing, which uses responses generated by a final checkpoint as empirical endpoints and measures the alignment of positive, negative, and composite objective gradients with those endpoints at earlier checkpoints. Across negative-only NPO, off-policy DPO, and on-policy PPO and GRPO, the paper reports different roles for negative learning: phase-dependent target shaping for out-of-distribution DPO endpoints, a regularization-like effect in some DPO phases, and an exploration-related opposing direction in PPO and GRPO. Based on these observations, the authors propose stage-aware controlled variants of DPO, PPO, and GRPO that change the strength of negative learning during training. Experiments across preference alignment, reasoning, and tool-use tasks show improvements over the corresponding baselines.

Critical strengths

S1. The paper addresses an important and under-explored question: whether negative learning has the same functional role across negative-only, off-policy preference, and on-policy reward objectives.

S2. Endpoint probing is a simple and flexible way to decompose the contributions of positive and negative objective components at the same model state.

S3. The empirical study covers multiple model families, objectives, endpoint distributions, and downstream settings.

Critical weaknesses

W1. The conclusions may depend on using the final model's own answers as the reference. The authors first collect answers from the final checkpoint, then check whether earlier gradients make those answers more likely. This tells us whether an update moves toward the model's eventual answers. However, it is less clear whether the reported role of negative learning would remain the same with a different set of reference answers. For example, would negative learning still appear to guide the model toward particular answers if those answers came from a teacher model or a separate training run? The authors could repeat the main analyses with these independent references and compare the results. Checking high- and low-quality reference answers separately would also help clarify what movement toward the endpoints means.

W2. The gradient analyzed in the paper may point in a different direction from the update actually used in training. The probing score uses the raw loss gradient, but AdamW changes the update direction through momentum and parameter-wise scaling. A gradient that appears to move toward the reference answers may therefore have a different effect after the optimizer processes it. For PPO and GRPO, the probe also leaves out the KL term that is included in training. Figure 1 checks raw-gradient steps, so it does not resolve these differences. The authors could compare the predicted endpoint-loss changes with the changes after actual optimizer steps, report how much the KL term contributes, and check whether the main conclusions about positive and negative learning still hold.

Questions for authors
  1. Can the authors evaluate endpoint probing against endpoints that are independent of the probed training trajectory, such as teacher-model responses, a separately trained final checkpoint, or endpoints sampled under multiple decoding temperatures? It would also be useful to report endpoint quality and test whether the main sign patterns persist for different endpoint subsets.
  2. Can the probing score be computed using the actual AdamW or policy-optimizer update direction rather than only the raw objective gradient? For PPO and GRPO, please clarify whether the KL contribution is included in the full update and report how large its gradient contribution is relative to the positive and negative surrogate components.
  3. The appendix gives sufficient conditions for sparse checkpoint scores to preserve their sign, but it does not establish that the checkpoint intervals in the experiments satisfy those conditions. Please report the largest parameter displacement or a denser-checkpoint analysis for representative settings, especially the 400-step PPO interval and the 10-step NPO interval.
  4. The paper interprets opposing endpoint alignment in PPO and GRPO as evidence that negative learning sustains exploration, while also noting that endpoint alignment alone does not establish discovery of better alternatives. Could the authors provide a direct exploration-related diagnostic, such as response diversity, novelty relative to the final endpoints, reward distribution, or coverage of alternative high-reward responses?
  5. For NPO, the probing study uses only the TOFU forget05 split and does not report a retention or general-utility metric. Since the discussion connects the results to unlearning, please clarify whether the goal is only to study negative-learning dynamics or also to support an unlearning claim; in the latter case, a retention evaluation would be useful.
  6. Please make the distinction between “endpoint alignment,” “target shaping,” “regularization,” and “exploration” explicit throughout the paper.
10 comments
AuthorsOpenReview ↗

Response to Reviewer e2gw (1/4)

We sincerely thank the reviewer for the thoughtful and encouraging feedback. Your positive assessment means a great deal to us, especially in such a highly competitive review cycle. We are also deeply grateful for your recognition of the importance of the question we study, the simplicity and flexibility of endpoint probing, and the breadth of our empirical study. We are glad to address your questions below and look forward to further discussion.

Q1 (W1 & Q1). The conclusions may depend on using the final model's own answers as the reference. It is less clear whether the reported role of negative learning would remain the same with a different set of reference answers. Can the authors evaluate endpoint probing against endpoints that are independent of the probed training trajectory, such as teacher-model responses, a separately trained final checkpoint, or endpoints sampled under multiple decoding temperatures? It would also be useful to report endpoint quality and test whether the main sign patterns persist for high- and low-quality endpoint subsets.

A1. We fully agree with your point and thank you for this suggestion. In fact, we have partially considered this issue and examined different endpoints within the same method, including in-distribution vs. out-of-distribution endpoints for DPO (Figure 3), sampled endpoints at temperatures 0.2 and 0.7 for DPO (Figure 16), and held-out vs. training-prompt endpoints for on-policy distillation (Figure 18). Our findings remain stable across these settings, except for the in-distribution vs. out-of-distribution contrast in DPO, whose rationale we discuss in Section 4.2 and which motivates our focus on out-of-distribution endpoints in subsequent analyses.

You further suggest endpoints that are independent of the probed trajectory, such as teacher-model responses and responses from a separately trained final checkpoint, as well as a separate analysis of high- and low-quality endpoints, which we find very interesting. Following your suggestion, we have conducted these experiments. We report endpoint-probing scores across checkpoints.

Teacher-model endpoints (OPD). Since on-policy distillation naturally involves a teacher, we replace the endpoints in Figure 18 (responses sampled from the final student) with responses sampled from the teacher (Qwen3-1.7B), while keeping everything else unchanged. Raw endpoint-probing scores on held-out prompts:

StepNEG (teacher)POS (teacher)TOT (teacher)NEG (student)POS (student)TOT (student)
0−9,1783,281−5,8794,1143,0997,223
10−5,3252,959−2,3621,3052,4213,737
30−1,1882,5141,3231,5252,1163,639
501511,5091,6615597971,356
70−518912394218402619
90−763792314241246
110−194636443125103228
150722603328620106
18013432145656−2134
210−1,204673−529356−25331

With teacher endpoints, positive learning stays aligned with the endpoints, while negative learning mostly opposes them. This matches the in-distribution case of DPO (Figure 3a). Intuitively, the teacher's responses serve as a direct training signal, similar to in-distribution endpoints in DPO, whereas the student's final responses reflect what the student generalizes to, similar to out-of-distribution endpoints (Appendix C.3).

Endpoints from a separately trained run (DPO). We train an independent run with a different seed, and probe its checkpoints using endpoints generated by the final checkpoint of the main run, so that the endpoints are independent of the probed trajectory. We compare the results with those obtained by probing the main run against its own endpoints (default), using out-of-distribution endpoints on Qwen3-1.7B:

StepNEG (default)POS (default)TOT (default)NEG (separate)POS (separate)TOT (separate)
30−19,48028,4799,038−24,53632,9148,378
60−4,26810,3246,056−6,11612,5296,413
90−4435,3454,895−1,0535,2294,176
120−656,4346,368−6946,5925,898
1507,253−5,9071,3446,223−4,1352,088
1806,096−4,0362,0635,629−3,8471,782
2106,372−5,4679095,957−4,8271,130
2406,426−5,4351,0025,279−4,641638
AuthorsOpenReview ↗

Response to Reviewer e2gw (2/4)

A1 (continued).

High- and low-quality endpoints (DPO). Using Qwen3-1.7B, we divide the default endpoints into winning and losing responses against the supervised fine-tuning baseline, as in our win-rate evaluation, and randomly sample 50 prompts from each group (seed 42, fixed before probing) so that both subsets have equal size. The two subsets are comparable in number of turns (2.3 vs. 2.6) and prompt length (median 80 vs. 89 tokens), so differences between them are not simply driven by prompt characteristics. The results are reported below:

StepNEG (high)POS (high)TOT (high)NEG (low)POS (low)TOT (low)
30−25,21933,3108,090−22,85130,3077,456
60−7,70513,6435,938−6,72011,9405,220
90−3,1647,2454,081−3,9077,4263,518
120−2,9758,7655,790−3,7458,9575,213
1502,980−3,130−1501,111−1,357−246
1802,836−1,1921,6441,399−3331,065
2102,844−2,0847601,050−1,01931
2403,145−2,7583871,719−1,779−60

Overall, these results are consistent with the conclusions in the paper. For OPD, teacher and student endpoints show a contrast similar to the in- vs. out-of-distribution contrast in DPO, which we will further discuss in the appendix. For endpoints from a separately trained run and for high- and low-quality endpoints, the sign patterns remain stable: negative learning first opposes the endpoints and then turns positive between steps 120 and 150, while positive learning shows the opposite trend.

More broadly, every conclusion holds within a certain scope. Our endpoints approximate the real target, and the real target itself can take different forms, naturally leading to different observations. We therefore carefully scope our conclusions in the paper, e.g., "for the out-of-distribution endpoints studied" (Line 412) and "in the on-policy PPO and GRPO settings studied" (Line 424). Nevertheless, our central claim holds across all these settings: the role of negative learning is far more complex than commonly assumed. To our knowledge, this has not been carefully examined in prior work.

Q2 (W2 & Q2). The gradient analyzed in the paper may point in a different direction from the update actually used in training. The probing score uses the raw loss gradient, but AdamW changes the update direction through momentum and parameter-wise scaling. A gradient that appears to move toward the reference answers may therefore have a different effect after the optimizer processes it. Figure 1 checks raw-gradient steps, so it does not resolve these differences. Can the probing score be computed using the actual AdamW or policy-optimizer update direction rather than only the raw objective gradient, and compared with the changes after actual optimizer steps?

A2. Thank you for this insightful question. We have conducted two additional experiments to examine the effect of the optimizer. First, we validate the first-order approximation under actual AdamW steps. Following Section 3.2 (Figure 1), we compare the endpoint-probing score with the exact one-step change in endpoint loss for DPO, but now after an actual AdamW update rather than a raw-gradient step. The squared Pearson correlations R2R^2 are reported below:

SettingSGD step (Figure 1)AdamW step
DPO, Qwen3-1.7B0.990.93

Although the correlation slightly decreases, it remains reliable under AdamW, indicating that endpoint probing closely tracks the actual update effect in practical training.

Second, to further justify this, we recompute endpoint probing using the AdamW update direction. We repeat the DPO (out-of-distribution) probing experiment on Qwen3-1.7B, but replace the raw gradient with the actual AdamW update direction. The results under the AdamW update direction are reported below, where scores are normalized and are therefore on a different scale from Figure 3c:

IntervalNEGPOSTOT
0→30−0.0130.1460.133
30→60−0.0090.1040.095
60→90−0.0090.0740.065
90→120−0.0050.0500.045
120→1500.112−0.0670.045
150→1800.026−0.0190.007
180→2100.032−0.0050.027
210→2400.048−0.0290.020

The main conclusions remain unchanged under AdamW. Positive learning drives endpoint fitting in the early phase while negative learning opposes it, and the two components exchange roles between steps 120 and 150, consistent with our original analysis (Figure 3c).

AuthorsOpenReview ↗

Response to Reviewer e2gw (3/4)

Q3 (W2 & Q2). For PPO and GRPO, the probe also leaves out the KL term that is included in training. Please clarify whether the KL contribution is included in the full update and report how large its gradient contribution is relative to the positive and negative surrogate components, and check whether the main conclusions about positive and negative learning still hold.

A3. Thank you for this careful question. Since our analysis focuses on positive and negative learning, we did not discuss the KL term in detail. We agree that, for the rigor of the paper, this is an oversight, and we will add the details in the revision. PPO and GRPO handle the KL term differently, so we clarify them separately below.

For PPO, the KL contribution is already included in our probing. Following the standard TRL implementation, the KL penalty is incorporated into the per-token rewards with an adaptive coefficient initialized at 0.2, and thus enters the advantages. Our probing uses exactly the same KL-shaped rewards and advantages, with the KL coefficient restored from each checkpoint, and decomposes the policy surrogate into positive and negative learning according to the sign of these advantages.

For GRPO, the KL term is an explicit loss term and can be probed separately. Following verl, GRPO adds a KL loss (low-variance estimator, coefficient 10−310^{-3}) that does not affect the advantages. Since gradients are additive, the probing scores of positive and negative learning remain unchanged, and the KL term contributes an additional component. Using Qwen3-0.6B on mathematical reasoning (the setting in Figure 15), we compare the total score with and without the KL term, and report the norm of the scaled KL gradient relative to those of the positive and negative components:

StepTOT (without KL)β\beta KLTOT (with KL)∥βgKL∥/∥gPOS∥\lVert\beta g_{\mathrm{KL}}\rVert / \lVert g_{\mathrm{POS}}\rVert∥βgKL∥/∥gNEG∥\lVert\beta g_{\mathrm{KL}}\rVert / \lVert g_{\mathrm{NEG}}\rVert
00.66900.6690.000%0.000%
100.780−4.11×10⁻⁵0.7800.017%0.069%
200.928−8.68×10⁻⁵0.9280.021%0.046%
300.368−6.52×10⁻⁵0.3680.014%0.017%

Overall, the KL contribution is negligible, and we will further clarify this in the revision.

Q4. The appendix gives sufficient conditions for sparse checkpoint scores to preserve their sign, but it does not establish that the checkpoint intervals in the experiments satisfy those conditions. Please report the largest parameter displacement or a denser-checkpoint analysis for representative settings, especially the 400-step PPO interval and the 10-step NPO interval.

A4. Thank you for this question. When designing our experiments, we had already verified that the current checkpoint intervals are reliable. Following your suggestion, we further probe PPO (Pythia-2.8B) every 100 steps, four times as densely as in the paper, and NPO (Llama3.2-3B, TOFU forget05) at every step from 0 to 60, with detailed results in a separate comment below. The signs are preserved at every checkpoint, so the current intervals do not miss any sign change.

In addition, following your suggestion, we record the parameter displacement ∥θ(k)−θ(k−1)∥2\lVert\theta^{(k)} - \theta^{(k-1)}\rVert_2 after each update for NPO. For this, we retrain the paper's NPO setup (same model, batches, optimizer, schedule, and clipping) and probe the intermediate steps on this re-run, which closely reproduces the paper's run (the per-update training loss differs by at most 0.052). We summarize each 10-step interval below (path length is the sum of per-update displacements). As we can see, the score remains positive at all 61 steps without any sign change:

IntervalPath lengthLargest single updateLowest scoreHighest scoreLargest change between adjacent stepsSign changes
0–100.53460.08662.35e72.62e71.57e60
10–200.62280.07282.09e72.56e71.33e60
20–300.50180.05394.67e62.09e72.18e60
30–400.37780.04378.14e54.67e68.46e50
40–500.23210.03001.50e58.14e51.44e50
50–600.10950.01566.55e31.50e53.41e40
Reviewer_s8f9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the functional role of negative learning within composite LLM fine-tuning objectives. An endpoint probing framework uses final-checkpoint responses as empirical targets and gradient inner products to isolate positive and negative learning contributions at each training checkpoint. The analysis reveals method- and phase-dependent roles: stable target shaping under negative-only training (NPO), a regularization-to-shaping transition in off-policy DPO, and an exploration-sustaining role in on-policy PPO/GRPO. Guided by these findings, controlled variants (cDPO, cPPO, cGRPO) that modulate negative learning strength achieve consistent improvements across preference alignment, reasoning, and tool-use tasks.

Critical strengths
  1. The paper fills a genuine gap: prior work analyzes negative learning in isolation or at a single update, but its trajectory-level role within composite objectives has been unexplored. The endpoint probing framework cleanly decomposes positive and negative contributions, revealing that negative learning has qualitatively different roles across algorithms (target shaping in DPO vs. exploration in PPO/GRPO) and training phases — a finding that challenges the dominant "target shaping" narrative.

  2. The experimental coverage is broad and systematic (four algorithms, four model families, multiple tasks), and the controlled variants demonstrate that the probing insights translate into robust practical improvements, not just a single tuned setting.

  3. The controlled variants (cDPO, cPPO, cGRPO) are robust across multiple scheduling configurations, lending credibility to the underlying analysis and showing the framework has direct practical value beyond diagnosis.

Critical weaknesses
  1. Endpoint probing is inherently retrospective — it requires the final checkpoint to construct reference targets, limiting its direct use as a prescriptive tool. The controlled methods still need schedule tuning on validation data, and clearer practical heuristics for schedule selection without running the full probing analysis would strengthen the contribution.

  2. The interpretations (regularization, target shaping, exploration) are plausible but descriptive: for example, "exploration" in PPO/GRPO is inferred from offsetting endpoint-aligned gradients, yet alternatives like gradient interference are not systematically ruled out. Targeted experiments to disambiguate these hypotheses would strengthen the explanatory claims.

  3. The core operation studied here—lowering the likelihood of selected outputs—has been extensively explored in the machine unlearning literature (e.g., gradient ascent on forget sets, NPO); it would be helpful to more explicitly discuss how the “negative learning” perspective and the insights obtained here differ from or build upon that existing body of work.

Questions for authors
  1. See the weaknesses above.
  2. For the DPO in-distribution vs. out-of-distribution contrast, have you considered endpoints at varying semantic distances from the training set to see whether target shaping emerges gradually or has a sharp threshold?
  3. Are there schedule configurations that substantially degrade performance below baseline, and what characterizes them?
9 comments
AuthorsOpenReview ↗

Response to Reviewer s8f9 (1/3)

We sincerely thank the reviewer for the thoughtful feedback and for recognizing our contribution to understanding the role of negative learning in LLM fine-tuning, as well as its practical value. We fully understand your concerns and are excited to discuss them further with you. We hope the responses below provide helpful clarification, and we look forward to continuing the discussion.

Q1 (W1, part 1). Endpoint probing is inherently retrospective — it requires the final checkpoint to construct reference targets, limiting its direct use as a prescriptive tool.

A1. We agree that endpoint probing is retrospective, and we view this as a contribution rather than a limitation. Since negative learning objectives specify no explicit target, their role can only be studied indirectly. Prior work has done so through single-update analysis (e.g., Ren & Sutherland, 2025) or purely qualitative arguments. Our retrospective approach is also indirect, but goes substantially further: by using final-checkpoint responses as empirical endpoints, it enables, to our knowledge, the first rigorous empirical characterization of negative learning at the trajectory level within composite objectives.

The resulting insights are also prescriptive. A large part of our paper is devoted to analyzing the phenomena revealed by probing, and these findings directly motivate the controlled variants in Section 5 (with further details in Appendix B.8), which consistently outperform baselines across methods and tasks without extensive tuning and require no probing at deployment. We therefore respectfully believe that the insights from probing can directly inform algorithm design, rather than being limited to post-hoc diagnosis.

Q2 (W1, part 2). The controlled methods still need schedule tuning on validation data, and clearer practical heuristics for schedule selection without running the full probing analysis would strengthen the contribution.

A2. We would like to clarify a possible misunderstanding: our scheduling rules are already general and require no further probing. Based on our analysis, we derive a small set of scheduling rules in Section 5, namely a linear positive-to-negative transition for cDPO, and periodic attenuation, constant scaling, and adaptive strengthening for cPPO and cGRPO. The latter three are designed to illustrate diverse ways of coordinating positive and negative learning, and each of them improves over the baseline in the settings where it is applied (Figure 7; Appendix B.8). In all subsequent experiments, these rules are applied directly, without first probing each setting.

The tunable parameters (e.g., t1,t2t_1, t_2 in cDPO) are introduced only for flexibility, which is standard practice in algorithm design. Each rule involves only one or two such parameters, and our methods are robust to them. Figures 19–21 show that a wide range of alternative settings also clearly outperform the baselines (e.g., cDPO achieves 83.56–87.91% win rates on Qwen3-1.7B vs. 72.14% for DPO).

The few configurations that fall below the baselines mainly involve overly abrupt transitions, which our analysis explicitly suggests avoiding (see A6). Choosing among the proposed rules with proper tuning is therefore sufficient for substantial gains, further confirming that these rules generalize well.

AuthorsOpenReview ↗

Response to Reviewer s8f9 (2/3)

Q3 (W2). The interpretations (regularization, target shaping, exploration) are plausible but descriptive: for example, "exploration" in PPO/GRPO is inferred from offsetting endpoint-aligned gradients, yet alternatives like gradient interference are not systematically ruled out. Targeted experiments to disambiguate these hypotheses would strengthen the explanatory claims.

A3. Gradient interference is in fact the reason we analyze positive and negative learning separately, and it does not conflict with our interpretation of exploration. Due to such potential interference, we directly probe their gradients, rather than relying on output-based measures such as likelihood changes, which only capture their joint effect. The presence of such interference motivates us to further analyze the specific behavior of negative learning, and exploration is precisely one such behavior we observe. The two therefore concern entirely different dimensions.

More generally, our study is conducted rigorously, following the standard workflow of scientific discovery: (i) observing phenomena, (ii) forming hypotheses, and (iii) validating them through targeted experiments. For example, in DPO, (i) early negative and late positive learning oppose endpoint fitting, (ii) suggesting a beneficial regularization-like role, and (iii) removing either one reduces win rates (Figure 4). The same logic applies to our analysis of PPO and GRPO. Our interpretations are therefore not merely descriptive.

Here, we further design an additional experiment to verify the exploration role of negative learning. Using PPO on Qwen3-1.7B, we branch from checkpoints at steps 120, 200, 280, and 360 and continue training for 40 steps with or without negative learning (Normal vs. w/o NEG), using the same prompts and seed. We then sample 8 responses for each of 500 held-out prompts and measure diversity (entropy, distinct-2) and novelty (embedding distance to the endpoints).

As shown below, removing negative learning consistently reduces response diversity and novelty, suggesting that the model quickly converges to its current responses without negative learning. This directly supports the exploration role of negative learning in PPO, rather than mere gradient interference.

Branch stepContinuationEntropyDistinct-2Novelty
120Normal1.32630.70750.1622
120w/o NEG1.14490.69330.1591
200Normal1.34320.70690.1623
200w/o NEG1.15130.69060.1598
280Normal1.35510.70580.1636
280w/o NEG1.15770.69270.1589
360Normal1.35790.70630.1632
360w/o NEG1.16830.69220.1598

We will add this table and the related discussion in our revision.

Q4 (W3). The core operation studied here—lowering the likelihood of selected outputs—has been extensively explored in the machine unlearning literature (e.g., gradient ascent on forget sets, NPO); it would be helpful to more explicitly discuss how the "negative learning" perspective and the insights obtained here differ from or build upon that existing body of work.

A4. Our work complements the unlearning literature, building upon it while differing in goal and scope. Unlearning mainly uses negative learning as a tool to remove private or harmful behavior while preserving utility, whereas we treat negative learning itself as the object of understanding. We analyze its mechanistic role much more broadly across post-training methods, where it interacts with other components, mainly positive learning. Our analysis starts from NPO to confirm prior understanding of negative-only learning. This further motivates our study of composite objectives, including DPO, PPO, and GRPO, yielding new findings that extend beyond LLM unlearning.

We appreciate the reviewer's kind suggestion to discuss the relationship with unlearning more explicitly. In fact, we have discussed this relationship in the paper, for example, in the Introduction (Lines 33–53), Section 2 (Lines 124–140), and Section 4.1 (Lines 208–232). We will further clarify it in the revision accordingly.

AuthorsOpenReview ↗

Response to Reviewer s8f9 (3/3)

Q5. For the DPO in-distribution vs. out-of-distribution contrast, have you considered endpoints at varying semantic distances from the training set to see whether target shaping emerges gradually or has a sharp threshold?

A5. This is a really interesting question, and we sincerely appreciate it. Following your suggestion, we conducted an additional experiment to examine how target shaping emerges as endpoints shift from out-of-distribution toward in-distribution. Using Qwen3-1.7B DPO, we probe with a mixed endpoint set consisting of 500 in-distribution prompts (held-out UltraFeedback) and 500 out-of-distribution prompts (our HH endpoints), where the in-distribution share pp ranges from 0.0 to 1.0 in steps of 0.1, so that the distance of the endpoints to the training data varies monotonically with pp.

The table below reports late-phase scores (averaged over steps 150–240) and the checkpoint interval within which negative learning flips from negative to positive ("–" if no flip occurs). For readability, we post the detailed per-step results in a separate comment below. Target shaping weakens gradually as the endpoints move closer to the training distribution. The late-phase score of negative learning decreases steadily from 3,268 (purely out-of-distribution) to −4,068 (purely in-distribution). The flip, which occurs at steps 120–150 for in-distribution shares up to 0.4, vanishes once the share exceeds a point between 0.4 and 0.5, where the in-distribution and out-of-distribution contributions roughly balance.

ID share0.00.10.20.30.40.50.60.70.80.91.0
NEG3,2682,5331,8001,068333−400−1,133−1,866−2,601−3,334−4,068
POS−2,364−1,570−777158101,6032,3963,1893,9844,7765,570
TOT9039631,0231,0831,1431,2031,2631,3231,3821,4421,502
Flip interval of NEG120–150120–150120–150120–150120–150––––––

We find this result quite interesting and will add the related discussion to the main text. Thank you again for this inspiring suggestion.

Q6. Are there schedule configurations that substantially degrade performance below baseline, and what characterizes them?

A6. Thank you for this question. Figures 19–21 already include configurations that fall below the baselines, e.g., cDPO (4000, 4000) and (4000, 5500) on Pythia-2.8B (59.50% and 59.28% vs. 76.71% for DPO), and cPPO (400, 0.08) and (2, 0.5) (54.72% and 54.29% vs. 55.96% for PPO).

As we can see, cPPO is relatively robust to the schedule, whereas cDPO is more sensitive. The degraded cDPO schedules share one pattern: the shift from positive to negative learning is too abrupt. In particular, the w/o POS ablation in Figure 4 corresponds to an abrupt switch to negative-only learning, which substantially reduces win rates. This is consistent with our analysis: both components matter throughout training, so their balance should be shifted gradually. We will add the related discussion to the main text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.