acceptodds
Under review as a conference paper at ICLR 2027

TV-Regulated OPD: Direction Matters in On-Policy Distillation

Abstract

On-Policy Distillation (OPD) facilitates the transfer of knowledge from domain expert to student in the post-training phase of Large Language Models (LLMs). However, the supervision signals in mainstream OPD methods suffer from high variance and noise which is generally instable during training. In this work, we systematically investigated what really matters to the performance and the fundamental mechanisms behind the instability during training. We found that retaining only the sign of token-level advantages is sufficient to achieve the performance comparable to standard OPD. Meanwhile, smoother and bounded advantages can stabilize the training process without sacrificing its performance. These motivated us to shape the advantages using the Total Variation (TV) and propose a robust TV regulated On-Policy Distillation (TV-OPD) method. Benefiting from the bounded and diminished advantages, TV-OPD exhibits stable training dynamics and steady late-stage performance. We conducted comprehensive experiments and found that, across various settings, TV-OPD consistently achieved better performance and lower variance in the late-stage of training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_u5NEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new OPD method that 1) removes the token-level relative advantage magnitude, 2) introduces a global update strength that is based on EMA of estimated TV. The authors motivated their proposal by a preliminary ablation experiment on the individual effects of advantage direction, relative magnitude, and global update strength. Compared with previous related work, their core argument is that not only token-level advantage magnitude destabilizes the OPD training, but also it is unnecessary as long as the implied global attenuation effect is properly retained. This observation-based claim is very interesting in itself, and the paper's theoretical elaboration on it is insightful, in particular the link between the gradient of TV loss and the sign-only gradient weight. Experiments corroborate the claim by comparing the proposed method with relevant baselines and show it retains similar performance.

Critical strengths
  1. The paper is well written, self-contained, and easy to follow.
  2. The preliminary ablation experiment studying the individual effects of advantage direction, relative magnitude, and global update strength provides enough insights itself, and the discovery that advantage magnitude is not as crucial as previously thought is noteworthy. It may also suggest a new direction for future research.
  3. The theoretical investigation of the relationship between a TV loss and their proposed method makes the proposal justified and less seemingly tricky.
Critical weaknesses
  1. The author additionally claims "TV-OPD achieves ... lower variance in the late-stage of training" in the abstract. However, if I am not mistaken, the sole empirical evidence is in Appendix E with a few data points. In the main paper, the training dynamics such as Figure 4 and Table 2 do not show conspicuously lower variance compared to baselines.
  2. The exact reasoning for moving from using the S as defined in the preliminary experiment to the proposed TV based scaling factor is ambiguous (see related questions in "Questions For Authors") and needs more clarity.
  3. The paper's general claim of the "non-crucialness of advantage magnitude" would have been more empirically solid if 1) more model families are used; 2) experiments on tasks beyond math problem solving are conducted.
Questions for authors
  1. There are actually two differences between the proposed global update strength and the naive advantage aggregation S: EMA and TV estimates. can the authors properly ablate the two components and compare with S with additional experiments?
  2. It seems one of reasons that motivate TV with EMA instead of the naive S is that TV is bounded whereas S is not. If this is true, can the authors make it more explictly stated.
  3. The proposed method only show performance gains over baselines in some settings. Can the authors provide more analysis and thoughts on why the method is not universally better? Also, if the performance gain itself is not so significant, what are the practical use cases where someone would prefer TV-OPD over Vanilla OPD?
3 comments
AuthorsOpenReview ↗

Response to Reviewer u5NE (Part 1/2)

Thank you for the careful review.

[Weakness 1] The abstract claims lower late-stage variance, but the limited Appendix E evidence and main training curves do not clearly establish it. Please provide stronger empirical support for this stability claim.

We strengthened the stability evidence with three additions: seed-wise curves make late-stage fluctuations within individual runs clearer; PeakDrop (Appendix E.2, Table 9) is smaller for TV-OPD than Raw and Sign on both model pairs; and the 38k response-length setting (Appendix E.3, Figure 10) shows stable convergence and a sustained advantage over Raw OPD with longer responses. The curves provide qualitative stability evidence, while PeakDrop quantifies retention; accordingly, the abstract now states improved late-stage performance retention.

[Weakness 2 & Question 2] The motivation for replacing the sequence-level magnitude S with a TV-based scale is unclear. Please explain this transition and state explicitly whether boundedness is a central reason for choosing TV.

Boundedness is a central reason for replacing SS with TV. Although relative token magnitudes are removed, S=T−1∑t∣Δt∣S=T^{-1}\sum_t|\Delta_t| remains unbounded and sensitive to extreme log-ratios. TV retains discrepancy-based scaling through the bounded statistic [1−eΔ]+∈[0,1][1-e^\Delta]_+\in[0,1], while matching the conditional-TV objective underlying sign supervision. This lets us regulate shared supervision strength without reintroducing raw magnitude outliers.

[Question 1] The proposed scale differs from the original magnitude aggregation in both its discrepancy statistic and EMA smoothing. Please ablate these two changes separately and compare the resulting variants with S.

We added a comparison with EMA removed from the TV controller. Using Appendix E.2's AIME24/25 LateMean protocol:

MethodJustRL AIME24JustRL AIME25Qwen AIME24Qwen AIME25
Full49.86 ± 0.7936.88 ± 1.8767.15 ± 0.6955.56 ± 0.20
Full without EMA50.00 ± 0.935.83 ± 0.567.50 ± 1.356.16 ± 0.7

Removing EMA gives similar LateMean scores, with no consistent advantage for either variant. Without EMA, Qwen improves slightly on both tasks, while JustRL improves on AIME24 and declines on AIME25. TV-based regulation therefore remains effective without EMA; we use EMA to smooth the discrepancy estimate across training steps.

AuthorsOpenReview ↗

Response to Reviewer u5NE (Part 2/2)

[Weakness 3] The magnitude observation would be more convincing with evidence from additional model families and tasks beyond mathematical reasoning. Please clarify how broadly the current experiments support the claim.

Appendix D (Figures 6–7, Table 8) replicates all six magnitude variants on the Qwen pair:

MethodAIME24 bestAIME25 bestTrajectory mean
Raw67.50 ± 2.3655.83 ± 2.3657.01 ± 0.32
Power-0.468.33 ± 3.5457.08 ± 1.7757.23 ± 0.33
Power-0.869.17 ± 1.1857.08 ± 1.7757.37 ± 0.61
Shuffle67.08 ± 0.5954.17 ± 2.3655.63 ± 0.58
Sequence-Constant65.42 ± 1.7759.17 ± 2.3656.44 ± 2.21
Sign67.50 ± 3.5457.08 ± 2.9557.01 ± 1.34

Sign matches Raw's trajectory mean (57.01%) and improves AIME25 best accuracy (57.08% vs. 55.83%), supporting the magnitude observation on a second teacher–student pair.

We additionally evaluated the six variants on LiveCodeBench (LCB) with Qwen, using best-checkpoint selection for each method:

ExperimentRaw OPDPower-0.4Power-0.8ShuffleSequence-ConstantSign
Qwen-LCB24.524.825.024.024.124.8

The five ablation variants score 24.0–25.0, compared with 24.5 for Raw OPD. Sign reaches 24.8 (+0.3 points), supporting the effectiveness of sign-only supervision on this coding benchmark. The evaluated model families remain Qwen and DeepSeek-Distill-Qwen.

[Question 3] TV-OPD improves over baselines only in some settings, with modest overall gains. Please explain these benchmark-dependent results and identify practical circumstances in which users should prefer it to Vanilla OPD.

TV-OPD does not dominate every benchmark, but achieves the highest six-benchmark mean on both model pairs. Benchmark-dependent gains also occur across the baselines: no single baseline performs best on every task. So the performance improvement of TV-OPD is convincing from this perspective.

TV-OPD is particularly useful when extreme token weights impair training. Sign supervision removes magnitude outliers, while historical TV feedback regulates the shared scale, promoting stable training and faster convergence. Gains can be smaller when Raw OPD already trains stably.

The new 38k experiment (Appendix E.3) illustrates this benefit during length extension. TV-OPD reaches a four-task mean of 33.85% at step 200, exceeding Raw OPD’s 33.44% at step 500, and improves to 35.16% at step 500. This earlier improvement and sustained lead support its practical value for longer reasoning rollouts.

Thank you for the response. I think my concerns are well addressed.

Reviewer_y4ZKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TV OPD replaces the sampled teacher to student log probability ratio with its sign, then applies a shared coefficient computed from a historical estimate of total variation. The motivation is that accurate relative advantage magnitudes may be less important than preserving their signs. The paper connects the sign signal to conditional TV minimization, derives a bounded estimator from probabilities already available during distillation, and evaluates the resulting method on two teacher and student pairs. Its most relevant empirical claim is improved retention later in training, rather than a large increase in the best observed accuracy.

Critical strengths
  1. The sign and magnitude decomposition is a productive way to study the supervision signal. The power transforms, sequence constant weights, and shuffled magnitudes probe different parts of the original coefficient rather than merely comparing two complete algorithms. Appendix D also includes a control preserving raw allocation within sign groups. These experiments give the direction only proposal a more substantial basis than the observation that clipping sometimes helps.

  2. The mathematical treatment is careful about several easy to miss details. Equations 10 and 11 obtain TV from a bounded statistic of sampled token probabilities, and Equation 12 uses the previous step's estimate rather than the same batch's estimate. Appendix B distinguishes fixed horizon expectations, random denominator pooling, predictable scaling, and token selection. These qualifications matter, and I appreciate that the paper does not present all of these estimators as unconditionally unbiased.

  3. The comparisons use the same rollout recipe within each model pair and include several relevant ways of modifying OPD coefficients. Table 1 is also interpreted sensibly: the authors explicitly describe the 0.67 and 0.65 point margins as competitive results, not statistically established superiority on every task.

Critical weaknesses
  1. The regulator is the principal addition beyond sign based training, but its proposed mechanism has not actually been measured. Section 6.3 states that the available exports contain neither the coefficient, conditional TV, nor the realized parameter update norm. Equation 12 rescales the gradient before AdamW; that is not equivalent to rescaling AdamW's parameter update, because its moment estimates change as well. Consequently, the accuracy curves cannot establish that a declining TV estimate is reducing the effective update magnitude in the intended way. Logging these quantities is essential here. A fixed attenuation schedule or a replayed coefficient sequence would also help determine whether feedback from the current training trajectory adds value beyond generic attenuation.

  2. The strongest late training analysis does not fully separate sign replacement from TV regulation. Table 2 includes Sign TV, but Section 6.3 identifies Raw OPD and TV OPD as the methods with complete coverage in the late window. Appendix E then reports LateMean and PeakDrop only for those two methods. That comparison supports the combined method over Raw, but cannot cleanly attribute retention to the regulator rather than to discarding magnitudes. The alpha sweep in Section 6.4 is useful, yet it selects the best checkpoint and therefore answers a different question. Please complete the direction only trajectory under the same horizon and report its retention statistics alongside the regulated version.

  3. Checkpoint selection limits what the main accuracy table demonstrates. Appendix C selects each run's highest six benchmark mean within steps 1 to 1000, and those same benchmarks supply the reported results. Using one shared checkpoint per run is preferable to selecting a different checkpoint for each task, but it still produces a selected validation score rather than an independent test estimate. With such small aggregate margins, selection opportunities and evaluation noise matter. Fixed terminal checkpoints, or selection on separate validation problems, would provide a more persuasive comparison. The late window analysis partly addresses this concern, but only for a subset of the methods and benchmarks.

Questions for authors
  1. Appendix B.7 describes optional chunk credit assignment, clamping, and sentinel exclusion, then carefully explains when these operations change the TV estimand. Which of these options were active in the reported runs? Please connect the actual configuration to the assumptions rather than leaving the reader to infer that connection.

  2. State the number of independent training runs and the seed pairing explicitly for each table and trajectory group. A sample standard deviation without its sample count is difficult to interpret, especially where later observations have reduced coverage.

  3. Table 4 lists four evaluation samples per problem, whereas Section 6.1 and Appendix C specify sixteen for the main tables and four for the dynamics. Label the table entry as a dynamics setting or list both protocols. The evaluation interval is also given only as “Periodic”; its numerical value matters for best checkpoint selection.

  4. The equivalence stated after Equation 9 needs a narrower scope. That equation derives the unweighted sign gradient, whereas Equation 7 multiplies the sign by S computed from the same sampled response. For a response containing one token, S is the absolute raw advantage, so Equation 7 recovers the raw log ratio rather than the TV sign estimator. Detaching S does not remove this sampling dependence. Please distinguish the Sequence Constant construction from the historical coefficient in Equation 12; this objection does not invalidate the latter.

4 comments
AuthorsOpenReview ↗

Response to Reviewer y4ZK (Part 1/2)

Thank you for the detailed review.

[Weakness 1] Scaling gradients before AdamW does not directly scale parameter updates, and the proposed mechanism has not been measured. Please log discrepancy, coefficients, and realized updates, and compare fixed or replayed schedules to isolate the value of feedback.

Appendix F.1 now includes the full Constant, Linear, and TV-OPD diagnostics alongside Sign-TV. Over steps 251–300, update-norm ratios to Sign-TV are 0.70–0.71 for TV-OPD, 0.93–0.95 for Constant, and 0.79–0.80 for Linear. TV-OPD's later ratios of 1.62–1.72 reflect the methods' different optimization histories: ckc_k is not a proportional learning-rate multiplier.

Under Appendix C's matched 625-step protocol, TV-OPD/Constant/Linear achieve four-task means of 52.37%/51.38%/50.71%. These results demonstrate the effectiveness of TV-OPD's scheduling mechanism.

[Weakness 2] The late-stage comparison omits complete Sign-TV trajectories, so it cannot separate magnitude removal from regulation. Please complete the direction-only runs at the same horizon and compare their retention with Full TV-OPD.

Appendix E.2 adds the matched Sign-TV comparison on both pairs. TV-OPD reduces AIME-mean PeakDrop from 3.10 ± 1.01 to 2.07 ± 0.29 on JustRL and 3.09 ± 0.15 to 1.98 ± 0.62 on Qwen. LateMean increases from 43.15 to 43.37 and 61.28 to 61.35, respectively.

[Weakness 3] Selecting and reporting checkpoints on the same benchmarks can inflate small performance differences. Please assess the ranking at fixed checkpoints or with separate validation data, and extend the late-window comparison beyond its current coverage.

Our additional validation ablation separates checkpoint selection from evaluation: checkpoints are selected on two benchmarks and evaluated on the remaining four. TV-OPD ranks first in most trials on both model pairs:

Model pairTV-OPD win rate
JustRL-1.5B73.3%
Qwen3-8B66.7%

At Qwen3-8B step 600, TV-OPD scores 68.25%, versus Raw 66.49%, ClipOPD 66.53%, PowerOPD 65.70%, and vOPD 67.14%, leading every baseline in both seeds.

At JustRL-1.5B step 600, TV-OPD achieves per-run six-benchmark means of 52.97% (seed 63) and 52.32% (seed 42), exceeding Raw OPD (52.10%/51.31%) by 0.87/1.01 percentage points. TV-OPD ranks first for seed 63; vOPD leads seed 42 (52.60%). The per-run fixed-step results are in Appendix E.5.

Appendix E.4 (Table 10) extends LateMean to all five methods and six benchmarks on both pairs. The six-benchmark averages are:

MethodJustRL-1.5B LateMeanQwen3-8B LateMean
Raw OPD52.2966.31
ClipOPD51.9666.34
PowerOPD51.8365.96
vOPD52.4966.34
TV-OPD (Ours)52.6666.45

Additionally, we want to clarify applying the same best-checkpoint selection rule to all methods a fair way to compare their attainable performance, since methods can converge at different rates and peak at different steps. A single fixed step may favor methods whose peaks happen to occur near that step.

[Question 1] Which optional preprocessing and masking operations in Appendix B.7 were used in the reported experiments? Please connect the actual run configurations to the assumptions under which the statistic estimates conditional TV.

The tokenwise conditional-TV identity holds under our preprocessing settings. Appendix C records clamping off and 1:1 token alignment, which preserve the original log-ratios. Automatic sentinel masking excluded no tokens in the available diagnostic logs. The scheduler uses a global token mean without sequence normalization. With response-only masking, this consistently estimates active-token average TV; variable length can introduce finite-batch bias, as explained in Appendix B.

AuthorsOpenReview ↗

Response to Reviewer y4ZK (Part 2/2)

[Questions 2–3] Please specify the run count, seed pairing, and coverage for every table and trajectory group. Also distinguish four- versus sixteen-response evaluation and give the numerical checkpoint-evaluation interval instead of “Periodic.”

Appendix C lists run counts and paired seeds; Table 4 now distinguishes the evaluation protocols. Main comparisons and ablations use seeds 42 and 63 (n=2n=2). Main results and the regulator-strength sweep use mean@16; training dynamics and retention use mean@4. The main-table checkpoint-selection interval is 25 steps, as for matched retention, six-benchmark LateMean, and Constant/Linear controls.

[Question 4] Equation 9 derives an unweighted sign gradient, whereas Equation 7 uses a scale computed from the same sampled response. Please narrow the equivalence claim and distinguish this sampling-dependent scale from the historical coefficient in Equation 12.

Eq. 9 applies to unweighted Sign supervision; Eq. 7 includes the current-response scale S(Y)S(Y). Detaching S(Y)S(Y) leaves its correlation with the sampled gradient intact, so Eq. 9 does not extend to that weighting.

Appendix A.2 establishes the conditional-TV gradient identity at fixed state occupancy and the unclipped on-policy point. The historical coefficient ck>0c_k>0 in Eq. 12 is fixed before current sampling, so it preserves this expected pre-optimizer direction.

Dear Reviewer,

Thank you so much for your thoughtful initial reviews!

With only two days left before the discussion window closes, I wanted to send a friendly reminder to check out the authors' rebuttal. They have put a lot of effort into answering the specific questions you raised.

Whenever you have a spare moment today, please drop a brief reply in the thread to let the authors know if their responses have addressed your concerns, or if any questions remain. Your timely feedback makes a huge difference in ensuring a fair and constructive review process!

Warm regards

Dear Reviewer,

We sincerely appreciate your thoughtful review. We would be truly grateful for any further feedback you may have. If any concerns remain unaddressed, we would be more than happy to provide further clarification.

Reviewer_DKjsOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines sign and magnitude in sampled-token supervision for on-policy distillation. It decomposes the log-probability difference between teacher and student into sign, relative token magnitude, and sequence-level scale. Ablations suggest that flattening or shuffling relative magnitudes preserves much of the benefit of standard OPD in the evaluated setting. Motivated by these findings, TV-OPD combines sign coefficients with a shared scale derived from a historical moving average of estimated conditional total variation. The paper connects the sign signal to the conditional TV gradient and derives a bounded sampled-token TV estimator. Across two model pairs and six mathematical reasoning benchmarks, TV-OPD achieves the highest reported macro-average in both settings, with modest gains over the strongest baselines. Additional analyses examine regulator sensitivity and late-stage retention.

Critical strengths

The strengths of this paper mainly lie in the following two aspects:

(1).Separating relative token weights from aggregate scale enables more targeted interventions than comparing complete loss functions. Flattening, shuffling, and direction-only ablations indicate that precise token magnitudes are not indispensable in the tested setting, clarifying which supervision components support learning.

(2).The fixed-state connection between sign coefficients and the TV gradient is sound under the stated assumptions. The bounded estimator reuses sampled-token log probabilities without additional teacher scoring. Explicit qualifications regarding state occupancy, variable-length pooling, and conditional versus sequence-level TV strengthen its interpretation.

Critical weaknesses

The weaknesses of this paper mainly lie in the following two aspects:

(1).The regulator sweep lacks tuned constant-scale and predetermined attenuation controls. Coefficient dispersion and performance retention do not directly establish how TV feedback affects AdamW updates or gradient variability, leaving the proposed mechanism insufficiently supported.

(2).The improvements over the strongest baselines are 0.67 and 0.65 percentage points, with checkpoints selected on the same benchmarks used for reporting. Unclear training-seed counts and incomplete late-stage direction-only comparisons further limit confidence in the performance and stability claims.

Questions for authors

The questions that require further clarification from the authors are as follows:

(1). Under the same training budget, how does TV-OPD compare with direction-only training using a tuned constant scale or a predetermined linear decay schedule? Please plot the TV estimate, applied scaling coefficient, and realized AdamW update norm for all three methods. Do decreases in the coefficient correspond to smaller or less variable parameter updates?

(2). How many independent training runs were used for each method, and what were their seeds? Were comparisons paired by seed? Please report per-run macro-average accuracy at a common predetermined training step. Does the ranking persist when checkpoints are selected on a separate validation set?

(3). Please report LateMean and PeakDrop for Raw OPD, Sign-TV, and TV-OPD on both model pairs using the windows defined in Appendix E. Which runs have incomplete coverage, and why? Does lower variance refer to variation across checkpoints, across seeds, or gradient estimation variance? If it refers to gradient variance, please measure it using independent batches at fixed model parameters.

(4). Were negative-branch clamping, chunk credit assignment, or current-token sentinel exclusion enabled in the reported runs? If so, please give the formula for the statistic supplied to the scheduler and identify the quantity it estimates. Does the conditional TV interpretation in Eq. (11) still hold under these settings, given the qualifications in Appendix B.7?

(5). Please compare the sign coefficient, sampled-token TV estimator, and historical scaling rule with the corresponding components in PowerOPD and prior TV-based distillation such as f-DISTILL. Which components are new, and which are adaptations of existing objectives or estimators?

4 comments
AuthorsOpenReview ↗

Response to Reviewer DKjs (Part 1/2)

Thank you for the constructive review.

[Weakness 1 & Question 1] How does TV-OPD compare with tuned constant scaling and predetermined linear decay under the same budget? Please measure TV, the applied coefficient, and realized AdamW updates to establish whether feedback reduces update magnitude or variability.

We added tuned Constant and predetermined Linear controls under the same 625-step JustRL budget. For these controls, Sign-TV, and TV-OPD, we tracked validation accuracy, TV estimates, applied coefficients, and realized AdamW updates. TV-OPD achieves the highest four-task mean:

MethodAIME24AIME25AIME26MATH-500Mean
Constant47.5034.1735.8388.0051.38
Linear45.8336.6733.3387.0050.71
TV-OPD49.1736.6735.8387.8052.37

Appendix F.1 reports the update diagnostics. Over steps 251–300, update-norm ratios relative to Sign-TV are:

MethodScale ruleUpdate ratio, seed 63Update ratio, seed 42
Signc=1c=11.0001.000
Constantc=0.1c=0.10.9470.932
Linear1→0.11\rightarrow0.10.7950.790
TV-OPDHistorical TV feedback0.7100.703

Constant scaling is largely canceled by Adam's normalization, while time-varying scaling interacts with its accumulated moments. TV-OPD's ratios later reach 1.62/1.72 over steps 501–625 despite ck<1c_k<1, reflecting different optimization histories rather than proportional update scaling. The constant baseline uses c=0.1c=0.1, selected from {0.1,0.2,0.3,0.4,0.5}\{0.1,0.2,0.3,0.4,0.5\} by the highest four-task mean validation score.

[Weakness 2 & Question 2] The small gains rely on checkpoints selected on the reporting benchmarks, and the number of training runs is unclear. Please specify seed pairing and provide per-run fixed-step scores and results using separate-validation checkpoint selection.

Appendix C specifies seed pairing and checkpoint selection. Per-seed results underlying Table 1 are below.

Model pairSeedBaselineBaseline score / stepTV-OPD score / stepDifference
JustRL 1.5B63Raw OPD54.07 / 100054.40 / 800+0.33
JustRL 1.5B42Raw OPD53.85 / 70054.87 / 975+1.02
Qwen3 8B63vOPD67.95 / 50068.69 / 625+0.74
Qwen3 8B42vOPD67.78 / 62568.34 / 625+0.56

In an additional validation ablation, we selected checkpoints on two benchmarks and evaluated on the remaining four. TV-OPD ranked first in most trials on both model pairs:

Model pairTV-OPD win rate
JustRL-1.5B73.3%
Qwen3-8B66.7%

We also report fixed-step results on both model pairs. At Qwen3-8B step 600, TV-OPD leads every baseline in both seeds:

MethodSeed 63Seed 42Mean
Raw OPD66.2366.7666.49
ClipOPD66.1366.9266.53
PowerOPD66.0965.3065.70
vOPD66.7967.5067.14
TV-OPD67.9768.5268.25

At JustRL-1.5B step 600, TV-OPD achieves 52.97% for seed 63 and 52.32% for seed 42, improving over Raw OPD by 0.87 and 1.01 percentage points, respectively. TV-OPD ranks first for seed 63; vOPD ranks first for seed 42 (52.60%). Appendix E.5 reports the per-run six-benchmark means:

MethodSeed 63Seed 42Mean
Raw OPD52.1051.3151.71
ClipOPD52.3250.6451.48
PowerOPD51.4951.6751.58
vOPD52.9352.6052.77
TV-OPD52.9752.3252.65

The six-benchmark LateMean comparison in Appendix E.4 ranks TV-OPD first on both model pairs.

Additionally, we want to clarify applying the same best-checkpoint selection rule to all methods a fair way to compare their attainable performance, since methods can converge at different rates and peak at different steps. A single fixed step may favor methods whose peaks happen to occur near that step.

AuthorsOpenReview ↗

Response to Reviewer DKjs (Part 2/2)

[Question 3] Please compare Raw, Sign-TV, and TV-OPD using Appendix E's LateMean/PeakDrop windows on both model pairs, and explain missing coverage. Clarify which variance is claimed; gradient variance requires independent batches at fixed parameters.

Appendix E.2 (Table 9) compares Raw, Sign-TV, and Full TV-OPD using matched windows. TV-OPD has the smallest mean PeakDrop on both pairs:

PairMethodLateMeanPeakDrop
JustRLRaw OPD42.33 ± 0.542.88 ± 0.83
JustRLSign43.15 ± 0.173.10 ± 1.01
JustRLTV-OPD43.37 ± 0.542.07 ± 0.29
QwenRaw OPD61.11 ± 0.882.64 ± 1.47
QwenSign61.28 ± 1.623.09 ± 0.15
QwenTV-OPD61.35 ± 0.441.98 ± 0.62

The original "variance" wording referred to validation-accuracy fluctuations across checkpoints. We have revised the abstract to emphasize improved late-stage performance retention, which PeakDrop measures.

[Question 4] Were clamping, chunk credit assignment, or current-token sentinel exclusion active in the reported runs? Please give the implemented scheduler statistic and explain whether its conditional-TV interpretation still holds under those settings.

The scheduler pools the original log-ratios as

D^k=∑imi[1−exp⁡(Δi)]+∑imi,Δi=log⁡p(ai∣si)q(ai∣si),ai∼q(⋅∣si).\widehat D_k=\frac{\sum_i m_i[1-\exp(\Delta_i)]_+}{\sum_i m_i},\qquad \Delta_i=\log\frac{p(a_i\mid s_i)}{q(a_i\mid s_i)},\quad a_i\sim q(\cdot\mid s_i).

The tokenwise conditional-TV identity holds under our preprocessing settings. Appendix C records clamping off and 1:1 token alignment, which preserve the original log-ratios. Automatic sentinel masking excluded no tokens in the available diagnostic logs. The scheduler uses a global token mean without sequence normalization. With response-only masking, this consistently estimates active-token average TV; variable length can introduce finite-batch bias, as explained in Appendix B.

[Question 5] How do the sign coefficient, sampled-token TV estimator, and historical scaling rule differ from PowerOPD and f-DISTILL? Please identify which elements are new and which adapt established objectives or estimators.

(1) Sign coefficient. Sign consistency is established in PowerOPD, which retains token-dependent magnitudes. We remove these magnitudes entirely; our controlled ablations provide evidence that sign-only supervision remains effective. The conditional-TV gradient identity is established mathematics.

(2) Sampled-token TV estimator. [1−exp⁡(A^t)]+[1-\exp(\hat A_t)]_+ adapts a standard one-sided TV identity to existing OPD scores, giving a bounded estimate without extra teacher scoring. f-DISTILL instead uses vocabulary-wide conditional TV losses. Our contribution here is the estimator's use for OPD regulation.

(3) Historical scaling rule. We propose the specific TV controller and its combination with sign-only supervision: a lagged EMA, normalized by a fixed initial reference, determines a clipped power-law scale shared across tokens. EMA is standard; our design fixes the scale before current sampling, avoiding coupling to the current sampled gradient.

Dear Reviewer,

Thank you so much for your thoughtful initial reviews!

With only two days left before the discussion window closes, I wanted to send a friendly reminder to check out the authors' rebuttal. They have put a lot of effort into answering the specific questions you raised.

Whenever you have a spare moment today, please drop a brief reply in the thread to let the authors know if their responses have addressed your concerns, or if any questions remain. Your timely feedback makes a huge difference in ensuring a fair and constructive review process!

Warm regards

Dear Reviewer,

We sincerely appreciate your thoughtful review. We would be truly grateful for any further feedback you may have. If any concerns remain unaddressed, we would be more than happy to provide further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.