- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a new OPD method that 1) removes the token-level relative advantage magnitude, 2) introduces a global update strength that is based on EMA of estimated TV. The authors motivated their proposal by a preliminary ablation experiment on the individual effects of advantage direction, relative magnitude, and global update strength. Compared with previous related work, their core argument is that not only token-level advantage magnitude destabilizes the OPD training, but also it is unnecessary as long as the implied global attenuation effect is properly retained. This observation-based claim is very interesting in itself, and the paper's theoretical elaboration on it is insightful, in particular the link between the gradient of TV loss and the sign-only gradient weight. Experiments corroborate the claim by comparing the proposed method with relevant baselines and show it retains similar performance.
Critical strengths
- The paper is well written, self-contained, and easy to follow.
- The preliminary ablation experiment studying the individual effects of advantage direction, relative magnitude, and global update strength provides enough insights itself, and the discovery that advantage magnitude is not as crucial as previously thought is noteworthy. It may also suggest a new direction for future research.
- The theoretical investigation of the relationship between a TV loss and their proposed method makes the proposal justified and less seemingly tricky.
Critical weaknesses
- The author additionally claims "TV-OPD achieves ... lower variance in the late-stage of training" in the abstract. However, if I am not mistaken, the sole empirical evidence is in Appendix E with a few data points. In the main paper, the training dynamics such as Figure 4 and Table 2 do not show conspicuously lower variance compared to baselines.
- The exact reasoning for moving from using the S as defined in the preliminary experiment to the proposed TV based scaling factor is ambiguous (see related questions in "Questions For Authors") and needs more clarity.
- The paper's general claim of the "non-crucialness of advantage magnitude" would have been more empirically solid if 1) more model families are used; 2) experiments on tasks beyond math problem solving are conducted.
Questions for authors
- There are actually two differences between the proposed global update strength and the naive advantage aggregation S: EMA and TV estimates. can the authors properly ablate the two components and compare with S with additional experiments?
- It seems one of reasons that motivate TV with EMA instead of the naive S is that TV is bounded whereas S is not. If this is true, can the authors make it more explictly stated.
- The proposed method only show performance gains over baselines in some settings. Can the authors provide more analysis and thoughts on why the method is not universally better? Also, if the performance gain itself is not so significant, what are the practical use cases where someone would prefer TV-OPD over Vanilla OPD?
3 comments
Response to Reviewer u5NE (Part 2/2)
[Weakness 3] The magnitude observation would be more convincing with evidence from additional model families and tasks beyond mathematical reasoning. Please clarify how broadly the current experiments support the claim.
Appendix D (Figures 6–7, Table 8) replicates all six magnitude variants on the Qwen pair:
| Method | AIME24 best | AIME25 best | Trajectory mean |
|---|---|---|---|
| Raw | 67.50 ± 2.36 | 55.83 ± 2.36 | 57.01 ± 0.32 |
| Power-0.4 | 68.33 ± 3.54 | 57.08 ± 1.77 | 57.23 ± 0.33 |
| Power-0.8 | 69.17 ± 1.18 | 57.08 ± 1.77 | 57.37 ± 0.61 |
| Shuffle | 67.08 ± 0.59 | 54.17 ± 2.36 | 55.63 ± 0.58 |
| Sequence-Constant | 65.42 ± 1.77 | 59.17 ± 2.36 | 56.44 ± 2.21 |
| Sign | 67.50 ± 3.54 | 57.08 ± 2.95 | 57.01 ± 1.34 |
Sign matches Raw's trajectory mean (57.01%) and improves AIME25 best accuracy (57.08% vs. 55.83%), supporting the magnitude observation on a second teacher–student pair.
We additionally evaluated the six variants on LiveCodeBench (LCB) with Qwen, using best-checkpoint selection for each method:
| Experiment | Raw OPD | Power-0.4 | Power-0.8 | Shuffle | Sequence-Constant | Sign |
|---|---|---|---|---|---|---|
| Qwen-LCB | 24.5 | 24.8 | 25.0 | 24.0 | 24.1 | 24.8 |
The five ablation variants score 24.0–25.0, compared with 24.5 for Raw OPD. Sign reaches 24.8 (+0.3 points), supporting the effectiveness of sign-only supervision on this coding benchmark. The evaluated model families remain Qwen and DeepSeek-Distill-Qwen.
[Question 3] TV-OPD improves over baselines only in some settings, with modest overall gains. Please explain these benchmark-dependent results and identify practical circumstances in which users should prefer it to Vanilla OPD.
TV-OPD does not dominate every benchmark, but achieves the highest six-benchmark mean on both model pairs. Benchmark-dependent gains also occur across the baselines: no single baseline performs best on every task. So the performance improvement of TV-OPD is convincing from this perspective.
TV-OPD is particularly useful when extreme token weights impair training. Sign supervision removes magnitude outliers, while historical TV feedback regulates the shared scale, promoting stable training and faster convergence. Gains can be smaller when Raw OPD already trains stably.
The new 38k experiment (Appendix E.3) illustrates this benefit during length extension. TV-OPD reaches a four-task mean of 33.85% at step 200, exceeding Raw OPD’s 33.44% at step 500, and improves to 35.16% at step 500. This earlier improvement and sustained lead support its practical value for longer reasoning rollouts.
Thank you for the response. I think my concerns are well addressed.
Response to Reviewer u5NE (Part 1/2)
Thank you for the careful review.
We strengthened the stability evidence with three additions: seed-wise curves make late-stage fluctuations within individual runs clearer; PeakDrop (Appendix E.2, Table 9) is smaller for TV-OPD than Raw and Sign on both model pairs; and the 38k response-length setting (Appendix E.3, Figure 10) shows stable convergence and a sustained advantage over Raw OPD with longer responses. The curves provide qualitative stability evidence, while PeakDrop quantifies retention; accordingly, the abstract now states improved late-stage performance retention.
Boundedness is a central reason for replacing S with TV. Although relative token magnitudes are removed, S=T−1∑t∣Δt∣ remains unbounded and sensitive to extreme log-ratios. TV retains discrepancy-based scaling through the bounded statistic [1−eΔ]+∈[0,1], while matching the conditional-TV objective underlying sign supervision. This lets us regulate shared supervision strength without reintroducing raw magnitude outliers.
We added a comparison with EMA removed from the TV controller. Using Appendix E.2's AIME24/25 LateMean protocol:
Removing EMA gives similar LateMean scores, with no consistent advantage for either variant. Without EMA, Qwen improves slightly on both tasks, while JustRL improves on AIME24 and declines on AIME25. TV-based regulation therefore remains effective without EMA; we use EMA to smooth the discrepancy estimate across training steps.