- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper argues that for long-context tasks trajectory level OPD score has weak alignment with verifier based rewards. They motivate this teacher-verifier disagreement on two representative long-context tasks by capturing the pairwise disagreement rate as a function of increasing prompt length. To address this disagreement, the authors introduce GC-OPD, which separately normalizes verifier rewards and trajectory level OPD scores for each group of rollouts and use their difference as signed teacher-verifier disagreement residual. Relative-advantage-based credit assignment (RACA) is used for distributing this residual score across tokens according to their relative OPD advantage. Experiments across two model scales and five benchmarks show higher average scores compared to baselines.
Critical strengths
- The motivation analysis quantifies pairwise disagreement between trajectory level OPD scores and verifier rewards and offers new insights on OPD for long-context tasks.
- The proposed method, GC-OPD, is simple and does not require additional rollouts/forward passes.
Critical weaknesses
- The performance gap between GC-OPD and baselines appear marginal and within noise. Table 6 in appendix reports the use of 3 random seeds. However, all results in the main paper report a single number without standard deviation or confidence intervals. The absence of standard deviation makes it difficult to assess if the gains over baselines are statistically significant or simply noise.
- Missing comparison against key baselines. The proposed method, despite using verified reward in its formulation, does not compare against GRPO or stronger verifier-aware OPD baselines like [1,2]. SG-OPD and Reward-Aware OPD are discussed in related work section, but not evaluated.
- Untuned baselines. Section 5 (lines 385-386) states the use of fixed 231 prompts disjoint from training set for tuning the beta coefficient. However, the baselines are not subjected to any hyper-parameter tuning and use the standard default configuration. To substantiate, ExOPD uses the default lambda = 1.25 (lines 889-890).
References
[1] Xu et. al., KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning, 2025.
[2] Zhang et. al., Reinforcement-aware Knowledge Distillation for LLM Reasoning, 2026.
Questions for authors
- Figure 1 shows the highest pairwise disagreement rate at 32-64K prompt length. However, the training dataset limits the prompt length to less than 32K with a large portion of the prompts <= 16K. Can the authors share an ablation study with varying input length prompts? Also, does the disagreement rate drop after training?
- Can the authors report standard deviation for results in Table 1?
- During training, what fraction of correct responses within a group receive p < 0 and similarly, what fraction of incorrect responses receive p > 0 (equation 7)?
I thank the authors for the detailed rebuttal and for sharing additional experiments. Some of my concerns are addressed, and I am increasing my soundness score to 2. However, some of my concerns still remain.
Marginal gains. With three seeds, the average gains over OPD is 0.88 (4B) and 0.80 (8B) pts, are lower than the gains reported in the paper. The margins over the strongest baselines are also small: 0.39 over SG-OPD and 0.51 over ExOPD on 8B, and 0.49 over ExOPD on 4B.
Training with a 16K prompt limit gives nearly the same results as the 32K limit for both OPD and GC-OPD, and the gain over OPD is similar in both settings. This trend suggests the improvement might not tied to the longer prompts where Figure 1 shows the highest disagreement. Moreover, in the 32–64K bin, most of the reduction in disagreement comes from OPD itself, and GC-OPD has a lower mean response F1 than OPD. The rebuttal also states that in the two shorter bins, GC-OPD has a higher disagreement rate than the base model. These are the lengths the model is trained on.
On binary tasks, 9.9% of correct responses receive a negative residual and 11.0% of incorrect responses receive a positive residual. This suggests that GC-OPD can suppress correct responses and promote incorrect ones, which direct reward addition does not do. The ablations in Tables 2 and 3 are all single runs with small differences, making it unclear whether the signed residual is more effective than direct reward addition.
In light of the above listed concerns, I will retain my rating.