acceptodds
Under review as a conference paper at ICLR 2027

Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning

Abstract

On-policy distillation (OPD) trains a student on its own responses using dense token-level guidance from a stronger teacher. In long-context tasks, however, token-level teacher support can favor locally plausible responses that omit evidence distributed across the input or violate global task constraints. Task-specific verifiers, in contrast, evaluate task completion at the response level and may return graded rewards that reflect partial success. We diagnose this mismatch on fixed responses from two representative long-context evidence-aggregation tasks. Across the analyzed input-length ranges, trajectory-level OPD scores show weaker alignment with verifier rewards at longer lengths, indicating teacher–verifier disagreement. Motivated by this observation, we introduce Group-Calibrated On-Policy Distillation (GC-OPD). GC-OPD separately normalizes verifier rewards and trajectory-level OPD scores within each rollout group and uses their difference as a signed teacher–verifier disagreement residual. Relative-advantage-based credit assignment (RACA) distributes this trajectory-level residual across tokens according to their relative OPD advantages while preserving the original OPD signal. Across five long-context benchmarks, GC-OPD achieves the highest average score among the evaluated methods for both Qwen3-4B and Qwen3-8B, improving upon vanilla OPD under the same training and evaluation setup. Controlled ablations show that the signed residual is more effective than either an additional OPD-derived term or direct group-normalized verifier reward addition, while RACA further improves over uniform token allocation. Together, these results demonstrate that group-relative residual calibration can incorporate verifier outcomes without discarding dense token-level guidance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, OPD, GRPO, Long Context Modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_64tjOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper argues that for long-context tasks trajectory level OPD score has weak alignment with verifier based rewards. They motivate this teacher-verifier disagreement on two representative long-context tasks by capturing the pairwise disagreement rate as a function of increasing prompt length. To address this disagreement, the authors introduce GC-OPD, which separately normalizes verifier rewards and trajectory level OPD scores for each group of rollouts and use their difference as signed teacher-verifier disagreement residual. Relative-advantage-based credit assignment (RACA) is used for distributing this residual score across tokens according to their relative OPD advantage. Experiments across two model scales and five benchmarks show higher average scores compared to baselines.

Critical strengths
  • The motivation analysis quantifies pairwise disagreement between trajectory level OPD scores and verifier rewards and offers new insights on OPD for long-context tasks.
  • The proposed method, GC-OPD, is simple and does not require additional rollouts/forward passes.
Critical weaknesses
  • The performance gap between GC-OPD and baselines appear marginal and within noise. Table 6 in appendix reports the use of 3 random seeds. However, all results in the main paper report a single number without standard deviation or confidence intervals. The absence of standard deviation makes it difficult to assess if the gains over baselines are statistically significant or simply noise.
  • Missing comparison against key baselines. The proposed method, despite using verified reward in its formulation, does not compare against GRPO or stronger verifier-aware OPD baselines like [1,2]. SG-OPD and Reward-Aware OPD are discussed in related work section, but not evaluated.
  • Untuned baselines. Section 5 (lines 385-386) states the use of fixed 231 prompts disjoint from training set for tuning the beta coefficient. However, the baselines are not subjected to any hyper-parameter tuning and use the standard default configuration. To substantiate, ExOPD uses the default lambda = 1.25 (lines 889-890). 



References

[1] Xu et. al., KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning, 2025.


[2] Zhang et. al., Reinforcement-aware Knowledge Distillation for LLM Reasoning, 2026.

Questions for authors
  1. Figure 1 shows the highest pairwise disagreement rate at 32-64K prompt length. However, the training dataset limits the prompt length to less than 32K with a large portion of the prompts <= 16K. Can the authors share an ablation study with varying input length prompts? Also, does the disagreement rate drop after training?
  2. Can the authors report standard deviation for results in Table 1?
  3. During training, what fraction of correct responses within a group receive p < 0 and similarly, what fraction of incorrect responses receive p > 0 (equation 7)?
1 comment

I thank the authors for the detailed rebuttal and for sharing additional experiments. Some of my concerns are addressed, and I am increasing my soundness score to 2. However, some of my concerns still remain.

  1. Marginal gains. With three seeds, the average gains over OPD is 0.88 (4B) and 0.80 (8B) pts, are lower than the gains reported in the paper. The margins over the strongest baselines are also small: 0.39 over SG-OPD and 0.51 over ExOPD on 8B, and 0.49 over ExOPD on 4B.

  2. Training with a 16K prompt limit gives nearly the same results as the 32K limit for both OPD and GC-OPD, and the gain over OPD is similar in both settings. This trend suggests the improvement might not tied to the longer prompts where Figure 1 shows the highest disagreement. Moreover, in the 32–64K bin, most of the reduction in disagreement comes from OPD itself, and GC-OPD has a lower mean response F1 than OPD. The rebuttal also states that in the two shorter bins, GC-OPD has a higher disagreement rate than the base model. These are the lengths the model is trained on.

  3. On binary tasks, 9.9% of correct responses receive a negative residual and 11.0% of incorrect responses receive a positive residual. This suggests that GC-OPD can suppress correct responses and promote incorrect ones, which direct reward addition does not do. The ablations in Tables 2 and 3 are all single runs with small differences, making it unclear whether the signed residual is more effective than direct reward addition.

In light of the above listed concerns, I will retain my rating.

Reviewer_M3i7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper provides an interesting way to provide calibration on the token allocation by combining outcome reward with dense distillation. They address the gap between teacher accuracy and task arruacy and propose GC-OPD, which calibrates the token level supervison with a standardized residual between OPD signals and verifier rewards. They report performance on five long-context benchmarks and that their proposed approach leads in overall benchmark performance.

Critical strengths
  1. They address an important gap between teacher accuracy and task accuracy, and offer an intuitive design that calibrates the token level supervison with the gap between OPD signals from group-normalized verifier rewards. The calibrations fits and enables partial graded reward and offer fine-grained supervision to distinguish different degrees of success
  2. They also provide additional ablation study over varied token allocation and show the effectiveness of their residual design.
Critical weaknesses
  1. The robustness of the proposed method remain insufficiently established. From Table 1, we see that the gains mainly comes from CorpusQA, especially for 4B model where the performance on CorpusQA accounts for almost the entire net improvement.
  2. The contribution over existing methods needs clearer justification. Uni-OPD already identifies disagreement between trajectory-level OPD scores and correctness. GC-OPD further introduces response-specific residuals between them and further distribute across tokens. Also, the paper does not isolate whether using graded rather than binary rewards contributes to its advantage over Uni-OPD.
  3. The residual comparison does not independently tune the direct-reward baseline. Table 2 compares residual calibration with direct reward addition, but both use the coefficient selected for GC-OPD. Since the correction signals can differ in magnitude, different setup may favor different optimal config. It’s unclear that GC-OPD is better than the baselines given a similar tuning budget.
  4. The experiments do not connect the long-context with the observed improvement. Figure 1 shows increasing OPD–verifier disagreement with input length, but the downstream results dont report on varied length. It’s unclear how the method helps addressing the specific failure pattern identified in Figure 1.
Questions for authors
  1. Why is the performance gain mainly saturated on CorpusQA? Will this narrow the experiment scope? Can you provide analysis on why the improvement is only significant on CorpusQA but not others?
  2. Can you report GC-OPD’s gains over other baselines with different context-length, and clarify whether larger gains truly occur where disagreement is greater?
  3. Since the performance gap is relatively small compared with baselines, are the baselines trained with their optimal configuration for fairer comparison?
  4. How would you isolate the benefit of partial graded reward from binary reward like Uni-OPD? This would help establish the importance of this distinction from binary outcome calibration.
Reviewer_q18AOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Group-Calibrated On-Policy Distillation (GC-OPD), a framework that incorporates verifier feedback into token-level on-policy distillation for long-context reasoning. The method constructs a signed residual between group-normalized verifier rewards and trajectory-level OPD scores, and introduces relative-advantage-based credit assignment (RACA) to distribute this residual across tokens. Experiments on five long-context benchmarks with Qwen3-4B and Qwen3-8B demonstrate improvements over conventional OPD baselines, supported by component ablations and teacher–verifier disagreement analyses.

Critical strengths

S1. The paper identifies a meaningful limitation of conventional OPD, where token-level teacher preferences may not align with verified task outcomes in long-context reasoning. The disagreement analyses on Multi-Table Extraction and High-Recall Retrieval provide useful empirical motivation for incorporating response-level verifier feedback.

S2. GC-OPD combines group-normalized verifier rewards with trajectory-level OPD scores through a signed residual while preserving the original token-level supervision. The proposed calibration can be integrated into existing OPD training pipelines without additional teacher or student forward passes, assuming verifier rewards are already available.

S3. The paper compares residual calibration against direct verifier reward addition and investigates different token-credit allocation strategies. These experiments help distinguish the contributions of the trajectory-level residual and the RACA mechanism.

Critical weaknesses

W1. My primary concern with this paper is the empirical performance of the proposed method. Although GC-OPD achieves the highest average scores, its improvements over vanilla OPD are only 1.16 and 1.09 points for Qwen3-4B and Qwen3-8B, respectively, with most gains coming from CorpusQA. Improvements on several other benchmarks are marginal or negative, raising concerns about the consistency of the proposed calibration across different long-context capabilities.

W2. The motivation relies on increasing teacher–verifier disagreement at longer input lengths, but training is restricted to 32K tokens while evaluation extends to 120K, without length-stratified performance analyses. Consequently, it remains unclear whether GC-OPD specifically mitigates the disagreement observed at longer contexts or primarily improves general response-level optimization.

W3. GC-OPD disables residual calibration when either group-level signal has negligible variance, which may occur frequently for binary verifier rewards when all sampled responses receive the same outcome. Given the substantial proportion of binary-reward tasks in the training mixture, the paper would benefit from analyzing how often calibration is activated and how sensitive its effectiveness is to rollout-group size and task characteristics.

Questions for authors
  1. Could the authors provide performance comparisons across different context-length ranges, particularly beyond 32K tokens? It would also be useful to examine whether GC-OPD actually reduces teacher–verifier disagreement after training.

  2. How frequently is residual calibration disabled due to insufficient within-group reward or OPD-score variation? Could the authors analyze this behavior across different task families and rollout-group sizes?

1 comment

Thank you for your responses and new results. I think most of my concerns have been addressed. I therefore change my assessment to weak acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.