acceptodds
Withdrawn from ICLR 2027

Rethinking Sequence-Level Feedback for On-Policy Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sequence-level feedback can guide on-policy distillation beyond the current token, but estimating future teacher–student disagreement more accurately does not ensure better task decisions. We investigate this gap in mathematical reasoning by separating feedback estimation, gradient noise, and task utility. For candidate-based credit over a 16-token future window on Qwen3-4B, matched comparisons identify loss normalization as the source of an apparent gradient-variance increase under centering. Averaging more continuations reduces initial gradient variance by 80.3%, yet yields no measured training benefit over 40 updates across three seeds; reproducible candidate rankings likewise do not improve held-out correctness over the student's choices. These findings motivate a complementary use of feedback: selecting where to provide supervision, with reference solutions supplying the token targets. We formulate Reference-supervised OPSD (RS-OPSD), which retains distillation on student-generated prefixes and adds reference-token supervision on problems whose sampled answers fail verification. In single-seed evaluations, RS-OPSD exceeds OPSD's final four-benchmark macro accuracy by 1.76 and 0.84 percentage points for Qwen3-4B and Qwen3-8B, respectively. However, mean online differences are negative, and random reference selection reaches a similar 4B endpoint, leaving the advantage of failure selection unresolved. Together, these results show why feedback precision, task value, and supervision selection require distinct evidence.

Primary area:
reinforcement learning
Keywords:
On Policy Distillation, Self-distillation, Sequence-level feedback, Credit assignment

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_tHf6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies when sequence-level feedback in OPSD translates into improved parameter updates and downstream math reasoning performance. The authors introduce RS-OPSD, which combines on-policy self-distillation with verification-gated reference cross-entropy on failed problems. The experiments on Qwen3-4B and Qwen3-8B show improvements of RS-OPSD compared to standard OPSD on math reasoning benchmarks.

Critical strengths

RS-OPSD is simple and easy to implement.

Critical weaknesses
  1. The paper reads like it was mostly written by AI, without careful check by human. There is clear evidence about this, such as lines 347-350 or the last sentence in the AI use statement. The content lacks intuition and explanation.
  2. There are several problems with the design of the diagnosis, method, and experiments/evaluation, such as:
  • The diagnosis is not connected to the actual method. Moreover, different analyses use different protocols and prefix sets, which makes it hard to have a conclusion about the actual limitations of OPSD.
  • The proposed method is not well justified by the results. For instance, in Figure 4, random selection achieves essentially the same performance as error selection.
  • The evaluation is very limited: only 1 seed, no same-protocol step-0 scores (lines 351-353).
Questions for authors

See Weaknesses.

Reviewer_UVqaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the relationship between the estimation quality of sequence-level feedback, gradient stability, and task performance in on-policy distillation. Through gradient auditing, short-horizon training experiments, and independent continuation evaluations, the authors find that the increased variance—observed after sensitivity-weighted centering—is primarily driven by normalization discrepancies. Furthermore, neither reducing initial gradient noise nor obtaining stable rankings of candidate responses yielded demonstrable improvements in training or decision-making outcomes. The paper introduces RS-OPSD, which retains distillation based on student-generated prefixes while incorporating cross-entropy supervision—using reference solutions—for instances where response verification fails. Experiments with Qwen3-4B and Qwen3-8B show average accuracy improvements of 1.76 and 0.84 percentage points, respectively, across four benchmarks. However, the average online differential is negative, and a random-selection baseline for the 4B model achieves a similar final performance; consequently, the incremental value of the failed selections themselves remains unestablished.

Critical strengths
  1. Through comparative analysis, the specific source of the increased gradient variance was identified. By holding the continuation samples constant while independently controlling the normalization denominator and the effective prefix support, the study reduced the variance ratio of the sensitivity-weighted centering (relative to the original estimate) from approximately 1.835 to 0.986. This provides compelling evidence that the observed increase in variance stems primarily from differences in normalization. This finding offers significant diagnostic value: it prevents the misattribution of gradient scaling—caused by loss reduction—to inherent instability in the estimator itself.

  2. A clear distinction was established between the information content of feedback and the actual gains in decision-making performance. Candidate actions were scored and evaluated using an independent set of continuations. Furthermore, the study incorporated controls—comparing the student’s candidate policy against a scenario with randomized score redistribution—to disentangle whether the scores contain meaningful information about the actions, and whether that information can be leveraged to outperform the student’s existing choices. This experimental design mitigates the optimistic bias that arises when the same samples are used for both selection and evaluation, and it avoids conflating ranking stability with actual task performance. This approach is valuable for enhancing the interpretability of experimental results and provides a robust validation framework for assessing sequence-level feedback.

Critical weaknesses
  1. The gains reported for the proposed method depend on a baseline objective function that is inherently problematic; the incremental value over standard distillation has not been established. In the primary experiments, the OPSD implementation truncates each forward KL term individually at a lower bound of 10−610^{-6}, while retaining negative contributions. As the paper’s own counterexample demonstrates, this approach can cause gradient descent to increase the probability of tokens that the teacher model disfavors. Consequently, the improvements observed after incorporating the cross-entropy term with a reference solution may stem from compensating for a flawed distillation objective, rather than from the inherent advantages of the proposed method itself. The absence of a control experiment—comparing the proposed method against standard KL distillation (without this specific truncation) while keeping the teacher model and other training settings constant—means this alternative explanation cannot be ruled out. This issue directly undermines the validity of the core methodological contribution and serves as a primary basis for recommending rejection.

  2. The primary performance results lack verifiable underlying evidence, and an anomaly regarding model identity within the evaluation pipeline remains unresolved. The original experiments report only aggregated scores, lacking checkpoints or per-question outputs; thus, it is impossible to independently verify whether the correct adapters were loaded during generation. Furthermore, a subsequent round of re-testing revealed an anomaly where different adapter weights yielded byte-for-byte identical outputs—a phenomenon that remains unexplained. While the authors were correct to withdraw these anomalous re-test results, the existing records from the original experiments are insufficient to rule out similar issues. Although this anomaly does not definitively prove that the original scores are incorrect, it leaves the core performance comparisons unverified, as it remains unclear whether the observed differences truly stem from the intended models being correctly loaded. This casts doubt on the credibility of the central empirical findings; resolving this requires fixing the evaluation pipeline and conducting a full re-run of the key comparative experiments.

Questions for authors

Does the method’s performance gain depend on the specific pruning objective? Could you provide a cross-comparison—holding the teacher model, dataset, sampling strategy, and optimization settings constant—that examines the effects of "truncating individual item contributions (at a 10−610^{-6} threshold)" versus "incorporating (or omitting) supervision from reference solutions"? This would help determine whether the gains from RS-OPSD persist under an unpruned KL-divergence objective, or if they primarily stem from compensating for anomalous gradients.

Can the core performance results be verified through reproducible re-runs? Could you clarify why different adapters yielded byte-for-byte identical outputs in subsequent tests? Furthermore, after confirming that the intended adapters were indeed active during generation, could you provide the checkpoints, item-by-item outputs, and results from re-runs of key comparisons to establish whether the performance differences reported in the original main experiments are reproducible?

Does the "failure-based selection" strategy offer value independent of additional supervision from reference solutions? Given that, in the current 4B-model experiments, random selection and failure-based selection yield similar final scores, could you conduct a multi-seed comparison (for both 4B and 8B models)—ensuring that the number of tokens used for reference-solution supervision, the loss weights, and the training budget are all matched—to determine whether failure-based selection provides a reproducible and practically significant performance boost?

Reviewer_SkGwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is barely readable, and honestly I tried my best to read it but fail to comprehend what it's trying to do.

Critical strengths

None

Critical weaknesses

The paper is terribly written, to the point that I genuinely suspect that this is an AI slop (the work, or at least the writing, is produced entirely using LLM). Some of the signs that I noticed:

  • In page 10, there's this sentence: "Author verification pending: Human authors must verify the final text, references, and this disclosure before submission.", which strongly resembles a left-over from LLM-generated text.
  • In Section 5.1: "Minerva was excluded from this reporting scope at the authors’ request after results were available;"
  • Appendix H.4 also sounds like LLM-generated text from code, and barely has any meaning.
Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.