- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates reasoning distillation for large language models. By constraining the student model, specifically, by minimizing the negative log-likelihood of the teacher’s output across all generated prefixes, the approach limits the student’s deviation from the teacher’s distribution while simultaneously optimizing for task rewards. The authors reformulate this constraint as a penalty for violations and propose a policy gradient decomposition, which allows the student to perform reasoning without requiring access to the teacher model during inference. Both mathematical analysis and empirical experiments demonstrate that this method improves adherence to teacher-imposed constraints and yields gains in reasoning success rates across the evaluated settings.
Critical strengths
This paper focuses on the phenomenon where a final answer is correct despite errors in the intermediate reasoning. It reports the answer accuracy, teacher constraint satisfaction rate, and reasoning success rate separately, which helps to distinguish performance across different dimensions. The experiments cover two model families, and the paper provides details on training configurations, prompts used for evaluation, and a small-scale manual review, making the assessment process easier to understand.
Critical weaknesses
1. Research Significance and Problem Framing
(1) The paper has yet to sufficiently establish its scientific contribution relative to existing research. The paper motivates its approach by highlighting cases where the final answer is correct despite flawed intermediate reasoning, pointing to the limitations of outcome-only reinforcement learning (RL) and "average teacher" consistency. However, the distinction between outcome supervision and process supervision, including the identification of erroneous reasoning steps and the use of teacher feedback to improve credit assignment in RL, is already a well-explored area. The paper must clearly articulate what specific, previously unaddressed problem it identifies and resolves, beyond what is already covered by existing methods in process supervision, selective distillation, and reward-aware distillation. Currently, the problem framing relies on a single example where the final answer happens to be correct, along with data on teacher cost distributions and limited evaluations by a "judge" model. While this evidence demonstrates that final-answer metrics are insufficient, it fails to establish that "compensation mechanisms within average supervision" constitute the primary bottleneck for current reasoning distillation, nor does it prove that the paper’s proposed "prefix constraints" are a necessary mechanism to overcome this bottleneck. The claims regarding the research gap are significantly stronger than the evidence provided.
(2) Improvements in "teacher consistency" have not been sufficiently translated into meaningful, independent gains in reasoning reliability. The most notable experimental gains appear in "Constraint Satisfaction", a metric derived directly from the teacher-cost statistics used during the model's training. While this confirms that the method successfully optimizes its surrogate objective, it does not independently prove that the mathematical reasoning has become more accurate or reliable. The RSR (Reasoning-Step Reliability/Correctness) metric is more closely aligned with the paper’s core application goals; however, the gains over strong baselines are generally modest, and the results lack repeated experiments or uncertainty estimates. Consequently, the current evidence supports the claim that the method can "adjust the relationship between accuracy and teacher likelihood," but it falls short of justifying broader conclusions, such as "significantly enhancing trustworthy reasoning" or "overcoming a fundamental trade-off." The practical value of these improvements to surrogate metrics needs to be substantiated through independent evaluations of process correctness, robust hidden-test-set evaluations, or formal verification.
(3) The assumption that "a single error invalidates the entire chain of reasoning" requires a more precise definition of its scope. In the context of a final mathematical proof, a critical inferential error can indeed render the entire proof invalid. However, the chain-of-thought generated by a model may include exploratory steps, backtracking, self-corrections, and discarded branches. Making an error during the exploration process is not the same as having a final conclusion that lacks valid support. The paper applies the "weakest-link" intuition to the entire token-generation process, evaluating the output based on the binary question, "Does it contain any errors?", without adequately distinguishing between uncorrected critical logical flaws, exploratory steps that were subsequently rectified, and redundant content that does not affect the final derivation. Consequently, the paper’s optimization and evaluation objectives may skew toward conservative, formulaic explanations, which do not necessarily reflect stronger, genuine reasoning capabilities.
2. Related Work and Baselines
A more substantive limitation of this paper is the lack of clear connections to, and fair comparisons with, representative, reliable methods in areas such as teacher supervision, reward-aware distillation, and fine-grained credit assignment. Consequently, the existing evidence is insufficient to substantiate the added value of the proposed constraint mechanism over prior research.
(1) The omission of relevant work, specifically research on the reliability of teacher supervision, reward-aware distillation, and fine-grained credit assignment, results in an inadequate set of baselines.
The paper primarily compares its approach against GKD, MiniLLM, GRPO, and a variant of GKD-GRPO with fixed weights, using these to highlight the tension between accuracy and teacher consistency. However, the existing literature has moved well beyond simple combinations involving fixed KL-divergence weighting; it now encompasses dynamic scheduling, selective imitation, the calibration of teacher reliability, and token-level reward attribution. The current set of comparisons covers only a fraction of this research landscape, which is insufficient to support broad claims regarding the limitations of existing methods or the advantages of the proposed approach.
The following works are closely related to the problem setting, technical motivation, or optimization objectives of this paper, yet they are neither cited nor included in the experimental comparisons:
-
KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning (2025). KDRL jointly optimizes GRPO and teacher reverse-KL supervision, investigating coefficient scheduling and reward-guided masking. A fixed-weight GKD-GRPO baseline therefore does not adequately represent existing approaches to combining distillation and reinforcement learning.
-
Reinforcement-aware Knowledge Distillation for LLM Reasoning (RLAD) (2026). RLAD addresses interference between teacher supervision and reward optimization through selective imitation using a trust-region ratio objective. Comparing against it would help establish whether the proposed method improves upon reward-aware policy optimization beyond simple KL weighting.
-
Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe (2026). Uni-OPD improves the reliability of trajectory-level teacher supervision through outcome-guided margin calibration. It directly relates to the paper’s objective of reconciling teacher consistency with task success.
-
Reward-Gated On-Policy Distillation (RG-OPD) (2026). RG-OPD uses verifier feedback and teacher–student likelihood differences to determine which trajectories receive teacher supervision. It provides a relevant comparison for filtering misleading teacher signals.
-
When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation (RA-OPD) (2026). RA-OPD checks whether trajectory-level distillation returns align with outcome rewards and filters unreliable updates. Its reward-alignment mechanism should be distinguished from and compared with the proposed fixed teacher-cost boundary.
-
Teacher-Guided Policy Optimization for LLM Distillation (TGPO) (2026). TGPO provides directional teacher guidance on student-visited prefixes and integrates it with RLVR. It is directly relevant to the paper’s claims concerning token-level guidance and stable optimization.
-
Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation (TSD-KD) (2026). TSD-KD combines teacher preference feedback with token-selective distillation to address limited student capacity and overly restrictive imitation. This comparison would help determine whether the proposed constraint offers benefits beyond selective supervision of important tokens.
-
TRAC: Teacher-Guided Token Reward with Adaptive Calibration for Robust Policy Optimization (2026). TRAC derives token-level rewards from teacher–student log-probability differences and calibrates teacher reliability at the problem, trajectory, and token levels before incorporating these signals into GRPO. It provides a relevant comparison for assessing whether a fixed teacher-cost boundary offers advantages over adaptively calibrated teacher supervision.
-
REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation (2026). REOPD uses token-level reliability weights and a batch-level adaptive budget to mitigate reward hacking and instability during reward extrapolation. Although its setting differs, the paper should discuss how its constraint mechanism differs from this reliability-adaptive approach.
-
Unlocking Token Rewards via Training-Free Reward Attribution (2026). This work derives token-level rewards from an existing differentiable reward model through gradient-based attribution, without training an additional token reward model. It provides a relevant comparison for evaluating fine-grained credit assignment and optimization efficiency, while accounting for its different supervision requirements.
The authors should, at a minimum, conduct fair comparisons with closely related methods. Otherwise, it remains unclear whether the observed gains stem from the unique design of the paper’s prefix-based constraints, or from more general techniques such as selective teacher feedback, reward calibration, or dense supervision.
(2) The discussion on process supervision and the distillation of reasoning structures is insufficient, and the analysis of associated costs is overly vague. While this paper aims to improve the effectiveness of intermediate reasoning, the experiments focus primarily on distribution distillation and reinforcement learning (RL) based on final answers; they lack direct comparisons with representative methods that target process correctness or reasoning structures. Relevant studies that warrant discussion include:
- Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations (ACL 2024): This work employs automatically constructed process supervision to train a reward model, which is then used for step-by-step RL, demonstrating that process supervision does not necessarily require manual, step-by-step human annotations.
- Free Process Rewards without Process Labels (2024): This research explores deriving implicit process-level rewards from outcome-level labels, a topic directly relevant to the paper’s framing of the costs associated with process supervision.
- RLKD: Distilling LLMs’ Reasoning via Reinforcement Learning (preprint, 2025). This approach employs reinforcement learning (RL) distillation guided by a reward function aligned with reasoning structures; it focuses on the transfer of reasoning patterns rather than merely ensuring consistency in the output text distribution.
- Reinforcement Learning via Self-Distillation (SDPO, 2026). This method derives dense learning signals through a self-teaching mechanism conditioned on feedback, offering an alternative approach to addressing sparse rewards and credit assignment. However, the specific nature of its supervisory conditioning differs from that proposed in this paper.
- On Teacher Hacking in Language Model Distillation (ICML 2025). While the specific setup differs from the current work, it shares a significant conceptual link regarding the risk that an agent might exploit flaws in the teacher model when optimizing against the teacher’s objectives.
The authors might reasonably highlight that their approach avoids training an additional Process Reward Model (PRM); however, they must justify this design choice by demonstrating its advantages in terms of performance, training costs, or the demand for supervisory resources. Characterizing other process-supervision methods as inherently reliant on expensive annotations, additional model training, or complex architectural modules fails to fully account for existing research on automated process supervision and low-cost, fine-grained reward mechanisms.
(3) The discussion of existing related work is insufficient to substantiate the technical claim that "hard constraints are the key to solving the problem."
Existing literature already offers various methods for integrating teacher feedback with task-specific rewards, including dynamic weighting, reward-aware masking, reliability gating, selective imitation, and token-level credit assignment. Consequently, the paper needs to clarify what specific value the "worst-case prefix constraint" adds beyond these established mechanisms. The current experiments simultaneously vary the definition of teacher costs, sequence aggregation methods, penalty formulations, and gradient estimation techniques. Furthermore, the lack of comparison with the aforementioned representative methods makes it impossible to rule out the possibility that simpler approaches, such as reliability calibration or localized supervision, could yield comparable gains.
In particular, it is essential to conduct comparisons using identical teacher models, student models, training datasets, and computational budgets. This should include representative baselines such as fixed soft regularization, adaptive teacher supervision, and process-level or token-level reward methods. Where feasible, different penalty formulations should be evaluated using consistent cost metrics. Only through such rigorous comparison can the observed performance differences be attributed specifically to the prefix constraint, rather than to variations in the intensity of teacher supervision or other implementation details.
3. Methodology
(1) The core prefix constraint fails to achieve the "protection of the weakest step" implied by the paper; it still permits significant local compensation. The constraint imposed in this paper is: , which is neither a constraint on the maximum cost of a single step nor a guarantee of the validity of individual reasoning steps.
A straightforward counterexample: let . Suppose the first 100 tokens each have a cost of 0.01, while the 101st token has a cost of 100. The average cost for all preceding prefixes is 0.01, and the average cost for the final prefix is:
Consequently, even with a local token of extremely high cost, the sequence still satisfies the paper’s prefix constraints. Low-cost text earlier in the sequence effectively "accumulates a buffer" that can be used to offset high-cost behavior later on.
Proposition 3.1, which states , further demonstrates that the allowable deviation for a single step grows linearly with its position in the sequence. For an error occurring near the end of a sequence, the worst-case upper bound remains . While the paper successfully prevents a prefix that has already exceeded the threshold from being "repaired" by subsequent text, it does not eliminate local compensation in a general sense, nor does it provide an upper bound on local risk that is independent of sequence length.
This distinction directly affects the validity of the paper’s title, motivation, and primary contributions. The authors should refine their claims regarding "anti-compensation" and investigate whether the proposed method is biased toward suppressing early-stage anomalies while remaining relatively insensitive to logical errors occurring later in the sequence.
(2) The link between the teacher’s negative log-likelihood (NLL) and logical validity is insufficient to support an interpretation based on "hard constraints. The teacher’s NLL measures the statistical predictability of a token, not the correctness of a mathematical step. High-probability generations may contain common errors, while low-probability generations could represent correct, albeit unconventional, solutions. Furthermore, this statistic is influenced by factors such as formatting, phrasing, the teacher model’s entropy, and the choice of tokenizer. For a fixed prefix, we have: & H(\pi(\cdot\mid s)) + D_{\mathrm{KL}}(\pi(\cdot\mid s)\Vert\mu(\cdot\mid s)).& Therefore, reducing the teacher's NLL is not equivalent to minimizing the teacher-student KL divergence; the student's conditional entropy also plays a role. Interpreting NLL directly as a measure of the degree of logical violation lacks sufficient justification.
Appendix E reports a Spearman correlation coefficient of 0.231, and a pooled AUC of 0.63 for samples where the model's answer is correct. This indicates that the proxy metric carries some information, but it is insufficient to establish it as a highly reliable signal for detecting logical violations. Furthermore, statistical significance should not be conflated with discriminative power. It is necessary to control for factors such as sequence length, stylistic expression, question difficulty, and formatting, and to analyze specific cases—such as those where the model is correct despite a low probability assigned by the teacher, or incorrect despite a high probability assigned by the teacher.
(3) Some of the structural contributions are not unique to the "prefix-average" constraint proposed in this paper; the claims regarding novelty are overstated. The principle that "once a violation occurs, the sequence is deemed infeasible" stems from tracking historical violations and does not inherently require a "prefix-average" formulation. Similarly, a cumulative sum of non-negative costs that exceeds a defined budget is also irreversible. The justification provided in Appendix D for excluding sliding-window constraints is also inadequate. While statistics from a current window might recover, one could instead constrain the "maximum cost observed across all historical windows up to the present." This construction would also ensure that historical violations remain irreversible, while more effectively targeting localized anomalies. Likewise, when the state is defined as the complete history, any budget variable derived from that history can be considered redundant within the mathematical state representation. This observation applies more broadly than just to the constraints proposed in this paper; the authors need to more clearly distinguish between general historical state representations, existing penalty-based constructions in safe reinforcement learning, and the specific technical contributions that are truly novel to this work.
(4) The fact that this approach is not equivalent to a fixed "soft" penalty does not, in itself, prove the necessity or superiority of the "hard" boundary design proposed in this paper. Proposition 3.1(b) compares a scalar soft penalty, based on cumulative cost, with a constraint based on the average cost of the worst-case prefix. Since the two approaches employ different cost functionals, it is not surprising that their optimal policies differ. This result does not preclude the use of soft penalties, hinge penalties, adaptive multipliers, or other risk-sensitive objectives applied to the same . For instance, the expression $ R(\tau)-\lambda[W(\tau)-d]_+$ could also be used to impose a penalty based on the same prefix-violation statistic.
The current experiments simultaneously vary the cost definition, the aggregation method, and the form of the penalty. Consequently, it is impossible to isolate whether the observed gains stem from the prefix statistics, the teacher’s NLL, the strong penalty, or the so-called "hard-constraint semantics." To substantiate the core design, comparisons must be conducted while keeping the cost definitions and statistics constant.
(5) There remains a significant gap between theoretical guarantees and practical training. Theorem 3.3 relies on conditions such as an infinite penalty () and the existence of an optimal policy with finite value. In contrast, the experiments employ a finite penalty (), parametric models, and approximate optimization via GRPO. The theorem provides no convergence guarantees for the actual algorithm, no bounds on the probability of constraint violation under finite penalties, and no guarantees regarding constraint satisfaction on the test distribution.
The paper itself acknowledges a decline in constraint satisfaction rates during testing. Therefore, claims such as "strictly safe during training," "satisfied almost surely," or "reliably preventing erroneous reasoning" require precise qualification to avoid erroneously attributing properties of a limiting MDP to the actual model. Furthermore, the argument that Lagrangian methods are "computationally infeasible" is insufficiently supported. Once the teacher’s costs are fixed, updating a small number of multipliers does not inherently necessitate expensive inner-loop optimization involving the teacher. If computational efficiency is a key advantage of this work, the authors should provide a baseline using adaptive multipliers, along with measurements of the associated overhead.
(6) The scope of the claims regarding variance reduction is limited, and details concerning the actual optimization process are insufficient. Appendix A.4 demonstrates, via conditional expectation, that the variance of the immediate gradient components is reduced, a valid application of the Rao–Blackwellization argument. However, the complete estimator also incorporates long-term return terms, as well as the covariances between these various components. Consequently, a reduction in the variance of individual immediate terms does not automatically imply a reduction in the total variance of the full policy gradient estimator, nor does it directly guarantee that GRPO training is more stable.
The paper does not report the variance of the full gradient, stability across different random seeds, or ablation studies involving the removal of the "immediate" (SINGLE) or "long-term" (LONG) components. Furthermore, while the main text presents the REINFORCE decomposition, the appendix mentions the use of in-group normalization and clipping; however, it fails to provide a clear explanation of how these mechanisms are integrated into the final loss function.
This detail is particularly significant for groups where all trajectories result in a violation (e.g., failing a constraint). If every trajectory incurs the same terminal penalty, the standard in-group relative advantage might be zero. It remains unclear whether the "immediate" terms can still provide meaningful updates in such cases, and how exactly they are reconciled within the GRPO framework, a point that requires both algorithmic clarification and empirical validation.
(7) Characterizing post-hoc threshold scanning as a simplification of hyperparameter selection during training conflates two distinct operations. Appendices E and G suggest that because the cost of each trajectory is recorded, one can perform a post-hoc scan of the threshold (), which is framed as being easier than selecting a soft regularization coefficient. However, filtering the outputs of a fixed model is fundamentally different from training distinct policies using different thresholds (). A training-time threshold alters the policy updates and the subsequent sampling distribution, effects that cannot be fully replicated by post-hoc filtering of a static set of trajectories.
Moreover, if such filtering were applied during deployment, it would necessitate computing the "teacher" cost, a process with resource requirements that differ from a scenario where a "student" model generates outputs independently (without invoking the teacher). The authors should clearly distinguish between hyperparameter selection during training, offline analysis, and deployment-time filtering; the ability to filter outputs should not be conflated with an advantage in training-time hyperparameter tuning.
4. Experiments
(1) The coverage of models and tasks is insufficient to support broad conclusions regarding current reasoning distillation methods. All primary student models are drawn from the Qwen2.5 and Llama-3.2 families, with parameter counts concentrated in the 0.5B–3B range; furthermore, both teacher and student models belong to the same model family. The study fails to verify whether the proposed methods remain effective when applied to newer models, established reasoning-oriented student models, or scenarios involving varying scales and capability gaps between teacher and student models.
While older models may be suitable for mechanistic research, this paper proposes a general-purpose reasoning distillation method, which necessitates more representative validation. Given that the Qwen3 series already includes small-scale models (e.g., 0.6B, 1.7B, and 4B), incorporating experiments with newer, comparably sized models would not necessarily require the use of massive-scale models.
The release of the Qwen3.5, 3.6, and 3.8 series by the Qwen team further highlights the temporal gap between this work and the current state of model development. At a minimum, the experimental evaluation should demonstrate that the proposed method yields performance gains on newer models with comparable computational requirements.
Mathematical evaluation is limited to the MATH and GSM benchmarks, while code generation is tested exclusively on MBPP. Since MATH remains challenging for the small-scale models used in this study, one cannot simply assume that these tasks have reached saturation. The evaluation lacks more demanding mathematical benchmarks (such as AIME or OlympiadBench), complex coding tasks, and scenarios requiring long-chain reasoning. Given that a core motivation of this paper concerns how local errors may be obscured by long sequences, the current evaluation lacks sufficient stratification by sequence length or specific validation for long-chain reasoning.
(2) The performance gains over relatively strong baselines are marginal, and there is a lack of statistical evidence to support claims of a significant advantage. In Table 1, the RSR (Reasoning Success Rate) achieved by the best configuration in this paper shows only modest improvements over existing strong baselines: On Qwen/MATH, the proposed method reaches 30.00%, compared to 29.48% for GKD-GRPO, an absolute gain of 0.52 percentage points. On Llama/MATH, it achieves 14.12%, versus 13.64% for SFT on teacher-generated CoT, a gain of 0.48 percentage points. On Qwen/GSM-Symbolic, the RSR increases from 64.36% (with GRPO) to 65.82%, a gain of 1.46 percentage points. On MBPP, the proposed method achieves 34.2%, compared to 34.0% for GKD-GRPO, a gain of merely 0.2 percentage points.
While these point estimates may have some value, the absence of results across multiple random seeds, confidence intervals, or paired statistical tests makes it impossible to determine their stability. In the case of MBPP, a 0.2 percentage-point difference corresponds to just one single problem out of 500. The improvements in "Chain-of-Thought Success" (CS) relative to pure GRPO, observed via this proxy metric, cannot substitute for rigorous statistical validation of gains in core reasoning success rates. The current evidence is insufficient to support strong claims such as "significant" or "comprehensively superior."
(3) The evaluation of core process quality relies too heavily on a single "judge" model, and the pairwise win rates do not consistently favor the proposed method across all experimental settings.
The RSR is determined by the DeepSeek-R1-Distill-Qwen-32B model. Although the authors conducted a manual review of 60 samples and employed a "double-order" pairwise comparison, which are valuable control measures, these steps remain insufficient to verify subtle differences across various methods and difficulty levels.
Further clarification is required regarding the number of human annotators, the sample sizes for each stratified subset, the specific types of errors encountered, and any potential biases the judge model may exhibit concerning response length, style, or the self-correction process. Process-based evaluation can also leverage expert-annotated data, such as that found in ProcessBench, to calibrate "judge" models or serve as a proxy for the cost of human (teacher) feedback. While such data is valuable for validating evaluation tools, it cannot serve as a direct substitute for assessing generative capabilities.
Furthermore, pairwise comparisons are conducted exclusively on samples where both methods provide correct answers; this constitutes a conditional evaluation, and both the intersection of the sample sets and the number of valid comparisons must be reported. In Figure 4, GKD-GRPO achieves a win rate of approximately 67.7% on the Qwen/MATH benchmark, outperforming the ~56% win rate reported for the method in this paper. Consequently, the results do not support the claim that the proposed approach consistently yields the highest reasoning quality across all settings.
(4) The reward evaluation in the code-generation experiments is highly susceptible to exploitation, which undermines its validity as evidence of general-purpose coding proficiency.
The coding prompts include unit tests, and the generated code is executed alongside these test assertions. The authors report that GRPO achieves a pass@1 score of 92.0% by leveraging these visible tests; however, the RSR (Robust Success Rate) is only 1.8%. This indicates that the primary execution-based metrics have been heavily exploited.
While this environment could serve as a stress test for "reward hacking," a rigorous assessment would require independent hidden test sets, isolated test evaluators, and a detailed analysis of specific "cheating" behaviors to determine whether the model has truly acquired generalizable programming capabilities. The paper also fails to adequately explain the underlying mechanism that allows the model to pass tests simply by reproducing the test code itself.
Compared to a relatively strong "soft-regularization" baseline, the proposed method shows an RSR increase of only 0.2 percentage points (from 34.0% to 34.2%), while the reported pass@1 actually declines from 50.4% to 45.6%. Therefore, these experiments fail to provide robust evidence for cross-domain advantages, nor do they support the broader claim that the method maintains high accuracy through pure reinforcement learning (RL) in the coding domain. Furthermore, a specialized evaluation protocol for RSR in coding tasks is required.
(5) The study lacks critical ablation analyses to identify the sources of performance gains; additionally, Figure 2 does not represent a direct comparison of constrained training approaches. The current experiments fail to sufficiently disentangle the individual contributions of the following factors:
- The choice between teacher negative log-likelihood (NLL) and Kullback-Leibler (KL) divergence-based costs.
- The differences between averaging costs over the "worst-case" prefix versus the entire sequence, as well as the use of maximum - single-step costs versus maximum costs over a sliding historical window.
- Discontinuous penalty for violations versus continuous/adaptive penalties based on the same statistic.
- "SINGLE" (exact expectation term) versus "LONG" (cumulative return term).
- Penalty strength (), training budget (), and generation length.
Specifically, Figure 2 is cited in the main text to argue that "worst-case prefix constraints" are superior to "cumulative budget" constraints. However, Appendix D reveals that this experiment involved sampling 256 trajectories per problem and subsequently applying different filters to the same pool of candidates. It does not compare the policies resulting from training under each constraint, nor does it evaluate end-to-end performance given an identical generation budget.
Furthermore, RSR (Resolved Success Rate) after filtering must be reported alongside the acceptance rate; otherwise, one could artificially inflate the conditional success rate simply by rejecting a large number of samples. This figure is insufficient to demonstrate that the proposed training mechanism outperforms training based on a cumulative budget.
(6) There is an internal consistency issue between the key figures in Appendix E and the main tables that requires urgent clarification. Appendix E states that the analysis is based on 12 training runs using Qwen/MATH, with 5,000 generations per run, totaling 60,000 trajectories, and defines RSR as cases where the "final answer is correct" and the "judge identifies no errors." The authors report that when samples are divided into quartiles based on teacher likelihood, the RSR increases monotonically from 30.1% in the lowest quartile to 59.6% in the highest. Assuming equal sample sizes across the four quartiles and a strictly monotonic increase, the overall RSR should be at least: However, in Table 1, the highest RSR reported for any training method on Qwen/MATH is only 30.00%. Similarly, Appendix E reports an FAC (Final Answer Correctness) that increases monotonically from 35.9% to 63.0%, implying an overall FAC of at least 42.675%; yet, the highest FAC reported for this configuration in the main table is only 34.86%.
If Appendix E corresponds to the same experimental runs, testing protocols, and metric definitions as those in the main table, these figures cannot both be correct. The authors must clarify whether different checkpoints, sample selection criteria, denominators, or evaluation settings were used, and provide reproducible subgroup counts and aggregate metrics. This issue directly concerns the core evidence supporting the validity of the "teacher-likelihood proxy." Until the methodology is clarified, this appendix cannot be considered a valid validation of the proxy. This observation pertains solely to the consistency of the reported data; no conclusions are drawn regarding the underlying cause.
(7) Insufficient information regarding reproducibility and computational fairness.
While the manuscript reports basic hyperparameters, such as learning rate, batch size, and number of epochs, several details critical to interpreting the results are missing. These include the exact size and partitioning of the mathematical training dataset, precise identifiers for the model checkpoints, training prompts, decoding settings for mathematical testing, random seeds, the full set of GRPO hyperparameters, and the specific implementation of the loss function involving decomposed gradients.
It should also be clarified whether the 3-round GKD (Generalized Knowledge Distillation) warm-up phase for Llama is included in the 20-round training total. Given that SFT (Supervised Fine-Tuning) uses 4 rounds and RL (Reinforcement Learning) uses 20, the number of epochs alone is insufficient to determine whether the comparison is fair; the total number of training tokens, as well as the computational budgets for generation and teacher-model inference, must be reported.
The initial FAC for student models in the tables is only 0–0.54%. It is necessary to distinguish between cases where the model fails to solve the problem correctly and cases where it fails due to answer formatting or extraction errors. The authors should report the "format compliance rate" and provide more robust answer-matching results.
Finally, the claim that the process takes "less than 23 hours on a single accelerator" lacks details regarding the hardware model, VRAM capacity, the actual computational cost of the teacher-model forward passes, and the total overhead of the proposed method; thus, the stated efficiency advantage is not adequately substantiated. For algorithms that rely on penalty-based sequencing, full-vocabulary immediate terms, and intra-group normalization, general training descriptions are insufficient to independently reproduce the results or verify whether the implementation aligns with the underlying theory.
Questions for authors
It is recommended to provide a point-by-point response to each of the issues raised in the "Weaknesses" section.
6 comments
2(1). Ten missing baselines. The review says we must "at a minimum" run fair comparisons against these. Here is what they are:
- TSD-KD (ICLR 2026). It is teacher-only distillation: the student trains on teacher signals with no task reward, like GKD and MiniLLM, which are in our Table 1 and among its own baselines. Its contribution is which tokens receive the teacher signal. Ours is on another axis: how the teacher term is aggregated and enforced once a task reward is present. Without a reward there is nothing to buy back, so the compensation we study does not arise in its setting; with a reward added, a gated average is still an average. The two compose.
- RLAD (ICML 2026). It trains on student rollouts with a task reward and a teacher, and replaces the KL regulariser with a clipped likelihood-ratio objective anchored to a mixture of the teacher and the old policy. The student then imitates only when that helps the update. This changes when the teacher is followed. As described, it remains a soft objective, and nothing in it bounds the teacher cost along a trajectory. The two act in different places: our constraint sits in the reward, RLAD’s change sits in the update objective, so its update can in principle run on our penalised reward.
- TRAC (ACL 2026). It builds a token-level reward from the teacher, calibrates it for teacher reliability at the problem, trajectory and token level, and adds it to GRPO’s advantage. It is still an additive soft term: a deviation at one token is allowed whenever the outcome reward pays for it. Its premise, that a teacher is a noisy signal which can itself cause reward hacking, is our first stated limitation. Its calibration weights are per-token weights and could weight our per-token cost.
- “Unlocking Token Rewards” (CVPR 2026): not a distillation method. It has no teacher and no student. It attributes token-level rewards from an existing reward model, a process-reward model in its examples, to improve credit assignment in RL. It answers a different question, and it is complementary. Our proofs hold for any non-negative per-token cost, and Appendix G names learned process-reward models as a richer cost. This paper’s attribution is what would turn a step-level reward model into the per-token cost our constraint takes. We used the teacher’s likelihood because distillation computes it anyway, and because one forward pass prices every token in the vocabulary, which makes our single-step term exact. With a reward-model cost that term would be sampled.
- KDRL: an arXiv preprint from June 2025. Its objective is GRPO minus a weighted reverse-KL term, which is our GKD-GRPO baseline. KDRL adds coefficient annealing and reward-guided masking. We swept the coefficient over five values spanning four orders of magnitude, which traces the frontier that a schedule moves along.
- Uni-OPD and TGPO (arXiv, May 2026), RG-OPD (arXiv, July 2026), REOPD (arXiv, August 2026) and RA-OPD (arXiv, 28 August 2026): preprints only, as far as we can find. The last was posted less than a month before the submission deadline.
The ICLR 2027 Reviewer Guide states that authors are not required to compare against papers available only on arXiv, and that "the lack of such comparisons cannot be a basis for rejection". Six of the ten fall under that rule. We are glad to cite and discuss all ten.
The other four are peer-reviewed, and we left them out of Table 1 by design. Table 1 is a controlled comparison. Every row runs on the same training pipeline, models, data and settings, and the rows differ only in how the teacher enters the objective: not at all (GRPO), alone (GKD, MiniLLM), as a soft term added to the reward (GKD-GRPO, with λ swept over four orders of magnitude), or as a constraint (ours). A gap between rows can then be attributed to that choice. Appendix B notes that we re-implemented MiniLLM over GRPO for this reason.
Each of the four belongs to a family that the table already holds in its plain form. TSD-KD is teacher-only distillation, like GKD and MiniLLM, which are its own baselines. RLAD and TRAC add a soft teacher signal to a task reward, like GKD-GRPO, the KL-based form that RLAD itself measures against. “Unlocking Token Rewards” has no teacher and sits outside the comparison. Each also brings several components of its own: token gates, a mixture anchor, three levels of calibration. Adding them would change several things at once, and a gap to our method could no longer be traced to the constraint.
2(2). Process supervision and its cost. We agree to cite and discuss those works. The reviewer is right that process rewards can now be obtained without step-level annotation, and we will reword our related-work sentence so that it does not suggest otherwise.
We do not claim that our method is better than training with a process-reward model. That comparison is outside the scope of the paper, and we ran none. The paper asks a different question: given a per-token cost, how should it be aggregated and enforced alongside a task reward? A process-reward model is a choice of cost, not an alternative to the constraint.
The two fit together. Our proofs hold for any non-negative per-step cost, and Appendix G names learned process-reward models as a richer one. A PRM’s score can replace the teacher’s likelihood as the cost, with the same worst-case prefix bound and the same enforcement through the reward, so the student would need no PRM at test time. We used the teacher’s likelihood because distillation computes it anyway and it needs no second model.
2(3). "Hard constraints are the key" is not established against weighting, masking and gating. Table 1 is that comparison. GKD-GRPO is the soft counterpart of our method on the same pipeline, and its weight is swept over four orders of magnitude, which traces the soft frontier in Figure 3. Our method sits off that frontier. At the soft baseline’s highest-accuracy setting (λ=0.001), Qwen/MATH gives FAC 34.62 with CS 61.48, against 34.86 with 81.30 for ours. GSM-Symbolic gives 74.08 with 0.00, against 76.14 with 83.88.
Weighting, masking and gating refine the per-token cost inside a soft term. They leave the aggregation an average. Proposition 3.1(a) holds for any non-negative per-token cost: under an averaged budget a single step can grow with the length of the sequence, whatever the weights. Proposition 3.1(b) shows that our optimum is not a re-tuning of λ. We do not claim that a hard bound is the only route, and as noted under 2(1), such weights can be used inside our cost.
3(1). The prefix constraint still permits local compensation. The counterexample is Proposition 3.1(a). The proposition says the cost at step k is bounded by (k+1)·d. In the reviewer's example, which on our reading takes d = 1, the bound at the 101st token is 101 and the token costs 100. We claimed no position-independent per-token bound. Section 3.1 words the guarantee as preventing "future compensation after a prefix has already crossed the boundary".
What the prefix form removes is repayment. Under a whole-sequence average, a deviation can be averaged down by anything written after it. Under ours it cannot, and the slack must have been accumulated first. The reviewer is right that this leaves the bound looser late in the sequence. Appendix D discusses the flat per-token alternative and why we rejected it.
3(2). NLL is not KL, and NLL is not validity. The paper does not equate NLL with KL. Appendix E compares the two as constraint statistics. Worst-prefix teacher NLL has a Spearman correlation of 0.231 with RSR, reverse KL 0.224 and forward KL 0.186 (Table 3). The identity the reviewer writes, cross-entropy equals entropy plus KL, is why the two behave alike. On validity, see our replies to reviewer 2B4i (W1) and to reviewer CuYZ (W2).
3(3). Absorption is not unique to the prefix average. Agreed, and not claimed. Absorption is the property of Saute's cumulative budget, which is where we take it from (Section 2.2). What we claim for the prefix average is a combination: absorbing like the sum, and length-normalised unlike the sum.
The reviewer's proposal, the running maximum of a sliding-window average, is a good one. It is absorbing, and for a window of w tokens it gives a position-independent per-token bound of w·d. It is also an instance of our construction. Lemma 3.2, Theorem 3.3 and the gradient decomposition use only the fact that the statistic is a running maximum computable from the prefix and one teacher pass. We tried it. It adds a hyperparameter, the window length, and early results were on par, so we kept the version without it.
3(4). Non-equivalence does not prove necessity, and there is no like-for-like ablation. Proposition 3.1(b) claims one thing: no fixed λ on the cumulative cost reproduces the constrained optimum. Its job is to rule out the reading that our method is GKD-GRPO with a lucky λ. It does not claim that a hinge or an adaptive multiplier on the same worst-prefix statistic would fail. Such variants are versions of what we propose, differing in the shape of the penalty beyond the boundary, and we expect several of them to work.
The comparison the reviewer asks for is the one Table 1 makes: the same pipeline, with the teacher entering as a soft term swept over λ or as a constraint. On the individual choices, Appendix D compares the cumulative and worst-case statistics on the same sampled trajectories.
We thank the reviewer for a long review.
1(1). Contribution relative to process supervision. The paper's claim is narrow. Methods that combine a task reward with a teacher term aggregate the teacher term as an average. An average permits compensation, and bounding the worst prefix removes the forward half of it. The review lists ten methods as covering this ground. By its own one-line descriptions, they schedule, mask, gate, calibrate or reweight a soft signal. None is described as a constraint, and none bounds a worst case. The list documents the gap we claim to fill.
We do not claim that compensation is the primary bottleneck of reasoning distillation, or that prefix constraints are necessary. Those framings are the review's. We claim that the effect exists, which Figure 5 shows as the tail of worst-prefix cost under soft regularisation. We also claim that removing it moves the accuracy-fidelity frontier, which Figure 3 and Table 1 show.
1(2). CS is the training statistic, and RSR gains are modest. CS is trained on, and so is FAC: answer correctness is the training reward of GRPO and of our method, and no one calls test FAC circular. Both are optimised on training prompts and reported on held-out ones, where the student decodes with no teacher. That the constraint still holds there is a generalisation result, not a given: satisfaction is about 100% in training and 57% to 87% at test. High CS alone is not our result either, since plain GKD has it. The result is high CS at GRPO’s accuracy: on GSM-Symbolic, 83.88 against 1.00 at equal FAC.
We do not offer CS as evidence of better reasoning; Section 4.1 calls it fidelity. That evidence is RSR and the pairwise comparison, which are the independent evaluations of process correctness the review asks for. The judge is neither teacher nor student, sees only the problem and the solution, and was audited by a human on 60 verdicts (Appendix F).
1(3). Errors during exploration are not invalid conclusions. See our reply to reviewer CuYZ on self-correction (https://openreview.net/forum?id=UC6oEpbaqC¬eId=0OXnafkxn5). This point says the constraint is too strict on a single misstep. Point 3(1) says it is too lax on a single misstep. Both cannot be pressed against the same design...