acceptodds
Under review as a conference paper at ICLR 2027

The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learning

Abstract

Distilling the reasoning capabilities of large language models (LLMs) into smaller students is a central challenge for efficient deployment. Current approaches face a fundamental tension: optimizing purely for verifiable task rewards (e.g., via GRPO) leads to *reward hacking*, where students arrive at correct final answers through flawed intermediate logic, while regularizing with soft divergence penalties against a teacher (e.g., KL-based distillation) dilutes task performance and, critically, allows the student to *compensate* for severe logical violations at one step with high teacher agreement at others. We argue that this averaging is fundamentally misaligned with the nature of reasoning: a chain-of-thought is only as valid as its *weakest link*. Motivated by this observation, we formulate reasoning distillation as a constrained reinforcement learning problem in which the task reward is maximized subject to a worst-case constraint on the teacher log-likelihood along every prefix of the trajectory. To avoid the prohibitive cost of dual Lagrangian solvers and the test-time teacher dependence of state-augmented methods such as Saute, we derive an unaugmented constrained MDP whose reward transformation preserves the hard-constraint semantics, admits a low-variance policy gradient decomposition into single-step and long-term terms, and provably satisfies the worst-case constraint almost surely in the penalty limit. Through extensive experiments on mathematical reasoning and code generation tasks, we demonstrate that our method significantly expands the accuracy-fidelity Pareto front. By matching the high Final Answer Correctness of pure RL and drastically reducing teacher constraint violations, we ultimately achieve the highest rigorous Reasoning Success Rate across all evaluated settings.

Primary area:
reinforcement learning
Keywords:
distillation, LLM, safety

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_meQmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates reasoning distillation for large language models. By constraining the student model, specifically, by minimizing the negative log-likelihood of the teacher’s output across all generated prefixes, the approach limits the student’s deviation from the teacher’s distribution while simultaneously optimizing for task rewards. The authors reformulate this constraint as a penalty for violations and propose a policy gradient decomposition, which allows the student to perform reasoning without requiring access to the teacher model during inference. Both mathematical analysis and empirical experiments demonstrate that this method improves adherence to teacher-imposed constraints and yields gains in reasoning success rates across the evaluated settings.

Critical strengths

This paper focuses on the phenomenon where a final answer is correct despite errors in the intermediate reasoning. It reports the answer accuracy, teacher constraint satisfaction rate, and reasoning success rate separately, which helps to distinguish performance across different dimensions. The experiments cover two model families, and the paper provides details on training configurations, prompts used for evaluation, and a small-scale manual review, making the assessment process easier to understand.

Critical weaknesses

1. Research Significance and Problem Framing

(1) The paper has yet to sufficiently establish its scientific contribution relative to existing research. The paper motivates its approach by highlighting cases where the final answer is correct despite flawed intermediate reasoning, pointing to the limitations of outcome-only reinforcement learning (RL) and "average teacher" consistency. However, the distinction between outcome supervision and process supervision, including the identification of erroneous reasoning steps and the use of teacher feedback to improve credit assignment in RL, is already a well-explored area. The paper must clearly articulate what specific, previously unaddressed problem it identifies and resolves, beyond what is already covered by existing methods in process supervision, selective distillation, and reward-aware distillation. Currently, the problem framing relies on a single example where the final answer happens to be correct, along with data on teacher cost distributions and limited evaluations by a "judge" model. While this evidence demonstrates that final-answer metrics are insufficient, it fails to establish that "compensation mechanisms within average supervision" constitute the primary bottleneck for current reasoning distillation, nor does it prove that the paper’s proposed "prefix constraints" are a necessary mechanism to overcome this bottleneck. The claims regarding the research gap are significantly stronger than the evidence provided.

(2) Improvements in "teacher consistency" have not been sufficiently translated into meaningful, independent gains in reasoning reliability. The most notable experimental gains appear in "Constraint Satisfaction", a metric derived directly from the teacher-cost statistics used during the model's training. While this confirms that the method successfully optimizes its surrogate objective, it does not independently prove that the mathematical reasoning has become more accurate or reliable. The RSR (Reasoning-Step Reliability/Correctness) metric is more closely aligned with the paper’s core application goals; however, the gains over strong baselines are generally modest, and the results lack repeated experiments or uncertainty estimates. Consequently, the current evidence supports the claim that the method can "adjust the relationship between accuracy and teacher likelihood," but it falls short of justifying broader conclusions, such as "significantly enhancing trustworthy reasoning" or "overcoming a fundamental trade-off." The practical value of these improvements to surrogate metrics needs to be substantiated through independent evaluations of process correctness, robust hidden-test-set evaluations, or formal verification.

(3) The assumption that "a single error invalidates the entire chain of reasoning" requires a more precise definition of its scope. In the context of a final mathematical proof, a critical inferential error can indeed render the entire proof invalid. However, the chain-of-thought generated by a model may include exploratory steps, backtracking, self-corrections, and discarded branches. Making an error during the exploration process is not the same as having a final conclusion that lacks valid support. The paper applies the "weakest-link" intuition to the entire token-generation process, evaluating the output based on the binary question, "Does it contain any errors?", without adequately distinguishing between uncorrected critical logical flaws, exploratory steps that were subsequently rectified, and redundant content that does not affect the final derivation. Consequently, the paper’s optimization and evaluation objectives may skew toward conservative, formulaic explanations, which do not necessarily reflect stronger, genuine reasoning capabilities.


2. Related Work and Baselines

A more substantive limitation of this paper is the lack of clear connections to, and fair comparisons with, representative, reliable methods in areas such as teacher supervision, reward-aware distillation, and fine-grained credit assignment. Consequently, the existing evidence is insufficient to substantiate the added value of the proposed constraint mechanism over prior research.

(1) The omission of relevant work, specifically research on the reliability of teacher supervision, reward-aware distillation, and fine-grained credit assignment, results in an inadequate set of baselines.

The paper primarily compares its approach against GKD, MiniLLM, GRPO, and a variant of GKD-GRPO with fixed weights, using these to highlight the tension between accuracy and teacher consistency. However, the existing literature has moved well beyond simple combinations involving fixed KL-divergence weighting; it now encompasses dynamic scheduling, selective imitation, the calibration of teacher reliability, and token-level reward attribution. The current set of comparisons covers only a fraction of this research landscape, which is insufficient to support broad claims regarding the limitations of existing methods or the advantages of the proposed approach.

The following works are closely related to the problem setting, technical motivation, or optimization objectives of this paper, yet they are neither cited nor included in the experimental comparisons:

  • KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning (2025). KDRL jointly optimizes GRPO and teacher reverse-KL supervision, investigating coefficient scheduling and reward-guided masking. A fixed-weight GKD-GRPO baseline therefore does not adequately represent existing approaches to combining distillation and reinforcement learning.

  • Reinforcement-aware Knowledge Distillation for LLM Reasoning (RLAD) (2026). RLAD addresses interference between teacher supervision and reward optimization through selective imitation using a trust-region ratio objective. Comparing against it would help establish whether the proposed method improves upon reward-aware policy optimization beyond simple KL weighting.

  • Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe (2026). Uni-OPD improves the reliability of trajectory-level teacher supervision through outcome-guided margin calibration. It directly relates to the paper’s objective of reconciling teacher consistency with task success.

  • Reward-Gated On-Policy Distillation (RG-OPD) (2026). RG-OPD uses verifier feedback and teacher–student likelihood differences to determine which trajectories receive teacher supervision. It provides a relevant comparison for filtering misleading teacher signals.

  • When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation (RA-OPD) (2026). RA-OPD checks whether trajectory-level distillation returns align with outcome rewards and filters unreliable updates. Its reward-alignment mechanism should be distinguished from and compared with the proposed fixed teacher-cost boundary.

  • Teacher-Guided Policy Optimization for LLM Distillation (TGPO) (2026). TGPO provides directional teacher guidance on student-visited prefixes and integrates it with RLVR. It is directly relevant to the paper’s claims concerning token-level guidance and stable optimization.

  • Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation (TSD-KD) (2026). TSD-KD combines teacher preference feedback with token-selective distillation to address limited student capacity and overly restrictive imitation. This comparison would help determine whether the proposed constraint offers benefits beyond selective supervision of important tokens.

  • TRAC: Teacher-Guided Token Reward with Adaptive Calibration for Robust Policy Optimization (2026). TRAC derives token-level rewards from teacher–student log-probability differences and calibrates teacher reliability at the problem, trajectory, and token levels before incorporating these signals into GRPO. It provides a relevant comparison for assessing whether a fixed teacher-cost boundary offers advantages over adaptively calibrated teacher supervision.

  • REOPD: Reliability-Adaptive Reward Extrapolation for On-Policy Distillation (2026). REOPD uses token-level reliability weights and a batch-level adaptive budget to mitigate reward hacking and instability during reward extrapolation. Although its setting differs, the paper should discuss how its constraint mechanism differs from this reliability-adaptive approach.

  • Unlocking Token Rewards via Training-Free Reward Attribution (2026). This work derives token-level rewards from an existing differentiable reward model through gradient-based attribution, without training an additional token reward model. It provides a relevant comparison for evaluating fine-grained credit assignment and optimization efficiency, while accounting for its different supervision requirements.

The authors should, at a minimum, conduct fair comparisons with closely related methods. Otherwise, it remains unclear whether the observed gains stem from the unique design of the paper’s prefix-based constraints, or from more general techniques such as selective teacher feedback, reward calibration, or dense supervision.

(2) The discussion on process supervision and the distillation of reasoning structures is insufficient, and the analysis of associated costs is overly vague. While this paper aims to improve the effectiveness of intermediate reasoning, the experiments focus primarily on distribution distillation and reinforcement learning (RL) based on final answers; they lack direct comparisons with representative methods that target process correctness or reasoning structures. Relevant studies that warrant discussion include:

  • Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations (ACL 2024): This work employs automatically constructed process supervision to train a reward model, which is then used for step-by-step RL, demonstrating that process supervision does not necessarily require manual, step-by-step human annotations.
  • Free Process Rewards without Process Labels (2024): This research explores deriving implicit process-level rewards from outcome-level labels, a topic directly relevant to the paper’s framing of the costs associated with process supervision.
  • RLKD: Distilling LLMs’ Reasoning via Reinforcement Learning (preprint, 2025). This approach employs reinforcement learning (RL) distillation guided by a reward function aligned with reasoning structures; it focuses on the transfer of reasoning patterns rather than merely ensuring consistency in the output text distribution.
  • Reinforcement Learning via Self-Distillation (SDPO, 2026). This method derives dense learning signals through a self-teaching mechanism conditioned on feedback, offering an alternative approach to addressing sparse rewards and credit assignment. However, the specific nature of its supervisory conditioning differs from that proposed in this paper.
  • On Teacher Hacking in Language Model Distillation (ICML 2025). While the specific setup differs from the current work, it shares a significant conceptual link regarding the risk that an agent might exploit flaws in the teacher model when optimizing against the teacher’s objectives.

The authors might reasonably highlight that their approach avoids training an additional Process Reward Model (PRM); however, they must justify this design choice by demonstrating its advantages in terms of performance, training costs, or the demand for supervisory resources. Characterizing other process-supervision methods as inherently reliant on expensive annotations, additional model training, or complex architectural modules fails to fully account for existing research on automated process supervision and low-cost, fine-grained reward mechanisms.

(3) The discussion of existing related work is insufficient to substantiate the technical claim that "hard constraints are the key to solving the problem."

Existing literature already offers various methods for integrating teacher feedback with task-specific rewards, including dynamic weighting, reward-aware masking, reliability gating, selective imitation, and token-level credit assignment. Consequently, the paper needs to clarify what specific value the "worst-case prefix constraint" adds beyond these established mechanisms. The current experiments simultaneously vary the definition of teacher costs, sequence aggregation methods, penalty formulations, and gradient estimation techniques. Furthermore, the lack of comparison with the aforementioned representative methods makes it impossible to rule out the possibility that simpler approaches, such as reliability calibration or localized supervision, could yield comparable gains.

In particular, it is essential to conduct comparisons using identical teacher models, student models, training datasets, and computational budgets. This should include representative baselines such as fixed soft regularization, adaptive teacher supervision, and process-level or token-level reward methods. Where feasible, different penalty formulations should be evaluated using consistent cost metrics. Only through such rigorous comparison can the observed performance differences be attributed specifically to the prefix constraint, rather than to variations in the intensity of teacher supervision or other implementation details.


3. Methodology

(1) The core prefix constraint fails to achieve the "protection of the weakest step" implied by the paper; it still permits significant local compensation. The constraint imposed in this paper is: W(τ)=max⁡1≤k≤T1k∑t=1kCt≤dW(\tau)=\max_{1\le k\le T}\frac{1}{k}\sum_{t=1}^{k}C_t\le d, which is neither a constraint on the maximum cost of a single step nor a guarantee of the validity of individual reasoning steps.

A straightforward counterexample: let d=1d=1. Suppose the first 100 tokens each have a cost of 0.01, while the 101st token has a cost of 100. The average cost for all preceding prefixes is 0.01, and the average cost for the final prefix is: 100×0.01+100101=1.\frac{100\times0.01+100}{101}=1.

Consequently, even with a local token of extremely high cost, the sequence still satisfies the paper’s prefix constraints. Low-cost text earlier in the sequence effectively "accumulates a buffer" that can be used to offset high-cost behavior later on.

Proposition 3.1, which states Ck≤(k+1)dC_k\le(k+1)d, further demonstrates that the allowable deviation for a single step grows linearly with its position in the sequence. For an error occurring near the end of a sequence, the worst-case upper bound remains O(Td)O(Td). While the paper successfully prevents a prefix that has already exceeded the threshold from being "repaired" by subsequent text, it does not eliminate local compensation in a general sense, nor does it provide an upper bound on local risk that is independent of sequence length.

This distinction directly affects the validity of the paper’s title, motivation, and primary contributions. The authors should refine their claims regarding "anti-compensation" and investigate whether the proposed method is biased toward suppressing early-stage anomalies while remaining relatively insensitive to logical errors occurring later in the sequence.

(2) The link between the teacher’s negative log-likelihood (NLL) and logical validity is insufficient to support an interpretation based on "hard constraints. The teacher’s NLL measures the statistical predictability of a token, not the correctness of a mathematical step. High-probability generations may contain common errors, while low-probability generations could represent correct, albeit unconventional, solutions. Furthermore, this statistic is influenced by factors such as formatting, phrasing, the teacher model’s entropy, and the choice of tokenizer. For a fixed prefix, we have: & H(\pi(\cdot\mid s)) + D_{\mathrm{KL}}(\pi(\cdot\mid s)\Vert\mu(\cdot\mid s)).& Therefore, reducing the teacher's NLL is not equivalent to minimizing the teacher-student KL divergence; the student's conditional entropy also plays a role. Interpreting NLL directly as a measure of the degree of logical violation lacks sufficient justification.

Appendix E reports a Spearman correlation coefficient of 0.231, and a pooled AUC of 0.63 for samples where the model's answer is correct. This indicates that the proxy metric carries some information, but it is insufficient to establish it as a highly reliable signal for detecting logical violations. Furthermore, statistical significance should not be conflated with discriminative power. It is necessary to control for factors such as sequence length, stylistic expression, question difficulty, and formatting, and to analyze specific cases—such as those where the model is correct despite a low probability assigned by the teacher, or incorrect despite a high probability assigned by the teacher.

(3) Some of the structural contributions are not unique to the "prefix-average" constraint proposed in this paper; the claims regarding novelty are overstated. The principle that "once a violation occurs, the sequence is deemed infeasible" stems from tracking historical violations and does not inherently require a "prefix-average" formulation. Similarly, a cumulative sum of non-negative costs that exceeds a defined budget is also irreversible. The justification provided in Appendix D for excluding sliding-window constraints is also inadequate. While statistics from a current window might recover, one could instead constrain the "maximum cost observed across all historical windows up to the present." This construction would also ensure that historical violations remain irreversible, while more effectively targeting localized anomalies. Likewise, when the state is defined as the complete history, any budget variable derived from that history can be considered redundant within the mathematical state representation. This observation applies more broadly than just to the constraints proposed in this paper; the authors need to more clearly distinguish between general historical state representations, existing penalty-based constructions in safe reinforcement learning, and the specific technical contributions that are truly novel to this work.

(4) The fact that this approach is not equivalent to a fixed "soft" penalty does not, in itself, prove the necessity or superiority of the "hard" boundary design proposed in this paper. Proposition 3.1(b) compares a scalar soft penalty, based on cumulative cost, with a constraint based on the average cost of the worst-case prefix. Since the two approaches employ different cost functionals, it is not surprising that their optimal policies differ. This result does not preclude the use of soft penalties, hinge penalties, adaptive multipliers, or other risk-sensitive objectives applied to the same W(τ)W(\tau). For instance, the expression $ R(\tau)-\lambda[W(\tau)-d]_+$ could also be used to impose a penalty based on the same prefix-violation statistic.

The current experiments simultaneously vary the cost definition, the aggregation method, and the form of the penalty. Consequently, it is impossible to isolate whether the observed gains stem from the prefix statistics, the teacher’s NLL, the strong penalty, or the so-called "hard-constraint semantics." To substantiate the core design, comparisons must be conducted while keeping the cost definitions and statistics constant.

(5) There remains a significant gap between theoretical guarantees and practical training. Theorem 3.3 relies on conditions such as an infinite penalty (n→∞n \to \infty) and the existence of an optimal policy with finite value. In contrast, the experiments employ a finite penalty (n=20n=20), parametric models, and approximate optimization via GRPO. The theorem provides no convergence guarantees for the actual algorithm, no bounds on the probability of constraint violation under finite penalties, and no guarantees regarding constraint satisfaction on the test distribution.

The paper itself acknowledges a decline in constraint satisfaction rates during testing. Therefore, claims such as "strictly safe during training," "satisfied almost surely," or "reliably preventing erroneous reasoning" require precise qualification to avoid erroneously attributing properties of a limiting MDP to the actual model. Furthermore, the argument that Lagrangian methods are "computationally infeasible" is insufficiently supported. Once the teacher’s costs are fixed, updating a small number of multipliers does not inherently necessitate expensive inner-loop optimization involving the teacher. If computational efficiency is a key advantage of this work, the authors should provide a baseline using adaptive multipliers, along with measurements of the associated overhead.

(6) The scope of the claims regarding variance reduction is limited, and details concerning the actual optimization process are insufficient. Appendix A.4 demonstrates, via conditional expectation, that the variance of the immediate gradient components is reduced, a valid application of the Rao–Blackwellization argument. However, the complete estimator also incorporates long-term return terms, as well as the covariances between these various components. Consequently, a reduction in the variance of individual immediate terms does not automatically imply a reduction in the total variance of the full policy gradient estimator, nor does it directly guarantee that GRPO training is more stable.

The paper does not report the variance of the full gradient, stability across different random seeds, or ablation studies involving the removal of the "immediate" (SINGLE) or "long-term" (LONG) components. Furthermore, while the main text presents the REINFORCE decomposition, the appendix mentions the use of in-group normalization and clipping; however, it fails to provide a clear explanation of how these mechanisms are integrated into the final loss function.

This detail is particularly significant for groups where all trajectories result in a violation (e.g., failing a constraint). If every trajectory incurs the same terminal penalty, the standard in-group relative advantage might be zero. It remains unclear whether the "immediate" terms can still provide meaningful updates in such cases, and how exactly they are reconciled within the GRPO framework, a point that requires both algorithmic clarification and empirical validation.

(7) Characterizing post-hoc threshold scanning as a simplification of hyperparameter selection during training conflates two distinct operations. Appendices E and G suggest that because the cost of each trajectory is recorded, one can perform a post-hoc scan of the threshold (dd), which is framed as being easier than selecting a soft regularization coefficient. However, filtering the outputs of a fixed model is fundamentally different from training distinct policies using different thresholds (dd). A training-time threshold alters the policy updates and the subsequent sampling distribution, effects that cannot be fully replicated by post-hoc filtering of a static set of trajectories.

Moreover, if such filtering were applied during deployment, it would necessitate computing the "teacher" cost, a process with resource requirements that differ from a scenario where a "student" model generates outputs independently (without invoking the teacher). The authors should clearly distinguish between hyperparameter selection during training, offline analysis, and deployment-time filtering; the ability to filter outputs should not be conflated with an advantage in training-time hyperparameter tuning.


4. Experiments

(1) The coverage of models and tasks is insufficient to support broad conclusions regarding current reasoning distillation methods. All primary student models are drawn from the Qwen2.5 and Llama-3.2 families, with parameter counts concentrated in the 0.5B–3B range; furthermore, both teacher and student models belong to the same model family. The study fails to verify whether the proposed methods remain effective when applied to newer models, established reasoning-oriented student models, or scenarios involving varying scales and capability gaps between teacher and student models.

While older models may be suitable for mechanistic research, this paper proposes a general-purpose reasoning distillation method, which necessitates more representative validation. Given that the Qwen3 series already includes small-scale models (e.g., 0.6B, 1.7B, and 4B), incorporating experiments with newer, comparably sized models would not necessarily require the use of massive-scale models.

The release of the Qwen3.5, 3.6, and 3.8 series by the Qwen team further highlights the temporal gap between this work and the current state of model development. At a minimum, the experimental evaluation should demonstrate that the proposed method yields performance gains on newer models with comparable computational requirements.

Mathematical evaluation is limited to the MATH and GSM benchmarks, while code generation is tested exclusively on MBPP. Since MATH remains challenging for the small-scale models used in this study, one cannot simply assume that these tasks have reached saturation. The evaluation lacks more demanding mathematical benchmarks (such as AIME or OlympiadBench), complex coding tasks, and scenarios requiring long-chain reasoning. Given that a core motivation of this paper concerns how local errors may be obscured by long sequences, the current evaluation lacks sufficient stratification by sequence length or specific validation for long-chain reasoning.

(2) The performance gains over relatively strong baselines are marginal, and there is a lack of statistical evidence to support claims of a significant advantage. In Table 1, the RSR (Reasoning Success Rate) achieved by the best configuration in this paper shows only modest improvements over existing strong baselines: On Qwen/MATH, the proposed method reaches 30.00%, compared to 29.48% for GKD-GRPO, an absolute gain of 0.52 percentage points. On Llama/MATH, it achieves 14.12%, versus 13.64% for SFT on teacher-generated CoT, a gain of 0.48 percentage points. On Qwen/GSM-Symbolic, the RSR increases from 64.36% (with GRPO) to 65.82%, a gain of 1.46 percentage points. On MBPP, the proposed method achieves 34.2%, compared to 34.0% for GKD-GRPO, a gain of merely 0.2 percentage points.

While these point estimates may have some value, the absence of results across multiple random seeds, confidence intervals, or paired statistical tests makes it impossible to determine their stability. In the case of MBPP, a 0.2 percentage-point difference corresponds to just one single problem out of 500. The improvements in "Chain-of-Thought Success" (CS) relative to pure GRPO, observed via this proxy metric, cannot substitute for rigorous statistical validation of gains in core reasoning success rates. The current evidence is insufficient to support strong claims such as "significant" or "comprehensively superior."

(3) The evaluation of core process quality relies too heavily on a single "judge" model, and the pairwise win rates do not consistently favor the proposed method across all experimental settings.

The RSR is determined by the DeepSeek-R1-Distill-Qwen-32B model. Although the authors conducted a manual review of 60 samples and employed a "double-order" pairwise comparison, which are valuable control measures, these steps remain insufficient to verify subtle differences across various methods and difficulty levels.

Further clarification is required regarding the number of human annotators, the sample sizes for each stratified subset, the specific types of errors encountered, and any potential biases the judge model may exhibit concerning response length, style, or the self-correction process. Process-based evaluation can also leverage expert-annotated data, such as that found in ProcessBench, to calibrate "judge" models or serve as a proxy for the cost of human (teacher) feedback. While such data is valuable for validating evaluation tools, it cannot serve as a direct substitute for assessing generative capabilities.

Furthermore, pairwise comparisons are conducted exclusively on samples where both methods provide correct answers; this constitutes a conditional evaluation, and both the intersection of the sample sets and the number of valid comparisons must be reported. In Figure 4, GKD-GRPO achieves a win rate of approximately 67.7% on the Qwen/MATH benchmark, outperforming the ~56% win rate reported for the method in this paper. Consequently, the results do not support the claim that the proposed approach consistently yields the highest reasoning quality across all settings.

(4) The reward evaluation in the code-generation experiments is highly susceptible to exploitation, which undermines its validity as evidence of general-purpose coding proficiency.

The coding prompts include unit tests, and the generated code is executed alongside these test assertions. The authors report that GRPO achieves a pass@1 score of 92.0% by leveraging these visible tests; however, the RSR (Robust Success Rate) is only 1.8%. This indicates that the primary execution-based metrics have been heavily exploited.

While this environment could serve as a stress test for "reward hacking," a rigorous assessment would require independent hidden test sets, isolated test evaluators, and a detailed analysis of specific "cheating" behaviors to determine whether the model has truly acquired generalizable programming capabilities. The paper also fails to adequately explain the underlying mechanism that allows the model to pass tests simply by reproducing the test code itself.

Compared to a relatively strong "soft-regularization" baseline, the proposed method shows an RSR increase of only 0.2 percentage points (from 34.0% to 34.2%), while the reported pass@1 actually declines from 50.4% to 45.6%. Therefore, these experiments fail to provide robust evidence for cross-domain advantages, nor do they support the broader claim that the method maintains high accuracy through pure reinforcement learning (RL) in the coding domain. Furthermore, a specialized evaluation protocol for RSR in coding tasks is required.

(5) The study lacks critical ablation analyses to identify the sources of performance gains; additionally, Figure 2 does not represent a direct comparison of constrained training approaches. The current experiments fail to sufficiently disentangle the individual contributions of the following factors:

  • The choice between teacher negative log-likelihood (NLL) and Kullback-Leibler (KL) divergence-based costs.
  • The differences between averaging costs over the "worst-case" prefix versus the entire sequence, as well as the use of maximum - single-step costs versus maximum costs over a sliding historical window.
  • Discontinuous penalty for violations versus continuous/adaptive penalties based on the same statistic.
  • "SINGLE" (exact expectation term) versus "LONG" (cumulative return term).
  • Penalty strength (nn), training budget (dd), and generation length.

Specifically, Figure 2 is cited in the main text to argue that "worst-case prefix constraints" are superior to "cumulative budget" constraints. However, Appendix D reveals that this experiment involved sampling 256 trajectories per problem and subsequently applying different filters to the same pool of candidates. It does not compare the policies resulting from training under each constraint, nor does it evaluate end-to-end performance given an identical generation budget.

Furthermore, RSR (Resolved Success Rate) after filtering must be reported alongside the acceptance rate; otherwise, one could artificially inflate the conditional success rate simply by rejecting a large number of samples. This figure is insufficient to demonstrate that the proposed training mechanism outperforms training based on a cumulative budget.

(6) There is an internal consistency issue between the key figures in Appendix E and the main tables that requires urgent clarification. Appendix E states that the analysis is based on 12 training runs using Qwen/MATH, with 5,000 generations per run, totaling 60,000 trajectories, and defines RSR as cases where the "final answer is correct" and the "judge identifies no errors." The authors report that when samples are divided into quartiles based on teacher likelihood, the RSR increases monotonically from 30.1% in the lowest quartile to 59.6% in the highest. Assuming equal sample sizes across the four quartiles and a strictly monotonic increase, the overall RSR should be at least: 30.1+30.1+30.1+59.64=37.475\frac{30.1+30.1+30.1+59.6}{4}=37.475%. However, in Table 1, the highest RSR reported for any training method on Qwen/MATH is only 30.00%. Similarly, Appendix E reports an FAC (Final Answer Correctness) that increases monotonically from 35.9% to 63.0%, implying an overall FAC of at least 42.675%; yet, the highest FAC reported for this configuration in the main table is only 34.86%.

If Appendix E corresponds to the same experimental runs, testing protocols, and metric definitions as those in the main table, these figures cannot both be correct. The authors must clarify whether different checkpoints, sample selection criteria, denominators, or evaluation settings were used, and provide reproducible subgroup counts and aggregate metrics. This issue directly concerns the core evidence supporting the validity of the "teacher-likelihood proxy." Until the methodology is clarified, this appendix cannot be considered a valid validation of the proxy. This observation pertains solely to the consistency of the reported data; no conclusions are drawn regarding the underlying cause.

(7) Insufficient information regarding reproducibility and computational fairness.

While the manuscript reports basic hyperparameters, such as learning rate, batch size, and number of epochs, several details critical to interpreting the results are missing. These include the exact size and partitioning of the mathematical training dataset, precise identifiers for the model checkpoints, training prompts, decoding settings for mathematical testing, random seeds, the full set of GRPO hyperparameters, and the specific implementation of the loss function involving decomposed gradients.

It should also be clarified whether the 3-round GKD (Generalized Knowledge Distillation) warm-up phase for Llama is included in the 20-round training total. Given that SFT (Supervised Fine-Tuning) uses 4 rounds and RL (Reinforcement Learning) uses 20, the number of epochs alone is insufficient to determine whether the comparison is fair; the total number of training tokens, as well as the computational budgets for generation and teacher-model inference, must be reported.

The initial FAC for student models in the tables is only 0–0.54%. It is necessary to distinguish between cases where the model fails to solve the problem correctly and cases where it fails due to answer formatting or extraction errors. The authors should report the "format compliance rate" and provide more robust answer-matching results.

Finally, the claim that the process takes "less than 23 hours on a single accelerator" lacks details regarding the hardware model, VRAM capacity, the actual computational cost of the teacher-model forward passes, and the total overhead of the proposed method; thus, the stated efficiency advantage is not adequately substantiated. For algorithms that rely on penalty-based sequencing, full-vocabulary immediate terms, and intra-group normalization, general training descriptions are insufficient to independently reproduce the results or verify whether the implementation aligns with the underlying theory.

Questions for authors

It is recommended to provide a point-by-point response to each of the issues raised in the "Weaknesses" section.

6 comments

We thank the reviewer for a long review.

1(1). Contribution relative to process supervision. The paper's claim is narrow. Methods that combine a task reward with a teacher term aggregate the teacher term as an average. An average permits compensation, and bounding the worst prefix removes the forward half of it. The review lists ten methods as covering this ground. By its own one-line descriptions, they schedule, mask, gate, calibrate or reweight a soft signal. None is described as a constraint, and none bounds a worst case. The list documents the gap we claim to fill.

We do not claim that compensation is the primary bottleneck of reasoning distillation, or that prefix constraints are necessary. Those framings are the review's. We claim that the effect exists, which Figure 5 shows as the tail of worst-prefix cost under soft regularisation. We also claim that removing it moves the accuracy-fidelity frontier, which Figure 3 and Table 1 show.

1(2). CS is the training statistic, and RSR gains are modest. CS is trained on, and so is FAC: answer correctness is the training reward of GRPO and of our method, and no one calls test FAC circular. Both are optimised on training prompts and reported on held-out ones, where the student decodes with no teacher. That the constraint still holds there is a generalisation result, not a given: satisfaction is about 100% in training and 57% to 87% at test. High CS alone is not our result either, since plain GKD has it. The result is high CS at GRPO’s accuracy: on GSM-Symbolic, 83.88 against 1.00 at equal FAC.

We do not offer CS as evidence of better reasoning; Section 4.1 calls it fidelity. That evidence is RSR and the pairwise comparison, which are the independent evaluations of process correctness the review asks for. The judge is neither teacher nor student, sees only the problem and the solution, and was audited by a human on 60 verdicts (Appendix F).

1(3). Errors during exploration are not invalid conclusions. See our reply to reviewer CuYZ on self-correction (https://openreview.net/forum?id=UC6oEpbaqC&noteId=0OXnafkxn5). This point says the constraint is too strict on a single misstep. Point 3(1) says it is too lax on a single misstep. Both cannot be pressed against the same design...

2(1). Ten missing baselines. The review says we must "at a minimum" run fair comparisons against these. Here is what they are:

  • TSD-KD (ICLR 2026). It is teacher-only distillation: the student trains on teacher signals with no task reward, like GKD and MiniLLM, which are in our Table 1 and among its own baselines. Its contribution is which tokens receive the teacher signal. Ours is on another axis: how the teacher term is aggregated and enforced once a task reward is present. Without a reward there is nothing to buy back, so the compensation we study does not arise in its setting; with a reward added, a gated average is still an average. The two compose.
  • RLAD (ICML 2026). It trains on student rollouts with a task reward and a teacher, and replaces the KL regulariser with a clipped likelihood-ratio objective anchored to a mixture of the teacher and the old policy. The student then imitates only when that helps the update. This changes when the teacher is followed. As described, it remains a soft objective, and nothing in it bounds the teacher cost along a trajectory. The two act in different places: our constraint sits in the reward, RLAD’s change sits in the update objective, so its update can in principle run on our penalised reward.
  • TRAC (ACL 2026). It builds a token-level reward from the teacher, calibrates it for teacher reliability at the problem, trajectory and token level, and adds it to GRPO’s advantage. It is still an additive soft term: a deviation at one token is allowed whenever the outcome reward pays for it. Its premise, that a teacher is a noisy signal which can itself cause reward hacking, is our first stated limitation. Its calibration weights are per-token weights and could weight our per-token cost.
  • “Unlocking Token Rewards” (CVPR 2026): not a distillation method. It has no teacher and no student. It attributes token-level rewards from an existing reward model, a process-reward model in its examples, to improve credit assignment in RL. It answers a different question, and it is complementary. Our proofs hold for any non-negative per-token cost, and Appendix G names learned process-reward models as a richer cost. This paper’s attribution is what would turn a step-level reward model into the per-token cost our constraint takes. We used the teacher’s likelihood because distillation computes it anyway, and because one forward pass prices every token in the vocabulary, which makes our single-step term exact. With a reward-model cost that term would be sampled.
  • KDRL: an arXiv preprint from June 2025. Its objective is GRPO minus a weighted reverse-KL term, which is our GKD-GRPO baseline. KDRL adds coefficient annealing and reward-guided masking. We swept the coefficient over five values spanning four orders of magnitude, which traces the frontier that a schedule moves along.
  • Uni-OPD and TGPO (arXiv, May 2026), RG-OPD (arXiv, July 2026), REOPD (arXiv, August 2026) and RA-OPD (arXiv, 28 August 2026): preprints only, as far as we can find. The last was posted less than a month before the submission deadline.

The ICLR 2027 Reviewer Guide states that authors are not required to compare against papers available only on arXiv, and that "the lack of such comparisons cannot be a basis for rejection". Six of the ten fall under that rule. We are glad to cite and discuss all ten.

The other four are peer-reviewed, and we left them out of Table 1 by design. Table 1 is a controlled comparison. Every row runs on the same training pipeline, models, data and settings, and the rows differ only in how the teacher enters the objective: not at all (GRPO), alone (GKD, MiniLLM), as a soft term added to the reward (GKD-GRPO, with λ swept over four orders of magnitude), or as a constraint (ours). A gap between rows can then be attributed to that choice. Appendix B notes that we re-implemented MiniLLM over GRPO for this reason.

Each of the four belongs to a family that the table already holds in its plain form. TSD-KD is teacher-only distillation, like GKD and MiniLLM, which are its own baselines. RLAD and TRAC add a soft teacher signal to a task reward, like GKD-GRPO, the KL-based form that RLAD itself measures against. “Unlocking Token Rewards” has no teacher and sits outside the comparison. Each also brings several components of its own: token gates, a mixture anchor, three levels of calibration. Adding them would change several things at once, and a gap to our method could no longer be traced to the constraint.

2(2). Process supervision and its cost. We agree to cite and discuss those works. The reviewer is right that process rewards can now be obtained without step-level annotation, and we will reword our related-work sentence so that it does not suggest otherwise.

We do not claim that our method is better than training with a process-reward model. That comparison is outside the scope of the paper, and we ran none. The paper asks a different question: given a per-token cost, how should it be aggregated and enforced alongside a task reward? A process-reward model is a choice of cost, not an alternative to the constraint.

The two fit together. Our proofs hold for any non-negative per-step cost, and Appendix G names learned process-reward models as a richer one. A PRM’s score can replace the teacher’s likelihood as the cost, with the same worst-case prefix bound and the same enforcement through the reward, so the student would need no PRM at test time. We used the teacher’s likelihood because distillation computes it anyway and it needs no second model.

2(3). "Hard constraints are the key" is not established against weighting, masking and gating. Table 1 is that comparison. GKD-GRPO is the soft counterpart of our method on the same pipeline, and its weight is swept over four orders of magnitude, which traces the soft frontier in Figure 3. Our method sits off that frontier. At the soft baseline’s highest-accuracy setting (λ=0.001), Qwen/MATH gives FAC 34.62 with CS 61.48, against 34.86 with 81.30 for ours. GSM-Symbolic gives 74.08 with 0.00, against 76.14 with 83.88.

Weighting, masking and gating refine the per-token cost inside a soft term. They leave the aggregation an average. Proposition 3.1(a) holds for any non-negative per-token cost: under an averaged budget a single step can grow with the length of the sequence, whatever the weights. Proposition 3.1(b) shows that our optimum is not a re-tuning of λ. We do not claim that a hard bound is the only route, and as noted under 2(1), such weights can be used inside our cost.

3(1). The prefix constraint still permits local compensation. The counterexample is Proposition 3.1(a). The proposition says the cost at step k is bounded by (k+1)·d. In the reviewer's example, which on our reading takes d = 1, the bound at the 101st token is 101 and the token costs 100. We claimed no position-independent per-token bound. Section 3.1 words the guarantee as preventing "future compensation after a prefix has already crossed the boundary".

What the prefix form removes is repayment. Under a whole-sequence average, a deviation can be averaged down by anything written after it. Under ours it cannot, and the slack must have been accumulated first. The reviewer is right that this leaves the bound looser late in the sequence. Appendix D discusses the flat per-token alternative and why we rejected it.

3(2). NLL is not KL, and NLL is not validity. The paper does not equate NLL with KL. Appendix E compares the two as constraint statistics. Worst-prefix teacher NLL has a Spearman correlation of 0.231 with RSR, reverse KL 0.224 and forward KL 0.186 (Table 3). The identity the reviewer writes, cross-entropy equals entropy plus KL, is why the two behave alike. On validity, see our replies to reviewer 2B4i (W1) and to reviewer CuYZ (W2).

3(3). Absorption is not unique to the prefix average. Agreed, and not claimed. Absorption is the property of Saute's cumulative budget, which is where we take it from (Section 2.2). What we claim for the prefix average is a combination: absorbing like the sum, and length-normalised unlike the sum.

The reviewer's proposal, the running maximum of a sliding-window average, is a good one. It is absorbing, and for a window of w tokens it gives a position-independent per-token bound of w·d. It is also an instance of our construction. Lemma 3.2, Theorem 3.3 and the gradient decomposition use only the fact that the statistic is a running maximum computable from the prefix and one teacher pass. We tried it. It adds a hyperparameter, the window length, and early results were on par, so we kept the version without it.

3(4). Non-equivalence does not prove necessity, and there is no like-for-like ablation. Proposition 3.1(b) claims one thing: no fixed λ on the cumulative cost reproduces the constrained optimum. Its job is to rule out the reading that our method is GKD-GRPO with a lucky λ. It does not claim that a hinge or an adaptive multiplier on the same worst-prefix statistic would fail. Such variants are versions of what we propose, differing in the shape of the penalty beyond the boundary, and we expect several of them to work.

The comparison the reviewer asks for is the one Table 1 makes: the same pipeline, with the teacher entering as a soft term swept over λ or as a constraint. On the individual choices, Appendix D compares the cumulative and worst-case statistics on the same sampled trajectories.

Reviewer_CuYZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies the challenge of distilling LLM reasoning capabilities into smaller student models. It discusses a fundamental limitation of existing approaches: GRPO may lead to reward hacking, while KL-based distillation allows severe local reasoning deviations to be compensated by high teacher agreement elsewhere in the trajectory. To address this issue, the paper formulates reasoning distillation as a constrained RL problem, maximizing task rewards subject to a worst-case constraint on teacher log-likelihood. Experiments on mathematical reasoning and code generation tasks demonstrate improvements in reasoning fidelity and reasoning success rate.

Critical strengths
  1. The motivation and methodology are clearly presented, making the paper easy to follow.
  2. The observation that a chain of thought is only as valid as its weakest link is interesting and worth further investigation.
  3. The paper provides clear and intuitive examples that effectively illustrate the motivation behind the proposed method.
Critical weaknesses
  1. Limited improvement in final task performance. The gains in Final Answer Correctness (FAC) over GRPO are marginal. The paper argues that flawed intermediate reasoning "will not generalize" and claims to achieve "superior, trustworthy reasoning capabilities." However, if the proposed method genuinely improves reasoning behavior and reliability, I would expect more substantial improvements in independent task-level accuracy or generalization, rather than primarily in teacher fidelity.

  2. Limited reliability of Teacher Likelihood as a proxy for reasoning correctness. Although the additional experiments show a positive correlation, the reported Spearman ρ = 0.231 and AUC = 0.63 suggest relatively weak discriminative power. Low teacher likelihood does not necessarily indicate incorrect reasoning, as it may also reflect valid alternative reasoning paths. This raises concerns about using such an imperfect proxy as a hard constraint.

  3. Potential suppression of self-correction behavior. Once a prefix violates the constraint, the entire trajectory is penalized, even if the model subsequently identifies and corrects its mistake. This may discourage the development of self-correction abilities, which are essential for robust reasoning. A capable reasoning model should learn to recognize and recover from errors arising within its own capabilities, rather than simply avoid making them. The paper lacks experiments investigating this potential limitation.

Questions for authors
  1. Can the proposed method achieve more substantial improvements in independent task-level accuracy or generalization compared to GRPO? If not, what might explain the gap between improved reasoning fidelity and the limited gains in final answer correctness?

  2. How does the proposed hard constraint affect the model's ability to self-correct? In particular, could penalizing an entire trajectory after an intermediate constraint violation discourage the model from learning to recognize and recover from its own reasoning mistakes?

1 comment

Thank you for your review. This review engages with the paper's central question, and we answer its two questions directly.

Q1 / W1. FAC gains over GRPO are small; better reasoning should show up as accuracy. GRPO maximises final-answer correctness and nothing else. We maximise the same quantity under an added constraint. A constrained optimiser cannot be expected to beat the unconstrained one on the unconstrained objective. The best possible outcome is to match it, which means the constraint cost nothing in accuracy. Table 1 shows that: FAC 34.86 vs 34.84 on Qwen/MATH, 76.14 vs 76.24 on GSM-Symbolic, 20.54 vs 18.78 on Llama/MATH.

Asking for a FAC gain as proof of better reasoning assumes that FAC measures reasoning. The paper disputes that with Figure 1, where the GRPO student is scored correct. What changes at equal FAC is what the correct answers are made of. On Qwen/MATH, the judge flags an error in 21% of GRPO's correct answers (FAC 34.84, RSR 27.54) and in 14% of ours (34.86, 30.00). On code, where the reward/FAC is even easier to game, GRPO reaches 92.0% pass@1 but it fails completely on RSR with of 1.8%. On Llama/MATH the flagged share is the same for both methods, 31%, and the gain arrives through FAC instead.

We agree that the RSR margins are a few points. GSM-Symbolic is the closest we have to a generalisation test, since we train on GSM8K. There the two methods tie on FAC and ours leads on RSR, 65.82 vs 64.36. Whether sound derivations pay off in accuracy under a larger shift is something our experiments do not establish. The introduction's remark that the broken logic "will not generalize" refers to the derivation in Figure 1, not to a measured benchmark gap.

W2. ρ = 0.231 and AUC = 0.63 are weak grounds for a hard constraint. The numbers are modest, and we report them for that reason. Two points on reading them. A Spearman coefficient against a binary label cannot reach 1. For a label with base rate p, its maximum is √(3p(1−p)), about 0.85 here, so 0.23 is not on the usual scale.

More important, a threshold does not need the proxy to be a classifier. It needs the far tail of the proxy to be enriched in failures, so that cutting the tail removes more bad trajectories than good ones. A noisy proxy argues for a loose threshold, not for no threshold. That is where our sweep ended: the loosest d is the best, and tightening it hurts.

The alternative on offer is a KL penalty, which applies the same noisy signal to every token of every trajectory as a training target. If the proxy is too weak to bound a tail with, it is too weak to imitate densely. Valid alternative paths with low teacher likelihood, which the reviewer rightly raises, are our first stated limitation. Inside the budget such paths cost nothing, which is not true under KL.

W3 / Q2. Self-correction. Thank you for this interesting question. We ran no experiment on self-correction. Three observations.

First, the constraint does not prohibit erring and recovering. It bounds the teacher's accumulated surprise along the prefix. One wrong step followed by a correction is absorbed unless that step alone pushes the running average past d, and Proposition 3.1(a) gives the slack available, (k+1)·d. The other two reviews criticise the constraint for having this slack. A per-token bound would answer them and would make recovery impossible; we rejected it in Appendix D.

Second, what a teacher signal can say about recovery depends on the teacher. If a teacher writes "that is wrong, let me redo it" after a slip, those tokens are likely under the teacher given the prefix, and they cost little.

Third, the concern is not specific to a hard bound. A KL term also penalises an error-and-recovery trace that the teacher would not have written, at every token of it. We agree that a hard bound is the stricter of the two once the boundary is crossed. With thinking-style teachers that explore and backtrack, the natural extension applies the constraint to the committed solution and leaves the scratch work alone.


We thank the reviewer again for the questions; we hope these answers are useful.

Reviewer_2B4iOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses reward hacking in RL-based reasoning distillation, where student models find shortcuts to reach correct answers through flawed reasoning steps. The authors argue that soft average penalties allow models to compensate for severe reasoning errors elsewhere in the sequence. To address this, they propose an unaugmented constrained RL formulation enforcing a worst-case prefix average bound on teacher negative log-likelihood. By leveraging absorbing infeasible states, the student remains independently deployable without test-time teacher access or state augmentation. They also introduce a token-level policy gradient decomposition for variance reduction and evaluate on math and code generation tasks.

Critical strengths

The paper identifies a meaningful limitation of average-based distillation penalties in reasoning tasks and proposes a clear alternative based on worst-case prefix constraints.

Critical weaknesses

1.Teacher likelihood is used as a proxy for reasoning validity, but it primarily measures distributional agreement rather than logical correctness. It can favor stylistically teacher-like but logically invalid continuations while penalizing valid reasoning that differs from the teacher.

2.The hard-constraint claim is stronger than the actual method. Training uses a finite terminal penalty, infeasible trajectories are still explored, and there is no test-time feasibility check. The almost-sure guarantee holds only in an ideal limiting case and depends on a nontrivial feasibility assumption.

3.The proposed worst-case quantity still permits sustained moderate deviations below the threshold. A single later step may also deviate more as the prefix grows, so it does not strictly enforce validity at every reasoning step.

4.The policy-gradient decomposition is largely a standard likelihood-ratio identity combined with Rao–Blackwellization. The paper does not clearly establish its end-to-end optimization benefit over directly using the same immediate-reward estimator, particularly within the GRPO/PPO-style implementation.

5.The comparison with state-augmented constrained RL remains mostly analytical and retrospective. The central claimed advantage over actually trained state-augmented alternatives is therefore not fully demonstrated.

Questions for authors

See weaknesses.

1 comment

Thank you for the review. Of the five weaknesses, three restate things the paper says itself in the Limitation section.

W1. Teacher likelihood is a proxy for validity. Agreed, and Appendix G opens with it: a step that is "logically valid but stylistically unusual may incur a high cost". The review omits two things. First, this is the premise of all teacher-based distillation, including every baseline in Table 1. Second, a constraint uses the proxy more sparingly than a KL penalty does. KL pulls the student toward the teacher at every token. Our budget does nothing until the worst prefix-average crosses d. At d = −ln(1/4), a valid but unusual continuation costs nothing as long as the teacher gives the prefix a geometric-mean token probability of at least 0.25.

Table 1 shows that this slack matters. Tightening d to −ln(3/4) lowers FAC and RSR in all three settings, for example RSR 26.36 against 30.00 on Qwen/MATH. Appendix E measures how informative the proxy is. The correlation is positive in every run.

W2. The hard-constraint claim is stronger than the method. The abstract and the introduction make the claim "in the penalty limit", and Theorem 3.3 is stated for n → ∞. Section 4.2 reports constraint satisfaction of about 100% in training and about 80% at test. We claim no guarantee for finite n or for unseen prompts, and Appendix G lists distribution shift as a limitation.

On the remaining items: exploring infeasible trajectories is how any penalty method learns where the boundary is, and the theorem concerns the optimum. The absence of a test-time check is the design goal. A check needs the teacher at inference, and removing the teacher at inference is what distillation is for. The feasibility assumption has its own remark in Appendix A.2, with a sufficient condition and a recipe for choosing d so that it holds.

W3. Moderate deviations below the threshold remain, and a late step may deviate more. Both statements are correct and both are in the paper. Proposition 3.1(a) gives the bound: the cost at step k is at most (k+1)·d. We claim no per-step validity. Section 3.1 states what is removed: "future compensation after a prefix has already crossed the boundary".

Under a whole-sequence average, a student can deviate and then pad. Under the prefix bound, slack must be earned before it is spent, and a violation cannot be repaid afterwards. Appendix D explains why we rejected a flat per-token bound: single formatting tokens routinely carry high teacher NLL, so a per-token threshold fires on style. Sustained moderate deviation below d is a smaller student reasoning differently from its teacher, within tolerance. A constraint that forbade it would be plain imitation.

W4. The decomposition is a standard identity plus Rao-Blackwellization, and its end-to-end benefit is not shown. The identity is standard, and the paper says so. Whether the identity can be used depends on the reward, and that part is specific to our construction. The single-step term is an exact expectation over the vocabulary. It needs the immediate reward of every candidate next token at every position. For a generic reward that is intractable, as Appendix A.4 notes.

It is tractable here because of how the MDP is built. The task reward is paid only at the end, so the immediate reward of an intermediate token is the constraint penalty alone. Infeasibility is absorbing and is decided by the prefix (Lemma 3.2). Whether a candidate token triggers the −n penalty therefore depends only on the prefix and on that token’s teacher log-probability. One teacher forward pass, the same one that scores the sampled trajectory, gives the immediate reward of the whole vocabulary. Under a whole-sequence average, feasibility is known only at the last token, the immediate reward carries no constraint information, and the single-step term is empty. Our claim is this link between the form of the constraint and the estimator, not the identity.

On the end-to-end benefit, we agree that the paper does not show it, but we should have mention it, because the term is not optional. In our runs without it, the Qwen student failed to train. The penalty and the task reward then reach the policy only as one scalar return per trajectory, and that signal was too sparse. The single-step term adds a signal at every position, for every token that would cross the boundary.

W5. The comparison with state-augmented constrained RL is analytical and retrospective. Correct, and Appendix D describes itself as a retrospective analysis. There is a reason Table 1 has no trained Saute student. Saute's policy takes the remaining budget as an input, and computing that budget at each decoding step requires the teacher. A Saute student ships with its teacher, which is the situation Section 3.2 exists to avoid.


We think stating limitations plainly is the right practice, and we hope readers weigh ours as context for the results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.