acceptodds
Under review as a conference paper at ICLR 2027

MILE: Milestone-Guided Reinforcement Learning for Mathematical Reasoning

Abstract

Reinforcement learning has shown strong potential for improving the mathematical reasoning capabilities of large language models (LLMs). However, existing methods often depend on sparse outcome feedback, manually designed process signals, or separately trained reward models, leaving expert reasoning trajectories underexploited as a rich source of dense supervision. We propose MILE, a milestone-guided reinforcement learning framework that models reasoning progress through intermediate states characterized by their future solvability. MILE extracts milestones from multiple expert solutions and constructs dense rewards for student reasoning transitions by measuring both progress and directional consistency. These rewards are incorporated into a two-stage GRPO optimization framework, enabling fine-grained transition-level learning followed by trajectory-level reasoning optimization. Experiments on mathematical reasoning benchmarks show that MILE consistently outperforms representative RL and distillation baselines. Further analyses validate the effectiveness of its components, scalability with expert trajectories, and improved reasoning efficiency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reasoning, Large Language Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_aquXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MILE, a framework that extracts milestones by estimating the future solvability of intermediate states in expert reasoning trajectories. It introduces progress and directional consistency rewards and employs a two-stage GRPO optimization strategy to improve mathematical reasoning performance. Experimental results demonstrate improvements across multiple mathematical reasoning benchmarks. However, the core design is closely related to existing value estimation and reward shaping methods, and there remains a gap between the theoretical guarantees and the practical algorithm. Overall, the paper presents relatively comprehensive experiments, but its methodological novelty and theoretical rigor require further improvement.

Critical strengths
  1. MILE integrates expert milestone extraction, progress reward estimation, directional consistency rewards, and two-stage GRPO optimization into a unified framework. Each component serves a distinct role in progress modeling, reward construction, and policy optimization. The overall design is clear and has practical feasibility.
  2. MILE achieves the highest average Pass@1 across multiple mathematical reasoning benchmarks, demonstrating the strong performance of the proposed method.
Critical weaknesses
  1. The idea of MILE is to evaluate intermediate reasoning states through Future Solvability and construct the Progress Reward based on value changes between consecutive states. However, this mechanism is highly similar to existing Value-Based Process Supervision and Potential-Based Reward Shaping methods. Moreover, the Direction Reward essentially imposes constraints based on the representation directions of expert trajectories, while the two-stage GRPO combines existing optimization strategies. Therefore, the paper's novelty primarily lies in the integration of existing techniques rather than the introduction of fundamentally new mechanisms.
  2. Several theoretical results in the paper rely on strong assumptions and do not fully align with the actual algorithm. For example, Theorem E.5 requires the retrieval bandwidth to decrease as the sample size increases, whereas the practical implementation uses a fixed retrieval temperature. The conditional zero-mean noise assumption in Assumption E.3 also be affected by selection bias in milestone extraction. Furthermore, Theorem E.8 only establishes the consistency of expert direction estimation without demonstrating that it leads to genuine reasoning progress. Theorem E.13 provides a no-cycling guarantee for the Safe Shaping reward, which is not used in the actual implementation. Therefore, the current theoretical analysis is insufficient to fully justify the effectiveness and stability of MILE's core reward mechanisms.
Questions for authors
  1. Theorem E.5 requires the kernel bandwidth to approach zero as the sample size increases, whereas the actual algorithm uses a fixed retrieval temperature of τr = 0.15. Since a fixed bandwidth may introduce irreducible smoothing bias, what is the corresponding finite-sample error bound?
  2. When the numerator of the Progress Reward is negative, increasing the number of tokens reduces the magnitude of the negative reward. Could the model exploit this mechanism by generating longer incorrect reasoning steps to reduce penalties, while using shorter steps with positive rewards to achieve a net positive reward?
  3. The paper uses False Progress to measure the discrepancy between high progress scores and actual improvements in solvability. However, this metric may be sensitive to the choice of threshold and the method used to estimate reference solvability. Could the author provides results under different thresholds and analyze the correlation between this metric and the final Pass@1 performance?
4 comments

We thank the reviewer for the detailed review. We respond to each point below. All changes to the paper are marked in red in the revised PDF.

Q1: Evaluating intermediate states by future solvability and building a progress reward from value changes between consecutive states is highly similar to existing value-based process supervision and potential-based reward shaping. The direction reward essentially imposes constraints from the representation directions of expert trajectories, and the two-stage GRPO combines existing optimization strategies. The novelty therefore appears to lie in integrating existing techniques, not in introducing a fundamentally new mechanism.

A1: MILE's main contribution lies in how it constructs progress supervision from multiple expert trajectories and uses it to guide student learning. This is achieved through three key components.

First, MILE selects sparse milestones from verified expert solutions at points where future solvability improves. This avoids the need to supervise every reasoning step and creates a reusable milestone pool containing states from multiple solution paths.

Second, MILE retrieves nearby milestones to estimate both the student's progress and the relevant expert direction. The progress reward measures changes in estimated solvability, while the direction reward encourages alignment with nearby expert transitions. This provides an additional form of supervision beyond value-based methods, guiding the student on how to make progress, not just whether progress has been made.

Third, MILE uses a two-stage training pipeline to apply these rewards at different levels. Stage 1 optimizes individual reasoning transitions from expert prefixes, while Stage 2 trains on complete solutions with final-answer supervision. The ablation results in Table 2 demonstrate the importance of these components. MILE-Full achieves 86.56% Overall Pass@1, compared with 83.68% using uniform milestone selection, 84.34% without Stage 1, and 82.86% without Stage 2. These results support the effectiveness of combining sparse milestone selection, retrieval-based rewards, and two-stage training.

Q2: Several theoretical results rest on strong assumptions and do not align with the actual algorithm. Theorem E.5 requires the retrieval bandwidth to shrink as the sample size grows, whereas the implementation uses a fixed retrieval temperature. The conditional zero-mean noise assumption of E.3 may be violated by selection bias in milestone extraction. Theorem E.8 establishes only the consistency of expert-direction estimation, without showing that it leads to genuine reasoning progress. Theorem E.13's no-cycling guarantee concerns a Safe Shaping reward that is not used in the implementation. The analysis therefore does not yet justify the effectiveness and stability of MILE's core reward mechanisms.

A2: We thank the reviewer for raising these questions. We clarify below how the theoretical results relate to our practical implementation.

  • E.5. Theorem E.5 establishes the convergence of the retrieval potential under sufficient coverage and shrinking-bandwidth conditions. In practice, we use a fixed retrieval temperature of 0.15. We provide a finite-sample analysis in our response to Q3 below.

  • E.3. Our theory assumes zero-mean annotation noise, while practical milestone selection uses finite-sample solvability estimates and may introduce selection bias. The implementation therefore approximates the idealized setting. Solvability-based selection achieves 86.56% Overall Pass@1, compared with 83.68% for uniform selection and 83.42% for dense milestones, suggesting that this approximation remains effective in practice.

  • E.8. Expert directions connect consecutive milestones selected based on improvements in future solvability, capturing how an expert moves toward a more solvable state. Theorem E.8 characterizes the estimation accuracy of these directions. Progress is measured separately through changes in estimated solvability and receives a larger reward weight (1.0 vs. 0.25 for direction, Table 6). Thus, the direction reward complements the progress signal by providing additional guidance on how to advance along expert trajectories.

  • E.13. Theorem E.13 analyzes a safe-shaping variant that is not included in our current implementation. We leave its practical integration to future work and have clarified in the discussion following Theorem E.13 that this reward is not used in any of our experiments (lines 1540–1542).

Q3: Theorem E.5 requires the kernel bandwidth to approach zero as the sample size grows, whereas the algorithm uses a fixed retrieval temperature τr=0.15\tau_r = 0.15. Since a fixed bandwidth may introduce irreducible smoothing bias, what is the corresponding finite-sample error bound?

A3: We agree that a fixed retrieval temperature τr=0.15\tau_r=0.15 can leave a non-vanishing smoothing bias. The implementation approximates the idealized setting of Theorem E.5, which requires a shrinking bandwidth.

For a fixed query ss, let Φ^(s)=∑iwi(s)p^(mi)\widehat\Phi(s)=\sum_i w_i(s)\widehat p(m_i) and assume the target solvability Φ⋆\Phi^\star is LL-Lipschitz. Under the idealized assumption that labels in [0,1][0,1] are conditionally independent and unbiased given the selected milestone locations, Hoeffding's inequality gives, with probability at least 1−δ1-\delta,

∣Φ^(s)−Φ⋆(s)∣≤Lr(τr;s)+log⁡(2/δ)2Meff(s).|\widehat{\Phi}(s)-\Phi^\star(s)| \le Lr(\tau_r;s)+\sqrt{\frac{\log(2/\delta)}{2M_{\mathrm{eff}}(s)}}.

Here, Meff(s)=1/∑iwi(s)2M_{\mathrm{eff}}(s)=1/\sum_i w_i(s)^2 is the effective number of retrieved milestones, and r(τr;s)=∑iwi(s)d(s,mi)r(\tau_r;s)=\sum_i w_i(s)d(s,m_i) is the weighted average retrieval distance. This bound retains the smoothing bias at fixed temperature. Selection based on noisy gains may violate the conditional assumptions, so the bound is not an unconditional guarantee for the implementation.

In practice, at τr=0.15\tau_r=0.15, the retrieval potential has a correlation of 0.76 with independently estimated student continuation success (see our response to Reviewer mNNm). Together with the selection ablations above, this supports the effectiveness of the practical approximation.

Q4: When the numerator of the progress reward is negative, increasing the number of tokens reduces the magnitude of the negative reward. Could the model exploit this by generating longer incorrect reasoning steps to dilute penalties while keeping positively rewarded steps short, and so obtain a net positive reward without net progress?

A4: We acknowledge that length normalization can introduce this failure mode. As analyzed in Appendix E.7, a short forward step followed by a longer backward step may receive a positive cumulative reward even when the potential returns to its initial value. However, we found its impact on training to be limited, for three reasons. First, the progress reward is grounded in teacher milestones, independent of the student's own predictions. For a sufficiently unproductive step, the decrease in potential can outweigh the effect of length normalization, so a longer step may still receive a larger penalty (lines 195–197). Second, each trajectory has a fixed token budget, meaning that generating unnecessarily long steps leaves fewer tokens to complete the solution. Third, in Stage 2, the dense rewards of trajectories with incorrect final answers are scaled by 0.05 (Table 6), substantially reducing the benefit of accumulating rewards without solving the problem. Together, these mechanisms help limit the incentive for reward exploitation.

Q5: False Progress measures the discrepancy between high progress scores and actual improvements in solvability, but it may be sensitive to the choice of threshold and to the method used to estimate reference solvability. Could the authors provide results under different thresholds and analyze the correlation between this metric and final Pass@1 performance?

A5: To evaluate the sensitivity of False Progress to the threshold, we recomputed the metric using the top 5%, 10%, and 20% of prefix scores, while keeping the reference checkpoint and the 1,024-token continuation budget fixed (Appendix B.3).

Top-score fractionFalse Progress (%)Pass@1
5%6.286.56
10%6.986.56
20%7.786.56

False Progress varies by only 1.5 percentage points as the threshold changes fourfold, suggesting that the metric is relatively stable across different cutoff choices. These results indicate that our conclusions are not sensitive to the specific threshold used. Across the ten configurations in Table 2, the reported means yield a Spearman correlation of -0.782 between False Progress and Overall Pass@1. This suggests that lower False Progress is associated with higher final accuracy.

AuthorsOpenReview ↗

Appreciation for Your Feedback

Thank you very much for your positive feedback and for acknowledging that your concerns have been fully addressed. We truly appreciate your time and careful consideration.

Since the main concerns are now resolved through our rebuttal and clarifications, we would sincerely appreciate it if you could kindly consider raising your score accordingly. This would mean a great deal to us.

Thank you for your detailed responses. My concerns have been largely addressed, and I have raised my rating to 3.

Reviewer_wtMDOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

MILE addresses the sparsity of final-answer rewards by deriving dense progress supervision from expert mathematical reasoning trajectories. It identifies milestones using answer-checked teacher continuations under several token budgets, then transfers their progress scores to student prefixes through a frozen encoder. Progress differences and alignment with aggregated expert directions guide two-stage GRPO training; MILE-Full additionally incorporates final-answer supervision. Experiments across five benchmarks, additional student backbones, component ablations, and token-budget studies report improvements over outcome-based RL, process-based RL, and distillation baselines.

Critical strengths
  • The reward construction is well motivated and carefully designed. It distinguishes useful progress from local step correctness, combines outcome-checked milestone annotations with progress and direction signals, and reuses a fixed milestone store without a separately trained reward model. The local-transition and full-trajectory training stages form a coherent approach to transferring expert reasoning into policy improvement.
  • The empirical evaluation is broad and supports a meaningful improvement.
Critical weaknesses

The semantic reliability of the embedding-based rewards remains incompletely established: averaging directions from distinct valid branches can favor a displacement between those branches over either valid direction. If an incorrect or unproductive step has that mean-aligned displacement, the direction component would prefer it. This is a potential failure mode, not demonstrated exploitation in the reported models. Final-answer supervision may mitigate it in Stage 2, but Stage 1 lacks that safeguard against reward hacking in this mode.

Questions for authors
  • Have branching cases been tested where an incorrect or unproductive student step aligns with the mean of distinct valid expert directions? Does the combined reward reliably rank valid branches above such steps, including during Stage 1?
  • Could the authors compare milestone-oriented prompting on the shared SFT or GRPO checkpoint against MILE under comparable inference budgets, applying the same prompt to MILE and using only demonstrations from training problems?
  • How was overlap between the training sources and evaluation benchmarks detected, including reformatted or paraphrased problems?
3 comments

We thank the reviewer for the constructive review. We address each point below and report the additional experiments we ran in response. All changes to the paper are marked in red in the revised PDF.

Q1: The semantic reliability of the embedding-based rewards is not fully established: averaging directions from distinct valid branches can favor a displacement between those branches over either valid direction, so an incorrect or unproductive step with that mean-aligned displacement would be preferred by the direction component. This is a potential failure mode, not demonstrated exploitation. Final-answer supervision may mitigate it in Stage 2, but Stage 1 has no such safeguard against reward hacking of this kind.

A1: We agree that uniformly averaging expert directions from different reasoning branches can produce a direction that falls between valid paths and may not correspond to meaningful progress. To address this, MILE uses soft weighting, assigning higher weights to milestones closer to the current student state. When the student is closer to a particular expert branch, milestones from that branch contribute more to the reference direction, encouraging the student to follow a consistent reasoning path. Steps aligned with this reference direction receive higher cosine similarity and thus higher direction rewards.

Our experiments support the effectiveness of this design. As shown in Table 2, adding the direction reward improves Overall Pass@1 from 84.58% to 86.56% and reduces False Progress from 12.1% to 6.9%, suggesting that it helps discourage plausible but unproductive reasoning steps. To further isolate the effect of soft weighting, we conducted an additional experiment replacing it with uniform averaging:

Direction aggregationSuccess rate (%)
Soft weighting (MILE)86.56
Uniform averaging75.60

Uniform averaging reduces performance by 10.96 percentage points. These results suggest that favoring nearby expert milestones helps maintain consistent reasoning directions and mitigates the failure mode raised by the reviewer.

Q2: Have branching cases actually been tested, in which an incorrect or unproductive student step aligns with the mean of distinct valid expert directions? Does the combined reward reliably rank valid branches above such steps, including during Stage 1?

A2: We did not observe this mean-aligned failure mode in the examples we examined. In Stage 1, each reasoning step is evaluated using both progress and direction rewards, with a larger weight assigned to progress (λp=1.0\lambda_p = 1.0 vs. λd=0.25\lambda_d = 0.25, Table 6). Even if an incorrect or unproductive step aligns well with the average expert direction, its low estimated progress can prevent it from receiving a high overall reward. To examine this, we compared the combined Stage 1 rewards for valid and invalid reasoning branches:

Reasoning branchStage 1 reward
Valid0.267
Invalid-0.023

Valid branches receive substantially higher rewards than invalid ones, suggesting that the combined reward can distinguish productive reasoning from misleading directional alignment. This distinction is particularly important in Stage 1, where final-answer gating is not applied.

Q3: Could milestone-oriented prompting on the shared SFT or GRPO checkpoint be compared against MILE under comparable inference budgets, applying the same prompt to MILE as well and using demonstrations drawn only from training problems?

A3: We thank the reviewer for suggesting this control experiment. We conducted the experiment as proposed, using a prompt containing three training problems, along with their solution trajectories and milestones, to encourage the model to take similarly meaningful reasoning steps. The same prompt was applied to all five checkpoints (Base, SFT-K, GRPO, MILE-Core, and MILE-Full) to ensure a fair comparison. All models were evaluated with the same generation limit of 4,096 new tokens per response, consistent with the main evaluation protocol in Table 7.

For comparison, the results without milestone prompting are the Overall Pass@1 scores reported in Table 1.

ModelPass@1 w/o prompting (%)Pass@1 w/ prompting (%)
Base model70.8471.6
SFT-K76.3877.6
GRPO81.0780.1
MILE-Core84.0679.2
MILE-Full86.5677.5

Interestingly, milestone prompting provides small improvements for the Base and SFT-K models, but reduces performance for the RL-trained checkpoints, particularly MILE-Core and MILE-Full. One possible explanation is that the additional prompting interferes with the reasoning behaviors learned during RL training. More importantly, simply providing milestones at inference time does not reproduce the gains achieved by MILE's training procedure. These results further support the importance of incorporating milestone-based rewards directly into training.

Q4: How was overlap between the training sources and the evaluation benchmarks detected, including reformatted or paraphrased versions of benchmark problems?

A4: We filtered training problems based on their original source. Both Big-Math and OpenR1-Math-220k provide source labels for each problem, allowing us to exclude all problems originating from the evaluation benchmarks. This filtering also covers reformatted or paraphrased versions, as these retain their original source labels even when the wording changes. We applied this filter before generating expert trajectories, ensuring that none of the excluded problems were used for milestone construction or any subsequent training stage.

AuthorsOpenReview ↗

Appreciation for Your Feedback

Thank you very much for your positive feedback and for acknowledging that your concerns have been fully addressed. We truly appreciate your time and careful consideration.

Since the main concerns are now resolved through our rebuttal and clarifications, we would sincerely appreciate it if you could kindly consider raising your score accordingly. This would mean a great deal to us.

Reviewer_mNNmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MILE, a RL method for mathematical reasoning guided by milestones. The approach collects multiple correct expert trajectories and utilizes a teacher model to generate subsequent steps, thereby evaluating whether an intermediate prefix, representing a state within the reasoning process, is likely to lead to a successful solution. Based on this evaluation, key milestones are identified. A fixed student encoder retrieves these milestones to construct reward signals that measure the progress and direction of the reasoning process. The student model is then optimized via a two-stage GRPO process: the first stage involves training based on local transitions derived from expert prefixes, while the second stage involves training based on complete reasoning trajectories. Across several mathematical reasoning benchmarks, MILE consistently outperforms various baseline methods based on reinforcement learning and knowledge distillation.

Critical strengths
  • Using multiple expert trajectories to provide intensive supervision during online reinforcement learning, without needing to query the teacher’s logits.

  • The paper includes various baseline comparisons and component ablations, alongside reported improvements across two student backbones.

Critical weaknesses
  • I'm not sure the core idea of "rewarding reasoning progress through future solvability" is innovative. Prior work [1] already defines progress through changes in future success probability, and [2] studies solve-rate gains of reasoning prefixes. Both are cited, but the paper does not clearly explain what is fundamentally different beyond milestone retrieval and the two-stage training pipeline. A more direct comparison with these works would help to establish the actual contribution.

  • In §2.2, future solvability is estimated from teacher continuations, not from the student policy being optimized. A budget is counted as solved if any of three continuations succeeds, and scores are averaged across different token budgets. An increase may therefore reflect shorter remaining solutions or sampling noise, rather than a genuinely better reasoning state for the student. The milestone threshold of 1/|B| is also as small as a one-budget change. Could the authors test whether the retrieved potential actually predicts student continuation success, using independent samples and controlled generation budgets?

  • In Appendix B.1, the retrieval entropy is divided by log⁡K\log K. This expression is undefined when K=1K=1, yet Figure 2(b) provides a value for this case. More importantly, even if all KK expert trajectories are identical, uniform retrieval weights would still yield the maximum normalized entropy—despite the actual path diversity being zero. Therefore, I argue that the entropy results presented in Section 3.4 and Appendix E.6 do not demonstrate that MILE effectively learns from diverse reasoning strategies. This metric requires revision, or it should be supplemented with an explicit analysis of trajectory diversity.

  • It is unclear whether the PRM baseline mentioned in Appendix D.3 incorporates the same components used in this paper? Namely, "direction rewards," first-stage training based on expert prefixes, "answer gating," and specific optimization budgets, which makes it difficult to isolate the contribution of the retrieval mechanism itself. Moreover, in the comparison with OPD, it should be noted that OPD utilizes online supervision from a single teacher model, whereas MILE relies on offline-constructed milestones (derived from multiple teacher models). The discussion on efficiency in the paper focuses primarily on the online training phase; Table 5 indicates that the offline construction process consumed 1,184 H200 GPU-hours, with an additional 320 GPU-hours required for MILE training. I suggest a more rigorous comparison that accounts for the total end-to-end cost.

  • There are several formatting issues that should be fixed. Line 107 has a syntax issue ("where is generated by multiple teacher reasoning models"). Appendix A says milestones are highlighted in red, while Table 3 uses bold. Table 2 describes MILE-Core as "w/o answer reward", but Table 9 uses "w/o answer gating", which may refer to different settings. Also, the values of λo\lambda_o (line 226) and γ\gamma (line 196) used in the reward equation do not appear to be specified in Table 6. The reported 16 evaluation seeds should be distinguished from independently repeated training runs.

[1] Setlur et al. Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning. ICLR 2025. https://proceedings.iclr.cc/paper_files/paper/2025/hash/98711dea460bdefe0e651ca23ec98ba-Abstract-Conference.html

[2] Zhou et al. From Correctness to Utility: Gain-Based Prefix Evaluation for LLM Reasoning. Findings of EMNLP 2026. https://aclanthology.org/2026.findings-emnlp.861/

Questions for authors

Refer to the Weaknesses section.

4 comments

We thank the reviewer for the careful reading. We respond to each point below. All changes to the paper are marked in red in the revised PDF.

Q1: Is "rewarding reasoning progress through future solvability" genuinely novel, given that [1] already defines progress as the change in future success probability and [2] studies solve-rate gains of reasoning prefixes? Beyond milestone retrieval and the two-stage pipeline, what is fundamentally different, and what does a direct comparison with these two works show?

A1: Future solvability is a common idea shared by MILE and prior work, but the key difference lies in how it is used to provide supervision. Specifically, [1] trains a process advantage verifier based on changes in future success probability under a prover policy, while [2] trains a prefix utility model using the improvement in solve rate conditioned on a reasoning prefix. In contrast, MILE selects a sparse set of milestones from verified expert solutions, focusing on points where future solvability improves. During training, rewards are obtained by retrieving nearby milestones, which provide two complementary signals: the change in estimated solvability and alignment with the expert direction. Our two-stage GRPO pipeline uses these signals at different levels, first supervising local transitions and then optimizing complete solutions. As shown in Table 1, MILE-Full achieves 86.56% Overall Pass@1, outperforming GRPO (81.07%) and OPD (84.10%) and establishing a new state of the art among the compared methods.

Q2: In §2.2 future solvability is estimated from teacher continuations, not from the student policy being optimized. A budget counts as solved if any of three continuations succeeds, scores are averaged across token budgets, and the milestone threshold 1/∣B∣1/|B| is no larger than a one-budget change. An increase may therefore reflect a shorter remaining solution or sampling noise, not a genuinely better reasoning state for the student. Does the retrieved potential actually predict student continuation success, under independent samples and controlled generation budgets?

A2: Future solvability measures whether a teacher model can complete a reasoning prefix within a given token budget. If at least one of three attempts succeeds, we consider the state solvable under that budget. The intuition is that states closer to the solution generally require fewer remaining reasoning steps and can therefore be solved under more budgets, resulting in higher future-solvability scores.

We agree that sampling noise is a valid concern, especially since the 1/81/8 milestone threshold corresponds to a single increment in the eight-budget score. To examine whether our milestones capture meaningful progress, we conducted an additional experiment using the student model as an independent evaluator. Specifically, we constructed milestones using the four teacher models from our main experiments (Llama 4 Scout, Qwen3.5-122B-A10B, Qwen2.5-72B-Instruct, and gpt-oss-120b) and sampled 100 trajectories. For each reasoning state, we generated independent continuations using the student model (Qwen3.5-4B) under a fixed token budget to estimate its continuation success rate V^π(s)\widehat V_\pi(s). We then measured the correlation between the retrieval potential Φ(s)\Phi(s) and V^π(s)\widehat V_\pi(s) across these states.

We observed a positive correlation of 0.76, suggesting that the retrieval potential captures meaningful information about a state's remaining difficulty that transfers from the teacher to the student. While this does not eliminate the effect of sampling variance, it provides empirical support for the usefulness of our milestone-based supervision.

Q3: The retrieval entropy of Appendix B.1 is divided by log⁡K\log K, which is undefined at K=1K=1, yet Figure 2(b) reports a value there. More importantly, KK identical expert trajectories with uniform retrieval weights would still attain the maximum normalized entropy even though path diversity is zero, so the entropy results in §3.4 and Appendix E.6 do not establish that MILE learns from diverse reasoning strategies. The metric should be revised, or supplemented with an explicit analysis of trajectory diversity.

A3: We thank the reviewer for pointing this out. At K=1K=1, cross-trajectory entropy is undefined under the original normalization. The value previously plotted at this point was instead the normalized entropy over milestones within the single trajectory. We have revised Appendix B.1 to define cross-trajectory retrieval entropy as zero when K=1K=1, ensuring that the metric consistently measures how retrieval weights are distributed across trajectories (lines 777–780).

We also agree that high retrieval entropy does not necessarily imply diverse reasoning strategies. Our metric measures how broadly retrieval draws from stored trajectories, without directly assessing their semantic diversity. Since our dataset contains trajectories generated by different teacher models, higher retrieval entropy indicates that the student draws supervision from a broader set of expert trajectories. We have clarified this distinction in Appendices B.1 (lines 787–792) and E.6 (lines 1453–1457).

To further examine the diversity of reasoning strategies, we conducted an additional analysis on 1,000 randomly sampled problems, covering 8,000 expert trajectories in total. For each problem, we asked GPT-6 to examine its eight trajectories and identify how many distinct solution strategies they represented, treating paraphrases or minor variations of the same approach as a single strategy. We found an average of 6.89 distinct strategies per problem, with a minimum of 5 and a maximum of 8. The results suggest that our expert trajectories contain substantial diversity in reasoning approaches, and provide additional support for the use of multiple expert trajectories in MILE.

Q4: Three concerns about attribution, fairness and cost. (i) Does the PRM baseline of Appendix D.3 carry the same components as MILE, namely the direction reward, Stage 1 on expert prefixes, answer gating, and the optimization budget, or is the gap attributable to something other than the retrieval mechanism? (ii) OPD uses online supervision from a single teacher, whereas MILE relies on offline milestones derived from multiple teachers, which is not an equal resource setting. (iii) The efficiency discussion covers only online training, while Table 5 reports 1,184 H200 GPU-hours for offline construction plus 320 for training. A rigorous comparison should account for the total end-to-end cost.

A4: The PRM baseline follows the same two-stage GRPO pipeline as MILE, with the same number of training rounds and the direction reward set to zero. It predicts future solvability from each reasoning prefix and uses the difference between consecutive predictions as the progress reward. We also apply the same answer gating mechanism to both methods. As shown in Table 10, PRM and MILE share the same training labels, SFT checkpoint, verifier, and rollout budget. We have added these details to Appendix D.3 (lines 1040–1045). This makes PRM a parametric counterpart to MILE, with the main difference being how the progress signal is obtained: PRM uses a learned value function, while MILE retrieves milestones from an offline store. The performance gap therefore helps isolate the contribution of milestone-based retrieval.

We acknowledge that OPD and MILE do not use identical teacher resources, as our OPD implementation uses a single teacher model, while MILE constructs milestones from four teachers. We include OPD because it represents a different approach to learning from teacher models that does not require an explicit reward model. We believe it provides a meaningful comparison with this alternative paradigm, although it is not a strictly resource-matched baseline.

We also clarify that MILE's total computational cost includes offline dataset construction. For the main Qwen3.5-4B experiment, the total cost before evaluation is 1,504 H200 GPU-hours (Table 5), including 1,184 hours for dataset construction and 320 hours for supervised warm-start training and both RL stages. Since the constructed dataset can be reused across students, the offline cost is paid only once, with an additional 320 GPU-hours per student under the same training configuration.

Q5: Several presentation issues should be fixed. Line 107 has a syntax error. Appendix A says milestones are highlighted in red while Table 3 uses bold. Table 2 calls MILE-Core "w/o answer reward" while Table 9 calls it "w/o answer gating", which may be different settings. λo\lambda_o (line 226) and γ\gamma (line 196) do not appear in Table 6. The 16 evaluation seeds should also be distinguished from independently repeated training runs.

A5: We thank the reviewer for pointing out these issues. We have corrected these issues in the revised paper. Specifically, line 107 now states that "each τ(k)\tau^{(k)} is generated by one of several teacher reasoning models." We have also corrected the description in Appendix A to match the bold formatting in Table 3 (line 740), renamed the ablation in Table 9 (line 1021; see also lines 1000–1005) to "w/o answer reward (≡\equiv MILE-Core)" for consistency with Table 2, and added λo=1.0\lambda_o = 1.0 and γ=0.25\gamma = 0.25 to Table 6 (lines 932–936).

For clarity, "w/o answer reward" (MILE-Core) removes both the additive final-answer reward and the final-answer-dependent gating. Its trajectory reward contains only the progress and direction terms:

RCore(τ)=∑t(λprtprog+λdrtdir).R_{\mathrm{Core}}(\tau)=\sum_t\left(\lambda_p r_t^{\mathrm{prog}}+\lambda_d r_t^{\mathrm{dir}}\right).

Equivalently, λo=0\lambda_o=0 and the gating factor is fixed to g(τ)=1g(\tau)=1 for every trajectory, regardless of final-answer correctness.

Regarding the evaluation seeds, each configuration was trained only once and evaluated using 16 different seeds, with one sample per problem per seed (Table 7). Therefore, the reported standard deviations reflect variation across evaluation seeds, not independent training runs. We have clarified this distinction in Appendix C (lines 944–947).

References

[1] Amrith Setlur et al. Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning. ICLR 2025. Paper

[2] Yuhang Zhou, Yixin Cao, and Guangnan Ye. From Correctness to Utility: Gain-Based Prefix Evaluation for LLM Reasoning. Findings of EMNLP 2026, pp. 18212–18231. Paper

AuthorsOpenReview ↗

Appreciation for Your Feedback

Thank you very much for your positive feedback and for acknowledging that your concerns have been fully addressed. We truly appreciate your time and careful consideration.

Since the main concerns are now resolved through our rebuttal and clarifications, we would sincerely appreciate it if you could kindly consider raising your score accordingly. This would mean a great deal to us.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.