- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MILE, a framework that extracts milestones by estimating the future solvability of intermediate states in expert reasoning trajectories. It introduces progress and directional consistency rewards and employs a two-stage GRPO optimization strategy to improve mathematical reasoning performance. Experimental results demonstrate improvements across multiple mathematical reasoning benchmarks. However, the core design is closely related to existing value estimation and reward shaping methods, and there remains a gap between the theoretical guarantees and the practical algorithm. Overall, the paper presents relatively comprehensive experiments, but its methodological novelty and theoretical rigor require further improvement.
Critical strengths
- MILE integrates expert milestone extraction, progress reward estimation, directional consistency rewards, and two-stage GRPO optimization into a unified framework. Each component serves a distinct role in progress modeling, reward construction, and policy optimization. The overall design is clear and has practical feasibility.
- MILE achieves the highest average Pass@1 across multiple mathematical reasoning benchmarks, demonstrating the strong performance of the proposed method.
Critical weaknesses
- The idea of MILE is to evaluate intermediate reasoning states through Future Solvability and construct the Progress Reward based on value changes between consecutive states. However, this mechanism is highly similar to existing Value-Based Process Supervision and Potential-Based Reward Shaping methods. Moreover, the Direction Reward essentially imposes constraints based on the representation directions of expert trajectories, while the two-stage GRPO combines existing optimization strategies. Therefore, the paper's novelty primarily lies in the integration of existing techniques rather than the introduction of fundamentally new mechanisms.
- Several theoretical results in the paper rely on strong assumptions and do not fully align with the actual algorithm. For example, Theorem E.5 requires the retrieval bandwidth to decrease as the sample size increases, whereas the practical implementation uses a fixed retrieval temperature. The conditional zero-mean noise assumption in Assumption E.3 also be affected by selection bias in milestone extraction. Furthermore, Theorem E.8 only establishes the consistency of expert direction estimation without demonstrating that it leads to genuine reasoning progress. Theorem E.13 provides a no-cycling guarantee for the Safe Shaping reward, which is not used in the actual implementation. Therefore, the current theoretical analysis is insufficient to fully justify the effectiveness and stability of MILE's core reward mechanisms.
Questions for authors
- Theorem E.5 requires the kernel bandwidth to approach zero as the sample size increases, whereas the actual algorithm uses a fixed retrieval temperature of τr = 0.15. Since a fixed bandwidth may introduce irreducible smoothing bias, what is the corresponding finite-sample error bound?
- When the numerator of the Progress Reward is negative, increasing the number of tokens reduces the magnitude of the negative reward. Could the model exploit this mechanism by generating longer incorrect reasoning steps to reduce penalties, while using shorter steps with positive rewards to achieve a net positive reward?
- The paper uses False Progress to measure the discrepancy between high progress scores and actual improvements in solvability. However, this metric may be sensitive to the choice of threshold and the method used to estimate reference solvability. Could the author provides results under different thresholds and analyze the correlation between this metric and the final Pass@1 performance?
4 comments
Q3: Theorem E.5 requires the kernel bandwidth to approach zero as the sample size grows, whereas the algorithm uses a fixed retrieval temperature . Since a fixed bandwidth may introduce irreducible smoothing bias, what is the corresponding finite-sample error bound?
A3: We agree that a fixed retrieval temperature can leave a non-vanishing smoothing bias. The implementation approximates the idealized setting of Theorem E.5, which requires a shrinking bandwidth.
For a fixed query , let and assume the target solvability is -Lipschitz. Under the idealized assumption that labels in are conditionally independent and unbiased given the selected milestone locations, Hoeffding's inequality gives, with probability at least ,
Here, is the effective number of retrieved milestones, and is the weighted average retrieval distance. This bound retains the smoothing bias at fixed temperature. Selection based on noisy gains may violate the conditional assumptions, so the bound is not an unconditional guarantee for the implementation.
In practice, at , the retrieval potential has a correlation of 0.76 with independently estimated student continuation success (see our response to Reviewer mNNm). Together with the selection ablations above, this supports the effectiveness of the practical approximation.
Q4: When the numerator of the progress reward is negative, increasing the number of tokens reduces the magnitude of the negative reward. Could the model exploit this by generating longer incorrect reasoning steps to dilute penalties while keeping positively rewarded steps short, and so obtain a net positive reward without net progress?
A4: We acknowledge that length normalization can introduce this failure mode. As analyzed in Appendix E.7, a short forward step followed by a longer backward step may receive a positive cumulative reward even when the potential returns to its initial value. However, we found its impact on training to be limited, for three reasons. First, the progress reward is grounded in teacher milestones, independent of the student's own predictions. For a sufficiently unproductive step, the decrease in potential can outweigh the effect of length normalization, so a longer step may still receive a larger penalty (lines 195–197). Second, each trajectory has a fixed token budget, meaning that generating unnecessarily long steps leaves fewer tokens to complete the solution. Third, in Stage 2, the dense rewards of trajectories with incorrect final answers are scaled by 0.05 (Table 6), substantially reducing the benefit of accumulating rewards without solving the problem. Together, these mechanisms help limit the incentive for reward exploitation.
Q5: False Progress measures the discrepancy between high progress scores and actual improvements in solvability, but it may be sensitive to the choice of threshold and to the method used to estimate reference solvability. Could the authors provide results under different thresholds and analyze the correlation between this metric and final Pass@1 performance?
A5: To evaluate the sensitivity of False Progress to the threshold, we recomputed the metric using the top 5%, 10%, and 20% of prefix scores, while keeping the reference checkpoint and the 1,024-token continuation budget fixed (Appendix B.3).
| Top-score fraction | False Progress (%) | Pass@1 |
|---|---|---|
| 5% | 6.2 | 86.56 |
| 10% | 6.9 | 86.56 |
| 20% | 7.7 | 86.56 |
False Progress varies by only 1.5 percentage points as the threshold changes fourfold, suggesting that the metric is relatively stable across different cutoff choices. These results indicate that our conclusions are not sensitive to the specific threshold used. Across the ten configurations in Table 2, the reported means yield a Spearman correlation of -0.782 between False Progress and Overall Pass@1. This suggests that lower False Progress is associated with higher final accuracy.
Appreciation for Your Feedback
Thank you very much for your positive feedback and for acknowledging that your concerns have been fully addressed. We truly appreciate your time and careful consideration.
Since the main concerns are now resolved through our rebuttal and clarifications, we would sincerely appreciate it if you could kindly consider raising your score accordingly. This would mean a great deal to us.
Thank you for your detailed responses. My concerns have been largely addressed, and I have raised my rating to 3.
Rebuttal
We thank the reviewer for the detailed review. We respond to each point below. All changes to the paper are marked in red in the revised PDF.
Q1: Evaluating intermediate states by future solvability and building a progress reward from value changes between consecutive states is highly similar to existing value-based process supervision and potential-based reward shaping. The direction reward essentially imposes constraints from the representation directions of expert trajectories, and the two-stage GRPO combines existing optimization strategies. The novelty therefore appears to lie in integrating existing techniques, not in introducing a fundamentally new mechanism.
A1: MILE's main contribution lies in how it constructs progress supervision from multiple expert trajectories and uses it to guide student learning. This is achieved through three key components.
First, MILE selects sparse milestones from verified expert solutions at points where future solvability improves. This avoids the need to supervise every reasoning step and creates a reusable milestone pool containing states from multiple solution paths.
Second, MILE retrieves nearby milestones to estimate both the student's progress and the relevant expert direction. The progress reward measures changes in estimated solvability, while the direction reward encourages alignment with nearby expert transitions. This provides an additional form of supervision beyond value-based methods, guiding the student on how to make progress, not just whether progress has been made.
Third, MILE uses a two-stage training pipeline to apply these rewards at different levels. Stage 1 optimizes individual reasoning transitions from expert prefixes, while Stage 2 trains on complete solutions with final-answer supervision. The ablation results in Table 2 demonstrate the importance of these components. MILE-Full achieves 86.56% Overall Pass@1, compared with 83.68% using uniform milestone selection, 84.34% without Stage 1, and 82.86% without Stage 2. These results support the effectiveness of combining sparse milestone selection, retrieval-based rewards, and two-stage training.
Q2: Several theoretical results rest on strong assumptions and do not align with the actual algorithm. Theorem E.5 requires the retrieval bandwidth to shrink as the sample size grows, whereas the implementation uses a fixed retrieval temperature. The conditional zero-mean noise assumption of E.3 may be violated by selection bias in milestone extraction. Theorem E.8 establishes only the consistency of expert-direction estimation, without showing that it leads to genuine reasoning progress. Theorem E.13's no-cycling guarantee concerns a Safe Shaping reward that is not used in the implementation. The analysis therefore does not yet justify the effectiveness and stability of MILE's core reward mechanisms.
A2: We thank the reviewer for raising these questions. We clarify below how the theoretical results relate to our practical implementation.
E.5. Theorem E.5 establishes the convergence of the retrieval potential under sufficient coverage and shrinking-bandwidth conditions. In practice, we use a fixed retrieval temperature of 0.15. We provide a finite-sample analysis in our response to Q3 below.
E.3. Our theory assumes zero-mean annotation noise, while practical milestone selection uses finite-sample solvability estimates and may introduce selection bias. The implementation therefore approximates the idealized setting. Solvability-based selection achieves 86.56% Overall Pass@1, compared with 83.68% for uniform selection and 83.42% for dense milestones, suggesting that this approximation remains effective in practice.
E.8. Expert directions connect consecutive milestones selected based on improvements in future solvability, capturing how an expert moves toward a more solvable state. Theorem E.8 characterizes the estimation accuracy of these directions. Progress is measured separately through changes in estimated solvability and receives a larger reward weight (1.0 vs. 0.25 for direction, Table 6). Thus, the direction reward complements the progress signal by providing additional guidance on how to advance along expert trajectories.
E.13. Theorem E.13 analyzes a safe-shaping variant that is not included in our current implementation. We leave its practical integration to future work and have clarified in the discussion following Theorem E.13 that this reward is not used in any of our experiments (lines 1540–1542).