- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes MORE, a self-evolving training framework that jointly optimizes reasoning mode selection and computational effort. The method adopts a two-stage training paradigm: Stage 1 initializes four reasoning modes through SFT, while Stage 2 introduces MATE-PPO to adaptively allocate reasoning modes and token budgets based on online difficulty estimation. Experiments across coding, mathematical, and general-reasoning benchmarks demonstrate improvements in both task performance and inference efficiency.
Critical strengths
- The motivation is clear and well presented. Jointly optimizing reasoning modes and computational budgets is intuitive and practically meaningful.
- The experimental results are promising. MORE achieves consistent improvements over AZR across different model scales and backbones, while substantially reducing response length and inference time.
- The experimental evaluation is relatively comprehensive, covering multiple benchmarks, model scales, and training dynamics.
Critical weaknesses
- The overall method is somewhat complicated, involving multiple training stages, reward components, and hyperparameters, making it less elegant than its core intuition.
- The ablation studies are insufficient. Although the paper evaluates the two training stages, it lacks fine-grained ablations of the key MATE-PPO components, such as the budget-deviation reward, mode-fit reward, and mode–budget joint reward. This makes it difficult to identify which components contribute most to the reported improvements.
Questions for authors
- What are the individual contributions of the reward components in MATE-PPO? In particular, both the budget-deviation reward and the mode–budget joint reward penalize deviations from the target budget. Could the authors clarify their distinct roles and provide supporting evidence?
- How sensitive is MORE to the manually designed mode-fit utility? Since the utility function encourages specific reasoning modes for different difficulty levels, could these predefined preferences limit the model's ability to discover better mode allocations?
- The paper demonstrates improvements in both accuracy and token efficiency. However, how does MORE behave under different accuracy–efficiency trade-offs? For example, could the authors provide an analysis of how varying the reward coefficients affects accuracy and token consumption?
5 comments
2. Answers to questions
Q1 — Individual contributions of the reward components, and the distinct roles of R_BD and R_MB.
The individual contributions are reported in Table R1 and analyzed in our response to W2. Here we focus on how R_BD and R_MB differ.
Conclusion. R_BD is the primary effort regulator over all rollouts, and R_MB is a secondary, mode-specific calibration over successful rollouts. Both read the same mode-conditioned budget bands; they differ in which rollouts they cover and in how strongly each mode–difficulty cell is weighted.
| R_BD | R_MB | |
|---|---|---|
| Applies to | every rollout, successful or failed | successful rollouts only |
| Budget band | mode-conditioned B±(d, m) | the same bands |
| Weight | per difficulty tier (λ_d) | per mode × difficulty cell (η(d, m)) |
| Shape | full over-length penalty; under-length penalty down-weighted to 0.1 for failed responses | symmetric penalty on (o + u) |
| Clip range | [−0.5, 0] | [−0.25, 0] |
R_BD is the only term that reaches failed rollouts. It keeps full over-length pressure on them but only one tenth of the under-length weight, which removes the incentive to lengthen unsuccessful reasoning just to reach the band. R_MB adds a mode-specific weight among successful rollouts: on hard tasks, for example, η is 0.08 for Direct but 0.03 for Long CoT, so a successful Direct answer that leaves its band costs more than a Long CoT one. R_MB is bounded by design (|γ·R_MB| ≤ 0.0125), and Fig. 13 shows that it stays a small correction during training.
The ablations in Table R1 support this division of labor (percentages average the four within-group relative changes):
- R_BD is the primary regulator. Adding it to R_MF + R_MB cuts response length and evaluation time by 33.5% each and raises accuracy in all four groups (+0.65 AVG).
- R_MB is not redundant with R_BD. Adding it to R_BD + R_MF cuts response length and evaluation time by a further 13.9% and 15.4%, with lower cost in every group and +0.3 AVG.
- R_MB acts on the mode–budget coupling. Added to R_BD alone, it raises AVG by 4.45 points but increases response length by 11.1%. Added to R_BD + R_MF, it reduces response length by 13.9%. Its efficiency gain therefore appears when mode choice is learned at the same time, which is the role it was designed for.
Q2 — Sensitivity to the hand-designed mode-fit utility.
Conclusion. Across the tested range, from removing U entirely to doubling it, accuracy stays within 0.7 points, so the predefined preferences do not cap accuracy. The learned allocation is also not a mechanical copy of U's ranking: it departs from that ranking wherever verifier evidence favors another mode. What U mainly sets is cost.
Evidence (new runs at β = 0, 0.5, and 2.0). Since R_MF = U(d, m) for correct responses, β scales every entry of U by the same factor: β = 0 removes the prior (the R_BD + R_MB run in Table R1), β = 1.0 is the default, and β = 2.0 doubles it. All other settings are fixed.
Table R2a. Accuracy (%) and mean response length (tokens) under different prior strengths (Qwen2.5-Coder-7B). RLA Avg averages the four benchmark groups.
| β | Code | Math | GSM8K | M-Pro | AVG | RLA Avg |
|---|---|---|---|---|---|---|
| 0 (no prior) † | 63.2 | 42.2 | 77.3 | 45.5 | 57.05 | 1567.2 |
| 0.5 † | 62.4 | 42.5 | 77.5 | 45.6 | 57.00 | 1127.4 |
| 1.0 (default) | 62.8 | 42.8 | 77.8 | 46.0 | 57.35 | 904.0 |
| 2.0 † | 61.9 | 42.1 | 76.9 | 45.8 | 56.68 | 846.5 |
Table R2b. Test-time mode allocation under different prior strengths. Each mode cell gives the shares of Direct / Short CoT / Long CoT / Code. Code Avg and Math Avg average the benchmarks in each group; AIME averages AIME 2024 and 2025. Acc. is accuracy (%).
| β | GSM8K | MMLU-Pro | Math Avg | Code Avg | LCB | LCB Acc. | AIME | AIME Acc. |
|---|---|---|---|---|---|---|---|---|
| 0 † | 0.15 / 0.20 / 0.33 / 0.32 | 0.15 / 0.26 / 0.33 / 0.26 | 0.14 / 0.26 / 0.44 / 0.16 | 0.08 / 0.21 / 0.44 / 0.27 | 0.12 / 0.23 / 0.38 / 0.27 | 37.7 | 0.07 / 0.22 / 0.52 / 0.19 | 17.4 |
| 0.5 † | 0.25 / 0.24 / 0.22 / 0.29 | 0.20 / 0.29 / 0.27 / 0.24 | 0.20 / 0.29 / 0.36 / 0.15 | 0.08 / 0.25 / 0.39 / 0.28 | 0.14 / 0.22 / 0.41 / 0.23 | 37.1 | 0.09 / 0.24 / 0.51 / 0.16 | 18.1 |
| 1.0 (default) | 0.28 / 0.25 / 0.18 / 0.29 | 0.22 / 0.30 / 0.24 / 0.24 | 0.23 / 0.30 / 0.32 / 0.15 | 0.12 / 0.25 / 0.36 / 0.27 | 0.15 / 0.21 / 0.35 / 0.29 | 38.0 | 0.10 / 0.23 / 0.48 / 0.19 | 17.7 |
| 2.0 † | 0.31 / 0.25 / 0.13 / 0.31 | 0.31 / 0.33 / 0.19 / 0.17 | 0.29 / 0.33 / 0.27 / 0.11 | 0.11 / 0.23 / 0.33 / 0.33 | 0.15 / 0.24 / 0.36 / 0.25 | 36.6 | 0.13 / 0.28 / 0.46 / 0.13 | 17.3 |
Analysis.
- Per-trajectory guarantee. U enters only for correct responses and lies in [−0.3, 0.3]. A correct solver response receives at least 1 − 0.3β − 0.3 − 0.0125, which is positive for every β < 2.29 (more than twice the default), and an incorrect one at most 0. A favored mode that fails therefore never outranks a disfavored mode that succeeds. U only reorders credit among correct responses; at the default β this reordering costs no accuracy relative to removing U (Table R2a).
- Accuracy. AVG stays between 56.68 and 57.35 whether the prior is absent, halved, default, or doubled, and no setting beats the default. Without U, every group stays within 0.6 points, while the default needs 42.8% fewer tokens and 43.9% less evaluation time (averages of the four within-group reductions). Cost falls monotonically as β increases.
- Where the prior acts. From β = 0.5 to 2.0, GSM8K, MMLU-Pro, and the math suites shift from Long CoT (−0.09, −0.08, −0.09) toward Direct (+0.06, +0.11, +0.09), as U intends. On LCB, the Long CoT + Code share moves only from 0.64 to 0.61; on AIME, Long CoT stays dominant (0.51 to 0.46). Accuracy on both shows no trend with β (LCB 36.6–38.0, AIME 17.3–18.1).
- Verifier-favored modes hold. GSM8K items fall mostly in the easy and medium tiers, where U ranks Short CoT above Code. Yet forced Code is 3.5 points more accurate there than forced Short CoT at nearly the same cost (91.6 vs. 87.7 tokens; submission, Table 12), and Code is MORE's most frequent GSM8K mode, holding about 0.30 at every β. In the coding suites, doubling the prior leaves Direct unchanged (0.12 to 0.11) and raises Code from 0.27 to 0.33.
- Training shows the same. After step 50, at least 95% of rollouts are easy or medium, yet Long CoT and Code, which U ranks lowest on easy tasks, take 41–54% of rollouts in every stream (submission, Fig. 7 and Fig. 16). U is also role-agnostic, but the output family's proposer and solver learn different preferred modes.
We thank the reviewer for finding the motivation clear and the evaluation comprehensive, and for the precise questions on the reward design. To answer them, we ran new controlled experiments: ablations over all combinations of the three shaping terms, and a sweep of the mode-fit coefficient. All new runs use Qwen2.5-Coder-7B and share MORE's Stage-1 checkpoint, task-generation settings, Stage-2 update budget, decoding settings, and checkpoint-selection rule (App. G.2). The reward-component ablations and their analysis are now also in Sec. 5.3 (Table 4) and App. I.7 (RQ7). In the tables, rows marked † are new runs; unmarked rows come from the original submission.
1. Response to the Critical Weaknesses
W1 — Overall method complexity.
Conclusion. Each component of MORE answers one decision the policy has to make, and the new ablations show that all three reward terms are needed.
MORE adds two decisions to a standard self-evolving loop: which reasoning form to use, and how much computation to spend in it. Stage 1 makes the four forms identifiable, and Stage 2 learns to choose among them. R_MF serves mode choice, R_BD serves effort, and R_MB calibrates effort per mode. All three are computed from quantities every rollout already produces (mode token, response length, verifier outcome), so they add no learned model and no extra rollouts.
Among all tested combinations, removing any one term lowers the four-group AVG by 0.3–0.65 points and raises cost (Table R1, in our response to W2). The default is not a fragile setting either: from a mode-fit weight of 0 to twice the default, AVG stays within 0.7 points (Table R2a, in our answer to Q2).
W2 — Fine-grained ablations of the MATE-PPO components.
Conclusion. We have added the requested ablations. Among all tested combinations, only the full return reaches both the highest four-group accuracy and the lowest cost. R_MF is the strongest single term for accuracy; R_MF and R_BD give most of the efficiency gain, and R_MB adds a further reduction.
Setup. Every Stage-2 run in this response keeps the role-specific execution feedback unchanged (R_acc for solvers, R_prop for proposers); we vary only the shaping terms, which the tables list. Each term has its own additive coefficient (Eq. 2). An ablation zeroes the omitted coefficients and leaves the active ones, mode-token generation, and controller updates unchanged. Table R1 covers all seven non-empty combinations.
Table R1. Reward-component ablations on Qwen2.5-Coder-7B (Sec. 5.3, Table 4; App. I.7). Accuracy in %. RLA (mean response length, tokens) and WT (evaluation time, min:s) are averaged over the four benchmark groups. Best values in bold.
Analysis.