acceptodds
Desk rejected at ICLR 2027

MORE: Mode-Optimized Reasoning with Self-Adaptive Effort for Self-Evolving Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-evolving training improves the performance of agents on difficult tasks through verifiable feedback, but it often applies costly deliberation uniformly across tasks. To address this problem, we propose Mode-Optimized Reasoning with Self-Adaptive Effort (MORE), a self-evolving training framework that adaptively selects both reasoning form and computational effort for each task. Specifically, MORE follows a two-stage training paradigm: (i) Stage 1 establishes a compact prior over direct answering, short and long chain-of-thought, and code-based reasoning, and (ii) Stage 2 self-evolves this prior with Mode-Adaptive and Test-time Effort PPO (MATE-PPO) on generated, verifiable task streams without curated data. By learning mode selection and token expenditure jointly, MORE allocates concise reasoning to tasks that permit it while retaining extended deliberation where it is useful. Across coding, mathematics, and general-reasoning benchmarks, MORE achieves consistent performance gains on Base and Coder backbones at model scales from 3B to 14B. During inference, MORE demonstrates higher accuracy and better efficiency, with markedly shorter response lengths and reduced inference time.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-evolving agent, Large Language Model application, LLM reinforcement learning, Efficiency of training and inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yaZPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes MORE, a self-evolving training framework that jointly optimizes reasoning mode selection and computational effort. The method adopts a two-stage training paradigm: Stage 1 initializes four reasoning modes through SFT, while Stage 2 introduces MATE-PPO to adaptively allocate reasoning modes and token budgets based on online difficulty estimation. Experiments across coding, mathematical, and general-reasoning benchmarks demonstrate improvements in both task performance and inference efficiency.

Critical strengths
  1. The motivation is clear and well presented. Jointly optimizing reasoning modes and computational budgets is intuitive and practically meaningful.
  2. The experimental results are promising. MORE achieves consistent improvements over AZR across different model scales and backbones, while substantially reducing response length and inference time.
  3. The experimental evaluation is relatively comprehensive, covering multiple benchmarks, model scales, and training dynamics.
Critical weaknesses
  1. The overall method is somewhat complicated, involving multiple training stages, reward components, and hyperparameters, making it less elegant than its core intuition.
  2. The ablation studies are insufficient. Although the paper evaluates the two training stages, it lacks fine-grained ablations of the key MATE-PPO components, such as the budget-deviation reward, mode-fit reward, and mode–budget joint reward. This makes it difficult to identify which components contribute most to the reported improvements.
Questions for authors
  1. What are the individual contributions of the reward components in MATE-PPO? In particular, both the budget-deviation reward and the mode–budget joint reward penalize deviations from the target budget. Could the authors clarify their distinct roles and provide supporting evidence?
  2. How sensitive is MORE to the manually designed mode-fit utility? Since the utility function encourages specific reasoning modes for different difficulty levels, could these predefined preferences limit the model's ability to discover better mode allocations?
  3. The paper demonstrates improvements in both accuracy and token efficiency. However, how does MORE behave under different accuracy–efficiency trade-offs? For example, could the authors provide an analysis of how varying the reward coefficients affects accuracy and token consumption?
5 comments

We thank the reviewer for finding the motivation clear and the evaluation comprehensive, and for the precise questions on the reward design. To answer them, we ran new controlled experiments: ablations over all combinations of the three shaping terms, and a sweep of the mode-fit coefficient. All new runs use Qwen2.5-Coder-7B and share MORE's Stage-1 checkpoint, task-generation settings, Stage-2 update budget, decoding settings, and checkpoint-selection rule (App. G.2). The reward-component ablations and their analysis are now also in Sec. 5.3 (Table 4) and App. I.7 (RQ7). In the tables, rows marked † are new runs; unmarked rows come from the original submission.

1. Response to the Critical Weaknesses

W1 — Overall method complexity.

Conclusion. Each component of MORE answers one decision the policy has to make, and the new ablations show that all three reward terms are needed.

MORE adds two decisions to a standard self-evolving loop: which reasoning form to use, and how much computation to spend in it. Stage 1 makes the four forms identifiable, and Stage 2 learns to choose among them. R_MF serves mode choice, R_BD serves effort, and R_MB calibrates effort per mode. All three are computed from quantities every rollout already produces (mode token, response length, verifier outcome), so they add no learned model and no extra rollouts.

Among all tested combinations, removing any one term lowers the four-group AVG by 0.3–0.65 points and raises cost (Table R1, in our response to W2). The default is not a fragile setting either: from a mode-fit weight of 0 to twice the default, AVG stays within 0.7 points (Table R2a, in our answer to Q2).

W2 — Fine-grained ablations of the MATE-PPO components.

Conclusion. We have added the requested ablations. Among all tested combinations, only the full return reaches both the highest four-group accuracy and the lowest cost. R_MF is the strongest single term for accuracy; R_MF and R_BD give most of the efficiency gain, and R_MB adds a further reduction.

Setup. Every Stage-2 run in this response keeps the role-specific execution feedback unchanged (R_acc for solvers, R_prop for proposers); we vary only the shaping terms, which the tables list. Each term has its own additive coefficient (Eq. 2). An ablation zeroes the omitted coefficients and leaves the active ones, mode-token generation, and controller updates unchanged. Table R1 covers all seven non-empty combinations.

Table R1. Reward-component ablations on Qwen2.5-Coder-7B (Sec. 5.3, Table 4; App. I.7). Accuracy in %. RLA (mean response length, tokens) and WT (evaluation time, min:s) are averaged over the four benchmark groups. Best values in bold.

Shaping termsCodeMathGSM8KM-ProAVGRLAWT
Stage 1 only (before Stage 2)52.219.970.039.145.301775.772:10
R_BD †58.434.275.342.552.601288.352:39
R_MF †60.940.476.044.855.531675.670:52
R_MB †53.625.372.841.148.201445.560:02
R_BD + R_MF †62.243.177.045.857.03956.540:20
R_MF + R_MB †61.742.676.945.656.701349.553:07
R_BD + R_MB †63.242.277.345.557.051567.263:29
Full MORE (all three)62.842.877.846.057.35904.034:41

Analysis.

  • Full return. Highest AVG, and the lowest response length and evaluation time in every group. Single groups can favor a subset (Code: R_BD + R_MB; Math: R_BD + R_MF), but no subset matches the full return on average accuracy and cost together.
  • Accuracy. R_MF is the strongest single term (55.53 AVG, vs. 52.60 for R_BD and 48.20 for R_MB). Removing R_BD from the full return costs the most (−0.65 AVG, lower in all four groups); removing R_MF or R_MB costs about 0.3 each.
  • Efficiency. Adding R_MF, R_BD, or R_MB to the other two terms cuts response length by 42.8%, 33.5%, and 13.9%, and evaluation time by 43.9%, 33.5%, and 15.4% (averages of within-group reductions, App. I.7).
  • Simpler rewards. No two-term variant matches the full return on both axes: the most accurate (R_BD + R_MB, 57.05) uses 1.7 times the tokens, and the most efficient (R_BD + R_MF, 57.03) costs more in every group. Every single-term variant stays below 56 AVG.

2. Answers to questions

Q1 — Individual contributions of the reward components, and the distinct roles of R_BD and R_MB.

The individual contributions are reported in Table R1 and analyzed in our response to W2. Here we focus on how R_BD and R_MB differ.

Conclusion. R_BD is the primary effort regulator over all rollouts, and R_MB is a secondary, mode-specific calibration over successful rollouts. Both read the same mode-conditioned budget bands; they differ in which rollouts they cover and in how strongly each mode–difficulty cell is weighted.

R_BDR_MB
Applies toevery rollout, successful or failedsuccessful rollouts only
Budget bandmode-conditioned B±(d, m)the same bands
Weightper difficulty tier (λ_d)per mode × difficulty cell (η(d, m))
Shapefull over-length penalty; under-length penalty down-weighted to 0.1 for failed responsessymmetric penalty on (o + u)
Clip range[−0.5, 0][−0.25, 0]

R_BD is the only term that reaches failed rollouts. It keeps full over-length pressure on them but only one tenth of the under-length weight, which removes the incentive to lengthen unsuccessful reasoning just to reach the band. R_MB adds a mode-specific weight among successful rollouts: on hard tasks, for example, η is 0.08 for Direct but 0.03 for Long CoT, so a successful Direct answer that leaves its band costs more than a Long CoT one. R_MB is bounded by design (|γ·R_MB| ≤ 0.0125), and Fig. 13 shows that it stays a small correction during training.

The ablations in Table R1 support this division of labor (percentages average the four within-group relative changes):

  • R_BD is the primary regulator. Adding it to R_MF + R_MB cuts response length and evaluation time by 33.5% each and raises accuracy in all four groups (+0.65 AVG).
  • R_MB is not redundant with R_BD. Adding it to R_BD + R_MF cuts response length and evaluation time by a further 13.9% and 15.4%, with lower cost in every group and +0.3 AVG.
  • R_MB acts on the mode–budget coupling. Added to R_BD alone, it raises AVG by 4.45 points but increases response length by 11.1%. Added to R_BD + R_MF, it reduces response length by 13.9%. Its efficiency gain therefore appears when mode choice is learned at the same time, which is the role it was designed for.

Q2 — Sensitivity to the hand-designed mode-fit utility.

Conclusion. Across the tested range, from removing U entirely to doubling it, accuracy stays within 0.7 points, so the predefined preferences do not cap accuracy. The learned allocation is also not a mechanical copy of U's ranking: it departs from that ranking wherever verifier evidence favors another mode. What U mainly sets is cost.

Evidence (new runs at β = 0, 0.5, and 2.0). Since R_MF = U(d, m) for correct responses, β scales every entry of U by the same factor: β = 0 removes the prior (the R_BD + R_MB run in Table R1), β = 1.0 is the default, and β = 2.0 doubles it. All other settings are fixed.

Table R2a. Accuracy (%) and mean response length (tokens) under different prior strengths (Qwen2.5-Coder-7B). RLA Avg averages the four benchmark groups.

βCodeMathGSM8KM-ProAVGRLA Avg
0 (no prior) †63.242.277.345.557.051567.2
0.5 †62.442.577.545.657.001127.4
1.0 (default)62.842.877.846.057.35904.0
2.0 †61.942.176.945.856.68846.5

Table R2b. Test-time mode allocation under different prior strengths. Each mode cell gives the shares of Direct / Short CoT / Long CoT / Code. Code Avg and Math Avg average the benchmarks in each group; AIME averages AIME 2024 and 2025. Acc. is accuracy (%).

βGSM8KMMLU-ProMath AvgCode AvgLCBLCB Acc.AIMEAIME Acc.
0 †0.15 / 0.20 / 0.33 / 0.320.15 / 0.26 / 0.33 / 0.260.14 / 0.26 / 0.44 / 0.160.08 / 0.21 / 0.44 / 0.270.12 / 0.23 / 0.38 / 0.2737.70.07 / 0.22 / 0.52 / 0.1917.4
0.5 †0.25 / 0.24 / 0.22 / 0.290.20 / 0.29 / 0.27 / 0.240.20 / 0.29 / 0.36 / 0.150.08 / 0.25 / 0.39 / 0.280.14 / 0.22 / 0.41 / 0.2337.10.09 / 0.24 / 0.51 / 0.1618.1
1.0 (default)0.28 / 0.25 / 0.18 / 0.290.22 / 0.30 / 0.24 / 0.240.23 / 0.30 / 0.32 / 0.150.12 / 0.25 / 0.36 / 0.270.15 / 0.21 / 0.35 / 0.2938.00.10 / 0.23 / 0.48 / 0.1917.7
2.0 †0.31 / 0.25 / 0.13 / 0.310.31 / 0.33 / 0.19 / 0.170.29 / 0.33 / 0.27 / 0.110.11 / 0.23 / 0.33 / 0.330.15 / 0.24 / 0.36 / 0.2536.60.13 / 0.28 / 0.46 / 0.1317.3

Analysis.

  • Per-trajectory guarantee. U enters only for correct responses and lies in [−0.3, 0.3]. A correct solver response receives at least 1 − 0.3β − 0.3 − 0.0125, which is positive for every β < 2.29 (more than twice the default), and an incorrect one at most 0. A favored mode that fails therefore never outranks a disfavored mode that succeeds. U only reorders credit among correct responses; at the default β this reordering costs no accuracy relative to removing U (Table R2a).
  • Accuracy. AVG stays between 56.68 and 57.35 whether the prior is absent, halved, default, or doubled, and no setting beats the default. Without U, every group stays within 0.6 points, while the default needs 42.8% fewer tokens and 43.9% less evaluation time (averages of the four within-group reductions). Cost falls monotonically as β increases.
  • Where the prior acts. From β = 0.5 to 2.0, GSM8K, MMLU-Pro, and the math suites shift from Long CoT (−0.09, −0.08, −0.09) toward Direct (+0.06, +0.11, +0.09), as U intends. On LCB, the Long CoT + Code share moves only from 0.64 to 0.61; on AIME, Long CoT stays dominant (0.51 to 0.46). Accuracy on both shows no trend with β (LCB 36.6–38.0, AIME 17.3–18.1).
  • Verifier-favored modes hold. GSM8K items fall mostly in the easy and medium tiers, where U ranks Short CoT above Code. Yet forced Code is 3.5 points more accurate there than forced Short CoT at nearly the same cost (91.6 vs. 87.7 tokens; submission, Table 12), and Code is MORE's most frequent GSM8K mode, holding about 0.30 at every β. In the coding suites, doubling the prior leaves Direct unchanged (0.12 to 0.11) and raises Code from 0.27 to 0.33.
  • Training shows the same. After step 50, at least 95% of rollouts are easy or medium, yet Long CoT and Code, which U ranks lowest on easy tasks, take 41–54% of rollouts in every stream (submission, Fig. 7 and Fig. 16). U is also role-agnostic, but the output family's proposer and solver learn different preferred modes.
Reviewer_5rWEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper adds four-level reasoning types (direct, short CoT, long CoT, and code) in inference time, then at inference time, the model can pick different reasoning types via mode controller, in this way it can reduce the total number of reasoning tokens and make it efficient in training. The experiments are done in range of 3B to 14B models. Results show MORE framework improves the average performance by 1.3 to 2.2 points over AZR approach, and also runs more efficiently than AZR approach.

Critical strengths
  1. This paper introduces a four modes for reasoning types for LLMs inference. Experiments show this approach can learn better and more efficient than AZR baseline.

  2. The complete pipeline design looks interesting (including mode controller, reward designs)

Critical weaknesses

In experiments, as we have another factor (task proposer) will affect the final performance, I’m not sure AZR also uses the same task proposer? And AZR and MORE are learning the same proposed tasks?

Questions for authors
  1. In Figure 3, please show the complete learning curve goes up to 450 steps as in Figure 4.

  2. In experiments, as we have another factor (task proposer) will affect the final performance, I’m not sure AZR also uses the same task proposer? And AZR and MORE are learning the same proposed tasks?

  3. The baseline of AZR also uses PPO algorithm?

  4. As you are using PPO algorithms, please add more information about critics training, e.g. warm up steps.

3 comments

We thank the reviewer for the positive assessment of the pipeline design and the efficiency results.

1. Response to the Critical Weaknesses

W1 — Is the task proposer the same, and do AZR and MORE learn from the same proposed tasks? Both AZR and MORE follow the self-evolving paradigm, in which one shared policy acts as both proposer and solver. During self-evolution, every training task is proposed by the model itself, and no pre-existing task dataset is used. The task-generation protocol is therefore the same, but the tasks are not. Both methods use the same task families and the same proposer learnability reward (Eq. 7), and holding this protocol fixed is an experimental control. Each policy generates its own tasks on-policy.

The key difference is that the MORE proposer is itself mode- and budget-aware. Each proposal is produced under a chosen reasoning mode and budget band and receives the mode–budget return (Eq. 10). As a result, MORE's task stream co-evolves with the way it allocates reasoning effort. Under the same Stage-1 initialization, the AZR training procedure reaches 52.4 AVG, while MORE reaches 57.3 (Table 3). The official AZR training logs are publicly available at https://wandb.ai/andrewzhao112/AbsoluteZeroReasoner.

2. Answers to questions

Q1 — Fig. 3 learning curves up to 450 steps. We have extended Fig. 3(a) to the full 450 steps for all three Coder scales. The longer horizon is consistent with the trends reported in the paper and further supports our long-horizon observations (Sec. 5.1, Fig. 4). From step 150 onward, MORE stays above the matched AZR curve at every checkpoint and every scale. It also reaches a higher in-distribution peak at every scale (3B: 48.5 vs. 45.0; 7B: 64.5 vs. 61.6; 14B: 76.8 vs. 73.6). Both methods plateau after roughly 300 steps, and the gap persists through step 450 (+4.2, +1.7, and +3.5 points).

Q2 — Same task proposer and same proposed tasks? Please see W1 above.

Q3 — Does the AZR baseline also use PPO? Yes. AZR optimizes critic-free Task-Relative REINFORCE++ advantages with the PPO clipped surrogate. MATE-PPO also uses a clipped-surrogate update, but it optimizes a different trajectory. Each trajectory begins with a likelihood-bearing mode token, and its return is conditioned on the chosen mode and its online budget band through the difficulty controller. A single clipped update therefore trains mode choice, response generation, and effort calibration jointly. Advantages are normalized separately over valid response tokens within each of the six task–role streams before the batches are concatenated for the policy update. We have provided a more detailed description of this computation in Appendix C.3 and aligned Algorithm 1 with this order of operations.

Q4 — Critic training details (e.g., warm-up steps). No learned critic is used, so critic training and critic warm-up are not applicable. The actor is configured with a base learning rate of 10−610^{-6} and zero learning-rate warm-up steps. Appendix C.3 now provides a more detailed description of the critic-free update and the actor optimization settings.

Thanks for the response! Most of my questions are addressed.

The reason I'm asking the critic questions is that if we can train a perfect critic, it will tell us several important things we care in this paper: 1. given a prompt, the critic score at first token will tell us how hard is this question (e.g. higher value means easy task), it can be served as Mode Controller; 2. the trajectory level of critic is also meaningful; 3. the end of trajectory will tell us the answer is correct or not.

AuthorsOpenReview ↗

Further Discussion on Critic-Based Mode–Effort Control

Thank you for the follow-up and for explaining the motivation behind the critic question! We agree that a well-calibrated critic could provide useful signals for difficulty estimation, reasoning progress, and outcome prediction.

The distinction we would emphasize is between estimating value and defining how reasoning should be controlled. A prompt-level value summarizes expected return under the current policy, but does not itself specify which reasoning mode to use or how accuracy should be balanced against computational effort. Using it as a controller would still require a mode-aware decision mechanism and an appropriate training objective. In practice, its calibration would also need to track the co-evolution of the policy and its generated tasks.

MORE addresses this control problem explicitly: it treats the reasoning mode as a policy action and jointly optimizes mode selection and effort through difficulty-conditioned budget bands and verifier-grounded rewards. Its contribution is therefore not merely estimating difficulty, but learning how reasoning form and computational effort should adapt together. A mode-conditioned critic would be a natural complement to this framework, potentially providing predictive guidance before and during generation.

Thank you again for the constructive discussion. If our clarifications and additional experimental results have addressed your concerns, we would be grateful if you would reconsider your score.

Reviewer_f2kfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MORE, a self-evolving reasoning framework that jointly optimizes reasoning mode selection and computational effort. The method uses two-stage training: mode-format SFT to initialize four reasoning modes (Direct, Short CoT, Long CoT, and Code), followed by MATE-PPO with difficulty-aware rewards and adaptive token budgets. Experiments across coding, mathematics, and general reasoning demonstrate improvements in accuracy and inference efficiency over several baselines, particularly Absolute Zero Reasoner (AZR).

Critical strengths
  1. Joint optimization of reasoning modes and computational effort. The paper addresses an important problem in efficient reasoning by combining reasoning-mode selection with adaptive token budgeting. The unified training framework provides a practical approach to balancing reasoning accuracy and computational efficiency.

  2. Promising performance and efficiency improvements. The experiments demonstrate improvements across multiple benchmarks and model sizes. In particular, MORE consistently reduces response length and inference time compared with AZR while maintaining or improving accuracy.

Critical weaknesses
  1. Insufficient component-level validation of the reward design. MATE-PPO combines correctness, budget-deviation, mode-fit, and mode–budget interaction rewards, but the paper does not isolate the contributions of the individual shaping terms. Although Section 6 notes the difficulty of preserving the coupled dynamics when ablating components, this leaves unclear which terms are necessary for the reported accuracy–efficiency improvements and whether a simpler reward would achieve similar results.

  2. Insufficient isolation of the benefit of joint mode–budget adaptation. Table 3 includes a useful shared-initialization control, with Base + Stage 1 + AZR reaching 52.4 AVG versus 57.3 for MORE. This supports the effectiveness of the complete training procedure. However, it does not distinguish the contributions of adaptive mode selection and adaptive token budgeting, or establish the additional benefit of coupling them. Since reasoning-mode selection and adaptive effort are closely related to existing approaches, matched mode-only and budget-only controls are important for clarifying the paper’s methodological contribution.

Questions for authors

Q1. Could the authors provide controlled ablations of the budget-deviation, mode-fit, and mode–budget interaction rewards, keeping the remaining controller and training settings fixed where possible? If removing a term makes the comparison ill-defined, please explain the specific dependency and consider a simpler replacement that preserves the training interface.

Q2. Building on the shared-initialization control in Table 3, how does MORE compare with adaptive mode selection under fixed budgets and adaptive budgeting under a fixed reasoning mode, using the same Stage-1 initialization and Stage-2 training budget? Please report both accuracy and computational cost to clarify the benefit of coupling the two.

3 comments

We thank the reviewer for the careful reading and for raising two central questions: which reward terms drive the accuracy–efficiency gains, and whether coupling mode and budget adaptation adds value beyond adapting either one alone. Both questions call for controlled evidence, so we conducted a new set of controlled experiments and analyses, presented below. All new runs use Qwen2.5-Coder-7B, start from the same Stage-1 checkpoint as MORE, and share its task-generation settings, Stage-2 update budget, decoding settings, and checkpoint-selection rule (App. G.2). For every run we report accuracy together with response length and evaluation time. We have also added these results and their analysis to the revised paper: the reward-component ablations to Sec. 5.3 (Table 4) and App. I.7 (RQ7), and the mode–budget controls to App. I.8 (RQ8).

In the tables below, rows marked † are new runs added for this response; unmarked rows come from the original submission.

1. Response to the Critical Weaknesses

W1 — Component-level validation of the reward design.

Conclusion. Each shaping term can be isolated cleanly, and all three are needed. Among all tested combinations, only the full return reaches the highest four-group accuracy and the lowest inference cost at the same time; every other combination is less accurate on average and more expensive.

Setup. Every Stage-2 run in this response, including all ablations and controls, keeps the role-specific execution feedback unchanged (R_acc for solvers, R_prop for proposers). We vary only the shaping terms on top of it, and the tables list only those terms. Each shaping term enters the return additively with its own coefficient (Eq. 2). For each ablation we set the coefficients of the omitted terms to zero, keep the active terms at their original coefficients, and leave mode-token generation and controller updates unchanged. Table R1 covers all seven non-empty combinations of the three terms. With these comparisons in place, we have removed the corresponding statement from Sec. 6.

Table R1. Reward-component ablations on Qwen2.5-Coder-7B (Sec. 5.3, Table 4; App. I.7). Accuracy in %. RLA is the mean response length in tokens and WT the evaluation time (min:s), both averaged over the four benchmark groups. Best values in bold.

Shaping termsCodeMathGSM8KM-ProAVGRLAWT
Stage 1 only (before Stage 2)52.219.970.039.145.301775.772:10
R_BD †58.434.275.342.552.601288.352:39
R_MF †60.940.476.044.855.531675.670:52
R_MB †53.625.372.841.148.201445.560:02
R_BD + R_MF †62.243.177.045.857.03956.540:20
R_MF + R_MB †61.742.676.945.656.701349.553:07
R_BD + R_MB †63.242.277.345.557.051567.263:29
Full MORE (all three)62.842.877.846.057.35904.034:41

Analysis.

The full return is best on both axes. It attains the highest AVG, and the lowest response length and evaluation time in every benchmark group. Individual groups can favor a subset (Code: R_BD + R_MB; Math: R_BD + R_MF), but no subset matches the full return on average accuracy and cost together.

The leave-one-out comparisons give each term a distinct role. Percentages below average the within-group reductions (App. I.7).

  • R_BD regulates effort across all rollouts. Adding it to R_MF + R_MB raises accuracy in all four groups (+0.65 AVG) and cuts response length and evaluation time by 33.5% each. It is the only term that also reaches failed rollouts.
  • R_MF delivers accuracy at low cost. It is the strongest single term (55.53 AVG). Added to R_BD + R_MB, it raises AVG by 0.3 while cutting response length by 42.8% and evaluation time by 43.9%.
  • R_MB refines mode-specific budgets. Added to R_BD + R_MF, it raises AVG by 0.3 and reduces response length and evaluation time by 13.9% and 15.4%, lowering both costs in every group.

A simpler reward does not match the full return. No variant with two shaping terms reaches it on both axes: the most accurate one (R_BD + R_MB, 57.05 AVG) uses 1.7 times the tokens, and the most efficient one (R_BD + R_MF, 57.03 AVG) still needs more tokens and more time in every group. Every variant with a single shaping term stays below 56 AVG. The three terms are therefore complementary rather than redundant.

W2 — Isolating the benefit of joint mode–budget adaptation.

Conclusion. Coupling the two adaptations yields gains that neither achieves alone. Against matched mode-only and budget-only controls, MORE is the most accurate configuration in every benchmark group, at a cost close to the cheaper control and well below the other.

Setup. Building on the shared-initialization control in Table 3, we trained a budget-only control (Long CoT forced, bands adaptive) and a mode-only control (modes adaptive, bands frozen) from the same Stage-1 checkpoint with the same Stage-2 update budget. Exact definitions are given in Q2.

Table R2. Mode–budget adaptation controls on Qwen2.5-Coder-7B (App. I.8, Table 14). Accuracy in %. RLA (tokens) and WT (min:s) are averaged over the four benchmark groups.

ConfigurationModeBudgetCodeMathGSM8KM-ProAVGRLAWT
Budget-only †fixed (Long CoT)adaptive59.340.777.645.755.83886.434:00
Mode-only †adaptivefixed57.437.575.445.153.851051.542:27
MORE (joint)adaptiveadaptive62.842.877.846.057.35904.034:41

Analysis.

  • Adaptive modes add accuracy that budget calibration alone cannot. Against budget-only, MORE gains 1.5 AVG points and improves all four groups (coding +3.5, mathematics +2.1) at about 2% higher response length and evaluation time. On MMLU-Pro it is both more accurate and 10.4% shorter.
  • Adaptive budgets add both accuracy and efficiency. Against mode-only, MORE gains 3.5 AVG points, improving every group, while cutting response length by 14.0% and evaluation time by 18.3%.
  • Coupling is the contribution. Both controls already exceed the shared-initialization control the reviewer highlighted (52.4 AVG), and joint adaptation goes furthest. This agrees with the inference-time comparison in the submission (Table 12): adaptive MORE leads on coding and stays within 1.3 points of the best fixed mode on every other suite while using 14–53% fewer tokens.

2. Answers to questions

Q1 — Controlled ablations of R_BD, R_MF, and R_MB, and their dependencies.

Yes. Table R1 (under W1) reports all seven combinations with the controller and training settings held fixed; these results are new and now appear in Sec. 5.3 and App. I.7. Removing any term is well defined, so no replacement reward is needed. The relevant dependencies are:

  • Mode prior. All three terms are indexed by the generated mode token (R_MF via U(d, m); R_BD and R_MB via the mode-conditioned bands), so all runs share the same Stage-1 checkpoint. The revised Sec. 6 states this dependency.
  • Shared budget state. R_BD and R_MB read the same bands; when either is removed, the controller still updates the bands with the same rule.
  • Outcome gating. R_MF and R_MB apply only to successful rollouts, while R_BD also reaches failed ones, so R_BD supplies an effort constraint the other two cannot replace.
  • Interaction. Added to R_BD alone, R_MB raises AVG by 4.45 points but lengthens responses by 11.1%; added to R_BD + R_MF, it shortens them by 13.9% (averages of the four within-group changes). We therefore report all seven combinations rather than leave-one-out runs alone.

The comparison with simpler rewards is given under W1.

Q2 — Mode-only and budget-only controls under matched initialization and training budget.

Yes; these results are new and now appear in App. I.8 (RQ8). Both controls share MORE's Stage-1 initialization, Stage-2 update budget, task-generation protocol, and evaluation settings.

  • Mode-only (adaptive modes, fixed budget). Adaptive mode selection and all three shaping terms keep their original coefficients, but every difficulty–mode band is frozen at its initialization (Table 10). Difficulty estimation and out-of-band penalties stay active; the bands are soft reward targets, not hard length limits.
  • Budget-only (adaptive budget, fixed mode). Long CoT is forced in Stage-2 training and evaluation; the adaptive bands, R_BD, and R_MB are kept, and β = 0. Long CoT is the most accurate fixed mode in every benchmark group (Table 12), which makes it the strongest single-mode reference.

Table R2 (under W2) reports accuracy, response length, and evaluation time for both controls.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer,

Thank you again for your constructive suggestions. We have provided point-by-point responses to all your questions, supported by detailed controlled experiments. Your recommendations on component-level validation and joint mode–budget adaptation have helped strengthen the paper’s rigor and completeness.

We would greatly appreciate your feedback on whether these responses and the additional evidence address your questions and concerns.

Sincerely,
The Authors

Comments on the paper

2 comments

We thank the reviewers for their insightful and constructive reviews of our manuscript. Reviewers f2kf and yaZP recognized that jointly optimizing reasoning modes and computational effort addresses an important and practically meaningful problem, and Reviewer 5rWE found the complete pipeline design, including the mode controller and the reward design, interesting. All the reviewers (f2kf, 5rWE, yaZP) also acknowledged that MORE improves over AZR in both accuracy and efficiency, and Reviewers f2kf and yaZP noted that our evaluation covers multiple benchmarks and model scales.

Based on all the reviews, here we provide a general response to the concerns raised by multiple reviewers. The individual responses are commented on below each review.

  • Regarding questions about experiments, we have addressed the concerns as follows:
    • We add controlled ablations over all seven combinations of the three reward-shaping terms (For Reviewer f2kf, yaZP)
    • We add matched mode-only and budget-only controls to isolate the benefit of joint mode–budget adaptation (For Reviewer f2kf)
    • We sweep the mode-fit coefficient to test sensitivity to the hand-designed utility and to trace accuracy–efficiency trade-offs (For Reviewer yaZP)
    • We extend the learning curves in Fig. 3(a) to the full 450 steps (For Reviewer 5rWE)
  • Regarding questions about clarification for details, we have addressed the concerns as follows:
    • We clarify the role of each component of MORE (For Reviewer yaZP)
    • We clarify the distinct roles of the budget-deviation and mode–budget rewards (For Reviewer yaZP)
    • We clarify the task proposer, which in MORE is mode- and budget-aware (For Reviewer 5rWE)
    • We clarify the policy optimization of MATE-PPO and its critic-free update (For Reviewer 5rWE)

We have also uploaded a revised version of our submission according to the suggestions provided by reviewers. The new ablations and controls are in Sec. 5.3 (Table 4) and App. I.7–I.8, the 450-step curves in Fig. 3(a), and the critic-free update in App. C.3.

We look forward to further feedback and discussion. Please feel free to let us know if further details/explanations are helpful or any other information needs to be supplemented .

Best,

Authors

Dear Reviewers,

Thank you very much for your efforts and constructive comments. As the discussion period is drawing to a close, we would like to kindly follow up on our response. Following your suggestions, we have added new controlled experiments and clarifications, and have incorporated them into the revised paper.

We would be grateful to know whether our response has addressed your concerns. If any questions remain, we would be glad to provide further details or clarifications. We would also greatly appreciate it if you could consider updating your assessment in light of the new results.

Thank you very much for your consideration.

Warm regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.