acceptodds
Under review as a conference paper at ICLR 2027

Efficient Agentic Reasoning Through Self-Regulated Simulative Planning

Abstract

How should an agent decide when and how to plan? A dominant approach builds the agent as a reactive policy with adaptive computation (e.g., chain-of-thought reasoning), trained end-to-end with the expectation that planning will emerge implicitly from sufficient data and compute. Without control over the presence, structure, or horizon of planning, however, these systems typically increase reasoning length dramatically during training, leading to inefficient token consumption that does not reliably translate to accuracy gains. We argue that efficient agentic reasoning benefits from a decomposition of decision-making into three interacting systems: **simulative reasoning** (System II) that grounds deliberation in future-state prediction using a world model, rather than unconstrained chain-of-thought; **self-regulation** (System III) that decides when and how deeply the agent plans at each turn through a learned **configurator**; and **reactive execution** (System I) that handles fine-grained reasoning and action. To test this, we develop SRAM (Self-Regulated Simulative Reasoning Agentic LLM), which realizes the configurator and simulative planning as distinct stages within an LLM's chain-of-thought reasoning, with the LLM itself serving as the world model in language space. We explore two implementations, v0.1-8B and v1.0-30B, both trained via supervised learning followed by reinforcement learning (RL). Across mathematical reasoning, scientific problem-solving, tabular data analysis, and web information seeking, SRAM-v0.1-8B and SRAM-v1.0-30B achieve overall Pass@1 competitive with systems at 120–357B and 685B–1T parameters, respectively, while SRAM-v1.0-30B consumes 25.8–94.9% fewer reasoning tokens than strong agentic LLMs of similar scale. Analysis reveals that RL increases average planning horizon by 22.8% while planning frequency grows only 2.0 percentage points, indicating that the model learns to plan *further ahead* rather than *more often*.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Agent Model, Adaptive Reasoning, Reinforcement Learning, World Model Planning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oQVdOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how to make agentic large language models more efficient and capable by replacing unconstrained chain-of-thought with a three-system decomposition: reactive execution (System I), simulative planning using the LLM as a language-space world model (System II), and learned self-regulation via a configurator (System III). The authors develop SR2AM\text{SR}^2\text{AM} (Self-Regulated Simulative Reasoning Agentic LLM) in two versions: v0.1-8B (using multi-module prompted inference) and v1.0-30B (using plan reconstruction from pretrained reasoning traces), trained via supervised fine-tuning followed by reinforcement learning. Across 11 benchmarks spanning math, science, tabular analysis, and web information-seeking, SR2AM\text{SR}^2\text{AM}-v1.0-30B achieves competitive Pass@1 performance compared to much larger models while consuming substantially fewer reasoning tokens. Further analysis reveals that reinforcement learning primarily deepens the planning horizon rather than increasing planning frequency.

Critical strengths

Principled Architecture & Efficiency: The decomposition into Systems I, II, and III elegantly addresses the growing issue of over-thinking and token inefficiency in agentic LLMs. SR2AM\text{SR}^2\text{AM}-v1.0-30B achieves strong accuracy while consuming 25.8% to 94.9% fewer reasoning tokens than comparable baselines.

Comprehensive Evaluation & Analysis: The paper provides a thorough empirical evaluation across 11 diverse benchmarks and includes insightful diagnostic analyses—such as evaluating world-model prediction accuracy and dissecting how reinforcement learning shifts planning behavior toward deeper horizons rather than unnecessary frequency.

Critical weaknesses

Language-Space World Model Limitations: While utilizing the LLM as a world model in language space is practical, it struggles notably on web tasks (achieving only 52.2% world model accuracy) due to high environmental uncertainty. A deeper discussion or mitigation for high-uncertainty domains would strengthen the work.

Over-Verification Tendencies: As highlighted in qualitative examples (e.g., Example 3 in Appendix M), the anticipatory planning mechanism can sometimes lead to excessive, unproductive verification loops when simple searches would suffice, indicating that the configurator's stopping criterion requires further calibration.

Questions for authors

Could the authors clarify how sensitive the configurator (System III) is to the quality of the base model used during plan reconstruction, and whether smaller base models (e.g., 8B) can reliably learn selective planning without requiring a 30B+ teacher?

Given that world-model prediction accuracy drops significantly on web tasks compared to math and science, what specific architectural or reward modifications might help the configurator dynamically curtail planning when environmental uncertainty spikes?

2 comments

We thank the reviewer for the positive assessment of the architecture and analysis, and for the constructive suggestions.

W1: Language-space world model limitations on web tasks.

Lower world-model accuracy on web tasks is expected given the inherent uncertainty of web search, and the system is designed to handle it. From RL, the configurator learns the shortest planning horizon for web tasks (1.86 steps, Figure 4), and SR²AM-v1.0-30B still achieves the highest XBench score among all 30B agentic LLMs (80.0) and competitive GAIA (76.0) while consuming fewer or equal tokens. The new perturbation experiment further confirms that predictions contribute functionally even under uncertainty: removing them lowers overall Pass@1 by 2.9 points (p=0.003) and lengthens episodes by 12%.

For future work, the reviewer's suggestion of dynamic reward modifications is well-taken. One concrete direction is to condition the configurator's planning-depth decision on an explicit uncertainty estimate, which would allow the system to curtail planning more aggressively when the world model is likely unreliable, rather than relying on static truncation.

W2: Over-verification tendencies (Example 3).

We appreciate the reviewer identifying this from our qualitative analysis. Even with occasional over-verification, SR²AM-v1.0-30B consumes 25.8–94.9% fewer reasoning tokens than comparable 30B agentic LLMs while achieving competitive or higher accuracy (Figure 3, Table 5), so over-verification does not erase the efficiency gains.

We view this as a future work direction rather than a fundamental limitation: further RL training (our models may be undertrained; Appendix F) or penalties for verification loops could reduce it. We will discuss this more explicitly in the camera-ready version.

Q1: How sensitive is the configurator to the base model quality for plan reconstruction? Can smaller models learn selective planning without a 30B+ teacher?

The experiments demonstrate that the decomposition works across different teacher and student configurations: v0.1 uses o4-mini to train 8B, v1.0 uses DeepSeek-V3.2 (685B) to train 30B, and in both cases the model learns selective planning. Figure 6 shows the benefit comes from data structure rather than teacher quality, as structured data outperforms unregulated data from the same teacher at every scale, with both o4-mini and Qwen3-Coder-480B.

Most directly relevant to the reviewer's question, Figure 5 tests whether a small model (Qwen3-8B) can learn selective planning through RL alone, without structured teacher data. Results show unregulated RL reaches 47.6 Pass@1 compared to SR²AM-v0.1-8B's 56.2, while consuming 52% more tokens.

This suggests the structured decomposition provides inductive bias that RL alone does not recover, and that the teacher's role is to supply structure instead of just reasoning content. Characterizing the minimum teacher quality for a given student size is an interesting question orthogonal to the paper's contribution.

Q2: What modifications might help the configurator curtail planning when uncertainty spikes?

Beyond the truncation strategies already in place (Sec 3.2), there are three possible directions:

  1. Explicit uncertainty signals (e.g., prediction entropy) as configurator input;
  2. Planning-cost terms in the RL reward that penalize long plans under low prediction accuracy;
  3. Explicitly modeling planning horizon per turn and supervising that decision, rather than just a binary plan/skip decision.

We will expand this discussion as future work in the camera-ready version.

We thank the reviewer again for the positive assessment and constructive suggestions. We have responded to the concerns and questions above, and would be glad to discuss any remaining questions in light of our response and new experiments.

Reviewer_Vq6ROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SR²AM, which integrates reactive execution, simulative planning, and self-regulation within a single LLM. Through supervised learning on structured trajectories followed by reinforcement learning, the model learns when to plan and how far ahead to plan. Experiments across four task domains demonstrate a favorable tradeoff between task accuracy and reasoning token consumption.

Critical strengths
  • The paper addresses an important problem: allocating reasoning effort effectively in multi-turn agents. Combining selective planning with explicit predictions of action outcomes is well motivated.
  • The plan reconstruction pipeline provides a practical way to incorporate structured planning into existing reasoning models.
  • Evaluation across multiple domains and component ablations support the approach. The finding that RL increases planning horizon with little change in planning frequency is particularly interesting.
Critical weaknesses
  • Plan–execution correspondence could be clearer. While executable tool actions are well defined, planned actions are expressed in natural language at variable levels of abstraction. It remains difficult to quantify how closely System I follows a plan and whether increased plan length consistently reflects deeper lookahead.
  • The semantics and functional role of states need further validation. Textual states may describe observations, intermediate conclusions, or desired outcomes. The prediction-alignment analysis is useful, but does not establish whether these representations causally guide execution beyond providing additional reasoning context.
  • The effect on exploration is insufficiently examined. Conditioning execution on a generated plan could encourage premature commitment to one solution path. Although replanning is supported, the paper provides limited quantitative evidence about trajectory diversity and recovery from misleading plans.
Questions for authors
  1. Could the authors quantify alignment between planned actions and executed actions, accounting for their potentially different temporal granularity?
  2. Would removing or perturbing predicted future states at inference time reduce performance? This could help establish their functional contribution.
  3. How does structured planning affect trajectory diversity and recovery from incorrect early assumptions, compared with unstructured reasoning under comparable computation budgets?
3 comments

We thank the reviewer for the thoughtful questions. Both Q1 and Q2 are now answered with new experiments in our General Response.

W1: Plan–execution correspondence could be clearer.

The new alignment analysis quantifies this directly. 87.6% of planned steps are carried out, and alignment does not decline for longer plans (92.0% for 3+ steps vs 84.4% for 2 steps), confirming that increased plan length reflects deeper executed lookahead rather than aspirational text.

W2: The semantics and functional role of states need further validation.

The new perturbation experiment directly tests this. Removing predicted states costs 2.9 points (p=0.003); swapping them costs 1.9 points (p=0.026). Together with 87.6% plan–execution alignment, these results establish that predictions causally guide action selection, not merely provide additional reasoning context.

W3: The effect on exploration is insufficiently examined.

We computed Pass@3 for SR²AM-v1.0-30B and the teacher CoT ablation after RL. The two reach effectively identical Pass@3: 82.5 vs 82.6 (p=0.93), while SR²AM leads on Pass@1 (one-sided p=0.03). This suggests that structured planning does not reduce the space of solvable problems, as both systems solve the same questions given multiple attempts. SR²AM, however, solves more on the first try. The improvement is in consistency, precisely the expected benefit of planning.

Additionally, the alignment analysis shows 12.4% of planned steps are abandoned when the environment reveals unexpected information, demonstrating recovery from misleading plans by replanning rather than persisting.

Q1: Quantify alignment between planned and executed actions?

Yes, the alignment is 87.6% overall. Please see General Response for the breakdown.

Q2: Would removing or perturbing predicted future states reduce performance?

Yes, removing leads to −2.9 overall Pass@1 (p=0.003); perturbing (i.e., swapping) leads to −1.9 (p=0.026). Please see General Response for more details.

Q3: How does structured planning affect trajectory diversity?

We measure trajectory diversity by Pass@3, and it is identical between the 3-system structured planning and unregulated reasoning (82.5 vs 82.6, p=0.93). The result shows structured planning improves first-attempt consistency without reducing exploration. The 12.4% plan-abandonment rate confirms the system's ability to adaptively recover from unexpected environment feedback.

Thank you for the further clarification, which has addressed some of my concerns. Having recently read many papers on planning and action, I believe this is a good paper overall and a valuable contribution to the community. I have therefore raised my score from 2 to 3. Best of luck!

We sincerely thank the reviewer for engaging with our response and new experiments, and for the updated assessment. We are glad the additional results addressed the concerns, and we appreciate the reviewer’s perspective that this is a valuable contribution to the planning and agentic reasoning community.

Reviewer_ewF8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an efficient agent reasoning framework named SR2AM, which decomposes the agent reasoning and action sequences into a hierarchy of components: 1) Configurator determines whether a new strategic planning is needed; 2) Planner generates a plan of proposed actions and predicted future states, if planning is needed according to the configurator; 3) Actor executes the action sequences following the plan. This amortised reasoning across different actions is expected to be more efficient than conventional agentic reasoning traces, which is unstructured, unconstrained and likely causes over-generation. Empirical results across 11 benchmarks spanning a variety of domains demonstrates SR2AM's competitive performance against baseline agent models, with reduced reasoning token consumption.

Critical strengths
  1. The paper is well-motivated and addresses an important problem. How to make agents reason more effectively with less token consumption is an intriguing problem which has practical economic significance, and this paper tackles this problem from a novel and promising angle of structured reasoning and planning.
  2. Positive empirical results across 11 benchmarks covering multiple domains, while using fewer reasoning tokens than several models of similar sizes. The analysis of experimental results and model behaviors are also informative and revealing that shows how the proposed architecture changes agent efficiency.
Critical weaknesses
  1. It remains unclear whether the model revises planned actions or considers alternative action sequences. For long-horizon tasks the agent performance critically depends on the model's plan simulation accuracy, but the paper did not analyze the correlation between task lengths and performance. While structured behavior scaffolding may improve reasoning efficiency due to the bias introduced, it also sacrifices flexibility and adaptability in different task environments.
  2. The planning simulation accuracy depends on the quality of the world model. However, the proposed framework reuses a general-purpose LLM to predict future world states, without explicitly training or validating it as a reliable world model. While the authors report prediction accuracy across several task categories, it remains unclear how reliably the LLM can simulate complex state transitions, particularly for smaller models with limited reasoning capabilities.
  3. Limited improvement over the original teacher COT while generating more tokens: Table 2 shows that the proposed model is only 0.9% above the teacher baseline (72.7% vs. 71.8%), but it generated significantly more tokens than baseline (5,672 vs. 4,555).
Questions for authors
  1. The performance of SR2AM critically relies on the simulation planner accuracy, but the simulation accuracy is expected to drop as the length of task increases. This creates a dilemma: long-horizon agents benefit more from reasoning token reduction, but due to the simulation accuracy drop, they may also be more vulnerable to inaccuracies in multi-step simulations. Does SR2AM maintain an accuracy-efficiency advantage as task horizons increase, particularly when world-model predictions become less reliable?
  2. Why does the S2RM model outperform the original teacher COT baseline by only 0.9% while using approximately 24.5% more reasoning tokens? This modest accuracy improvement comes at a substantial computational cost, raising concerns about the practical value of the proposed architecture. The authors could provide a more thorough explanation of this result and evaluate whether SR2AM offers meaningful advantages.
  3. The evaluation of the configurator accuracy remains insufficient, existing results do not establish whether the configurator learns to make effective planning decisions. Specifically, does the configurator correctly identifies when planning is beneficial versus unnecessary? This is critical to the accuracy-efficiency tradeoff.
3 comments

We thank the reviewer for the detailed feedback. We address each point below, referencing the new experiments in our General Response.

W1: Whether the model revises plans; structured scaffolding sacrifices flexibility.

The new plan-execution alignment analysis shows 87.6% of planned steps are carried out, with 12.4% adaptively abandoned, mainly in web tasks where search results contradict the predicted outcomes. This shows that as designed, the actor (System I) references the plan while the configurator overrides plans based on environment feedback. The qualitative examples (e.g., Appendix M, Example 1) demonstrate replanning after discovering the initial tally needs cross-validation.

W2: The LLM is reused as a world model without explicit training or validation.

The paper's contribution is the three-system decomposition, not a novel world model. We use a general-purpose LLM, which is a world model in the text space [1], to test whether the decomposition can be implemented in an end-to-end model. The fact that it does means better world models (a complementary direction) would only improve the framework. We have discussed this in Appendix I and will clarify in the camera-ready version.

We also did validate the accuracy of LLM-as-world-model: Table 1 shows 64.6% prediction accuracy with a clear success/failure gap (78.6% vs 51.6%), and the new perturbation experiment establishes causal contribution of the world model's prediction (−2.9 points when removed, p=0.003). For web tasks (52.2% accuracy), plans are only 1.86 steps on average (Figure 4), suggesting adaptive response to uncertainty.

W3: Limited improvement over teacher CoT while generating more tokens.

We would like to clarify that Table 3 is a data-construction ablation, not a system-level comparison. "Original Teacher CoT" is our own data without the structured decomposition, not an external baseline. The practical value is in Figure 2 and Table 5: SR²AM-v1.0-30B competes with 685B–1T models at 25.8–94.9% fewer reasoning tokens. The fact that the 3-system structure improves a model performing at such a level is testament to its practical value.

Results confirm that the 3-system structure provides a better RL substrate:

  1. After RL, the full decomposition significantly outperforms all variants (p<0.05);
  2. At 8B, structured data outperforms unregulated data from the same teacher at every scale (Figure 6);
  3. Through RL, this compounds to 56.2 vs 47.6 overall Pass@1 with 34% fewer tokens (Figure 5).

Q1: Does SR²AM maintain an accuracy-efficiency advantage as task horizons increase?

Web information-seeking tasks are the longest-horizon and most uncertain category in our evaluation (52.2% world model accuracy), and the accuracy-efficiency advantage holds there: SR²AM-v1.0-30B achieves the highest XBench score among all 30B agentic LLMs (80.0) and competitive GAIA (76.0), while consuming on average 48.0% fewer reasoning tokens on web tasks than MiroThinker-v1.5-30B, the strongest baseline (Figure 3).

The system maintains its advantage on long-horizon tasks because the configurator actively adapts by learning shorter planning horizons for uncertain domains (web: 1.86 steps vs. math: 2.32, Figure 4). Rather than a fixed planning strategy that would degrade with horizon, the system modulates planning depth, which is precisely the intended role of System III.

Q2: Why only 0.9% improvement?

Addressed above in response to W3 (ablation vs. system-level comparison).

Q3: Does the configurator correctly identify when planning is beneficial?

The System III ablation (Table 3) directly tests this: forcing the model to always plan lowers post-RL accuracy from 72.7 to 68.6 while increasing reasoning tokens. This shows selective invocation contributes to both accuracy and efficiency. We agree that finer-grained analysis (e.g., oracle hindsight) would be valuable and discuss this in Appendix I. We also note that measuring an ablation's overall effect is the standard evidence level in prior effort-adaptive work [2, 3].

[1] https://aclanthology.org/2023.emnlp-main.507.pdf

[2] https://arxiv.org/pdf/2505.11896

[3] https://arxiv.org/pdf/2505.16854

I'd like to thank the authors for their detailed responses. While the responses clarified some of the points raised in my original review, I still have fundamental concerns about the reliability and accuracy of the proposed approach, for example planning accuracy and simulation fidelity. I don't think that the paper is ready to be accepted in its current form. Therefore, I will maintain my original rating.

We thank the reviewer for their response. We note that our rebuttal and new experiments directly address the two concerns mentioned:

  • Planning accuracy: Table 1 validates world-model prediction accuracy (64.6% overall, 78.6% in successful vs. 51.6% in failed trajectories). The new plan-execution alignment analysis further shows 87.6% of planned steps are carried out, and remains stable across plan lengths.
  • Simulation fidelity: The new inference-time perturbation experiment establishes that predicted states causally contribute to performance (−2.9 points when removed, p = 0.003).

We'd like to emphasize that the paper’s contribution is the three-system decomposition and the evidence that it yields better accuracy-efficiency tradeoffs, not a perfect world model. The results show the decomposition helps even with an imperfect LLM-based world model, and better world models would only strengthen the framework (as discussed in our W2 response and Appendix I).

We would welcome any specific aspects of these results that the reviewer finds insufficient, so that we can address them concretely.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response and New Experiments

We thank all reviewers for their constructive feedback. We are encouraged that all three find the problem important (ewF8, Vq6R, oQVd), the approach novel and well-motivated (ewF8, oQVd), and the evaluation comprehensive (oQVd). We conducted new experiments during the discussion period to address shared concerns.

New Experiment 1: Predicted future states materially influence decisions

We intervened on predicted future states at inference time by removing predictions or swapping them for those from other questions. The results are below:

ConditionPass@1Δ (95% CI)p-value (one-sided)Turns/episode
Original72.7——19.0
Remove predictions69.8−2.9 [−5.0, −0.8]0.00321.3 (+12%)
Swap predictions (from other questions)70.8−1.9 [−3.8, +0.0]0.02620.7 (+9%)

Removing predicted states lowers Pass@1 by 2.9 points (p=0.003) and lengthens episodes by 12%. Swapping them costs 1.9 points (p=0.026). Once removed from history, the model stops writing those predictions (0% of plans contain one). This suggests that predicted states causally guide action selection, not just provide additional reasoning context.

New Experiment 2: Plans are followed and adaptively revised

We quantified plan–execution alignment across all test trajectories using the same LLM judge as Table 1. Overall, 87.6% [87.0, 88.1] of planned steps are carried out. Below are the breakdown of alignment rates across plan lengths:

Plan lengthShare of plansAlignment rate
1 step22.9%99.3% [98.9, 99.6]
2 steps62.5%84.4% [83.0, 86.0]
3 steps9.3%92.7% [91.2, 94.0]
4 steps3.6%91.0% [87.9, 93.5]
5+ steps1.7%91.8% [88.9, 94.9]

The results suggest alignment does not decline for longer plans, confirming deeper executed lookahead. The lower 2-step rate comes from web tasks where the agent drops planned actions when search results are unexpected, which aligns with the design intention of flexible replanning.

AuthorsOpenReview ↗

For Area Chairs: Summary of our responses

Dear Area Chairs,

Thank you for handling our submission and for encouraging reviewer engagement. As the discussion period closes, we briefly summarize the current state.

We conducted two new experiments during the discussion period: 1) an inference-time ablation of predicted future states (removing leads to −2.9 points, p = 0.003) and 2) a plan–execution alignment analysis (alignment rate 87.6%). Both support the paper’s claims.

We summarize the reviewers' engagement with our responses below:

  • Reviewer Vq6R engaged with our response, confirmed the new experiments addressed their concerns, raised their score from 2 to 3, and described this as “a good paper overall and a valuable contribution to the community.”
  • Reviewer oQVd has not responded; their concerns (web-task limitations, over-verification) were addressed in our rebuttal with concrete evidence.
  • Reviewer ewF8 responded but did not engage with the specific evidence; their stated concerns (planning accuracy, simulation fidelity) are directly addressed by our new experiments and existing analyses (Table 1).

We are committed to incorporating the two new experiments and the clarifications described in our rebuttal into the camera-ready version.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.