acceptodds
Under review as a conference paper at ICLR 2027

Not All Turns Are Equally Hard: Adaptive Thinking Budgets For Efficient Multi-Turn Reasoning in Agents

Abstract

As LLM reasoning performance plateaus, improving inference-time compute efficiency is crucial to mitigate overthinking and long thinking traces even for simple queries. Prior approaches including length regularization, adaptive routing and difficulty-based budget allocation primarily focus on single-turn settings and fail to address the sequential dependencies inherent in multi-turn reasoning. We formulate multi-turn reasoning as a sequential compute allocation problem and model it as a multi-objective Markov Decision Process. We propose TAB: Turn-Adaptive Budgets, a budget allocation policy that takes as input the conversation history and learns to maximize task accuracy while respecting global per-problem token constraints, and consequently learns to adaptively allot smaller budgets to easier turns and save an appropriate number of tokens for the crucial harder reasoning steps. We train and analyze the behavior of TAB on an auxiliary hierarchical plan-and-execute system for multi-turn math reasoning where solution trajectories are inexpensive to collect. Here, TAB saves up to 35% tokens and 30% latency over static and off-the-shelf LLM budget baselines. We then transfer the same TAB policy to environment-grounded agentic workflows spanning sequential tool-use, simulated user-agent dialogue and multi-step terminal tasks, where it generalizes and maintains accuracy while saving up to 20% tokens and 15% latency, without ever being trained on them.

Primary area:
reinforcement learning
Keywords:
Multi-Turn Reasoning, Efficient Reasoning, Compute Efficiency, GRPO, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_R7JGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Turn-Adaptive Budgets (TAB), a GRPO-trained policy that dynamically allocates reasoning budgets across turns in LLM agents to improve computational efficiency. Trained on multi-turn mathematical reasoning, TAB generalizes to agentic tasks without additional fine-tuning. Experiments show up to 35% token savings on math tasks and 20% on agentic benchmarks while maintaining or improving accuracy.

Critical strengths
  1. easy to follow
  2. important problem
Critical weaknesses

Although applying adaptive reasoning budgets to multi-turn agentic settings is interesting, the core method is relatively straightforward. TAB essentially trains an LLM-based budget allocator using standard GRPO with a reward combining task accuracy and token efficiency. Both difficulty-aware budget allocation and reinforcement learning for compute efficiency have been explored in previous studies.

Questions for authors

please see weakness

Reviewer_Vgh7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TAB, a model-based Budgeter fine-tuned using GRPO. At each turn it outputs a difficulty level from 0 to 4, which is mapped to the Solver's budget control: token limits for L1-Qwen3-8B-Exact, or reasoning-effort levels for GPT-5.4-nano. TAB is trained on multi-turn math reasoning and evaluated on both math and environment-grounded agentic tasks.

Critical strengths
  1. The design is simple and modular, working across Solvers. Because the Budgeter outputs difficulty levels rather than raw token counts, it plugs into different Solvers (a length-controlled open model and an API model with reasoning-effort settings) without retraining.
  2. Zero-shot transfer: Training only on decomposed math and evaluating on tool-use, user-agent dialogue and terminal tasks avoids expensive agentic rollouts.
Critical weaknesses
  1. The gains over off-the-shelf LLM allocator are small and have no statistical support. The improvements from TAB over off-the-shelf LLM allocator are small across benchmarks, especially on agentic tasks. The paper reports no seeds, confidence intervals or significance tests to support that the improvements are significant.
  2. No learned baselines: All baselines are fixed budgets or untrained prompted LLM judges, and those judges are poorly calibrated. Existing works on agentic routing to improve efficiency in multi-turn agents are discussed in the paper, but none of them are used as baselines in the experiment.
Questions for authors

See the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.