acceptodds
Under review as a conference paper at ICLR 2027

ARC: Trajectory-Aware Adaptive Reasoning Control via Latent-State Dynamics

Abstract

Large Reasoning Models (LRMs) have demonstrated strong capabilities in solving complex tasks through extended chain-of-thought reasoning. However, existing approaches to efficient reasoning primarily rely on truncation, compression, or local step-level signals, limiting adaptive computation allocation across the reasoning trajectory. Our empirical analysis shows that local signals can be ambiguous across different reasoning stages, whereas hidden states capture richer information about the evolution of reasoning. Building on this observation, we propose Adaptive Reasoning Controller (ARC), a lightweight trajectory-aware controller that leverages the hidden-state history of a frozen reasoning model to select among four discrete actions—Deep, Normal, Compress, or Conclude—for regulating subsequent computation. ARC introduces fewer than 2.8M trainable parameters, a negligible overhead over the frozen reasoning model. ARC is initialized through supervised learning on latent reasoning trajectories using pseudo-action labels derived from PRM signals, and is further optimized with Group Relative Policy Optimization (GRPO) using final-answer correctness and reasoning cost as learning signals. Under this hierarchical formulation, the frozen reasoning model serves as the low-level reasoning policy, whereas ARC learns a high-level strategy for adaptive computation allocation. Experiments across 4 LRMs and 4 benchmarks show that ARC reduces reasoning tokens by 32.8% on average and up to 55.3%, while increasing macro-averaged accuracy from 58.43% to 58.79%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Reasoning Models, Adaptive Reasoning, Test-Time Compute

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sqztOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ARC, a trajectory-aware adaptive reasoning framework that uses hidden-state histories to dynamically control the reasoning process of frozen LLMs. A lightweight controller selects among four actions and is trained through PRM-based supervised initialization followed by GRPO. Experiments across multiple models and reasoning benchmarks demonstrate substantial token savings while maintaining comparable overall accuracy.

Critical strengths
  1. The motivation is clear and well presented. Using hidden-state histories to capture reasoning dynamics provides an intuitive motivation for trajectory-aware computation control.
  2. The method is relatively lightweight. The controller requires fewer than 2.8M trainable parameters and does not require fine-tuning the underlying reasoning model.
Critical weaknesses
  1. The experimental evaluation is largely limited to mathematical reasoning. Most benchmarks focus on mathematics (AIME24, AIME25, and AMC23). This limits the evidence for the generalization of ARC to other reasoning domains, such as coding and instruction following.
  2. The presentation could be further improved. For example, Figure 3 contains excessive whitespace and conveys relatively limited information, while some mathematical formulations could be presented more clearly.
  3. The accuracy–efficiency trade-off is not consistently favorable. Although ARC reduces token consumption across all evaluated settings, it still incurs noticeable accuracy degradation on some benchmarks (e.g., AMC23 on DeepSeek-R1-Distill-Qwen-1.5B).
Questions for authors
  1. How important is the HMM-based pseudo-label construction? Would simpler PRM thresholds or alternative initialization strategies achieve comparable results after GRPO training?
  2. How reliably do the four control actions affect the actual reasoning behavior? For example, does selecting Deep consistently lead to more extensive reasoning, while Compress and Conclude reduce reasoning length as intended? A brief quantitative analysis would be helpful.
1 comment

Thank you for this suggestion.

  1. HMM-based pseudo-label construction.
    Our motivation is that similar PRM scores may correspond to different reasoning states depending on their sequence context. HMM incorporates score observations and state transitions, whereas independent thresholding does not explicitly use these temporal dependencies. However, without a controlled ablation, we cannot establish HMM’s independent contribution or rule out comparable performance from simpler initialization strategies after GRPO. We will consider comparing these alternatives under identical training settings, evaluating performance both before and after GRPO.

  2. Effects and reliability of the control actions.
    Appendices D.1 and D.2 provide qualitative evidence of behavior consistent with the action tags. In D.2, after [Deep-Step], the model further develops a previously proposed time-dilation correction and completes the calculation correctly. In D.1, [Normal-Step] is followed by further verification, while [Conclude-Step] is followed by a concise final answer.

    These cases support alignment between tag semantics and reasoning behavior, but do not establish overall reliability, directly demonstrate Compress, or quantify average length reductions. We agree that quantitative analysis would strengthen the evidence and will consider comparing actions under identical reasoning prefixes, measuring subsequent token counts, steps to completion, and accuracy.

Reviewer_MgTPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies efficient reasoning of Large Reasoning Models (LRMs). Existing adaptive-reasoning methods either truncate at fixed depths, compress text post-hoc, or use local step-level signals (PRM scores, confidence thresholds). The paper argues that local signals are ambiguous: the same instantaneous PRM score can correspond either to "sufficient reasoning, stop now" or to "deep but still unfinished reasoning, keep going". Layer-wise t-SNE visualizations of hidden states show structured temporal patterns across reasoning steps, suggesting the trajectory of latent states carries information that single-step signals do not.

To this end, Adaptive Reasoning Controller (ARC) is proposed as a lightweight controller (< 2.8M params, trained in SFT/GRPO stages) that observes the hidden-state history of a frozen reasoner and selects a discrete action after each reasoning step. The action is converted into a natural-language instruction tag inserted into the context before the next step is generated.

On the main benchmark grid of 4 reasoning backbones × 4 benchmarks (Table 1), ARC(GRPO) reduces generated reasoning tokens in all 16 cells (−17.33% to −55.30%, mean −32.78%) while nudging macro-averaged accuracy up by +0.36 pp over CoT. ARC(GPRO) also consistently outperforms DEER and DynaSor on the accuracy–efficiency trade-off, and improves over ARC(SFT) which incurs a −3.14 pp accuracy drop.

Critical strengths

1 The motivational analysis is substantive. The ambiguity of local PRM scores and the layer-wise t-SNE evidence that hidden states separate across reasoning steps together make a non-trivial case that trajectory-level signals carry information not present in instantaneous step scores.

2 The two-stage training recipe is well-motivated. HT-SFT on PRM-derived pseudo-actions gives the controller a structured behavioral prior (Eq. 9), which CA-GRPO then refines against the actual downstream reward. The paper is explicit that pseudo-actions are not treated as ground truth, and the ablation that ARC(SFT) alone incurs a −3.14 pp accuracy drop correctly shows that RL refinement is doing real work rather than just copying the HMM prior.

Critical weaknesses

1 The empirical evidence rests on tiny benchmarks. AIME24 and AIME25 each contain 30 problems, AMC23 contains 40, and GPQA-Diamond contains 198. The paper evaluates at temperature = 0.6, top_p = 0.95 but does not report pass@k, standard deviation, or any form of resampling / bootstrap. One problem flip equals 3.33 pp on AIME and 2.5 pp on AMC. Table 1 shows:

  • Qwen3-4B, AIME24: ARC 70.00 vs CoT 66.67 = +1 problem out of 30.
  • Qwen3-4B, AMC: ARC 97.50 vs CoT 93.70 = +1–2 problems out of 40.
  • Qwen3-8B, AIME24: ARC 76.67 vs CoT 70.00 = +2 problems out of 30.
  • DeepSeek-R1-Distill-Qwen-7B, AIME24: ARC 50.00 vs CoT 53.33 = −1 problem out of 30.
  • DeepSeek-R1-Distill-Qwen-1.5B, AMC: ARC 67.50 vs CoT 72.50 = −2 problems out of 40.

At pass@1 at T=0.6, each cell has substantial variance; a single-seed point estimate on 30 problems is simply not powerful enough to support the ±3 pp claims this paper makes. The paper's own claim "+0.36 pp mean accuracy" is well below the standard error of any single AIME cell. A proper evaluation would report pass@k or move to larger benchmarks (MATH-500, Olympiad-Bench, LiveCodeBench).

2 Appendix A.1 reveals that the four actions Deep/Normal/Compress/Conclude are implemented as in-context tags such as "[Compress Step]: The last reasoning step is likely redundant or repetitive. Avoid repeating it; make the next step brief and only add new useful information." The frozen reasoner is assumed to follow these instructions. But the controller cannot force the model to compress; it can only ask. Instruction-following reliability of 1.5B–8B reasoners on domain-specific control tags is far from 100%. The entire efficiency story therefore depends on the base model's instruction-following behavior with these specific tags on these specific tasks. If the model fails to follow the instruction, ARC spends an action with no effect. The base model's instruction-following reliability on these actions is never evaluated in the experiment.

Questions for authors

Please refer to the weakness section.

1 comment

Thank you for this suggestion. 1. Statistical reliability of the evaluation:
We conducted five repeated evaluations of both ARC and CoT using Qwen3-4B at temperature = 0.6 and top_p = 0.95, reporting across-run standard deviations and pass@1–5. After averaging the five responses per question, we performed 100,000 paired question-level bootstrap resamples to estimate 95% confidence intervals. The table below presents the AIME25 and AMC23 subset of this supplementary evaluation.

Accuracy is in percent; SD and accuracy differences are in percentage points. ΔAcc denotes ARC − CoT.

DatasetARC accuracy ± SDCoT accuracy ± SDΔAcc [95% CI]Generated-token reduction
AIME2562.00 ± 2.9862.67 ± 7.23−0.67 [−6.00, +4.67]15.36%
AMC2391.50 ± 3.3592.50 ± 2.50−1.00 [−6.50, +4.00]15.52%

On these two benchmarks, ARC reduced generated tokens by approximately 15%. The accuracy intervals include zero but do not establish equivalence. We will narrow our accuracy-preservation claims accordingly. These intervals are conditional on the observed generations and unadjusted for multiple comparisons; token measurements exclude controller overhead.

2. Reliability of instruction following:
We acknowledge this limitation. ARC’s four actions guide generation through natural-language instructions, and their effectiveness depends on the underlying model’s instruction-following ability. Their intended execution is therefore not guaranteed. We will explicitly acknowledge this limitation and identify action-level compliance evaluation and improved control reliability as directions for future research.

Reviewer_MKzPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ARC, a lightweight controller with under 2.8M parameters that makes large reasoning models think more efficiently without changing the models themselves. After each reasoning step, ARC reads the frozen model's history of hidden states and chooses one of four actions: Deep, Normal, Compress, or Conclude. That choice steers the next step through a short in-context tag. ARC is first trained with supervised learning on labels derived from PRM scores, then refined with GRPO using rewards for answer correctness and token efficiency. Across four models and four benchmarks, it cuts reasoning tokens by 32.8% on average (up to 55.3%) while slightly improving accuracy, from 58.43% to 58.79%.

Critical strengths
  1. Clear motivation. The paper shows that a single local PRM score can mean different things in different trajectories, and its t-SNE analysis finds structured patterns in hidden states across reasoning steps. This makes a solid case for treating computation control as a sequential decision problem rather than a thresholding problem.
  2. Lightweight and non-invasive. The controller has only 2.1–2.8M parameters and the reasoner stays frozen. It steers generation through soft in-context tags instead of hard truncation, so it is cheap to train and simple to attach to existing models.
  3. Well-designed training pipeline. PRM scores are used only as a weak prior for initialization, with HMM-calibrated thresholds and smoothing. The actual policy is learned with GRPO from downstream outcomes. The reward makes correctness dominate: the efficiency bonus applies only to correct answers, it is bounded at ±0.3, and there is an explicit penalty for concluding too early. This prevents "short but wrong" shortcuts.
Critical weaknesses
  1. Small accuracy gains. The average improvement is only +0.36 pp, and some settings get worse. There are already some GRPO-based efficient reasoning methods that can achieve improvements in both accuracy and efficiency, such as AutoThink and ThinkPrune.
  2. Narrow evaluation scope. Only four benchmarks are used, three of them competition math. The method is not tested on code or other domains, or against a wider range of efficient-reasoning baselines such as length-penalized RL fine-tuning or budget-forcing methods.
  3. Heuristic pseudo-labels and many hyperparameters. The PRM-to-action mapping, the HMM thresholds, the smoothing window, and the reward coefficients (λ_tok, c_tok, λ_con, α, b_min) are hand-chosen. The paper gives limited sensitivity analysis for them.
Questions for authors

The benchmark selection range in the current experiment is relatively narrow. Could out-of-domain experiments and AIME26 experiments be added?

3 comments

Thank you for suggesting broader evaluation. We have conducted additional experiments on AIME26 and LiveCodeBench-v6 using Qwen3-4B, extending the evaluation to code generation and a more recent mathematical benchmark. We also compare against DEER and Dynasor under the configurations shown below.

MethodAIME26 Acc. (%) ↑Tokens ↓LiveCodeBench-v6 Acc. (%) ↑Tokens ↓
CoT73.3314,032.170.7110,591.6
DEER60.0013,470.465.304,357.5
Dynasor53.3313,169.763.104,845.2
Ours73.3311,121.568.538,947.8
  1. AIME26. Our method maintains the CoT baseline’s accuracy of 73.33% while reducing token usage by 20.74%. In these evaluated configurations, DEER and Dynasor reduce tokens by 4.00% and 6.15%, respectively, with accuracy decreasing to 60.00% and 53.33%. Our method therefore achieves both higher accuracy and fewer generated tokens than these two baselines on AIME26.

  2. Cross-domain evaluation. On LiveCodeBench-v6, our method reduces token usage by 15.52%, with accuracy decreasing from 70.71% to 68.53% (−2.18 pp). DEER and Dynasor achieve larger token reductions, but incur larger accuracy drops of 5.41 pp and 7.61 pp, respectively. Our method preserves more accuracy at a higher token cost in these settings.

We agree that the modest average accuracy improvement should not be interpreted as a consistent improvement across all configurations. The additional results clarify this trade-off: our method preserves accuracy while reducing tokens on AIME26, whereas code generation exhibits a measurable accuracy cost. We will incorporate these results and make the scope and limitations of our claims explicit in the revision.

Thank you for your reply. The new results have addressed part of my concerns, but the issues of insignificant improvement and hyperparameter analysis remain unresolved.

Therefore, I will keep my score unchanged.

Thank you for raising this concern. We would like to clarify four points:

  1. HMM thresholds are data-derived. They are computed from adjacent fitted emission means, rather than manually selected cutoffs (Appendix B.1.1). Figure 6 compares K=2,3,4K=2,3,4, an independent GMM, and two temporal-order controls, supporting the temporal modeling and state-count choice.

  2. Pseudo-actions serve only as an initialization prior. After SFT, GRPO refines the controller using final-answer correctness and reasoning cost. Appendix C.1/Table 3 demonstrates improvement over the SFT-initialized controller, although this does not isolate sensitivity to the initial mapping.

  3. The reward design prioritizes correctness. Token-efficiency rewards apply only to correct trajectories and are bounded within [−0.3,0.3][-0.3,0.3], compared with correctness rewards of +1/−1+1/-1. The per-problem token baseline and its lower bound are adaptive rather than fixed.

  4. The pseudo-label smoothing window is w=3w=3. It averages the current and two preceding PRM scores, using available scores for the initial steps. This differs from the inference-time persistence window, which is 1 (Appendix B.3).

We acknowledge that these diagnostics and safeguards do not replace downstream sensitivity analysis of label construction and reward/baseline hyperparameters. We will explicitly report w=3w=3 and clarify the scope of the existing evidence in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.