acceptodds
Under review as a conference paper at ICLR 2027

Learn to Scale Reasoning at Test Time Via Reinforcement Learning

Abstract

Scaling test-time computation has emerged as a powerful paradigm for Large Language Models (LLMs). The effectiveness of sequential test-time scaling depends on whether models can effectively utilize additional inference budgets for further reasoning. However, we show that RLVR-trained reasoning models struggle to benefit from additional inference computation, even when larger budgets are available. In this work, we investigate whether RL training can teach LLMs to benefit from additional inference computation. To analyze how training changes the use of additional computation, we introduce state coverage as a proxy for in-context exploration and find that incentivizing additional computation during training can broaden coverage but also induce repetition. These findings motivate Learning to scale INference with lEss repetition (\method), a reward-shaping approach that encourages continued reasoning on unsuccessful attempts while controlling redundancy. Experiments show that LINE enables Qwen3-4B-Base to translate additional inference computation into accuracy gains, while also improving average mathematical reasoning accuracy by 4.4 percentage points over GSPO and strengthening out-of-domain generalization. Broader evaluations show that \method improves reasoning performance across multiple models and benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BsXbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LINE, a reward-shaping approach for improving sequential test-time scaling in LLM reasoning. It encourages incorrect reasoning trajectories to use additional computation while penalizing excessive repetition. The authors also introduce state-coverage metrics to analyze in-context exploration. Experiments across several models and reasoning benchmarks show accuracy improvements over standard RLVR baselines.

Critical strengths
  1. The proposed reward shaping is simple and can be integrated into existing RL training frameworks.

  2. The paper provides extensive empirical evaluations, including comparisons with several RL and reward-shaping baselines, as well as improvements across multiple models.

Critical weaknesses
  1. Encouraging longer reasoning, length-based reward shaping, and exploration-oriented RL have been investigated in prior work (e3, Demystifying Long CoT, DIVER, LASER-D, IBPO...) LINE appears to combine a computation incentive with an n-gram repetition penalty. My main concern is that it's unclear what technical contribution this combination provides beyond existing methods. Also, I wonder why extending an incorrect reasoning trajectory should lead to better reasoning? An incorrect response that terminates early does not necessarily benefit from additional computation. Moreover, the target budget is defined as the reference-policy computation plus a fixed increment, without a justification for why this should be an appropriate reasoning length. The construction of the reference budget is also not specified.

  2. The proposed state coverage is measured through proxies such as distinct n-grams and equations, rather than meaningful reasoning states. Increases in absolute coverage may result from longer generations. In fact, Appendix E.1 shows that normalized distinct equation counts are lower for LINE than for GSPO across the reported budgets. Thus, the experiments do not show that LINE improves reasoning through more effective in-context state exploration.

Questions for authors
  1. What is the key technical distinction between LINE and existing length-based reward shaping or exploration-oriented RL approaches?

  2. How exactly are the reference computation budget and target increment determined? What justifies encouraging incorrect trajectories to reach this target length, rather than learning when additional reasoning is beneficial?

3, How are distinct mathematical states or equations identified and normalized? Given the lower normalized equation coverage, I wonder what evidence supports the claim that LINE enables more effective state exploration rather than simply longer, less repetitive outputs?

1 comment

Authors, please post a rebuttal if you want me to reaccess the paper.

Reviewer_Lv4EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LINE, a reward-shaping method for making better use of extra reasoning time. It penalizes incorrect responses that stop before a per-problem length target, and adds a penalty for repetition. The paper tests the method across several models and RL algorithms, and uses state coverage to study how reasoning changes.

Critical strengths

The gains across GRPO, GRPO-CH, and GSPO are encouraging. They suggest that the recipe can help beyond one particular training setup. The computation-only comparison is also useful: adding repetition control raises the math average from 52.2 to 53.8 and lowers the reported training cost. This gives a concrete reason to use the two reward terms together.

Critical weaknesses
  1. My main concern is what LINE adds beyond the closest reward designs. Demystifying [1] already combines length incentives for incorrect responses with repetition penalties. LASER-DE [2] also reduces the penalty for incorrect responses that pass a target length. The paper cites both and includes a cosine baseline. Still, I find the contribution beyond these ideas hard to pin down, especially without a direct comparison with LASER-DE.

  2. The ablations help, but they leave this question open. The computation-only comparison supports adding repetition control. Figure 9 varies the target increment, and Appendix E.8 studies repetition settings. These experiments do not isolate the per-problem reference target, the one-sided linear penalty, or the choice to leave correct responses free of length shaping. These choices could matter, but the current results do not tell us which ones account for the gains over prior methods. I would be more convinced by a few direct comparisons that change one of these choices at a time.

Questions for authors
  1. Could you compare LINE with LASER-DE under the same training setup? Keeping the repetition control and training budget the same would help isolate the length reward. Please also clarify how the cosine baseline was adapted and tuned.

  2. How is the reference length for each problem obtained? Which checkpoint is used, how many responses are sampled, and how are their lengths combined? A comparison with a fixed target would help show whether the per-problem reference is needed.

References

[1] Yang et al. Demystifying Long Chain-of-Thought Reasoning. ICML 2025. https://proceedings.mlr.press/v267/yang25ae.html

[2] Liu et al. Learn to Reason Efficiently with Adaptive Length-based Reward Shaping. 2025. Section 5.3 (LASER-DE). https://arxiv.org/abs/2505.15612

Reviewer_r9JvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies why RLVR-trained reasoning models fail to benefit from additional test-time compute: incorrect responses often stop with budget remaining, and raising the inference budget or forcing continuation ("Wait" cues) barely improves accuracy. The authors first add a simple reward RcompR_{comp} that encourages incorrect trajectories to use more tokens, and observe that while this improves accuracy, the extra tokens are partly spent on repetition, as shown by a drop in the distinct n-gram ratio. They therefore propose LINE, which combines RcompR_{comp} with a thresholded redundancy penalty RredR_{red}.

Critical strengths
  1. The discussion of different coverage metrics, while not initially proposed in this paper, is an interesting metric for measuring the model's trajectories. Feel like the author could provide a more detailed discussion about what each of these metrics means for RL with qualitative examples.
Critical weaknesses

Presentation

I read the paper carefully, but I still find its presentation difficult to follow, even though the method and idea are extremely simple. Here, I will list multiple issues with the presentation.

  1. RcompR_{comp} is important for the story in Section 3, but it is discussed in detail only in Section 4.
  2. How δB\delta B is selected is not discussed in the main paper.
  3. How the redundancy function and score D(τ)D(\tau) are defined is not discussed in the main paper. While Table 1 suggests that DD in the main experiment reuses the coverage metrics in Section 3.3, this is not stated directly.

The presentation feels scattered and lacks important details throughout.

Method and Results

  1. RcompR_{comp} is essentially a length bonus on failures. Under GRPO/GSPO's group-normalized advantage, RcompR_{comp} only re-ranks incorrect rollouts by length, so it directly encourages longer unsuccessful trajectories. Table 1 shows that most of the gain comes from this term alone, so the method mainly trades reasoning length for accuracy, which is not surprising. It is also close in spirit to the correctness-conditioned cosine length reward (Yang et al., 2025b).

  2. The redundancy evidence is circular. Adding an n-gram penalty and then showing improved n-gram diversity (DNC, DNR; Fig. 5, Fig. 7) is trivial, since the metric is what is being optimized.

  3. The claim on the RcompR_{comp} baseline is that it reduces the n-gram diversity massively (Fig. 8), and the authors claim that it reduces the diversity. This actually kind of counter with the results in Fig 7, where it increase the diversity on other coverage metrics. Low N-gram diversity does not necessarily mean the trajectory is repetitive in its reasoning structure, especially since Fig. 7 shows improvement over other coverage metrics. So the entire goal of this paper to propose LINE to reduce n-gram redundancy is questionable as a goal. Perhaps focusing on Distinct reasoning behavior type count (RBC), as a diversity measure over the reasoning states covered semantically, can be a more important metric. However, there is no evidence that LINE improves this over the simple RcompR_{comp} baseline.

Overall, the entire claim of the paper is the novelty of adding the redundancy reward to let the model think longer and perform better. While the method is trivial and not necessarily novel, the strong empirical results support its usefulness, which the paper does not show. The scale of the experiments is also incredibly small, and the authors do not report the standard deviation across runs, making the empirical findings less reliable.

Questions for authors

Comment

  1. For the presentation, in Section 3, there are many specific numbers and references to later results in 3.1 and 3.2. The referenced figures and tables are a bit far in the paper, and going back and forth makes reading a bit difficult. Perhaps add a teaser figure that highlights the main results (increasing reasoning length helps very little to performance by default grpo/gspo setting), or just add one sentence to cover it.
  2. What does each coverage metric (DNC, TD, DEC, RBC) actually capture, and is there evidence that higher values indicate better trajectories rather than just longer ones (Fig. 4 caption notes length is not controlled)? Moreover, since LINE directly penalizes n-gram repetition, n-gram metrics (DNC, DNR) are no longer independent evidence of better exploration (Goodhart's law); the paper's own Fig. 11 shows that directly optimizing coverage leads to reward hacking.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.