- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces LINE, a reward-shaping approach for improving sequential test-time scaling in LLM reasoning. It encourages incorrect reasoning trajectories to use additional computation while penalizing excessive repetition. The authors also introduce state-coverage metrics to analyze in-context exploration. Experiments across several models and reasoning benchmarks show accuracy improvements over standard RLVR baselines.
Critical strengths
-
The proposed reward shaping is simple and can be integrated into existing RL training frameworks.
-
The paper provides extensive empirical evaluations, including comparisons with several RL and reward-shaping baselines, as well as improvements across multiple models.
Critical weaknesses
-
Encouraging longer reasoning, length-based reward shaping, and exploration-oriented RL have been investigated in prior work (e3, Demystifying Long CoT, DIVER, LASER-D, IBPO...) LINE appears to combine a computation incentive with an n-gram repetition penalty. My main concern is that it's unclear what technical contribution this combination provides beyond existing methods. Also, I wonder why extending an incorrect reasoning trajectory should lead to better reasoning? An incorrect response that terminates early does not necessarily benefit from additional computation. Moreover, the target budget is defined as the reference-policy computation plus a fixed increment, without a justification for why this should be an appropriate reasoning length. The construction of the reference budget is also not specified.
-
The proposed state coverage is measured through proxies such as distinct n-grams and equations, rather than meaningful reasoning states. Increases in absolute coverage may result from longer generations. In fact, Appendix E.1 shows that normalized distinct equation counts are lower for LINE than for GSPO across the reported budgets. Thus, the experiments do not show that LINE improves reasoning through more effective in-context state exploration.
Questions for authors
-
What is the key technical distinction between LINE and existing length-based reward shaping or exploration-oriented RL approaches?
-
How exactly are the reference computation budget and target increment determined? What justifies encouraging incorrect trajectories to reach this target length, rather than learning when additional reasoning is beneficial?
3, How are distinct mathematical states or equations identified and normalized? Given the lower normalized equation coverage, I wonder what evidence supports the claim that LINE enables more effective state exploration rather than simply longer, less repetitive outputs?
Authors, please post a rebuttal if you want me to reaccess the paper.