ROAST: Reducing Overthinking in Reasoning via Advantage Reshaping and State-aware Termination

Large reasoning models trained via Reinforcement Learning with Verifiable Rewards (RLVR) exhibit exceptional capabilities in complex tasks, yet they increasingly suffer from overthinking. This phenomenon traps them in redundant cycles and excessive self-denial. Existing methods that apply scalar penalties to reasoning length or steps struggle to balance efficiency and accuracy. Furthermore, the absence of structural metrics and the sparsity of binary rewards in Group Relative Policy Optimization (GRPO) severely hinder effective policy updates. To address this, we propose ROAST (Reducing Overthinking via Advantage reshaping and State-aware Termination). First, we introduce a structural prior that utilizes a suffix automaton to detect repeated substrings and high local repetition density. This mechanism safely terminates the reasoning process, significantly reducing generation length without compromising accuracy. Using the detected repetition patterns, we reshape the GRPO advantage for precise credit assignment: it rewards the shortest on-policy correct trajectories while applying relative down-weighting exclusively to the truncated redundant suffixes. This guarantees a positive net advantage for correct answers, mitigating gradient noise. Experiments demonstrate that ROAST effectively mitigates endless self-denial and path-switching, slashing reasoning lengths by 46-56% while preserving overall performance.