acceptodds
Under review as a conference paper at ICLR 2027

Do Not Go Gentle into Repetition: Escaping Overthinking with Repetition-Aware Reinforcement Learning

Abstract

Recent reasoning models achieve strong performance through reinforcement learning, but often generate excessively long traces containing redundant verification and repetitive loops. Existing efficiency-oriented methods mainly rely on length rewards or position-agnostic truncation, which may interrupt productive reasoning and can even assign negative advantages to correct trajectories. We propose , petition-aware reinocement lerng for efficient reasoning. REFRAIN uses repetition patterns to identify unproductive reasoning and construct shorter training rollouts. It further introduces correctness-preserving optimization to avoid distorting the learning signal of successful trajectories. Experiments across mathematical reasoning benchmarks and model scales show that REFRAIN improves both training and inference efficiency while maintaining or improving reasoning accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_K86gOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3(+1)
Soundness
3
Read review
Paper summary

This paper introduces REFRAIN to address the issue of repetitive loops in reinforcement learning (RL) reasoning. The method employs LRS (Loop-Removal Strategy) to identify truncation points and generate "recovery rollouts," and subsequently trains the model using "sign-preserving efficiency advantage" and "prefix-trie credit assignment." Experimental results demonstrate that this approach reduces both the number of generated tokens and the per-step training time, while maintaining or improving accuracy.

Critical strengths

The paper identifies a concrete and specific failure mode: models may become trapped in repetitive verification or cyclic generation even after successfully completing valid reasoning—a finding that goes beyond the simplistic assumption that "longer outputs are necessarily worse." The entropy pattern shown in Figure 1 provides meaningful diagnostic evidence for this phenomenon. The methodology is described comprehensively, encompassing formal analysis, algorithmic details, and an extensive set of ablation studies.

Critical weaknesses
  1. LRS has not been validated as a reliable proxy for ineffective reasoning. The method lacks a minimum repetition length, semantic constraints, or confidence thresholds. Repeated use of formulas, necessary verification, and corrective recalculation may all trigger LRS. Appendix C provides only two successful examples, without human annotations or a false-positive analysis. I recommend conducting blind human annotation on complete reasoning trajectories and reporting precision, recall, and F1 for detecting harmful repetition.

  2. The contribution of prefix-trie averaging has not been isolated. The Mask-All ablation in Table 8 removes all gradients from shared prefixes, demonstrating only that shared-prefix feedback should not be discarded, rather than that trie averaging is superior to standard trajectory-level credit assignment. A more appropriate ablation would be REFRAIN-TrajCredit, which keeps the rollouts, boundaries, hints, advantages, and computational budget unchanged, while replacing trie averaging with standard trajectory-wise credit assignment. Appendix E indicates that the two approaches differ only when mixed-sign advantages and PPO clipping occur simultaneously, yet the paper does not report how frequently such cases arise.

Questions for authors

1. Could the authors provide a human-annotated repetition evaluation that distinguishes harmful loops, necessary verification, and corrective recalculation, and report precision, recall, F1, and the intervention harm rate?

2. Could the authors include a standard trajectory-level credit assignment baseline and report the frequency of mixed-sign advantages at shared nodes and the proportion of cases where PPO clipping is triggered?

2 comments
AuthorsOpenReview ↗

Response to Reviewer K86g [1/2]

W1: Reliability of LRS

As clarified in Sec. 4.2, LRS is a lightweight diagnostic of recurrent token-level behavior, rather than a perfect semantic classifier of harmful repetition versus useful verification.

Recovery and harm analysis. Our existing Appendix G (Fig. 5) compares REFRAIN with REFRAIN-Random, REFRAIN-Resample, and JET in terms of Recovery Rate and Harm Rate. REFRAIN generally achieves more stable recovery quality, with lower harm rates than its random and resampling variants and a greater ability to recover incorrect trajectories.

Comparison with a learned detector. We additionally compare LRS (minimum threshold = 8) against WordSaladChopper (WSC, EMNLP'25). We adapt and retrain WSC's original classifier-based implementation for Qwen3.5-4B. Both methods operate on identical original trajectories with matched recovery settings.

DatasetTAcc. WSC / LRS (%)Length WSC / LRSTrigger WSC / LRS (%)
GSM8K0.677.86 / 77.942,851 / 2,7320.15 / 91.28
GSM8K0.079.53 / 79.763,065 / 2,5601.36 / 92.19
MATH-5000.687.80 / 88.0012,273 / 10,6322.40 / 54.00
MATH-5000.066.00 / 68.8015,769 / 10,12511.20 / 77.00
GPQA0.664.14 / 66.6721,885 / 19,1268.59 / 38.89
GPQA0.051.01 / 52.5323,389 / 12,96629.80 / 82.32
AIME250.656.25 / 58.3325,814 / 26,4710.00 / 32.50

Despite triggering substantially more frequently, LRS achieves higher accuracy than WSC in all seven settings and shorter final responses in six, while preserving or improving accuracy relative to the original trajectories.

These results support LRS as a practical intervention signal, rather than establishing its semantic detection precision. Together with the recovery/harm analysis in Fig. 5, they suggest that repetition-aware intervention can mitigate unproductive reasoning without systematically degrading correctness.

We have expanded Appendix J with WSC reproduction details, matched intervention protocols, and complete accuracy, length, and trigger-rate results.

AuthorsOpenReview ↗

Response to Reviewer K86g [2/2]

W2&Q2:Contribution of Trie Credit

To isolate the contribution of Trie credit assignment, we add REFRAIN-TrajCredit (Mask-Hint), which uses standard trajectory-level advantages while keeping all other training components unchanged.

MethodAccuracy (%) ↑#Tokens ↓
REFRAIN-Mask-All39.209,745
REFRAIN-TrajCredit (Mask-Hint)41.159,615
REFRAIN (Trie Credit)42.478,844

Compared with TrajCredit, Trie credit improves accuracy by 1.32 percentage points while reducing response length by 8.0%. This provides direct evidence for the benefit of inverse-occurrence weighting. Mask-All further shows that simply removing shared-prefix gradients is less effective. We have also incorporated Linear-Mask-Hint into the extended ablations in Appendix H (Tables 7 and 8), including its configuration, benchmark-wise results, and comparison with Mask-All and the default Trie Credit.

Mechanism analysis. We clarify that the difference between Trie and trajectory-level credit does not arise from PPO clipping, which is inactive in our training setting (train_batch_size = mini_batch_size = 32). Our implementation applies inverse-occurrence weighting to each trajectory occurrence:

Ai,tTrie=AiDi,t,A_{i,t}^{\mathrm{Trie}}=\frac{A_i}{D_{i,t}},

where Di,tD_{i,t} is the occupancy of the corresponding shared-prefix node. Consequently, the aggregate contribution of a shared node is scaled by its inverse occupancy, even without PPO clipping. This applies to all shared nodes, while mixed-sign downstream advantages represent a particularly important case of conflicting feedback.

In our TrajCredit training, 66.7% of training token occurrences belong to shared nodes, and 18.6% of shared-token occurrences correspond to nodes with mixed-sign downstream advantages (equivalent to 12.4% of all training tokens). The mean occupancy is comparable between TrajCredit and REFRAIN (1.503 vs. 1.526).

We have corrected the derivation in Appendix E and revised the related claims to reflect the actual inverse-occurrence weighting implementation (Section 3.3 Eq. 11/12, Algorithm 5). The previous derivation inadvertently introduced a summation before occupancy normalization, leading to an incorrect equivalence claim under the unclipped objective.

Q1: Human-annotated repetition evaluation

Thank you for this suggestion. We agree that human annotation would provide valuable evidence for distinguishing harmful repetition from useful verification and corrective recalculation.

Given our limited computational and human resources during the rebuttal period, we have prioritized additional empirical comparisons and ablation studies. In particular, we have compared LRS with WordSaladChopper (WSC, EMNLP'25), adapting and retraining its original classifier for Qwen3.5-4B. Under matched recovery settings, LRS achieves higher accuracy in all seven evaluation settings and shorter responses in six, supporting its practical effectiveness as an intervention signal.

We acknowledge that this comparison does not replace human-annotated evaluation of semantic detection precision. We will make every effort to conduct a small-scale human annotation study during the rebuttal period, including an analysis of harmful repetition, useful verification, and corrective recalculation.


We hope that our additional experiments and clarifications have adequately addressed your concerns and strengthened the evidence supporting our contributions. We sincerely appreciate your thoughtful feedback and hope you will kindly reconsider your assessment in light of these updates.

Reviewer_FaLhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Reasoning models can overthink (i.e. produce lengthy CoT before answering) and whose thinking can repeat completed work. This paper proposes REFRAIN, an RL method for efficient reasoning that detects repetitive tokens in training rollouts, branches before the cycle repeats, inserts an answer cue and generates a recovery continuation. It combines these branches with a length-sensitive advantage and shared-prefix credit averaging.

Critical strengths

Useful and practical result: the 4B model improves or matches GRPO accuracy on every reported math benchmark while reducing tokens effectively. It also seems to be more accuracy-preserving than other efficient RL baselines (despite at the expense of less reduction on thinking length).

Reasonable experiment controls: the paper performs ablation studies and in the Appendix details the baseline details and the porting (i.e. implementation) details.

Critical weaknesses

Task type: most of the datasets involved in the paper are math-focused (except for GPQADiamond used in OOD test); would like to see the result on other non-math reasoning benchmarks (for instance, benchmarks involving tool use, searching, etc.).

Model use: despite the paper performing RL experiments on models of 3 different sizes. (1) the largest model is only of 4B size; would like to see the algorithm being applied to slightly larger (e.g. 7B-8B dense) models -- which should be possible for the 4 A800 resource that the paper authors have access to; (2) certain model like DeepSeek-R1-Distill-Qwen-1.5B is a bit old and out-dated.

Questions for authors
  • What continuation cap governs the main REFRAIN runs, and does Figure. 4 include repetition detection, prefix processing, and all recovery generation?

  • With identical recovery branches and rewards, how does the usual trajectory credit compare with trie averaging, and how frequently does mixed-sign clipping occur?

4 comments
AuthorsOpenReview ↗

Response to Reviewer FaLh [1/2]

W1: Lack of non-math evaluation (task type)

Thank you for this suggestion. To assess generalization beyond mathematical reasoning, we additionally evaluate REFRAIN on code-generation benchmark HumanEval+. While it does not fully represent interactive agentic tasks, it provides a practical non-math evaluation.

We use a 16K token budget and four generations per problem, reporting mean@4 accuracy, pass@4, and average response length.

ModelMethodmean@4 (%) ↑pass@4 (%) ↑#Tokens ↓
Qwen3-1.7BGRPO78.585.42,670
Qwen3-1.7BREFRAIN78.789.62,723
Qwen3.5-4BGRPO86.393.9915
Qwen3.5-4BREFRAIN86.793.9792

REFRAIN maintains or improves code-generation accuracy across both model scales. In particular, on Qwen3.5-4B, it achieves 13.4% fewer tokens with slightly higher mean@4 accuracy. On Qwen3-1.7B, REFRAIN improves pass@4 by 4.2 percentage points, with only a 2.0% increase in response length. These results provide complementary evidence that REFRAIN generalizes beyond mathematical reasoning, although full agentic evaluation remains outside the scope of this work.

W2: Model scales and architectures (model use)

Thank you for recognizing the strengths of our Qwen3.5-4B results. We include the earlier DeepSeek-R1-Distill-Qwen-1.5B primarily to enable direct end-to-end comparisons with existing baselines.

Regarding scalability, we believe repetition-aware intervention has potential beyond the evaluated model scales, as repetitive reasoning is not exclusive to small models and can also occur in larger systems. However, we acknowledge that its effectiveness on larger models requires empirical validation.

Given our computational constraints, we are additionally evaluating Qwen3.5-9B to investigate whether repetition remains prevalent at larger scales and whether LRS continues to provide a useful diagnostic signal.

Repetition analysis on larger models. To examine whether repetition remains relevant at larger scales, we additionally analyze Qwen3.5-2B, 4B, and 9B using 16 rollouts per problem. For each problem containing both correct and incorrect trajectories, we compare their average LRS lengths.

ModelAMC23: Higher LRS in Incorrect TrajectoriesMean ΔLRS
Qwen3.5-2B13/16 (81.3%)+1,174
Qwen3.5-4B16/19 (84.2%)+1,095
Qwen3.5-9B22/28 (78.6%)+497.5

Notably, even at 9B, incorrect trajectories exhibit higher LRS in 78.6% of comparable AMC23 problems, with a mean difference of approximately 498 tokens. A similar tendency appears on HMMT26 (5/6 comparable problems). These observations indicate that substantial LRS differences between correct and incorrect trajectories remain observable in larger models, suggesting that LRS may still provide a useful signal for identifying promising recovery intervention points beyond the scales used for REFRAIN training.

AuthorsOpenReview ↗

Response to Reviewer FaLh [2/2]

Q1: Continuation cap and cost in Figure 4

1. Continuation cap. All training experiments use the same 1,024-token recovery continuation cap, with a maximum total response length of 8,192 tokens, including the retained prefix and inserted hint. Thus, the recovery budget is consistently constrained across methods.

2. Cost in Figure 4. We report VeRL's original end-to-end training step time, which includes full and recovery rollout generation, LRS localization, reward and advantage computation, and policy optimization. Therefore, the reported cost already accounts for the additional operations introduced by REFRAIN.

Both the suffix automaton and prefix trie have linear time and space complexity in the processed sequence length, introducing lightweight overhead compared with the computationally intensive rollout generation. The overall training time is primarily dominated by rollout generation rather than these auxiliary data structures.

Q2: Contribution of Trie Credit

To isolate the contribution of Trie credit assignment, we add REFRAIN-TrajCredit (Mask-Hint), which uses standard trajectory-level advantages while keeping all other training components unchanged.

MethodAccuracy (%) ↑#Tokens ↓
REFRAIN-Mask-All39.209,745
REFRAIN-TrajCredit (Mask-Hint)41.159,615
REFRAIN (Trie Credit)42.478,844

Compared with TrajCredit, Trie credit improves accuracy by 1.32 percentage points while reducing response length by 8.0%. This provides direct evidence for the benefit of inverse-occurrence weighting. Mask-All further shows that simply removing shared-prefix gradients is less effective. We have also incorporated Linear-Mask-Hint into the extended ablations in Appendix H (Tables 7 and 8), including its configuration, benchmark-wise results, and comparison with Mask-All and the default Trie Credit.

Mechanism analysis. We clarify that the difference between Trie and trajectory-level credit does not arise from PPO clipping, which is inactive in our training setting (train_batch_size = mini_batch_size = 32). Our implementation applies inverse-occurrence weighting to each trajectory occurrence:

Ai,tTrie=AiDi,t,A_{i,t}^{\mathrm{Trie}}=\frac{A_i}{D_{i,t}},

where Di,tD_{i,t} is the occupancy of the corresponding shared-prefix node. Consequently, the aggregate contribution of a shared node is scaled by its inverse occupancy, even without PPO clipping. This applies to all shared nodes, while mixed-sign downstream advantages represent a particularly important case of conflicting feedback.

In our TrajCredit training, 66.7% of training token occurrences belong to shared nodes, and 18.6% of shared-token occurrences correspond to nodes with mixed-sign downstream advantages (equivalent to 12.4% of all training tokens). The mean occupancy is comparable between TrajCredit and REFRAIN (1.503 vs. 1.526).

We have corrected the derivation in Appendix E and revised the related claims to reflect the actual inverse-occurrence weighting implementation (Section 3.3 Eq. 11/12, Algorithm 5). The previous derivation inadvertently introduced a summation before occupancy normalization, leading to an incorrect equivalence claim under the unclipped objective.


We hope that our additional experiments and clarifications have adequately addressed your concerns and strengthened the evidence supporting our contributions. We sincerely appreciate your thoughtful feedback and hope you will kindly reconsider your assessment in light of these updates.

Reviewer_FaLhOpenReview ↗

Thanks for the careful rebuttal

Thanks for the careful rebuttal. Most of my concerns are resolved so I would be happy to increase my score.

AuthorsOpenReview ↗

Thanks & Additional Qwen3.5-9B Results (W2)

We sincerely thank you for your positive reassessment and for increasing your rating. We greatly appreciate your constructive feedback.

To further address your concern about model scalability (W2), we have now completed REFRAIN training on Qwen3.5-9B (70 steps). We evaluate the trained model under a 16K token budget with T=0.6 (mean@4), using the same evaluation protocol.

BenchmarkBase Acc. (%)Base TokensREFRAIN Acc. (%)REFRAIN Tokens
HMMT2615.9113,88930.3014,283
AIME2520.8315,69757.5010,119
AIME2615.8315,94266.679,379
AMC2358.7513,88996.884,238
Average27.8314,85462.849,505

Compared with the base model, REFRAIN improves average accuracy by 35.01 percentage points while reducing average response length by 36.0%. Accuracy improves substantially across all four benchmarks, with particularly large gains on AIME25 (+36.67 pp) and AIME26 (+50.84 pp).

These results extend our training experiments from 4B to 9B, providing encouraging evidence that REFRAIN can effectively scale to larger reasoning models while achieving a favorable accuracy–efficiency trade-off.

We sincerely appreciate your valuable feedback and hope these additional results further strengthen your assessment of our work.

Reviewer_3Cn4OpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2(+1)
Presentation
3(+1)
Soundness
3
Read review
Paper summary

This paper proposes REFRAIN, a repetition-aware reinforcement learning method for reducing overthinking in reasoning models. It identifies repetitive patterns to construct shorter training rollouts and introduces a correctness-preserving optimization scheme to prevent efficiency incentives from assigning negative advantages to correct trajectories. Experiments across mathematical reasoning benchmarks and model scales show improved training and inference efficiency while maintaining or improving reasoning accuracy.

Critical strengths
  1. The paper uses repetition structure to identify intervention points, providing a more targeted alternative to fixed-position truncation or length-based penalties. This offers a clear and interpretable approach to reducing redundant computation.

  2. The method explicitly addresses negative advantages assigned to correct trajectories and conflicting feedback on shared prefixes. The correctness-aligned auxiliary advantage and prefix-trie aggregation form a coherent design, supported by an analysis of their interaction with policy clipping.

  3. The experiments cover three model configurations, mathematical reasoning benchmarks, and an out-of-domain scientific reasoning benchmark. The paper evaluates accuracy, response length, and training time, with ablations examining the intervention location, transition cue, and advantage design.

Critical weaknesses
  1. The training experiments are limited to models with 1.5B–4B parameters. It remains unclear whether exact repetition is a sufficiently common and consequential bottleneck in larger or better-optimized reasoning models. Evaluating such models and quantifying the fraction of computation attributable to repetition would help establish the method’s broader practical relevance.

  2. Insufficient empirical positioning against closely related methods. Although the paper discusses several efficient-reasoning approaches, the comparisons leave some important gaps. For example, S-GRPO is directly relevant to early-exit rollout construction but is not included in the main experimental comparison. A matched comparison would help isolate the benefit of repetition-aware intervention from that of constructing additional early-exit trajectories. And miss important related works, such as:
    (1) Reconsidering Overthinking: Penalizing Internal and External Redundancy in CoT Reasoning. (2) Does Your Reasoning Model Implicitly Know When to Stop Thinking?

3,Limited evidence for the statistical reliability of some gains. Several improvements over GRPO are small: on Qwen3-1.7B, average mathematical accuracy increases by only 0.40 percentage points, while GPQA accuracy is unchanged. The reported repeated generations do not establish robustness across independent training runs. Multiple training seeds and uncertainty estimates for the main comparisons are needed to distinguish consistent improvements from sampling and training variability.

Questions for authors
  1. How reliably does the repetition detector distinguish unproductive loops from useful verification or repeated mathematical expressions? Could the authors provide an error analysis and clarify whether repetition remains a substantial source of inefficiency in larger, stronger reasoning models?

  2. Are the improvements over GRPO consistent across independent training seeds?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 3Cn4 [1/2]

W1: Broader practical relevance

Given our computational constraints, we are additionally evaluating Qwen3.5-9B to investigate whether repetition remains prevalent at larger scales and whether LRS continues to provide a useful diagnostic signal.

Repetition analysis on larger models. To examine whether repetition remains relevant at larger scales, we additionally analyze Qwen3.5-2B, 4B, and 9B using 16 rollouts per problem. For each problem containing both correct and incorrect trajectories, we compare their average LRS lengths.

ModelAMC23: Higher LRS in Incorrect TrajectoriesMean ΔLRS
Qwen3.5-2B13/16 (81.3%)+1,174
Qwen3.5-4B16/19 (84.2%)+1,095
Qwen3.5-9B22/28 (78.6%)+497.5

Notably, even at 9B, incorrect trajectories exhibit higher LRS in 78.6% of comparable AMC23 problems, with a mean difference of approximately 498 tokens. A similar tendency appears on HMMT26 (5/6 comparable problems). These observations indicate that substantial LRS differences between correct and incorrect trajectories remain observable in larger models, suggesting that LRS may still provide a useful signal for identifying promising recovery intervention points beyond the scales used for REFRAIN training.

W2: Lack of baselines and citations

Thank you for highlighting these relevant works. S-GRPO was not included in our original comparison because its code and checkpoints were unavailable. Nevertheless, our existing REFRAIN-Random ablation isolates the benefit of repetition-aware intervention from simply introducing additional early-exit trajectories.

We have now reproduced S-GRPO following its paper and evaluated SAGE-RL (ICML'26), the method proposed in Does Your Reasoning Model Implicitly Know When to Stop Thinking? using its recently released official training code under the same Qwen3-1.7B setting.

MethodAcc. (%) ↑#Tokens ↓Step Time (s) ↓
S-GRPO38.148,795167
SAGE-RL42.4410,193729
REFRAIN42.478,844301

REFRAIN outperforms S-GRPO by 4.33 accuracy points and achieves comparable accuracy to SAGE-RL with 13.2% fewer tokens and 58.7% lower training step time.

We also observe a notable entropy decline in SAGE-RL (w2r1): its average policy entropy drops from 0.268 (steps 1–60) to 0.161 (steps 61–119), whereas REFRAIN remains stable (0.279 → 0.275) and S-GRPO increases (0.281 → 0.326).

We have also added Reconsidering Overthinking (Hong et al., 2025) to Sec. 2.2. While it reduces internal and external redundancy through semantic- and post-answer-based penalties, REFRAIN identifies token-level repetition boundaries to construct recovery rollouts for RL training. We prioritize direct comparisons with S-GRPO and SAGE-RL, as they are more closely related to our rollout-construction setting.

AuthorsOpenReview ↗

Response to Reviewer 3Cn4 [2/2]

W3&Q2: Consistency across independent training seeds

We agree that independent training seeds are important for establishing the robustness of small accuracy differences. Our current evaluation uses multiple decoding seeds but a single training seed per method. Due to limited computational resources, conducting multiple independent training runs remains challenging. Nevertheless, we will make every effort to include additional training seeds during the rebuttal period. Meanwhile, the larger improvement on Qwen3.5-4B and the consistent efficiency gains provide encouraging evidence, although cross-training-seed variability remains to be assessed.

Q1: Can LRS distinguish harmful repetition from useful verification?

As clarified in Sec. 4.2, LRS is a lightweight diagnostic of recurrent token-level behavior, not a semantic classifier that perfectly separates harmful repetition from useful verification.

To evaluate its practical utility, we additionally compare LRS (threshold=8) with WordSaladChopper (WSC, EMNLP'25), whose original classifier-based approach we adapted and retrained for Qwen3.5-4B. Both methods use identical original trajectories and recovery settings.

DatasetTAcc. WSC/LRS (%)Length WSC/LRSTrigger WSC/LRS (%)
GSM8K0.677.86 / 77.942,851 / 2,7320.15 / 91.28
GSM8K0.079.53 / 79.763,065 / 2,5601.36 / 92.19
MATH-5000.687.80 / 88.0012,273 / 10,6322.40 / 54.00
MATH-5000.066.00 / 68.8015,769 / 10,12511.20 / 77.00
GPQA0.664.14 / 66.6721,885 / 19,1268.59 / 38.89
GPQA0.051.01 / 52.5323,389 / 12,96629.80 / 82.32
AIME250.656.25 / 58.3325,814 / 26,4710.00 / 32.50

Despite triggering substantially more frequently, LRS achieves higher accuracy in all seven settings and shorter final responses in six, while preserving or improving accuracy relative to the original responses. This supports LRS as a useful intervention signal, although it does not establish semantic detection precision.

Furthermore, our existing recovery/harm analysis (Appendix G, Fig. 5) shows that repetition-aware recovery provides more stable recovery quality and lower harm rates. We have expanded the discussion in Appendix J.


We hope that our additional experiments and clarifications have adequately addressed your concerns and strengthened the evidence supporting our contributions. We sincerely appreciate your thoughtful feedback and hope you will kindly reconsider your assessment in light of these updates.

AuthorsOpenReview ↗

Additional Qwen3.5-9B Training Results (W1)

To further address your concern about the practical relevance of repetition-aware training in larger models, we have now completed REFRAIN training on Qwen3.5-9B (70 steps). We evaluate the trained model under a 16K token budget with T=0.6 (mean@4), using the same evaluation protocol.

BenchmarkBase Acc. (%)Base TokensREFRAIN Acc. (%)REFRAIN Tokens
HMMT2615.9113,88930.3014,283
AIME2520.8315,69757.5010,119
AIME2615.8315,94266.679,379
AMC2358.7513,88996.884,238
Average27.8314,85462.849,505

Compared with the base model, REFRAIN improves average accuracy by 35.01 percentage points while reducing average response length by 36.0%. Accuracy improves substantially across all four benchmarks.

Combined with our earlier observation that incorrect trajectories exhibit larger LRS values in 22/28 comparable AMC23 problems on Qwen3.5-9B, these results provide encouraging evidence that LRS-guided recovery remains promising beyond the smaller model scales evaluated in our original submission and demonstrate that REFRAIN can be successfully trained at the 9B scale, achieving substantial accuracy improvements alongside shorter average responses.

We sincerely appreciate your constructive feedback and hope these additional results help address your concerns regarding the scalability and practical relevance of our method.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Manuscript Revisions

We sincerely thank the AC and reviewers for their constructive feedback. During the rebuttal period, we conducted additional experiments and revised the manuscript to address the main concerns regarding empirical comparisons, generalization, robustness, and methodological clarity.

1. Stronger baselines and practical effectiveness. We added comparisons with S-GRPO, SAGE-RL, and WordSaladChopper (WSC). REFRAIN outperforms S-GRPO by 4.33 pp in accuracy and matches SAGE-RL's accuracy with 13.2% fewer tokens and 58.7% lower training step time. Compared with WSC under matched recovery settings, LRS achieves higher accuracy in all seven evaluated settings.

2. Generalization to larger models and non-math tasks. We extended REFRAIN training to Qwen3.5-9B, achieving +35.01 pp accuracy and 36.0% fewer tokens relative to the base model. This complements our original Qwen3.5-4B results (+4.21 pp and 13.2% fewer tokens over GRPO). Additional HumanEval+ experiments demonstrate competitive code-generation accuracy with efficiency gains, particularly on Qwen3.5-4B.

3. Robustness across independent training seeds. We conducted three independent training runs for GRPO and REFRAIN on Qwen3-1.7B. REFRAIN consistently reduces generation length across all seeds (8.9% on average) while maintaining comparable accuracy (42.15% vs. 41.97%).

4. Reliability of repetition-aware intervention. We conducted a 90-span model-assisted annotation study with independent human adjudication. Harmful repetition accounts for 86.7% of LRS-selected spans versus 33.3% of length-matched random spans. We further analyzed 759 trajectories using ROSCOE and ReasonEval as complementary automated references.

5. Mechanism validation and manuscript corrections. We added a controlled trajectory-credit baseline, showing that Trie Credit improves accuracy by 1.32 pp while reducing tokens by 8.0%. We also clarified the inverse-occurrence weighting mechanism, corrected its derivation, expanded the ablations and WSC analysis, clarified the recovery budget and training-time measurement, and strengthened the related work.

Overall, these additions strengthen the evidence for REFRAIN's central contribution: using repetition structure to construct informative recovery rollouts and improve the accuracy–efficiency trade-off in reasoning RL. LRS serves as a lightweight structural proxy rather than requiring perfect semantic classification of harmful repetition.

We appreciate the reviewers' valuable suggestions and hope these additional experiments and revisions assist the AC in assessing our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.