- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 3
Read review
Paper summary
This paper introduces REFRAIN to address the issue of repetitive loops in reinforcement learning (RL) reasoning. The method employs LRS (Loop-Removal Strategy) to identify truncation points and generate "recovery rollouts," and subsequently trains the model using "sign-preserving efficiency advantage" and "prefix-trie credit assignment." Experimental results demonstrate that this approach reduces both the number of generated tokens and the per-step training time, while maintaining or improving accuracy.
Critical strengths
The paper identifies a concrete and specific failure mode: models may become trapped in repetitive verification or cyclic generation even after successfully completing valid reasoning—a finding that goes beyond the simplistic assumption that "longer outputs are necessarily worse." The entropy pattern shown in Figure 1 provides meaningful diagnostic evidence for this phenomenon. The methodology is described comprehensively, encompassing formal analysis, algorithmic details, and an extensive set of ablation studies.
Critical weaknesses
-
LRS has not been validated as a reliable proxy for ineffective reasoning. The method lacks a minimum repetition length, semantic constraints, or confidence thresholds. Repeated use of formulas, necessary verification, and corrective recalculation may all trigger LRS. Appendix C provides only two successful examples, without human annotations or a false-positive analysis. I recommend conducting blind human annotation on complete reasoning trajectories and reporting precision, recall, and F1 for detecting harmful repetition.
-
The contribution of prefix-trie averaging has not been isolated. The Mask-All ablation in Table 8 removes all gradients from shared prefixes, demonstrating only that shared-prefix feedback should not be discarded, rather than that trie averaging is superior to standard trajectory-level credit assignment. A more appropriate ablation would be REFRAIN-TrajCredit, which keeps the rollouts, boundaries, hints, advantages, and computational budget unchanged, while replacing trie averaging with standard trajectory-wise credit assignment. Appendix E indicates that the two approaches differ only when mixed-sign advantages and PPO clipping occur simultaneously, yet the paper does not report how frequently such cases arise.
Questions for authors
1. Could the authors provide a human-annotated repetition evaluation that distinguishes harmful loops, necessary verification, and corrective recalculation, and report precision, recall, F1, and the intervention harm rate?
2. Could the authors include a standard trajectory-level credit assignment baseline and report the frequency of mixed-sign advantages at shared nodes and the proportion of cases where PPO clipping is triggered?
2 comments
Response to Reviewer K86g [2/2]
W2&Q2:Contribution of Trie Credit
To isolate the contribution of Trie credit assignment, we add REFRAIN-TrajCredit (Mask-Hint), which uses standard trajectory-level advantages while keeping all other training components unchanged.
| Method | Accuracy (%) ↑ | #Tokens ↓ |
|---|---|---|
| REFRAIN-Mask-All | 39.20 | 9,745 |
| REFRAIN-TrajCredit (Mask-Hint) | 41.15 | 9,615 |
| REFRAIN (Trie Credit) | 42.47 | 8,844 |
Compared with TrajCredit, Trie credit improves accuracy by 1.32 percentage points while reducing response length by 8.0%. This provides direct evidence for the benefit of inverse-occurrence weighting. Mask-All further shows that simply removing shared-prefix gradients is less effective. We have also incorporated Linear-Mask-Hint into the extended ablations in Appendix H (Tables 7 and 8), including its configuration, benchmark-wise results, and comparison with Mask-All and the default Trie Credit.
Mechanism analysis. We clarify that the difference between Trie and trajectory-level credit does not arise from PPO clipping, which is inactive in our training setting (train_batch_size = mini_batch_size = 32). Our implementation applies inverse-occurrence weighting to each trajectory occurrence:
where is the occupancy of the corresponding shared-prefix node. Consequently, the aggregate contribution of a shared node is scaled by its inverse occupancy, even without PPO clipping. This applies to all shared nodes, while mixed-sign downstream advantages represent a particularly important case of conflicting feedback.
In our TrajCredit training, 66.7% of training token occurrences belong to shared nodes, and 18.6% of shared-token occurrences correspond to nodes with mixed-sign downstream advantages (equivalent to 12.4% of all training tokens). The mean occupancy is comparable between TrajCredit and REFRAIN (1.503 vs. 1.526).
We have corrected the derivation in Appendix E and revised the related claims to reflect the actual inverse-occurrence weighting implementation (Section 3.3 Eq. 11/12, Algorithm 5). The previous derivation inadvertently introduced a summation before occupancy normalization, leading to an incorrect equivalence claim under the unclipped objective.
Q1: Human-annotated repetition evaluation
Thank you for this suggestion. We agree that human annotation would provide valuable evidence for distinguishing harmful repetition from useful verification and corrective recalculation.
Given our limited computational and human resources during the rebuttal period, we have prioritized additional empirical comparisons and ablation studies. In particular, we have compared LRS with WordSaladChopper (WSC, EMNLP'25), adapting and retraining its original classifier for Qwen3.5-4B. Under matched recovery settings, LRS achieves higher accuracy in all seven evaluation settings and shorter responses in six, supporting its practical effectiveness as an intervention signal.
We acknowledge that this comparison does not replace human-annotated evaluation of semantic detection precision. We will make every effort to conduct a small-scale human annotation study during the rebuttal period, including an analysis of harmful repetition, useful verification, and corrective recalculation.
We hope that our additional experiments and clarifications have adequately addressed your concerns and strengthened the evidence supporting our contributions. We sincerely appreciate your thoughtful feedback and hope you will kindly reconsider your assessment in light of these updates.
Response to Reviewer K86g [1/2]
W1: Reliability of LRS
As clarified in Sec. 4.2, LRS is a lightweight diagnostic of recurrent token-level behavior, rather than a perfect semantic classifier of harmful repetition versus useful verification.
Recovery and harm analysis. Our existing Appendix G (Fig. 5) compares REFRAIN with REFRAIN-Random, REFRAIN-Resample, and JET in terms of Recovery Rate and Harm Rate. REFRAIN generally achieves more stable recovery quality, with lower harm rates than its random and resampling variants and a greater ability to recover incorrect trajectories.
Comparison with a learned detector. We additionally compare LRS (minimum threshold = 8) against WordSaladChopper (WSC, EMNLP'25). We adapt and retrain WSC's original classifier-based implementation for Qwen3.5-4B. Both methods operate on identical original trajectories with matched recovery settings.
Despite triggering substantially more frequently, LRS achieves higher accuracy than WSC in all seven settings and shorter final responses in six, while preserving or improving accuracy relative to the original trajectories.
These results support LRS as a practical intervention signal, rather than establishing its semantic detection precision. Together with the recovery/harm analysis in Fig. 5, they suggest that repetition-aware intervention can mitigate unproductive reasoning without systematically degrading correctness.
We have expanded Appendix J with WSC reproduction details, matched intervention protocols, and complete accuracy, length, and trigger-rate results.