- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper detects textual-repetition loops by comparing top-k candidate sets at historical positions that emitted the same token. Experiments across 25 models evaluate detection, while interventions on nine models demonstrate reduced loops and token consumption under greedy decoding.
Critical strengths
The evaluation covers multiple model families and includes controls for generation length, token counts, and repetitive successful generations. Reporting answer retention alongside accuracy helps assess intervention harm.
Critical weaknesses
-
The contribution primarily establishes an empirical association between candidate-set recurrence and textual loops. The experiments mainly identify which signal recognizes established repetition, without sufficiently investigating the origin of this association or providing deeper insights into loop formation and persistence. The scientific contribution therefore remains limited.
-
Compared with text-counter force-close, accuracy improves by only 0.33 percentage points, with essentially the same token consumption. The large headline savings are relative to no intervention, while differing history-reset policies further complicate attribution of the incremental benefit.
-
End-to-end intervention benefits are demonstrated only under greedy decoding. Under recommended sampling, loops are substantially rarer and early detection deteriorates; intervention outcomes under sampling and computational overhead relative to the text counter remain insufficiently evaluated.
Questions for authors
-
What further evidence explains the source of candidate-set recurrence and its relationship to loop formation or persistence?
-
Does the advantage over text-counter force-close persist with matched operating points and identical history-reset policies?
-
Under recommended sampling, what are the net effects on accuracy, answer retention, token consumption, and runtime?
3 comments
Response to Weakness 2 and Question 2: Token savings with matched history handling
Thank you for highlighting the need to compare the two methods under the same history-handling policy.
With matched operating points and identical history handling, Jac still used fewer generated tokens than the text counter. With history retained and with history reset, it generated 0.48M (95% interval 0.14–0.80M) and 0.32M (0.04–0.59M) fewer tokens, respectively. Jac had a slightly higher PASS rate under both history settings, but the 95% intervals for the differences included zero. Retention was at least 99.8% in every condition.
As you noted, the paper used different history handling for the two methods: Jac retained detector history, whereas the counter (K = 10) reset it. To examine the effect of this difference, we evaluated both methods with history retained and with history reset. Each setting had to follow its no-intervention counterpart up to its first detection, so we used two models whose outputs are reproducible regardless of batching: DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B. Under retained history, the detector continues using its accumulated state after each insertion: Jac's same-token candidate-set history and moving average, or the counter's occurrence counts of 30-token spans. Under reset history, that state is cleared whenever an insertion ends and accumulated again from subsequent generation. After detection, all four settings used the same force-close action: inserting the end-of-thinking delimiter and continuing answer generation.
The counter threshold was chosen on validation by scanning K = 2–30 and taking, as the representative, the smallest K whose number of false positives on correct trajectories does not exceed that of Jac (K = 15 and K = 13, respectively). This is the earliest-first-alert operating point on the same untreated trajectories among the K values satisfying the validation false-positive constraint. Every smaller K exceeded Jac's false positives on validation.
| Policy | History handling | PASS (%) ↑ | Retention (%) ↑ | Tokens (M) ↓ |
|---|---|---|---|---|
| Text counter | Retain | 64.32 | 99.81 | 9.04 |
| Text counter | Reset | 64.48 | 100.00 | 8.88 |
| Ours (Jac) | Retain | 64.99 | 99.91 | 8.56 |
| Ours (Jac) | Reset | 64.94 | 99.91 | 8.55 |
PASS and Retention are as in Table 1 of the paper, and the table pools the two models' test results (974 questions each, 1,948 generations in total). Intervals in the text are question-level 95% bootstrap intervals. Generated tokens include answer generation after force-close.
Jac's results were nearly unaffected by history handling (PASS 64.99% vs. 64.94%; 8.56M vs. 8.55M generated tokens), whereas the counter was slightly better on PASS, Retention, and generated tokens when its history was reset. Thus, in this experimental setting, Jac held the advantage over the counter regardless of history handling, generating fewer tokens while maintaining similar PASS and Retention.
Response to Weakness 3 and Question 3: Intervention outcomes and runtime under recommended sampling settings
Thank you for asking how the method performs under recommended sampling settings, including its computational cost.
Overall, the gains from stopping loops were modest under the recommended sampling settings because loops were rare. Detection and intervention nevertheless worked under these settings.
We evaluated Qwen3.5-4B on the 974 test questions three times each under sampling (2,922 outputs in total). The same seed was applied to all methods for each question and replicate, and there were 13 loop trajectories under no intervention.
| Policy | PASS (%) ↑ | LOOP (%) ↓ | Retention (%) ↑ | Token change (%) ↓ | Time change (%) ↓ |
|---|---|---|---|---|---|
| No intervention | 79.88 | 0.44 | 100.00 | — | — |
| Text counter (K = 10) | 79.91 | 0.17 | 99.70 | −0.54 | +1.55 |
| Ours (Jac) | 79.98 | 0.07 | 100.00 | −0.10 | +7.11 |
PASS, LOOP, and Retention are as in Table 1 of the paper, and token and time values are changes relative to no intervention. The items above are reported by pooling the three replicates. Jac uses the paper's setting, the text counter detects a trajectory when the same 30-token span has occurred 10 times, and both methods force-close while retaining detector history.
Jac retained all baseline-correct answers and detected 11 of the 13 baseline loop trajectories, 3 of which became correct. With the text counter, 4 became correct, but 7 baseline-correct answers were lost, 5 of which became loop trajectories.
Runtime was measured on four NVIDIA B300 GPUs with vLLM 0.28.0 and PyTorch 2.13.0 (CUDA 13.0), processing one request at a time per GPU. Each GPU ran the same subset of questions with all methods. Jac's runtime increased by about 7% relative to no intervention. Extracting candidate sets and comparing them with history adds computation at every decoding step, while rare loop trajectories leave little generation to save.
Response to Weakness 1 and Question 1: Study scope and empirical contribution
We sincerely thank you for the time and care you devoted to reviewing our paper, as well as for your thoughtful feedback and the specific concerns you raised. Your comments have helped us clarify the scope of our findings and identify key comparisons that warranted closer examination.
In this work, we did not investigate the underlying mechanisms of loop formation and persistence, nor the origin of candidate-set recurrence. Instead, our goal was to leverage next-token probability distributions to build a practical, on-the-fly loop detection algorithm and verify its generalizability across diverse models and tasks. To this end, we evaluated our method on 25 models and three datasets using a shared detector configuration. Furthermore, Appendix C demonstrates that our signal distinguishes loop trajectories from those that eventually yield correct answers, even after both have met the textual-repetition criterion—indicating that this separation captures more than a simple binary repetition check. This practical algorithm and its broad empirical validation constitute our core contribution, while we leave a deeper investigation into the underlying mechanisms to future work.