acceptodds
Under review as a conference paper at ICLR 2027

PrefixRLVR: Mitigating Catastrophic Repetition in Long-Context RAG

12 views

Abstract

Abstract Modern end-to-end agentic workflows rely on robust long-context generation for multi-step autonomous tasks. However, sustained long-form outputs make LLMs prone to catastrophic repetition, which is particularly evident in open-source models with smaller parameter sizes, resulting in wasted compute and stalled downstream pipelines. Prior work mitigates repetition through decoding strategies, attention interventions, or training-based methods without assessing their impact on general capabilities, leaving the risk of catastrophic forgetting unaddressed. In this work, we introduce De-Rp, accompanied by a character-level detector that precisely localizes the onset of catastrophic repetition. Through systematic analyses across a range of open-source models, we find that catastrophic repetition is not merely a random decoding failure but can be induced by specific trigger prefixes that are shared across models of different series and scales. Hidden-state analysis reveals a repetition-associated signal near the prefix boundary that is shared across models. Building on this insight, we propose PrefixRLVR, a reinforcement-learning-based approach that combines training from these high-risk prefixes with length-aware advantage scaling. Experimental results show that PrefixRLVR reduces catastrophic repetition rates by approximately 49–330× across the evaluated backbones while preserving general model capabilities. Further long-context evaluations demonstrate consistent generalization gains, suggesting that PrefixRLVR offers an effective path toward more reliable long-context generation. Code and data are released at: https://anonymous.4open.science/r/PrefixRLVR.

Primary area:
generative models
Keywords:
Reinforcement Learning, Reward Sparsity, Catastrophic Repetition, Long-Context RAG

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nCBzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies catastrophic repetition in long-context retrieval-augmented generation. It introduces DE-RP, a stress benchmark containing 731 failure-enriched long-context RAG prompts, together with a detector that identifies periodic word-level repetition and retrospectively returns a character-offset boundary. Based on this detector, the authors observe that prefixes preceding detected repetition substantially increase the probability of subsequent repetition. They further report that prefixes mined from one model can induce failures in several other model families and that repetition-associated hidden-state signals become prominent near the detected prefix boundary. The proposed PrefixRLVR method combines two components. Prefix-Guided Rollout (PGR) conditions RLVR rollouts on mined high-risk prefixes, increasing the probability that a rollout group contains both positive and negative verifier outcomes. Length-Aware Advantage Estimation (LAAE) rescales group-relative advantages according to continuation length in order to discourage a shortcut in which the policy avoids repetition by terminating early.

Across Qwen3-0.6B, Qwen3-4B, Qwen3-8B, and Llama-3.1-8B, PrefixRLVR substantially reduces detector-defined repetition on DE-RP. Compared with prefix-conditioned GRPO alone, LAAE also reduces the observed shortening of non-repetitive responses. The paper additionally reports results on standard reasoning and instruction-following benchmarks, RULER, LaRA, LongWeave, and YaRN-extrapolated 64K/128K DE-RP variants.

Critical strengths
  1. The method produces large and consistent anti-repetition gains on the evaluated stress benchmark. PrefixRLVR reduces DE-RP repetition from 9.90% to 0.03% on Qwen3-0.6B, from 3.44% to 0.07% on Qwen3-4B, from 4.10% to 0.03% on Qwen3-8B, and from 14.16% to 0.09% on Llama-3.1-8B. The models are evaluated using the original prompts rather than trigger-prefixed inputs, so the gains are not restricted to the exact conditioning format used during training. The continuation experiments and mining comparison also provide meaningful behavioral evidence that small-model failure prefixes can be useful for other models.

  2. The paper identifies an important optimization shortcut and presents evidence that LAAE moderates it. Prefix-conditioned GRPO without LAAE reduces repetition but shortens non-repetitive outputs by approximately 35–40% on several backbones. On Qwen3-4B, adding LAAE changes the reported length shift from −40.3% to +0.7% while retaining a very low repetition rate. The binary-gate, ungated-scaling, and positive-only ablations exhibit substantially larger positive or negative length drift. This is consequential because an anti-repetition method is of limited value if it mainly learns to terminate before completing the answer.

Critical weaknesses
  1. The DE-RP quality study evaluates only 100 Qwen3-4B queries for which both Base and PrefixRLVR already produce non-repetitive answers. This clean-clean selection is a useful conditional sanity check, but it excludes cases in which the method changes the repetition outcome and therefore does not measure end-to-end utility over the test population. Matching the mean length of non-repetitive answers also does not establish completeness, relevance, faithfulness, or correct citation use. Table 4 presents a more serious comparison issue. Several LaRA and LongWeave Base scores marked with † or ‡ come from the original benchmark papers, whereas PrefixRLVR scores are obtained from the authors’ local pipeline. These results may differ in checkpoint revision, prompt formatting, decoding, data processing, or LLM-judge version. In the LongWeave rows, the published Overall score and the locally measured repetition statistics also appear to come from different generations, so their relationship cannot be interpreted directly. Several reported LaRA improvements are extremely small, such as 46.45 to 46.50, and no uncertainty is provided. Consequently, Table 4 supports the conclusion that repetition robustness transfers to some long-context settings, but it does not yet establish a general improvement in long-context capability.

  2. The LAAE component ablations are useful, but detailed comparisons are concentrated on Qwen3-4B and do not fully compare the proposed transformation with a simple, well-specified length or EOS correction under the same training and quality-evaluation budget. The reward-side correction shown in Figure 4 is not reported with the same final repetition, length, and answer-quality metrics as the full method. The LZ comparison uses one fixed configuration rather than characterizing the quality–length–repetition trade-off. The related-work framing also needs correction. Tsai et al. (2008), cited in support of entropy-based repetition evaluation, is a medical image-quality paper and is unrelated to text repetition. The statement that prior repetition-mitigation work does not assess capability preservation is too broad: LZ Penalty explicitly evaluates capability preservation, Xu et al. (2022) evaluates perplexity and generation quality after repetition-focused training, and Dong et al. (2025) evaluates downstream code-generation benchmarks. Similarly, the claim that prior work typically intervenes only after repetition becomes observable overlooks training-based approaches such as DITTO. These issues do not invalidate PrefixRLVR, but they overstate the novelty and motivation in its current form.

Questions for authors
  1. Since the paper states that local Base evaluations were already conducted, why are published Base scores used for several LaRA and LongWeave comparisons in Table 4? Please provide local Base scores using exactly the same checkpoint revision, prompting, decoding, data processing, and judge as PrefixRLVR. Are LongWeave Overall, Rep., and n-gram values computed from the same generated outputs for both variants?
  2. Can the authors report quality measurements on an unconditioned or randomly sampled DE-RP test population, including cases in which only one model produces a repetitive response? Per-dimension results, paired uncertainty, and evaluation of completeness and faithfulness would clarify whether repetition reduction preserves useful answers.
  3. How many independent training runs were performed for each method? Please report raw repetition-failure counts, prompt-clustered confidence intervals, and training-run variability for the principal results. For general benchmarks, are the observed changes larger than normal evaluation or checkpoint variance?
  4. What exactly is the reward-side length correction labeled GRPO+RP in Figure 4, how was it tuned, and what are its final repetition, length, and answer-quality results? A compute-matched comparison with a simple EOS/minimum-length or length-distribution correction would materially affect my assessment of LAAE.
  5. How were the 200 human-evaluation samples selected, did they include outputs from post-training models, and what was the annotator agreement before discussion? How sensitive are the main results to the exclusion of periods 2–5 and to alternative detector thresholds?
  6. Please replace the unrelated Tsai et al. image-quality citation, qualify the claims about prior capability evaluation and post-onset intervention, and update incomplete bibliographic information. How does PrefixRLVR differ empirically from training-based anti-repetition methods such as DITTO under a comparable evaluation?
3 comments

As the author have not submitted the rebuttal, I will keep my initial score.

Thanks for the detailed comments and for the recognition of our work. We have also run several additional experiments. Our point-by-point responses are below. All new results will be added to the revision, some overstated claims will be corrected, and the revised PDF will be uploaded before the end of the discussion period.

Q1. We originally used the published scores as a conservative choice. These scores were produced by the benchmark authors with the official pipelines, and our local Base scores are lower than the published ones in 4 of 5 cases. In the original paper, the Base Overall score was taken from the original benchmark paper, while Rep. and n-gram were computed from our local generations. This mixed two evaluation pipelines, which was our oversight. Therefore, in the revision, all Base scores in Table 4 are replaced by local results, using the same checkpoint, prompt, decoding, data processing, and judge as PrefixRLVR.

Numbers (published Base in the original paper / local Base / PrefixRLVR):

  • Llama-3.1-8B, LaRA-LC 32K / 128K: 66.58 / 55.10 ; 65.48 / 53.82 ; 67.94 / 55.57
  • Llama-3.1-8B, LongWeave: 20.25 / 16.23 / 25.69
  • Qwen3-8B, LongWeave: 27.17 / 25.82 / 31.98
  • Qwen3-4B, LongWeave: 19.18 / 21.78 / 27.83

Q2. On Qwen3-4B, we used GPT-4o, the same judge as in the original paper, to score answers on five dimensions from 1 to 5 (the prompt will be included in the revision). Intervals are 95% CIs from a paired bootstrap over questions.

Random 100 questions, not filtered by repetition status (Base / PrefixRLVR / difference [95% CI]):

  • Relevance: 3.92 / 4.17 / +0.25 [0.10, 0.41]
  • Faithfulness: 2.82 / 3.11 / +0.29 [0.13, 0.46]
  • Completeness: 2.71 / 2.85 / +0.14 [0.02, 0.26]
  • Coherence: 3.67 / 4.28 / +0.61 [0.43, 0.80]
  • Citation quality: 2.63 / 3.04 / +0.41 [0.25, 0.58]
  • Overall: 3.15 / 3.49 / +0.34 [0.24, 0.44]

All five dimensions improve significantly, including completeness and faithfulness. There is no evidence that reducing repetition comes at the cost of informativeness.

Q3

Due to GPU compute constraints, each method was trained only once on each backbone, and the revision will state this limitation explicitly. The same recipe was trained independently on 4 backbones, with consistent results. Raw failure counts (731 prompts x 8 = 5,848 generations) with prompt-clustered 95% CIs (Base vs. PrefixRLVR):

  • Qwen3-0.6B: 579 (9.90% [8.93, 10.89]) vs. 2 (0.03% [0.00, 0.09])
  • Qwen3-4B: 201 (3.44% [2.72, 4.19]) vs. 4 (0.07% [0.02, 0.14])
  • Qwen3-8B: 240 (4.10% [3.21, 5.04]) vs. 2 (0.03% [0.00, 0.09])
  • Llama-3.1-8B: 828 (14.16% [12.48, 15.87]) vs. 5 (0.09% [0.02, 0.19])

Q4

  • P+GRPO+RP (reward-side correction): within each rollout group, we compute the mean and the population standard deviation of the lengths of all non-repetitive responses. A non-repetitive response whose length is below the mean minus k standard deviations has its reward changed from 1 to 0, and GRPO advantages are then computed from the modified rewards (no penalty is applied if a group has fewer than 2 non-repetitive responses). This baseline directly rewrites the verifier reward labels, whereas LAAE leaves the reward unchanged and only adjusts the magnitude of the advantages. The curve in Figure 4 of the original paper corresponds to k = 1.5: during training, the length kept drifting and all metrics were unsatisfactory, so no checkpoint was saved. For the rebuttal, we ran full training and evaluation with k = 2, with results shown below.
  • P+Dr.GRPO (aggregation correction): removes the per-response length normalization and the standard-deviation normalization in GRPO (Liu et al., 2025).

Both baselines use exactly the same prefix data, number of training steps, and other hyperparameters as PrefixRLVR. Results on Qwen3-4B (failures / repetition rate / length change / overall quality):

  • Base: 201 / 3.44% / - / 3.150
  • P+GRPO+RP (k = 2): 7 / 0.12% / -18.8% / 3.318
  • P+Dr.GRPO: 39 / 0.67% / -9.8% / 3.282
  • PrefixRLVR: 4 / 0.07% / +0.7% / 3.490

The reward-side correction is very sensitive to the threshold: with k = 1.5 the length grows out of control, and with k = 2 responses are still 18.8% shorter, with both repetition and quality worse than PrefixRLVR. Dr.GRPO removes most of the shortening (from -40.3% to -9.8%), but its repetition rate is about 10 times that of PrefixRLVR, and its quality is lower by 0.21 [0.11, 0.31]. Only LAAE achieves low repetition, unchanged length, and higher quality at the same time.

Q5

  • Samples: randomly drawn from Qwen3-0.6B Base outputs on DE-RP and labeled by humans until there were 100 repetitive and 100 non-repetitive samples, then compared with the detector results. Both sampling and labeling were independent of the detector.
  • Agreement: before discussion, only 2 of the 200 samples were labeled inconsistently (99.0% agreement), and both were finally judged repetitive.
  • Post-training models: the human-evaluation samples do not include outputs from post-trained models. To rule out the possibility that the policy exploits blind spots of the detector, we ran a sensitivity analysis on the 5,848 Qwen3-4B generations. After relaxing the rule to 3 occurrences, Base failures increase from 201 to 225, while PrefixRLVR stays at 4. When detecting only the excluded periods 2 to 5, Base has 24 failures and PrefixRLVR has 0.

Q6

Our revision will remove the unrelated references, complete the bibliographic information, and qualify some overstated claims.

DITTO: we reproduced DITTO on Qwen3-4B (official "nl" loss, mixed 50/50 with MLE). A repeated-sentence probe shows that the implementation is correct: the probability of a repeated sentence decays with each repetition (0.689, 0.335, 0.168, 0.084), consistent with the trend in the original paper. On DE-RP, however, its repetition rate is 24.23% (1417/5848), higher than Base, and its overall quality is 2.294, also lower than Base. We believe the reason is that DITTO learns from offline synthetic sentence-level repetitions, whereas loops in long RAG outputs mostly occur at the level of lists, citations, and sub-sentence spans, and come entirely from real samples rather than constructed ones. PrefixRLVR instead optimizes directly on high-risk prefixes generated by the model itself, and therefore generalizes better.

Reviewer_rxCVOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PrefixRLVR to mitigate catastrophic repetition in long-context generation. It identifies transferable trigger prefixes that increase repetition risk across different LLMs and trains models on these high-risk states using reinforcement learning with verifiable rewards. The method combines prefix-guided rollout to improve training efficiency with length-aware advantage estimation to prevent models from avoiding repetition by generating shorter responses. Experiments show a 49–330× reduction in catastrophic repetition rates while largely preserving general capabilities and improving performance on long-context tasks.

Critical strengths
  1. The paper found that specific prefixes can significantly increase the probability of catastrophic repetition in different series and different scale models, and provided further evidence through hidden state analysis, providing a basis for training for high-risk states.
  2. PrefixRLVR combines prefix-guided training and length-aware advantage estimation, reducing the catastrophic repetition rate by approximately 49-330 times, and maintaining the original performance on benchmarks such as inference, encoding, and instruction following. The experimental results are quite convincing.
Critical weaknesses
  1. The method mainly involves extracting the easily triggering repetitive prefixes from the existing repetitive cases. However, the model may exhibit other repetitive patterns when dealing with new tasks or different prompts. Can this method handle the repetitive situations that were not encountered during training?
Questions for authors
  1. Does a decrease in repetition truly indicate an improvement in the quality of the generated content? Besides the repetition rate and the length of the response, can we further evaluate the completeness, correctness, and information content of the response? In particular, we need to rule out the possibility that although the model no longer repeats, the generated content becomes insufficient.
  2. The paper mainly explores the triggering prefixes from the existing repetitive cases. If different types of repetitive loops occur during the test, or if a different prompt word and task are used, will the method still be effective?
1 comment

We sincerely appreciate your recognition of our work. We have added several validation experiments. All new results will be included in the revision, and the revised PDF will be uploaded before the end of the discussion period.

For Q1:

On Qwen3-4B, we used GPT-4o, the same judge as in the original paper, and kept the original five-dimension evaluation framework (relevance, faithfulness, completeness, coherence, and citation quality, each scored from 1 to 5). We refined the scoring criteria: each dimension has clear anchors for scores 1, 3, and 5, and the judge is explicitly told not to give higher scores for length, formatting, or citation count by themselves. Intervals are 95% CIs from a paired bootstrap over questions.

Random 100 questions, not filtered by repetition status (Base / PrefixRLVR / difference [95% CI]):

  • Relevance: 3.92 / 4.17 / +0.25 [0.10, 0.41]
  • Faithfulness: 2.82 / 3.11 / +0.29 [0.13, 0.46]
  • Completeness: 2.71 / 2.85 / +0.14 [0.02, 0.26]
  • Coherence: 3.67 / 4.28 / +0.61 [0.43, 0.80]
  • Citation quality: 2.63 / 3.04 / +0.41 [0.25, 0.58]
  • Overall: 3.15 / 3.49 / +0.34 [0.24, 0.44]

All five dimensions improve significantly, including completeness and faithfulness. This shows that the information content and correctness of the answers do not drop when repetition is reduced.

21 questions where Base repeats: PrefixRLVR improves the overall score by +0.94 [0.71, 1.18], with 19 wins, 1 tie, and 1 loss, and none of the 21 answers is repetitive. In other words, answers that used to fall into a loop not only become normal answers, but also useful ones.

On the possibility of "no repetition, but less content": this can indeed happen, for example with P+GRPO, which does not use LAAE; LAAE solves this problem well. P+GRPO, trained with prefixes only, shortens answers by 40.3%, and its completeness (2.60) is even lower than that of Base (2.71). PrefixRLVR keeps the length almost unchanged (+0.7%), and its completeness is higher than that of P+GRPO by +0.25 [0.16, 0.35]. On LongWeave, which requires long outputs, the gap is even clearer: P+GRPO scores 18.92, lower than Base (21.78), while PrefixRLVR scores 27.83. This is exactly what LAAE does: it stops the model from avoiding repetition by saying less, and so keeps the answers complete.

For Q2:

Thank you for raising this question about generalization. We answer it from three angles.

(1) Different prompts. On Qwen3-4B, we used two additional prompts that differ substantially from the training template. Prompt A uses a role setting and requires a citation in every sentence. Prompt B asks for a report with a fixed four-part structure. We sampled 8 times per prompt, for 5,848 generations in total (Base failures (rate) / PrefixRLVR failures (rate) / PrefixRLVR length change):

  • Original template: 201 (3.44%) / 4 (0.07%) / +0.7%
  • Prompt A: 185 (3.16%) / 5 (0.09%) / +5.2%
  • Prompt B: 94 (1.61%) / 3 (0.05%) / -1.3%

With the new prompts, PrefixRLVR still has only 3 to 5 failures, almost the same as with the original template, and its confidence intervals do not overlap with those of Base at all. The n-gram repetition rate also drops from about 14% to about 6%. Compared with Base under the same prompt, the length changes by +5.2% and -1.3%, so there is no shortening. This directly shows that PrefixRLVR does not rely on the training template.

(2) Different tasks. Both the tasks and the prompts in LongWeave are completely different from our training data. On LongWeave, PrefixRLVR reduces the repetition rate by 60.3% to 87.6% and improves Overall by 23.9% to 58.3% over the local Base. In addition, on long-context tasks unrelated to repetition, such as RULER and LaRA, PrefixRLVR maintains or slightly improves performance.

(3) Repetition patterns not seen in training. Our training setup already tests this:

  • All training prefixes come from Qwen3-0.6B, while Qwen3-4B, Qwen3-8B, and Llama-3.1-8B show their own repetition loops at test time, which never appear in training.
  • Test queries do not overlap with training queries, and no prefix is added at test time; we use the original prompts directly.
  • All models are trained only on 32K inputs, yet on longer 64K and 128K inputs, the repetition rate still drops clearly.

We believe this generalization comes from the way PrefixRLVR is trained: it does not memorize any specific repeated text, but learns how to keep generating normal content from high-risk states where the model itself is likely to enter a loop.

Thank you again for your recognition and constructive comments. These additional experiments make the conclusions of our paper more complete.

Reviewer_YycpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies catastrophic repetition in long-context RAG. It introduces the DE-RP and a detector for locating repetition onset, and finds that prefixes preceding these failures can induce repetition across models. PrefixRLVR uses prefixes mined from a small model for GRPO rollouts, together with length-aware advantage scaling to mitigate the shortcut of producing shorter answers. Experiments on Qwen and Llama show substantial reductions in repetition on unprefixed test prompts, maintained performance on general benchmarks, and improvements on several long-context evaluations.

Critical strengths

Simple P+GRPO clearly shortcuts by truncating outputs to avoid looping. The LAAE term largely fixes this response-shortening hacking.To some extent, the LongWeave gains show that preserving length translates into genuine downstream quality rather than just gaming the repetition score.

Critical weaknesses
  1. The abstract and introduction suggest that earlier methods did not examine their impact on general capabilities. However, LZ Penalty already evaluates repetition alongside GPQA and AIME, reporting capability results.

  2. It is still unclear whether the recovered answers are actually good. In Appendix C.4, the quality check only looks at prompts where both the base and fine-tuned models were already non-repetitive, completely skipping the exact cases where the method is supposed to help most: when the base model loops and the trained model recovers.

Questions for authors
  1. The existing baselines are useful, but they leave the advantage over simpler training alternatives unclear. P+GRPO already achieves similarly low repetition rates, so the main benefit of LAAE appears to be its effect on response length and quality. A matched comparison against an existing length-bias correction would help establish whether the additional design is necessary. The clean-response SFT baseline also does not represent training methods that directly penalize repetition. Testing one such alternative on Qwen3-4B, with comparable data and training budget, would make the contribution more convincing.

  2. Could the authors report the locally evaluated LongWeave baseline scores under the same setup as the trained models? How do these results compare with the published baselines used in Table 4?

3 comments

We appreciate for the careful review and for recognizing that LAAE corrects the length-shortening shortcut of P+GRPO. To address the remaining concerns, we conducted several new experiments, all of which will be included in the revision. We agree that LZ Penalty does assess capability preservation on several benchmarks; our evaluation covers a broader range of benchmarks and settings. We will revise and qualify the overstated claims, and the revised PDF will be uploaded before the discussion period ends.

For W2:

We scored Qwen3-4B answers with GPT-4o, the same judge used in the paper. We kept the original five-dimension framework (relevance, faithfulness, completeness, coherence, and citation quality, each on a 1-to-5 scale) and refined the scoring criteria: each dimension now has explicit anchors for scores 1, 3, and 5, and the judge is instructed not to reward length, formatting, or citation count by themselves. The full judge prompt will be added to the revision. All intervals below are paired-bootstrap 95% CIs over questions.

Questions where Base falls into a loop (21 questions), which is exactly the case the highlighted (Base / PrefixRLVR / gain [95% CI]):

  • Relevance: 3.05 / 3.90 / +0.86 [0.43, 1.33]
  • Faithfulness: 2.43 / 2.95 / +0.52 [0.19, 0.81]
  • Completeness: 2.52 / 2.90 / +0.38 [0.00, 0.76]
  • Coherence: 1.86 / 4.00 / +2.14 [1.81, 2.43]
  • Citation quality: 2.10 / 2.90 / +0.81 [0.33, 1.19]
  • Overall: 2.39 / 3.33 / +0.94 [0.71, 1.18]

PrefixRLVR wins on 19 of these 21 questions (1 tie, 1 loss), and every one of its 21 answers is free of repetition. In other words, PrefixRLVR does not merely break the loop: the recovered answers are also significantly more relevant, more faithful, and better cited.

On a random sample of 100 questions, with no filtering by repetition status, PrefixRLVR raises the overall score from 3.15 to 3.49 (+0.34 [0.24, 0.44]), with significant gains on all five dimensions.

For Q1:

(a) Length-bias corrections. We added two length-correction baselines trained under exactly the same conditions as PrefixRLVR (same prefix data, number of steps, and other hyperparameters):

  • P+GRPO+RP (reward-side correction): in each rollout group, a non-repetitive response shorter than the mean length of the non-repetitive responses minus k standard deviations receives reward 0 instead of 1. The curve in Figure 4 used k = 1.5; its length drifted throughout training and no checkpoint was kept, so we additionally trained with k = 2.
  • P+Dr.GRPO (aggregation correction): GRPO without the per-response length normalization and without the standard-deviation normalization (Liu et al., 2025).

Qwen3-4B results (failures out of 5,848 / repetition rate / length change vs. Base / overall quality / completeness):

  • Base: 201 / 3.44% / - / 3.150 / 2.71
  • P+GRPO: 2 / 0.03% / -40.3% / 3.332 / 2.60
  • P+GRPO+RP (k = 2): 7 / 0.12% / -18.8% / 3.318 / 2.76
  • P+Dr.GRPO: 39 / 0.67% / -9.8% / 3.282 / 2.79
  • PrefixRLVR: 4 / 0.07% / +0.7% / 3.490 / 2.85

Although P+GRPO also reaches a very low repetition rate, PrefixRLVR delivers higher overall quality (+0.16 [0.08, 0.24]), and the largest gap is in completeness (+0.25 [0.16, 0.35]). P+GRPO is in fact slightly less complete than Base (-0.11 [-0.23, 0.01]): the shortcut of producing shorter answers is paid for in missing content. On LongWeave, which requires long outputs, this cost becomes much larger: P+GRPO scores 18.92, below the local Base (21.78), whereas PrefixRLVR reaches 27.83.

Neither standard length correction closes this gap. The reward-side correction is fragile with respect to its threshold: k = 1.5 causes uncontrolled length growth, while k = 2 still shortens responses by 18.8%. Dr.GRPO removes most of the shortening, but it leaves about 10 times more repetition failures than PrefixRLVR and lower quality (-0.21 [0.11, 0.31]). PrefixRLVR is the only method that simultaneously achieves near-zero repetition, preserved length, and the best answer quality.

(b) A training method that directly penalizes repetition. We reproduced DITTO on Qwen3-4B (official "nl" loss, mixed 50/50 with MLE), using the same training corpus as the SFT baseline. A repeated-sentence probe confirms that the reproduction behaves as intended: the probability of a repeated sentence decays with each copy (0.689, 0.335, 0.168, 0.084), matching the trend in the DITTO paper. On DE-RP, however, DITTO reaches a repetition rate of 24.23% (1417/5848), higher than Base, and an overall quality of 2.294, also below Base. We attribute this to a mismatch in training signal: DITTO learns from offline, synthetic copies of whole sentences, whereas loops in long RAG outputs mostly arise in lists, citations, and sub-sentence spans, and come entirely from real model generations rather than constructed data. PrefixRLVR trains directly on high-risk prefixes produced by the model itself, which is why it generalizes better.

For Q2:

The original paper used the published scores as a conservative choice: they were obtained by the benchmark authors with the official pipeline, and most of our local Base scores are lower. In the revision, all Base scores in Table 4 are replaced with local results obtained with the same checkpoint, prompt, decoding, data processing, and judge as PrefixRLVR.

LongWeave Overall (published Base / local Base / PrefixRLVR):

  • Llama-3.1-8B: 20.25 / 16.23 / 25.69
  • Qwen3-8B: 27.17 / 25.82 / 31.98
  • Qwen3-4B: 19.18 / 21.78 / 27.83

Against the local Base, PrefixRLVR improves LongWeave Overall by 23.9% to 58.3%.

Thanks for the reply. After comprehensive consideration, I am still concerned that the existing evidence cannot guarantee the strength of the conclusions, so I have decided to maintain the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.