- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper identifies the problem of repetitive copying in large language models, where in long-context settings they copy text from the prompt in their reasoning traces. The paper argues that copying information from the context is only useful when that information overlaps with key evidence necessary for the question, and proposes a reward formulation that encourages copying relevant information and discourages copying irrelevant information. Experiments across models, context lengths, and tasks show improvements over accuracy-only rewards.
Critical strengths
The problem setting of excessive copying from the context is relevant and has implications for both performance and efficiency. The paper shows this behaviour across models and context length, and highlights how erroneously copying irrelevant information correlates with poor performance. The proposed reward shaping fix, GEAR, appears to improve performance and reduce reasoning length across tasks.
Critical weaknesses
The accuracy gains from GEAR reward shaping are small (1.5-4.6%), without any uncertainty estimates or variance. Due to the high variance in RL training, this small difference might not hold with repeated training, especially when considering the large impact of hyperparameters on GEAR performance (most beta/alpha changes drop accuracy to that of standard GSPO). In a related but smaller point: in multiple places there are claims of correlations and significance without corresponding statistical tests.
No other reward shaping methods are tested, making it unclear if other methods like length-based reward shaping or other evidence-based rewards would provide similar improvements. Methods like LongRLVR are cited in the related work but not compared against.
Questions for authors
-
Can you provide statistical tests or confidence intervals for the correlation and significance claims? For example: the relationship between overlap rate and accuracy (Figure 2, lines 206-215), and between grounding ratio and correctness (Figure 3, lines 251-257). This would help substantiate the claim of "significantly more selective copying" on lines 51-53.
-
Did you test GEAR on existing tasks without the evidence-annotations? In section 4.2 you propose a way to create new QA pairs with the evidence by construction, but I’m curious if the alternative mentioned on line 307 (post-hoc labeling which parts of a document are relevant) would allow you to apply GEAR to more existing datasets.
-
Are the RL results from one training run each, or averaged across multiple seeds?
4 comments
Response to Reviewer qjjk (2/2)
The results show that GEAR achieves the highest aggregate score among the tested methods at both context ranges. Under the same training data, backbone, and RL-update budget, the LongRLVR adaptation has lower aggregate scores than vanilla GSPO under both evaluation protocols. GEAR computes its reward directly from free-form reasoning without requiring an additional evidence-ID output protocol. In addition, we compare the methods' mean complete-response lengths on RULER ≤32k, as shown below (Appendix B.2, Table 8).
| Method | Avg. response length (tokens) |
|---|---|
| LongRLVR, adapted protocol | 2,438 |
| LongRLVR, ordinary protocol | 2,744 |
| Length penalty, λ=0.1 | 89 |
| Length penalty, λ=0.01 | 24 |
| GSPO | 2,853 |
| GSPO + GEAR | 2,185 |
On the one hand, training with length penalties directly causes the model's reasoning ability to collapse. On the other hand, GEAR still has the shortest mean response length among GEAR, GSPO, and LongRLVR, further indicating that GEAR achieves a better balance between output length and task performance.
Q3. Statistical tests and confidence intervals (Original question 1)
Can you provide statistical tests or confidence intervals for the correlation and significance claims? For example: the relationship between overlap rate and accuracy (Figure 2, lines 206-215), and between grounding ratio and correctness (Figure 3, lines 251-257). This would help substantiate the claim of "significantly more selective copying" on lines 51-53.
A3. We have added Spearman correlation tests for the figures, using the revised results for Figure 2a. The table below reports the correlation coefficient ρ and p value (Appendix B.8, Table 16).
| Two variables | Correlation coefficient ρ | p value |
|---|---|---|
| Input overlap and answer correctness (Figure 2a) | −0.42 | |
| Grounding ratio and answer correctness (Figure 3a) | +0.29 | |
| Input overlap and reasoning length (Figure 2b) | +0.27 |
The results show statistically significant correlations between the pairs of variables discussed above in the paper.
Q4. Existing QA data and post-hoc labels (Original question 2)
Did you test GEAR on existing tasks without the evidence-annotations? In section 4.2 you propose a way to create new QA pairs with the evidence by construction, but I’m curious if the alternative mentioned on line 307 (post-hoc labeling which parts of a document are relevant) would allow you to apply GEAR to more existing datasets.
A4. We have not conducted training experiments that retain existing QA pairs and add evidence annotations post hoc. Methodologically, GEAR's reward computation does not require questions to be generated through our data construction pipeline: once corresponding evidence annotations are available for existing QA pairs, they can be used to compute the GEAR reward. Post-hoc annotation could therefore extend GEAR to more existing datasets in principle.
We would like to emphasize that the main contributions of this paper are to identify and analyze repetitive copying in long-context reasoning and to propose an evidence-based reward design to improve reasoning behavior. To validate this design, we automatically construct training data with evidence annotations and demonstrate its effectiveness on long-context reasoning benchmarks.
Developing a reliable post-hoc evidence annotation pipeline for existing QA pairs is a valuable extension, whose annotation quality and training effectiveness require dedicated evaluation. This paper focuses on proposing and validating the GEAR reward design; a systematic comparison of such data extension pipelines is left to future work.
Q5. Number of independent RL runs (Original question 3)
Are the RL results from one training run each, or averaged across multiple seeds?
A5. The results in the original paper were obtained from a single training run per setting. We have added results for Qwen3.5-35B-A3B using three training seeds each for GSPO and GEAR; please see A1 and Appendix B.1 (Table 6) of the revised manuscript for details.
Thanks again for your constructive suggestions. Hope our explanations can ease your concerns. Please let us know if there is more to clarify.
I appreciate the additional experiments: in particular the three-seed RL experiments alleviate my concerns about training variance, and the statistical tests provide stronger support for the reported correlations. The additional reward-shaping baselines are also appreciated.
I have small concerns about the length-penalty baselines as they appear to produce extremely short responses (24-89 tokens on RULER), and the weaker penalty results in shorter responses than the stronger one. This suggests some training instability and that better performance may be possible with better hyper-parameters, although the setup may still be representative of a standard approach to length-penalty reward shaping.
Overall, the additional experiments have strengthened the empirical evidence for GEAR, and I am happy to increase my score from 2 to 3.
Thank you very much for raising the score! We agree with your observation about instability in length-penalized training. One possible reason may be that a simple length penalty provides a broad incentive to shorten responses, which may also penalize necessary, high-quality reasoning. However, GEAR was developed to address repetitive copying, a common issue in long-context models identified through our analysis. It therefore offers a more targeted objective for improving long-context capabilities, which can help make training more effective. We will investigate the causes of this instability and provide a more detailed analysis in future versions.
Thanks once again for your encouraging feedback. Have a great day!
Response to Reviewer qjjk (1/2)
Response to Reviewer qjjk
Thank you for recognizing the relevance of repetitive copying to long-context reasoning and the potential of GEAR to improve task performance while reducing reasoning length, and for your constructive suggestions on reward baselines, training variability, and statistical validation. We address your concerns point by point below.
Q1. Gain size, variability, and hyperparameters (Weakness 1)
A1. To address this concern, we have added three-seed experiments for GSPO and GEAR on Qwen3.5-35B-A3B and report the mean and sample standard deviation below (Appendix B.1, Table 6 of the revised manuscript).
GEAR improves the aggregate benchmark score over GSPO by 2.4 and 2.6 percentage points at ≤32k and full ≤128k, respectively. Taken together, the cross-seed means and sample standard deviations of both methods provide further empirical support for the reliability of GEAR's performance gains across the three evaluated seeds.
Regarding hyperparameter sensitivity, we acknowledge that poorly chosen α and β can induce reward hacking, for example by incentivizing indiscriminate copying to collect overlap rewards. However, our chosen hyperparameter setting yields higher aggregate scores than accuracy-only GSPO on all three model scales—Qwen3.5-9B, Qwen3.5-27B, and Qwen3.5-35B-A3B—at both ≤32k and full ≤128k. Together with the consistent gains in the new multi-seed experiments, we believe these results provide sufficient empirical evidence for GEAR's advantage.
We address the questions about correlations in A3.
Q2. Other reward-shaping baselines (Weakness 2)
A2. We have added LongRLVR and length-penalty reward baselines on Qwen3.5-35B-A3B, using the same training data, backbone, and RL-update budget as GEAR. For LongRLVR, we directly train the initial model using the method's format (chunked input and evidence-ID output), and evaluate it with both the adapted protocol and the ordinary benchmark protocol; the length penalties use λ=0.1 and λ=0.01, following the reward definition and token-counting convention in Appendix B.2 (Eq. 4). The comparison results are shown below (Table 7).