- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies answer anchoring in reasoning models: the tendency to repeat an earlier answer even when it is incorrect, despite extensive RLVR training. The authors introduce Context-Replay Gap, a token-level metric that measures how token log-likelihood changes when a previous response is added to the context. They then propose Context-Replay Guided Tuning (CRT), which applies negative log-likelihood (NLL) supervision only to tokens with the lowest gaps. Experiments on three reasoning model families and four mathematical reasoning benchmarks show consistent accuracy gains over standard SFT and other token-selection baselines. Behavioral analyses further suggest that CRT reduces reliance on previous answers while largely preserving sensitivity to prior reasoning.
Critical strengths
- Context-Replay Gap provides a novel way to quantify how previous responses influence token predictions. The resulting CRT method is straightforward to implement and improves reasoning accuracy through selective NLL supervision.
- The improvements are consistent across three model families and four mathematical reasoning benchmarks. The paper also includes extensive ablations and behavioral interventions to examine how the method works.
Critical weaknesses
- Context-Replay Gap measures the influence of the entire previous response, so it does not isolate the effect of the previous answer itself. It is therefore unclear whether selecting low-gap tokens specifically addresses answer anchoring or captures broader contextual effects.
- CRT improves accuracy but also produces substantially longer reasoning trajectories. The reported ablations do not fully separate the benefits of reduced answer anchoring from those of additional inference computation.
Questions for authors
- What evidence would help distinguish whether Context-Replay Gap captures answer anchoring specifically or broader contextual effects, such as changes in response length or reasoning behavior?
- Could you compare CRT with the baselines under matched inference token budgets? This would help clarify how much of the accuracy gain comes from longer reasoning.
- Since CRT applies NLL supervision to self-generated trajectories without correctness rewards, how do correct and incorrect trajectories each contribute to the gains? Would training only on correct trajectories yield similar results?
- CRT improves single-turn accuracy, but the additional gains from multi-turn self-correction appear relatively small. How should we interpret this finding in relation to the claim that CRT reduces answer anchoring, as opposed to improving reasoning ability more generally?
4 comments
(Q1). Context-Replay Gap measures the influence of the entire previous response, so it does not isolate the effect of the previous answer itself. It is therefore unclear whether selecting low-gap tokens specifically addresses answer anchoring or captures broader contextual effects.
(W1). What evidence would help distinguish whether Context-Replay Gap captures answer anchoring specifically or broader contextual effects, such as changes in response length or reasoning behavior?
-
Length-Controlled. We do not entirely rule out the possibility that CRT benefits from other factors. However, these reasoning-related factors are difficult to quantify. For the readily measurable effect of response length, our preceding rebuttal shows that CRT can improve reasoning accuracy while keeping inference length nearly unchanged.
-
Reduced Answer Copying. We further evaluate the length-controlled models from the preceding analysis under the two-pass setting. Even after controlling for length, the CRT variants retain a strong tendency to correct prior errors, achieving wrong-answer copying rates of 69.32 and 55.87, respectively. This suggests that CRT’s resistance to answer anchoring persists independently of increased inference length.
| Model | P1 Acc. | P2 Acc. | Δ Acc. | Copy | P1 Correct | Copy | P1 Wrong | P2 Correct | P1 Wrong |
|---|---|---|---|---|---|---|
| Base | 59.58 | 59.79 | +0.21 | 97.64 | 81.96 | 3.99 |
| no CRT,20k-length-RL, 156step | 61.67 | 61.25 | −0.42 | 97.47 | 83.97 | 2.99 |
| CRT, 18K-length-RL, 156step | 62.71 | 63.54 | +0.83 | 96.64 | 69.32 | 7.67 |
| CRT, 20k-length-RL, 314step | 63.65 | 64.27 | +0.63 | 95.09 | 55.87 | 10.32 |
- Answer vs. Reasoning. We agree that the raw context-replay gap may capture broad contextual differences, including response length and changes in reasoning behavior. To isolate answer anchoring more directly, we added a crossed 2$\times$2 intervention that independently varies the correctness of the previous reasoning and terminal answer while holding the question, Q-A1-Q structure, and evaluation protocol fixed. The results appear in Table 12.
| Constructed history | Base Acc. | CRT Acc. |
|---|---|---|
| Correct reasoning + correct answer | 88.82 | 82.89 |
| Correct reasoning + wrong answer | 59.21 | 70.39 |
| Wrong reasoning + correct answer | 51.32 | 38.82 |
| Wrong reasoning + wrong answer | 11.84 | 31.58 |
| Answer-correctness main effect | +34.54 pp | +9.87 pp |
| Reasoning-correctness main effect | +42.43 pp | +41.45 pp |
For each answer contrast, we hold the question, reasoning trace, and prompt structure fixed, replacing only the terminal answer. We also mask answer-equivalent spans in the reasoning and check for residual leakage. CRT reduces the resulting answer effect from to pp, while leaving the reasoning main effect nearly unchanged at versus pp. If CRT broadly downweighted prior context, both effects would contract. Instead, CRT preserves sensitivity to reasoning correctness while substantially reducing sensitivity to the terminal answer.
- Broader Reasoning Improvement. However, we also find evidence that CRT may also induce broader reasoning improvements beyond answer anchoring. Using a broad-first extraction rule, we identify the model’s first answer within a single reasoning attempt, defined as either the first boxed answer or the first textual answer declaration , and compute accuracy based on these answers. CRT models also improve under this metric, suggesting that CRT training may also enhance reasoning capabilities beyond iterative correction.
| Model | AIME24 | AIME25 | HMMT | HMMT-Nov | Avg. | Δ vs. Base |
|---|---|---|---|---|---|---|
| Base | 62.29 | 55.31 | 36.30 | 41.51 | 48.85 | — |
| No CRT-20k s156 | 63.65 | 57.81 | 36.88 | 39.58 | 49.48 | +0.63 |
| CRT-18k s156 | 66.25 | 58.75 | 36.25 | 42.29 | 50.89 | +2.04 |
| CRT-20k s314 | 66.25 | 58.75 | 40.00 | 42.19 | 51.80 | +2.95 |
| Main CRT s50 | 66.72 | 60.52 | 40.57 | 46.35 | 53.54 | +4.69 |
Overall, performance largely follows the ordering Base < CRT-18k < CRT-20k < Main CRT.
(Q3) Since CRT applies NLL supervision to self-generated trajectories without correctness rewards, how do correct and incorrect trajectories each contribute to the gains? Would training only on correct trajectories yield similar results?
We compared training on all-positive versus all-negative examples. The all-positive variant performed slightly better, but the difference was not significant.
| Training data | Step | AIME24 | AIME25 | 2-set Avg |
|---|---|---|---|---|
| ACE Base | 0 | 72.40 | 65.73 | 69.07 |
| Original data | 40 | 74.58 | 71.88 | 73.23 |
| Original data | 50 | 74.58 | 70.94 | 72.76 |
| AllPos | 40 | 74.79 | 71.88 | 73.33 |
| AllPos | 50 | 75.10 | 71.77 | 73.44 |
| AllNeg | 40 | 75.83 | 70.73 | 73.28 |
| AllNeg | 50 | 74.58 | 69.38 | 71.98 |
(Q4) CRT improves single-turn accuracy, but the additional gains from multi-turn self-correction appear relatively small. How should we interpret this finding in relation to the claim that CRT reduces answer anchoring, as opposed to improving reasoning ability more generally?
Our fourth analysis addressing Q1/W1, Broader Reasoning Improvement, evaluates the accuracy of the model’s first answer claim. We next examine answer anchoring within a single reasoning trajectory by comparing the first and final answer claims in each response. Using template-based extraction, we identify the first and final answer claims in each response and compute their respective accuracies. A larger Broad-final minus Broad-first gain indicates a stronger net tendency to move beyond an early answer claim and reach the correct final conclusion. CRT consistently exhibits larger gains than its corresponding Base model: +4.30 pp versus +3.40 pp for JustRL and +6.38 pp versus +5.26 pp for ACE. These results suggest that CRT reduces answer anchoring not only across turns but also within a single reasoning trajectory.
| Family / Model | Broad-first | Broad-final | Δ | Relative gain |
|---|---|---|---|---|
| JustRL Base | 48.85 | 52.25 | +3.40pp | +6.96% |
| JustRL No CRT-20k-RL 156step | 49.48 | 52.76 | +3.28pp | +6.63% |
| JustRL CRT-20k-RL 314step | 51.80 | 55.62 | +3.82pp | +7.37% |
| JustRL CRT | 53.54 | 57.84 | +4.30pp | +8.03% |
| ACE Base | 54.48 | 59.74 | +5.26pp | +9.66% |
| ACE CRT | 57.81 | 64.19 | +6.38pp | +11.04% |
Response to Authors
Thank you for the detailed rebuttal and additional experiments. The length-controlled results and behavioral analyses help address some of my concerns.
However, while CRT appears to reduce reliance on previous answers, it remains unclear whether mitigating answer anchoring is the primary driver of the accuracy gains, given the substantial improvements in first-answer accuracy and relatively modest self-correction gains. Thus, my main concern regarding the underlying mechanism remains only partially resolved.
length-controlled ablations
Because CRT naturally increases response length, we additionally optimize CRT with an RL objective that improves reward while penalizing excessively long generations. In the table, “CRT, 20k-length-RL, 314step” denotes CRT followed by length-constrained RL using a 20K-token length setting, evaluated at training step 314; the 18K variant uses a tighter length setting. “no CRT, 20k-length-RL” applies the same RL objective without CRT, controlling for gains from reward optimization alone, while “CRT, no RL” is the original CRT model without length control. At nearly unchanged length (+2.4%), CRT improves four-benchmark average accuracy by 2.20 percentage points, whereas RL without CRT yields only a 0.51-point gain. Larger gains still require some additional inference computation: longer CRT length-RL training improves accuracy by 3.38 points with only a 7.8% token increase, substantially below the 56.2% increase incurred by CRT-only training.
Table: length-controlled ablations.