acceptodds
Under review as a conference paper at ICLR 2027

Fine Until Fine-Tuned: Repeated Solutions Make Reasoning Fragile

Abstract

Recipes such as s1 and LIMO teach a model to reason with little data by showing it the same thousand or fewer worked solutions many times over. Judged when that training ends, the repetition looks harmless. But reasoning models are often trained again, and we find that repetition leaves their reasoning fragile to that next stage, even when the stage has nothing to do with reasoning. We fine-tuned Qwen3.5-9B-Base on its own correct solutions to competition math problems, either drilling a few hundred of them about eight times each or showing many more once; with the same amount of training, both solve about 95% of held-out problems. A single pass of ordinary instruction tuning leaves the once-trained model where it was, while the drilled one falls to 86.0%, and harsher later stages take it to 59.3% or below. A third model that visited the drilled problems just as often, with a new solution at every visit, was unharmed, so the damage comes from seeing the same texts again rather than from having few problems. The break recurs with a stronger model's traces, in further training runs and on other models and tasks. It is also cheap to undo: the reasoning is suppressed rather than erased, and five updates of reasoning training bring almost all of it back, as does brief training on the reasoning format with almost no mathematics. Fresh solutions prevented the damage, and so did replaying 6.25% of the original solutions in a gentler later stage, so our claim concerns later training without such replay. Sharpening alone does not explain the break, since a model sharpened three-quarters as much without repetition was unharmed, and on a skill the base model could not perform within a token budget, repetition mainly cost learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
continual learning, large language models, reasoning, supervised fine-tuning, data repetition, catastrophic forgetting, post-training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3FyxOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies whether repeatedly fine-tuning on identical reasoning traces makes reasoning vulnerable to subsequent training. All models perform similarly at handoff, but only the drilled model loses substantial reasoning accuracy after later instruction or answer-only training.

Critical strengths
  1. The paper shows that two models with the same immediate reasoning accuracy can encode that capability with very different robustness to later training, which is useful for refinement of how data efficiency should be assessed.

  2. The paper does not only identify the problem: it shows that varying solution traces, replaying a small amount of reasoning data, or briefly restoring the reasoning format can prevent or repair the failure.

Critical weaknesses
  1. The base model already scores 94.7%, and reasoning training provides almost no handoff improvement. On the synthetic task where the base lacks the procedure, repetition mainly harms acquisition rather than causing the same post-training collapse.

  2. The evaluation is only performed on particular reasoning format, only qwen's <think> block and uses a selected subset of MATH-500, boarder reasoning tasks and models need to be included to strengthen the claims.

Questions for authors
  1. Does replay remain effective under the harsher later stages?

  2. Can the main design be replicated in a setting where reasoning training produces a substantial capability gain at handoff?

1 comment
AuthorsOpenReview ↗

Reply to Reviewer 3Fyx

Thank you for the review. In short: replay remains effective under the harsher stages, and the paper now states its scope precisely: reasoning the model already has, the setting s1 and LIMO target.

Q1. Does replay still work under the harsher stages? Yes (Appendix F, registered). We replayed the drilled model's own training texts in 6.25% of each batch.

  • Instruction stress test. The drilled model keeps 91.5% instead of 59.3%. Its loss from handoff is 3.8 points [1.8, 5.9] instead of 36.1, which is 89% less.
  • Answer-only stress test. It keeps 55.8% instead of 22.5%. Its loss is 39.6 points [35.7, 43.6] instead of 72.9, which is 46% less.

The once-trained model lost nothing in either stress test, so we did not rerun it with replay.

Q2. Can the design be repeated where reasoning training gives a real gain at handoff? We tried the most natural candidate, and it does not provide such a setting.

We trained an instruct model (Qwen2.5-7B-Instruct, the family s1 uses) on gpt-oss-120b's traces. It lost forced accuracy instead of gaining it (69.6% to 61–65%). One epoch then took even the untrained student from 69.6% to 25.3%. Because the reasoning block is not native to that model's format, we also measured free accuracy. It fell as far (69.5% to 27.3%). That stage harms this chat model whatever its training, so the setting cannot test repetition in either direction.

The synthetic skill is our setting with a large gain, and there repetition mainly cost learning, as you say. The paper therefore states that the effect concerns reasoning the model could already do. What reasoning training changes in our setting is the default: the base model opens its reasoning block in 39% of free answers, the drilled and once-trained models in all of them.

Breadth: one reasoning format and a subset of MATH-500. Nemotron 3 Nano is another model family with its own chat template (it also reasons in a think block), and it, the arithmetic task and AIME were in the submission. The break appears in all of them: two model families, two scales, a stronger model's traces, three independent runs and a second task. The limitations state the remaining breadth: mathematics with integer answers and one arithmetic task.

Scope: released weights. One new result sharpens the claim and gives a remedy: from merged weights with a new adapter, one epoch no longer harms the drilled model, while the instruction stress test still costs it 9.4 points more than the once-trained model (general response, part 1). The title and abstract now state this condition, and the abstract says that every replication continues the adapter.

The writing. The paper is restructured: a Method section with design tables, results ordered by question, one name per stage, and 9 appendices by question instead of 15.

Reviewer_gi52OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether training a language model to reason by repeatedly cycling through a small set of worked solutions ("drilling", as in s1 and LIMO) makes its reasoning fragile to subsequent, unrelated fine-tuning. Using Qwen3.5-9B-Base with LoRA, the authors compare three regimes under the same 140-update compute budget: (1) drilled, (2) once-trained, and (3) fresh-solution. While all three achieve around 95% forced accuracy on MATH-500 at the end of reasoning training ("handoff"), subsequent instruction tuning causes the drilled model's accuracy to drop significantly, whereas the once-trained and fresh-solution models remain stable. The authors further show that this drop is a form of suppression rather than erasure, as it can be reversed by brief format-only training on thinking blocks or prevented by replaying 6.25% of the reasoning data.

Critical strengths
  1. Clean controlled comparisons and empirical transparency. The fresh-solution control cleanly isolates repeating identical solution texts from having few problems, and the pre-registered experimental protocols with transparent reporting of falsified hypotheses are commendable.
  2. Practical findings and simple remedies. The paper provides useful empirical observations on how repeated SFT behaves under subsequent fine-tuning, along with simple remedies such as sampling diverse solutions per problem or mixing 6.25% replay data.
Critical weaknesses
  1. Mismatch between the motivating pipeline (s1/LIMO) and the experimental setup (saturated base model and inverted pipeline).
    • Pipeline inversion: The paper motivates its study via s1 and LIMO, yet s1 and LIMO fine-tune already instruction-tuned models as a final reasoning stage, rather than training a Base model on reasoning first and then overwriting the same LoRA adapter with general instruction tuning or unformatted answer-only targets.
    • Ceiling effect on MATH-500 and disappearance of the effect when learning a new skill: Qwen3.5-9B-Base already achieves 94.7% forced accuracy on MATH-500 before any training, so the reasoning stage yields zero accuracy gain at handoff. Crucially, when the authors test a task where the base model actually learns a new skill from near zero (Section 6, Base-7 multiplication), the post-handoff fragility gap vanishes: from handoff to the later stage, the drilled model loses fewer points than the once-trained model. This raises doubts about whether post-handoff fragility holds when reasoning SFT genuinely improves capability rather than re-weighting an already saturated base model.
  2. Obscure writing style and fragmented, log-like organization.
    • Elliptical prose: The paper is written in an overly compressed, aphoristic style (for example, "whatever later separates the trained models must come from how the solutions were presented" and "Nor is the break a quirk of our setup"), and introduces core concepts inline within narrative paragraphs rather than in a clear, structured methodology section.
    • Disconnected experimental flow: The main text reads like a chronological stitching of experimental records across 15 appendices (Appendices A to O). Many experiments (such as the arithmetic task, anchor arms, and habit controls) are summarized without self-contained definitions in the main body, making it difficult to follow the logical connection between sections.
Questions for authors
  1. Starting from Instruct models or non-saturated tasks: Does the post-handoff fragility still occur when (a) starting from an Instruct model (as in s1 and LIMO), or (b) training on a math task where the base model is not already at a 95% ceiling (so that reasoning SFT yields a positive handoff gain, unlike MATH-500 where the gain is 0% and unlike the Base-7 task where post-handoff extra loss is -4.2 points)?
  2. Merged LoRA weights or full fine-tuning: If the reasoning LoRA adapter is merged into the base model before initializing a fresh LoRA adapter for the later stage (matching how a downstream user fine-tunes a released checkpoint), or if full fine-tuning is used, does the drilled model still break?
  3. Inference-time EOS suppression: Since 97% of the answer-only failures end the turn inside the thinking block and 20 format-only updates recover 97% to 98% of the accuracy, does simply masking or penalizing the EOS token before the closing thinking tag at inference time restore accuracy without retraining?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer gi52 (1/2)

Thank you for a detailed and precise review. Your Q2 sharpened the paper's main claim.

Q2. Merged weights with a fresh adapter, or full fine-tuning. From merged weights with a new adapter, one epoch no longer harms the drilled model, and with full fine-tuning it barely does. With a new adapter, the instruction stress test still does, in one run. The title and abstract now lead with this condition, and the introduction states it.

We merged the adapters into the base weights in bfloat16 (the merged weights reproduce Tinker's log-probabilities to 0.015 nats per token) and trained a new LoRA adapter, or every weight of the text model, with warm-up, cosine decay and clipping in the one-epoch stage; the new-adapter stress test kept the paper's constant rate. The table gives the drilled model's extra loss over the once-trained model, in points.

Later stageAdapter continuedNew adapterFull fine-tuning
One epoch of No Robots9.6 [6.9, 12.4]0.2 [−2.0, 2.5]1.5 [−0.9, 4.0]
Instruction stress test36.29.4 [6.4, 12.3]18.9 [14.1, 23.5]*
  • First row. The continued adapter is at a constant rate; under the standard schedule it gives 9.2 [6.2, 12.2], so the schedule is not what matters. The new adapter and full fine-tuning use the standard schedule (the new adapter at a constant rate gives −0.2).
  • The 9.4 is at the paper's constant rate. The other two models lose nothing in that stage.
  • *The 18.9 is one run of 20 updates at peak 3e-5 on the standard schedule. Every model loses accuracy in that stage, so we do not rest the claim on it.

Two registered controls show that neither the different software nor the output head explains the protection (general response, part 1). The paper now states plainly which case breaks in one epoch (a continued adapter) and which does not.

Q3. Masking the end-of-turn token. Under the registered scoring, masking recovers under half of the loss in both stress tests, as we predicted (Appendix D.3). Until the reasoning block closes, we give the tokens that open a new user turn, and end-of-text, a logit bias of −100.

  • After the instruction stress test, accuracy rises from 57.7% to 74.1%. That recovers 0.43 [0.35, 0.51] of the gap to handoff.
  • After the answer-only stress test, it recovers nothing (−0.05 [−0.11, 0.01]). Unable to stop, the model writes to the token limit in 97% of samples, boxing answer after answer.

Five updates of reasoning training recover nearly everything, so what is lost is suppressed and not erased.

AuthorsOpenReview ↗

Reply to Reviewer gi52 (2/2)

Q1 and W1. An instruct model, or a task where reasoning training adds accuracy. We tried an instruct student and did not get a setting where reasoning training adds accuracy. The paper now claims the effect only for reasoning the model could already do.

We trained Qwen2.5-7B-Instruct, the family s1 fine-tunes, on gpt-oss-120b's traces in its own chat template. The traces did not add accuracy: forced accuracy went from 69.6% to 61–65%. We then put the untrained student through the same epoch. It fell from 69.6% to 25.3%, and its free accuracy fell as far (69.5% to 27.3%; Appendix C.4). That stage harms this chat model whatever its training, so the setting cannot test repetition, in either direction.

The paper therefore states its scope precisely: the effect concerns reasoning the model could already do, which is what s1 and LIMO aim to draw out. Within that scope it replicates on Nemotron 3 Nano, a model that is already post-trained.

W1. The ceiling and the synthetic skill. Your reading is right, and the paper now says so in the abstract and the limitations. On the base-7 task the drilled model learns much less (37.9% against 66.3%), so there repetition's cost is in learning. We present that task as the boundary of the claim (Section 7, Appendix G).

W2. The writing. Thank you for quoting specific sentences. The second is gone. The first is now stated as an inference, right after the design it follows from (Section 1), and the aphoristic style has gone with both. The general response (part 2, point 6) lists what changed: a Method section that defines the three models, the later stages and the measures before the results; results ordered by question; one name per stage; 9 appendices by question instead of 15. Pre-registration is now mentioned only where it changes how a result should be read.

Reviewer_G6wrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how repeatedly training on identical reasoning traces affects the robustness of reasoning capabilities during subsequent finetuning. The paper discovers that although repeated training can obtain strong immediate performance gain, it inevitably makes reasoning more vulnerable to later finetuning, even on unrelated tasks. The paper also shows that using diverse solutions for the same problems largely avoids this issue. The affected reasoning ability can also be quickly recovered through additional training, suggesting that it is temporarily suppressed rather than lost completely.

Critical strengths
  • The studied problem is interesting. The paper identifies an interesting and practically relevant problem in reasoning finetuning that is often overlooked.

  • The experiments are carefully designed to isolate the effect of repeated reasoning traces from limited problem diversity.

  • The findings are supported by extensive experiments across different models and settings.

  • The recovery experiments provide interesting insights and practical ways to mitigate the problem. I actually find this finding quite useful.

Critical weaknesses
  • The underlying mechanism why this phenomena occur remains unclear. The analysis offers some clues but does not fully explain why repeated training leads to greater fragility.

  • The experiments mainly use LoRA, and some of the strongest effects appear under aggressive finetuning settings. The relevance to more realistic training pipelines is not fully studied. I think more finetuning settings can better support the wide existence of such phenomena.

  • The severity of degradation varies considerably across experiments, and the paper provides limited insight into what determines this variation.

Questions for authors

See the weakness section.

1 comment
AuthorsOpenReview ↗

Reply to Reviewer G6wr

Thank you for the review. In short: we ran the realistic settings you asked for, added an account of the mechanism whose registered predictions held, and measured what does and does not set the severity. One clarification on the summary first, because it changes how the paper reads: repetition gives no accuracy gain at handoff. All three models are at about 95%. What reasoning training changes is the default to reason: the base model opens its reasoning block in 39% of free answers, the drilled and once-trained models in all of them.

Mechanism (Section 5; general response, part 2). Drilling doubles the norm of the adapter's U factors, so every later update of the same adapter moves the weights further. A new adapter starts from zero and removes this, and then one epoch does no harm.

  • What was predicted and held. We registered that five stress-test updates would move the drilled weights at least twice as far, and observed 2.4× (1.7× without the output head, short of 2×).
  • What it covers. The account explains why continuing the drilled adapter amplifies the break. Why the drilled weights are more fragile to begin with is the open question the paper now poses.
  • Why repetition enlarges the factors: our reading is that the same texts give the same gradients, which add up over passes instead of partly cancelling.

LoRA and realistic pipelines (Section 4; table in the general response, part 1). We reran the later stages from merged weights with a new LoRA adapter or with full fine-tuning: the one-epoch stage under warm-up, cosine decay and clipping, the new-adapter stress tests at the paper's constant rate.

  • With the adapter continued, one epoch breaks the drilled model under either schedule: an extra loss of 9.6 points at a constant rate and 9.2 with the standard schedule.
  • With a new adapter or full fine-tuning, the same epoch does little or no harm: extra losses of 0.2 [−2.0, 2.5] and 1.5 [−0.9, 4.0]. So the break under ordinary training is specific to continuing the adapter, and starting a new one is a remedy.
  • With a new adapter, the instruction stress test still costs the drilled model 9.4 [6.4, 12.3] points more, so the drilled weights themselves remain more fragile.

Two registered controls show that neither the different software nor the output head explains the protection.

What sets the severity (Section 3, Appendix C).

  • The later stage. On instruction data the size follows mostly the learning rate (Figure 2a). At the same rate the answer-only stress test is far harsher, so the data matter too.
  • The dose. 1.1, 3.3, 5.0 and 7.7 passes over the same solutions lose 1, 1, 9.5 and 36 points in the instruction stress test.
  • The drilled set. At a fixed stage and dose, the extra loss runs from 12.6 to 61.1 points across drilled sets: always a break, of varying size.

Following your third point, we also measured each stored adapter's U factors without the output head (post hoc). Every rank-32 adapter drilled for about eight passes has a norm between 14.4 and 15.2, while its loss in the instruction stress test runs from 12.4 to 62.1 points. Every other rank-32 adapter is at most 8.6, except the five-pass one, at 11.7, in the gap like its loss. So factor size marks repetition, while the size of the break also depends on the drilled set. The paper accordingly claims the direction of the effect, which held in every run.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (to all reviewers and the AC) (1/2)

We thank the reviewers. In short: we ran every experiment the reviews asked for, and they make the result sharper. When the reasoning adapter itself is trained further, one epoch of instruction tuning costs the drilled model 9.2 points more than the once-trained model with warm-up and cosine decay (9.6 at a constant rate). The same experiments show how to avoid it: merged into the base weights and given a new adapter, as a downstream user often would, the drilled model comes through that epoch unharmed. Its weights still carry a measurable fragility: a harsher stage costs them 9.4 points more than the others. The revised title, abstract and introduction state this condition. The new title is "Fine Until Fine-Tuned Further: Repeated Solutions Make a Reasoning Adapter Fragile".

Part 1 covers whether the break survives realistic fine-tuning, and whether replay still works under harsh stages. Part 2 covers why repetition makes the reasoning fragile, the test-time probe, the missing gain at handoff, and the writing.

Each new experiment was registered before its data, apart from a few analyses that the paper marks as post hoc. Appendix I lists all 17 new predictions with their outcomes, and the records are in records/rebuttal.md in the updated supplement.

1. Released weights (Section 4, Table 4; Appendix C). We merged the three 9B adapters into the base weights (the merged weights reproduce Tinker's per-token log-probabilities to 0.015 nats/token), then ran the later stages with a newly initialized LoRA adapter, or with full fine-tuning; the one-epoch stage used warm-up, cosine decay and clipping. The table gives the drilled model's extra loss over the once-trained model, in points. Both stress tests ran at the paper's constant rate with either adapter; the full fine-tuning stress test ran on the standard schedule (†).

Later stageContinued adapterNew adapterFull fine-tuning
One epoch, standard schedule9.2 [6.2, 12.2]0.2 [−2.0, 2.5]1.5 [−0.9, 4.0]
Instruction stress test36.29.4 [6.4, 12.3]18.9 [14.1, 23.5]†
Answer-only stress test73.110.5 [7.4, 13.7]†–
  • Continued adapter: the schedule does not matter. Under warm-up, cosine decay and clipping the drilled model loses 9.2 [6.4, 11.9] points, as at a constant rate (9.4), and the once-trained model 0.0 [−1.4, 1.5] (registered: at most 3; held).
  • New adapter or full fine-tuning: one epoch does little or no harm. With full fine-tuning the drilled model loses 1.9 [0.1, 3.7] points, 1.5 [−0.9, 4.0] more than the once-trained model.
  • A harsher stage still finds the drilled weights. With a new adapter, the instruction stress test costs the drilled model 9.3 [6.7, 11.9] points while the others lose none.
  • † In the two harshest stages the drilled model again loses the most, but every model loses much of its accuracy there and the two models without repetition also differ (by 23.5 and 12.6 points), so we do not rest the claim on them.
  • Neither the software nor the output head explains the protection. Two registered controls, run on the drilled model only. Continuing its own adapter in the new software still breaks it: a loss of 6.7 [4.3, 9.2] points (predicted at least 5; 9.4 on Tinker, where no part of the adapter had to be frozen). A new adapter that also covers the output head still protects it: 95.7% to 97.2% (predicted a loss of at most 3).

2. Replay under the stress tests (Appendix F). Replay remains effective. With 6.25% of each batch replayed, the drilled model keeps 91.5% instead of 59.3% after the instruction stress test, and 55.8% instead of 22.5% after the answer-only stress test: its losses are cut by 89% and 46%.

AuthorsOpenReview ↗

General response (2/2)

3. Why the continued adapter breaks (Section 5). We now have an account at the level of the weights, and its two registered predictions held.

  1. A LoRA adapter adds a product of two matrices, UV, to each weight matrix, and the larger these factors, the further one update moves the weights. The drilled adapter's U factors have twice the norm of the once-trained adapter's (27.8 against 14.2).
  2. So the same later update should move the drilled weights further. We registered at least 2× after five stress-test updates and observed 2.4× (0.80% against 0.33%). Without the output head the ratio is 1.7×, short of 2×.
  3. A new adapter starts from U = 0 and should move every model alike. It does: 1.03 to 1.05×, against 1.6× for the continued adapter on the same post hoc measure (mean over matrices, without the head).

The account explains why continuing the adapter amplifies the break. Why the drilled weights are more fragile to begin with is the open question the paper now poses (Section 5).

At handoff the drilled adapter has moved the base weights 3.5× as far as the once-trained one. Across the 18 stored 9B models from the submission whose stress-test loss we know, every model drilled for about eight passes moved them by 2.0–2.8%, and every other model by at most 1.2%. The measure also sorts correctly the sharpened model, which the output measure places near the drilled models and the weights place with the robust ones. And a registered prediction held: a five-pass model moved its weights into the gap (1.35%), and it broke partly (9.5 points).

4. A test-time probe (Appendix D.3). Forbidding the model to end its turn recovers under half of the loss in both stress tests, as predicted: 0.43 of the gap after the instruction stress test and nothing after the answer-only stress test (−0.05), where the model writes until the token limit. Five updates of reasoning training recover nearly everything.

5. No gain at handoff (Section 7, Appendix C.4). Reasoning training adds no accuracy at handoff here, as gi52 and 3Fyx note. We tried a setting where it should: Qwen2.5-7B-Instruct on gpt-oss-120b's traces. It added none (forced accuracy fell from 69.6% to 61–65%). One epoch then took even the untrained student from 69.6% to 25.3%, and its free accuracy fell as far (69.5% to 27.3%): that stage harms this chat model whatever its training, so the setting cannot test repetition. The paper therefore states its scope precisely: reasoning the model can already do, which is what s1 and LIMO aim to draw out.

6. The writing.

  • A Method section defines the three models, the four later stages, forced and free accuracy, and extra loss before any result.
  • Results are ordered by question: does it break, does it survive release, why, what is lost, how to prevent it. Each stage and measure has one name throughout.
  • The 15 appendices in order of execution are now 9 by question.
  • The aphorisms and internal codes are gone from the main text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.