acceptodds
Under review as a conference paper at ICLR 2027

When Fine-Tuning Gains Reverse: Separating Prefix and Continuation Effects

Abstract

Supervised fine-tuning (SFT) can improve accuracy at a short output budget yet reduce it at a longer one, relative to the same initial checkpoint. To locate the change, we cross initial and fine-tuned prefix sources with both continuation models, comparing one stage while holding the other fixed. Our models are adapted with LoRA on 500 GSM8K training examples. On 50 generated two-variable linear systems, three Llama runs outperform the initial checkpoint at 192 tokens but underperform it at 768; the larger loss follows the prefix source. On all 1,319 GSM8K test questions, the tested Llama run again shows a prefix-driven loss, whereas three Qwen runs improve mainly through continuation. We then remove terminal end-of-sequence (EOS) targets during training. Exchanging the learned stopping probabilities and conditional content distributions assigns 74.7–81.1% of the algebra ablation gain to stopping. EOS ablation also improves MATH-500 and HumanEval accuracy, although recovery is incomplete in some settings. Checkpoint comparisons should therefore match the intended output budget; crossed continuations help identify which stage changed.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, supervised fine-tuning, output budgets, stopping behavior, model evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_GpK5OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how SFT gains depend on the output budget. It crosses initial and adapted checkpoints as prefix generators and continuation models, then exchanges stopping probabilities and conditional content distributions between standard and EOS-ablated SFT. Experiments find a budget reversal for Llama on synthetic algebra and attribute most of the EOS-ablation improvement to stopping in the tested algebra and MBPP settings.

Critical strengths

The matched EOS ablations and four-cell exchanges provide more informative controls than endpoint accuracy alone. The main algebra and EOS comparisons include three training seeds, and the question-level bootstrap analyses keep repeated samples grouped within each question. The paper also explicitly distinguishes recovery through EOS ablation from the full SFT deficit.

Critical weaknesses

W1. The empirical contribution remains limited to a narrow case study of a damaging adaptation regime. A consistent positive-to-negative budget reversal is demonstrated on only 50 synthetic algebra questions. The additional models do not consistently reproduce this pattern, and the original GSM8K subset does not exhibit the predicted budget trajectories (Table 2; Appendices C.1 and E). All main adaptations use the same 500-example corpus and learning rate. On MATH-500, mean standard-SFT accuracy drops from 43.10% to 15.53% for Llama and from 63.20% to 43.27% for Qwen, while EOS ablation recovers only 12.94% and 6.19% of these deficits, respectively (Appendix A.2). These results leave unresolved whether the approach provides a broadly useful diagnostic or primarily characterizes a specific training failure. Establishing the broader significance of the findings requires validation-selected SFT controls and evidence beyond the synthetic reversal. W2. The stage attribution is insufficiently validated against native stopping behavior. The protocols in Appendices C and G allow prefixes that have already terminated to receive nonempty continuations, which can replace the extracted answer. This issue is consequential: in the 50-question GSM8K diagnostic, 584 of 800 initial Llama prefixes and 699 of 800 adapted Llama prefixes have already terminated by the handoff. The reported source and continuation effects may therefore partly reflect the restarting of completed answers. The paper acknowledges this limitation but neither quantifies its contribution to the stage ordering nor demonstrates that the ordering persists when EOS is absorbing. The greedy MATH-500 NLL diagnostic in Appendix A.4 excludes early-ending pairs and therefore does not resolve this concern. Supporting the diagnostic interpretation requires a direct control with absorbing EOS.

Questions for authors
  1. Do the budget reversal and stage ordering persist for a validation-selected SFT checkpoint or under a modest learning-rate variation, rather than only the reported fixed recipe?
  2. What are the four-cell contrasts when terminated prefixes remain absorbing, and how much of the current GSM8K stage ordering comes from reopened responses?
1 comment
AuthorsOpenReview ↗

Adaptation Settings and EOS Handling

Thank you for the comments. We tested a lower learning rate, validation-selected checkpoints, and a handoff that leaves completed responses unchanged.

W1 and Question 1 — Adaptation setting and scope

Both settings reduce the long-budget loss, while all three seeds still improve at the short budget and lose accuracy at the long budget. On the original 50 algebra questions, with 16 samples per question, the lower learning rate gives mean gains of +10.625 pp at 192 tokens and −2.375 pp at 768 tokens. Validation selection gives +11.250 and −2.000 pp at the same budgets.

We lowered the learning rate from 1e−4 to 5e−5 and kept 100 training steps, the corpus, LoRA, and other optimization and batch settings fixed. For validation selection, candidates come from steps 0, 25, 50, 75, and 100 of the original 1e−4 runs. Selection by 192-token accuracy on 200 other GSM8K training questions chooses steps 50, 75, and 50. These questions exclude the adaptation corpus and official test split. The selected checkpoints are evaluated on algebra, so both task and budget change.

We also repeat the stage exchange for these checkpoints, generating prefixes for up to 192 tokens and allowing up to 576 more tokens for continuation. Source effects change the prefix-generating model while fixing the continuation model. Continuation effects change the continuation model while fixing the prefixes. Effects calculated from seed-averaged cells are below, in pp.

SettingSource effect 1Source effect 2Continuation effect 1Continuation effect 2Margin and 95% interval
Lower learning rate−2.917−2.792−0.417−0.2922.375 [0.167, 3.042]
Short-budget validation selection−2.292−2.375−0.333−0.4171.875 [−0.251, 2.417]

The margin measures how much the smaller source-effect magnitude exceeds the larger continuation-effect magnitude. It is positive in every seed. The lower-learning-rate interval is above zero, while the validation-selected interval includes zero. Bootstrap resampling is by question and keeps all samples, seeds, and cells together, including the shared initial reference.

Supplementary Tables 1–2 and 4 give the counts and individual contrasts. All effects use the exchange’s own diagonals. Their mean differences are −3.208 and −2.708 pp, compared with −2.375 and −2.000 pp under native generation.

On full-test GSM8K, Llama’s larger loss follows the prefix source (−7.127/−6.710 pp), while Qwen gains mainly through continuation (+4.321 to +5.099 pp). Other settings do not all show the budget reversal. On MATH-500, standard SFT reduces accuracy from 43.10% to 15.53% for Llama and from 63.20% to 43.27% for Qwen. EOS ablation closes 12.94% and 6.19% of those gaps.

W2 and Question 2 — Absorbing EOS and reopened responses

The larger source effect remains when completed responses are left unchanged. We tested initial Llama and the original d100 seed-1 checkpoint on 200 fixed GSM8K test questions, with four samples per question and a 768-token total allowance. The two evaluations reuse the same stored prefixes and all continuations of active prefixes.

HandoffAbsorbing-EOS source effectsAbsorbing-EOS continuation effectsMargin and 95% interval
96−6.000 / −5.375 pp−0.250 / +0.375 pp5.000 [0.500, 8.250] pp
192−6.500 / −5.750 pp−0.250 / +0.500 pp5.250 [0.375, 9.500] pp
384−6.250 / −5.500 pp−0.250 / +0.500 pp5.000 [1.250, 8.250] pp

Source effects fix the initial/adapted continuation models, respectively. Continuation effects fix the initial/adapted sources. All three margin intervals are above zero.

At 192 tokens, the source gaps fall from 56/49 to 52/46 successful responses under absorbing EOS, reductions of 7.1%/6.1%. The margin decreases by 0.250 pp, or 4.5%. Supplementary Tables 3 and 5 report the four-cell counts and reopening effects.

These percentages apply to the matched 200-question collection. The 584/800 and 699/800 ended-response counts in your review concern the original 50-question subset. Full-test GSM8K retains its separate population of 5,276 responses per cell.

Reviewer_Q1XXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates why SFT can improve accuracy at a short output budget while reducing it at a longer budget, and introduces controlled exchanges to distinguish changes in prefix generation, continuation, and stopping behavior. Using Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct adapted with LoRA on 500 GSM8K examples, the authors cross initial and fine-tuned prefix sources with both continuation checkpoints, keeping each stored prefix unchanged across receivers. On 50 generated algebra problems, three Llama adaptations gain approximately 11–13 percentage points at 192 tokens but lose 9–12 points at 768 tokens, with larger losses associated with the prefix source; on the full GSM8K test set, the tested Llama seed again exhibits a prefix-dominated loss, whereas three Qwen seeds improve mainly through continuation. A matched training ablation masks terminal end-of-sequence (EOS) targets, and a second factorial exchange swaps learned stopping probabilities and conditional non-EOS content distributions, assigning 74.7–81.1% of the algebra EOS-ablation recovery, rather than the original fine-tuning deficit, to stopping. Additional mathematical and code evaluations show benefits from EOS ablation over standard SFT, although recovery relative to the initial model is incomplete in several settings.

Critical strengths

The crossed prefix–continuation design provides an interpretable diagnostic beyond endpoint accuracy by holding the receiving checkpoint fixed when comparing prefix sources, explicitly reporting interactions, and avoiding an unjustified unique allocation when the replacement orders disagree.

The EOS investigation combines a matched training intervention that preserves the original loss-normalization denominator with stopping–content policy exchanges and same-history local interventions, providing substantially more informative evidence than a correlation between response length and accuracy alone.

The empirical reporting includes exact response counts, question-clustered bootstrap intervals for the stage comparisons, extraction-sensitivity analyses, full-test GSM8K evaluation, and mathematical/code transfer tests, while clearly distinguishing family-specific findings and incomplete recovery.

Critical weaknesses

The stage attribution is established at a single 192-token handoff that can resume previously terminated responses, leaving it unclear whether the reported ordering persists when EOS remains absorbing and the handoff occurs at other positions.

  • Using the existing initial and Llama d100 seed-1 checkpoints on 200 fixed GSM8K test questions with four responses per question, could you repeat the four-cell prefix exchange at 96, 192, and 384 tokens, keeping the total allowance at 768 and comparing the current handoff with a variant that leaves EOS-terminated prefixes unchanged. Report all four controlled contrasts, the contributions from terminated versus active prefixes, and paired question-bootstrap intervals; this is an inference-only robustness test requiring no retraining.

Although the paper correctly qualifies its denominator, the 74.7–81.1% stopping attribution compares standard and EOS-ablated d200 checkpoints rather than the initial and d100 checkpoints exhibiting the headline reversal, so it does not directly quantify stopping’s contribution to that original degradation.

  • Could you reuse the existing policy-exchange implementation to cross the initial versus standard-SFT d100 stopping gates and conditional content distributions for all three Llama seeds on the original 50 algebra questions, with 16 responses per question at both 192- and 768-token caps. Check that the diagonal policies reproduce native scores within sampling variability, then report the four contrasts, interactions, and signed stopping/content allocations with question-bootstrap intervals, directly testing whether restoring the initial stopping gate while retaining fine-tuned content repairs the long-budget loss.

The use of one fixed GSM8K-500 corpus and a single learning-rate recipe leaves robustness to adaptation strength unresolved, particularly given that Llama’s mean MATH-500 accuracy falls from 43.10% initially to 15.53% after only 50 standard-SFT steps.

Questions for authors

PLease see above.

3 comments
AuthorsOpenReview ↗

Handoff Controls and the Original Budget Reversal

Thank you for the specific suggestions. We ran the handoff controls and initial–d100 policy exchange, and tested the learning rate and checkpoint-selection settings.

1. Handoff position, absorbing EOS, and group contributions

The larger source effect persists at 96-, 192-, and 384-token handoffs with absorbing EOS. We use initial Llama and standard-SFT d100 seed 1 on 200 fixed GSM8K test questions, four samples per question, and a 768-token total allowance.

At each position, both evaluations reuse the stored prefixes and active continuations. With absorbing EOS, completed responses keep their terminal scores. The difference between the evaluations is the net effect of restarting those responses.

HandoffOriginal protocol, margin and 95% intervalAbsorbing EOS, margin and 95% interval
965.625 [1.000, 8.750] pp5.000 [0.500, 8.250] pp
1925.500 [0.375, 9.500] pp5.250 [0.375, 9.500] pp
3845.125 [0.625, 8.125] pp5.000 [1.250, 8.250] pp

The margin subtracts the larger absolute continuation effect from the smaller absolute source effect. Question-bootstrap draws retain each question’s four samples and all four cells. All six reported intervals are above zero.

At 192 tokens, the two source differences with absorbing EOS are −52 and −46 successful responses. Ended prefixes contribute −40 to each. Active prefixes contribute −12 and −6. Restarting completed responses increases the gap by four successes under the initial continuation model and three under the adapted model. Each source generates its own active group.

Supplementary Tables 1–2 and 6–7 provide all four cells, controlled contrasts, interactions, group counts, and intervals. This 200-question collection is separate from the original full-test GSM8K results, which use 5,276 responses per cell.

2. Initial–d100 stopping–content exchange

Restoring initial stopping while retaining adapted content closes 70.9%, 82.7%, and 68.0% of the long-budget gaps across seeds. We use all three Llama d100 checkpoints, the original 50 algebra questions, 16 samples per question, and both budgets.

A uses initial stopping and content. B changes stopping, C changes content, and D changes both. Unmixed policies run through the same implementation differ from native scores by one or two successes (0.125–0.250 pp). All paired difference intervals include zero. These runs use independent decoding streams with matching questions, sample counts, temperature, top-p, budget, and scorer. The initial-policy result is shared across seeds.

All effects use this exchange’s own cells. At 768 tokens, switching from D to C restores initial stopping probabilities while keeping adapted content. The fraction of the gap closed is (C−D)/(A−D).

SeedDiagonal gap, A−DNet success increase, C−DFraction of gap closedRemaining gap, A−CNet accuracy increase and 95% interval, pp
1866170.9%257.625 [5.125, 10.375]
2756282.7%137.750 [5.125, 10.625]
31006868.0%328.500 [5.750, 11.250]

Under both conditional content policies, adapted stopping improves accuracy at 192 tokens and reduces it at 768 tokens. With initial stopping restored, adapted content leaves gaps of 25/13/32 successes.

Supplementary Tables 3–4 and 8 give diagonal checks, four-cell counts, both stopping and content contrasts, interactions, and signed symmetric allocations with question-bootstrap intervals. The allocations average replacement orders and sum to D−A. The fractions above describe the D-to-C path.

The paper’s 74.7%–81.1% figure concerns the separate d200 comparison. Its denominator is the gain from standard SFT to EOS-ablated SFT.

3. Adaptation strength and selection

The lower-learning-rate control uses 5e−5 for 100 steps with the same GSM8K-500 corpus and other LoRA, optimization, and batch settings. Validation selection uses the original 1e−4 trajectories at steps 0/25/50/75/100. Selection by 192-token accuracy on 200 other GSM8K training questions chooses steps 50/75/50, separately for each seed. Validation excludes adaptation and official test questions.

On algebra, mean accuracies at 192/768 tokens are 33.250%/72.500% for the lower learning rate and 33.875%/72.875% for validation selection, compared with 22.625%/74.875% initially. Every tested seed retains positive short-budget and negative long-budget gains. Supplementary Table 5 reports the seed-level point estimates.

Selection is on GSM8K and evaluation on algebra, so both task and budget change. These controls vary learning rate and selection while keeping the training corpus fixed. MATH-500 still shows substantial degradation after EOS ablation.

AuthorsOpenReview ↗

Supplementary results for Reviewer Q1XX

Table 1. Handoff cells

Each cell contains 800 responses. Continuation allowances are 672/576/384 tokens for handoffs at 96/192/384 tokens. Both EOS treatments reuse active continuations.

HandoffEOS handlingInitial / InitialInitial / AdaptedAdapted / InitialAdapted / Adapted
96Original600597544548
96Absorbing586584538541
192Original604602548553
192Absorbing592590540544
384Original605603554558
384Absorbing598596548552

Table 2. Ended and active contributions

Contributions are adapted-source minus initial-source success counts within each group, with the full 800-response population as denominator. Active populations depend on source.

HandoffEnded prefixes, initial / adaptedCorrect ended prefixes, initial / adaptedEnded contribution, either receiverActive contribution, initial receiverActive contribution, adapted receiverTotal source difference, initial / adapted receiver
96300 / 350200 / 175−25−23−18−48 / −43
192500 / 550360 / 320−40−12−6−52 / −46
384650 / 690485 / 445−40−10−4−50 / −44

Table 3. Diagonal reproduction

Native and exchange runs use independent decoding random streams with matching evaluation settings. The initial reproduction is shared across adaptation seeds.

ModelBudgetOriginal native successesUnmixed exchange successesDifference, pp95% paired question-bootstrap interval, pp
Initial192181182+0.125[−1.750, 1.875]
Adapted seed 1192274273−0.125[−1.750, 1.500]
Adapted seed 2192286287+0.125[−1.500, 1.750]
Adapted seed 3192269270+0.125[−1.375, 1.750]
Initial768599601+0.250[−1.875, 2.500]
Adapted seed 1768513515+0.250[−1.625, 2.125]
Adapted seed 2768527526−0.125[−1.875, 1.625]
Adapted seed 3768503501−0.250[−2.250, 1.750]

Table 4. Policy-exchange cells

Each cell contains 800 responses. Content means the conditional non-EOS distribution. The stopping contrasts are B−A and D−C. The content contrasts are C−A and D−B.

SeedBudgetABCD
1192182243215273
1768601532576515
2192182251222287
2768601540588526
3192182238209270
3768601520569501

Table 5. Adaptation controls

Each cell contains 800 responses. Native initial scores are 181/599 at 192/768 tokens. Mean gains are +10.625/−2.375 pp for the lower learning rate and +11.250/−2.000 pp for selection.

SettingSeedCorrect at 192 tokensCorrect at 768 tokensShort-budget gain, ppLong-budget gain, pp
Lower learning rate1263577+10.250−2.750
Lower learning rate2268584+10.875−1.875
Lower learning rate3267579+10.750−2.500
Short-budget validation selection1270580+11.125−2.375
Short-budget validation selection2275588+11.750−1.375
Short-budget validation selection3268581+10.875−2.250

Table 6. Handoff contrasts and interactions

Entries give point estimates and 95% paired question-bootstrap intervals in pp. Source effects fix the initial/adapted continuation models. Continuation effects fix the initial/adapted sources. Full-test GSM8K retains its original counts of 3,983/3,972/3,607/3,618.

HandoffEOS handlingSource effect 1Source effect 2Continuation effect 1Continuation effect 2Interaction
96Original−7.000 [−10.500, −3.500]−6.125 [−9.625, −2.625]−0.375 [−1.875, 1.125]+0.500 [−1.125, 2.125]+0.875 [−1.125, 2.875]
96Absorbing−6.000 [−9.500, −2.622]−5.375 [−9.000, −1.872]−0.250 [−1.625, 1.125]+0.375 [−1.000, 1.750]+0.625 [−1.125, 2.375]
192Original−7.000 [−11.250, −2.875]−6.125 [−10.500, −1.750]−0.250 [−1.750, 1.250]+0.625 [−0.750, 2.125]+0.875 [−1.250, 3.000]
192Absorbing−6.500 [−10.875, −2.250]−5.750 [−10.125, −1.375]−0.250 [−1.375, 1.000]+0.500 [−0.125, 1.250]+0.750 [−0.750, 2.125]
384Original−6.375 [−9.750, −3.000]−5.625 [−9.000, −2.125]−0.250 [−1.750, 1.250]+0.500 [−0.875, 2.000]+0.750 [−1.250, 2.750]
384Absorbing−6.250 [−9.500, −3.000]−5.500 [−8.875, −2.125]−0.250 [−1.250, 0.750]+0.500 [0.125, 1.000]+0.750 [−0.375, 1.875]

Table 7. Group-contribution intervals

Entries give point estimates and 95% paired question-bootstrap intervals in pp under absorbing EOS.

HandoffEnded contributionActive contribution, initial receiverActive contribution, adapted receiver
96−3.125 [−5.125, −1.125]−2.875 [−6.375, 0.875]−2.250 [−5.875, 1.375]
192−5.000 [−7.375, −2.750]−1.500 [−5.250, 2.375]−0.750 [−4.500, 3.250]
384−5.000 [−7.125, −3.000]−1.250 [−3.875, 1.500]−0.500 [−3.250, 2.250]

Table 8. Policy contrasts, interactions, and allocations

Entries give point estimates and 95% paired question-bootstrap intervals in pp. The interaction is D−C−B+A. Each symmetric allocation averages its two corresponding contrasts.

Seed / BudgetStopping effect 1Stopping effect 2Content effect 1Content effect 2InteractionStopping allocationContent allocation
1 / 192+7.625 [5.000, 10.500]+7.250 [5.125, 9.625]+4.125 [2.375, 6.250]+3.750 [1.750, 5.875]−0.375 [−2.875, 2.000]+7.4375 [5.313, 9.750]+3.9375 [2.438, 5.563]
1 / 768−8.625 [−11.750, −5.750]−7.625 [−10.375, −5.125]−3.125 [−5.000, −1.375]−2.125 [−4.000, −0.250]+1.000 [−1.750, 4.000]−8.1250 [−10.625, −5.813]−2.6250 [−3.875, −1.500]
2 / 192+8.625 [6.000, 11.500]+8.125 [5.625, 10.875]+5.000 [2.750, 7.375]+4.500 [2.500, 6.625]−0.500 [−3.625, 2.625]+8.3750 [6.250, 10.563]+4.7500 [3.313, 6.250]
2 / 768−7.625 [−10.500, −4.875]−7.750 [−10.625, −5.125]−1.625 [−3.125, −0.250]−1.750 [−3.500, −0.122]−0.125 [−2.000, 1.875]−7.6875 [−10.375, −5.188]−1.6875 [−3.000, −0.438]
3 / 192+7.000 [4.625, 9.625]+7.625 [5.250, 10.125]+3.375 [1.625, 5.375]+4.000 [2.125, 6.000]+0.625 [−2.125, 3.128]+7.3125 [5.313, 9.500]+3.6875 [2.375, 5.125]
3 / 768−10.125 [−13.375, −7.000]−8.500 [−11.250, −5.750]−4.000 [−6.625, −1.750]−2.375 [−4.125, −0.750]+1.625 [−0.875, 4.500]−9.3125 [−12.063, −6.688]−3.1875 [−4.875, −1.625]

Thank you again for your careful review. We would be happy to clarify any remaining questions.

Reviewer_TQ6COpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies a reversal in apparent SFT gains. A fine-tuned checkpoint can outperform its initial checkpoint under a short output budget yet underperform it when allowed to generate for longer. The authors cross the initial and fine-tuned checkpoints as prefix generators and continuation models to determine where the change arises. On synthetic algebra problems, Llama’s long-budget loss follows the fine-tuned prefix source; a similar prefix-driven loss appears for one Llama run on full GSM8K, whereas three Qwen runs benefit mainly from the fine-tuned continuation model. Removing terminal EOS targets mitigates the long-budget degradation. The results show that checkpoints should be compared at the output budget intended for deployment.

Critical strengths
  1. The paper raises an important issue. It shows that choosing a checkpoint based on a short output budget can lead to a model that does worse at the budget actually used in deployment.

  2. The diagnostic is simple and useful. Swapping prefixes and continuations between the two checkpoints shows where the difference comes from, which endpoint accuracy alone cannot reveal. Splitting successes into retained and recovered answers also makes the results easy to understand.

  3. The paper is well written and easy to follow.

Critical weaknesses
  1. It is unclear how general the reversal is. The clear short-to-long reversal is shown mainly for three Llama-3.1-8B LoRA runs on 50 synthetic algebra problems. Qwen does not show the same reversal on this task, and the full GSM8K stage analysis uses only one Llama training seed. More model families, training setups, and tasks are needed to support broader claims.

  2. The stage-level results depend on how the handoff is done. Prefixes from another checkpoint may be out of distribution for the continuation model, and the prefix/continuation comparison is run only with a handoff at 192 tokens. Testing other handoff points would show whether the conclusions hold.

  3. The EOS result explains the policy, not the training mechanism. The 74.7%–81.1% figure comes from swapping parts of two trained policies. It shows that stopping matters more than content within this swap, but it does not measure how much of the training effect comes from the EOS loss itself. The paper is mostly careful about this, but the abstract's wording could be read as a causal claim.

  4. The mechanism is tested on fewer tasks than the endpoint gains. The stopping/content analysis covers only algebra and MBPP. MATH-500 and HumanEval show endpoint gains, but under different training and decoding settings, so they do not show that the same stopping mechanism is behind those gains.

Questions for authors

Please refer to the Weaknesses.

2 comments
AuthorsOpenReview ↗

Scope, Handoff Robustness, and Stopping Attribution

Thank you for the comments on the evaluation budgets, handoff, stopping analysis, and task coverage.

1. Scope of the reversal

All three original Llama algebra seeds improve at 192 tokens and lose accuracy at 768 tokens, with gains of +11.000 to +13.125 pp and −12.000 to −9.000 pp.

We added a lower learning rate of 5e−5 for 100 steps, keeping the corpus, LoRA, and other optimization settings fixed. We also select checkpoints from the original 1e−4 trajectories at steps 0/25/50/75/100, using 192-token accuracy on 200 separate GSM8K training questions. The selected steps are 50/75/50.

Across three seeds, mean algebra gains at 192/768 tokens are +10.625/−2.375 pp for the lower learning rate and +11.250/−2.000 pp for validation selection. All six seed-level estimates retain the sign change, with smaller long-budget losses. Selection is on GSM8K and evaluation on algebra, so both task and budget change. Supplementary Table 1 gives the counts.

On full-test GSM8K, the tested Llama checkpoint has a larger source loss, while three Qwen checkpoints gain through continuation. The same budget reversal does not occur in every model and task setting.

2. Handoff protocol

We tested initial Llama and d100 seed 1 on 200 GSM8K test questions, four samples per question, with a total allowance of 768 tokens. The two EOS evaluations reuse stored prefixes and active continuations. Absorbing EOS keeps ended responses at their terminal scores.

HandoffSource effects with absorbing EOSContinuation effects with absorbing EOSSource-ordering margin and 95% interval
96−6.000 / −5.375 pp−0.250 / +0.375 pp5.000 [0.500, 8.250] pp
192−6.500 / −5.750 pp−0.250 / +0.500 pp5.250 [0.375, 9.500] pp
384−6.250 / −5.500 pp−0.250 / +0.500 pp5.000 [1.250, 8.250] pp

The margin is the smaller absolute source effect minus the larger absolute continuation effect. Paired question-bootstrap resampling retains each question’s samples and all cells. The larger source effect persists at all three positions.

Prompts and prefix token IDs are passed unchanged. Cross-checkpoint histories can still differ from the continuation model’s own histories. The local NLL analysis covers retained active trajectories. The new controls address handoff position and restarting completed responses.

3. Interpretation of the stopping allocation

We revised the manuscript abstract and separate abstract text to read

In the 200-step algebra comparison, exchanging policy factors between standard-SFT and EOS-ablated checkpoints assigns 74.7–81.1% of the accuracy gain from EOS ablation to differences in stopping probabilities.

The percentage allocates the gain between factors of two trained policies. The matched training ablation removes terminal EOS supervision while preserving the loss-normalization denominator. Gradients measure local sensitivity at the evaluated checkpoint. One-step EOS suppression raises success before incorrect termination by 6.88 pp for Llama and 5.63 pp for Qwen. Before correct termination, it lowers success by 2.81 and 1.25 pp. Continuing can therefore recover an error or spoil a correct answer.

4. Mechanism and endpoint evidence

The original policy exchanges measure algebra final accuracy and MBPP prefix value under the initial continuation model. The new initial–d100 Qwen exchange on 684 common active GSM8K validation histories gives content allocations of 3.1250–3.4375 pp and stopping allocations of 0.8750–1.1875 pp, normalized by all 800 source responses.

MATH-500 and HumanEval use the same d50 checkpoints. EOS ablation raises MATH-500 accuracy by 3.57 pp for Llama and 1.23 pp for Qwen. On HumanEval, the gains are 6.68 and 6.07 pp, respectively. The original algebra exchange uses d200. MBPP uses Llama d100 and Qwen d200. Decoding settings differ across tasks.

EOS ablation leaves substantial losses on MATH-500, while every ablated HumanEval checkpoint exceeds its initial reference. We report the factor allocations for the exchanged policies and the endpoint gains for these two benchmarks.

AuthorsOpenReview ↗

Supplementary results

Table 1. Adaptation controls

Each seed has 800 algebra responses. The initial reference is 181/800 at 192 tokens and 599/800 at 768 tokens. Validation questions exclude the 500 adaptation examples and official test questions. Intermediate checkpoints come from the original 100-step trajectories.

SettingCorrect responses at 192 tokens, seeds 1–3Correct responses at 768 tokens, seeds 1–3Mean short-budget gainMean long-budget gain
Lower learning rate263 / 268 / 267577 / 584 / 579+10.625 pp−2.375 pp
Short-budget validation selection270 / 275 / 268580 / 588 / 581+11.250 pp−2.000 pp

Table 2. Endpoint gains

Both benchmarks use the same d50 checkpoints without further training.

BenchmarkLlama, EOS ablation minus standard SFTQwen, EOS ablation minus standard SFT
MATH-500, mean response accuracy+3.57 pp+1.23 pp
HumanEval, mean response accuracy+6.68 pp+6.07 pp

Thank you again for your careful review. We would be happy to clarify any remaining questions.

Reviewer_rsJLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how supervised fine-tuning gains change with the generation budget. It crosses initial and fine-tuned prefix sources with both continuation models to separate changes in generated histories from changes in continuation. A second experiment exchanges stopping probabilities and conditional content distributions between standard and EOS-ablated checkpoints. The experiments identify different stage effects for Llama and Qwen and show that stopping accounts for most of the EOS-ablation gain on algebra.

Critical strengths
  1. The crossed experiments provide useful controlled comparisons. Reusing each stored prefix under both continuation models helps locate where fine-tuning changes performance. The stopping-content exchange goes beyond a standard EOS ablation by measuring the effects of changing each probability component separately.

  2. The experimental reporting is careful. The paper includes full GSM8K evaluation, multiple training seeds for the central algebra experiments, and question-level uncertainty estimates. It also reports settings that do not reproduce the headline reversal and remaining losses after EOS ablation. These results make the scope of the findings clear.

Critical weaknesses
  1. The main weakness is the limited scientific advance over existing work. The EOS Decision study (arXiv:2010.07174) already separates content generation from stopping and compares training with and without EOS supervision. Zhang et al. (2025) (arXiv:2505.07961) already document non-monotonic accuracy with inference budget after SFT and adjust EOS probabilities to control the accuracy-length tradeoff. The latter studies excessive continuation, so it does not establish the premature-stopping result here, but it already covers much of the broader motivation.

  2. The paper's new contribution is therefore concentrated in the two crossed experiments and their measured effects. These establish useful differences between particular checkpoints, but provide little explanation of why Llama's change is dominated by the prefix source while Qwen's is dominated by continuation, or when either pattern should occur. The main adaptation setting uses LoRA on 500 GSM8K examples; the additional benchmark results extend the endpoint benefit of EOS ablation without establishing the same stage-level explanation. The study offers a detailed diagnosis, but the additional understanding remains too limited to support acceptance.

Questions for authors

Do the existing experiments identify a property of the initial models or their learned stopping behavior that explains the different stage effects in Llama and Qwen? Evidence connecting these effects to an identifiable condition would strengthen the scientific contribution.

2 comments
AuthorsOpenReview ↗

Contribution and Llama–Qwen Stage Differences

Thank you for the comments. We added policy exchanges and same-task measurements to investigate the different Llama and Qwen stage effects.

W1 — Contribution relative to prior work

Newman et al. study EOS supervision and length extrapolation, and Zhang et al. study stopping control and the accuracy–length trade-off. We find that the stopping change in the original Llama d100 adaptation has opposite effects at the two budgets. With either conditional content policy fixed, adapted stopping probabilities improve accuracy at 192 tokens and reduce it at 768 tokens.

The exchange covers three Llama d100 seeds, the original 50 algebra questions, and 16 samples per question. At 768 tokens, restoring initial stopping while retaining adapted content gives the following results.

SeedInitial policyAdapted policyAdapted content with initial stoppingNet success increaseFraction of the diagonal gap closed
16015155766170.9%
26015265886282.7%
36015015696868.0%

The net gains are 7.625, 7.750, and 8.500 pp, with 95% paired question-bootstrap intervals of [5.125, 10.375], [5.125, 10.625], and [5.750, 11.250] pp. Unmixed-policy scores differ from the original native scores by one or two successes. Effects use the new exchange’s own cells.

The original counts also show why earlier answer timing alone cannot explain the loss. Adapted prefixes have 88–105 more correct answers at token 192, but produce 175–195 fewer recoveries and 78–107 fewer final successes under the initial continuation model.

W2 and the author question — Llama and Qwen on the same task

We use two disjoint sets of 200 GSM8K training questions, both outside the adaptation corpus. We use the first for checkpoint selection and the two measurements below, and the second for held-out stage comparisons. Each has four samples per question, a 192-token handoff, and a 768-token total allowance.

For each response ending within 192 tokens, we compare its terminal score with the score after continuation by the fixed initial model. EOS is masked for the first continuation step. The net benefit counts rescued errors minus spoiled correct answers. The first measurement below is the adapted-source benefit minus the initial-source benefit, divided by all 800 responses. It combines how often responses end, which responses end, and whether continuing helps.

The second measurement compares adapted and initial continuation models on the same active prefixes from the initial source. We divide the success-count difference by all 800 responses. Ended responses contribute zero.

SettingChange in net benefit of continuing ended prefixes, ppGain from adapted continuation on fixed active prefixes, pp
Llama standard SFT, seed 1+5.375 [3.000, 7.750]+0.125 [−1.375, 1.625]
Qwen standard SFT, seed 1+0.125 [−1.375, 1.625]+4.375 [2.500, 6.375]
Qwen standard SFT, seed 2−0.125 [−1.375, 1.125]+4.000 [2.250, 5.750]
Qwen standard SFT, seed 30.000 [−1.250, 1.250]+4.625 [2.875, 6.500]

Brackets give 95% intervals. Llama’s net benefit from continuing ended prefixes increases after adaptation, with a small point-estimate gain from changing the continuation model. Qwen shows little change in the first measurement and larger gains from adapted continuation.

The Qwen stopping–content exchange on 684 common active validation histories gives content allocations of 3.1250–3.4375 pp and stopping allocations of 0.8750–1.1875 pp. These contributions use all 800 source responses as the denominator. The larger content point estimates connect Qwen’s fixed-history gain to its adapted conditional content distribution.

On separate questions with absorbing EOS, standard Llama has a source-ordering margin of 5.750 pp [2.000, 7.000]. The three Qwen continuation-ordering margins are 4.125 [1.000, 5.125], 3.875 [0.750, 4.875], and 4.375 [1.250, 5.375] pp. Supplementary Tables 1–5 give policy counts, full contrasts, and additional Llama settings.

The Llama exchange identifies a stopping change that reduces long-budget success. Qwen’s fixed-history gain has a larger point-estimate allocation to content. These measurements explain part of the observed difference on the same task. Their use to predict stage ordering remains untested, since active-history quality and stage interactions also affect the ordering.

The paper’s 74.7%–81.1% allocation concerns standard-SFT versus EOS-ablated d200 checkpoints. The d100 fractions above describe closure of the initial–adapted gap, and the Qwen allocations describe receiver gains on common histories.

Supplementary results for Reviewer rsJL

The original full-test GSM8K comparison uses 5,276 responses per cell. Llama source effects are −7.127/−6.710 pp and continuation effects −0.208/+0.208 pp. Qwen continuation gains span +4.321 to +5.099 pp. The new held-out collection is separate.

Changing stopping can alter subsequent histories and tokens. In the original d200 same-history interventions, one-step EOS suppression increases success before incorrect termination by 6.88/5.63 pp for Llama/Qwen, and decreases it before correct termination by 2.81/1.25 pp.

Table 1. Qwen policy-factor exchange

Counts concern 684 common active histories. A uses initial stopping and content, B changes stopping, C changes content, and D changes both. Each allocation averages its two replacement-order contrasts, divides by all 800 source responses, and multiplies by 100. Allocations sum to (D−A)/8 pp.

SeedABCDStopping allocation and 95% interval, ppContent allocation and 95% interval, pp
15635715895981.0625 [−0.313, 2.375]3.3125 [2.125, 4.563]
25635705885950.8750 [−0.313, 2.000]3.1250 [1.813, 4.500]
35635725906001.1875 [0.125, 2.250]3.4375 [2.188, 4.750]

Table 2. Held-out stage effects

The source margin subtracts the larger absolute continuation effect from the smaller absolute source effect. The continuation margin exchanges their roles. Intervals use paired question-bootstrap resampling.

SettingSource effects, ppContinuation effects, ppSource-ordering margin and 95% interval, ppContinuation-ordering margin and 95% interval, pp
Llama standard SFT, seed 1−6.500 / −6.250+0.250 / +0.5005.750 [2.000, 7.000]−6.250 [−8.375, −4.500]
Qwen standard SFT, seed 1+0.500 / +0.625+4.750 / +4.875−4.375 [−6.500, −3.125]4.125 [1.000, 5.125]
Qwen standard SFT, seed 2+0.625 / +0.625+4.500 / +4.500−3.875 [−6.125, −2.625]3.875 [0.750, 4.875]
Qwen standard SFT, seed 3+0.750 / +0.750+5.125 / +5.125−4.375 [−6.750, −3.250]4.375 [1.250, 5.375]

Table 3. Additional Llama probes

Both use seed 1. The lower learning rate is 5e−5 for 100 steps. Validation selection chooses step 50 from the original 1e−4 trajectory.

SettingChange in ended-history continuation-value contribution and 95% interval, ppReceiver improvement on fixed active histories and 95% interval, pp
Llama lower learning rate, seed 1+0.875 [−0.875, 2.625]+3.000 [1.625, 4.500]
Llama validation selection, seed 1+1.000 [−0.625, 2.625]+1.000 [−0.125, 2.250]

Table 4. Held-out exchange counts

Each cell contains 800 responses. Initial histories and their initial continuations are shared within each family. These GSM8K train questions are separate from the 200 GSM8K test questions in the handoff control and the 50 algebra questions in the adaptation-setting exchange.

SettingInitial / InitialInitial / AdaptedAdapted / InitialAdapted / Adapted
Llama standard SFT, seed 1589591537541
Llama lower learning rate, seed 1589608594613
Llama validation selection, seed 1589598582591
Qwen standard SFT, seed 1663701667706
Qwen standard SFT, seed 2663699668704
Qwen standard SFT, seed 3663704669710

Table 5. Additional Llama stage effects

Both continuation-margin intervals include zero. These comparisons cover fixed checkpoints from two families. The Qwen seeds share an initial model.

SettingSource effects, ppContinuation effects, ppSource-ordering margin and 95% interval, ppContinuation-ordering margin and 95% interval, pp
Llama lower learning rate, seed 1+0.625 / +0.625+2.375 / +2.375−1.750 [−3.875, −0.500]1.750 [−1.500, 2.625]
Llama validation selection, seed 1−0.875 / −0.875+1.125 / +1.125−0.250 [−2.625, 1.000]0.250 [−2.375, 1.375]

Comments on the paper

1 comment

Thank you to the authors for the responses and additional results, and thank you to all reviewers for your reviews and engagement so far.

I would like to draw the reviewers' attention to the authors' responses and the additional results addressing the reviewers' concerns. If you get a chance to look them over, your thoughts would be very welcome, including whether any concerns remain or you would like to update your assessment.

Thank you again for your time on this paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.