acceptodds
Under review as a conference paper at ICLR 2027

SFT-as-Context Mitigates Forgetting in Supervised Fine-tuning

Abstract

Supervised fine-tuning (SFT) equips large language models (LLMs) with specialized capabilities, but often comes at the cost of forgetting the general capabilities of their parent models (i.e., the pretrained models before fine-tuning). This trade-off is especially limiting for queries that require both specialized and general capabilities. We introduce SFT-as-context, a training-free method in which the parent model uses the SFT model's response as context to answer the query. This allows the parent model to acquire fine-tuned capabilities from the SFT response through in-context learning while preserving its own general capabilities. Across 19 parent-SFT model pairs and 11 benchmarks, SFT-as-context remains close to the SFT models on fine-tuned capabilities, with gaps of only 2.2 and 2.1 percentage points on AIME 2024 and LiveCodeBench and 2.0 macro MAE on NutriBench-English, while staying within 2.2 percentage points of the parent models on general capabilities on average. Remarkably, it can solve queries requiring both fine-tuned and general capabilities, even when neither the parent nor SFT model succeeds alone. This approach also extends beyond parent-SFT pairs: responses from a small open-source SFT model can improve a strong close-source LLM, outperforming either model alone. Furthermore, we use a Bayesian framework to derive theoretical guarantees that bound the error of SFT-as-context relative to the SFT model on fine-tuned capabilities and to the parent model on general capabilities. In addition, we visualize the attention weights and find that the parent model attends more to useful SFT responses and less to irrelevant ones, suggesting that selective attention helps the parent model use the SFT response through in-context learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
catastrophic forgetting, supervised fine-tuning, in-context learning, Bayesian analysis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1LFcOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose SFT-as-context, a training-free, inference-time method in which the SFT model first answers the query and the parent model then produces the final answer conditioned on the query, the SFT response, and a fixed instruction template. The method is evaluated on 19 parent–SFT pairs and 11 benchmarks. It recovers most fine-tuning gains while largely preserving general capabilities. The authors further show that responses from a small open-source SFT model can improve a proprietary model, provide a Bayesian analysis with error bounds, and analyze attention patterns over the SFT response.

Critical strengths

The paper studies a setting where a single query requires both fine-tuned and general capabilities. Because neither model alone can succeed here, combining the two models is valuable, and the setting itself provides a useful reference for future research.

Critical weaknesses
  1. The method requires both models to be available at serving time and performs two sequential decoding passes, so latency, memory, and compute all increase substantially relative to deploying the SFT model alone. The only overhead analysis (Appendix I) counts second-pass decode tokens, which ignores prefill cost over the long SFT response, additional weight memory, and end-to-end latency/throughput.

  2. The contribution is incremental and targets a problem setting that is shrinking. EFT (Mitchell et al., 2023) and Proxy-Tuning (Liu et al., 2024a) already combine fine-tuned and base models at inference time without retraining; the actual novelty is restricted to black-box text-only access. More importantly, post-training recipes are shifting from SFT toward RL, and the paper's own Table 5 finds that RL post-training induces negligible forgetting. The authors, however, offer no analysis or discussion of what this implies for the method's applicability. The study therefore rests on an application scenario that the community may gradually abandon.

Questions for authors
  1. Can you report end-to-end latency, GPU memory footprint, and serving throughput of SFT-as-context versus deploying the SFT model alone?

  2. The claim in Section 2.2 that Qwen3-8B drops from 99.6% to 10.7% on acknowledging non-food queries is not verifiable from any table (Table 1 reports averages). Could you release a full numeric table?

8 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1/5)

We thank you for your thoughtful and constructive review.

W1 & Q1:

The method requires both models to be available at serving time and performs two sequential decoding passes, so latency, memory, and compute all increase substantially relative to deploying the SFT model alone. The only overhead analysis (Appendix I) counts second-pass decode tokens, which ignores prefill cost over the long SFT response, additional weight memory, and end-to-end latency/throughput.

Can you report end-to-end latency, GPU memory footprint, and serving throughput of SFT-as-context versus deploying the SFT model alone?

A1: We thank you for highlighting the importance of measuring actual inference latency. In response, we conducted additional experiments to directly compare the end-to-end latency and decoding throughput of the parent model, the SFT model, and SFT-as-context (SAC) on all benchmarks involved. We demonstrate that SAC is highly efficient, with its second inference pass much cheaper than the first pass, incurring less than 5% additional latency in most (9 out of 11) of the datasets.

Experimental setup. For benchmarks where we tested math models, we used the parent model Qwen2.5-7B-Instruct and its SFT model OpenThinker3-7B, randomly sampling 100 questions from each benchmark (except AIME 2024, which contains only 30 questions, all of which were used). For benchmarks where we tested coding models, we use the parent model OpenCodeReasoning-Nemotron-7B, randomly sampling 100 questions from each benchmark. For NutriBench, we use the parent model Gemma-4-E4B-it and our fine-tuned version of it, sampling 50 examples from each of NutriBench-English, NutriBench-Non-English, and NutriBench-Non-Food (150 examples in total). All experiments are conducted using the Transformers library on a single NVIDIA A6000 GPU, processing one query at a time to ensure consistent latency measurements.

The end-to-end latency (seconds) of the methods is summarized in the table below. Each cell reports the result in the format of "mean ± std; median [min, max]". The fifth column is the sum of the third and the fourth columns. Comparing the fifth column with the third column results in the additional cost (percent of mean latency) in the final column. Importantly, for the second pass, all input processing (including GPU prefill) time is included in the latency.

BenchmarkParentSFTSFT-as-Context (Second Pass)SFT-as-Context (Both Passes)Additional Latency over SFT
AIME 202425.36 ± 14.84; 23.99 [10.08, 90.90]467.15 ± 278.65; 323.76 [129.34, 966.06]11.64 ± 17.77; 6.51 [0.24, 91.15]478.79 ± 285.63; 331.28 [141.71, 1054.95]+2.5%
MathIF20.27 ± 94.99; 10.66 [0.19, 957.86]247.87 ± 234.53; 160.65 [18.01, 958.53]6.29 ± 6.72; 4.05 [0.22, 31.19]254.16 ± 235.84; 163.64 [18.30, 987.03]+2.5%
IFEval26.05 ± 133.48; 4.28 [0.15, 954.75]307.06 ± 382.44; 103.12 [7.76, 957.30]27.01 ± 133.18; 4.59 [0.09, 950.78]334.07 ± 435.36; 108.06 [8.32, 1906.12]+8.8%
MGSM6.74 ± 3.91; 5.61 [2.09, 26.06]156.73 ± 222.36; 81.20 [11.37, 956.25]4.35 ± 4.22; 3.69 [0.24, 20.71]161.08 ± 224.14; 82.41 [11.63, 963.94]+2.8%
LiveCodeBench16.34 ± 93.94; 6.52 [1.27, 945.60]314.88 ± 225.17; 292.98 [28.31, 955.22]7.06 ± 5.20; 5.62 [1.12, 26.10]321.94 ± 228.28; 302.40 [30.72, 963.81]+2.2%
LiveCodeBenchIF5.39 ± 2.71; 5.03 [1.63, 18.63]334.08 ± 225.03; 351.19 [24.74, 955.55]7.88 ± 6.10; 5.60 [1.76, 33.65]341.96 ± 228.38; 355.28 [26.60, 958.37]+2.4%
SQuAD2.00.20 ± 0.19; 0.13 [0.08, 1.54]145.49 ± 313.40; 17.85 [2.44, 952.26]0.24 ± 0.15; 0.20 [0.11, 0.89]145.73 ± 313.41; 18.04 [2.59, 953.14]+0.2%
FQuAD2.01.41 ± 0.75; 1.35 [0.13, 4.94]158.55 ± 324.50; 19.77 [2.29, 952.32]0.57 ± 0.53; 0.34 [0.13, 2.82]159.12 ± 324.83; 20.13 [2.45, 954.34]+0.4%
CoQA0.89 ± 0.83; 0.66 [0.14, 5.19]119.56 ± 276.72; 18.81 [1.71, 942.66]0.75 ± 1.07; 0.42 [0.22, 6.96]120.31 ± 276.71; 19.52 [2.04, 943.00]+0.6%
ACPBench19.95 ± 87.20; 10.50 [2.35, 880.56]249.67 ± 360.77; 57.88 [9.49, 934.01]12.08 ± 88.02; 1.24 [0.38, 882.22]261.75 ± 389.82; 59.74 [9.93, 1763.79]+4.8%
NutriBench25.68 ± 22.94; 16.03 [4.09, 89.76]35.12 ± 66.87; 19.76 [3.68, 425.18]19.28 ± 17.43; 11.55 [4.33, 97.90]54.40 ± 68.74; 37.31 [8.36, 456.86]+54.9%

In 9 out of 11 datasets, SAC incurs less than 5% additional mean latency compared with the SFT model, despite requiring two sequential inference passes. Only on IFEval and on Nutribench did SAC incur a higher latency overhead of 8.8% and 54.9%. The result shows that the second inference pass is much cheaper than the first pass. The effect of mitigating forgetting and maintaining fine-tuned capabilities is worth this modest cost.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2/5)

In the same experiment, we additionally measure decoding throughput (tokens per second). Throughput is calculated as the total number of generated tokens divided by the corresponding generation time.

BenchmarkParentSFTSFT-as-Context
AIME 202439.6335.5638.99
MathIF37.0136.7038.60
IFEval35.7935.2535.88
MGSM39.7436.7938.78
LiveCodeBench36.0436.4237.92
LiveCodeBenchIF39.0836.3637.76
SQuAD2.034.7634.9926.29
FQuAD2.039.1934.9332.99
CoQA36.8435.3029.66
ACPBench36.9035.0934.39
NutriBench19.3619.5119.41

We observe that SAC maintains almost the same throughput as the parent and SFT runs. Unlike approaches that introduce additional computation during token generation, such as CPR, which uses an auxiliary MLP to predict token-mixing probabilities and reports a 23.0%-31.5% reduction in throughput compared to SFT model inference (Table 5 from the CPR paper), SAC does not modify the underlying decoding procedure so it does not reduce throughput.

Although SQuAD2.0 and CoQA seemingly suffers from the prefill time of SFT-as-context, we note that the average SFT-as-context responses are very short (6.4 and 22.2 tokens, respectively), so the lower throughput reflects the impact of the prefill time. However, the mean latencies of the second pass of SFT-as-context for these two datasets are below 1 second, way faster than the SFT pass that on average takes over 100 seconds.

We acknowledge that SFT-as-context requires access to both the parent and SFT models, resulting in additional model-weight storage compared with SFT-only inference. However, SAC does not require both models to be loaded into GPU memory simultaneously. Since the two inference passes are sequential, the SFT model can be unloaded after generating its response, allowing the parent model to reuse the same GPU memory for the second pass. Consequently, SAC can operate with peak model-weight GPU memory comparable to that of a single model, at the cost of additional model-loading overhead.

In contrast, popular token-level model combination methods require access to both models during decoding, making it more difficult to avoid simultaneously loading two models without substantial performance overhead. Thus, while SAC requires additional model storage compared with SFT-only inference, its sequential execution provides greater flexibility in managing GPU memory than token-level combination methods.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 3/5)

W2 (First Half):

The contribution is incremental and targets a problem setting that is shrinking. EFT (Mitchell et al., 2023) and Proxy-Tuning (Liu et al., 2024a) already combine fine-tuned and base models at inference time without retraining; the actual novelty is restricted to black-box text-only access.

A2 (First Half): We respectfully disagree that the novelty of our method is limited to black-box, text-only access. While EFT and Proxy-Tuning also combine pre-trained and fine-tuned models at inference time, they differ from our method in both their problem formulation and their mechanism for combining model capabilities.

First, EFT and Proxy-Tuning are not directly applicable to our setting. Both methods are designed around a completely different, three-model configuration consisting of a large pre-trained model, a smaller pre-trained model, and the fine-tuned version of the smaller model. They use the logit differences between the two smaller models to steer the large model during decoding. In contrast, our setting involves only two models: a parent model and its fine-tuned counterpart. Therefore, the original formulations of EFT and Proxy-Tuning cannot be directly applied.

Second, adapting these methods to our two-model setting does not effectively mitigate forgetting. A natural adaptation is to combine the parent and fine-tuned models through token-level logit manipulation. However, as demonstrated in Appendix H and our additional experiments in response to Reviewer SJ3d, such methods fail to effectively mitigate forgetting while preserving fine-tuned capabilities. The fundamental challenge is that the fine-tuned model's output distribution reflects both its newly acquired specialized capabilities and its degraded general capabilities. Consequently, directly increasing the influence of the fine-tuned model's logits can reinforce both desirable specialized behaviors and undesirable forgetting-related behaviors, without explicitly distinguishing between them.

In contrast, our method combines the two models at the response level instead of at the token level, allowing the parent model to selectively leverage specialized information while retaining its own general capabilities. By providing the fine-tuned model's response as context, SAC allows the parent model to use its in-context learning capability to selectively leverage specialized information while retaining its own general capabilities. This distinction is supported by our experiments showing that SAC can correct incorrect fine-tuned responses and, in some settings, outperform both models individually.

Reviewer_SJ3dOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the degradation of general capabilities following supervised fine-tuning, focusing particularly on requests that require both specialized knowledge and general capabilities such as instruction following and multilingual responses. It introduces SFT-as-Context (SAC), a two-stage inference method that requires no additional training. First, the SFT model generates a candidate answer. Its parent model then produces the final answer using the original request, the candidate answer, and relevant instructions, preserving useful content while making necessary corrections. The method requires only a text input–output interface and can also use a stronger model other than the original parent model to generate the final answer. Experiments cover 19 parent–SFT model pairs and 11 benchmarks spanning mathematics, coding, nutrition estimation, and general capabilities. SAC retains 92.0% of the fine-tuning gains while closing 94.2% of the general-capability gap between the parent and SFT models.

Critical strengths
  1. The paper provides a simple, practical way to combine specialized and general capabilities, validated across multiple domains. SAC can be applied to existing model checkpoints through a text interface, without modifying the training process or accessing model weights or logits. Across the evaluated model pairs, it preserves most of the specialized performance while substantially improving general behaviors degraded by SFT.

  2. The paper evaluates the composition of capabilities within a single answer, demonstrating benefits beyond selecting either existing response. Tables 7 and 8 evaluate joint success rates and compare SAC with an oracle selector that optimally chooses between the parent and SFT models’ responses. This comparison supports the paper’s motivation: an answer with correct specialized content may still violate user constraints, and choosing between two existing answers does not always resolve this conflict. Ablation experiments using a model’s own response as context further support the importance of complementarity between models.

Critical weaknesses
  1. The practical efficiency evaluation does not measure SAC’s actual latency. Appendix I and Table 10 analyze generated token counts rather than comparing end-to-end execution times. SAC requires two sequential model calls, and the second call must also process an input containing both the original task and the reference answer. The paper does not report measured response latency, how much longer SAC takes to produce the final answer compared with either model alone, or whether this overhead affects the practical user experience.

  2. Some conclusions about general-capability recovery rely on metrics with limited coverage or insufficient validation. Appendix A.5 evaluates behavior on MGSM based on the language used in the answer prefix, rather than assessing the quality of the complete multilingual response. For the nutrition task, the paper heuristically assesses language consistency and correct identification of non-food inputs by detecting phrases such as “contribute to” and “contributes to.” These checks may be sensitive to paraphrasing or output style, particularly because SAC regenerates the final answer. These checks alone cannot establish whether the full response maintains language consistency or whether the model handles irrelevant inputs correctly at the semantic level.

  3. Additional comparative analysis with two closely related approaches would be helpful. First, “CPR for LLMs: Critical-Point Routing against Catastrophic Forgetting in Domain Adaptation,” which the paper already cites, also mitigates forgetting through collaboration between a parent model and an SFT expert, but uses a token-level routing mechanism that requires training. Second, “Mixture-of-Agents Enhances Large Language Model Capabilities” generates a final answer from other models’ responses through a text interface, making its operation similar to SAC. A comparative analysis of CPR, MoA-style two-stage aggregation, and SAC is therefore recommended.

Questions for authors
  1. Clarification and request for additional evidence on latency: Beyond the token counts reported in Appendix I, are end-to-end latency comparisons available? Please compare the parent model alone, the SFT model alone, and SAC on both a long-reasoning task and NutriBench. Report median and tail latencies, with a breakdown of the time spent on first-stage generation, second-stage input processing, and second-stage generation.

  2. Clarification and validation request regarding the recovery metrics: For the nutrition task, how closely does phrase-based detection agree with independent assessments of whether the full response maintains language consistency and correctly handles non-food inputs? Please examine representative samples and report agreement rates or error rates separately for Parent, SFT, and SAC to verify whether the reported recovery gains hold under semantic evaluation.

8 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1/6)

We thank you for your thoughtful and constructive review.

W1 & Q1:

The practical efficiency evaluation does not measure SAC's actual latency. Appendix I and Table 10 analyze generated token counts rather than comparing end-to-end execution times. SAC requires two sequential model calls, and the second call must also process an input containing both the original task and the reference answer. The paper does not report measured response latency, how much longer SAC takes to produce the final answer compared with either model alone, or whether this overhead affects the practical user experience.

Clarification and request for additional evidence on latency: Beyond the token counts reported in Appendix I, are end-to-end latency comparisons available? Please compare the parent model alone, the SFT model alone, and SAC on both a long-reasoning task and NutriBench. Report median and tail latencies, with a breakdown of the time spent on first-stage generation, second-stage input processing, and second-stage generation.

A1: We thank you for highlighting the importance of measuring actual inference latency. In response, we conducted additional experiments to directly compare the end-to-end latency and decoding throughput of the parent model, the SFT model, and SFT-as-context (SAC) on all benchmarks involved. We demonstrate that SAC is highly efficient, with its second inference pass much cheaper than the first pass, incurring less than 5% additional latency in most (9 out of 11) of the datasets.

Experimental setup. For benchmarks where we tested math models, we used the parent model Qwen2.5-7B-Instruct and its SFT model OpenThinker3-7B, randomly sampling 100 questions from each benchmark (except AIME 2024, which contains only 30 questions, all of which were used). For benchmarks where we tested coding models, we use the parent model OpenCodeReasoning-Nemotron-7B, randomly sampling 100 questions from each benchmark. For NutriBench, we use the parent model Gemma-4-E4B-it and our fine-tuned version of it, sampling 50 examples from each of NutriBench-English, NutriBench-Non-English, and NutriBench-Non-Food (150 examples in total). All experiments are conducted using the Transformers library on a single NVIDIA A6000 GPU, processing one query at a time to ensure consistent latency measurements.

The end-to-end latency (seconds) of the methods is summarized in the table below. Each cell reports the result in the format of "mean ± std; median [min, max]". The fifth column is the sum of the third and the fourth columns. Comparing the fifth column with the third column results in the additional cost (percent of mean latency) in the final column. Importantly, for the second pass, all input processing (including GPU prefill) time is included in the latency.

BenchmarkParentSFTSFT-as-Context (Second Pass)SFT-as-Context (Both Passes)Additional Latency over SFT
AIME 202425.36 ± 14.84; 23.99 [10.08, 90.90]467.15 ± 278.65; 323.76 [129.34, 966.06]11.64 ± 17.77; 6.51 [0.24, 91.15]478.79 ± 285.63; 331.28 [141.71, 1054.95]+2.5%
MathIF20.27 ± 94.99; 10.66 [0.19, 957.86]247.87 ± 234.53; 160.65 [18.01, 958.53]6.29 ± 6.72; 4.05 [0.22, 31.19]254.16 ± 235.84; 163.64 [18.30, 987.03]+2.5%
IFEval26.05 ± 133.48; 4.28 [0.15, 954.75]307.06 ± 382.44; 103.12 [7.76, 957.30]27.01 ± 133.18; 4.59 [0.09, 950.78]334.07 ± 435.36; 108.06 [8.32, 1906.12]+8.8%
MGSM6.74 ± 3.91; 5.61 [2.09, 26.06]156.73 ± 222.36; 81.20 [11.37, 956.25]4.35 ± 4.22; 3.69 [0.24, 20.71]161.08 ± 224.14; 82.41 [11.63, 963.94]+2.8%
LiveCodeBench16.34 ± 93.94; 6.52 [1.27, 945.60]314.88 ± 225.17; 292.98 [28.31, 955.22]7.06 ± 5.20; 5.62 [1.12, 26.10]321.94 ± 228.28; 302.40 [30.72, 963.81]+2.2%
LiveCodeBenchIF5.39 ± 2.71; 5.03 [1.63, 18.63]334.08 ± 225.03; 351.19 [24.74, 955.55]7.88 ± 6.10; 5.60 [1.76, 33.65]341.96 ± 228.38; 355.28 [26.60, 958.37]+2.4%
SQuAD2.00.20 ± 0.19; 0.13 [0.08, 1.54]145.49 ± 313.40; 17.85 [2.44, 952.26]0.24 ± 0.15; 0.20 [0.11, 0.89]145.73 ± 313.41; 18.04 [2.59, 953.14]+0.2%
FQuAD2.01.41 ± 0.75; 1.35 [0.13, 4.94]158.55 ± 324.50; 19.77 [2.29, 952.32]0.57 ± 0.53; 0.34 [0.13, 2.82]159.12 ± 324.83; 20.13 [2.45, 954.34]+0.4%
CoQA0.89 ± 0.83; 0.66 [0.14, 5.19]119.56 ± 276.72; 18.81 [1.71, 942.66]0.75 ± 1.07; 0.42 [0.22, 6.96]120.31 ± 276.71; 19.52 [2.04, 943.00]+0.6%
ACPBench19.95 ± 87.20; 10.50 [2.35, 880.56]249.67 ± 360.77; 57.88 [9.49, 934.01]12.08 ± 88.02; 1.24 [0.38, 882.22]261.75 ± 389.82; 59.74 [9.93, 1763.79]+4.8%
NutriBench25.68 ± 22.94; 16.03 [4.09, 89.76]35.12 ± 66.87; 19.76 [3.68, 425.18]19.28 ± 17.43; 11.55 [4.33, 97.90]54.40 ± 68.74; 37.31 [8.36, 456.86]+54.9%
AuthorsOpenReview ↗

Rebuttal by Authors (Part 2/6)

In 9 out of 11 datasets, SAC incurs less than 5% additional mean latency compared with the SFT model, despite requiring two sequential inference passes. Only on IFEval and on Nutribench did SAC incur a higher latency overhead of 8.8% and 54.9%. The result shows that the second inference pass is much cheaper than the first pass. The effect of mitigating forgetting and maintaining fine-tuned capabilities is worth this modest cost.

In the same experiment, we additionally measure decoding throughput (tokens per second). Throughput is calculated as the total number of generated tokens divided by the corresponding generation time.

BenchmarkParentSFTSFT-as-Context
AIME 202439.6335.5638.99
MathIF37.0136.7038.60
IFEval35.7935.2535.88
MGSM39.7436.7938.78
LiveCodeBench36.0436.4237.92
LiveCodeBenchIF39.0836.3637.76
SQuAD2.034.7634.9926.29
FQuAD2.039.1934.9332.99
CoQA36.8435.3029.66
ACPBench36.9035.0934.39
NutriBench19.3619.5119.41

We observe that SAC maintains almost the same throughput as the parent and SFT runs. Unlike approaches that introduce additional computation during token generation, such as CPR, which uses an auxiliary MLP to predict token-mixing probabilities and reports a 23.0%-31.5% reduction in throughput compared to SFT model inference (Table 5 from the CPR paper), SAC does not modify the underlying decoding procedure so it does not reduce throughput.

Although SQuAD2.0 and CoQA seemingly suffers from the prefill time of SFT-as-context, we note that the average SFT-as-context responses are very short (6.4 and 22.2 tokens, respectively), so the lower throughput reflects the impact of the prefill time. However, the mean latencies of the second pass of SFT-as-context for these two datasets are below 1 second, way faster than the SFT pass that on average takes over 100 seconds.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 3/6)

W2 & Q2:

Some conclusions about general-capability recovery rely on metrics with limited coverage or insufficient validation. Appendix A.5 evaluates behavior on MGSM based on the language used in the answer prefix, rather than assessing the quality of the complete multilingual response. For the nutrition task, the paper heuristically assesses language consistency and correct identification of non-food inputs by detecting phrases such as "contribute to" and "contributes to." These checks may be sensitive to paraphrasing or output style, particularly because SAC regenerates the final answer. These checks alone cannot establish whether the full response maintains language consistency or whether the model handles irrelevant inputs correctly at the semantic level.

Clarification and validation request regarding the recovery metrics: For the nutrition task, how closely does phrase-based detection agree with independent assessments of whether the full response maintains language consistency and correctly handles non-food inputs? Please examine representative samples and report agreement rates or error rates separately for Parent, SFT, and SAC to verify whether the reported recovery gains hold under semantic evaluation.

A2: We thank you for raising this concern. We would like to clarify that our heuristics are designed to detect specific, observable forgetting behaviors induced by fine-tuning, rather than to serve as comprehensive measures of response quality. We construct these metrics based on the task instructions and the characteristic failure patterns observed in the SFT model. For NutriBench, we further provide manual validation to show that the heuristics accurately capture the intended behaviors, with a 99% agreement with human annotations. We also provide LLM-as-a-judge prompts that can more robustly handle the results, allowing us to achieve 100% agreement with human annotations.

1. MGSM

We formulate MGSM queries following the official evaluation implementation in OpenAI's simple-evals. Each prompt explicitly specifies the language-dependent prefix required for the final answer. For example, the English instruction is:

"Solve this math problem. Give the reasoning steps before giving the final answer on the last line by itself in the format of 'Answer:'. Do not add anything other than the integer answer after 'Answer:'."

Instructions in other languages similarly asks explicitly for the word "answer" in that language. Importantly, these instructions specify the format of the final answer but do not explicitly require the reasoning steps to be written in the same language as the question. Therefore, checking the final-answer prefix directly measures compliance with an explicit formatting requirement, rather than relying on an implicit expectation about the language of the entire response.

We agree that this metric does not assess the language consistency or semantic quality of the full reasoning trace. However, it is appropriate for evaluating the specific instruction-following behavior targeted in this experiment, and we do not interpret it as a comprehensive measure of multilingual response quality.

Reviewer_EaEVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SFT-as-Context, a training-free approach to mitigating catastrophic forgetting in supervised fine-tuning. It uses the SFT model's response as context for the parent model, aiming to combine specialized and general capabilities. Experiments demonstrate promising capability recovery, supported by theoretical analysis and attention visualization.

Critical strengths
  1. Simple and practical approach: The method requires no additional training or model modification and can be applied through standard text interfaces.
  2. Broad experimental validation: The evaluation covers multiple model families and tasks, demonstrating promising results in preserving fine-tuned capabilities while recovering general capabilities.
Critical weaknesses
  1. Inference overhead: The method requires two sequential model inferences, potentially introducing substantial latency and computational costs. Without an efficiency analysis, its practical advantages over alternative approaches remain unclear.
  2. Limited evidence for genuine capability transfer: The results do not sufficiently distinguish whether the parent model acquires specialized reasoning capabilities or primarily reuses the SFT model's answers. Additionally, some evaluation metrics rely on simple heuristics, raising concerns about the reliability of the reported capability recovery.
Questions for authors
  1. Inference Efficiency: What are the end-to-end latency, token consumption, and inference costs compared with standalone parent and SFT models? How do these costs affect practical deployment?
  2. Capability Transfer: To what extent does the parent model genuinely leverage specialized knowledge rather than copying or reformulating the SFT response? Could the authors analyze cases where the SFT response is incorrect or provide controlled experiments distinguishing answer copying from reasoning transfer?
  3. Evaluation Reliability: Some metrics, particularly language consistency and non-food acknowledgment, rely on heuristic phrase matching. How reliable are these heuristics? Could the authors validate them through human annotation or more robust evaluation methods?
9 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1/4)

We thank you for your thoughtful and constructive review.

W1 & Q1:

Inference overhead: The method requires two sequential model inferences, potentially introducing substantial latency and computational costs. Without an efficiency analysis, its practical advantages over alternative approaches remain unclear.

Inference Efficiency: What are the end-to-end latency, token consumption, and inference costs compared with standalone parent and SFT models? How do these costs affect practical deployment?

A1: We thank you for highlighting the importance of efficiency analysis. In response, we conducted additional experiments to directly compare the end-to-end latency and decoding throughput of the parent model, the SFT model, and SFT-as-context (SAC) on all benchmarks involved. We demonstrate that SAC is highly efficient, with its second inference pass much cheaper than the first pass, incurring less than 5% additional latency in most (9 out of 11) of the datasets. The additional token consumption is also low, as Table 10 in our paper shows.

Experimental setup. For benchmarks where we tested math models, we used the parent model Qwen2.5-7B-Instruct and its SFT model OpenThinker3-7B, randomly sampling 100 questions from each benchmark (except AIME 2024, which contains only 30 questions, all of which were used). For benchmarks where we tested coding models, we use the parent model OpenCodeReasoning-Nemotron-7B, randomly sampling 100 questions from each benchmark. For NutriBench, we use the parent model Gemma-4-E4B-it and our fine-tuned version of it, sampling 50 examples from each of NutriBench-English, NutriBench-Non-English, and NutriBench-Non-Food (150 examples in total). All experiments are conducted using the Transformers library on a single NVIDIA A6000 GPU, processing one query at a time to ensure consistent latency measurements.

The end-to-end latency (seconds) of the methods is summarized in the table below. Each cell reports the result in the format of "mean ± std; median [min, max]". The fifth column is the sum of the third and the fourth columns. Comparing the fifth column with the third column results in the additional cost (percent of mean latency) in the final column. Importantly, for the second pass, all input processing (including GPU prefill) time is included in the latency.

BenchmarkParentSFTSFT-as-Context (Second Pass)SFT-as-Context (Both Passes)Additional Latency over SFT
AIME 202425.36 ± 14.84; 23.99 [10.08, 90.90]467.15 ± 278.65; 323.76 [129.34, 966.06]11.64 ± 17.77; 6.51 [0.24, 91.15]478.79 ± 285.63; 331.28 [141.71, 1054.95]+2.5%
MathIF20.27 ± 94.99; 10.66 [0.19, 957.86]247.87 ± 234.53; 160.65 [18.01, 958.53]6.29 ± 6.72; 4.05 [0.22, 31.19]254.16 ± 235.84; 163.64 [18.30, 987.03]+2.5%
IFEval26.05 ± 133.48; 4.28 [0.15, 954.75]307.06 ± 382.44; 103.12 [7.76, 957.30]27.01 ± 133.18; 4.59 [0.09, 950.78]334.07 ± 435.36; 108.06 [8.32, 1906.12]+8.8%
MGSM6.74 ± 3.91; 5.61 [2.09, 26.06]156.73 ± 222.36; 81.20 [11.37, 956.25]4.35 ± 4.22; 3.69 [0.24, 20.71]161.08 ± 224.14; 82.41 [11.63, 963.94]+2.8%
LiveCodeBench16.34 ± 93.94; 6.52 [1.27, 945.60]314.88 ± 225.17; 292.98 [28.31, 955.22]7.06 ± 5.20; 5.62 [1.12, 26.10]321.94 ± 228.28; 302.40 [30.72, 963.81]+2.2%
LiveCodeBenchIF5.39 ± 2.71; 5.03 [1.63, 18.63]334.08 ± 225.03; 351.19 [24.74, 955.55]7.88 ± 6.10; 5.60 [1.76, 33.65]341.96 ± 228.38; 355.28 [26.60, 958.37]+2.4%
SQuAD2.00.20 ± 0.19; 0.13 [0.08, 1.54]145.49 ± 313.40; 17.85 [2.44, 952.26]0.24 ± 0.15; 0.20 [0.11, 0.89]145.73 ± 313.41; 18.04 [2.59, 953.14]+0.2%
FQuAD2.01.41 ± 0.75; 1.35 [0.13, 4.94]158.55 ± 324.50; 19.77 [2.29, 952.32]0.57 ± 0.53; 0.34 [0.13, 2.82]159.12 ± 324.83; 20.13 [2.45, 954.34]+0.4%
CoQA0.89 ± 0.83; 0.66 [0.14, 5.19]119.56 ± 276.72; 18.81 [1.71, 942.66]0.75 ± 1.07; 0.42 [0.22, 6.96]120.31 ± 276.71; 19.52 [2.04, 943.00]+0.6%
ACPBench19.95 ± 87.20; 10.50 [2.35, 880.56]249.67 ± 360.77; 57.88 [9.49, 934.01]12.08 ± 88.02; 1.24 [0.38, 882.22]261.75 ± 389.82; 59.74 [9.93, 1763.79]+4.8%
NutriBench25.68 ± 22.94; 16.03 [4.09, 89.76]35.12 ± 66.87; 19.76 [3.68, 425.18]19.28 ± 17.43; 11.55 [4.33, 97.90]54.40 ± 68.74; 37.31 [8.36, 456.86]+54.9%

In 9 out of 11 datasets, SAC incurs less than 5% additional mean latency compared with the SFT model, despite requiring two sequential inference passes. Only on IFEval and on Nutribench did SAC incur a higher latency overhead of 8.8% and 54.9%. The result shows that the second inference pass is much cheaper than the first pass. The effect of mitigating forgetting and maintaining fine-tuned capabilities is worth this modest cost.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2/4)

In the same experiment, we additionally measure decoding throughput (tokens per second). Throughput is calculated as the total number of generated tokens divided by the corresponding generation time.

BenchmarkParentSFTSFT-as-Context
AIME 202439.6335.5638.99
MathIF37.0136.7038.60
IFEval35.7935.2535.88
MGSM39.7436.7938.78
LiveCodeBench36.0436.4237.92
LiveCodeBenchIF39.0836.3637.76
SQuAD2.034.7634.9926.29
FQuAD2.039.1934.9332.99
CoQA36.8435.3029.66
ACPBench36.9035.0934.39
NutriBench19.3619.5119.41

We observe that SAC maintains almost the same throughput as the parent and SFT runs. Unlike approaches that introduce additional computation during token generation, such as CPR (another recent forgetting mitigation method, EMNLP 2026), which uses an auxiliary MLP to predict token-mixing probabilities and reports a 23.0%-31.5% reduction in throughput compared to SFT model inference (Table 5 from the CPR paper), SAC does not modify the underlying decoding procedure so it does not reduce throughput.

Although SQuAD2.0 and CoQA seemingly suffers from the prefill time of SFT-as-context, we note that the average SFT-as-context responses are very short (6.4 and 22.2 tokens, respectively), so the lower throughput reflects the impact of the prefill time. However, the mean latencies of the second pass of SFT-as-context for these two datasets are below 1 second, way faster than the SFT pass that on average takes over 100 seconds.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 3/4)

W2 & Q2:

Limited evidence for genuine capability transfer: The results do not sufficiently distinguish whether the parent model acquires specialized reasoning capabilities or primarily reuses the SFT model's answers.

Capability Transfer: To what extent does the parent model genuinely leverage specialized knowledge rather than copying or reformulating the SFT response? Could the authors analyze cases where the SFT response is incorrect or provide controlled experiments distinguishing answer copying from reasoning transfer?

A2: We thank you for raising this important question. We agree that distinguishing genuine capability transfer from simply copying the SFT model's responses is important. In the paper, we have provided three controlled experiments that demonstrate the parent model's ability to selectively leverage information from the SFT model instead of blindly copying its answers.

First, during SFT-as-context, the parent model can ignore the hallucinated, wrong response from the SFT model. This is evident in the NutriBench-Non-Food benchmark. SFT models frequently invent hallucinated nutrition values for irrelevant queries, such as estimating 5 grams of carbohydrates for a greeting "Hello, world!" (Figure 1 of the paper). When this is provided as context, the parent model can correctly identify the irrelevance and ignore the wrong response from the SFT model. This is a general capability beyond copying or reformulating. This capability allows SFT-as-context to recover the rate of acknowledging non-food queries from 22.6% of the SFT model to 98.0%, matching the parent performance of 98.8 with a recovery rate of 98.9%.

Second, we have examined cases where the SFT model provides incorrect answers at a large scale with more challenging tasks. Specifically, as shown in Figure 2 of the paper, we evaluate SFT models on three QA benchmarks (SQuAD, FQuAD, and CoQA), where the SFT model sometimes produces incorrect answers, resulting in worse performance than the parent model. We observe that the parent model can correct these errors even when the incorrect SFT responses are provided in its context. In the results of all these datasets from all models, there are 5,313 total cases where the parent model itself can answer the query correctly, but the SFT model provides a wrong answer. During SFT-as-context, 78.8% of these cases are flipped so that the correct answer is provided in the end. This demonstrates that the parent model does not simply adopt the SFT model's answers, but can instead evaluate and selectively use the information provided.

Third, we show that SFT-as-context can outperform both the parent and SFT models individually. As shown in Table 2 of the paper, SFT-as-context achieves a lower mean absolute error (MAE) than either model alone. For example, when using Qwen3-4B as context, the parent model has the lowest MAE of 14.08, lower than the parent's own performance of 17.44 and the SFT model's performance of 16.46. This improvement cannot be explained by simply copying the SFT model's predictions. Instead, it suggests that the parent model can combine its existing capabilities with information from the SFT model to produce more accurate predictions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.