- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose SFT-as-context, a training-free, inference-time method in which the SFT model first answers the query and the parent model then produces the final answer conditioned on the query, the SFT response, and a fixed instruction template. The method is evaluated on 19 parent–SFT pairs and 11 benchmarks. It recovers most fine-tuning gains while largely preserving general capabilities. The authors further show that responses from a small open-source SFT model can improve a proprietary model, provide a Bayesian analysis with error bounds, and analyze attention patterns over the SFT response.
Critical strengths
The paper studies a setting where a single query requires both fine-tuned and general capabilities. Because neither model alone can succeed here, combining the two models is valuable, and the setting itself provides a useful reference for future research.
Critical weaknesses
-
The method requires both models to be available at serving time and performs two sequential decoding passes, so latency, memory, and compute all increase substantially relative to deploying the SFT model alone. The only overhead analysis (Appendix I) counts second-pass decode tokens, which ignores prefill cost over the long SFT response, additional weight memory, and end-to-end latency/throughput.
-
The contribution is incremental and targets a problem setting that is shrinking. EFT (Mitchell et al., 2023) and Proxy-Tuning (Liu et al., 2024a) already combine fine-tuned and base models at inference time without retraining; the actual novelty is restricted to black-box text-only access. More importantly, post-training recipes are shifting from SFT toward RL, and the paper's own Table 5 finds that RL post-training induces negligible forgetting. The authors, however, offer no analysis or discussion of what this implies for the method's applicability. The study therefore rests on an application scenario that the community may gradually abandon.
Questions for authors
-
Can you report end-to-end latency, GPU memory footprint, and serving throughput of SFT-as-context versus deploying the SFT model alone?
-
The claim in Section 2.2 that Qwen3-8B drops from 99.6% to 10.7% on acknowledging non-food queries is not verifiable from any table (Table 1 reports averages). Could you release a full numeric table?
8 comments
Rebuttal by Authors (Part 2/5)
In the same experiment, we additionally measure decoding throughput (tokens per second). Throughput is calculated as the total number of generated tokens divided by the corresponding generation time.
| Benchmark | Parent | SFT | SFT-as-Context |
|---|---|---|---|
| AIME 2024 | 39.63 | 35.56 | 38.99 |
| MathIF | 37.01 | 36.70 | 38.60 |
| IFEval | 35.79 | 35.25 | 35.88 |
| MGSM | 39.74 | 36.79 | 38.78 |
| LiveCodeBench | 36.04 | 36.42 | 37.92 |
| LiveCodeBenchIF | 39.08 | 36.36 | 37.76 |
| SQuAD2.0 | 34.76 | 34.99 | 26.29 |
| FQuAD2.0 | 39.19 | 34.93 | 32.99 |
| CoQA | 36.84 | 35.30 | 29.66 |
| ACPBench | 36.90 | 35.09 | 34.39 |
| NutriBench | 19.36 | 19.51 | 19.41 |
We observe that SAC maintains almost the same throughput as the parent and SFT runs. Unlike approaches that introduce additional computation during token generation, such as CPR, which uses an auxiliary MLP to predict token-mixing probabilities and reports a 23.0%-31.5% reduction in throughput compared to SFT model inference (Table 5 from the CPR paper), SAC does not modify the underlying decoding procedure so it does not reduce throughput.
Although SQuAD2.0 and CoQA seemingly suffers from the prefill time of SFT-as-context, we note that the average SFT-as-context responses are very short (6.4 and 22.2 tokens, respectively), so the lower throughput reflects the impact of the prefill time. However, the mean latencies of the second pass of SFT-as-context for these two datasets are below 1 second, way faster than the SFT pass that on average takes over 100 seconds.
We acknowledge that SFT-as-context requires access to both the parent and SFT models, resulting in additional model-weight storage compared with SFT-only inference. However, SAC does not require both models to be loaded into GPU memory simultaneously. Since the two inference passes are sequential, the SFT model can be unloaded after generating its response, allowing the parent model to reuse the same GPU memory for the second pass. Consequently, SAC can operate with peak model-weight GPU memory comparable to that of a single model, at the cost of additional model-loading overhead.
In contrast, popular token-level model combination methods require access to both models during decoding, making it more difficult to avoid simultaneously loading two models without substantial performance overhead. Thus, while SAC requires additional model storage compared with SFT-only inference, its sequential execution provides greater flexibility in managing GPU memory than token-level combination methods.
Rebuttal by Authors (Part 3/5)
W2 (First Half):
The contribution is incremental and targets a problem setting that is shrinking. EFT (Mitchell et al., 2023) and Proxy-Tuning (Liu et al., 2024a) already combine fine-tuned and base models at inference time without retraining; the actual novelty is restricted to black-box text-only access.
A2 (First Half): We respectfully disagree that the novelty of our method is limited to black-box, text-only access. While EFT and Proxy-Tuning also combine pre-trained and fine-tuned models at inference time, they differ from our method in both their problem formulation and their mechanism for combining model capabilities.
First, EFT and Proxy-Tuning are not directly applicable to our setting. Both methods are designed around a completely different, three-model configuration consisting of a large pre-trained model, a smaller pre-trained model, and the fine-tuned version of the smaller model. They use the logit differences between the two smaller models to steer the large model during decoding. In contrast, our setting involves only two models: a parent model and its fine-tuned counterpart. Therefore, the original formulations of EFT and Proxy-Tuning cannot be directly applied.
Second, adapting these methods to our two-model setting does not effectively mitigate forgetting. A natural adaptation is to combine the parent and fine-tuned models through token-level logit manipulation. However, as demonstrated in Appendix H and our additional experiments in response to Reviewer SJ3d, such methods fail to effectively mitigate forgetting while preserving fine-tuned capabilities. The fundamental challenge is that the fine-tuned model's output distribution reflects both its newly acquired specialized capabilities and its degraded general capabilities. Consequently, directly increasing the influence of the fine-tuned model's logits can reinforce both desirable specialized behaviors and undesirable forgetting-related behaviors, without explicitly distinguishing between them.
In contrast, our method combines the two models at the response level instead of at the token level, allowing the parent model to selectively leverage specialized information while retaining its own general capabilities. By providing the fine-tuned model's response as context, SAC allows the parent model to use its in-context learning capability to selectively leverage specialized information while retaining its own general capabilities. This distinction is supported by our experiments showing that SAC can correct incorrect fine-tuned responses and, in some settings, outperform both models individually.
Rebuttal by Authors (Part 1/5)
We thank you for your thoughtful and constructive review.
A1: We thank you for highlighting the importance of measuring actual inference latency. In response, we conducted additional experiments to directly compare the end-to-end latency and decoding throughput of the parent model, the SFT model, and SFT-as-context (SAC) on all benchmarks involved. We demonstrate that SAC is highly efficient, with its second inference pass much cheaper than the first pass, incurring less than 5% additional latency in most (9 out of 11) of the datasets.
Experimental setup. For benchmarks where we tested math models, we used the parent model Qwen2.5-7B-Instruct and its SFT model OpenThinker3-7B, randomly sampling 100 questions from each benchmark (except AIME 2024, which contains only 30 questions, all of which were used). For benchmarks where we tested coding models, we use the parent model OpenCodeReasoning-Nemotron-7B, randomly sampling 100 questions from each benchmark. For NutriBench, we use the parent model Gemma-4-E4B-it and our fine-tuned version of it, sampling 50 examples from each of NutriBench-English, NutriBench-Non-English, and NutriBench-Non-Food (150 examples in total). All experiments are conducted using the Transformers library on a single NVIDIA A6000 GPU, processing one query at a time to ensure consistent latency measurements.
The end-to-end latency (seconds) of the methods is summarized in the table below. Each cell reports the result in the format of "mean ± std; median [min, max]". The fifth column is the sum of the third and the fourth columns. Comparing the fifth column with the third column results in the additional cost (percent of mean latency) in the final column. Importantly, for the second pass, all input processing (including GPU prefill) time is included in the latency.
In 9 out of 11 datasets, SAC incurs less than 5% additional mean latency compared with the SFT model, despite requiring two sequential inference passes. Only on IFEval and on Nutribench did SAC incur a higher latency overhead of 8.8% and 54.9%. The result shows that the second inference pass is much cheaper than the first pass. The effect of mitigating forgetting and maintaining fine-tuned capabilities is worth this modest cost.