acceptodds
Under review as a conference paper at ICLR 2027

Should SFT Prefer the Familiar? Diversity Collapse and Recovery via Model Merging

Abstract

Supervised fine-tuning (SFT) is widely used for downstream adaptation and often serves as initialization for further reinforcement learning (RL). Recently, several methods have improved SFT by upweighting from tokens that are already familiar to the current policy. Some approaches such as DFT prioritize high-probability tokens within external data, while some approaches train on self-generated trajectories, which naturally contain tokens that are familiar to the current model. Although these methods have good post-SFT performance, we empirically find that preferring familiar tokens in SFT can substantially reduce policy diversity and lead to worse subsequent RL performance. We further show theoretically that such methods create a self-reinforcing dynamic that progressively contracts the model's effective support, causing the model to become overly concentrated and reduce the diversity. To mitigate this effect, we propose a simple recovery approach: merging the fine-tuned model with its pretrained base model before RL. This approach can restore diversity lost during fine-tuning while preserving the gains in post-SFT performance. Across multiple mathematical reasoning and code generation benchmarks, base-model merging consistently outperforms recent SFT variants and standard regularization approaches such as KL-divergence regularization, yielding improvements in both post-SFT accuracy and subsequent RL performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Supervised Fine-Tuning, Reinforcement Learning, Diversity Collapse, Model Merging

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_78jiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies supervised fine-tuning methods that give more weight to tokens with high probability under the model. It finds that this preference can improve Pass@1 while reducing entropy, Pass@8, and gains from later reinforcement learning. A simplified theoretical analysis describes how the model distribution can become concentrated. The paper then applies base model merging (BMM), which averages the parameters of the fine-tuned and base models. Experiments on mathematical reasoning and code generation show improved accuracy and subsequent RL performance.

Critical strengths
  1. The evaluation covers several useful measures of model quality. The paper reports Pass@1, Pass@8, token entropy, answer-category entropy, and performance after RL. These measures reveal costs that a Pass@1-only evaluation could miss. The experiments cover multiple models and both mathematical reasoning and code generation. This makes the empirical findings useful for selecting an SFT method.

  2. BMM gives consistent practical gains over DFT. Across the four math models, it improves average Pass@1 and Pass@8 by 3.63 and 7.46 percentage points, respectively (Table 2). The code and subsequent RL results provide further support for its usefulness. The merge itself requires no additional training. The comparisons with regularization and early stopping also strengthen the empirical case.

Critical weaknesses
  1. The larger benefit of BMM for DFT remains insufficiently explained. The authors argue that DFT leaves more room for diversity recovery and that merging retains its task capabilities. This is plausible, but the experiments do not clearly separate entropy recovery from changes in task capability. Table 3(a) illustrates the gap. For Qwen2.5-Coder-3B, SFT and DFT both have an average Pass@1 of 69.13 before merging. After merging, their scores are 69.43 and 72.81. Thus, a higher starting average accuracy does not explain this case, and the difference already exists before RL. Figure 4 confirms that BMM increases DFT’s entropy, but does not show why its benefit exceeds that of SFT+BMM. Since this combination is the proposed remedy, its advantage needs more direct analysis.

  2. The connection between lower entropy and lower Pass@8 needs stronger evidence. The theoretical analysis explains concentration under a simplified model. It does not determine whether probability is removed from correct or incorrect responses. Lower entropy can therefore accompany either better or worse accuracy. For Qwen3-4B, DFT raises average Pass@1 from 25.22 to 26.70, but lowers Pass@8 from 46.22 to 40.42 (Table 1). The intuition that reduced diversity limits exploration is reasonable. However, the current analysis does not show which problems lose correct-response probability or whether BMM restores it on those problems. This limits support for the proposed explanation of the loss and recovery.

Questions for authors
  1. What evidence explains the larger improvement from DFT+BMM than from SFT+BMM, beyond an increase in entropy? A comparison of their performance at similar entropy levels would help. Please also clarify whether the advantage remains when both methods use separately selected merge coefficients. Appendix D.3 reports the coefficient sweep for DFT+BMM.

  2. Which problems account for the drop in Pass@8 under DFT, and does BMM improve success rates on those same problems? A plot of changes in entropy and correct-response frequency for each problem would help explain the aggregate results. A temperature sweep could further test whether greater sampling diversity alone recovers the lost performance.

4 comments

Thanks for the detailed response! We will address your questions below.

  1. "What evidence explains the larger Pass@1 improvement from DFT+BMM than from SFT+BMM, beyond an increase in entropy?"

This is a great question. We will answer this question by discussing Pass@1 and Pass@8 improvements. We have done a per-problem analysis of Qwen3-4B-Base rollouts across the five math benchmarks, with 16 responses for each problem.

Pass@1 improvement: First, since DFT has lower answer-category entropy, DFT concentrates its probability on one answer, so it has more failure problems than SFT with only 0–1 correct responses out of 16. We count the number of failure problems that have only 0-1 correct responses.

SFTDFT
Failure Problems (0–1 correct out of 16)512586

We then find that these failures are where the Pass@1 gain of merging comes from. For example, for DFT, merging raises the overall Pass@1 by 6.18, and 4.60 of this gain comes from the failure problems that Pass@1 on them rises from 0.7 to 8.1. Since DFT has more these failure problems, its Pass@1 improvement could be higher.

Moreover, beyond the entropy, we can find that DFT's failures are more often solvable by the base model. Among each model's failure problems, we count how often the base model can solve them with at least 3 correct responses per problem.

Correct responsesModelProblemsBase model solves them (at least 3 correct responses)Pass@1 before → after mergingContribution to the Pass@1 gain
0SFT4212.6%0.0 → 1.8+0.87
DFT51811.6%0.0 → 5.2+2.90
1SFT9138.3%6.3 → 13.4+0.54
DFT6849.1%6.2 → 32.4+1.69

The table shows two things. (a). First, DFT's failure problems are more often solvable by the base model. For example, the base model solves 11.6% of the problems where DFT has no correct response, compared with 2.6% for SFT. (b) Second, merging recovers much more Pass@1 on DFT's failure problems: from 0.0 to 5.2 and from 6.2 to 32.4 for DFT, compared with 0.0 to 1.8 and 6.3 to 13.4 for SFT. We believe this is because DFT keeps reinforcing its own distribution, which moves the model further away from the base model than SFT does. As a result, many of DFT's failures are problems that the base model can solve, but whose correct answer DFT has lost during over-concentration. Merging with the base model brings this answer back. SFT stays closer to the base model, so it loses less of what the base model knows, and merging has less to recover. This is why DFT+BMM improves Pass@1 much more than SFT+BMM.

Pass@8 improvement: Pass@8 improvement is more direct. In fact, since we rollout 16 responses for each problem and estimate Pass@8, the Pass@8 improvement is almost entirely determined by the failure problems with 0–1 correct responses before merging. (For example, improving 0 correct to 1 correct will increase Pass@8 from 0 to 0.5, but improving 4 correct to 10 correct only increases Pass@8 from 0.962 to 1.) Hence, since we have identified that DFT has more failure problems and increases more on these problems by base model merging, a larger Pass@8 improvement is expected.

  1. "Please also clarify whether the advantage remains when both methods use separately selected merge coefficients."

We swept the base weight from 0.1 to 0.9 in steps of 0.1 for both SFT+BMM and DFT+BMM, and selected each method's coefficient by the best Pass@1 + Pass@8. The table shows the Pass@1/Pass@8, and the selected base weight is shown in parentheses.

ModelSFT+BMMDFT+BMM
Qwen2.5-Math-1.5B24.60 / 49.10 (0.7)27.84 / 50.00 (0.7)
Qwen3-1.7B-Base22.30 / 43.60 (0.8)25.18 / 43.48 (0.7)
Qwen2.5-3B18.32 / 39.66 (0.5)22.08 / 42.96 (0.9)

With separately selected coefficients, DFT+BMM still has a clear Pass@1 advantage of 2.88 to 3.76 points on every model, with similar or higher Pass@8.

  1. "Lower entropy and Lower Pass@8 need more empirical evidence. It does not determine whether probability is removed from correct or incorrect responses. Lower entropy can therefore accompany either better or worse accuracy."

Thank you for this question. As discussed in point 1, Pass@8 is almost entirely determined by the failure problems with 0–1 correct responses out of 16. Hence, due to the lower answer-category entropy and more concentrated answers, DFT have more failure problems and then lower Pass@8. In contrast, the correct probability removed from correctly concentrated responses does not influence the Pass@8 a lot. Thus, lower entropy can imply lower Pass@8.

  1. "Also, which problems account for the drop in Pass@8 under DFT, and does BMM improve success rates on those same problems?"

The Pass@8 drop comes from the failure problems where DFT has only 0–1 correct responses and a lower Pass@8 than SFT (175 problems on Qwen3-4B-Base). We checked whether BMM restores success on these same problems. After merging, DFT+BMM recovers both Pass@1 (from 1.3 to 25.4) and Pass@8 (from 10.1 to 61.9) on these problems, and reaches the highest Pass@1 among all models on these problems.

Qwen3-4B-BaseBaseSFTDFTSFT+BMMDFT+BMM
Pass@123.722.11.322.525.4
Pass@875.977.710.169.061.9
  1. "A temperature sweep could further test whether greater sampling diversity alone recovers the lost performance."

We run the temperature sweep from T = 0.4 to T = 1.6 on Qwen3-4B-Base, and provide the Pass@1/Pass@8 in the following table.

Qwen3-1.7B-BaseT = 0.4T = 0.7 (default)T = 1.0T = 1.3T = 1.6
SFT20.62 / 39.2618.02 / 36.6413.58 / 31.844.70 / 17.181.46 / 8.98
DFT19.52 / 29.7220.08 / 33.6620.12 / 35.7020.24 / 36.8019.54 / 36.92
DFT+BMM24.72 / 39.9023.56 / 40.0623.78 / 42.7622.84 / 41.2821.20 / 42.16

Over all temperatures, the best Pass@8 of DFT+BMM (42.76) is clearly higher than that of DFT (36.92), and DFT+BMM has the best Pass@1 at every temperature. Hence, greater sampling diversity alone does not recover what BMM recovers.

Thanks again for your time and efforts! We will be happy to answer any further questions you may have.

Thank you for the additional analyses and experiments. The per-problem results, separately tuned merging coefficients, and temperature sweep substantially address my concerns. While some uncertainty remains about the mechanistic explanation, I am raising my score to 3.

Thanks for your constructive comments and for raising your score! We’re glad the additional analyses and experiments addressed your concerns, and we will include them in the revised version.

Reviewer_CFAPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates how SFT methods that favor familiar tokens affect diversity and subsequent RL performance. It provides theoretical analyses under simplified settings and evaluates base-model merging, which linearly interpolates a fine-tuned checkpoint with its pretrained initialization. Experiments on mathematical reasoning and code generation show improvements in post-SFT accuracy and final RL performance.

Critical strengths
  1. The paper addresses the important distinction between immediate SFT performance and the quality of the resulting initialization for RL.

  2. Base-model merging is simple, requires no additional training, and produces consistent accuracy improvements across the evaluated backbones.

  3. The experiments include useful controls for regularization strength, learning rate, early stopping, interpolation coefficients, and merging partners.

Critical weaknesses

Weaknesses

  1. The theoretical analysis and discussion of related methods have a narrow scope.
    The theoretical results concern DFT, TALR, and ProFit under simplified assumptions, leaving unclear how far the explanation extends beyond these probability-based token weighting rules. Relevant approaches such as GEM [1], which introduces a game-theoretic objective to preserve diversity, and Proximal Supervised Fine-Tuning [2], which constrains policy drift, are not discussed. The paper should clarify which classes of SFT objectives its conclusions apply to and explain how its analysis relates to these alternative formulations.

  2. The measurements do not sufficiently establish recovery of useful diversity.
    Final-answer entropy cannot distinguish different reasoning strategies that reach the same correct answer, while lower entropy can reflect improved correctness. Token entropy likewise does not distinguish useful alternatives from uncertainty. Higher final RL performance also does not isolate improved exploration. For example, in Table 3, Qwen2.5-Coder-3B gains 1.31 points from RL with DFT+BMM versus 1.83 with DFT, although BMM starts higher. Analysis of distinct successful reasoning strategies and comparisons accounting for initial performance would strengthen the proposed explanation.

  3. The model scale and family coverage are limited.
    All experiments use Qwen models between 1.5B and 4B parameters, with code experiments reaching only 3B. It remains unclear whether the observed collapse and recovery persist in larger models with stronger initial capabilities or in other model families. Evaluation on a larger backbone and an independent model family would strengthen the generality of the findings.

  4. The merging evaluation covers a limited range of algorithms and settings.
    The experiments vary interpolation coefficients and merging partners but do not compare different merging algorithms. In particular, Appendix D.4’s DFT+SFT setting would support a comparison with TIES-Merging [3] using the same checkpoints and evaluating both accuracy and diversity. This would clarify how the merging rule affects recovery. Furthermore, the evaluation remains confined to single-turn mathematical reasoning and code generation; merging checkpoints trained with different interaction formats, such as single-turn and multi-turn tasks, remains unexplored.

References

[1] Ziniu Li et al. Preserving Diversity in Supervised Fine-Tuning of Large Language Models. ICLR, 2025.

[2] Wenhong Zhu et al. Proximal Supervised Fine-Tuning. ICLR, 2026.

[3] Prateek Yadav et al. TIES-Merging: Resolving Interference When Merging Models. NeurIPS, 2023.

Questions for authors

See weaknesses.

5 comments

We thank the reviewer for the thoughtful and constructive comments. We will address your questions below.

  1. "The paper should clarify which classes of SFT objectives its conclusions apply to and explain how its analysis relates to these alternative formulations like GEM and Proximal SFT."

Thank you for this suggestion. Our analysis mainly focuses on token-reweighted likelihood objectives proposed in recent SFT variants (DFT, ProFit, TALR) ∑tw(pθ(yt∣x,y<t))log⁡pθ(yt∣x,y<t)\sum_t w(p_\theta(y_t\mid x,y_{<t})) \log p_\theta(y_t\mid x,y_{<t}), where the weight ww is relatively large when the pθ(yt∣x,y<t)p_\theta(y_t\mid x,y_{<t}) is high. This weight amplifies tokens the model already prefers, which is the reason for the concentration in our analysis.

Different from DFT, ProFit, and TALR, GEM [1] and Proximal SFT [2] are outside of this class. GEM changes the training objective to keep the output distribution diverse, so it will not have such concentration. Proximal SFT limits how far the policy can move from a reference model at each update, and the distribution shift happened in their paper is mainly due to overtraining multiple epochs of SFT. Both of these papers do not prefer the familiar token during training, hence not leading to the concentration we mentioned in our paper. Compared to them, our analysis mainly focuses on the reweighted variants of SFT which will lead to severe concentration even with a single epoch.

2. "Final-answer entropy cannot distinguish different reasoning strategies that reach the same correct answer. Analysis of distinct successful reasoning strategies would strengthen the proposed explanation."

This is a great suggestion. For Qwen3-4B-Base, we take all correct responses of each model on 619 problems where both SFT and DFT have at least 2 correct responses. An LLM judge (gpt-5-mini) is used to categorize each model's correct solutions to a problem by their solution path, where two solutions take different paths if they differ in the key intermediate steps, the lemma or the formula they use. The results are shown in the following table.

Qwen3-4B-BaseAverage correct responsesDistinct solution paths
SFT11.52.31
DFT12.91.79
SFT+BMM10.92.15
DFT+BMM13.51.93

As we can see, although DFT has more correct responses than SFT, it uses fewer distinct successful solution paths. BMM restores part of them. These results show that BMM recovers the useful diversity measured by distinct successful reasoning paths.

  1. "Higher final RL performance also does not isolate improved exploration. Comparisons accounting for initial performance would strengthen the proposed explanation."

Thanks for this question. We compare RL gains directly across all six RL settings (average Pass@1, before → after RL).

ModelDFTDFT+BMM
Qwen2.5-Math-1.5B26.52 → 28.90 (+2.38)28.46 → 33.72 (+5.26)
Qwen2.5-3B17.88 → 20.04 (+2.16)20.78 → 25.18 (+4.40)
Qwen3-1.7B-Base20.08 → 21.40 (+1.32)23.56 → 26.22 (+2.66)
Qwen3-4B-Base26.70 → 29.26 (+2.56)32.88 → 36.08 (+3.20)
Qwen2.5-Coder-1.5B63.79 → 65.63 (+1.84)66.84 → 69.54 (+2.70)
Qwen2.5-Coder-3B69.13 → 70.96 (+1.83)72.81 → 74.12 (+1.31)

In most of the settings, DFT+BMM gains more from RL than DFT. We further show the Pass@1 RL improvement on MATH-OAI since the improvement is clearer for our models.

ModelDFTDFT+BMM
Qwen2.5-Math-1.5B63.0 → 66.5 (+3.5)63.5 → 71.7 (+8.2)
Qwen2.5-3B48.9 → 52.6 (+3.7)52.4 → 61.4 (+9.0)
Qwen3-1.7B-Base53.1 → 56.4 (+3.3)57.8 → 62.3 (+4.5)
Qwen3-4B-Base63.7 → 65.5 (+1.8)69.7 → 74.2 (+4.5)

As shown in the table, DFT+BMM gains more from RL than DFT on all four models, which validates that DFT+BMM can have larger further RL gain compared to DFT. Hence, by isolating the higher initial performance, we still show that BMM leads to larger RL gain.

  1. "Evaluation on a larger backbone and an independent model family would strengthen the generality of the findings."

To make our evaluation more comprehensive, we added two larger backbones, Qwen3-8B-Base and Qwen2.5-Coder-7B, and two different model families, Falcon3-3B-Base and Granite-4.1-3B-Base (Pass@1 / Pass@8).

ModelSFTDFTSFT+BMMDFT+BMM
Qwen3-8B-Bas27.30 / 48.7832.68 / 48.4428.32 / 50.6033.76 / 53.14
Falcon3-3B-Base16.06 / 35.2817.98 / 32.2217.56 / 36.1819.00 / 34.62
Granite-4.1-3B-Base19.68 / 41.2421.06 / 34.5821.20 / 42.8224.20 / 43.48

The same pattern holds on every model. DFT raises Pass@1 but lowers Pass@8, and BMM recovers DFT's Pass@8 much more than SFT's (for example, +8.90 vs +1.58 on Granite-4.1-3B-Base). DFT+BMM also has a higher Pass@1 than other approaches on all models.

  1. "Appendix D.4's DFT+SFT setting would support a comparison with TIES-Merging using the same checkpoints and evaluating both accuracy and diversity."

Thank you for this suggestion. On Qwen2.5-Math-1.5B and Qwen3-1.7B-Base, we compared standard weight merging and SLERP for DFT+BMM and DFT+SFT. For DFT+SFT, we additionally consider TIES for merging. We measured accuracy and the number of distinct successful solution paths using LLM-as-Judge.

Qwen2.5-Math-1.5BPass@1 / Pass@8Solution paths
DFT+BMM (TA)28.46 / 46.881.90
DFT+BMM (SLERP)28.94 / 47.141.91
DFT+SFT (TA)24.94 / 45.842.04
DFT+SFT (SLERP)24.76 / 46.262.05
DFT+SFT (TIES)26.12 / 46.502.06
Qwen3-1.7B-BasePass@1 / Pass@8Solution paths
DFT+BMM (TA)23.56 / 40.062.04
DFT+BMM (SLERP)24.10 / 41.982.03
DFT+SFT (TA)21.03 / 40.222.10
DFT+SFT (SLERP)21.30 / 40.122.10
DFT+SFT (TIES)19.40 / 37.661.95

Across different merging approaches, the Pass@1, Pass@8 and diversity does not change a lot. Hence, the merging rule does not have a very clear influence on the final result.

  1. "Furthermore, the evaluation remains confined to single-turn mathematical reasoning and code generation; merging checkpoints trained with different interaction formats, such as single-turn and multi-turn tasks, remains unexplored."

Thank you for this suggestion. We first checked that our findings extend beyond single-turn evaluation, using the Follow-up QA task of the multi-turn MathChat benchmark [3]. It contains three-turn conversations built on GSM8K test problems. The first turn asks the original problem, and the next two turns ask follow-up questions that build on the previous answer. We evaluate the same single-turn-trained Qwen3-1.7B-Base model and grade the answer of every turn (Pass@1 / Pass@8).

Qwen3-1.7B-BaseTurn 1Turn 2Turn 3All 3 turns correct
SFT73.7 / 94.043.3 / 72.732.0 / 62.520.2%
DFT73.1 / 89.042.8 / 65.231.9 / 56.119.7%
SFT+BMM65.1 / 93.732.8 / 70.324.0 / 58.713.2%
DFT+BMM77.7 / 93.246.9 / 71.236.5 / 63.724.4%

As shown in the table, DFT also has significantly lower Pass@8 in every turn, and BMM restores DFT's Pass@8. Also, DFT+BMM has the best Pass@1 over other approaches.

Moreover, merging checkpoints trained with different interaction formats is an interesting direction. It requires training extra checkpoints on multi-turn data, while our paper focuses on a simple, training-free remedy BMM for recovering the diversity loss. Hence, this direction is outside the scope of this paper, and we leave it as future work.

Thanks again for your time and efforts! We will be happy to answer any further questions you may have.

References

[1]. Li et al, Preserving Diversity in Supervised Fine-Tuning of Large Language Models, 2025.

[2]. Zhu et al, Proximal Supervised Fine-Tuning, 2026.

[3]. Liang et al. MathChat: Benchmarking Mathematical Reasoning and Instruction Following in Multi-Turn Interactions, 2024.

Dear Reviewer CFAP,

Thank you again for your time and effort! As the discussion period approaches its deadline, we would appreciate it if you could let us know whether our responses have addressed your concerns. If you have more concerns, we are also happy to address them.

Best,

The Authors

Reviewer_ZomUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the impact of favoring familiar tokens during supervised fine-tuning (SFT). The authors find that recent SFT methods can improve initial accuracy but reduce policy diversity and limit the effectiveness of later reinforcement learning (RL). The paper provides a theoretical explanation of how favoring high-probability tokens can reduce diversity and proposes Base Model Merging, which combines the fine-tuned model with its pretrained base model through linear interpolation before RL.

Critical strengths
  1. The paper studies an important problem in SFT-to-RL training, highlighting the trade-off between initial SFT accuracy and the potential for further improvement through RL.

  2. The theoretical analysis provides useful insights into why favoring familiar tokens can lead to diversity collapse. The unified framework that covers several SFT variants is also interesting.

  3. The proposed BMM method is simple and computationally efficient. The experimental results show consistent improvements across multiple models and benchmarks.

Critical weaknesses

Overall, I find the paper well-motivated, clearly written, and supported by solid theoretical analysis and experimental results. I only have two concerns:

  1. Merging model checkpoints through weight interpolation has already been explored in prior work (e.g., [1]) to mitigate diversity collapse and improve subsequent RL performance. Although this paper provides a useful theoretical perspective on familiarity-biased SFT, the proposed BMM approach is similar to existing model-merging methods.

[1] Dang, X., Baek, C., Wen, K., Kolter, Z., and Raghunathan, A. (2025). Weight ensembling improves reasoning in language models. arXiv preprint arXiv:2504.10478.

  1. The paper argues that reduced diversity limits further improvement through RL. However, BMM improves both initial accuracy and diversity, making it unclear whether the better RL performance mainly comes from increased diversity or a better initialization.
Questions for authors
  1. In Figure 7, α=0.6\alpha = 0.6 seems to perform slightly better than α=0.5\alpha = 0.5 on both Pass@1 and Pass@8. Why was α=0.5\alpha = 0.5 chosen as the default merging coefficient?

  2. Figures 2 and 8 contain many small subplots, making the text and axis labels difficult to read. Could the authors consider enlarging the subplots?

  3. Table 8 uses MATH500?

  4. Section 4 mentions sampling 16 responses per prompt (Line 210), while Table 9 (Line 1114) reports 8 rollouts per prompt. Could the authors clarify this difference?

3 comments

We thank the reviewer for the positive review and the helpful comments. We will address your questions below.

  1. "Merging model checkpoints through weight interpolation has already been explored in prior work (e.g., [1]) to mitigate diversity collapse and improve subsequent RL performance."

Thank you for pointing out this related work. We acknowledge that weight interpolation itself is not a new technique, and we also mention the prior work [1] in our related work section. Compared to their methods that apply BMM for SFT, our contribution is to find that recent SFT variants that favor familiar tokens such as DFT, TALR, and ProFit, improve Pass@1 over standard SFT but suffer from a much more severe diversity collapse, and we explain this collapse theoretically. We then show that BMM effectively addresses it, which keeps or further improves Pass@1 while recovering some lost diversity. This gives a stronger initial checkpoint for subsequent RL. In our experiment, we find that DFT+BMM can achieve better performance than SFT+BMM, and a similar BMM technique also works well for other variants of SFT that prefer the familiar token. Hence, even if the BMM technique is not brand-new, we think the contribution of our paper is important and novel.

  1. "BMM improves both initial accuracy and diversity, making it unclear whether the better RL performance mainly comes from increased diversity or a better initialization."

Thanks for this question. We compare RL gains directly across all six RL settings (average Pass@1, before → after RL).

ModelDFTDFT+BMM
Qwen2.5-Math-1.5B26.52 → 28.90 (+2.38)28.46 → 33.72 (+5.26)
Qwen2.5-3B17.88 → 20.04 (+2.16)20.78 → 25.18 (+4.40)
Qwen3-1.7B-Base20.08 → 21.40 (+1.32)23.56 → 26.22 (+2.66)
Qwen3-4B-Base26.70 → 29.26 (+2.56)32.88 → 36.08 (+3.20)
Qwen2.5-Coder-1.5B63.79 → 65.63 (+1.84)66.84 → 69.54 (+2.70)
Qwen2.5-Coder-3B69.13 → 70.96 (+1.83)72.81 → 74.12 (+1.31)

In most of the settings, DFT+BMM gains more from RL than DFT. We further show the Pass@1 RL improvement on MATH-OAI since the improvement is clearer for our models.

ModelDFTDFT+BMM
Qwen2.5-Math-1.5B63.0 → 66.5 (+3.5)63.5 → 71.7 (+8.2)
Qwen2.5-3B48.9 → 52.6 (+3.7)52.4 → 61.4 (+9.0)
Qwen3-1.7B-Base53.1 → 56.4 (+3.3)57.8 → 62.3 (+4.5)
Qwen3-4B-Base63.7 → 65.5 (+1.8)69.7 → 74.2 (+4.5)

As shown in the table, DFT+BMM gains more from RL than DFT on all four models, which validates that DFT+BMM can have a larger further RL gain compared to DFT. Hence, by isolating the higher initial performance, we still show that BMM leads to a larger RL gain.

  1. "Why was 0.5 chosen as the default merging coefficient?"

Thank you for the question. We chose 0.5 because it is a common and generally strong choice that works well across all our models, so we use it as a single default. As shown in our response to Reviewer 78ji (point 2), the best coefficient varies across models (from 0.5 to 0.9), and DFT+BMM outperforms SFT+BMM in Pass@1 at each model's best coefficient. Hence, we use 0.5 as a simple default without tuning. In practice, model merging is computationally cheap, so it will be beneficial to select the best coefficient on a validation set.

  1. "Figures 2 and 8 contain many small subplots, making the text and axis labels difficult to read."

Thank you for the suggestion. We will enlarge the subplots and their fonts in the revised version.

  1. "Table 8 uses MATH500?" and "Section 4 mentions sampling 16 responses per prompt (Line 210), while Table 9 (Line 1114) reports 8 rollouts per prompt."

Thank you for pointing these out! Table 8 is MATH-OAI, and the number of rollouts per prompt in Table 9 should be 16. We will correct both in the revised version.

Thanks again for your time and efforts! We will be happy to answer any further questions you may have.

References:

[1] Dang et al, Weight ensembling improves reasoning in language models, 2025.

Dear Reviewer ZomU,

Thank you again for your time and effort! As the discussion period approaches its deadline, we would appreciate it if you could let us know whether our responses have addressed your concerns. If you have more concerns, we are also happy to address them.

Best,

The Authors

Thanks for the response. My major concerns have been addressed so I would like to keep my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.