- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies supervised fine-tuning methods that give more weight to tokens with high probability under the model. It finds that this preference can improve Pass@1 while reducing entropy, Pass@8, and gains from later reinforcement learning. A simplified theoretical analysis describes how the model distribution can become concentrated. The paper then applies base model merging (BMM), which averages the parameters of the fine-tuned and base models. Experiments on mathematical reasoning and code generation show improved accuracy and subsequent RL performance.
Critical strengths
-
The evaluation covers several useful measures of model quality. The paper reports Pass@1, Pass@8, token entropy, answer-category entropy, and performance after RL. These measures reveal costs that a Pass@1-only evaluation could miss. The experiments cover multiple models and both mathematical reasoning and code generation. This makes the empirical findings useful for selecting an SFT method.
-
BMM gives consistent practical gains over DFT. Across the four math models, it improves average Pass@1 and Pass@8 by 3.63 and 7.46 percentage points, respectively (Table 2). The code and subsequent RL results provide further support for its usefulness. The merge itself requires no additional training. The comparisons with regularization and early stopping also strengthen the empirical case.
Critical weaknesses
-
The larger benefit of BMM for DFT remains insufficiently explained. The authors argue that DFT leaves more room for diversity recovery and that merging retains its task capabilities. This is plausible, but the experiments do not clearly separate entropy recovery from changes in task capability. Table 3(a) illustrates the gap. For Qwen2.5-Coder-3B, SFT and DFT both have an average Pass@1 of 69.13 before merging. After merging, their scores are 69.43 and 72.81. Thus, a higher starting average accuracy does not explain this case, and the difference already exists before RL. Figure 4 confirms that BMM increases DFT’s entropy, but does not show why its benefit exceeds that of SFT+BMM. Since this combination is the proposed remedy, its advantage needs more direct analysis.
-
The connection between lower entropy and lower Pass@8 needs stronger evidence. The theoretical analysis explains concentration under a simplified model. It does not determine whether probability is removed from correct or incorrect responses. Lower entropy can therefore accompany either better or worse accuracy. For Qwen3-4B, DFT raises average Pass@1 from 25.22 to 26.70, but lowers Pass@8 from 46.22 to 40.42 (Table 1). The intuition that reduced diversity limits exploration is reasonable. However, the current analysis does not show which problems lose correct-response probability or whether BMM restores it on those problems. This limits support for the proposed explanation of the loss and recovery.
Questions for authors
-
What evidence explains the larger improvement from DFT+BMM than from SFT+BMM, beyond an increase in entropy? A comparison of their performance at similar entropy levels would help. Please also clarify whether the advantage remains when both methods use separately selected merge coefficients. Appendix D.3 reports the coefficient sweep for DFT+BMM.
-
Which problems account for the drop in Pass@8 under DFT, and does BMM improve success rates on those same problems? A plot of changes in entropy and correct-response frequency for each problem would help explain the aggregate results. A temperature sweep could further test whether greater sampling diversity alone recovers the lost performance.
4 comments
- "Also, which problems account for the drop in Pass@8 under DFT, and does BMM improve success rates on those same problems?"
The Pass@8 drop comes from the failure problems where DFT has only 0–1 correct responses and a lower Pass@8 than SFT (175 problems on Qwen3-4B-Base). We checked whether BMM restores success on these same problems. After merging, DFT+BMM recovers both Pass@1 (from 1.3 to 25.4) and Pass@8 (from 10.1 to 61.9) on these problems, and reaches the highest Pass@1 among all models on these problems.
| Qwen3-4B-Base | Base | SFT | DFT | SFT+BMM | DFT+BMM |
|---|---|---|---|---|---|
| Pass@1 | 23.7 | 22.1 | 1.3 | 22.5 | 25.4 |
| Pass@8 | 75.9 | 77.7 | 10.1 | 69.0 | 61.9 |
- "A temperature sweep could further test whether greater sampling diversity alone recovers the lost performance."
We run the temperature sweep from T = 0.4 to T = 1.6 on Qwen3-4B-Base, and provide the Pass@1/Pass@8 in the following table.
| Qwen3-1.7B-Base | T = 0.4 | T = 0.7 (default) | T = 1.0 | T = 1.3 | T = 1.6 |
|---|---|---|---|---|---|
| SFT | 20.62 / 39.26 | 18.02 / 36.64 | 13.58 / 31.84 | 4.70 / 17.18 | 1.46 / 8.98 |
| DFT | 19.52 / 29.72 | 20.08 / 33.66 | 20.12 / 35.70 | 20.24 / 36.80 | 19.54 / 36.92 |
| DFT+BMM | 24.72 / 39.90 | 23.56 / 40.06 | 23.78 / 42.76 | 22.84 / 41.28 | 21.20 / 42.16 |
Over all temperatures, the best Pass@8 of DFT+BMM (42.76) is clearly higher than that of DFT (36.92), and DFT+BMM has the best Pass@1 at every temperature. Hence, greater sampling diversity alone does not recover what BMM recovers.
Thanks again for your time and efforts! We will be happy to answer any further questions you may have.
Thank you for the additional analyses and experiments. The per-problem results, separately tuned merging coefficients, and temperature sweep substantially address my concerns. While some uncertainty remains about the mechanistic explanation, I am raising my score to 3.
Thanks for your constructive comments and for raising your score! We’re glad the additional analyses and experiments addressed your concerns, and we will include them in the revised version.
Thanks for the detailed response! We will address your questions below.
This is a great question. We will answer this question by discussing Pass@1 and Pass@8 improvements. We have done a per-problem analysis of Qwen3-4B-Base rollouts across the five math benchmarks, with 16 responses for each problem.
Pass@1 improvement: First, since DFT has lower answer-category entropy, DFT concentrates its probability on one answer, so it has more failure problems than SFT with only 0–1 correct responses out of 16. We count the number of failure problems that have only 0-1 correct responses.
We then find that these failures are where the Pass@1 gain of merging comes from. For example, for DFT, merging raises the overall Pass@1 by 6.18, and 4.60 of this gain comes from the failure problems that Pass@1 on them rises from 0.7 to 8.1. Since DFT has more these failure problems, its Pass@1 improvement could be higher.
Moreover, beyond the entropy, we can find that DFT's failures are more often solvable by the base model. Among each model's failure problems, we count how often the base model can solve them with at least 3 correct responses per problem.
The table shows two things. (a). First, DFT's failure problems are more often solvable by the base model. For example, the base model solves 11.6% of the problems where DFT has no correct response, compared with 2.6% for SFT. (b) Second, merging recovers much more Pass@1 on DFT's failure problems: from 0.0 to 5.2 and from 6.2 to 32.4 for DFT, compared with 0.0 to 1.8 and 6.3 to 13.4 for SFT. We believe this is because DFT keeps reinforcing its own distribution, which moves the model further away from the base model than SFT does. As a result, many of DFT's failures are problems that the base model can solve, but whose correct answer DFT has lost during over-concentration. Merging with the base model brings this answer back. SFT stays closer to the base model, so it loses less of what the base model knows, and merging has less to recover. This is why DFT+BMM improves Pass@1 much more than SFT+BMM.
Pass@8 improvement: Pass@8 improvement is more direct. In fact, since we rollout 16 responses for each problem and estimate Pass@8, the Pass@8 improvement is almost entirely determined by the failure problems with 0–1 correct responses before merging. (For example, improving 0 correct to 1 correct will increase Pass@8 from 0 to 0.5, but improving 4 correct to 10 correct only increases Pass@8 from 0.962 to 1.) Hence, since we have identified that DFT has more failure problems and increases more on these problems by base model merging, a larger Pass@8 improvement is expected.
We swept the base weight from 0.1 to 0.9 in steps of 0.1 for both SFT+BMM and DFT+BMM, and selected each method's coefficient by the best Pass@1 + Pass@8. The table shows the Pass@1/Pass@8, and the selected base weight is shown in parentheses.
With separately selected coefficients, DFT+BMM still has a clear Pass@1 advantage of 2.88 to 3.76 points on every model, with similar or higher Pass@8.
Thank you for this question. As discussed in point 1, Pass@8 is almost entirely determined by the failure problems with 0–1 correct responses out of 16. Hence, due to the lower answer-category entropy and more concentrated answers, DFT have more failure problems and then lower Pass@8. In contrast, the correct probability removed from correctly concentrated responses does not influence the Pass@8 a lot. Thus, lower entropy can imply lower Pass@8.