- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates a tension in Dynamic Fine-Tuning: improved single-sample accuracy can coincide with reduced high-budget pass rates. It analyzes local softmax updates and proposes Reference-Aware Dynamic Fine-Tuning (RA-DFT), which uses frozen base-model token probabilities to modulate detached cross-entropy weights and retain a recovery floor for reference-supported tokens. Experiments on five models and six mathematical reasoning benchmarks compare SFT, DFT, DFT with KL anchoring, and RA-DFT. The method improves benchmark-averaged pass@k and majority voting over DFT, while token-level diagnostics and ablations support the intended selective preservation of alternative continuations.
Critical strengths
-
The paper addresses a consequential evaluation failure with consistent empirical improvements. Table 2 shows that RA-DFT improves benchmark-averaged pass@k over DFT for all five models and every reported sampling budget. The pass@64 gains range from 5.90 to 10.55 points, while majority-voting performance also improves. This is meaningful evidence that the method improves useful sampled reasoning rather than simply maximizing dispersion, and it strongly supports acceptance.
-
The intervention is well specified and receives several complementary checks. Equation (9), the recovery-floor ablation in Table 3, and the DFT+KL comparison help identify which parts of the objective matter. Appendix D's paired substitutions further support the distinction between low- and high-reference-probability positions. These diagnostics add explanatory value to a method that preserves standard supervised training and single-model inference.
Critical weaknesses
-
The incremental advantage over anchoring needs stronger statistical and tuning support. Appendix E fixes DFT+KL's coefficient at 0.05 and evaluates all methods at temperature 1.0, without reporting training-seed uncertainty or paired confidence intervals. Several RA-DFT–DFT+KL margins are small; on Qwen3-8B, RA-DFT's pass@64 is actually lower. The evidence for improvement over DFT is stronger than the evidence for a superior overall trade-off over a suitably tuned anchoring or decoding baseline.
-
The local analysis is suggestive rather than a full explanation of coverage changes. Equations (4)–(6) describe a direct logit update; ordinary cross-entropy also suppresses competing logits, so these identities alone do not explain the DFT-specific empirical pattern. Full-network updates and interactions across positions are not modeled. Appendix D provides useful supporting evidence, but its two-family substitution study reports relative effects without sample counts or uncertainty. This limits how firmly the paper can attribute benchmark gains to preserving viable reasoning branches.
Questions for authors
-
Clarification: How were the fixed KL coefficient and decoding temperature chosen? Are already-computed KL-strength or temperature comparisons available that show the low-/high-budget trade-off is not specific to these choices?
-
Clarification: How many independent training runs underlie Tables 1–3? Can paired uncertainty from existing evaluation outputs be reported, particularly for RA-DFT versus DFT+KL and the smaller low-budget differences?
-
Clarification: For Appendix D's substitution experiment, how many problems, positions, and continuations were used in each bucket, and what are the absolute reference/substituted accuracies? These details would determine how strongly the diagnostic supports the mechanism.