acceptodds
Under review as a conference paper at ICLR 2027

Reference-Aware Dynamic Fine-Tuning for Multi-Sample Reasoning

Abstract

Dynamic Fine-Tuning (DFT) has recently shown that a simple detached reweighting of supervised fine-tuning can substantially improve average sampled accuracy on reasoning tasks. We identify a complementary limitation: the same DFT checkpoints that improve average-sample performance can degrade high-k pass rates, sometimes falling below the base model under multi-sample decoding. We trace this tension to a local distributional effect of the DFT update. By increasing the demonstrated-token probability, DFT also contracts high-probability non-reference alternatives through the softmax; this sharpening is beneficial when alternatives are local noise, but can reduce pass@k when alternatives correspond to viable reasoning continuations. Motivated by this mechanism, we propose Reference-Aware Dynamic Fine-Tuning (RA-DFT), an objective that uses the frozen base model's teacher-forced probability as a local confidence signal. RA-DFT attenuates DFT-style sharpening at low-reference-probability positions, where alternatives are more likely to support multi-sample coverage, while retaining strong sharpening at high-reference-probability positions. The method preserves the standard supervised fine-tuning pipeline and only changes the token-level loss weight. Across five models and six mathematical reasoning benchmarks, RA-DFT consistently improves over DFT across the evaluated pass@k range and improves majority voting, while preserving the average-sample gains that make DFT a strong baseline. Compared with a DFT+KL anchoring baseline, RA-DFT yields a better overall low-k/high-k balance, suggesting an advantage of reference-conditioned selective sharpening over global base anchoring.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Supervised Fine-Tuning, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oTp6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates a tension in Dynamic Fine-Tuning: improved single-sample accuracy can coincide with reduced high-budget pass rates. It analyzes local softmax updates and proposes Reference-Aware Dynamic Fine-Tuning (RA-DFT), which uses frozen base-model token probabilities to modulate detached cross-entropy weights and retain a recovery floor for reference-supported tokens. Experiments on five models and six mathematical reasoning benchmarks compare SFT, DFT, DFT with KL anchoring, and RA-DFT. The method improves benchmark-averaged pass@k and majority voting over DFT, while token-level diagnostics and ablations support the intended selective preservation of alternative continuations.

Critical strengths
  1. The paper addresses a consequential evaluation failure with consistent empirical improvements. Table 2 shows that RA-DFT improves benchmark-averaged pass@k over DFT for all five models and every reported sampling budget. The pass@64 gains range from 5.90 to 10.55 points, while majority-voting performance also improves. This is meaningful evidence that the method improves useful sampled reasoning rather than simply maximizing dispersion, and it strongly supports acceptance.

  2. The intervention is well specified and receives several complementary checks. Equation (9), the recovery-floor ablation in Table 3, and the DFT+KL comparison help identify which parts of the objective matter. Appendix D's paired substitutions further support the distinction between low- and high-reference-probability positions. These diagnostics add explanatory value to a method that preserves standard supervised training and single-model inference.

Critical weaknesses
  1. The incremental advantage over anchoring needs stronger statistical and tuning support. Appendix E fixes DFT+KL's coefficient at 0.05 and evaluates all methods at temperature 1.0, without reporting training-seed uncertainty or paired confidence intervals. Several RA-DFT–DFT+KL margins are small; on Qwen3-8B, RA-DFT's pass@64 is actually lower. The evidence for improvement over DFT is stronger than the evidence for a superior overall trade-off over a suitably tuned anchoring or decoding baseline.

  2. The local analysis is suggestive rather than a full explanation of coverage changes. Equations (4)–(6) describe a direct logit update; ordinary cross-entropy also suppresses competing logits, so these identities alone do not explain the DFT-specific empirical pattern. Full-network updates and interactions across positions are not modeled. Appendix D provides useful supporting evidence, but its two-family substitution study reports relative effects without sample counts or uncertainty. This limits how firmly the paper can attribute benchmark gains to preserving viable reasoning branches.

Questions for authors
  1. Clarification: How were the fixed KL coefficient and decoding temperature chosen? Are already-computed KL-strength or temperature comparisons available that show the low-/high-budget trade-off is not specific to these choices?

  2. Clarification: How many independent training runs underlie Tables 1–3? Can paired uncertainty from existing evaluation outputs be reported, particularly for RA-DFT versus DFT+KL and the smaller low-budget differences?

  3. Clarification: For Appendix D's substitution experiment, how many problems, positions, and continuations were used in each bucket, and what are the absolute reference/substituted accuracies? These details would determine how strongly the diagnostic supports the mechanism.

Reviewer_ttR6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies an interesting tension in Dynamic Fine-tuning (DFT): DFT improves the average-sample preformance while degrade the high-k pass rate. To resolve this, the author proposes Reference-based reweighting method, which guide the DFT through taking frozen base model's prob as the local confidence signal, thereby attenunating sharpening selectively. The method is implemented and validated with multiple math benchmarks across few model families.

Critical strengths
  • Clear writing and well-motivated method. The paper is easy to follow and explain the problem clearly: DFT improves pass@1 but degrades high-k pass rates. The method follows logically from the observation, the local softmax analysis, and the "non-selective sharpening" framing.

  • Simple and low-overhead. RA-DFT changes only the per-token loss weight and keeps the standard SFT/DFT pipeline. It needs no extra data, reward model, or on-policy sampling. Base probabilities can be precomputed offline, and training costs 1.32× DFT, less than the DFT+KL baseline (1.43×).

  • Comprehensive experiments with key controls and detailed analysis. The evaluation covers five models from three families on six math benchmarks, with full pass@k and maj@k curves. DFT+KL serves as a global-anchoring control. Additional diagnostics include bucket-wise alternative mass, paired-substitution validation, on-policy canonical leakage, and component ablations.

Critical weaknesses
  • No variance or significance reporting across random seeds. All results come from a single training run. There are no standard deviations or confidence intervals. Results over multiple seeds (e.g., 3) with mean ± std are needed to show the improvements are statistically meaningful.

Missing sampling-temperature control. All evaluations use a fixed temperature of T = 1.0. DFT produces a visibly sharper distribution, so the simplest remedy for its high-k degradation would be to sample DFT at a higher temperature (e.g., T ∈ {1.2, 1.4}). Without this comparison, it is unclear whether RA-DFT offers benefits beyond what inference-time temperature tuning already provides. Pass@k curves for DFT across temperatures would directly address this.

Questions for authors

See the part "critial weakness". I suggest the author resolve the quesions regarding random seeds and temeperature.

Reviewer_RhR4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

RA-DFT addresses the degradation of high-k sampling performance observed after DFT. It reweights token-level cross-entropy using probabilities from the frozen initial model, combining confidence-based attenuation with a recovery floor. Experiments cover five models and six mathematical reasoning benchmarks.

Critical strengths
  • The trade-off between single-sample accuracy and multi-sample success is practically relevant.

  • The method is simple and consistently improves benchmark-averaged pass@k over DFT, with supporting diagnostics, ablations, and majority-voting results.

Critical weaknesses
  • The proposed mechanism is not sufficiently established. Equations (8)–(9) combine attenuation, a recovery floor, and a fallback rule, but their respective roles remain unclear. With (\alpha=0), removing attenuation still achieves 65.87 pass@64, compared with 67.24 for the full method and 59.02 for DFT. This suggests that the recovery floor may explain much of the improvement.

  • The reference-confidence assumption needs stronger validation. The frozen initial model’s probability is used as a proxy for whether alternatives are useful. The substitution experiments provide support, but sensitivity to reference-model quality and calibration remains unexplored.

  • Comparisons and statistical reporting are incomplete. Important related baselines such as iw-SFT and PSFT are missing, DFT+KL uses only one coefficient, and training-seed variability and confidence intervals are not reported.

Questions for authors
  • Can you better justify the specific weighting and fallback rules, and disentangle attenuation from recovery across multiple models?

  • How sensitive is performance to the choice and calibration of the frozen reference model?

  • Can you provide comparisons with iw-SFT, PSFT, and tuned DFT+KL, together with uncertainty estimates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.