acceptodds
Under review as a conference paper at ICLR 2027

Learnable Reference Fine-Tuning for Rollout-Free Mathematical Reasoning

Abstract

Reinforcement learning can improve mathematical reasoning in large language models, but large-scale online rollouts are expensive. We study rollout-free fine-tuning from verified solutions alone, where the absence of negative examples leaves the contribution of each demonstration difficult to calibrate. We propose Learnable Reference Fine-Tuning(LRFT), which jointly trains a target policy and a reference policy on verified demonstrations. The reference models the demonstration distribution, and target-to-reference likelihood ratios modulate token-level target updates. Unlike target-only self-weighting or fixed-reference regularization, LRFT learns its update scale from the verified data. Across four checkpoints from two Qwen families and five mathematical-reasoning benchmarks, LRFT achieves the highest mean Pass@1 among all evaluated methods for every checkpoint and the best performance in most reported Pass@k comparisons. Relative to DFT, it also shows lower measured KL divergence from the base policy and less polarized gold-token probabilities. These results suggest that a demonstration-fitted reference provides an effective rollout-free learning signal while limiting distributional shift.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Supervised Fine-Tuning, Offline Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LWHCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Learnable Reference Fine-Tuning (LRFT), a roll-out-free method for teaching models mathematical reasoning. The method combines a stochastic loss from a reference SFTed model with a ratio between the current policy model and the reference model, motivated by importance sampling. The authors demonstrate their results on various Qwen variants for mathematical reasoning, where their approach outperforms related work in most cases.

Critical strengths
  • Well Written: In my opinion, the paper is well written and well motivated, making it easy to follow, and the method is straightforward.
  • Promising Results: The results of the paper are very promising, and they back up the claims made in the paper. To the best of my knowledge, the baselines they compare themselves against are sensible.
Critical weaknesses

Major: -No confidence intervals: In order to get scientifically meaningful results, I would urge the authors to report some kind of confidence intervals around the reported results. I understand that running multiple seeds might not be feasible; in that case, I recommend reporting 95% bootstrapped confidence intervals. This would further strengthen the paper and help readers assess the significance of the results.

Minor:

  • Different model Family: All the models here are tested on Qwen LLM families. While the results look promising so far, I would be interested to know whether this method also generalises to other model families (they may still be small models; no problem), or whether this is mainly an artefact of the Qwen series.
Questions for authors

Q1: Different Tasks: I understand that the focus of this paper is on mathematical reasoning. However, the proposed method seems very general. How does it generalise to other reasoning tasks, such as MMLU-Pro or other non-mathematical benchmarks? In my opinion, if you can show improvements over SFT not just in maths but more generally in instruction tuning/distillation, the paper would be more relevant.

Q2: Different mathematical perspective: To me, the proposed loss looks like a stochastically weighted combination of (1) the reference model with normal SFT loss and (2) the DPO loss with positive-only examples. I might be wrong here, but I would encourage the authors to have a closer look at this, as additional motivation may arise from this.

Q3: Hyperparameter search: While I appreciate the ablation experiment at the end, I wanted to enquire how the hyperparameters for the sampling, as well as, e.g. the learning rates, were found, also for the baselines.

Reviewer_jdsyOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Learnable Reference Fine-Tuning (LRFT), a rollout-free method for improving mathematical reasoning using only verified demonstrations. The method jointly trains a target policy and a learnable reference policy, using their token-level likelihood ratios to reweight the target's updates. To prevent the two policies from evolving identically, the authors introduce stochastic Beta mixing. Experiments on four Qwen checkpoints across five mathematical reasoning benchmarks show improvements over SFT, DFT, and ASFT, with smaller measured distributional shifts than DFT.

Critical strengths
  • Interesting problem. Improving reasoning capabilities without expensive online rollouts is an important and practical research direction.
  • Simple and intuitive approach. Using a learned reference to modulate the contribution of positive demonstrations is an interesting extension of existing rollout-free fine-tuning methods.
  • Promising empirical results. LRFT achieves the highest average Pass@1 across all four evaluated checkpoints, with particularly notable improvements on Qwen2.5-Math-7B.
Critical weaknesses
  • (Main) Unclear whether the improvements reflect genuine learning. All experiments are conducted on Qwen models, which have been shown to improve substantially under spurious or uninformative reward signals (Spurious Rewards). This raises an important alternative explanation: the observed gains may primarily reflect better elicitation of reasoning capabilities already present in the base models, rather than the proposed reference-guided learning mechanism. Although the cited work studies a different training algorithm, evaluating LRFT on models that do not exhibit similar spurious-reward improvements would provide much stronger evidence for its generality.
  • Insufficient explanation of stochastic mixing. The proposed Beta mixing has the same expected gradient as fixed mixing for given policy parameters. Since fixed 0.5 mixing can keep the two identically initialized policies synchronized, its poor performance does not establish that token-level stochasticity is necessary. Simpler ways of breaking symmetry are not evaluated.
  • Potentially weak baseline tuning. SFT and DFT substantially degrade performance on some Qwen3 checkpoints, suggesting sensitivity to optimization settings. Without learning-rate sweeps or comparisons at matched KL divergence, it is difficult to determine how much of LRFT's improvement comes from its objective versus more conservative updates.
Questions for authors
  • Given the findings on spurious rewards in Qwen mathematical reasoning models, can the authors evaluate LRFT on model families such as Llama3 or OLMo2, where spurious rewards have not produced comparable improvements? How can we distinguish genuine learning from elicitation of existing capabilities?
  • Can the authors provide better-tuned SFT and DFT baselines, including learning-rate sweeps and comparisons at matched KL divergence from the base model?
  • What is the additional training cost of maintaining a learnable reference policy, and how does LRFT compare with the baselines under a matched training compute budget?
Reviewer_a6UbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Learnable Reference Fine-Tuning (LRFT), a rollout-free fine-tuning method for improving mathematical reasoning using only verified positive demonstrations. The method jointly trains a target policy and a learnable reference policy, where the reference is optimized through maximum likelihood and its token-level likelihoods are used to adaptively scale the target policy's updates. Additionally, token-wise Beta mixing is introduced to encourage distinct update patterns between the two policies. The authors evaluate LRFT on four Qwen checkpoints across five mathematical reasoning benchmarks, comparing it against several supervised and reference-based fine-tuning baselines. Experimental results demonstrate improvements in average Pass@1 and competitive Pass@k performance.

Critical strengths

Simple and practical method design: LRFT introduces a relatively simple optimization framework by using a learnable reference policy to adaptively scale token-level updates. The method relies only on standard teacher-forced training with verified demonstrations, making it straightforward to integrate into existing fine-tuning pipelines.

Critical weaknesses
  1. Insufficient justification and validation of the core mechanism: The target-to-reference likelihood ratio is motivated by importance sampling, but its use as an optimization objective remains largely heuristic, since the training data are not sampled from the learned reference policy. Moreover, the experiments do not sufficiently isolate the contribution of the learnable reference policy. Although the ablation results support the effectiveness of token-wise Beta mixing, its proposed gradient-decorrelation mechanism lacks direct empirical validation. These limitations weaken the justification for the key methodological innovations.

  2. Limited experimental validation and computational analysis: The experiments are restricted to mathematical reasoning on four Qwen checkpoints. Some improvements over strong baselines are relatively small, making their robustness uncertain. Furthermore, LRFT requires maintaining two trainable policies, but the paper lacks a systematic comparison of training costs and computational efficiency against simpler baselines, leaving its training-efficiency advantages insufficiently established.

Questions for authors
  1. Theoretical justification: Since the training data are not sampled from the learned reference policy, what justifies using the target-to-reference likelihood ratio as an optimization objective? Can the authors provide further theoretical and empirical evidence explaining why this ratio should improve upon standard SFT?

  2. Necessity of the learnable reference: Can the authors provide a controlled ablation comparing the learnable reference with a frozen or lagged reference under the same ratio-guided objective? This would help clarify whether the improvements specifically arise from learning the reference policy.

  3. Role of Beta mixing: Although the ablation results demonstrate the effectiveness of token-wise Beta mixing, can the authors provide direct evidence that it reduces gradient correlation or prevents the target and reference policies from evolving synchronously?

  4. Robustness and computational cost: Can the authors report results across multiple training seeds, particularly for cases where improvements over ASFT are small? Additionally, how does LRFT compare with SFT and DFT in terms of training time and GPU memory consumption?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.