- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces Learnable Reference Fine-Tuning (LRFT), a roll-out-free method for teaching models mathematical reasoning. The method combines a stochastic loss from a reference SFTed model with a ratio between the current policy model and the reference model, motivated by importance sampling. The authors demonstrate their results on various Qwen variants for mathematical reasoning, where their approach outperforms related work in most cases.
Critical strengths
- Well Written: In my opinion, the paper is well written and well motivated, making it easy to follow, and the method is straightforward.
- Promising Results: The results of the paper are very promising, and they back up the claims made in the paper. To the best of my knowledge, the baselines they compare themselves against are sensible.
Critical weaknesses
Major: -No confidence intervals: In order to get scientifically meaningful results, I would urge the authors to report some kind of confidence intervals around the reported results. I understand that running multiple seeds might not be feasible; in that case, I recommend reporting 95% bootstrapped confidence intervals. This would further strengthen the paper and help readers assess the significance of the results.
Minor:
- Different model Family: All the models here are tested on Qwen LLM families. While the results look promising so far, I would be interested to know whether this method also generalises to other model families (they may still be small models; no problem), or whether this is mainly an artefact of the Qwen series.
Questions for authors
Q1: Different Tasks: I understand that the focus of this paper is on mathematical reasoning. However, the proposed method seems very general. How does it generalise to other reasoning tasks, such as MMLU-Pro or other non-mathematical benchmarks? In my opinion, if you can show improvements over SFT not just in maths but more generally in instruction tuning/distillation, the paper would be more relevant.
Q2: Different mathematical perspective: To me, the proposed loss looks like a stochastically weighted combination of (1) the reference model with normal SFT loss and (2) the DPO loss with positive-only examples. I might be wrong here, but I would encourage the authors to have a closer look at this, as additional motivation may arise from this.
Q3: Hyperparameter search: While I appreciate the ablation experiment at the end, I wanted to enquire how the hyperparameters for the sampling, as well as, e.g. the learning rates, were found, also for the baselines.