- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper presents an offline method in which a language model is SFT’d on a reweighted dataset of on-policy samples. The authors propose and combine various reweighting methods.
Critical strengths
None
Critical weaknesses
The paper reads in a rather roundabout way, where it is not clear upfront that the method is fully offline, unlike the related work. Thus, it is unclear whether their proposed problem space “Language-model post-training increasingly reuses generated trajectories to allocate a limited optimization budget” is relevant. The paper does not compare their method to the base model itself or standard methods that can be applied to verifiable settings like GSM8K, such as SFT on the dataset filtered to successes (i.e. a single iteration of STaR [1]), more standard on-policy methods like GRPO [2], or newer off-policy RL methods [3] (which the paper appeared to pitch itself as from the introduction). Thus, it is hard to evaluate whether the proposed methods provide valuable insights over existing ones.
Moreover, all experiments are run using nonstandard settings (generation length of 512, greedy decoding), producing subpar evaluations ~66% as the best experimental result compared to the official base model evaluation at ~73% [4], which invalidate all of the results. It would also be helpful to see error bars on all results, as the numbers are fairly close in the ablations.
[1] https://arxiv.org/abs/2203.14465 [2] https://arxiv.org/abs/2402.03300 [3] https://arxiv.org/abs/2602.19362 [4] https://qwen.ai/blog?id=qwen2.5-llm
Questions for authors
None