- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a method for dynamic data selection in reinforcement finetuning (RFT) called Order-RFT. Order-RFT assigns a score to each data point based on its staleness and its gradient direction’s alignment with the EMA of the student’s gradient direction from the last batch. The method is novel and well motivated by the fact that existing methods take place after expensive rollout generation or are static. The paper presents experiments in the math domain across two model scales (1.5B and 7B) and four standard competition math benchmarks. The results show that Order RFT outperforms baselines. The paper also presents several theoretical results and proofs as to the influence of different components of their algorithm such as age.
Critical strengths
The motivation is strong, and the paper even verifies from the beginning with controlled experiments in the introduction that reusing cached scores does pose a significant problem. The method is presented with good theoretical discussion and positive results across a broad set of math benchmarks.
Critical weaknesses
Key details for reproducibility are missing.
- Section 4 says optimizer, response-sampling settings, maximum generation length, etc. are shared but does not actually identify the optimizer, response-sampling settings, or maximum generation length.
- The reproducibility statement on p. 10 claims that training settings and software versions are in the appendix; the appendix actually contains only theoretical derivations, not an experimental configuration section. It is claimed that all methods use the same rollout budget, but the actual total rollout budget is never stated.
- There is no discussion of the training dataset beyond that DeepScaleR is used; details on the total size of the training set used, any filtering, etc. is missing
- The “normalized projected alignment” approach in the ablations is never formally defined, which is concerning as that is apparently what is actually used in Order-RFT.
Evaluation has two issues:
- A key strong baseline appears to be missing. Song et al. in FastCuRL showed that on the DeepScaleR dataset, doing static data selection based on prompt length does pretty well; their performance is comparable on the same benchmarks to Order-RFT’s with the same 1.5B base model although they have differences in training procedure. A baseline with prompt length based data selection would strengthen this paper.
- No variance estimates for the main results, making it difficult to understand whether the gain from Order RFT is significant.
Song, Mingyang, Mao Zheng, Zheng Li, Wenjie Yang, Xuan Luo, Yue Pan, and Feng Zhang. "FastCuRL: Curriculum Reinforcement Learning with Stage-wise Context Scaling for Efficient Training R1-like Reasoning Models." In EMNLP (Findings), pp. 8856-8866. 2025.
Questions for authors
- Have you tried any static filtering approaches based on prompt length?
- What is “normalized projected alignment score”?
- Are you able to release more details about your setup such as optimizer choice, response-sampling settings, maximum generation length, etc.?
- How does the theory justify the cached ranking actually used to select training examples? Proposition 3.3 bounds only freshly computed scores, while Theorem 3.4 addresses refresh delays without bounding how inaccurate cached scores become between refreshes.
- Are you able to provide any variance estimates for the main results?