acceptodds
Withdrawn from ICLR 2027

Order-RFT: Gradient Alignment and Age-Aware Data Selection for Efficient Reinforcement Fine-Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement fine-tuning (RFT) improves the reasoning ability of large models, but rollout generation dominates its training cost. Data selection lowers this cost by prioritizing high-utility score samples, yet it faces an efficiency–adaptivity trade-off. Cached-score methods—whether static or dynamically updated—are cheap, but they lag behind the evolving utility in RFT. This risks a stale-utility exclusion problem: samples with low cached scores may remain unselected even after becoming useful, leaving their scores uncorrected. Conversely, refreshing scores for all samples consumes the rollouts that selection aims to save. To tackle this problem, we propose Order-RFT, a plug-and-play framework that estimates utility before rollout generation while tracking evolving utility through scheduled re-examination. It combines two complementary components: (i) a historical gradient-alignment estimator that approximates each sample’s marginal contribution to policy improvement; and (ii) a linear age bonus mechanism that prioritizes both cached utility and time since the last observation, enabling samples with cached scores to be reassessed. We establish approximation error bounds for the estimator and re-examination guarantees for the age mechanism. Experiments on several benchmarks show that Order-RFT achieves higher matching with refreshed sample-utility rankings and outperforms both static and dynamic baselines.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Reinforcement fine-tuning, large language models, data pruning, data selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_vsh4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a method for dynamic data selection in reinforcement finetuning (RFT) called Order-RFT. Order-RFT assigns a score to each data point based on its staleness and its gradient direction’s alignment with the EMA of the student’s gradient direction from the last batch. The method is novel and well motivated by the fact that existing methods take place after expensive rollout generation or are static. The paper presents experiments in the math domain across two model scales (1.5B and 7B) and four standard competition math benchmarks. The results show that Order RFT outperforms baselines. The paper also presents several theoretical results and proofs as to the influence of different components of their algorithm such as age.

Critical strengths

The motivation is strong, and the paper even verifies from the beginning with controlled experiments in the introduction that reusing cached scores does pose a significant problem. The method is presented with good theoretical discussion and positive results across a broad set of math benchmarks.

Critical weaknesses

Key details for reproducibility are missing.

  • Section 4 says optimizer, response-sampling settings, maximum generation length, etc. are shared but does not actually identify the optimizer, response-sampling settings, or maximum generation length.
  • The reproducibility statement on p. 10 claims that training settings and software versions are in the appendix; the appendix actually contains only theoretical derivations, not an experimental configuration section. It is claimed that all methods use the same rollout budget, but the actual total rollout budget is never stated.
  • There is no discussion of the training dataset beyond that DeepScaleR is used; details on the total size of the training set used, any filtering, etc. is missing
  • The “normalized projected alignment” approach in the ablations is never formally defined, which is concerning as that is apparently what is actually used in Order-RFT.

Evaluation has two issues:

  1. A key strong baseline appears to be missing. Song et al. in FastCuRL showed that on the DeepScaleR dataset, doing static data selection based on prompt length does pretty well; their performance is comparable on the same benchmarks to Order-RFT’s with the same 1.5B base model although they have differences in training procedure. A baseline with prompt length based data selection would strengthen this paper.
  2. No variance estimates for the main results, making it difficult to understand whether the gain from Order RFT is significant.

Song, Mingyang, Mao Zheng, Zheng Li, Wenjie Yang, Xuan Luo, Yue Pan, and Feng Zhang. "FastCuRL: Curriculum Reinforcement Learning with Stage-wise Context Scaling for Efficient Training R1-like Reasoning Models." In EMNLP (Findings), pp. 8856-8866. 2025.

Questions for authors
  • Have you tried any static filtering approaches based on prompt length?
  • What is “normalized projected alignment score”?
  • Are you able to release more details about your setup such as optimizer choice, response-sampling settings, maximum generation length, etc.?
  • How does the theory justify the cached ranking actually used to select training examples? Proposition 3.3 bounds only freshly computed scores, while Theorem 3.4 addresses refresh delays without bounding how inaccurate cached scores become between refreshes.
  • Are you able to provide any variance estimates for the main results?
Reviewer_LK1MOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a data selection framework called Sequential Reinforcement Fine-tuning, aiming to address the problem of outdated utility exclusion caused by cache fractions during reinforcement fine-tuning. This method combines historical gradient alignment estimation with a linear age reward mechanism. However, after careful review, I believe that the core idea and method design of this work are highly dependent on existing dynamic data importance measurement and related reinforcement fine-tuning data selection research, lacking substantive theoretical or methodological innovation. It is a clearly incremental work, and therefore, it is recommended to reject its acceptance.

Critical strengths
  1. The structure of the paper is clear and the logic is coherent. It provides a detailed background review on the trade-off issue between computational cost and utility assessment in the process of fine-tuning data selection.

  2. The experimental part offers abundant comparative data, covering the performance under different computational budgets. The charts are presented in an intuitive manner, which helps readers understand the surface advantages of the method in specific mathematical reasoning tasks.

Critical weaknesses
  1. The historical gradient alignment estimator proposed in the paper essentially is a direct adoption and fine-tuning of the existing dynamic data importance measurement methods. The dynamic data importance measurement has already proposed sample importance assessment based on the current batch gradient alignment. This paper merely replaces it with an exponential moving average of historical gradients. This modification is an extremely basic engineering adjustment and does not propose a new learning paradigm or profound algorithmic insights.

  2. Introducing time decay or age penalty to break the stale state of the cache has long been a mature and common heuristic strategy in the fields of machine learning data resampling and curriculum learning. Simply superimposing it on the existing gradient alignment scores does not constitute a new method with independent contributions. It is more like a patch added to fix the flaws of the basic method.

  3. Although the paper provides error boundary proofs, these proofs are highly dependent on strong assumptions such as gradient Lipschitz constants, and its theoretical conclusions are mainly a repetitive derivation of the existing sequential sampling framework, and does not provide new theoretical guarantees for the specific strategy evolution dynamics of reinforcement fine-tuning.

  4. The experiments are only conducted on narrow tasks with clear binary correctness rewards such as mathematical reasoning, and fail to prove the effectiveness of this method in open-domain generation tasks that require complex fine-grained reward model evaluation. Considering the nature of its method as a simple concatenation of existing technologies, its generalization ability in more extensive scenarios is seriously questionable.

Questions for authors

See weakness

Reviewer_Jn2rOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Order-RFT, a prompt-selection method for reinforcement fine-tuning that combines cached gradient-alignment scores with an age bonus before rollout generation. The age bonus encourages re-examination of prompts that might otherwise remain excluded because of stale scores. Experiments on mathematical reasoning report improvements at both 1.5B and 7B scales. However, the connection between the analyzed scoring rule and the evaluated implementation, as well as the reliability of the reported gains, requires clarification.

Critical strengths
  • Addresses a practical limitation of cached selection through a simple re-examination mechanism.

  • Includes useful ablations against age-only selection, random exploration, and periodic refresh.

  • Reports improvements across both model scales and all four benchmarks.

Critical weaknesses
  • The connection between the analyzed method and the evaluated implementation is unclear. The formal analysis considers raw EMA gradient alignment, whereas the best-performing variant in Table 2 uses normalized projected alignment. This modification improves pass@1 by 0.7 percentage points, equal to the gain from adding raw alignment to age-only selection. However, the normalization and projection procedures, their individual contributions, and whether this variant is used in Table 1 are insufficiently specified. This ambiguity complicates both attribution of the gains and assessment of the theory’s applicability to the evaluated method.

  • The robustness of the reported gains is insufficiently established. Average improvements over DIEM are approximately 1.0 and 1.2 percentage points for the 1.5B and 7B models, respectively. Although Figure 1 reports averages over three runs, Tables 1 and 2 provide neither uncertainty estimates nor an explicit number of independent training runs. Given the modest margins, training-seed variability and evaluation uncertainty are needed to assess whether these improvements are reliable.

Questions for authors
  • Which scoring variant is used in Table 1? How are normalization and projection implemented, what does each contribute, and which theoretical results remain applicable to the resulting score?

  • How many independent training runs support Tables 1 and 2? Could the authors report variation across training seeds and evaluation uncertainty for both the main comparisons and ablations?

Reviewer_VwS2OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Order-RFT first samples a uniform candidate window and then selects the top-qq prompts using cached historical gradient-alignment scores and a linear age bonus. It generates responses only for the selected prompts, whose gradients are used to update the policy, the historical reference, and the corresponding cached scores.

The paper presents an approximation-error bound for a fresh raw alignment score relative to a one-step, full-window GRPO surrogate influence quantity, and claims re-examination guarantees for the age mechanism. Experiments on two DeepSeek-R1-Distill-Qwen model scales and four mathematical reasoning benchmarks report improvements over static selection baselines and DIEM.

Critical strengths
  1. The motivation is clear and practically meaningful. The dependency between selection and score refresh can create an exclusion loop in which low cached scores prevent the observations needed to correct them.

  2. Pre-rollout selection has a direct computational motivation: selecting prompts before generating responses can avoid spending rollout resources on candidates that would subsequently be discarded.

  3. The evaluation includes two model scales, four mathematical reasoning benchmarks, and controls such as age-only selection, alignment without age, uniform exploration, and periodic refresh. The main-table gains are directionally consistent. The paper also appropriately distinguishes the GRPO surrogate from the correctness-reward objective, the full-window influence quantity from the actual top-qq update, and coverage from convergence to the uniform-pool objective.

Critical weaknesses

1 The best-performing scorer is undefined, leaving the relationship between the analyzed method and the experimental results unclear

Equations (8)-(9) and Algorithm 1 define a raw EMA alignment score and its clipped value:

\qquad Y_i(t)=\operatorname{clip}_{[-B_u,B_u]}\bigl(u_t(i)\bigr).$$ However, Table 2 reports "Normalized projected alignment + age" as the best-performing variant, with a revisit median of 12, coverage of 94%, and an "Avg. pass@1" value of 86.2. The accompanying text refers to the variant achieving these results as the full method. The defined raw EMA alignment plus age instead obtains 16, 89%, and 85.5, respectively. I could not find a definition of the normalized projected scorer anywhere in the provided 18-page manuscript. The projection space or matrix construction, dimensionality, normalization rule, and treatment of zero gradients are unspecified. It is also unclear which scorer is used in Table 1 and the two Order-RFT curves in Figure 1. Consequently, the manuscript does not establish whether the method analyzed in Section 3 is the method responsible for the main empirical gains. The authors need to define the transformations and explain whether Proposition 3.3 extends to the implemented scorer, including their effect on the relevant scores and rankings. The actual values of $\lambda$ and $B_u$, and the score-scale calibration procedure required by Equation (14), are also missing. The paper states that the normalized pilot's $\lambda=1$ is not a raw-score default. Although the reproducibility statement says that numerical training details and software versions are provided in the appendix, Appendices A-C do not supply a complete experimental configuration. Finally, obtaining per-prompt gradient alignments is part of the central efficiency claim. An aggregate batch gradient alone does not provide the individual inner products in Equation (9). The authors should describe their implementation, for example through per-sample or microbatch differentiation, directional derivatives, or projected approximations, and report the additional time and memory required for scoring and maintaining the EMA. Such scores may be computed efficiently without storing every full per-prompt gradient, but the absence of extra candidate rollouts does not establish negligible scoring overhead. **2 The proof of the re-examination guarantee contradicts the main-text argument and itself** Theorem 3.4, Equations (16)-(17), concerns the waiting time until a target prompt is actually selected and its cache is refreshed. For the bound in Equation (17), which does not restrict the old-competitor cohort, the main text argues that each target-containing window that fails to select the target instead selects and refreshes $q$ old competitors. Appendix C.4 explicitly rejects this charging argument and gives only the geometric waiting time for entry into the candidate window. There is also a direct inconsistency within Appendix C.4. To make the time indices explicit, Equation (14) defines the priority as $$p_t^i=\widehat{u}_t^{\\,i}+\beta\bigl(t-\tau_t^i\bigr), \qquad \beta=\frac{\lambda q}{n}.$$ Let $\tau_i$ be the target's fixed last-selection time while it remains unselected, so that $\tau_t^i=\tau_i$. Under $|\widehat{u}_t^{\\,k}|\le B_u$ and $\beta>0$, define $$h=\left\lceil\frac{2B_u}{\beta}\right\rceil+1.$$ At a subsequent selection step, any competitor with $\tau_t^j\ge\tau_i+h$ satisfies $$\begin{aligned} p_t^i-p_t^j &=\widehat{u}_t^{\\,i}-\widehat{u}_t^{\\,j} +\beta\bigl(\tau_t^j-\tau_i\bigr)\\\\ &\ge -2B_u+\beta h\\\\ &>0. \end{aligned}$$ Thus a competitor refreshed at or after the cutoff cannot subsequently outrank the still-unselected target. Lines 918-920 state this property, but lines 924-925 then argue that competitors can repeatedly acquire fresh timestamps and continue to block the target. That assertion contradicts the priority relation above. Candidate-window entry does not itself generate a rollout or refresh the cache. Therefore, its expected waiting time $n/s$ cannot be used as the expected waiting time for a fresh utility observation. The example on page 18 describes the expected five iterations until a window visit as an "observation" while acknowledging that the target need not survive top-$q$ selection. This confuses window entry with selection and cache refresh. As written, Appendix C.4 does not establish the conditional bound in Equation (16) or the binomial-tail bound in Equation (17). The corresponding use of these bounds in Corollary 3.5 needs to be checked. This is a defect in the supplied proof; it does not demonstrate that the theorem itself is false. The main-text charging argument appears amenable to formalization under the stated bounded-score and positive-age assumptions. A separate gap concerns the introduction's claim that the composite ranking error is bounded by the sum of two terms. Proposition 3.3 bounds the approximation error of a fresh raw score, whereas Theorem 3.4 bounds refresh delay. Combining them into an error bound for the implemented cached, age-adjusted ranking requires an additional argument relating utility drift, cache staleness, and ranking error. The authors should provide that result or narrow the claim. **3 The DIEM batch definition and resource accounting are insufficiently specified** The experimental setup states that methods selecting $q$ prompts are compared under matched generation budgets and that methods observing all $s$ prompts are charged for additional rollouts. Table 1 states that all methods use identical rollout budgets. These statements can be compatible, but the manuscript does not explain how the comparison is implemented. In particular, does DIEM observe $q=128$ prompts or the full $s=512$ candidate window? With $G=8$, the two possibilities generate $qG=1024$ or $sG=4096$ responses per generation batch, respectively. Neither batch definition should be assumed from the current description. If DIEM observes the full window, the authors should explain how generation-batch counts, update-batch sizes, and optimization steps are adjusted to match the cumulative budget. If it observes $q$ prompts, they should explain the source of the claimed generation savings under that comparison protocol. The introduction also claims comparable performance using approximately 80% of DIEM's rollout budget. Figure 1(d), however, labels $0.8\times$ as a time-cost ratio relative to the cached baseline, with DIEM at approximately $1.05\times$ and the other Order-RFT curve at approximately $1.02\times$. Time cost, generated-response count, and generated-token count are distinct measures. The plotted time ratio does not by itself substantiate the claimed rollout-budget ratio. The configurations corresponding to the two Order-RFT curves, with endpoint scores around 86.2 and 85.2, and their relationship to Tables 1 and 2 also need to be identified. For the key comparisons, please report cumulative generated responses, generated tokens, optimization updates, and GPU time, including selection and scoring overhead. Please also state whether static-baseline preprocessing and warmup costs are included. Quality at matched cumulative generation budget and resources needed to reach comparable quality should be reported separately. **4 The reported gains and mechanism metrics lack sufficient statistical and operational detail** Table 1 reports improvements over DIEM of approximately 0.6-1.6 percentage points. These may be meaningful, but Tables 1 and 2 do not specify the number of training seeds or provide cross-run statistics or uncertainty estimates. Figure 1's statement that its curves are averaged over three runs does not establish the protocol used for the tables. For small evaluation sets such as AIME24 and AMC23, please specify the decoding protocol, the number of generations per problem, and how pass@1 is estimated. These details are needed to assess the reliability of the reported differences under training and evaluation randomness; their absence does not imply that the gains are insignificant. The operational definition of "fresh utility" in Figure 1 is unclear. Is it the counterfactual influence in Equation (7), a fresh full-window gradient-alignment proxy, or another quantity? These alternatives support different interpretations of the estimator-error and ranking-correlation results. Table 2 likewise does not specify how "stale-useful" samples are identified, the coverage horizon, the unit of the revisit median, the treatment of samples not revisited within the horizon, or the evaluation sets included in "Avg. pass@1". The table includes uniform-exploration and periodic-refresh controls, but their exploration ratios, refresh periods, tuning procedures, and generation-cost accounting are missing. These omissions prevent the reader from evaluating the claimed advantage of age-based scheduling over simpler exploration and refresh strategies. Clarifying the configurations, metric definitions, and statistics for the existing experiments would address this concern without requiring a substantially expanded study.
Questions for authors

Q1. Please define the normalized projected scorer and identify the scorer used in Table 1, every Table 2 variant, and each Order-RFT curve in Figure 1. How does Proposition 3.3 apply to the implemented scorer? Please provide λ\lambda, BuB_u, the score-calibration procedure, and the time and memory overhead of computing alignments and maintaining the EMA.

Q2. Please reconcile Theorem 3.4 with Appendix C.4 and complete the proofs of Equations (16)-(17) for actual selection and cache refresh. Please check Corollary 3.5 accordingly. Which result establishes the claimed composite ranking-error bound for cached, age-adjusted priorities?

Q3. What batch does DIEM observe, and how are cumulative generated responses, tokens, updates, and training time matched or accounted for? What resource measure and reference configuration does the "80%" claim use? How do the Figure 1 configurations correspond to Tables 1 and 2?

Q4. Please provide the training-seed statistics and evaluation protocol for the existing main experiments and key ablations, define the mechanism metrics, and document the exploration and periodic-refresh parameters and their costs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.