acceptodds
Under review as a conference paper at ICLR 2027

Beyond Solvability: Task Learnability as a Static Prior for LLM RL Post-Training

Abstract

Reinforcement learning (RL) has become a central post-training paradigm for eliciting reasoning capabilities in large language models, yet uniform task sampling allocates compute without regard to differences in how tasks respond to optimization. Existing task-valuation methods mostly rely on snapshot-based signals such as current pass rate or reward, which estimate how solvable a task is under the current policy. However, tasks with similar current solvability can still differ substantially in how positively they respond to further training. We study this residual axis as task learnability: a regime-conditional measure of expected positive response to continued training under a fixed RL post-training regime. By analyzing per-task reward trajectories, we find that learnability is reproducible across independently sampled training contexts and predictive of downstream utility. To make this signal practical before training begins, we propose TrajVal, a probe-based estimator that approximates per-task learnability from a short probe run and two endpoint evaluations. TrajVal can be used either as a standalone static prior for task sampling or as a multiplicative prior for existing online schedulers. Experiments on mathematical and logical reasoning benchmarks across multiple model scales show that TrajVal improves data efficiency over uniform sampling and provides complementary gains when combined with online scheduling methods.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Task Learnability, Task Selection, RL Post-Training, Data Efficiency, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kV7rOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that task learnability, the expected positive response to continued RL, is distinct from current solvability. The authors propose TRAJVAL, which estimates learnability from a short probe run and two endpoint evaluations, then uses the score as a static sampling prior or as a multiplicative prior for online schedulers. On math and logic with Qwen3-1.7B/4B, TRAJVAL improves over uniform sampling and gives additional gains when combined with BOTS and GRESO.

Critical strengths
  1. Distinguishing learnability from snapshot solvability is an interesting and sensible framing for RL data allocation, and the paper shows that this signal is reproducible across independently sampled training contexts.

  2. TRAJVAL is lightweight and easy to plug in, and evaluating it both standalone and as a multiplicative prior for online schedulers (BOTS, GRESO) across math and logic benchmarks is convincing.

Critical weaknesses
  1. In Table 1, the performance improvements seem somewhat marginal. It is less convincing to conclude that the proposed method brings substantial gains, which may be because the score formulation is heuristic rather than a principled derivation.

  2. The novelty seems somewhat limited, and the advantage over existing methods remains unclear. Adaptive sampling and rollout allocation methods have been widely explored, but this paper does not empirically compare with many of them, and neglects several closely related works, not limited to [1-7].

[1] Small generalizable prompt predictive models can steer efficient RL post-training of large reasoning models

[2] Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models

[3] TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning

[4] Improving data efficiency for LLM reinforcement fine-tuning through difficulty-targeted online data selection and rollout replay

[5] Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse

[6] Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration

[7] Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocation

Questions for authors
  1. In Table 1, several gains over BOTS/GRESO are below 1 point. Can the authors clarify why these improvements are sufficient to support learnability as a distinct axis, especially after accounting for the extra probe overhead?

  2. Can the authors discuss the conceptual advantages over [1-7], and empirically compare with at least some of them?

  3. The score s(x)=(1−pre)⋅max⁡(0,Δ)s(x)=(1-\text{pre})\cdot\max(0,\Delta) is a heuristic product. Why is this form preferred over using Δ\Delta alone, or the slope/R2R^2 statistics already analyzed in Section 3?

5 comments

We sincerely thank Reviewer kV7r for the encouraging assessment and insightful literature pointers. Below, we clarify the statistical significance and overhead of our gains, provide a taxonomy and new empirical comparisons with [1–7] (specifically GPS [1] and DPS [2]), and justify our multiplicative score formulation.


1. W1 & Q1: Validating Learnability as a Distinct Axis — Statistically Significant Gains (p<0.05p<0.05), Up to +28%–44% Controlled Within-Bin Improvements, and Zero Extra Overhead When Plugged into Online Schedulers

  • Statistical Significance & Pronunciation on Hard Benchmarks (W1): While macro-averaged Best Acc gains over strong online baselines can appear numerically modest, paired tt-tests on Qwen3-1.7B Math confirm that improvements are statistically significant across all settings: standalone TrajVal over Random (p=0.011p=0.011), BOTS+TrajVal over BOTS (p=0.042p=0.042), and GRESO+TrajVal over GRESO (p=0.028p=0.028), all <0.05< 0.05. On challenging competition benchmarks (Tables 9–10), gains are substantial (e.g., +1.5% to +3.0% on AIME24/25 and OlympiadBench), while main-training steps to reach baseline peak (S2B) drop to 60.0%–95.0%.
  • Why Gains Over BOTS/GRESO Support a Distinct Axis After Accounting for Probe Overhead (Q1): BOTS and GRESO react to instantaneous or recent policy states: BOTS maintains online posteriors of current difficulty, while GRESO filters out zero-advantage prompts from recent rollouts. Neither models how positively a task responds to continued future optimization. Crucially, regarding extra probe overhead: once the static TrajVal prior is computed on a dataset (8.10 GPU-hours on 1.7B), plugging it into BOTS, GRESO, or transfer targets (4B) incurs zero re-estimation overhead and only 0.01% per-step sampling latency (Table 7). For instance, on Qwen3-4B Math, sharing the static prior with BOTS and GRESO cuts their main training time by 25.0% (saving 7.91 GPU-hours per run) while boosting Best Acc (+1.02% and +1.08%) and AUC.
  • Direct Controlled Proof of Orthogonality (§5.3, Table 2): Our primary proof that learnability is distinct from solvability is the controlled within-bin study (§5.3), which holds initial pass rate (pre\text{pre}) strictly identical between sub-pools. Across all six bin–domain pairs, high-Δ\Delta tasks systematically beat low-Δ\Delta tasks—boosting Low-pre Math Best Acc by +28.4% relative (0.338→0.4340.338 \rightarrow 0.434) and Low-pre Logic AUC by +44.5% relative (0.0946→0.13670.0946 \rightarrow 0.1367).

2. W2 & Q2: Conceptual Taxonomy of [1–7] & New Experiments on GPS [1] and DPS [2] — TrajVal Boosts DPS Accuracy by +1.94% and Cuts Training Steps by 35.3%

We thank you for pointing out [1–7], which we have studied carefully and will integrate into Related Work. While all seven study non-uniform compute allocation in RLVR, they intervene at distinct decision levels:

  1. Intra-Step Rollout Budget Allocation & Reuse (TRACE [3], AR3PO [5], DARS [6], Knapsack RL [7]): DARS [6], Knapsack RL [7], and AR3PO [5] decide how many rollouts nn to generate or reuse after a prompt is sampled, while TRACE [3] allocates budgets across root rollouts and prefix branches. Because TrajVal operates strictly at the inter-step task-sampling level (prioritizing which prompts enter the batch), it is orthogonal and can directly supply the root task-sampling distribution for these rollout-allocation frameworks.
  2. Online Solvability & Transition-Predictive Task Scheduling (GPS [1], DPS [2], DOTS [4]): GPS [1] (small predictive models), DPS [2] (dynamics-predictive state transitions), and DOTS [4] (difficulty-targeted selection with replay) adjust task-sampling probabilities online using historical rollout feedback. Conceptually, they belong to Dynamic Task-Value Scheduling (§2), targeting instantaneous or next-step difficulty/solvability. TrajVal instead provides an offline multi-epoch learnability prior that guides allocation from step 0—resolving the cold-start noise before online predictors accumulate sufficient rollout history.

New Empirical Comparisons with GPS [1] and DPS [2]: To verify this complementarity beyond BOTS and GRESO, we evaluated GPS [1] and DPS [2] both standalone and composed multiplicatively with TrajVal on Qwen3-1.7B Math:

MethodBest Acc (5-step mean) ↑\uparrowAUC ↑\uparrowS2B ↓\downarrow
Random (BASE-GRPO)0.39670.3620–
Standalone TrajVal (Ours)0.41050.373065.66%
GPS [1] (History-based difficulty predictor)0.40800.3617–
GPS [1] + TrajVal (Ours)0.40820.3709 (+0.92%)90.12%
DPS [2] (Next-step state-transition predictor)0.39530.3571–
DPS [2] + TrajVal (Ours)0.4147 (+1.94%)0.3764 (+1.93%)64.71%

Standalone TrajVal outperforms both standalone GPS (0.4105 vs. 0.4080) and DPS (0.3953). More importantly, composing TrajVal as a prior consistently accelerates and improves both online methods: it boosts GPS's AUC from 0.3617→0.37090.3617 \rightarrow 0.3709 (90.12% S2B), and substantially elevates DPS by +1.94% Best Acc (0.3953→0.41470.3953 \rightarrow 0.4147) and +1.93% AUC while reaching DPS's peak in 64.71% of the steps. This confirms that offline learnability provides vital complementary signal even alongside explicit online transition models.

3. Q3: Why the Multiplicative Score Beats Δ\Delta-Only, Additive, and Dense Slope/R2R^2 — A Logical-AND Gate That Avoids 10×10\times Checkpoint Evaluation Overhead

We agree s(x)=(1−pre(x))max⁡(0,Δ(x))s(x) = (1 - \text{pre}(x))\max(0, \Delta(x)) is an intuitive proxy rather than a theoretical optimum, but its multiplicative endpoint form is specifically preferred over Δ\Delta-only, additive, and dense slope/R2R^2 alternatives for clear empirical and computational reasons:

DomainScoring VariantFormula s(x)s(x)S2B ↓\downarrowBest Acc ↑\uparrowAUC ↑\uparrow
MathAdditive Variant12[(1−pre)+max⁡(0,Δ)]\frac{1}{2}[(1-\text{pre}) + \max(0,\Delta)]90.91%0.40010.3592
MathΔ\Delta-only (Table 3)max⁡(0,Δ(x))\max(0, \Delta(x))85.86%0.40860.3660
MathTrajVal (Product)(1−pre)max⁡(0,Δ)(1-\text{pre})\max(0,\Delta)65.66%0.41050.3730
LogicΔ\Delta-only (Table 3)max⁡(0,Δ(x))\max(0, \Delta(x))78.26%0.18640.1641
LogicTrajVal (Product)(1−pre)max⁡(0,Δ)(1-\text{pre})\max(0,\Delta)73.91%0.19040.1663
  1. Why Product beats Δ\Delta-only and Additive: Δ\Delta-only over-prioritizes tasks with high probe gains that are already nearly mastered (e.g., 0.75→0.950.75 \rightarrow 0.95), which quickly saturate during main RL. Conversely, an Additive rule allows high headroom (1−pre=1.01-\text{pre}=1.0) to compensate for zero learning response (Δ=0\Delta=0). The Multiplicative form acts as a conjunctive (logical-AND) gate requiring both positive responsiveness and remaining headroom, slashing Math S2B by 20.2 points over Δ\Delta-only (85.86%→65.66%85.86\% \rightarrow 65.66\%) and 25.3 points over Additive.
  2. Why Endpoint Δ\Delta is preferred over Section 3's Slope and R2R^2: Endpoint Δ\Delta captures the direction and magnitude of reward change using only two full-pool evaluations. Fitting trajectory-level slope and R2R^2 requires evaluating intermediate checkpoints: on our 17k-task pool, 20 passes would cost approximately 43 h at the measured 2.15 h per pass, compared with 4.3 h for two endpoint passes. Figures 4 and 8 show that endpoint-derived profiles largely preserve the overall structure of full-trajectory behavior. Together with the downstream gains over Δ\Delta-only scoring, this supports using the inexpensive endpoint signal, combined with headroom, for task allocation.
Reviewer_vWBYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies task selection for RL post-training and distinguishes task learnability from current solvability. Learnability measures how positively a task responds to continued training under a fixed regime. The authors show that tasks with similar initial performance can have different training trajectories and that these differences are somewhat reproducible.

The paper proposes TrajVal, which estimates learnability using a short probe run and assigns each task a sampling weight based on its observed improvement and remaining headroom. TrajVal can be used alone or combined with existing online schedulers. Experiments on math and logic reasoning show improved convergence speed and modest improvements in final performance.

Critical strengths
  1. The solvability vs. learnability distinction is intuitive and practically relevant. The within-solvability-bin analysis provides useful evidence that initial performance does not fully capture downstream training value.

  2. The empirical evaluation is reasonably thorough. The paper evaluates multiple domains, model scales, and scheduling methods, with several useful ablations and analyses.

Critical weaknesses
  1. The notion of learnability feels somewhat circular, and the conceptual novelty is limited. Learnability is defined through improvement during training, and TrajVal estimates it using improvement during a smaller probe training run. It is therefore not surprising that tasks improving during the probe tend to be useful for subsequent training. Overall, this seems close to using short-horizon training dynamics as a data selection signal.

  2. TrajVal is fairly heuristic and the empirical gains are modest. The score s(x)=(1−rˉearly(x))max⁡(0,Δ(x))s(x)=(1-\bar{r}_{\mathrm{early}}(x))\max(0,\Delta(x)) and subsequent weighting scheme are reasonable but lack strong justification. Most of the benefit appears in convergence speed, while improvements in final accuracy are generally small.

Questions for authors
  1. What experiment best distinguishes TrajVal from the simpler strategy of running a short probe and directly prioritizing tasks with positive reward improvement?

  2. When the probe run and endpoint evaluation costs are included, how much end-to-end compute does TrajVal actually save relative to uniform sampling and the online baselines?

  3. How sensitive are the results to the specific score $s(x)=(1-\bar{r}_{\mathrm{early}}(x))\max(0,\Delta(x)) $ compared with simpler alternatives such as ranking directly by Δ(x)\Delta(x)?

6 comments

We sincerely thank Reviewer vWBY for the thoughtful feedback. Below, we address your conceptual and empirical questions with new controlled ablations, statistical significance tests, and an end-to-end wall-clock compute breakdown.


1. W1: Resolving "Circularity" — 97% of Tasks Are Never Trained on During Probing, Yet Their Cross-Task Generalization Response Predicts Utility Beyond Solvability

We agree with your intuition: if a task's true training response were already known, prioritizing it would be natural. However, in RL post-training, this creates a chicken-and-egg dilemma: task selection aims to save RL compute, yet observing per-task dynamics ordinarily requires rollouts and policy updates across the full pool DD (17,398 tasks). TrajVal is non-circular for three reasons:

  1. Zero-Training Cross-Task Estimation for 97% of the Pool: TrajVal trains θprobe\theta_{\text{probe}} on only ∣P∣=512|P|=512 uniformly sampled tasks (≈3%\approx 3\% of DD). The remaining 97% of tasks (x∈D∖Px \in D \setminus P) are never trained on during probing; their gain Δ(x)=post(x)−pre(x)\Delta(x) = \text{post}(x) - \text{pre}(x) measures how positively they respond to policy updates induced by other tasks from the same distribution.
  2. Cross-Context Reproducibility: It is not guaranteed a priori that response to a small subset P1P_1 holds under an independent subset P2P_2 or full-pool training. Yet, §3.2 (Fig. 6) shows high per-task agreement across independent 512-task probes (Cohen's κ=0.879\kappa = 0.879 on Qwen3-1.7B, κ=0.872\kappa = 0.872 on Llama-3.2-3B).
  3. Orthogonality to Initial Solvability: In our controlled within-solvability-bin analysis (§5.3, Table 2), even when sub-pools have identical average initial pass rates (e.g., Avg pre=0.0016\text{Avg pre} = 0.0016 in Low-pre Math), the high-Δ\Delta sub-pool beats the low-Δ\Delta sub-pool by +28.4% relative Best Acc (0.338→0.4340.338 \rightarrow 0.434) and +44.5% relative AUC on Logic (0.0946→0.13670.0946 \rightarrow 0.1367).

Thus, our contribution lies in showing that regime-conditional learnability is a context-robust property orthogonal to static solvability, and can be cheaply estimated for an entire dataset via a 3% probe before main RL begins.


2. W2, Q1 & Q3: Score Justification & Sensitivity — Conjunctive Score Outperforms Positive-Only (1[Δ>0]\mathbb{1}[\Delta>0]), Δ\Delta-Only, and Additive Variants Across All Metrics

To directly answer Q1 (distinguishing TrajVal from simply prioritizing tasks with positive reward improvement) and Q3 (sensitivity to s(x)s(x) vs. simpler alternatives like ranking by Δ\Delta), we compare five scoring rules on Qwen3-1.7B under identical probe runs, soft-weighting (α=0.3,ϵ=0.05\alpha=0.3, \epsilon=0.05), and training budgets:

Scoring FormulationFormula s(x)s(x)Math Best Acc ↑\uparrowMath AUC ↑\uparrowMath S2B ↓\downarrowLogic Best Acc ↑\uparrowLogic AUC ↑\uparrowLogic S2B ↓\downarrow
Random (Baseline)Uniform0.39670.3620–0.18260.1544–
Positive-only (Q1)1[Δ(x)>0]\mathbb{1}[\Delta(x) > 0]0.40350.360989.19%–––
Additive Variant (W2)12[(1−pre)+max⁡(0,Δ)]\frac{1}{2}[(1-\text{pre}) + \max(0,\Delta)]0.40010.359290.91%–––
pre\text{pre}-only (Table 3)1−pre(x)1 - \text{pre}(x)0.40840.367585.86%0.19040.162579.08%
Δ\Delta-only (Q3, Table 3)max⁡(0,Δ(x))\max(0, \Delta(x))0.40860.366085.86%0.18640.164178.26%
TrajVal (Ours)(1−pre)max⁡(0,Δ)(1-\text{pre})\max(0,\Delta)0.41050.373065.66%0.19040.166373.91%
  1. Why Positive-only (1[Δ>0]\mathbb{1}[\Delta>0]) is insufficient (Q1): Binary selection treats a noisy marginal gain (0.0→0.060.0 \rightarrow 0.06) identically to a strong response (0.0→0.500.0 \rightarrow 0.50). Incorporating magnitude in Δ\Delta-only improves all three metrics.
  2. Why Δ\Delta-only is insufficient & Headroom is vital (Q3): Ranking purely by Δ\Delta over-allocates compute to nearly mastered tasks (pre=0.75→post=0.95\text{pre}=0.75 \rightarrow \text{post}=0.95) that quickly saturate during main RL. Multiplying by headroom (1−pre)(1-\text{pre}) in TrajVal cuts Math S2B by 20.2 points (85.86%→65.66%85.86\% \rightarrow 65.66\%) and improves all six Math/Logic metrics.
  3. Why Multiplicative beats Additive (W2): Under Additive, stubbornly unlearned tasks (pre=0,Δ=0\text{pre}=0, \Delta=0) still get a high score (0.50.5) purely from headroom. Multiplication acts as a logical AND gate—requiring both responsiveness and headroom—boosting Best Acc (0.4001→0.41050.4001 \rightarrow 0.4105) and S2B (90.91%→65.66%90.91\% \rightarrow 65.66\%). Meanwhile, probabilistic sampling with power compression (α∈{0.1,0.3,0.5}\alpha \in \{0.1, 0.3, 0.5\}, Table 11) ensures low sensitivity to fine-grained score fluctuations.
  4. Statistical Significance & Gain Size (W2): Paired significance tests confirm that Best Acc gains of standalone TrajVal over Random (p=0.011p = 0.011), BOTS+TrajVal over BOTS (p=0.042p = 0.042), and GRESO+TrajVal over GRESO (p=0.028p = 0.028) are all statistically significant (p<0.05p < 0.05), with up to +2.1% to +3.0% gains on competition math (AIME24/25, Tables 9–10).

3. Q2: End-to-End Compute vs. Uniform & Online Baselines — 8.10 h Upfront Cost Yields 15.7%–34.0% Net Wall-Clock Savings Under Multi-Run and Cross-Scale Reuse

We measure end-to-end wall-clock time on the same 8-GPU hardware (Math, 17k pool). TrajVal's upfront preparation takes 8.10 wall-clock hours total: 3.80 h for the 20-epoch probe (∣P∣=512|P|=512) + 4.30 h for two full-pool vLLM forward passes. (Note: BOTS also requires >6 h upfront inference from external 7B/32B models to initialize difficulty priors [Appendix A.5], but we conservatively charge 0 h prep to all baselines below).

Scale & ScenarioMethodBest Acc ↑\uparrowAUC ↑\uparrowS2B ↓\downarrowMain Train Time (to Base Peak)Upfront PrepTotal End-to-End Time
Qwen3-1.7BRandom / BOTS / GRESO39.67 / 40.42 / 40.28%.3620 / .3648 / .3621100%16.66 h0 h16.66 h
(Isolated 1-Epoch)Random + TrajVal (In-regime)41.05%0.373065.66%10.94 h (-34.3%)8.10 h19.04 h (+14.3%)
BOTS / GRESO + TrajVal40.81% / 40.83%0.3703 / 0.368195% / 90%15.83 h / 14.99 h0 h (shared)15.83 h / 14.99 h
Qwen3-4BRandom / BOTS / GRESO50.60 / 50.76 / 50.62%.4829 / .4767 / .4695100%27.23 h0 h27.23 h
(Prior Reuse)Random + TrajVal (Reused 1.7B)51.80%0.488966.00%17.97 h (-34.0%)0 h (reused)17.97 h (-9.26 h)
Random + TrajVal (In-regime 4B)51.90%0.493160.00%16.34 h (-40.0%)(4B probe)16.34 h main (-10.89 h)
BOTS / GRESO + TrajVal51.78% / 51.70%0.4863 / 0.479475.00%20.42 h (-25.0%)0 h (shared)20.42 h (-6.81 h)
  • Isolated 1-Epoch Small Run: On 1.7B Math (1 epoch = 16.66 h), the 8.10 h upfront cost exceeds the 5.72 h saved in main training (19.04 h vs. 16.66 h). Thus, we do not claim net wall-clock savings for a single, isolated 1-epoch small-model run—though TrajVal reaches a higher accuracy (41.05%) and AUC (0.3730) that baselines never attain. (Moreover, Fig. 9 shows probe rankings stabilize at ρ≥0.85\rho \ge 0.85 by epoch 14/20, allowing the 3.80 h probe cost to be cut by ~30%).

  • Net Savings via Scale, Online Plug-in & Amortization: (1) Cross-Scale Transfer (1.7B →\rightarrow 4B): Reusing the 1.7B prior on 4B costs 0 h additional prep and cuts 4B training time by 34.0% (saving 9.26 h). Across a joint 1.7B+4B workflow, total time drops from 16.66+27.23=43.89 h16.66 + 27.23 = \mathbf{43.89\text{ h}} to 8.10+10.94+17.97=37.01 h8.10 + 10.94 + 17.97 = \mathbf{37.01\text{ h}} (saving 6.88 h or 15.7% net wall-clock time). (2) Online Baselines: Sharing the static prior with BOTS and GRESO on 4B cuts their training time by 25.0% (saving 6.81 h per run) while boosting Best Acc by 1.02–1.08 percentage points. (3) Pool-Size Scaling: Because probe cost O(T∣P∣)O(T|P|) is fixed (∣P∣=512|P|=512), expanding ∣D∣|D| by 5×5\times shrinks the probe's relative exposure from ~59% to ~12% of one epoch.

Reviewer_t5g8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies task selection for LLM reinforcement learning. TrajVal combines a prompt's improvement in success rate with its initial headroom to assign sampling weights. It trains a probe on a small subset, then evaluates the full training pool before and after the probe training. The resulting weights stay fixed during main training and can be combined with an online sampler. Experiments on Math and Logic show improvements over several sampling baselines. Presentation: 3 - Clear and easy to follow. The motivation, oracle study, and practical method are presented clearly. Informativeness: 2 - Something interesting or useful. The analyses show that tasks with similar initial success rates can differ in training value. Soundness: 2 - Minor or fixable concerns. The results support the method, but the total cost and comparison with prior trajectory-based selection need further evaluation.

Critical strengths

S1. The oracle experiment and analysis support the motivation. On Qwen3-1.7B Math, the oracle subset uses about 41% of the 2,048-task diagnostic pool and matches full-pool performance in 3.6 times fewer main-training steps. Oracle weighting also improves accuracy from 38.7% to 41.0% at the same main-training budget (Figure 3). In Table 2, high-gain groups perform better in all six comparisons with matched average initial success rates. The analyses of probe subsets and later checkpoints help explain when the signal remains useful.

S2. Performance is consistently good in the main experiments. TrajVal improves Best Acc and AUC on both Math and Logic, for Qwen3-1.7B and Qwen3-4B (Table 1). The gains appear with uniform sampling, BOTS, and GRESO, averaged over two seeds. Its gains on top of BOTS and GRESO show that the prior can also help existing samplers. The ablations and reuse experiments further support the method.

Critical weaknesses

W1. The efficiency comparison leaves out the cost of computing the weights. Steps-to-Baseline counts main-training steps, but excludes probe training and the evaluations needed to score the training pool. Appendix B.5 discusses these costs, but does not report an end-to-end timing comparison. The 0.01% overhead in Table 7 covers only the sampling operation. Appendix B.4 uses a 20-epoch probe. If that also applies to the main experiments, the 512-task probe entails 10,240 prompt exposures, about 59% as many as the one-epoch Math main run (Table 4), before endpoint scoring. This is not a GPU-time estimate, but suggests that the omitted cost could be substantial. This matters for the comparison with pre-only in Table 3: the gain in Math peak accuracy is small, and Logic accuracy is tied. The clearer benefits are fewer main-training steps and higher AUC. A comparison at the same total compute budget is needed to judge whether the extra scoring cost pays off. It should include preparation costs for both methods and explain how reusing the weights changes the cost.

W2. The comparison with prior trajectory-based selection is incomplete. The paper cites LIMR, which already selects RL examples using reward trajectories, and notes that it requires full-pool training. However, there is no empirical comparison. The external baselines are uniform sampling, BOTS, and GRESO. The endpoint ablations help explain the score, but do not address this comparison. Since the authors emphasize replacing full-pool training with a small probe, a comparison with LIMR's selection rule under the same RL setup would help show what TrajVal gains in accuracy or total cost.

Questions for authors

Q1. After including probe training and scoring costs, how does total GPU time to reach the same accuracy compare with pre-only? Please include each method's preparation costs. One representative setting would be sufficient. If the weights are reused, please state how their cost is shared across runs. Q2. Could you compare with LIMR's selection rule in one representative setting, using the same RL setup and reporting accuracy and total GPU time? Q3. The gains on Llama are less consistent than on Qwen, and the 8B experiment only tests transfer of an existing prior. Could you provide results on a larger model or a MoE model, given that all evaluated models are dense?

4 comments

We sincerely thank Reviewer t5g8 for the constructive feedback and rigorous cost analysis. Following your suggestions, we conducted end-to-end wall-clock evaluations, benchmarked directly against LIMR, and performed a new in-regime experiment on Qwen3-8B.


W1 & Q1: End-to-End GPU Time vs. pre-only — Upfront Cost (8.1h) is Offset by Faster Main Training, Yielding 15%–39% Net Time Savings Under Reuse

You are completely right that Steps-to-Baseline (S2B) measures main-training steps and omits upfront prior construction. We report an end-to-end wall-clock evaluation on the exact same hardware below.

Preparation Cost Breakdown (1.7B, 17k Pool): Constructing the TrajVal prior takes 8.10 GPU-hours in total, comprising 3.80 h for the 20-epoch probe training (∣P∣=512|P|=512) and 4.30 h for the two full-pool forward inference passes (≈\approx 2.15 h per pass). Note that pre-only also requires one full-pool inference pass (2.15 h) to compute initial pass rates. For maximum transparency, the table below reports total wall-clock time to reach the random-sampling baseline peak both conservatively (charging 0 h prep to pre-only) and realistically (charging 2.15 h to pre-only on 1.7B):

ModelMethodBest AccS2BSteps to BaseMain Train TimePrior Prep TimeTotal Time (Conservative / Realistic)
1.7Bpre-only40.84%85.86%214.714.31 h0 h / 2.15 h14.31 h / 16.46 h
1.7BTrajVal (in-regime)41.05%65.66%164.210.94 h8.10 h19.04 h
4Bpre-only50.84%107.50%263.429.26 h0 h (if reused)29.26 h
4BTrajVal (reused 1.7B prior)51.80%66.00%161.717.97 h0 h (reused)17.97 h (-38.6%)

1. Isolated Single-Run (1.7B): When charging the full 8.10 h prep cost to a single 1-epoch run, TrajVal takes 19.04 h vs. pre-only's 14.31 h (conservative) or 16.46 h (realistic). Thus, we do not claim net wall-clock savings for an isolated, 1-epoch small-model run—though TrajVal still cuts main-training time by 23.5% (10.94 h vs. 14.31 h), improves Best Acc (+0.21%), and boosts AUC (0.3730 vs. 0.3675). (Note: As shown in Fig. 9, probe rankings already stabilize at ρ≥0.85\rho \ge 0.85 by epoch 14/20, allowing the 3.80 h probe cost to be further reduced by ~30%).

2. Amortization via Cross-Scale & Multi-Run Reuse (1.7B →\rightarrow 4B): The static 1.7B prior transfers directly to the 4B model with zero re-estimation cost, reaching the baseline peak in 17.97 h (saving 11.29 h or 38.6% of wall-clock time) while boosting Best Acc by +0.96% (51.80% vs. 50.84%). Across a joint development workflow of training 1.7B and 4B models where the prior is computed once:

  • Tpre-only=14.31+29.26=43.57 hT_{\text{pre-only}} = 14.31 + 29.26 = \mathbf{43.57\text{ h}} (conservative, 0 h prep) or 45.72 h\mathbf{45.72\text{ h}} (realistic)
  • TTrajVal=8.10 (one-time prep)+10.94+17.97=37.01 hT_{\text{TrajVal}} = 8.10\text{ (one-time prep)} + 10.94 + 17.97 = \mathbf{37.01\text{ h}}

Even under the most conservative accounting, TrajVal saves 6.56 hours (15.1% net compute) while achieving superior final accuracy on both scales. Furthermore, because the probe cost O(T∣P∣)O(T|P|) is independent of full pool size ∣D∣|D|, expanding ∣D∣|D| by 5×5\times shrinks the probe's relative exposure from ~59% to just ~12% of one epoch, and the one-time 8.10 h cost is similarly amortized when reused across multiple seeds or online schedulers (BOTS/GRESO).

W2 & Q2: Empirical Comparison with LIMR — TrajVal Cuts Preparation Cost by 4×4\times–33×33\times and Achieves +4.96% Higher Accuracy

LIMR faces a severe scaling bottleneck: collecting full-pool reward trajectories over 20 epochs on our 17k pool requires ~126 GPU-hours, whereas TrajVal's 512-task probe takes only 3.80 h (a 33.2×33.2\times reduction in probe training, decoupled from ∣D∣|D|).

For a direct, controlled comparison on Qwen3-1.7B (Math), we evaluate on the 2,048-task diagnostic pool D\mathcal{D} over 20 epochs. We run full 20-epoch training on D\mathcal{D} (15.2 h) to obtain LIMR's trajectories and select K=272K=272 tasks via its original threshold rule. For alignment, TrajVal estimates priors using the 512-task probe (3.8 h) and selects either the top-272 tasks (TrajVal Hard Subset) or reweights the full pool (TrajVal Soft Sampling):

MethodAllocation TypePrep CostActive PoolBest AccMain Time to Best Acc
Uniform (Full Pool)Uniform0 h2,04848.18%64h 16m
LIMRHard Subset15.2 h (Full Pool)27244.34%5h 05m
TrajVal (Hard Subset)Hard Subset3.8 h (Probe)27244.89%4h 19m
TrajVal (Soft Sampling)Soft Prior3.8 h (Probe)2,04849.30%25h 02m
  1. Superior Signal at 1/41/4 Prep Cost: Under the exact same 272-task hard-filtering regime, TrajVal outperforms LIMR in both Best Acc (44.89% vs. 44.34%) and convergence speed (4h 19m vs. 5h 05m), while requiring only 25% of LIMR's preparation time (3.8 h vs. 15.2 h).
  2. Breaking the Hard-Pruning Ceiling: Hard subset selection (both LIMR and TrajVal-Hard) discards 87% of tasks, sacrificing diversity and plateauing below the full-pool uniform baseline (48.18%). By retaining full-pool coverage via soft weighting, TrajVal (Soft Sampling) surpasses LIMR by +4.96% Best Acc (49.30% vs. 44.34%) and beats the uniform baseline while cutting main training time by 61.0% (25h 02m vs. 64h 16m).

Q3: Generalization to In-Regime Qwen3-8B & Llama Analysis — In-Regime 8B Prior Further Boosts Accuracy (+1.59%) and Speed (76.8% S2B)

Why Llama-3.2-3B Gains Are Less Uniform: Qwen3 undergoes a dedicated reasoning pre-training stage on synthetic math and code, whereas Llama-3.2-3B is a general-purpose base model. Consequently, Llama-3.2-3B starts with much lower initial pass rates on competition math/logic (Table 5), yielding sparser positive rewards during the short probe run. This increases endpoint estimation variance and narrows the optimization headroom.

New In-Regime Qwen3-8B Experiment: While training large MoE models via RL exceeds our 8-GPU node budget during the rebuttal window, we directly addressed your concern that our original 8B experiment only tested prior transfer by running a new in-regime TrajVal experiment on Qwen3-8B (estimating the prior directly on the 8B model):

Qwen3-8B (Math)Best Acc ↑\uparrowAUC ↑\uparrowS2B ↓\downarrow
Random (BASE-GRPO)0.53420.4999–
+ TrajVal (Transferred 4B prior)0.54840.510987.5%
+ TrajVal (In-regime 8B prior)0.55010.513276.8%

Estimating the prior in-regime on Qwen3-8B improves Best Acc by +1.59% over Random and further outperforms the transferred 4B prior across all metrics, cutting S2B from 87.5% down to 76.8%. This confirms that TrajVal scales natively and effectively to larger models. We will integrate all new experiments and wall-clock analyses into the revision.

Dear Reviewer t5g8,

Thank you again for your time and constructive review. As the discussion period is ending, here is a brief summary of our responses:

  • End-to-end cost (W1 & Q1): We now report preparation-inclusive wall-clock time against pre-only. When the prior is reused across scales, TrajVal saves 15.1%–19.1% in total time with higher accuracy.
  • LIMR (W2 & Q2): Under the same RL setup, TrajVal matches or exceeds LIMR's selection rule at about 25% of its preparation cost.
  • Larger models (Q3): A new in-regime experiment on Qwen3-8B improves Best Acc by 1.59 points over Random.

The cost analysis has been added to Appendix B.8 (Table 13, pp. 24–25) of the revised PDF.

We would be grateful to know whether these address your concerns, and we are happy to clarify anything further.

AuthorsOpenReview ↗

Eager to discuss

Dear Reviewer t5g8,

With less than 7 hours left in the discussion period, we are eager to hear from you. We have posted a detailed rebuttal and a revised PDF but have not yet received a response, so even a brief reply on whether our answers resolve your concerns would be greatly appreciated.

For your convenience, all three points you raised have already been addressed in our responses above: (1) End-to-end cost (W1 & Q1); (2) Comparison with LIMR (W2 & Q2); (3) Larger models (Q3).

If these address your concerns, we would be grateful if you could consider updating your assessment accordingly. Thank you again for your time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Updates and Reviewer Discussion

Dear Senior Area Chairs, Area Chairs, and Reviewers,

Thank you for the time and care you devoted to reviewing Submission 20011. As the discussion period closes, we would like to summarize the main updates and the status of the discussion. Changes incorporated into the revised PDF are highlighted.

Main rebuttal updates

We are grateful for the questions raised during review, which helped us strengthen the paper. Below we summarize the experiments and analyses we added during the rebuttal.

  • Our end-to-end timing (t5g8 W1/Q1, vWBY Q2) includes preparation costs and uses the same hardware. An isolated small-model run takes longer once preparation is included. Reusing the prior across scales saves total time without re-estimation. The prior also shortens BOTS and GRESO main training while improving peak accuracy.
  • In our comparisons with existing methods (t5g8 W2/Q2, kV7r W2/Q2), TrajVal slightly outperformed LIMR under matched hard selection at lower preparation cost. Full-pool soft weighting widened the accuracy gain. We added GPS and DPS comparisons, found that TrajVal improved DPS peak accuracy and convergence, and discussed all seven suggested papers.
  • In the score ablations (vWBY W2/Q1/Q3, kV7r Q3), the product score reached the Math baseline peak sooner than positive-only, gain-only, and additive variants. Paired tests in the reported Math setting showed significant gains for standalone TrajVal and for TrajVal added to BOTS or GRESO.
  • On the core learnability claim (vWBY W1, kV7r follow-up), gains for tasks outside the probe set reflect updates induced by other tasks. The signal was reproducible across independent probes, and sub-pools with higher raw gains performed better in every comparison matched for initial pass rate.
  • In the Qwen3-8B experiment (t5g8 Q3), estimating the prior directly improved peak accuracy over uniform sampling and outperformed the transferred 4B prior on all reported metrics.

Review discussion status

We appreciate the reviewers’ positive feedback on the evidence for learnability beyond initial solvability (t5g8 S1, vWBY S1), its reproducibility (kV7r S1), and the evaluation across domains, scales, and schedulers (S2 in all three reviews). We also summarize below how the exchanges with each reviewer developed during the rebuttal.

  • t5g8 recognized the consistency of the main results. We responded to their questions on cost, LIMR, and scaling and have not yet received a follow-up.
  • vWBY acknowledged that the additional analyses addressed some concerns. Their concern about methodological novelty remained, and they kept their rating unchanged.
  • kV7r said their concerns were partially addressed and asked for a more principled derivation. We clarified the evidence behind the learnability claim and confirmed the revised PDF upload. We have not received a further reply.

We hope this summary accurately reflects the current state of the submission and how the review discussion developed. Thank you for considering the reviews, the revised PDF, and the supplementary results together in your final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.