- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that task learnability, the expected positive response to continued RL, is distinct from current solvability. The authors propose TRAJVAL, which estimates learnability from a short probe run and two endpoint evaluations, then uses the score as a static sampling prior or as a multiplicative prior for online schedulers. On math and logic with Qwen3-1.7B/4B, TRAJVAL improves over uniform sampling and gives additional gains when combined with BOTS and GRESO.
Critical strengths
-
Distinguishing learnability from snapshot solvability is an interesting and sensible framing for RL data allocation, and the paper shows that this signal is reproducible across independently sampled training contexts.
-
TRAJVAL is lightweight and easy to plug in, and evaluating it both standalone and as a multiplicative prior for online schedulers (BOTS, GRESO) across math and logic benchmarks is convincing.
Critical weaknesses
-
In Table 1, the performance improvements seem somewhat marginal. It is less convincing to conclude that the proposed method brings substantial gains, which may be because the score formulation is heuristic rather than a principled derivation.
-
The novelty seems somewhat limited, and the advantage over existing methods remains unclear. Adaptive sampling and rollout allocation methods have been widely explored, but this paper does not empirically compare with many of them, and neglects several closely related works, not limited to [1-7].
[1] Small generalizable prompt predictive models can steer efficient RL post-training of large reasoning models
[2] Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models
[3] TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning
[4] Improving data efficiency for LLM reinforcement fine-tuning through difficulty-targeted online data selection and rollout replay
[5] Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse
[6] Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration
[7] Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocation
Questions for authors
-
In Table 1, several gains over BOTS/GRESO are below 1 point. Can the authors clarify why these improvements are sufficient to support learnability as a distinct axis, especially after accounting for the extra probe overhead?
-
Can the authors discuss the conceptual advantages over [1-7], and empirically compare with at least some of them?
-
The score is a heuristic product. Why is this form preferred over using alone, or the slope/ statistics already analyzed in Section 3?
5 comments
2. W2 & Q2: Conceptual Taxonomy of [1–7] & New Experiments on GPS [1] and DPS [2] — TrajVal Boosts DPS Accuracy by +1.94% and Cuts Training Steps by 35.3%
We thank you for pointing out [1–7], which we have studied carefully and will integrate into Related Work. While all seven study non-uniform compute allocation in RLVR, they intervene at distinct decision levels:
- Intra-Step Rollout Budget Allocation & Reuse (TRACE [3], AR3PO [5], DARS [6], Knapsack RL [7]): DARS [6], Knapsack RL [7], and AR3PO [5] decide how many rollouts to generate or reuse after a prompt is sampled, while TRACE [3] allocates budgets across root rollouts and prefix branches. Because TrajVal operates strictly at the inter-step task-sampling level (prioritizing which prompts enter the batch), it is orthogonal and can directly supply the root task-sampling distribution for these rollout-allocation frameworks.
- Online Solvability & Transition-Predictive Task Scheduling (GPS [1], DPS [2], DOTS [4]): GPS [1] (small predictive models), DPS [2] (dynamics-predictive state transitions), and DOTS [4] (difficulty-targeted selection with replay) adjust task-sampling probabilities online using historical rollout feedback. Conceptually, they belong to Dynamic Task-Value Scheduling (§2), targeting instantaneous or next-step difficulty/solvability. TrajVal instead provides an offline multi-epoch learnability prior that guides allocation from step 0—resolving the cold-start noise before online predictors accumulate sufficient rollout history.
New Empirical Comparisons with GPS [1] and DPS [2]: To verify this complementarity beyond BOTS and GRESO, we evaluated GPS [1] and DPS [2] both standalone and composed multiplicatively with TrajVal on Qwen3-1.7B Math:
| Method | Best Acc (5-step mean) | AUC | S2B |
|---|---|---|---|
| Random (BASE-GRPO) | 0.3967 | 0.3620 | – |
| Standalone TrajVal (Ours) | 0.4105 | 0.3730 | 65.66% |
| GPS [1] (History-based difficulty predictor) | 0.4080 | 0.3617 | – |
| GPS [1] + TrajVal (Ours) | 0.4082 | 0.3709 (+0.92%) | 90.12% |
| DPS [2] (Next-step state-transition predictor) | 0.3953 | 0.3571 | – |
| DPS [2] + TrajVal (Ours) | 0.4147 (+1.94%) | 0.3764 (+1.93%) | 64.71% |
Standalone TrajVal outperforms both standalone GPS (0.4105 vs. 0.4080) and DPS (0.3953). More importantly, composing TrajVal as a prior consistently accelerates and improves both online methods: it boosts GPS's AUC from (90.12% S2B), and substantially elevates DPS by +1.94% Best Acc () and +1.93% AUC while reaching DPS's peak in 64.71% of the steps. This confirms that offline learnability provides vital complementary signal even alongside explicit online transition models.
3. Q3: Why the Multiplicative Score Beats -Only, Additive, and Dense Slope/ — A Logical-AND Gate That Avoids Checkpoint Evaluation Overhead
We agree is an intuitive proxy rather than a theoretical optimum, but its multiplicative endpoint form is specifically preferred over -only, additive, and dense slope/ alternatives for clear empirical and computational reasons:
| Domain | Scoring Variant | Formula | S2B | Best Acc | AUC |
|---|---|---|---|---|---|
| Math | Additive Variant | 90.91% | 0.4001 | 0.3592 | |
| Math | -only (Table 3) | 85.86% | 0.4086 | 0.3660 | |
| Math | TrajVal (Product) | 65.66% | 0.4105 | 0.3730 | |
| Logic | -only (Table 3) | 78.26% | 0.1864 | 0.1641 | |
| Logic | TrajVal (Product) | 73.91% | 0.1904 | 0.1663 |
- Why Product beats -only and Additive: -only over-prioritizes tasks with high probe gains that are already nearly mastered (e.g., ), which quickly saturate during main RL. Conversely, an Additive rule allows high headroom () to compensate for zero learning response (). The Multiplicative form acts as a conjunctive (logical-AND) gate requiring both positive responsiveness and remaining headroom, slashing Math S2B by 20.2 points over -only () and 25.3 points over Additive.
- Why Endpoint is preferred over Section 3's Slope and : Endpoint captures the direction and magnitude of reward change using only two full-pool evaluations. Fitting trajectory-level slope and requires evaluating intermediate checkpoints: on our 17k-task pool, 20 passes would cost approximately 43 h at the measured 2.15 h per pass, compared with 4.3 h for two endpoint passes. Figures 4 and 8 show that endpoint-derived profiles largely preserve the overall structure of full-trajectory behavior. Together with the downstream gains over -only scoring, this supports using the inexpensive endpoint signal, combined with headroom, for task allocation.
We sincerely thank Reviewer kV7r for the encouraging assessment and insightful literature pointers. Below, we clarify the statistical significance and overhead of our gains, provide a taxonomy and new empirical comparisons with [1–7] (specifically GPS [1] and DPS [2]), and justify our multiplicative score formulation.
1. W1 & Q1: Validating Learnability as a Distinct Axis — Statistically Significant Gains (p<0.05), Up to +28%–44% Controlled Within-Bin Improvements, and Zero Extra Overhead When Plugged into Online Schedulers