- Rating
- 3(+2)
- Confidence
- 2(−1)
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BanditSched, an adaptive data scheduling framework for RLVR based on non-stationary contextual bandits and Thompson Sampling. The method predicts prompt utility using competence statistics, difficulty information, and question embeddings, while employing frontier probing to update estimates for challenging prompts. Experiments are conducted on mathematical reasoning, coding, and logical reasoning tasks.
Critical strengths
- The paper addresses an important practical problem in RLVR: dynamically selecting informative prompts to reduce unnecessary rollouts.
- The method is evaluated across multiple model sizes and reasoning tasks.
Critical weaknesses
- The motivation and justification for the proposed design are insufficient. The method assumes that pass rates, advantage magnitudes, and question embeddings can effectively predict prompt utility, but their relationship with actual learning gains is not adequately established. In particular, larger advantages do not necessarily translate into greater improvements in model performance.
- Several important design choices and implementation details remain unclear. For example, the choice of the historical window in Eq. (4) and the computation of this feature for prompts sampled at different frequencies are not sufficiently explained. Moreover, Appendix D.1 introduces an embedding projection trained on 2K labeled examples, but its training objective and procedure are inadequately specified, limiting reproducibility.
- BanditSched combines contextual Thompson Sampling, a composite reward, sliding windows, embedding projection, and frontier probing, resulting in a rather complex framework. Although ablations are provided, the additional benefits of these components over simpler data selection strategies are not sufficiently demonstrated.
- Although Appendix E.2 reports approximately 1.8× wall-clock speedup, the primary efficiency metric remains rollout count rather than actual time to reach a target accuracy. End-to-end timing comparisons with simple filtering baselines are missing, and preprocessing costs and generated token counts are not adequately analyzed.
- Figure 2(a) shows a sharp performance drop for all methods near step 400, whereas Appendix E.9 reports a decline of only approximately 0.7 percentage points and attributes it to output truncation. This apparent inconsistency requires clarification, including the cause of the drop and which checkpoints were used for the final reported results.
- The discussion of related work is incomplete, particularly regarding MoPPS [1] and CurES [2], which already explore Bayesian estimation, online prompt difficulty prediction, bandit-based selection, and efficient curriculum learning. The paper does not sufficiently distinguish its contributions from these studies or provide direct experimental comparisons.
- The mathematical reasoning evaluation could be more comprehensive and up to date. The experiments primarily rely on MATH-500, AIME 2024, and Minerva Math, without evaluating on more recent benchmarks such as AIME 2025, AIME 2026 and MathArena/hmmt_feb_2026. Given the claimed improvements on challenging and out-of-distribution problems, evaluations on newer and more challenging benchmarks would provide stronger evidence of generalization and robustness.
[1] Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?
[2] CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs.
Questions for authors
See Weaknesses.
10 comments
Response to Reviewer YGGs (Part 2)
(Following above Part 1)
W2: Several important design choices and implementation details remain unclear. For example, ...
A2: Both gaps are real and both are now closed.
The lag in . We use steps, now stated in §3.2, in the implementation details, and in the shared-hyperparameter list. Appendix E.1 gives the bookkeeping: each prompt keeps a short append-only list of
pairs written whenever the EMA updates that prompt, with entries older than evicted except the most recent one; is read as the EMA recorded at the newest step not later than , or the initial value 0.5 if the prompt has never been observed. Expected list length is , so storage is .
Your question about prompts sampled at different frequencies has a clean answer that the submission should have stated: because changes only when a prompt is observed, is exactly zero for any prompt not observed during the last steps. The feature is therefore never undefined, never extrapolated, and never scale-distorted for infrequently selected prompts; a prompt observed several times inside the window accumulates the full change over those observations. Table 7 adds a sweep and two variants:
| Variant | MATH-500 | AIME | Eff. |
|---|---|---|---|
| 64.3 ± 0.6 | 27.5 ± 1.2 | 0.38 | |
| 64.6 ± 0.5 | 28.0 ± 1.1 | 0.36 | |
| (default) | 64.8 ± 0.5 | 28.3 ± 1.1 | 0.35 |
| 64.7 ± 0.5 | 28.1 ± 1.1 | 0.36 | |
| 64.4 ± 0.6 | 27.7 ± 1.2 | 0.37 | |
| removed from the context | 64.5 ± 0.6 | 27.8 ± 1.2 | 0.37 |
| Explicit staleness coordinate added () | 64.8 ± 0.5 | 28.3 ± 1.1 | 0.35 |
Accuracy varies by less than 0.5 points over . We also tested an explicit staleness coordinate , which changes nothing measurable, consistent with the first point: a zero together with a low already identifies a stale prompt.
The embedding projection. New Appendix E.2 specifies it in full. Let be the mean-pooled final-layer hidden state over the problem statement ( = 2048 / 3584 / 2560 / 4096 for Qwen2.5-3B / 7B and Qwen3-4B / 8B). We learn one linear map by attaching two heads to , discarded afterwards: a topic head trained with softmax cross-entropy over annotated subject tags, and a difficulty head trained with ordinal (cumulative-link) cross-entropy over the ordered difficulty levels, which respects their ordering rather than treating them as unrelated classes. The objective is
with and .
Labels: 2,000 DAPO-Math-17K questions carrying both a difficulty annotation and a subject tag (7 subjects, 5 levels) for math; 2,000 TACO problems with the tag set collapsed to 8 groups and the dataset's 4 difficulty levels for code; 2,000 generated puzzles with the generator's template family as topic (6 classes) and its 3 levels for logic. All labels are dataset metadata; no reference solution, generated response, or verifier outcome is used, so no answer information enters the scheduler, and the 2,000 annotated questions remain in the training pool.
Optimization: hidden states extracted once and cached; AdamW, learning rate , weight decay , batch size 128, at most 50 epochs, 80/20 split of , early stopping on validation topic macro-F1 with patience 5. Under two minutes on a single GPU once the hidden states are cached; total preprocessing including the forward pass is 0.05 hours for the 17K math pool and generates no responses. Held-out metrics for math: macro-F1 0.71 (topic), 5-way accuracy 0.58 (difficulty). When no annotations are available, the unsupervised PCA projection is a drop-in substitute costing 0.7 points, and 1,000 labels already recover most of the benefit of the full budget.
Three further details that were underspecified and are now fixed: an epoch for the diversity centroid is a block of consecutive steps, with the centroid accumulated over questions selected within the block and reset at its start; when a dataset provides no difficulty annotation, is set to a constant, which leaves the coordinate uninformative while keeping fixed across domains; and the two filtering baselines in Appendix F.5 use the same EMA, the same , and the same neutral initialization as our method, with no refresh mechanism added, since adding one would import part of our method into them.
Response to Reviewer YGGs (Part 3)
(Following above Part 2)
W3: BanditSched combines contextual Thompson Sampling, a composite reward, sliding windows, embedding projection, and frontier probing, resulting in a rather complex framework. Although ablations are provided, the additional benefits of these components over simpler data selection strategies are not sufficiently demonstrated.
A3: Thank you for highlighting it. Our original ablations showed what is lost when a component is removed, but did not sufficiently establish the relative gain over a simpler method that never had it. To address this gap, new Appendix F.6 reframes the evidence incrementally, from Full-GRPO upward (Table 24):
| # | Configuration | MATH-500 | Eff. | Additional machinery |
|---|---|---|---|---|
| 1 | Full-GRPO + DAPO | 63.8 ± 0.6 | 1.00 | none |
| 2 | + EMA pass-rate band (Band-Filter) | 63.5 ± 0.6 | 0.44 | one scalar per question |
| 3 | + greedy value ranking in the band | 63.9 ± 0.6 | 0.41 | closed-form score |
| 4 | Contextual TS on competence and difficulty | 62.9 ± 0.7 | 0.49 | 5x5 ridge posterior |
| 5 | + question embedding () | 63.4 ± 0.6 | 0.43 | frozen encoder, 64-d head |
| 6 | + diversity bonus | 63.6 ± 0.6 | 0.39 | running centroid |
| 7 | + frontier probing (full) | 64.8 ± 0.5 | 0.35 | periodic probe batches |
Row 4 is a regression in both accuracy and efficiency: a linear model over four scalars is a worse description of utility than a direct band rule, and the contextual formulation only pays off once the embedding gives it something to generalize over. Reporting this says precisely which part of the complexity is load-bearing. Rows 4 and 5 also explain an apparent inconsistency between Tables 3 and 23 that the submission did not flag: removing the embedding removes three things at once (the context block, the diversity term, and the frontier rule, which degenerates to uniform probing), which is why its effect exceeds the sum of the individual ablations. This explanation now lives in Appendix F.6 rather than in the main text.
Table 25 adds four simpler schedulers a reader might reasonably propose instead:
| Scheduler | MATH-500 | AIME | Eff. | Offline cost |
|---|---|---|---|---|
| Static offline subset ranked by (fixed 20%) | 63.2 ± 0.7 | 26.1 ± 1.4 | 0.52 | 0.17 |
| Hand-designed difficulty curriculum | 63.0 ± 0.8 | 25.9 ± 1.5 | 0.61 | 0.00 |
| Non-contextual per-prompt UCB | 62.7 ± 0.7 | 26.0 ± 1.4 | 0.57 | 0.00 |
| EMA band filter with difficulty annealing | 63.8 ± 0.6 | 27.0 ± 1.3 | 0.42 | 0.00 |
| BanditSched | 64.8 ± 0.5 | 28.3 ± 1.1 | 0.35 | 0.00 |
None reaches a rollout ratio below 0.41. The static offline subset is the closest simple analogue of what we do online and the clearest illustration of why adaptivity matters: it pays a full offline sweep and the subset it fixes at the start is stale by mid-training.
On complexity as a runtime concern, we measured it. Per training step on the 17K pool with : posterior update 2.1 ms, scoring all questions 0.6 ms, context construction 1.4 ms, frontier similarity search 38 ms every 10 steps. Against a mean step time of 73.8 s, the scheduler accounts for 0.006% of step time, or 0.011% including the amortized search. Persistent state is one matrix, one 69-vector, and four scalars plus a 64-d embedding per question: 4.7 MB in FP32. Finally, for a reader who wants fewer moving parts, the "lite" configuration (no diversity term, no difficulty feature, and at defaults) reaches 63.9 with a rollout ratio of 0.40, so most of the efficiency benefit survives a two-hyperparameter version.
W4: Although Appendix E.2 reports approximately 1.8× wall-clock speedup, the primary efficiency metric remains rollout count rather than actual time to reach a target accuracy. End-to-end timing comparisons with simple filtering baselines are missing, and preprocessing costs and generated token counts are not adequately analyzed.
A4: Agreed on all three counts. New Appendix F.4 reports hours to the same accuracy target for every method, broken into generation (including probes), policy updates, preprocessing, and the scheduler's own computation, on Qwen2.5-3B with MATH (Table 21):
| Method | Steps to | Gen. | Update | Preproc. | Sched. | Total (speedup) |
|---|---|---|---|---|---|---|
| Full-GRPO + DAPO | 272 | 8.60 | 1.43 | 0.00 | 0.00 | 10.03 (1.00x) |
| Random-20% | 358 | 7.00 | 1.88 | 0.00 | 0.00 | 8.88 (1.13x) |
| E2H | 301 | 5.00 | 1.58 | 0.00 | 0.00 | 6.58 (1.52x) |
| SEC | 289 | 4.90 | 1.52 | 0.00 | 0.01 | 6.43 (1.56x) |
| DEPO | 268 | 4.30 | 1.41 | 2.10 | 0.00 | 7.81 (1.28x) |
| MoPPS | 259 | 4.12 | 1.36 | 0.00 | 0.02 | 5.50 (1.82x) |
| CurES | 252 | 3.74 | 1.32 | 0.00 | 0.03 | 5.09 (1.97x) |
| CROPI | 251 | 3.80 | 1.32 | 1.40 | 0.02 | 6.54 (1.53x) |
| Band-Filter | 262 | 4.00 | 1.38 | 0.00 | 0.01 | 5.39 (1.86x) |
| Band-Filter + greedy value | 248 | 3.80 | 1.30 | 0.00 | 0.01 | 5.11 (1.96x) |
| BanditSched | 198 | 3.26 + 0.07 | 1.04 | 0.05 | 0.09 | 4.51 (2.22x) |
Response to Reviewer YGGs (Part 1)
W1: The motivation and justification for the proposed design are insufficient. The method assumes that pass rates, advantage magnitudes, and question embeddings can effectively predict prompt utility, but their relationship with actual learning gains is not adequately established. In particular, larger advantages do not necessarily translate into greater improvements in model performance.
A1: Thanks for your criticism: the submission asserted the link between the selection signal and learning value and verified it only indirectly through end-task accuracy. New Appendix B.3 tests it directly, and §3.1 now separates what is proved from what is supported. The four studies are summarized here; Reviewer mgj9's W3 is the same concern and our corresponding response above gives the same evidence in more detail.
Pass rates and advantage magnitudes. The relation between pass rate and advantage magnitude is an identity, not an assumption: with binary rewards Aˉi,t=2qi,t(1−qi,t), so a uniform group contributes exactly zero to the gradient. What needed establishing is the step from there to learning gain, and you are right that larger advantages do not necessarily mean larger improvements. The measurements:
Question embeddings. For the embeddings the relevant claim is weaker and the paper now states it as such: the embedding is not asserted to predict utility by itself, but to let observations transfer between related prompts. New Appendix F.2 measures this directly by comparing predicted and realized group success rates, stratified by how many times the prompt had been observed (Table 16):
All three methods agree once a prompt has been seen five or more times, the regime where a per-prompt posterior is well determined. They separate sharply in the regime that dominates the run: under a budget of about three visits per prompt, more than half of all selection decisions concern prompts observed at most once, where a per-prompt posterior can only return its prior. New Appendix F.12 adds the complementary result that the offline retrieval quality of the embedding predicts final accuracy across eleven representations with Spearman 0.91.
What the revision claims. The limitations section now states that roughly 40% of the rank variation in one-step gain is unexplained by the signal, that it says nothing about whether an update helps on the test distribution, and that our gradient-agreement check is a weak test rather than a guarantee.