acceptodds
Under review as a conference paper at ICLR 2027

BanditSched: Adaptive Data Scheduling with Contextual Bandits for Efficient Reasoning Reinforcement Learning

10 views

Abstract

Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for eliciting reasoning capabilities in large language models. As the model's competence evolves during training, so does the value of each training problem, and data selection must adapt to this non-stationary process. Existing methods rely on offline profiling, coarse-grained curricula, or filtering after rollouts have already been generated. We observe that the reward-based learning signal of a problem is governed by its current pass rate: groups with intermediate success rates yield nonzero mean absolute advantages, while groups with uniformly correct or incorrect responses yield zero advantages. Motivated by this observation, we formulate RLVR data selection as a *non-stationary contextual bandit* problem, where each training problem is an arm and its context encodes competence statistics, difficulty, and reasoning-pattern information. We propose BanditSched, a Thompson Sampling-based framework that (1) predicts prompt utility before allocating rollouts, (2) periodically probes previously attempted low-success problems to refresh estimates near the model's competence frontier without policy updates, and (3) combines advantage magnitude, competence-edge membership, and question-space diversity into a composite reward. A sliding-window reward model tracks the changing utilities, with its window length guided by an estimation–drift trade-off. Experiments on mathematical reasoning, code generation, and logical puzzles with Qwen2.5 and Qwen3 models show that BanditSched improves rollout efficiency while matching or exceeding the final accuracy of strong baselines, with the largest gains on hard and out-of-distribution tasks. Specifically, BanditSched reaches 95% of Full-GRPO's final accuracy using 30–35% of its online rollouts and completes full training runs with an approximately wall-clock speedup. Controlled comparisons further show that contextual scheduling improves efficiency beyond pass-rate band filtering alone.

Primary area:
reinforcement learning
Keywords:
Data Scheduling, Reinforcement Learning, Contextual Bandits

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YGGsOpenReview ↗
Rating
3(+2)
Confidence
2(−1)
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BanditSched, an adaptive data scheduling framework for RLVR based on non-stationary contextual bandits and Thompson Sampling. The method predicts prompt utility using competence statistics, difficulty information, and question embeddings, while employing frontier probing to update estimates for challenging prompts. Experiments are conducted on mathematical reasoning, coding, and logical reasoning tasks.

Critical strengths
  1. The paper addresses an important practical problem in RLVR: dynamically selecting informative prompts to reduce unnecessary rollouts.
  2. The method is evaluated across multiple model sizes and reasoning tasks.
Critical weaknesses
  1. The motivation and justification for the proposed design are insufficient. The method assumes that pass rates, advantage magnitudes, and question embeddings can effectively predict prompt utility, but their relationship with actual learning gains is not adequately established. In particular, larger advantages do not necessarily translate into greater improvements in model performance.
  2. Several important design choices and implementation details remain unclear. For example, the choice of the historical window τ\tau in Eq. (4) and the computation of this feature for prompts sampled at different frequencies are not sufficiently explained. Moreover, Appendix D.1 introduces an embedding projection trained on 2K labeled examples, but its training objective and procedure are inadequately specified, limiting reproducibility.
  3. BanditSched combines contextual Thompson Sampling, a composite reward, sliding windows, embedding projection, and frontier probing, resulting in a rather complex framework. Although ablations are provided, the additional benefits of these components over simpler data selection strategies are not sufficiently demonstrated.
  4. Although Appendix E.2 reports approximately 1.8× wall-clock speedup, the primary efficiency metric remains rollout count rather than actual time to reach a target accuracy. End-to-end timing comparisons with simple filtering baselines are missing, and preprocessing costs and generated token counts are not adequately analyzed.
  5. Figure 2(a) shows a sharp performance drop for all methods near step 400, whereas Appendix E.9 reports a decline of only approximately 0.7 percentage points and attributes it to output truncation. This apparent inconsistency requires clarification, including the cause of the drop and which checkpoints were used for the final reported results.
  6. The discussion of related work is incomplete, particularly regarding MoPPS [1] and CurES [2], which already explore Bayesian estimation, online prompt difficulty prediction, bandit-based selection, and efficient curriculum learning. The paper does not sufficiently distinguish its contributions from these studies or provide direct experimental comparisons.
  7. The mathematical reasoning evaluation could be more comprehensive and up to date. The experiments primarily rely on MATH-500, AIME 2024, and Minerva Math, without evaluating on more recent benchmarks such as AIME 2025, AIME 2026 and MathArena/hmmt_feb_2026. Given the claimed improvements on challenging and out-of-distribution problems, evaluations on newer and more challenging benchmarks would provide stronger evidence of generalization and robustness.

[1] Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?

[2] CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs.

Questions for authors

See Weaknesses.

10 comments
AuthorsOpenReview ↗

Response to Reviewer YGGs (Part 1)

W1: The motivation and justification for the proposed design are insufficient. The method assumes that pass rates, advantage magnitudes, and question embeddings can effectively predict prompt utility, but their relationship with actual learning gains is not adequately established. In particular, larger advantages do not necessarily translate into greater improvements in model performance.

A1: Thanks for your criticism: the submission asserted the link between the selection signal and learning value and verified it only indirectly through end-task accuracy. New Appendix B.3 tests it directly, and §3.1 now separates what is proved from what is supported. The four studies are summarized here; Reviewer mgj9's W3 is the same concern and our corresponding response above gives the same evidence in more detail.

Pass rates and advantage magnitudes. The relation between pass rate and advantage magnitude is an identity, not an assumption: with binary rewards Aˉi,t=2qi,t(1−qi,t)\bar{A}_{i,t} = 2\sqrt{q_{i,t}(1-q_{i,t})}, so a uniform group contributes exactly zero to the gradient. What needed establishing is the step from there to learning gain, and you are right that larger advantages do not necessarily mean larger improvements. The measurements:

  • Against per-prompt gradient norms: Pearson 0.73, Spearman 0.78, pooled over four checkpoints and 1,536 prompts. Strong but well below one, as expected given score-function, clipping, and token-aggregation effects.
  • Against realized one-step gains: at four frozen checkpoints we build 40 candidate batches spanning a range of batch-mean Aˉ\bar{A}, apply one GRPO update each, and measure the change on a held-in 512-question probe set. Spearman ρ=0.63\rho = 0.63 for batch-mean Aˉ\bar{A}, 0.58 for our predicted utility, and 0.29 for mean annotated difficulty, the quantity bucket-level curricula rely on. Top decile of batches: +0.42+0.42 points; bottom decile: −0.03-0.03.
  • Against an oracle, with a negative control: in a reduced setting (N=2,000N=2{,}000, T=100T=100) an oracle that observes the true Aˉ\bar{A} of every question at every step reaches 59.7, BanditSched 59.1, uniform 56.9, and anti-selection (choosing the lowest predicted utility) 53.4. Inverting the score costs 3.5 points, which establishes direction and not merely correlation, and BanditSched recovers 79% of the uniform-to-oracle gap at a seventh of the oracle's rollout cost.

Question embeddings. For the embeddings the relevant claim is weaker and the paper now states it as such: the embedding is not asserted to predict utility by itself, but to let observations transfer between related prompts. New Appendix F.2 measures this directly by comparing predicted and realized group success rates, stratified by how many times the prompt had been observed (Table 16):

Prior observations of the promptMoPPSCurESBanditSchedShare of decisions
00.310.300.1932%
10.240.230.1721%
2 to 40.170.170.1526%
5 or more0.130.130.1321%

All three methods agree once a prompt has been seen five or more times, the regime where a per-prompt posterior is well determined. They separate sharply in the regime that dominates the run: under a budget of about three visits per prompt, more than half of all selection decisions concern prompts observed at most once, where a per-prompt posterior can only return its prior. New Appendix F.12 adds the complementary result that the offline retrieval quality of the embedding predicts final accuracy across eleven representations with Spearman 0.91.

What the revision claims. The limitations section now states that roughly 40% of the rank variation in one-step gain is unexplained by the signal, that it says nothing about whether an update helps on the test distribution, and that our gradient-agreement check is a weak test rather than a guarantee.

AuthorsOpenReview ↗

Response to Reviewer YGGs (Part 2)

(Following above Part 1)

W2: Several important design choices and implementation details remain unclear. For example, ...

A2: Both gaps are real and both are now closed.

The lag τ\tau in Δp^i,t\Delta \hat{p}_{i,t}. We use τ=20\tau = 20 steps, now stated in §3.2, in the implementation details, and in the shared-hyperparameter list. Appendix E.1 gives the bookkeeping: each prompt keeps a short append-only list of (s,p^i,s)(s, \hat{p}_{i,s})

pairs written whenever the EMA updates that prompt, with entries older than τ\tau evicted except the most recent one; p^i,t−τ\hat{p}_{i,t-\tau} is read as the EMA recorded at the newest step not later than t−τt - \tau, or the initial value 0.5 if the prompt has never been observed. Expected list length is 1+τB/N≈1.151 + \tau B/N \approx 1.15, so storage is O(N)O(N).

Your question about prompts sampled at different frequencies has a clean answer that the submission should have stated: because p^i,t\hat{p}_{i,t} changes only when a prompt is observed, Δp^i,t\Delta\hat{p}_{i,t} is exactly zero for any prompt not observed during the last τ\tau steps. The feature is therefore never undefined, never extrapolated, and never scale-distorted for infrequently selected prompts; a prompt observed several times inside the window accumulates the full change over those observations. Table 7 adds a sweep and two variants:

VariantMATH-500AIMEEff.
τ=5\tau = 564.3 ± 0.627.5 ± 1.20.38
τ=10\tau = 1064.6 ± 0.528.0 ± 1.10.36
τ=20\tau = 20 (default)64.8 ± 0.528.3 ± 1.10.35
τ=50\tau = 5064.7 ± 0.528.1 ± 1.10.36
τ=100\tau = 10064.4 ± 0.627.7 ± 1.20.37
Δp^\Delta\hat{p} removed from the context64.5 ± 0.627.8 ± 1.20.37
Explicit staleness coordinate added (d=70d = 70)64.8 ± 0.528.3 ± 1.10.35

Accuracy varies by less than 0.5 points over τ∈[10,50]\tau \in [10, 50]. We also tested an explicit staleness coordinate ai,t=min⁡{1,(t−tilast)/100}a_{i,t} = \min\{1, (t - t_i^{\mathrm{last}})/100\}, which changes nothing measurable, consistent with the first point: a zero Δp^\Delta\hat{p} together with a low σ^2\hat{\sigma}^2 already identifies a stale prompt.

The embedding projection. New Appendix E.2 specifies it in full. Let hi\mathbf{h}_i be the mean-pooled final-layer hidden state over the problem statement (dhd_h = 2048 / 3584 / 2560 / 4096 for Qwen2.5-3B / 7B and Qwen3-4B / 8B). We learn one linear map P∈R64×dhP \in \mathbb{R}^{64 \times d_h} by attaching two heads to ei=Phi\mathbf{e}_i = P\mathbf{h}_i, discarded afterwards: a topic head trained with softmax cross-entropy over annotated subject tags, and a difficulty head trained with ordinal (cumulative-link) cross-entropy over the ordered difficulty levels, which respects their ordering rather than treating them as unrelated classes. The objective is

L=1∣M∣∑i∈M[CE(WκPhi,κi)+λd,OrdCE(wδ⊤Phi,δi)]+λr∣P∣F2,\mathcal{L} = \frac{1}{|\mathcal{M}|}\sum_{i \in \mathcal{M}}\Big[\mathrm{CE}(W_\kappa P \mathbf{h}_i, \kappa_i) + \lambda_d\\,\mathrm{OrdCE}(\mathbf{w}_\delta^\top P \mathbf{h}_i, \delta_i)\Big] + \lambda_r \\|P\\|_F^2, with λd=1\lambda_d = 1 and λr=10−4\lambda_r = 10^{-4}.

Labels: 2,000 DAPO-Math-17K questions carrying both a difficulty annotation and a subject tag (7 subjects, 5 levels) for math; 2,000 TACO problems with the tag set collapsed to 8 groups and the dataset's 4 difficulty levels for code; 2,000 generated puzzles with the generator's template family as topic (6 classes) and its 3 levels for logic. All labels are dataset metadata; no reference solution, generated response, or verifier outcome is used, so no answer information enters the scheduler, and the 2,000 annotated questions remain in the training pool.

Optimization: hidden states extracted once and cached; AdamW, learning rate 10−310^{-3}, weight decay 10−210^{-2}, batch size 128, at most 50 epochs, 80/20 split of M\mathcal{M}, early stopping on validation topic macro-F1 with patience 5. Under two minutes on a single GPU once the hidden states are cached; total preprocessing including the forward pass is 0.05 hours for the 17K math pool and generates no responses. Held-out metrics for math: macro-F1 0.71 (topic), 5-way accuracy 0.58 (difficulty). When no annotations are available, the unsupervised PCA projection is a drop-in substitute costing 0.7 points, and 1,000 labels already recover most of the benefit of the full budget.

Three further details that were underspecified and are now fixed: an epoch for the diversity centroid is a block of ⌈N/B⌉=133\lceil N/B \rceil = 133 consecutive steps, with the centroid accumulated over questions selected within the block and reset at its start; when a dataset provides no difficulty annotation, d~i\widetilde{d}_i is set to a constant, which leaves the coordinate uninformative while keeping dd fixed across domains; and the two filtering baselines in Appendix F.5 use the same EMA, the same ηp\eta_p, and the same neutral initialization as our method, with no refresh mechanism added, since adding one would import part of our method into them.

AuthorsOpenReview ↗

Response to Reviewer YGGs (Part 3)

(Following above Part 2)

W3: BanditSched combines contextual Thompson Sampling, a composite reward, sliding windows, embedding projection, and frontier probing, resulting in a rather complex framework. Although ablations are provided, the additional benefits of these components over simpler data selection strategies are not sufficiently demonstrated.

A3: Thank you for highlighting it. Our original ablations showed what is lost when a component is removed, but did not sufficiently establish the relative gain over a simpler method that never had it. To address this gap, new Appendix F.6 reframes the evidence incrementally, from Full-GRPO upward (Table 24):

#ConfigurationMATH-500Eff.Additional machinery
1Full-GRPO + DAPO63.8 ± 0.61.00none
2+ EMA pass-rate band (Band-Filter)63.5 ± 0.60.44one scalar per question
3+ greedy value ranking in the band63.9 ± 0.60.41closed-form score
4Contextual TS on competence and difficulty62.9 ± 0.70.495x5 ridge posterior
5+ question embedding (d=69d = 69)63.4 ± 0.60.43frozen encoder, 64-d head
6+ diversity bonus63.6 ± 0.60.39running centroid
7+ frontier probing (full)64.8 ± 0.50.35periodic probe batches

Row 4 is a regression in both accuracy and efficiency: a linear model over four scalars is a worse description of utility than a direct band rule, and the contextual formulation only pays off once the embedding gives it something to generalize over. Reporting this says precisely which part of the complexity is load-bearing. Rows 4 and 5 also explain an apparent inconsistency between Tables 3 and 23 that the submission did not flag: removing the embedding removes three things at once (the context block, the diversity term, and the frontier rule, which degenerates to uniform probing), which is why its effect exceeds the sum of the individual ablations. This explanation now lives in Appendix F.6 rather than in the main text.

Table 25 adds four simpler schedulers a reader might reasonably propose instead:

SchedulerMATH-500AIMEEff.Offline cost
Static offline subset ranked by Aˉ\bar{A} (fixed 20%)63.2 ± 0.726.1 ± 1.40.520.17
Hand-designed difficulty curriculum63.0 ± 0.825.9 ± 1.50.610.00
Non-contextual per-prompt UCB62.7 ± 0.726.0 ± 1.40.570.00
EMA band filter with difficulty annealing63.8 ± 0.627.0 ± 1.30.420.00
BanditSched64.8 ± 0.528.3 ± 1.10.350.00

None reaches a rollout ratio below 0.41. The static offline subset is the closest simple analogue of what we do online and the clearest illustration of why adaptivity matters: it pays a full offline sweep and the subset it fixes at the start is stale by mid-training.

On complexity as a runtime concern, we measured it. Per training step on the 17K pool with d=69d = 69: posterior update 2.1 ms, scoring all questions 0.6 ms, context construction 1.4 ms, frontier similarity search 38 ms every 10 steps. Against a mean step time of 73.8 s, the scheduler accounts for 0.006% of step time, or 0.011% including the amortized search. Persistent state is one 69×6969 \times 69 matrix, one 69-vector, and four scalars plus a 64-d embedding per question: 4.7 MB in FP32. Finally, for a reader who wants fewer moving parts, the "lite" configuration (no diversity term, no difficulty feature, WW and KK at defaults) reaches 63.9 with a rollout ratio of 0.40, so most of the efficiency benefit survives a two-hyperparameter version.

W4: Although Appendix E.2 reports approximately 1.8× wall-clock speedup, the primary efficiency metric remains rollout count rather than actual time to reach a target accuracy. End-to-end timing comparisons with simple filtering baselines are missing, and preprocessing costs and generated token counts are not adequately analyzed.

A4: Agreed on all three counts. New Appendix F.4 reports hours to the same accuracy target for every method, broken into generation (including probes), policy updates, preprocessing, and the scheduler's own computation, on Qwen2.5-3B with MATH (Table 21):

MethodSteps to a⋆a^\starGen.UpdatePreproc.Sched.Total (speedup)
Full-GRPO + DAPO2728.601.430.000.0010.03 (1.00x)
Random-20%3587.001.880.000.008.88 (1.13x)
E2H3015.001.580.000.006.58 (1.52x)
SEC2894.901.520.000.016.43 (1.56x)
DEPO2684.301.412.100.007.81 (1.28x)
MoPPS2594.121.360.000.025.50 (1.82x)
CurES2523.741.320.000.035.09 (1.97x)
CROPI2513.801.321.400.026.54 (1.53x)
Band-Filter2624.001.380.000.015.39 (1.86x)
Band-Filter + greedy value2483.801.300.000.015.11 (1.96x)
BanditSched1983.26 + 0.071.040.050.094.51 (2.22x)
Reviewer_QSBSOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces BANDITSCHED, which formulates RLVR data selection as a non-stationary contextual bandit problem to dynamically target prompts at the model's evolving competence edge. By leveraging Thompson Sampling for pre-rollout utility prediction and incorporating lightweight frontier probing to refresh stale estimates, the framework avoids wasting compute on uninformative examples. The main contributions lie in this proactive scheduling paradigm and its empirical validation across Qwen2.5 and Qwen3 models, demonstrating that it achieves comparable final accuracy using only 30–35% of the online rollouts compared to full-data GRPO baselines.

Critical strengths

1.The paper addresses a critical computational bottleneck in RLVR by shifting data selection from reactive, post-hoc filtering to proactive, pre-rollout utility prediction. This paradigm shift is highly practical, as it prevents the expenditure of expensive LLM generation compute on uninformative prompts before rollouts are allocated.

2.By periodically re-evaluating historically failed prompts that exhibit embedding similarity to currently solvable problems—without executing policy-gradient updates—the framework ensures that promising instances are not permanently neglected as the model's competence evolves.

3.The empirical evaluation is comprehensive and rigorous, demonstrating consistent rollout reductions (30–35%) across multiple model scales (3B to 8B) and diverse reasoning domains (mathematics, code, and logic).

Critical weaknesses

1.Training dynamics of advantages: the paper lacks evaluation of how batch-level advantage evolves during training compared to Full GRPO baselines. Tracking advantage over time would provide direct mechanistic evidence that the scheduler successfully prevents the signal collapse typical in standard RLVR.

2.Lack of per-problem selection frequency statistics. The paper does not provide a fine-grained distribution of how often individual problems are selected during training (e.g., the number of problems sampled exactly once, twice, three times, etc.). Given that the core contribution is an adaptive data scheduler, such a breakdown would offer crucial insight into whether the bandit concentrates rollouts on a small subset of high-utility problems or maintains broader exploration across the dataset.

Questions for authors

see weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer QSBS (Part 1)

W1: Training dynamics of advantages: the paper lacks evaluation of how batch-level advantage evolves during training compared to Full GRPO baselines. Tracking advantage over time would provide direct mechanistic evidence that the scheduler successfully prevents the signal collapse typical in standard RLVR.

A1: Thanks for the insightful suggestion, and the result became one of the most useful additions. New Appendix F.15 tracks the batch-level mean absolute advantage ∣Aˉ∣t=B−1∑xi∈BtAˉi,t|\bar{A}|_t = B^{-1}\sum_{x_i \in \mathcal{B}_t} \bar{A}_{i,t} and the fraction of generated groups that are degenerate (pass rate 0 or 1, hence exactly zero gradient), over the Qwen2.5-3B math run. For Full-GRPO and Random-20% both quantities are computed over all groups generated at the step, before DAPO-style dynamic sampling discards the degenerate ones, since those groups have already been paid for; BanditSched does not resample, so for it the two populations coincide. Table 35:

MethodQuantityStep 2550100200300400
Full-GRPO + DAPO∥Aˉ∥t\|\bar{A}\|_t0.320.300.280.240.210.19
degenerate frac.0.600.620.640.680.710.74
Random-20%∥Aˉ∥t\|\bar{A}\|_t0.310.290.270.230.200.18
degenerate frac.0.610.630.650.690.720.75
Band-Filter∥Aˉ∥t\|\bar{A}\|_t0.560.590.580.550.520.50
degenerate frac.0.330.300.310.340.370.39
BanditSched∥Aˉ∥t\|\bar{A}\|_t0.580.620.650.640.620.61
degenerate frac.0.310.270.240.250.270.28

This is signal collapse in the precise sense the preliminaries make available: under uniform selection the learning signal falls by about 40% over the run and nearly three groups in four are degenerate by the end, because as the policy improves more questions leave the intermediate pass-rate region in both directions and nothing brings them back. Under BanditSched the signal rises over the first hundred steps, as the scheduler learns which questions sit on the frontier, and then stays nearly flat, because the frontier itself keeps moving and the scheduler follows it. The comparison with Band-Filter is also informative: band filtering recovers much of the effect but drifts worse over the second half of training (0.59 down to 0.50, degenerate fraction 0.30 up to 0.39), which is the staleness of unrefreshed EMAs appearing as a measurable quantity rather than as an argument.

The table also accounts quantitatively for the headline number, which the submission only asserted. Averaged over the steps preceding the common target, Full-GRPO must generate 2.84 groups for every gradient-bearing group, against 1.36 for BanditSched, a per-step generation ratio of 1.36/2.84=0.481.36/2.84 = 0.48. BanditSched reaches the target at step 198 against 272 for Full-GRPO, a ratio of 0.73. The product is 0.48×0.73=0.350.48 \times 0.73 = 0.35, the rollout ratio in Table 1. The efficiency gain therefore decomposes into roughly two thirds from generating fewer wasted groups and one third from needing fewer gradient steps. Section 4.4 carries a short "Advantage dynamics" paragraph with the headline numbers; the full table and the decomposition are in Appendix F.15.

AuthorsOpenReview ↗

Response to Reviewer QSBS (Part 2)

(Following above Part 1)

W2: Lack of per-problem selection frequency statistics. The paper does not provide a fine-grained distribution of how often individual problems are selected during training (e.g., the number of problems sampled exactly once, twice, three times, etc.). Given that the core contribution is an adaptive data scheduler, such a breakdown would offer crucial insight into whether the bandit concentrates rollouts on a small subset of high-utility problems or maintains broader exploration across the dataset.

A2: Agreed, and this also help us correct an accounting ambiguity in the submission. New Appendix F.9 gives the full distribution over the 400-step Qwen2.5-3B math run, in which T⋅B=51,200T \cdot B = 51{,}200 selection slots are allocated (Table 28):

Times selectedBanditSched: questions% of selected% of rolloutsFull-GRPO: questions% of rollouts
012,180n/a0.02,8000.0
182017.01.64,1208.0
2 to 31,01021.04.75,11024.0
4 to 71,09022.611.33,55037.1
8 to 151,01521.122.41,28025.8
16 to 3163013.127.81405.1
32 or more2555.332.200.0

BanditSched: mean 10.6 and median 6 over selected questions, maximum 143, Gini 0.52 over the selected set and 0.86 over the pool. Full-GRPO: mean 3.6 over contributing questions, maximum 14, Gini 0.42 over the pool.

To answer the question directly: both, in a specific proportion. A third of the rollouts go to the 5.3% of selected questions used at least 32 times, which is what one would want from a scheduler that has found a set of questions on the competence frontier and keeps training on them while they remain there. At the same time 38% of the selected questions are used three times or fewer, the exploratory tail produced by posterior sampling and the diversity term.

Two further statistics show that the concentration tracks the intended quantity. Questions selected more than 16 times spend on average 61% of their observed steps with an EMA inside the competence band, against 24% for questions selected exactly once. And the heavy bucket is not fixed early: of the 255 questions selected 32 or more times, 68% receive their first selection after step 100.

An accounting clarification. The submission compared "4,820 for BanditSched" with "14,200 (83.5%) for Full-GRPO" without noting that the two are not measured the same way. For BanditSched, which generates one group per selected question and keeps it, 4,820 is both the number of questions touched and the number that contributed a gradient. For Full-GRPO, dynamic sampling generates groups for many more questions than it keeps, so essentially the whole pool is touched; 14,200 is the number of questions that ever contributed a gradient-bearing group, and the remaining 2,800 produced only uniform groups whenever sampled, paying generation cost without contributing to any update. Appendix F.8 now states this, and the earlier average-visit figures, which did not reconcile with 400×128400 \times 128 selections, have been removed.

Dear Reviewer QSBS,

Thank you for suggesting these two analyses. With one day left in the discussion, we wanted to check whether the additions answer your questions. Appendix F.15 tracks the batch-level mean absolute advantage throughout training for both BanditSched and Full-GRPO. Appendix F.9 reports how often individual questions were selected and shows that, even among the 255 most frequently selected questions, 68% were first selected after step 100. If these results address your concerns, would you consider raising your rating? We would welcome any further questions.

Best regards,

The Authors

Reviewer_mgj9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to solve the zero advantage problem in RLVR via a novel data selection framework. Unlike previous methods that adopt a generate-then-select paradigm, the proposed framework formulates data selection as a non‑stationary contextual bandit problem. Specifically, it adopts Thompson Sampling for prompt selection, which is conditioned on a composite reward of advantage, pass rate, and diversity. It also proposes a frontier probing method to identify samples near the reasoning boundary during training, to compensate for Thompson Sampling. Lastly, it also employs an estimation-drift proxy to determine the sampling window. The proposed method is evaluated with four base models (Qwen series) across math, coding, and logical reasoning tasks, outperforming several baselines.

Critical strengths
  1. This paper offers an efficient pre-rollout data selection solution for RLVR.

  2. Modeling sample selection as a non‑stationary contextual bandit is interesting and does alleviate the computational burden.

  3. The proposed method is evaluated via extensive ablations and analyses.

Critical weaknesses
  1. In the proposed framework, approximately 4,820 of 17K questions are selected, which might neglect informative candidates.

  2. While ablations verify that the composite reward function brings performance gains, it is hand‑crafted and empirically designed, and the current weighted combination might be suboptimal.

  3. The mean absolute advantage is a coarse proxy for learning value. Rollouts within the same group may induce model updates in different directions and do not guarantee better performance.

Questions for authors
  1. The proposed frontier probing strategy relies on question embedding. How does the embedding quality affect the framework's performance?

  2. Are all hyperparameters determined on Qwen2.5‑3B math and reused for other models and tasks?

7 comments
AuthorsOpenReview ↗

Response to Reviewer mgj9 (Part 1)

W1: In the proposed framework, approximately 4,820 of 17K questions are selected, which might neglect informative candidates.

A1: Thank you, this is the right question to ask of any method that selects a subset. New Appendix F.10 examines it from three directions. We should say at the outset that some informative candidates are indeed left behind; what the evidence shows is that recovering them does not help within a fixed budget.

(1) The rollout budget binds long before coverage does. At each step the scheduler can serve B=128B=128 questions. Counting only questions observed at least once, so that their EMA reflects evidence rather than the neutral prior, the number whose EMA lies inside the competence band (0.2,0.8)(0.2, 0.8) is:

Step50100200300400
Observed questions with EMA in band7121,1801,6202,0602,290
Multiple of the batch size B=128B=1285.6x9.2x12.7x16.1x17.9x

Even at its smallest, the supply of questions known to be informative exceeds what a step can consume by a factor of 5.6, so the scheduler never has to reach for an unexamined question because it has run out of examined ones.

(2) The unselected pool is enriched in uninformative questions, though not purely so. We sampled 1,000 questions uniformly from the 12,180 never selected and generated a group of G=16G=16 responses for each with the final Qwen2.5-3B policy, as a diagnostic outside the training run (these groups update nothing). The same diagnostic was run on 1,000 selected questions:

Outcome with the final policyNever-selected sampleSelected sample
q=0q=0 (all incorrect, zero gradient)62.8%31.4%
q=1q=1 (all correct, zero gradient)22.6%27.2%
inside (0.2,0.8)(0.2, 0.8)14.6%41.4%

The selected pool is about 2.8x richer in band questions (41.4% against 14.6%), and the never-selected pool correspondingly about 1.5x enriched in questions that would have produced no gradient (85.4% against 58.6%). Extrapolating, roughly 1,780 of the never-selected questions would nonetheless have been informative, which Appendix F.10 states and the limitations section repeats.

(3) Forcing coverage raises coverage but not accuracy. Four variants were added to Table 27 that reserve batch slots for uniformly sampled questions or add a bonus for never-selected ones:

VariantMATH-500AIMEEff.Coverage
Default64.8 ± 0.528.3 ± 1.10.3528.4%
One uniform question per batch64.5 ± 0.527.9 ± 1.10.3632.7%
Eight uniform questions per batch64.6 ± 0.528.0 ± 1.10.3841.2%
Thirty-two uniform questions per batch64.2 ± 0.627.4 ± 1.20.4563.5%
Coverage bonus for never-selected questions64.7 ± 0.528.1 ± 1.10.3736.8%

Coverage more than doubles in the most aggressive variant, accuracy does not improve, and the rollout ratio degrades monotonically from 0.35 to 0.45. This follows from point (1): with far more band questions available than slots, swapping a high-utility question for a random one trades a likely-informative group for a likely-degenerate one.

(4) Coverage is a function of the budget, not a fixed property of the method. There is no hard exclusion, and coverage grows monotonically (8.9% at step 50, 14.7% at step 100, 28.4% at step 400). Extending training to T=800T=800 steps raises coverage to 44.6% and MATH-500 accuracy to 65.6 ± 0.5, against 64.3 ± 0.6 for Full-GRPO at the same 800 steps. Appendix E.3 now also explains why coverage keeps growing: an unobserved question keeps the neutral EMA of 0.5, which sits inside the band, so unobserved questions become relatively more attractive as the observed ones are mastered.

Section 4.4 carries a two-sentence summary and points to the appendix. The limitations section notes that a setting in which coverage binds before the budget does, for instance a much smaller training pool, is different from the one we study.

W2: While ablations verify that the composite reward function brings performance gains, it is hand‑crafted and empirically designed, and the current weighted combination might be suboptimal.

A2: Agreed. New Appendix F.14 replaces the implicit assumption that the weights are well chosen with two pieces of evidence.

(1) The weight surface is flat near the default. Table 14 only examined the three corners of the simplex; Table 34 now fills in the interior (Qwen2.5-3B, three seeds):

(α,β,γ)(\alpha, \beta, \gamma)MATH-500AIMEEff.
(1.0, 0.0, 0.0)63.4 ± 0.626.5 ± 1.30.42
(0.0, 1.0, 0.0)63.8 ± 0.527.0 ± 1.20.40
(0.0, 0.0, 1.0)60.6 ± 0.923.8 ± 1.60.71
(0.5, 0.5, 0.0)64.0 ± 0.527.1 ± 1.20.38
(0.6, 0.3, 0.1)64.6 ± 0.528.0 ± 1.10.36
(0.5, 0.3, 0.2) default64.8 ± 0.528.3 ± 1.10.35
(0.4, 0.4, 0.2)64.7 ± 0.528.1 ± 1.10.35
(0.4, 0.3, 0.3)64.5 ± 0.627.8 ± 1.20.36
(1/3, 1/3, 1/3)64.3 ± 0.627.6 ± 1.20.37
(0.3, 0.3, 0.4)64.1 ± 0.627.3 ± 1.20.38
(0.2, 0.2, 0.6)62.9 ± 0.726.1 ± 1.40.44
AuthorsOpenReview ↗

Response to Reviewer mgj9 (Part 2)

(Following above Part 1)

Every configuration with α∈[0.4,0.6]\alpha \in [0.4, 0.6], β∈[0.3,0.4]\beta \in [0.3, 0.4], γ∈[0.1,0.3]\gamma \in [0.1, 0.3] sits within 0.3 points of the default and within 0.01 of its rollout ratio. Performance degrades only when one term dominates, and the one that must not dominate is the diversity term, the only one carrying no information about current competence.

(2) Tuning the weights online instead of fixing them. We added a variant with no manual choice: the simplex is discretized at spacing 0.1 (66 vectors), each treated as an arm of a discounted EXP3 bandit; the weight vector is resampled every 10 steps and credited with the change in accuracy on a fixed held-in probe set of 512 training questions evaluated every 20 steps (1.2% extra generation). Result: 64.9 ± 0.5 / 28.4 ± 1.1 / Eff. 0.34, matching the fixed defaults within one standard deviation. Averaged over the second half of training the arm probabilities concentrate on (0.47, 0.32, 0.21), within 0.03 of the fixed (0.5, 0.3, 0.2) on every coordinate. We keep the fixed weights as the default because they avoid the probe-set overhead.

(3) Beyond a linear combination. Table 33 already bears on this: replacing the fixed linear form by a learned two-layer posterior over the same signals raises the offline rank correlation from 0.71 to 0.74 and accuracy by 0.1 points, at 4.8% extra wall-clock cost. The linear combination is therefore not optimal, but the headroom above it is small relative to its cost in this setting, and Appendix F.14 now says so.

In the main text, §3.2 states only that the weights are fixed once and points to the appendix, so the method section stays free of tuning detail.

W3: The mean absolute advantage is a coarse proxy for learning value. Rollouts within the same group may induce model updates in different directions and do not guarantee better performance.

A3: Thank you, this is an insightful point, and because it concerns the premise the whole method rests on, we have tested that premise rather than asserted it. New Appendix B.3 reports four studies aimed at exactly this question, and §3.1 now separates what the signal guarantees from what it does not. Your specific concern about updates pulling in different directions is addressed in Study 3 below, where we measure it directly.

To be precise about the claim: a nonzero Aˉi,t\bar{A}_{i,t} is necessary for a prompt to contribute to the GRPO gradient, and this is algebra rather than an empirical regularity, since a uniform group has every A^i,t(g)\hat{A}_{i,t}^{(g)} identically zero. It is not sufficient for a large improvement, for exactly the reasons you give. The question is how far the signal goes beyond necessity.

Study 1, per-prompt gradient norms. At checkpoints 50/150/250/350 we recompute, for each of the 128 prompts in that step's batch, the per-prompt GRPO loss and the ℓ2\ell_2 norm of its gradient with respect to the final transformer block and the LM head. Pearson correlation with Aˉi,t\bar{A}_{i,t}: 0.76, 0.74, 0.71, 0.69 (0.73 pooled); Spearman 0.78 pooled. Strong but well below one, as expected given the score-function, clipping, and token-aggregation factors.

Study 2, one-step learning gains. Gradient norms measure the size of an update, not its usefulness. At the same checkpoints we freeze θt\theta_t, build 40 candidate batches stratified so their batch-mean Aˉ\bar{A} spans [0.05,0.95][0.05, 0.95], apply exactly one GRPO update per candidate batch, and record the change in accuracy on a fixed probe set of 512 training-pool questions disjoint from all candidate batches (Table 5):

Predictor of one-step probe gainSpearman ρ\rhopp
Batch mean absolute advantage0.63<10−4< 10^{-4}
Predicted utility r~t\tilde{r}_t used by BanditSched0.58<10−4< 10^{-4}
Fraction of batch with EMA inside the band0.51<10−4< 10^{-4}
Mean annotated difficulty level0.290.0002
Mean question length in tokens0.070.38
Random ordering0.010.90

The top decile of batches by Aˉ\bar{A} improves the probe set by 0.42 points on average, against −0.03-0.03 for the bottom decile. Note the contrast with annotated difficulty (ρ=0.29\rho = 0.29), the quantity bucket-level curricula use.

Study 3, your "different directions" concern, measured directly. Using the same per-prompt gradients, the mean pairwise cosine similarity within a batch is 0.031 ± 0.004 for Full-GRPO, 0.036 ± 0.005 for Band-Filter, and 0.039 ± 0.004 for BanditSched; the ratio of the averaged batch gradient norm to the mean per-prompt norm is 0.21, 0.23, 0.24. Selecting high-advantage prompts therefore does not increase disagreement among the per-prompt updates. If anything the selected prompts are mildly more aligned, consistent with their concentrating on a narrower region of the competence frontier.

AuthorsOpenReview ↗

Response to Reviewer mgj9 (Part 3)

(Following above Part 2)

Study 4, an oracle upper bound and a negative control. The three studies above are correlational. In a reduced setting where recomputing the true advantage of every question at every step is affordable (N=2,000N = 2{,}000, T=100T = 100, B=64B = 64, G=8G = 8), Table 6 reports:

Selection ruleMATH-500Online rollout groups
Base model48.40
Anti-selection (lowest predicted utility)53.4 ± 1.06,400
Uniform56.9 ± 0.86,400
BanditSched59.1 ± 0.76,560
Oracle-Aˉ\bar{A} (impractical upper bound)59.7 ± 0.746,400

Inverting the score costs 3.5 points relative to uniform, which shows the signal carries directional information rather than merely correlating with outcomes, and BanditSched recovers 79% of the gap between uniform selection and the oracle while generating 7x fewer rollouts than the oracle needs (6,560 groups against 46,400).

What the paper now claims. Appendix B.3 states that roughly 40% of the rank variation in one-step gain is left unexplained by Aˉ\bar{A}, and the limitations section repeats it. The signal is nevertheless substantially more predictive than difficulty metadata, directionally correct, and close to an oracle that observes it exactly. Section 3.1 carries a short qualitative version of this, without numbers, and the composite reward is now presented as a consequence of the signal being partial rather than as an unmotivated design.

Q1: The proposed frontier probing strategy relies on question embedding. How does the embedding quality affect the framework's performance?

A4: Table 30 compared representations differing in several ways at once, so new Appendix F.12 degrades a single representation in a controlled way. We replace the default embedding by e~i(κ)∝ei+κξi\widetilde{\mathbf{e}}_i^{(\kappa)} \propto \mathbf{e}_i + \kappa \pmb{\xi}_i, with ξi\pmb{\xi}_i Gaussian noise drawn once per question before training at per-coordinate scale matched to the embedding, so κ\kappa is the noise-to-signal ratio (Table 31):

Noise κ\kappaP@10 IDP@10 OODMATH-500CD-OODProbe conversion
0 (default)0.820.7464.8 ± 0.522.1 ± 1.823%
0.250.790.7164.7 ± 0.521.8 ± 1.822%
0.50.730.6664.4 ± 0.621.0 ± 1.919%
1.00.650.5863.9 ± 0.620.1 ± 2.015%
2.00.520.4663.4 ± 0.719.3 ± 2.011%
pure noise0.110.1063.0 ± 0.718.8 ± 2.18%
no embeddingn/an/a63.1 ± 0.618.9 ± 2.18%

Three observations, and they answer the question for frontier probing specifically.

Degradation is graceful, not brittle. Halving the OOD neighbourhood precision costs 1.4 points on MATH-500, and a completely uninformative embedding returns the method to the no-embedding variant rather than below it: the scheduler falls back on competence features and the ridge penalty shrinks the random coordinates.

Frontier probing is the component most sensitive to embedding quality, which is expected since the frontier rule is defined entirely through cosine similarity. Its conversion rate falls from 23% all the way to the 8% of uniform probing as the embedding degrades, while main-batch selection degrades far more slowly. The held-out OOD splits consequently suffer more than MATH-500, by 3.3 against 1.8 points over the full range, consistent with our claim that the OOD gains come from tracking the frontier.

The offline diagnostic is predictive, so quality can be screened cheaply. Across the eleven representations in Tables 30 and 31, the Spearman correlation between P@10 OOD and final MATH-500 accuracy is 0.91, so candidate representations can be screened by nearest-neighbour retrieval without running RL.

Table 32 adds a label-budget study, since embedding quality here is partly a function of annotation: 0 labels (unsupervised PCA) gives 64.1 ± 0.6, 500 gives 64.4 ± 0.6, 1,000 gives 64.6 ± 0.5, and 2,000 and 4,000 both give 64.8. One thousand labels recover most of the benefit, and the unsupervised fallback still improves on no embedding, so the method does not depend on a large annotation budget. The limitations section now states the corresponding failure mode: on a domain where question semantics are not captured by the base model's hidden states, the OOD advantage should be expected to shrink.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for the detailed and constructive reading. Every point raised led to a concrete change in the paper. All changes are highlighted in blue, and all section, table, and figure numbers below refer to the revised version. To keep the main text within the page limit, the main text states each new conclusion and the appendix carries the evidence; the one related-work discussion that could not fit has been moved to a supplementary related-work section. Where the new material lives:

Reviewer pointWhere it is addressed
mgj9 W1, coverage of the 4,820 selected questionsApp. F.10, plus four new rows in Table 27
mgj9 W2, hand-crafted reward weightsApp. F.14 (weight-simplex sweep and an online tuner)
mgj9 W3 / YGGs W1, is the utility signal predictive?App. B.3 (four studies, including an oracle and a negative control)
mgj9 Q1, embedding qualityApp. F.12 (controlled degradation and label-budget studies)
mgj9 Q2, hyperparameters§4.1 and App. E.5 (transfer study)
QSBS W1, advantage dynamics§4.4 and App. F.15
QSBS W2, per-problem selection frequencyApp. F.9
YGGs W2, lag τ\tau and embedding projection§3.2, App. E.1 and App. E.2
YGGs W3, complexity vs simpler strategiesApp. F.6 (incremental build-up, four simpler schedulers, measured overhead)
YGGs W4, wall-clock rather than rollout countApp. F.4 (time to target and token accounting)
YGGs W5, the drop near step 400Figure 2(a) regenerated, App. F.17, checkpoint policy in §4.1
YGGs W6, MoPPS and CurESApp. A (positioning), baselines in Tables 1, 2, 10, 11, 12, 17, 19, and App. F.2
YGGs W7, recent benchmarks§4.2 and App. F.1 (AIME 2025, AIME 2026, HMMT Feb 2026)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.