acceptodds
Withdrawn from ICLR 2027

FEAST: Fair and Efficient Adaptive Sampling for Task-Free On-Policy Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language agents trained from self-generated experience must allocate a limited rollout budget across heterogeneous capabilities. A common strategy is to prioritize capabilities with strong current learning signals, but in on-policy learning these signals are endogenous: past allocation changes the policy and therefore the evidence available for future allocation. This can create a feedback loop in which under-trained capabilities receive increasingly fewer opportunities to improve. We formalize this effect through the Quota-Efficiency Law (QEL), which shows that allocation history can change learning efficiency even under matched capability-specific exposure. Motivated by this, we introduce FEAST, a task-free adaptive sampling framework that separates opportunity allocation from compute allocation. FEAST dynamically discovers capability clusters while preserving their learning histories, maintains continued sampling opportunities across clusters, and reserves a rollout budget for each selected cluster. Within each budget, it progressively concentrates computation on prompts with informative reward variation. Across GURU-18K and SMC, FEAST improves average performance over competitive adaptive sampling baselines under matched rollout budgets, while better preserving learning progress on weaker capabilities. Ablation studies further support the complementary roles of cross-cluster opportunity protection and within-cluster selective allocation. Code will be released for reproducibility.

Primary area:
reinforcement learning
Keywords:
reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_6fwsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes FEAST, a task-free sampling framework for multi-capability on-policy reinforcement learning. It introduces the Quota-Efficiency Law (QEL) to analyze how allocation history can affect learning gains even at matched capability-specific exposure. FEAST maintains persistent atomic-unit histories, dynamically groups these units into capability clusters, reserves rollout budgets for selected clusters, and progressively allocates rollouts within each cluster based on reward variance. Experiments on GURU-18K and SMC with Qwen3-1.7B and 8B report higher average performance than the compared GRPO sampling baselines under matched rollout budgets, with ablations examining the two allocation levels.

Critical strengths
  1. The paper addresses a practical problem in multi-capability reinforcement learning: weaker capabilities may be repeatedly overlooked during training. Combining opportunity protection across capability clusters with selective rollout allocation within each cluster is a sensible approach.
  2. The evaluation covers two datasets, two model sizes, and several sampling baselines. FEAST improves average performance under a fixed rollout budget, and the ablations provide some evidence for the benefits of both allocation levels.
Critical weaknesses
  1. The presentation needs improvement. The abstract and introduction do not clearly convey the main contribution or the reasoning behind the design, and the main text makes the full training procedure difficult to follow. The figures offer limited help in understanding the method.
  2. The connection between QEL and FEAST is not sufficiently established. The result that allocation history reduces learning efficiency relies on strong assumptions whose applicability is not examined in the experiments. FEAST’s guarantee mainly establishes a positive probability of cluster selection; it does not show that the method meaningfully mitigates the effects described by QEL.
Questions for authors
  1. How is the GRPO loss normalized for response groups containing 2, 6, and 22 rollouts? Does allocating more rollouts also increase a prompt’s weight in the policy update?
  2. Could the authors walk through one FEAST training iteration with a concrete example and clarify the relationships among atomic units, dynamic capability clusters, budget allocation, and feedback updates in Figure 3? How do the modules in the figure correspond to Equations (5)–(8)?
4 comments
AuthorsOpenReview ↗

To Reviewer 6fws(Part I)

We sincerely thank you for recognizing the practical problem FEAST addresses and the breadth of its evaluation. Your questions helped us explain the GRPO weighting, walk through one update, and make the connection between QEL and FEAST explicit. We would be glad to discuss any remaining concerns.

(Q1) How is the GRPO loss normalized for response groups containing 2, 6, and 22 rollouts? Does allocating more rollouts also increase a prompt’s weight in the policy update?

Thank you for asking how the GRPO loss is normalized for response groups containing 2, 6, and 22 rollouts, which helped us state the advantage normalization and loss aggregation explicitly in Appendix A.2.6.

  • (GRPO normalization) On how is the GRPO loss normalized, GRPO normalizes rewards within each prompt's full response group in the same way for 2, 6, or 22 responses.
    • The implementation computes the group mean Rˉx\bar R_x and sample standard deviation sxs_x, then forms Axi=(Rxi−Rˉx)/(sx+ϵadv)A_{xi}=(R_{xi}-\bar R_x)/(s_x+\epsilon_{\mathrm{adv}}) with a small stabilizer ϵadv\epsilon_{\mathrm{adv}}. Appendix A.2.6 now states this advantage, and Appendix A.2.4 specifies the 2, 6, and 22 group sizes and groups all rollouts of a prompt before this reward-centered update.
  • (Prompt weighting) Allocating more rollouts does increase a prompt’s weight in the policy update, because each retained response contributes one term to the loss.
    • The policy loss averages tokens within each response and responses within an optimization minibatch. For valid responses B\mathcal B, the aggregation, now Equation (28) in Appendix A.2.6, is
    ∑(x,i)∈B1Txi∑t=1Txiℓxitclip(θ).\sum_{(x,i)\in\mathcal B}\frac{1}{T_{xi}} \sum_{t=1}^{T_{xi}}\ell^{\mathrm{clip}}_{xit}(\theta).
- Here $\ell^{\mathrm{clip}}$ is the clipped GRPO surrogate and $T_{xi}$ counts valid response tokens. - Constant-reward groups have zero task advantages and contribute no task gradient (Appendix A.2.4).
AuthorsOpenReview ↗

To Reviewer 6fws(Part II)

(W1) The presentation needs improvement. The abstract and introduction do not clearly convey the main contribution or the reasoning behind the design, and the main text makes the full training procedure difficult to follow. The figures offer limited help in understanding the method.

(Q2) Could the authors walk through one FEAST training iteration with a concrete example and clarify the relationships among atomic units, dynamic capability clusters, budget allocation, and feedback updates in Figure 3? How do the modules in the figure correspond to Equations (5)–(8)?

Thank you for asking us to walk through one FEAST training iteration with a concrete example, which helped us add a step-by-step walkthrough to Appendix A.2.6 and map each module in Figure 3 to its equation.

  • (Walkthrough and equation mapping) As for how do the modules in the figure correspond to Equations (5)–(8) (Equations (3)–(6) in the revised paper), each module in Figure 3 maps to one step of a FEAST update (Table R4.1).

    • The policy stays fixed during generation and is then updated by GRPO.
    • Table R4.1 is now Table 5 in Appendix A.2.6, which combines Algorithm 1 and the rollout counts of Appendix A.2.4 into one walkthrough.

    Table R4.1. One FEAST update in the main configuration (64 atomic units, four selected clusters, 1,024 rollouts), with the matching parts of the paper.

    StepWhat FEAST doesPaper reference
    Retain historiesKeep 64 persistent atomic units and their statesSection 4.1
    RegroupEvery 10 updates, regroup units by current-policy representations and smoothed learning trendsEquation (3)
    Compute prioritiesCombine the exploration floor, rollout shortfall, and learning evidence (reward variation ui,tu_{i,t}, average reward mi,tm_{i,t}, and its recent change ℓi,t\ell_{i,t})Equation (4)
    Select and reserveSelect four clusters without replacement, reserving 256 rollouts eachEquations (2), (5)
    Allocate progressivelyIn each cluster, 2 rollouts for 32 prompts, 4 more for the top 16, and 16 more for the top 8 (256 rollouts)Equation (6)
    Update the policyTrain on 128 prompt groups: 64 with 2 responses, 32 with 6, and 32 with 22 (1,024 responses)Equation (28), Algorithm 1
    Update feedbackUpdate unit histories. For example, pilot rewards 0 and 1 give an average reward of 0.5 and a squared reward difference of 1Figure 3, Appendix A.2
  • (Relationships among modules) In terms of the relationships among atomic units, dynamic capability clusters, budget allocation, and feedback updates, atomic units keep histories, clusters group units for selection, the reserved budgets mechanism fixes each selected cluster's budget, and feedback refreshes the unit histories.

    • As specified in Appendix A.2, average reward and reward variation are computed from the common initial pilot with prompts weighted equally. Trends are updated from rollouts with reward variation, and shortfalls from all generated rollouts.
  • (Presentation) The revised introduction now states the main contribution and the reasoning behind the design, and the Figure 3 caption maps each module to its equation.

    • The introduction now follows the path from how past allocation shapes current rewards, to QEL, and then to FEAST's separation of opportunity and compute allocation. The walkthrough in Table R4.1 and the loss definition above have been added to Appendix A.2.6 (Table 5 and Equation (28)) and are referenced from Section 4.3.
AuthorsOpenReview ↗

To Reviewer 6fws(Part Ⅲ)

(W2) The connection between QEL and FEAST is not sufficiently established. The result that allocation history reduces learning efficiency relies on strong assumptions whose applicability is not examined in the experiments. FEAST’s guarantee mainly establishes a positive probability of cluster selection; it does not show that the method meaningfully mitigates the effects described by QEL.

Thank you for noting that the connection between QEL and FEAST is not sufficiently established, which helped us make this connection explicit and test it on GURU reasoning tasks.

  • (Connection between QEL and FEAST) The link is direct: QEL shows that allocation history changes learning efficiency at matched exposure, so FEAST adapts allocation to history-dependent feedback.

    • History dependence makes current learnability of a certain capability partly a product of past allocation. In the decomposition of Theorem 1 (Appendix A.1.1), direct learning and cross-capability transfer are both evaluated at the policy produced by earlier updates. Low average reward (mi,tm_{i,t}) or little reward variation (ui,tu_{i,t}) on an under-allocated capability can therefore reflect its allocation path. A scheduler that excludes capabilities by these current values alone can reinforce that path (Figure 2), as SEC-Greedy in paper Figure 5 illustrates.
    • FEAST's priority combines current learning evidence with allocation history. In the learning-evidence surplus of Equation (4), ui,tΦ(mi,t)u_{i,t}\Phi(m_{i,t}) favors units whose prompts the current policy solves only sometimes, and the learning trend ∣ℓi,t∣|\ell_{i,t}| responds to both improvement and regression, so changes from signed cross-capability transfer also shift priority. The rollout shortfall ai,ta_{i,t} raises the priority of units whose past rollouts fell below an equal share, the under-allocation that the suppressive regime links to lower efficiency.
    • Opportunity protection lets FEAST re-measure the current policy's performance on different capabilities as the policy evolves. Positive sampling weights give a geometric waiting-time bound for eligible cluster selection (Proposition 4, Appendix A.2.3), and reserved budgets prevent a selected cluster's rollouts from being reallocated to higher-variance prompts elsewhere. This lets FEAST re-estimate the average reward, reward variation, and recent reward change of low-priority clusters under the current policy. Atomic-unit histories persist across regrouping, so shortfall and trend statistics carry over as capability clusters change.
  • (Assumptions and their applicability) Regarding the strong assumptions whose applicability is not examined in the experiments, QEL identifies the problem, and FEAST does not rely on these assumptions. We also add two experiments to prove that QEL holds on reasoning tasks.

    • The assumptions of Theorem 1 and Proposition 2 only explain when allocation history lowers learning efficiency. FEAST's priorities are computed from observed pilot rewards and rollout counts, and its waiting-time bound (Proposition 4) needs only positive, bounded sampling weights, which the exploration floor keeps positive. FEAST's gains are established empirically on GURU-18K and SMC at both model scales (paper Tables 2–3) and in the ablations (Table R4.5).
    • On GURU Math, matched Math exposure yields different Math scores under different allocation paths. We added a fixed Math quota experiment (Appendix A.7.2) in which the varied factor is the Math share of each update, 12.5% or 50%. Both runs start from the same model and see the same Math prompts in the same order, so the 12.5% run reaches the same Math exposure later, after more Code and Science training. The 12.5% run scores higher on MATH-500 (Table R4.2), so matched Math exposure does not determine the Math score, consistent with QEL. The direction matches positive transfer from Code and Science, which the signed transfer term in Theorem 1 allows.

    Table R4.2. Same Math exposure under two allocation paths. Both runs train Qwen3-1.7B-Base on the same 512 Math prompts in the same order and differ only in the Math share of each update. The 95% CI comes from a paired question-level bootstrap.

    Math share per updateMath promptsCode + Science promptsUpdatesMATH-500
    12.5%5123,5843261.20
    50%512512855.40
    Difference (12.5% − 50%), 95% CIN/AN/AN/A+5.80 [2.00, 9.80]
AuthorsOpenReview ↗

To Reviewer 6fws(Part Ⅳ)

  • (Beyond positive selection probability) Beyond a positive probability of cluster selection, FEAST bounds the expected waiting time, the exploration floor shortens the longest waits in training, and the reserved budgets make sure each cluster's capability is sufficiently trained.

    • Proposition 4 (Appendix A.2.3) bounds the expected waiting time and its tail probability for eligible cluster selection. After selection, reserved budgets prevent its rollouts from being reallocated to other clusters. Within the cluster, each prompt slot goes to an atomic unit with probability proportional to its priority, which includes the rollout shortfall (Appendix A.2.2).
    • The exploration floor shortens the longest waits while every atomic unit stays covered. We added an analysis of atomic-unit service (Appendix A.7.5), comparing Full FEAST with FEAST w/o exploration floor (ϵ=0\epsilon=0) on the same 64 atomic units. Both runs serve every atomic unit. Compared with FEAST w/o exploration floor, FEAST substantially reduces the P90 and maximum waits (Table R4.3), indicating that the floor protects the units that would otherwise wait longest.

    Table R4.3. Atomic-unit coverage and waiting times over 100 updates on the same 64 units. Service means receiving at least one prompt. Waiting statistics summarize each unit's longest unserved interval.

    MetricFull FEASTFEAST w/o exploration floor (ϵ=0\epsilon=0)
    Units served at least once64/6464/64
    Longest wait, full training: median / P90 / max (updates)10 / 20.7 / 598 / 52.5 / 73
  • (Mitigation of QEL effects) With the reserved budgets mechanism, FEAST keeps a Zebra advantage at equal Zebra exposure and after further shared training, indicating that the mechanism mitigates the effects described by QEL.

    • We added a matched-exposure experiment on GURU Zebra (Appendix A.7.2) comparing Full FEAST with FEAST w/o reserved budgets mechanism, which keeps cluster selection but lets high-variance prompts in one cluster take rollouts from other selected clusters. Zebra receives fewer rollouts under FEAST w/o reserved budgets mechanism, so we give it one extra update until both models have seen the same number of Zebra rollouts. Both models then train on the same Zebra, Code, and Science prompts in the same order.
    • After this matching, Full FEAST scores far higher than FEAST w/o reserved budgets mechanism (Table R4.4). Equal Zebra exposure therefore does not give equal Zebra performance, which is the history dependence QEL describes. Further shared updates that spend half their rollouts on Zebra narrow the gap without closing it: after four of them, FEAST w/o reserved budgets mechanism still scores far below what Full FEAST had already reached at the matched count. The deficit from earlier under-allocation is therefore hard to recover: closing the gap under a fixed budget would require taking even more rollouts from other capabilities.

    Table R4.4. Zebra scores at equal Zebra exposure. After the counts are matched, both models train on the same prompts, half of them from Zebra. The 95% CI comes from a paired question-level bootstrap.

    StageCumulative Zebra rollouts per modelFull FEASTFEAST w/o reserved budgets mechanismGap, 95% CI
    After matching rollout counts3,36031.878.78+23.09 [20.82, 25.49]
    After two shared updates4,38432.6813.53+19.15 [16.92, 21.47]
    After four shared updates5,40833.5417.41+16.13 [14.11, 18.18]
    • FEAST w/o reserved budgets mechanism also has a lower Avg. than Full FEAST (Table R4.5), suggesting that the protection benefits overall performance.

    Table R4.5. Avg. after removing or changing one FEAST component. All runs use Qwen3-1.7B-Base, 100 updates, and 1,024 rollouts per update.

    Variant1.7B Avg.
    Full FEAST36.32
    FEAST w/o inter-cluster fairness: global sampling without atomic units or clusters35.66
    FEAST w/o intra-cluster efficiency: eight responses per prompt, no later rounds34.66
    FEAST w/o reserved budgets mechanism: high-variance prompts can take rollouts from other selected clusters32.56
    FEAST w/o capability clusters: four atomic units, 32 prompts each34.01
    FEAST w/o dynamic clustering: fixed initial clusters35.65
    FEAST w/o exploration floor (ϵ=0\epsilon=0)35.70
    FEAST w/o rollout shortfall (λa=0\lambda_a=0)35.94
    FEAST w/o learning trend (γ=0\gamma=0)35.85

Thank you again for helping us clarify the training procedure and the link between QEL and FEAST. We would be happy to discuss any remaining questions.

Reviewer_dDr5OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper focuses on task-free on-policy learning and studies how past training allocation decisions change the policy and, consequently, the learning signals used for future allocation. This feedback can leave some capabilities with insufficient training opportunities. To address this issue, the paper proposes FEAST, which dynamically groups training prompts using model representations and learning feedback while maintaining persistent histories for finer-grained units. It adjusts cluster sampling priorities to preserve training opportunities for under-allocated capabilities and reserves a rollout budget for each selected cluster. Within each cluster, additional rollouts are allocated to prompts with higher reward variance. The paper also analyzes allocation-history effects at matched capability-specific exposure. Experiments on GURU-18K and SMC compare FEAST with existing sampling methods, while ablation studies examine cluster-level allocation and within-cluster rollout allocation.

Critical strengths
  1. Empirical improvements. FEAST improves average performance over the evaluated GRPO sampling baselines on GURU-18K and SMC under matched online rollout budgets. Although the gains are modest in some settings, they suggest practical benefits from the proposed allocation strategy.

  2. Conceptually simple design. The method reserves budgets for selected clusters and allocates additional rollouts within each cluster based on reward variance. The overall idea is easy to understand and appears feasible to incorporate into an existing GRPO training pipeline without modifying the model architecture.

Critical weaknesses
  1. Limited methodological novelty. Group-based adaptive sampling and variance-guided rollout allocation have already been explored in prior work, including ADS and VIGOR. FEAST combines these ideas with dynamic regrouping and opportunity protection, but the paper does not sufficiently establish the contribution of these additions beyond a refinement of existing sampling strategies. More targeted ablations, particularly fixed versus dynamic grouping and separate removal of the protection mechanisms, would help clarify what is substantively new.

  2. Insufficient comparison with the closest alternative. VIGOR is directly relevant to FEAST’s within-cluster rollout allocation, yet the paper does not report a separately identified VIGOR baseline. The “w/o Inter-Fairness” ablation retains progressive allocation and partially addresses this comparison, but its equivalence to VIGOR is unclear. Moreover, the full method improves over this ablation by only 0.66 percentage points in a single-seed experiment. Together with the modest gains on GURU-18K, this leaves limited evidence that the added grouping and opportunity-protection mechanisms provide a reliable improvement over simpler alternative

Questions for authors
  1. Comparison with VIGOR. Is the “w/o Inter-Fairness” ablation equivalent to VIGOR under the same training settings? If not, what are the differences, and can the authors provide a direct comparison to establish the benefit of FEAST’s additional mechanisms?

  2. Ablation of individual mechanisms. Can the authors isolate the effects of dynamic regrouping, the exploration floor, and rollout-shortfall compensation while keeping the remaining components unchanged? The current ablation removes the entire capability-grouping structure, making it difficult to identify which additions are responsible for the gains.

  3. Evidence of opportunity protection. Beyond the domain-level allocation curves, do the training logs show improved exposure coverage or shorter waiting times for under-allocated atomic units? Since selecting a cluster does not guarantee exposure to every member unit, such evidence would clarify whether the proposed protection benefits the units it is intended to support.

  4. End-to-end computational cost. How does FEAST compare with the main baselines in total training time or GPU-hours, including representation extraction and regrouping? Given the modest gains on GURU-18K, this comparison would help assess whether the improvements justify the additional computation.

5 comments
AuthorsOpenReview ↗

To Reviewer dDr5(Part I)

We sincerely thank you for recognizing FEAST's gains under matched rollout budgets and its simple design. Your suggestions helped us compare FEAST with VIGOR, isolate each protection mechanism, and measure atomic-unit service and training cost. We would be glad to discuss any remaining concerns.

(W2) Insufficient comparison with the closest alternative. VIGOR is directly relevant to FEAST’s within-cluster rollout allocation, yet the paper does not report a separately identified VIGOR baseline. The “w/o Inter-Fairness” ablation retains progressive allocation and partially addresses this comparison, but its equivalence to VIGOR is unclear. Moreover, the full method improves over this ablation by only 0.66 percentage points in a single-seed experiment. Together with the modest gains on GURU-18K, this leaves limited evidence that the added grouping and opportunity-protection mechanisms provide a reliable improvement over simpler alternative

(Q1) Comparison with VIGOR. Is the “w/o Inter-Fairness” ablation equivalent to VIGOR under the same training settings? If not, what are the differences, and can the authors provide a direct comparison to establish the benefit of FEAST’s additional mechanisms?

Thank you for asking whether the w/o Inter-Fairness ablation is equivalent to VIGOR under the same training settings, which helped us add a direct comparison with VIGOR and clarify the reliability of FEAST's gains.

  • (Equivalence to VIGOR) The w/o Inter-Fairness ablation is similar but not exactly equivalent to VIGOR [1]: it applies FEAST's [2,4,16] increments in a global pool to align the budget, while VIGOR uses [2,4,8,16].

    • Appendix A.3 specifies this ablation, and Section 4.3 describes FEAST as placing variance-guided progressive allocation inside the reserved budgets.
  • (Direct comparison) In a direct comparison to establish the benefit of FEAST’s additional mechanisms, FEAST outperforms the published VIGOR allocation rule at the same rollout budget.

    • We added a comparison with the published VIGOR allocation rule (Appendix A.7.3). The comparator applies VIGOR's [2,4,8,16] increments over all 128 prompts, following Sections 4.2 and 5.1 of [1], while FEAST applies [2,4,16] inside four reserved budgets. Both use 1,024 rollouts per update (Table R3.1).
    • The two methods share variance-guided allocation and differ in FEAST's cluster structure and reserved budgets, so FEAST's higher Avg. supports the benefit of these additional mechanisms.

    Table R3.1. FEAST versus the VIGOR allocation rule at the same budget. Qwen3-1.7B-Base, 100 updates, 1,024 rollouts per update. Both allocate rollouts progressively by reward variance. FEAST does so inside four reserved budgets, and VIGOR does so over all 128 prompts.

    MethodRollouts added per roundPrompts per roundAvg.
    Full FEAST[2,4,16]32/16/8 in each of four clusters36.32
    VIGOR rule[2,4,8,16]128/64/32/16 in one global pool35.58
  • (Fairness across capabilities) FEAST's main goal is fairness across capabilities: it leaves no benchmark far behind the best scheduler, whereas ADS and BOTS stall on Zebra.

    • FEAST's largest gap to the best scheduler on any of the nine benchmarks is the smallest among the six GRPO schedulers at both scales (Table R3.2). At 1.7B, FEAST is never the lowest on any benchmark. At 8B, it is the best or tied for best on four of nine.
    • ADS and BOTS rely on current success estimates, which are near zero for Zebra at the start of training, so Zebra barely improves under them at 1.7B and stays far below FEAST (paper Table 2).

    Table R3.2. Largest gap to the best scheduler. For each method, we find the benchmark where it trails the best of the six GRPO schedulers the most and report that gap. A smaller gap means the method does not fall far behind on any of the nine benchmarks (paper Tables 7–8).

    ModelRandomTACADSDUMPBOTSFEAST
    1.7B7.0812.1932.798.9329.663.73
    8B4.563.7322.355.069.532.15
  • (Reliability of the gains) FEAST's gains over Random have question-level 95% CIs above zero at both model scales, supporting a reliable improvement over simpler alternative.

    • FEAST also has a smaller seed SD at both scales, suggesting more stable performance across seeds (Table R3.3).
    • Repeating the single-seed ablations with more training seeds is future work.

    Table R3.3. FEAST versus Random on GURU-18K (Appendix A.7.1). The 95% CI of the paired difference comes from a question-level bootstrap.

    ModelRandomFEASTFEAST − Random95% CI
    1.7B35.43 ± 0.1836.18 ± 0.13+0.75[0.20, 1.31]
    8B50.65 ± 1.0151.69 ± 0.08+1.04[0.30, 1.79]
AuthorsOpenReview ↗

To Reviewer dDr5(Part II)

  • (Size of the gains) Behind the modest gains on GURU-18K in Avg., FEAST's gains on individual benchmarks are substantial.

    • Avg. weights the nine benchmarks equally, so a gain on one benchmark contributes one ninth of its size to Avg.
    • Large gains on individual benchmarks, such as FinQA at 1.7B and CodeIO at 8B, each add less than one point to Avg. (Table R3.4, computed from the per-seed scores in paper Tables 7–8).

    Table R3.4. Benchmarks with large FEAST gains (Appendix A.7.1). Gain is the FEAST − Random difference averaged over training seeds. Each benchmark contributes one ninth of its gain to Avg.

    ModelBenchmarkGainContribution to Avg.
    1.7BFinQA+7.08+0.79
    8BCodeIO+4.56+0.51
    8BMultiHierTT+3.61+0.40

[1] Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning. arXiv preprint 2026.

(W1) Limited methodological novelty. Group-based adaptive sampling and variance-guided rollout allocation have already been explored in prior work, including ADS and VIGOR. FEAST combines these ideas with dynamic regrouping and opportunity protection, but the paper does not sufficiently establish the contribution of these additions beyond a refinement of existing sampling strategies. More targeted ablations, particularly fixed versus dynamic grouping and separate removal of the protection mechanisms, would help clarify what is substantively new.

(Q2) Ablation of individual mechanisms. Can the authors isolate the effects of dynamic regrouping, the exploration floor, and rollout-shortfall compensation while keeping the remaining components unchanged? The current ablation removes the entire capability-grouping structure, making it difficult to identify which additions are responsible for the gains.

Thank you for asking us to isolate the effects of dynamic regrouping, the exploration floor, and rollout-shortfall compensation, which helped us separate the contribution of each FEAST mechanism and clarify what is new beyond ADS and VIGOR.

  • (Novelty beyond ADS and VIGOR) FEAST goes beyond a refinement of existing sampling strategies by separating opportunity allocation from compute allocation: ADS [1] and VIGOR [2] let one current utility signal decide both, so high-utility prompts can absorb the budget of weak capabilities.
    • A single signal creates a feedback loop, and QEL explains why this matters. By QEL, current feedback partly results from past allocation. When the same feedback decides both which capabilities train and how many rollouts they get, weak feedback reduces a capability's rollouts, and fewer rollouts keep its feedback weak. FEAST therefore decides opportunities from allocation history, through the exploration floor, rollout shortfall, and learning trends kept in persistent atomic units, and spends compute by reward variance only inside each reserved budget.
    • ADS falls into this loop. ADS favors clusters with higher observed success, so a capability with low initial success keeps receiving few rollouts. At 1.7B, ADS leaves Logic near its initial score, and FEAST scores higher than ADS on Logic at both model scales (paper Table 2). Section 5.2.2 of the paper links this gap to repeatedly favoring capabilities that already provide useful feedback.
    • VIGOR lacks opportunity protection. VIGOR allocates rollouts by reward variance in a global pool, so high-variance prompts can take rollouts from weak capabilities. FEAST achieves a higher Avg. than the VIGOR rule at the same budget (Table R3.1), and removing the reserved budgets mechanism gives the largest drop among the ablations (Table R3.5), suggesting that separating the two allocation levels is central to FEAST's gains.
AuthorsOpenReview ↗

To Reviewer dDr5(Part Ⅲ)

  • (Isolated effects of individual mechanisms) Dynamic regrouping, the exploration floor, rollout-shortfall compensation, and the reserved budgets mechanism each contribute to FEAST's performance, as shown by the suggested fixed versus dynamic grouping and separate removal of the protection mechanisms.

    • We added ablations in which each variant removes or changes one component of Full FEAST and keeps the others unchanged (Table R3.5). The FEAST w/o inter-cluster fairness and FEAST w/o intra-cluster efficiency rows come from Table 4 of the original submission, the remaining variants are now in revised Table 4 (Section 5.2.4), and Appendix A.3 defines each variant. FEAST w/o dynamic clustering keeps the initial clusters throughout training, and FEAST w/o reserved budgets mechanism keeps cluster selection and pilot rollouts, but in the later rounds high-variance prompts in one cluster can take rollouts from other selected clusters.
    • FEAST w/o dynamic clustering lowers Avg. at 1.7B. This indicates that updating groups with current-policy representations and learning trends helps allocation as capabilities change.
    • FEAST w/o exploration floor (ϵ=0\epsilon=0) lowers Avg. The floor provides the positive selection probability in Proposition 4, so this result indicates that keeping every eligible cluster selectable improves training.
    • FEAST w/o rollout shortfall (λa=0\lambda_a=0) lowers Avg., showing that raising the priority of under-allocated units helps, as the history dependence in QEL suggests.
    • FEAST w/o reserved budgets mechanism gives the largest drop among the 1.7B variants. This suggests that the reserved budgets mechanism is the most important of the tested mechanisms.
    • FEAST w/o capability clusters and FEAST w/o inter-cluster fairness, which removes the whole capability structure, also lower Avg., indicating that the capability structure itself contributes.

    Table R3.5. Avg. after removing or changing one FEAST component. All runs use Qwen3-1.7B-Base, 100 updates, and 1,024 rollouts per update.

    Variant1.7B Avg.
    Full FEAST36.32
    FEAST w/o inter-cluster fairness: global sampling without atomic units or clusters35.66
    FEAST w/o intra-cluster efficiency: eight responses per prompt, no later rounds34.66
    FEAST w/o reserved budgets mechanism: high-variance prompts can take rollouts from other selected clusters32.56
    FEAST w/o capability clusters: four atomic units, 32 prompts each34.01
    FEAST w/o dynamic clustering: fixed initial clusters35.65
    FEAST w/o exploration floor (ϵ=0\epsilon=0)35.70
    FEAST w/o rollout shortfall (λa=0\lambda_a=0)35.94
    FEAST w/o learning trend (γ=0\gamma=0)35.85

[1] Learning at the Right Pace: Adaptive Data Scheduling Improves LLM Reinforcement Learning. arXiv preprint 2026.

[2] Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learning. arXiv preprint 2026.

Reviewer_4zznOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies adaptive allocation of on-policy training experience across heterogeneous capabilities. It introduces the Quota–Efficiency Law, which analyzes learning gains at matched capability-specific exposure. The paper also proposes a framework combining persistent atomic-unit histories, dynamic capability clustering, protected cluster budgets, and variance-guided progressive rollout allocation. However, I am not sufficiently convinced that the theoretical analysis and empirical evidence establish the distinctive contribution of the proposed mechanisms.

Critical strengths

The paper proposes a new framework combining persistent atomic-unit histories, dynamic capability clustering, protected cluster budgets, and variance-guided progressive rollout allocation, which considers history, learning signal, fairness, and efficiency structure in the adaptive allocation.

Critical weaknesses
  1. The theoretical contribution and its connection to the proposed algorithm are limited. Appendix A.1 attempts to explain QEL using optimization theory. However, I do not see a clear need for this theoretical treatment, and the conclusion relies on numerous assumptions that are poorly motivated and difficult to justify.

  2. The ablations do not isolate the components most relevant to the claimed contribution. While this paper proposes a new framework considering history, learning signal, fairness, and efficiency, the ablation studies do not clearly show the significance of each design in the framework.

Questions for authors
  1. It would be helpful to present Section 3 better. The formulation and model are not clearly introduced in Section 3; for example, why does Vk(θ)V_k(\theta) have a Lipschitz gradient, and how do we estimate g^j,t\widehat{g}_{j,t}?

  2. Could you explain more about the Inter-Fairness ablation study? Now it only writes, "removes the capability structure and samples 128 prompts globally". I wonder if it keeps the sampling procedure the same as sampling from different atomic units in one cluster (as in eq 6), or if it also ignores the atomic unit structure. Also, could you explain the main necessity of using clusters, because it seems we can also apply the same Inter Fairness directly to atomic units.

  3. The rollout setting in section 4.3 is progressive increments [2,4,16]. Have you tried a larger rollout at the beginning, because looking at the variance of 2 rollouts may introduce large noise.

4 comments
AuthorsOpenReview ↗

To Reviewer 4zzn(Part I)

We sincerely thank you for recognizing how FEAST combines history, learning signals, fairness, and efficiency in one allocation framework. Your comments helped us clarify Section 3, isolate each design, and examine the initial pilot size. We would be glad to discuss any remaining concerns.

(W1) The theoretical contribution and its connection to the proposed algorithm are limited. Appendix A.1 attempts to explain QEL using optimization theory. However, I do not see a clear need for this theoretical treatment, and the conclusion relies on numerous assumptions that are poorly motivated and difficult to justify.

(Q1) It would be helpful to present Section 3 better. The formulation and model are not clearly introduced in Section 3; for example, why does Vk(θ)V_k(\theta) have a Lipschitz gradient, and how do we estimate g^j,t\hat{g}_{j,t}?

Thank you for noting that it would be helpful to present Section 3 better, which helped us rewrite Section 3 around the problem that FEAST addresses and make its connection to FEAST explicit.

  • (Revised Section 3) We have revised Section 3 to pose the problem that FEAST addresses: a capability's learning efficiency depends on allocation history.
    • Section 3 now defines VkV_k and the matched-exposure learning efficiency in one equation, explains why equal exposure can give unequal gains and when sparse allocation hurts, and ends with learning-opportunity fairness, the principle FEAST builds on. The formal setup, assumptions, and proofs remain in Appendix A.1.
  • (Lipschitz gradient) The revised Section 3 no longer involves the Lipschitz gradient of Vk(θ)V_k(\theta), and you can find it in Theorem 1 (Appendix A.1.1).
    • The Lipschitz gradient and bounded second moments are standard assumptions in convergence analyses of stochastic gradient methods [1, 2]. In Theorem 1, they only bound the remainder RN\mathcal R_N of the path decomposition.
  • (Estimating g^j,t\hat g_{j,t}) The revised Section 3 no longer involves g^j,t\widehat g_{j,t}, and you can find it in Appendix A.1.1.
    • In practice, g^j,t\widehat g_{j,t} is the GRPO policy gradient from current-policy rollouts of capability jj, so no separate estimator is needed.
    • Its conditional mean gj,tg_{j,t} enters only the analysis in Theorem 1. FEAST's scheduler uses pilot rewards and exposure statistics, which it observes directly during training.
  • (Connection to the proposed algorithm) The theory's connection to the proposed algorithm is direct: QEL describes the phenomenon that a capability's learning efficiency depends on allocation history. FEAST's history-dependent allocation responds to this phenomenon.
    • History dependence makes current learnability of a certain capability partly a product of past allocation. In the decomposition of Theorem 1 (Appendix A.1.1), direct learning and cross-capability transfer are both evaluated at the policy produced by earlier updates. Low average reward (mi,tm_{i,t}) or little reward variation (ui,tu_{i,t}) on an under-allocated capability can therefore reflect its allocation path. A scheduler that excludes capabilities by these current values alone can reinforce that path (Figure 2), as SEC-Greedy in paper Figure 5 illustrates.
    • FEAST's priority combines current learning evidence with allocation history. In the learning-evidence surplus of Equation (4), ui,tΦ(mi,t)u_{i,t}\Phi(m_{i,t}) favors units whose prompts the current policy solves only sometimes, and the learning trend ∣ℓi,t∣|\ell_{i,t}| responds to both improvement and regression, so changes from signed cross-capability transfer also shift priority. The rollout shortfall ai,ta_{i,t} raises the priority of units whose past rollouts fell below an equal share, the under-allocation that the suppressive regime links to lower efficiency.
    • Opportunity protection lets FEAST re-measure the current policy's performance on different capabilities as the policy evolves. Positive sampling weights give a geometric waiting-time bound for eligible cluster selection (Proposition 4, Appendix A.2.3), and reserved budgets prevent a selected cluster's rollouts from being reallocated to higher-variance prompts elsewhere. This lets FEAST re-estimate the average reward, reward variation, and recent reward change of low-priority clusters under the current policy. Atomic-unit histories persist across regrouping, so shortfall and trend statistics carry over as capability clusters change.
AuthorsOpenReview ↗

To Reviewer 4zzn(Part II)

  • (Assumptions) Regarding the concern that the conclusion relies on assumptions that are poorly motivated and difficult to justify, QEL identifies the problem, and FEAST does not rely on these assumptions.

    • The assumptions of Theorem 1 and Proposition 2 only explain when allocation history lowers learning efficiency. FEAST's priorities are computed from observed pilot rewards and rollout counts, and its waiting-time bound (Proposition 4) needs only positive, bounded sampling weights, which the exploration floor keeps positive. FEAST's gains are established empirically on GURU-18K and SMC at both model scales (paper Tables 2–3) and in the ablations (Table R2.3).
  • (Empirical need for the theory) On GURU Math and GURU Zebra, matched exposure does not give matched performance, so history dependence appears in practice and shows a clear need for this theoretical treatment.

    • On GURU Math, matched Math exposure yields different Math scores under different allocation paths. We added a fixed Math quota experiment (Appendix A.7.2) in which the varied factor is the Math share of each update, 12.5% or 50%. Both runs start from the same model and see the same Math prompts in the same order, so the 12.5% run reaches the same Math exposure later, after more Code and Science training. The 12.5% run scores higher on MATH-500 (Table R2.1), so matched Math exposure does not determine the Math score, consistent with QEL. The direction matches positive transfer from Code and Science, which the signed transfer term in Theorem 1 allows.

    Table R2.1. Same Math exposure under two allocation paths. Both runs train Qwen3-1.7B-Base on the same 512 Math prompts in the same order and differ only in the Math share of each update. The 95% CI comes from a paired question-level bootstrap.

    Math share per updateMath promptsCode + Science promptsUpdatesMATH-500
    12.5%5123,5843261.20
    50%512512855.40
    Difference (12.5% − 50%), 95% CIN/AN/AN/A+5.80 [2.00, 9.80]
    • On GURU Zebra, equal exposure does not give equal performance, and the deficit from earlier under-allocation is hard to recover. We added a matched-exposure experiment (Appendix A.7.2) comparing Full FEAST with FEAST w/o reserved budgets mechanism, which keeps cluster selection but lets high-variance prompts in one cluster take rollouts from other selected clusters. Zebra receives fewer rollouts under FEAST w/o reserved budgets mechanism, so we give it one extra update until both models have seen the same number of Zebra rollouts. Both models then train on the same Zebra, Code, and Science prompts in the same order. After this matching, Full FEAST scores far higher (Table R2.2), so equal Zebra exposure does not give equal Zebra performance, which is the history dependence QEL describes. Further shared updates that spend half their rollouts on Zebra narrow the gap without closing it: after four of them, FEAST w/o reserved budgets mechanism still scores far below what Full FEAST had already reached at the matched count. The deficit from earlier under-allocation is therefore hard to recover: closing the gap under a fixed budget would require taking even more rollouts from other capabilities.

    Table R2.2. Zebra scores at equal Zebra exposure. After the counts are matched, both models train on the same prompts, half of them from Zebra. The 95% CI comes from a paired question-level bootstrap.

    StageCumulative Zebra rollouts per modelFull FEASTFEAST w/o reserved budgets mechanismGap, 95% CI
    After matching rollout counts3,36031.878.78+23.09 [20.82, 25.49]
    After two shared updates4,38432.6813.53+19.15 [16.92, 21.47]
    After four shared updates5,40833.5417.41+16.13 [14.11, 18.18]

[1] Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming. SIAM Journal on Optimization, 2013.

[2] Optimization Methods for Large-Scale Machine Learning. SIAM Review, 2018.

AuthorsOpenReview ↗

To Reviewer 4zzn(Part Ⅲ)

(W2) The ablations do not isolate the components most relevant to the claimed contribution. While this paper proposes a new framework considering history, learning signal, fairness, and efficiency, the ablation studies do not clearly show the significance of each design in the framework.

(Q2) Could you explain more about the Inter-Fairness ablation study? Now it only writes, "removes the capability structure and samples 128 prompts globally". I wonder if it keeps the sampling procedure the same as sampling from different atomic units in one cluster (as in eq 6), or if it also ignores the atomic unit structure. Also, could you explain the main necessity of using clusters, because it seems we can also apply the same Inter Fairness directly to atomic units.

Thank you for asking us to explain more about the Inter-Fairness ablation study and why we use clusters, which helped us define each ablation precisely and isolate the contribution of each design.

  • (Inter-Fairness ablation) The w/o Inter-Fairness ablation in paper Table 4 removes both atomic units and clusters, so it also ignores the atomic unit structure.

    • It samples prompts globally without replacement and applies progressive allocation in a global pool.
  • (Necessity of clusters and significance of each design) Every tested design contributes to FEAST's performance, which shows the significance of each design in the framework: removing clusters or any single mechanism lowers Avg.

    • We added ablations in which each variant removes or changes one component of Full FEAST and keeps the others unchanged (Table R2.3). The FEAST w/o inter-cluster fairness and FEAST w/o intra-cluster efficiency rows come from Table 4 of the original submission, the remaining variants are now in revised Table 4 (Section 5.2.4), and Appendix A.3 defines each variant. FEAST w/o capability clusters applies the same scheduler directly to atomic units, selecting four units with 32 prompts each. FEAST w/o reserved budgets mechanism keeps cluster selection and pilot rollouts, but in the later rounds high-variance prompts in one cluster can take rollouts from other selected clusters.
    • Applying the same Inter Fairness directly to atomic units (FEAST w/o capability clusters) lowers Avg. A possible reason is that each reserved budget then covers one unit's small prompt pool, leaving little room for intra-cluster efficiency. Clusters are therefore needed as the unit of budget sharing, while atomic units keep their own histories across regrouping.
    • FEAST w/o rollout shortfall (λa=0\lambda_a=0) lowers Avg., and so does FEAST w/o dynamic clustering, which keeps the initial clusters instead of regrouping by learning trends. This indicates that using allocation history in priorities and grouping improves allocation.
    • FEAST w/o exploration floor (ϵ=0\epsilon=0) and FEAST w/o reserved budgets mechanism both lower Avg., suggesting that protecting learning opportunities improves overall performance.
    • FEAST w/o intra-cluster efficiency, which gives every selected prompt eight responses in one round, lowers Avg., so concentrating rollouts on prompts with reward variation uses the budget more effectively.

    Table R2.3. Avg. after removing or changing one FEAST component. All runs use Qwen3-1.7B-Base, 100 updates, and 1,024 rollouts per update.

    Variant1.7B Avg.
    Full FEAST36.32
    FEAST w/o inter-cluster fairness: global sampling without atomic units or clusters35.66
    FEAST w/o intra-cluster efficiency: eight responses per prompt, no later rounds34.66
    FEAST w/o reserved budgets mechanism: high-variance prompts can take rollouts from other selected clusters32.56
    FEAST w/o capability clusters: four atomic units, 32 prompts each34.01
    FEAST w/o dynamic clustering: fixed initial clusters35.65
    FEAST w/o exploration floor (ϵ=0\epsilon=0)35.70
    FEAST w/o rollout shortfall (λa=0\lambda_a=0)35.94
    FEAST w/o learning trend (γ=0\gamma=0)35.85
AuthorsOpenReview ↗

To Reviewer 4zzn(Part Ⅳ)

(Q3) The rollout setting in section 4.3 is progressive increments [2,4,16]. Have you tried a larger rollout at the beginning, because looking at the variance of 2 rollouts may introduce large noise.

Thank you for asking whether we tried a larger rollout at the beginning, which helped us test how sensitive FEAST is to the initial pilot size.

  • (Larger rollout at the beginning) A four-response initial pilot performs similarly to Full FEAST at the same budget, so the noise from looking at the variance of 2 rollouts has little effect on performance.

    • We added a run with a four-response initial pilot (Appendix A.7.4), which uses [4,4,8] increments in place of [2,4,16], keeping 1,024 rollouts per update and FEAST's cluster allocation, so only the within-cluster schedule changes (Table R2.4).
    • The close Avg. suggests that two-response pilots already identify informative prompts for the later rounds.
    • Eight responses per prompt is the extreme case of a larger first round, with no later concentration. Its lower Avg. suggests that short pilots free budget for concentrating rollouts on prompts with reward variation.

    Table R2.4. Rollout schedules at the same budget. Each update starts with 128 prompts and generates 1,024 rollouts. A schedule [a,b,c] gives a rollouts to every prompt, then b more to the half with the highest reward variance in each cluster, then c more to the top quarter.

    MethodScheduleRollouts per updateAvg.
    Full FEAST[2,4,16]128×2 + 64×4 + 32×16 = 1,02436.32
    Four-response pilot[4,4,8]128×4 + 64×4 + 32×8 = 1,02436.26
    Eight responses per prompt, no later rounds (FEAST w/o intra-cluster efficiency)[8]128×8 = 1,02434.66

Thank you again for helping us clarify Section 3 and complete the ablations. We would be happy to discuss any remaining questions.

Reviewer_VfEhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses endogenous nonstationarity in on-policy RLVR (RL with verifiable rewards) training, where allocation decisions change the policy and thus future learning signals. The authors formalize this through the Quota-Efficiency Law (QEL) showing allocation history affects learning efficiency even at matched exposure. They introduce FEAST - Fair and Efficient Adaptive Sampling for Task-Free On-Policy Learnin that discovers capability clusters, protects learning opportunities across clusters with reserved budgets, and concentrates computation on informative prompts within each cluster. Theoretical results are complemented with an experimental evaluation aimed to demonstrate QEL at matched task exposure, FEAST’s performance, and computational cost.

Critical strengths

The Quota-Efficiency Law provides novel formal characterization of how allocation history affects learning efficiency at matched capability-specific exposure. This is a contribution towards understanding on-policy learning dynamics where prior work treated utility as intrinsic rather than path-dependent. The decomposition showing cross-capability transfer terms depend on evolving policy explains why greedy allocation can reinforce neglect of weak capabilities. This theoretical foundation justifies the fairness mechanism. Thus the authors provide principled justification for protecting under-allocated capabilities and explain the feedback loop that causes utility-driven methods to fail.

Critical weaknesses

Empirical results do not looks fully convincing. See questions for authors.

Questions for authors
  1. Could you comment on statistical significance (and if statistically significant - how substantial/marginal the differences are) for the FEAST vs Random comparisons in Table 2? The SD values suggest there might be no real improvements.
  2. Why does FEAST underperform Random on Code at 8B (79.69 vs 80.81)? Or is this difference negligible? Under what conditions do fairness mechanisms hurt performance on strong capabilities?
  3. The QEL experiments use simple environments - have you validated the theory on more complex reasoning tasks similar to GURU-18K?
3 comments
AuthorsOpenReview ↗

To Reviewer VfEh (Part I)

We sincerely thank you for recognizing QEL's formal account of how allocation history affects learning efficiency. Your questions helped us quantify FEAST's gains, explain the Code trade-off, and test QEL on GURU reasoning tasks. We would be glad to discuss any remaining concerns.

(W1) Empirical results do not looks fully convincing. See questions for authors.

(Q1) Could you comment on statistical significance (and if statistically significant - how substantial/marginal the differences are) for the FEAST vs Random comparisons in Table 2? The SD values suggest there might be no real improvements.

Thank you for asking about statistical significance and how substantial/marginal the differences are, which helped us clarify both the reliability and the size of FEAST's gains.

  • (Statistical significance) For the FEAST vs Random comparisons in Table 2, the differences reach statistical significance at both model scales, with question-level 95% CIs above zero.

    • Regarding the concern that the SD values suggest there might be no real improvements, the SDs in Table 2 describe each method's variation across training seeds. Significance is assessed on the paired difference, which Table R1.1 reports with its interval. FEAST also has a smaller seed SD at both scales, suggesting more stable performance across seeds. We would like to respectfully clarify that the per-seed results in Tables 7–8 of the original submission already show FEAST above Random for every training seed at both scales.

    Table R1.1. FEAST versus Random on GURU-18K (Appendix A.7.1). The 95% CI of the paired difference comes from a question-level bootstrap.

    ModelRandomFEASTFEAST − Random95% CI
    1.7B35.43 ± 0.1836.18 ± 0.13+0.75[0.20, 1.31]
    8B50.65 ± 1.0151.69 ± 0.08+1.04[0.30, 1.79]
  • (Magnitude of the gains) On how substantial/marginal the differences are, FEAST achieves substantial gains on individual benchmarks, while the overall difference reflects equal one-ninth weighting across the nine benchmarks.

    • Avg. weights the nine benchmarks equally, so a gain on one benchmark contributes one ninth of its size to Avg.
    • Large gains on individual benchmarks, such as FinQA at 1.7B and CodeIO at 8B, each add less than one point to Avg. (Table R1.2, computed from the per-seed scores in paper Tables 7–8).

    Table R1.2. Benchmarks with large FEAST gains (Appendix A.7.1). Gain is the FEAST − Random difference averaged over training seeds. Each benchmark contributes one ninth of its gain to Avg.

    ModelBenchmarkGainContribution to Avg.
    1.7BFinQA+7.08+0.79
    8BCodeIO+4.56+0.51
    8BMultiHierTT+3.61+0.40
AuthorsOpenReview ↗

To Reviewer VfEh (Part II)

(Q2) Why does FEAST underperform Random on Code at 8B (79.69 vs 80.81)? Or is this difference negligible? Under what conditions do fairness mechanisms hurt performance on strong capabilities?

Thank you for asking why FEAST underperforms Random on Code and when fairness mechanisms hurt performance on strong capabilities, which helped us locate the source of the Code decrease and clarify the trade-off that fairness involves.

  • (Code decrease at 8B) On why does FEAST underperform Random on Code at 8B, the decrease comes mainly from HumanEval, and reduced exposure to its prompt format is a possible explanation.

    • MBPP remains nearly unchanged. The domain-level allocation in paper Figure 6 (Appendix A.4) shows a lower overall Code rollout share under FEAST, which may reduce training exposure to the prompt format of HumanEval.
    • Even with this decrease, FEAST's largest gap to the best scheduler on any benchmark at 8B is the smallest among the six schedulers (Table R1.3).

    Table R1.3. Largest gap to the best scheduler. For each method, we find the benchmark where it trails the best of the six GRPO schedulers the most and report that gap. A smaller gap means the method does not fall far behind on any of the nine benchmarks (paper Tables 7–8).

    ModelRandomTACADSDUMPBOTSFEAST
    1.7B7.0812.1932.798.9329.663.73
    8B4.563.7322.355.069.532.15
  • (When fairness can hurt strong capabilities) As for under what conditions do fairness mechanisms hurt performance on strong capabilities, this can happen when redistributing a fixed budget reduces training exposure to task-specific formats that a strong capability relies on.

    • From a Pareto perspective, allocating a fixed rollout budget across capabilities can produce trade-offs between their scores. At 8B, gains on Logic, Simulation, and Tabular outweigh the HumanEval decrease and raise Avg., the cross-domain trade-off that Section 5.2.2 of the paper notes.
AuthorsOpenReview ↗

To Reviewer VfEh (Part Ⅲ)

(Q3) The QEL experiments use simple environments - have you validated the theory on more complex reasoning tasks similar to GURU-18K?

Thank you for asking whether we validated the theory on more complex reasoning tasks similar to GURU-18K, which helped us extend the QEL evidence from controlled environments (Section 5.1 and Appendix A.5) to GURU reasoning data with two new experiments.

  • (QEL on GURU reasoning tasks) QEL also holds on more complex reasoning tasks similar to GURU-18K: on both GURU Math and GURU Zebra, matched exposure to a capability does not give matched performance.

    • On GURU Math, matched Math exposure yields different Math scores under different allocation paths. We added a fixed Math quota experiment (Appendix A.7.2) in which the varied factor is the Math share of each update, 12.5% or 50%. Both runs start from the same model and see the same Math prompts in the same order, so the 12.5% run reaches the same Math exposure later, after more Code and Science training. The 12.5% run scores higher on MATH-500 (Table R1.4), so matched Math exposure does not determine the Math score, consistent with QEL. The direction matches positive transfer from Code and Science, which the signed transfer term in Theorem 1 allows.
    • On GURU Zebra, equal exposure does not give equal performance, and the deficit from earlier under-allocation is hard to recover. We added a matched-exposure experiment (Appendix A.7.2) comparing Full FEAST with FEAST w/o reserved budgets mechanism, which keeps cluster selection but lets high-variance prompts in one cluster take rollouts from other selected clusters. Zebra receives fewer rollouts under FEAST w/o reserved budgets mechanism, so we give it one extra update until both models have seen the same number of Zebra rollouts. Both models then train on the same Zebra, Code, and Science prompts in the same order. After this matching, Full FEAST scores far higher (Table R1.5), so equal Zebra exposure does not give equal Zebra performance, which is the history dependence QEL describes. Further shared updates that spend half their rollouts on Zebra narrow the gap without closing it: after four of them, FEAST w/o reserved budgets mechanism still scores far below what Full FEAST had already reached at the matched count. The deficit from earlier under-allocation is therefore hard to recover: closing the gap under a fixed budget would require taking even more rollouts from other capabilities.

    Table R1.4. Same Math exposure under two allocation paths. Both runs train Qwen3-1.7B-Base on the same 512 Math prompts in the same order and differ only in the Math share of each update. The 95% CI comes from a paired question-level bootstrap.

    Math share per updateMath promptsCode + Science promptsUpdatesMATH-500
    12.5%5123,5843261.20
    50%512512855.40
    Difference (12.5% − 50%), 95% CIN/AN/AN/A+5.80 [2.00, 9.80]

    Table R1.5. Zebra scores at equal Zebra exposure. After the counts are matched, both models train on the same prompts, half of them from Zebra. The 95% CI comes from a paired question-level bootstrap.

    StageCumulative Zebra rollouts per modelFull FEASTFEAST w/o reserved budgets mechanismGap, 95% CI
    After matching rollout counts3,36031.878.78+23.09 [20.82, 25.49]
    After two shared updates4,38432.6813.53+19.15 [16.92, 21.47]
    After four shared updates5,40833.5417.41+16.13 [14.11, 18.18]

Thank you again for helping us make the empirical evidence more precise. We would welcome any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.