- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes FEAST, a task-free sampling framework for multi-capability on-policy reinforcement learning. It introduces the Quota-Efficiency Law (QEL) to analyze how allocation history can affect learning gains even at matched capability-specific exposure. FEAST maintains persistent atomic-unit histories, dynamically groups these units into capability clusters, reserves rollout budgets for selected clusters, and progressively allocates rollouts within each cluster based on reward variance. Experiments on GURU-18K and SMC with Qwen3-1.7B and 8B report higher average performance than the compared GRPO sampling baselines under matched rollout budgets, with ablations examining the two allocation levels.
Critical strengths
- The paper addresses a practical problem in multi-capability reinforcement learning: weaker capabilities may be repeatedly overlooked during training. Combining opportunity protection across capability clusters with selective rollout allocation within each cluster is a sensible approach.
- The evaluation covers two datasets, two model sizes, and several sampling baselines. FEAST improves average performance under a fixed rollout budget, and the ablations provide some evidence for the benefits of both allocation levels.
Critical weaknesses
- The presentation needs improvement. The abstract and introduction do not clearly convey the main contribution or the reasoning behind the design, and the main text makes the full training procedure difficult to follow. The figures offer limited help in understanding the method.
- The connection between QEL and FEAST is not sufficiently established. The result that allocation history reduces learning efficiency relies on strong assumptions whose applicability is not examined in the experiments. FEAST’s guarantee mainly establishes a positive probability of cluster selection; it does not show that the method meaningfully mitigates the effects described by QEL.
Questions for authors
- How is the GRPO loss normalized for response groups containing 2, 6, and 22 rollouts? Does allocating more rollouts also increase a prompt’s weight in the policy update?
- Could the authors walk through one FEAST training iteration with a concrete example and clarify the relationships among atomic units, dynamic capability clusters, budget allocation, and feedback updates in Figure 3? How do the modules in the figure correspond to Equations (5)–(8)?
4 comments
To Reviewer 6fws(Part II)
(W1) The presentation needs improvement. The abstract and introduction do not clearly convey the main contribution or the reasoning behind the design, and the main text makes the full training procedure difficult to follow. The figures offer limited help in understanding the method.
(Q2) Could the authors walk through one FEAST training iteration with a concrete example and clarify the relationships among atomic units, dynamic capability clusters, budget allocation, and feedback updates in Figure 3? How do the modules in the figure correspond to Equations (5)–(8)?
Thank you for asking us to
walk through one FEAST training iteration with a concrete example, which helped us add a step-by-step walkthrough to Appendix A.2.6 and map each module in Figure 3 to its equation.
(Walkthrough and equation mapping) As for
how do the modules in the figure correspond to Equations (5)–(8)(Equations (3)–(6) in the revised paper), each module in Figure 3 maps to one step of a FEAST update (Table R4.1).
- The policy stays fixed during generation and is then updated by GRPO.
- Table R4.1 is now Table 5 in Appendix A.2.6, which combines Algorithm 1 and the rollout counts of Appendix A.2.4 into one walkthrough.
Table R4.1. One FEAST update in the main configuration (64 atomic units, four selected clusters, 1,024 rollouts), with the matching parts of the paper.
Step What FEAST does Paper reference Retain histories Keep 64 persistent atomic units and their states Section 4.1 Regroup Every 10 updates, regroup units by current-policy representations and smoothed learning trends Equation (3) Compute priorities Combine the exploration floor, rollout shortfall, and learning evidence (reward variation , average reward , and its recent change ) Equation (4) Select and reserve Select four clusters without replacement, reserving 256 rollouts each Equations (2), (5) Allocate progressively In each cluster, 2 rollouts for 32 prompts, 4 more for the top 16, and 16 more for the top 8 (256 rollouts) Equation (6) Update the policy Train on 128 prompt groups: 64 with 2 responses, 32 with 6, and 32 with 22 (1,024 responses) Equation (28), Algorithm 1 Update feedback Update unit histories. For example, pilot rewards 0 and 1 give an average reward of 0.5 and a squared reward difference of 1 Figure 3, Appendix A.2 (Relationships among modules) In terms of
the relationships among atomic units, dynamic capability clusters, budget allocation, and feedback updates, atomic units keep histories, clusters group units for selection, the reserved budgets mechanism fixes each selected cluster's budget, and feedback refreshes the unit histories.
- As specified in Appendix A.2, average reward and reward variation are computed from the common initial pilot with prompts weighted equally. Trends are updated from rollouts with reward variation, and shortfalls from all generated rollouts.
(Presentation) The revised introduction now states the main contribution and
the reasoning behind the design, and the Figure 3 caption maps each module to its equation.
- The introduction now follows the path from how past allocation shapes current rewards, to QEL, and then to FEAST's separation of opportunity and compute allocation. The walkthrough in Table R4.1 and the loss definition above have been added to Appendix A.2.6 (Table 5 and Equation (28)) and are referenced from Section 4.3.
To Reviewer 6fws(Part Ⅲ)
(W2) The connection between QEL and FEAST is not sufficiently established. The result that allocation history reduces learning efficiency relies on strong assumptions whose applicability is not examined in the experiments. FEAST’s guarantee mainly establishes a positive probability of cluster selection; it does not show that the method meaningfully mitigates the effects described by QEL.
Thank you for noting that
the connection between QEL and FEAST is not sufficiently established, which helped us make this connection explicit and test it on GURU reasoning tasks.
(Connection between QEL and FEAST) The link is direct: QEL shows that allocation history changes learning efficiency at matched exposure, so FEAST adapts allocation to history-dependent feedback.
- History dependence makes current learnability of a certain capability partly a product of past allocation. In the decomposition of Theorem 1 (Appendix A.1.1), direct learning and cross-capability transfer are both evaluated at the policy produced by earlier updates. Low average reward () or little reward variation () on an under-allocated capability can therefore reflect its allocation path. A scheduler that excludes capabilities by these current values alone can reinforce that path (Figure 2), as SEC-Greedy in paper Figure 5 illustrates.
- FEAST's priority combines current learning evidence with allocation history. In the learning-evidence surplus of Equation (4), favors units whose prompts the current policy solves only sometimes, and the learning trend responds to both improvement and regression, so changes from signed cross-capability transfer also shift priority. The rollout shortfall raises the priority of units whose past rollouts fell below an equal share, the under-allocation that the suppressive regime links to lower efficiency.
- Opportunity protection lets FEAST re-measure the current policy's performance on different capabilities as the policy evolves. Positive sampling weights give a geometric waiting-time bound for eligible cluster selection (Proposition 4, Appendix A.2.3), and reserved budgets prevent a selected cluster's rollouts from being reallocated to higher-variance prompts elsewhere. This lets FEAST re-estimate the average reward, reward variation, and recent reward change of low-priority clusters under the current policy. Atomic-unit histories persist across regrouping, so shortfall and trend statistics carry over as capability clusters change.
(Assumptions and their applicability) Regarding the
strong assumptions whose applicability is not examined in the experiments, QEL identifies the problem, and FEAST does not rely on these assumptions. We also add two experiments to prove that QEL holds on reasoning tasks.
- The assumptions of Theorem 1 and Proposition 2 only explain when allocation history lowers learning efficiency. FEAST's priorities are computed from observed pilot rewards and rollout counts, and its waiting-time bound (Proposition 4) needs only positive, bounded sampling weights, which the exploration floor keeps positive. FEAST's gains are established empirically on GURU-18K and SMC at both model scales (paper Tables 2–3) and in the ablations (Table R4.5).
- On GURU Math, matched Math exposure yields different Math scores under different allocation paths. We added a fixed Math quota experiment (Appendix A.7.2) in which the varied factor is the Math share of each update, 12.5% or 50%. Both runs start from the same model and see the same Math prompts in the same order, so the 12.5% run reaches the same Math exposure later, after more Code and Science training. The 12.5% run scores higher on MATH-500 (Table R4.2), so matched Math exposure does not determine the Math score, consistent with QEL. The direction matches positive transfer from Code and Science, which the signed transfer term in Theorem 1 allows.
Table R4.2. Same Math exposure under two allocation paths. Both runs train Qwen3-1.7B-Base on the same 512 Math prompts in the same order and differ only in the Math share of each update. The 95% CI comes from a paired question-level bootstrap.
Math share per update Math prompts Code + Science prompts Updates MATH-500 12.5% 512 3,584 32 61.20 50% 512 512 8 55.40 Difference (12.5% − 50%), 95% CI N/A N/A N/A +5.80 [2.00, 9.80]
To Reviewer 6fws(Part Ⅳ)
(Beyond positive selection probability) Beyond
a positive probability of cluster selection, FEAST bounds the expected waiting time, the exploration floor shortens the longest waits in training, and the reserved budgets make sure each cluster's capability is sufficiently trained.
- Proposition 4 (Appendix A.2.3) bounds the expected waiting time and its tail probability for eligible cluster selection. After selection, reserved budgets prevent its rollouts from being reallocated to other clusters. Within the cluster, each prompt slot goes to an atomic unit with probability proportional to its priority, which includes the rollout shortfall (Appendix A.2.2).
- The exploration floor shortens the longest waits while every atomic unit stays covered. We added an analysis of atomic-unit service (Appendix A.7.5), comparing Full FEAST with FEAST w/o exploration floor () on the same 64 atomic units. Both runs serve every atomic unit. Compared with FEAST w/o exploration floor, FEAST substantially reduces the P90 and maximum waits (Table R4.3), indicating that the floor protects the units that would otherwise wait longest.
Table R4.3. Atomic-unit coverage and waiting times over 100 updates on the same 64 units. Service means receiving at least one prompt. Waiting statistics summarize each unit's longest unserved interval.
Metric Full FEAST FEAST w/o exploration floor () Units served at least once 64/64 64/64 Longest wait, full training: median / P90 / max (updates) 10 / 20.7 / 59 8 / 52.5 / 73 (Mitigation of QEL effects) With the reserved budgets mechanism, FEAST keeps a Zebra advantage at equal Zebra exposure and after further shared training, indicating that the mechanism mitigates
the effects described by QEL.
- We added a matched-exposure experiment on GURU Zebra (Appendix A.7.2) comparing Full FEAST with FEAST w/o reserved budgets mechanism, which keeps cluster selection but lets high-variance prompts in one cluster take rollouts from other selected clusters. Zebra receives fewer rollouts under FEAST w/o reserved budgets mechanism, so we give it one extra update until both models have seen the same number of Zebra rollouts. Both models then train on the same Zebra, Code, and Science prompts in the same order.
- After this matching, Full FEAST scores far higher than FEAST w/o reserved budgets mechanism (Table R4.4). Equal Zebra exposure therefore does not give equal Zebra performance, which is the history dependence QEL describes. Further shared updates that spend half their rollouts on Zebra narrow the gap without closing it: after four of them, FEAST w/o reserved budgets mechanism still scores far below what Full FEAST had already reached at the matched count. The deficit from earlier under-allocation is therefore hard to recover: closing the gap under a fixed budget would require taking even more rollouts from other capabilities.
Table R4.4. Zebra scores at equal Zebra exposure. After the counts are matched, both models train on the same prompts, half of them from Zebra. The 95% CI comes from a paired question-level bootstrap.
Stage Cumulative Zebra rollouts per model Full FEAST FEAST w/o reserved budgets mechanism Gap, 95% CI After matching rollout counts 3,360 31.87 8.78 +23.09 [20.82, 25.49] After two shared updates 4,384 32.68 13.53 +19.15 [16.92, 21.47] After four shared updates 5,408 33.54 17.41 +16.13 [14.11, 18.18]
- FEAST w/o reserved budgets mechanism also has a lower Avg. than Full FEAST (Table R4.5), suggesting that the protection benefits overall performance.
Table R4.5. Avg. after removing or changing one FEAST component. All runs use Qwen3-1.7B-Base, 100 updates, and 1,024 rollouts per update.
Variant 1.7B Avg. Full FEAST 36.32 FEAST w/o inter-cluster fairness: global sampling without atomic units or clusters 35.66 FEAST w/o intra-cluster efficiency: eight responses per prompt, no later rounds 34.66 FEAST w/o reserved budgets mechanism: high-variance prompts can take rollouts from other selected clusters 32.56 FEAST w/o capability clusters: four atomic units, 32 prompts each 34.01 FEAST w/o dynamic clustering: fixed initial clusters 35.65 FEAST w/o exploration floor () 35.70 FEAST w/o rollout shortfall () 35.94 FEAST w/o learning trend () 35.85
Thank you again for helping us clarify the training procedure and the link between QEL and FEAST. We would be happy to discuss any remaining questions.
To Reviewer 6fws(Part I)
We sincerely thank you for recognizing the practical problem FEAST addresses and the breadth of its evaluation. Your questions helped us explain the GRPO weighting, walk through one update, and make the connection between QEL and FEAST explicit. We would be glad to discuss any remaining concerns.
(Q1) How is the GRPO loss normalized for response groups containing 2, 6, and 22 rollouts? Does allocating more rollouts also increase a prompt’s weight in the policy update?