acceptodds
Under review as a conference paper at ICLR 2027

When Is Reuse Worth Replacing Exploration? Quality-Constrained Rollout Allocation for Stateful Agentic Reinforcement Learning

Abstract

Agentic reinforcement learning improves language models through multi-step environment interaction, but costly rollouts can yield little reward contrast, while restarting trajectories repeatedly discards reusable progress. Efficient training therefore requires deciding how much experience to acquire and when to generate, branch, or replay it. We propose QCAR, a quality-constrained rollout allocation framework for stateful agentic reinforcement learning. QCAR organizes observed task progress and explored transitions into a state graph, using initial Probe rollouts to estimate continuation success and identify valuable Fork boundaries. These estimates guide allocation across Fresh Generation, Fork, and Replay, with quality feedback limiting unproductive reuse. Balanced policy updates account for unequal sampling and shared prefixes, while observed outcomes recalibrate the predictor as the policy evolves. Across two models and four benchmarks, QCAR improves final accuracy over GRPO in all eight settings, averaging percentage points (% relative). QCAR yields up to % more reward-contrasting groups, with gains in all six model–environment comparisons, while fresh-token consumption decreases in five, by up to %. These results demonstrate the value of graph-informed, quality-constrained experience acquisition for stateful agentic reinforcement learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Reinforcement Learning, Adaptive Rollout Allocation, Graph-Guided Prefix Exploration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GXXeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces QCAR, a quality-constrained framework for allocating rollouts in stateful agentic reinforcement learning. Using initial probe trajectories and graph-conditioned predictions, QCAR distributes the training budget among fresh trajectory generation, prefix branching, and replay. Quality constraints aim to prevent ineffective reuse, while the policy update procedure accounts for differences in group size and shared trajectory prefixes. Experiments exhibits an average accuracy gain of 2.32% over GRPO and a reduction in fresh-token usage of up to 8.35%.

Critical strengths
  1. Efficient exploration and rollout allocation is an important problem to be studied in agentic RL, where expensive interactions do not always yield useful learning signals. Improving the use of collected trajectories therefore has clear practical value.

  2. The experiments on outcome prediction, stalled continuations, and reward contrast help motivate the proposed approach. They provide empirical support for considering graph-derived information when allocating rollout resources.

Critical weaknesses
  1. (critical) AUROC on seen tasks provides only partial evidence of the predictor’s usefulness. It does not guarantee whether predictions remain reliable on unfamiliar task structures or under the changing policy distribution encountered during training.

  2. QCAR relies on predicted probabilities to allocate rollouts, yet the paper does not evaluate their calibration. This matters because inaccurate estimates may distort resource allocation. In particular, states whose potential is underestimated may receive less exploration, reducing opportunities to correct those estimates and potentially reinforcing the initial errors.

  3. (critical) Prior work, including [1], has already investigated how to estimate the utility of exploration. Although QCAR addresses rollout allocation during training, the paper does not clearly clarify which aspects of its formulation or mechanism constitute a substantive advance over related exploration-aware approaches.

  4. The evaluation includes GRPO but ignores relevant agentic RL methods, such as GiGPO [2], as well as alternative rollout-allocation and trajectory-reuse strategies. These comparisons are needed to examine the benefit of QCAR’s proposed allocation mechanism.

[1] Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization [2] Group-in-Group Policy Optimization for LLM Agent Training

Questions for authors

How does the proposed value estimator perform on unseen task structures, and how stable are its predictions as the policy distribution changes during training? Can the authors provide direct evidence of its reliability in online allocation decisions?

2 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/2)

Online prediction on new instructions and changing policies

Thank you for highlighting predictor reliability under instruction novelty and policy evolution. We added an audit of the actual online neural allocator, using probabilities recorded before expansion and before fitting the current update’s outcomes.

The new-instruction cohort contains 736 distinct instructions absent from all earlier predictor updates and 1,482 bare-root outcomes. Within-policy-update AUROC is 0.827 on this cohort and 0.802 across all post-initialization roots, with a pooled AUROC of 0.805. These results support task discrimination on newly encountered instructions and under the policy making the allocation decision.

The complete post-initialization root cohort contains 3,168 outcomes from 1,584 previously unselected game files. Instruction novelty is determined against the predictor’s earlier online fitting history, and all audited predictions precede their corresponding outcomes.

We also evaluate forward transfer in time: a monotone affine-logit calibrator is fitted on a preceding policy window, frozen, and evaluated on the following window. Across four future windows on first-instruction encounters, ECE improves from 0.06801 to 0.04847, and Brier score improves from 0.12580 to 0.12368. The forward test cohort contains 994 outcomes from 493 instructions. This evaluates probability adaptation on subsequent allocation decisions as the policy changes.

Calibration and protection against under-exploration

We expanded probability evaluation to include Brier score, ECE, a recent-success-rate baseline, and forward temporal testing. The recent-rate baseline measures global policy progress, while within-update AUROC evaluates task-specific ranking.

Fresh exploration is maintained independently of the predicted value. Every candidate task receives at least four fresh trajectories, including two reward-blind probes. Tasks with low predicted success therefore continue to supply new observations for policy learning and predictor recalibration. Allocation combines predicted utility with uncertainty, coverage, and an exploration bonus.

For prefixes, we record candidate probabilities, selected anchors, restoration outcomes, and exact lineage. These records connect predicted opportunities to executed continuations and their observed rewards.

Executed Fork decisions and graph utility

We added complete pre-selection and execution-trace auditing. All 160 selected Fork requests match the actual restored parent/boundary lineage. These executions produce 35 parent–Fork reward contrasts and convert 14 otherwise homogeneous task groups into mixed groups, relative to the same batches’ fresh trajectories. These measurements verify both faithful execution of the selected continuations and their contribution to additional reward-contrasting training groups.

We further isolate the graph utility multiplier while retaining text prediction, auxiliary-head fitting, credit, and actor budget:

Graph utilityTraining dataMean validation successesMixed groups
Learned multiplierALFWorld25.2023/96
Unit multiplierALFWorld24.1021/96

Training tasks are drawn from the 3,553-task ALFWorld pool, and evaluation uses 134 unseen tasks. Means average the same scheduled evaluation checkpoints. The learned graph utility multiplier improves both mean validation performance and mixed-group yield, supporting its contribution to informative acquisition and subsequent policy learning.

We will explain the implementation roles explicitly: the text model supplies continuation-success probabilities, the graph head modulates acquisition utility, and exact token lineage verifies prefix reuse and determines suffix credit.

AuthorsOpenReview ↗

Rebuttal (Part 2/2)

Controlled acquisition and policy-learning results

We added acquisition baselines and an acquisition-by-credit factorial experiment to evaluate how stateful acquisition contributes to policy learning.

Under the same wall-clock budget, QCAR achieves the highest validation accuracy among the compared methods.

MethodTraining dataSuccesses within shared wall time
QCARALFWorld109/134
GRPOALFWorld103/134
Fixed Root/ForkALFWorld90/134
Dynamic mixed-group samplingALFWorld65/134

Each run receives 12,537 seconds, approximately 3.482 hours, on eight reserved H800 GPUs. Scores are the latest completed evaluations within that allocation on the same 134 held-out tasks. Inclusive time covers initialization, generation, environment interaction/restoration, predictor fitting, Replay rescoring, optimization, and evaluation.

QCAR exceeds Fixed Root/Fork by 19 successful tasks, supporting the complete adaptive acquisition-and-learning scheme over fixed branching in this setting.

Fully mixed batches require substantial additional generation. During matched-update experiments, dynamic sampling and QCAR achieve similar mean validation successes: 38.40/134 versus 38.00/134. Dynamic sampling consumes 3.52× the fresh generated tokens and 2.17× the inclusive time. It generates 4,736 trajectories, retains 1,280, and discards 3,456 to obtain 160/160 mixed training groups. QCAR obtains 81/160 mixed groups with substantially less generation. Mean validation successes are averaged across shared scheduled evaluation checkpoints.

We also added fixed-pool filtering, which generates 256 candidates and retains 128 per actor update. QCAR produces 48/96 mixed groups versus 40/96, a 20% increase, while mean validation successes remain close at 23.00/134 versus 22.00/134. Filtering generates 1,536 trajectories and discards half.

Acquisition and policy credit work together. The factorial control crosses fixed/adaptive Root-Fork acquisition with ordinary/balanced+sibling credit:

AcquisitionCreditMean validation successesMixed groups
Fixed Root/ForkOrdinary33.0045/160
Fixed Root/ForkBalanced + sibling32.6044/160
Adaptive Root/ForkOrdinary32.4047/160
Adaptive Root/ForkBalanced + sibling34.4047/160

All arms use the same Qwen3-8B initialization, actor-training budget, KL coefficient, and policy-GPU topology. Under fixed acquisition, the credit schemes yield similar validation performance. The combination of adaptive acquisition and balanced+sibling credit achieves the highest mean. Within adaptive acquisition, balanced+sibling credit improves the mean from 32.40/134 to 34.40/134, while both configurations produce 47/160 mixed groups, indicating a benefit in how acquired contrast is used during optimization.

Replay also contributes within QCAR.

SettingMean validation successes (/134)Endpoint successes (/134)Mixed groups
QCAR w/o Replay102.80124175/1600
QCAR w/ Replay106.50127191/1600

Replay occupies part of the fixed actor budget. Its inclusion improves mean and endpoint validation performance while increasing mixed groups by 9.1%.

Contribution relative to exploration-aware optimization

We will make the connection to Learning to Explore/EAPO explicit. EAPO uses information-value rewards and exploration/completion-aware optimization. QCAR contributes a training-time experience-acquisition layer that allocates a retained budget among fresh generation, verified-prefix Fork, and bounded-age Replay, with quality feedback and provenance-aware credit.

This layer coordinates how much experience to acquire, where to continue, when to reuse, and how reused tokens contribute to the update. The observable graph organizes acquisition opportunities; exact lineage enables verified restoration and suffix credit.

We will organize related work by mechanism: GiGPO for policy credit, Knapsack RL for rollout budgets, and InfoTree/Prefix Sampling for tree or prefix exploration. The online prediction audit, executed-Fork accounting, acquisition controls, credit factorial, and Replay ablation jointly evaluate these aspects of quality-constrained, stateful experience acquisition.

Reviewer_jp3UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes QCAR, a quality-constrained rollout allocation framework for stateful agentic reinforcement learning. The main idea is to improve training efficiency by adaptively allocating experience among Fresh Generation, Fork, and Replay, rather than repeatedly sampling independent trajectories. QCAR constructs state graphs from agent-environment interactions and uses Probe rollouts to estimate continuation success and identify promising branching states. It further introduces quality-aware allocation, Balanced-TreeRPO to handle unequal group sizes and shared prefixes, and online predictor recalibration. Experiments are conducted using Qwen3.5-4B and Qwen3-8B on ALFWorld, WebShop, Natural Questions, and HotpotQA. QCAR consistently outperforms GRPO across all eight settings, with an average accuracy improvement of 2.32 percentage points. It also increases reward-contrasting groups by up to 53.5% and reduces fresh-token consumption in five of six model-environment settings.

Critical strengths

S1. The paper addresses an important challenge in agentic RL: allocating expensive rollout budgets while preserving informative training signals. I find the motivation convincing, particularly the observation that independent rollouts can repeatedly explore unproductive states or discard useful intermediate progress. Incorporating graph-based progress and exploration coverage into rollout allocation is an interesting direction, and distinguishing observable state graphs from exact trajectory lineage is a reasonable design choice for stateful environments.

S2. QCAR improves accuracy over GRPO across both evaluated models and all four benchmarks. The improvements in reward-contrasting groups also provide useful evidence supporting the motivation.

Critical weaknesses

W1. Although the paper discusses related approaches such as Knapsack RL, InfoTree, and Prefix Sampling, the main experimental comparisons are limited to GRPO and PCL. These methods do not fully represent existing work on adaptive rollout allocation and stateful prefix exploration. Direct comparisons with more relevant methods would be important for establishing the advantage of QCAR. Moreover, QCAR introduces several changes simultaneously, including graph-guided allocation, Fork/Replay reuse, and Balanced-TreeRPO. Table 3 shows that removing the balanced update reduces WebShop accuracy from 42.80% to 35.70%, which is a substantial difference. This makes it difficult to determine how much of the improvement over GRPO is attributable specifically to the proposed rollout allocation strategy rather than changes to policy optimization. A more controlled comparison, such as GRPO with the balanced update but without adaptive allocation, would help clarify the contribution.

W2. I also have concerns about the efficiency evaluation. Table 2 reports reductions in fresh-token consumption of at most 8.35%, with one setting consuming 4.17% more tokens than GRPO. However, this accounting excludes environment execution, state restoration, predictor fitting, Replay rescoring, and policy optimization. Since QCAR introduces additional computation and bookkeeping, fresh-token savings alone may not establish an end-to-end efficiency improvement. Reporting wall-clock training time or total computational cost would make the efficiency claims more convincing.

Questions for authors

Q1. Could the authors provide comparisons with more directly related rollout allocation or prefix exploration methods, such as Knapsack RL, InfoTree, or Prefix Sampling, under matched computational budgets? Additionally, would it be possible to evaluate GRPO with Balanced-TreeRPO but without graph-guided allocation, to better isolate the improvement attributable to the allocation strategy?

  1. I would appreciate further clarification regarding the Replay mechanism. Section 3.3 states that replayed trajectories are rescored under the current policy before optimization. However, since these trajectories were originally sampled from older policies, recomputing their log-probabilities does not necessarily eliminate the distribution mismatch. How sensitive is performance to Replay staleness, and how is the resulting off-policy bias controlled? A no-Replay ablation or sensitivity analysis would help assess this concern.
2 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/2)

Separating acquisition from policy credit

We followed your suggestion and added a factorial control crossing fixed/adaptive Root-Fork acquisition with ordinary/balanced+sibling credit. All four arms use the same Qwen3-8B initialization, actor-training budget, KL coefficient, and four-policy-GPU topology.

Balanced+sibling credit has little effect under fixed acquisition and improves validation performance under QCAR’s adaptive acquisition. With fixed Root/Fork sampling, mean validation successes remain close: 33.00/134 with ordinary credit and 32.60/134 with balanced+sibling credit, with 45/160 and 44/160 mixed groups, respectively. With adaptive acquisition, balanced+sibling credit raises mean validation successes from 32.40/134 to 34.40/134, while both arms produce 47/160 mixed groups. The corresponding adaptive-acquisition endpoint scores are 53/134 and 58/134.

AcquisitionTraining dataCreditMean validation successes (/134)Mixed groups
Fixed Root/ForkALFWorldOrdinary33.0045/160
Fixed Root/ForkALFWorldBalanced + sibling32.6044/160
Adaptive Root/ForkALFWorldOrdinary32.4047/160
Adaptive Root/ForkALFWorldBalanced + sibling34.4047/160

Training tasks are drawn from the 3,553-task ALFWorld pool, and evaluation uses 134 unseen tasks. Means average the same scheduled evaluation checkpoints. Each arm trains on 16 tasks and 128 trajectories per update with KL coefficient 0.001. This Root/Fork factorial control is separate from the original WebShop ablation in Table 3.

These results support coordinating acquisition and policy credit. Under fixed acquisition, the two credit schemes yield similar validation performance and mixed-group counts. Under adaptive acquisition, balanced+sibling credit improves validation performance with the same number of mixed groups, indicating that its benefit extends beyond acquiring reward contrast to how that contrast is used during optimization.

Fixed acquisition uses uniform quotas and uniformly selected eligible Fork anchors. Ordinary credit retains native GRPO normalization; balanced+sibling credit assigns task/type mass and adds sibling comparison on divergent suffixes. These weights affect the policy surrogate, while all four cells use the same native row-mean KL.

End-to-end resource efficiency

We expanded efficiency reporting from fresh tokens to inclusive wall-clock time and policy quality within a common resource budget. The measurement includes initialization, generation, environment execution/restoration, predictor fitting, Replay rescoring, policy optimization, and evaluation.

Under the same 12,537-second allocation, approximately 3.482 hours, QCAR solves 109/134 tasks, compared with 103/134 for GRPO, 90/134 for Fixed Root/Fork, and 65/134 for dynamic mixed-group sampling.

MethodTraining dataSuccesses within shared wall time
QCARALFWorld109/134
GRPOALFWorld103/134
Fixed Root/ForkALFWorld90/134
Dynamic mixed-group samplingALFWorld65/134

Each run reserves eight H800 GPUs, and each score is the latest completed validation within the common budget on the same 134 held-out tasks. QCAR exceeds the fixed Root/Fork control by 19 tasks and achieves the highest validation accuracy among these methods within the allocation.

The matched-update comparison exposes the additional generation required by dynamic sampling. During experiments, dynamic sampling and QCAR achieve similar mean validation success counts: 38.40/134 versus 38.00/134. Dynamic sampling uses 2,208,379 fresh generated tokens and 3.482 hours, compared with 627,510 tokens and 1.605 hours for QCAR—3.52× the tokens and 2.17× the inclusive time.

Dynamic sampling retains 160/160 mixed training groups, compared with 81/160 for QCAR, but generates 4,736 trajectories, retaining 1,280 and discarding 3,456. Its complete candidate pool contains 176/592 mixed groups. Fully mixed retained batches therefore require substantial additional acquisition, while matched-update validation performance remains close. Mean validation successes are averaged across shared scheduled evaluation checkpoints; inclusive time covers launch through completion of the evaluations included in this comparison.

AuthorsOpenReview ↗

Rebuttal (Part 2/2)

Fixed-pool filtering

We also evaluated a simpler filtering control that generates 256 candidate trajectories and retains 128 per actor update, prioritizing mixed groups.

During experiments, QCAR and filtering achieve similar mean validation success counts across shared evaluation checkpoints: 23.00/134 and 22.00/134, respectively. QCAR produces 48/96 mixed training groups, compared with 40/96 for filtering, corresponding to 20% more mixed groups. Filtering generates 1,536 trajectories, retaining 768 and discarding 768.

Thus, fixed-pool filtering generates twice as many candidates as it retains, yet produces fewer mixed training groups than QCAR, with similar validation performance. Together with the dynamic-sampling comparison, this distinguishes the contrast retained for optimization from the generation required to acquire it.

Replay ablation and off-policy considerations

Following your suggestion, we compared QCAR w/o Replay and QCAR w/ Replay on ALFWorld.

Including Replay raises mean validation successes from 102.80/134 to 106.50/134 and endpoint successes from 124/134 to 127/134. Mixed training groups increase from 175/1600 to 191/1600, a 9.1% increase, while the number of distinct mixed tasks remains 56.

SettingTraining dataMean validation successes (/134)Endpoint successes (/134)Mixed groupsDistinct mixed tasks
QCAR w/o ReplayALFWorld102.80124175/160056
QCAR w/ ReplayALFWorld106.50127191/160056

Means average all scheduled validations during these experiments. The matched cohort contains 64 training tasks, and each update trains on 16 tasks and 128 trajectories. Replay occupies part of this fixed actor budget, replacing fresh generation in those slots. The results support its contribution within QCAR: additional reward contrast is accompanied by improved validation performance, with the same distinct mixed-task coverage.

Rescoring computes the reference token probabilities under the current policy and holds them fixed during optimization. Replay eligibility restricts task identity and policy age, the quality guard regulates reuse opportunities, and fresh-sampling minima preserve current-policy exploration.

For the theoretical analysis, let PrepP_{\mathrm{rep}} denote the distribution of Replay trajectories admitted to optimization and PvP_v the current-policy trajectory distribution. For a fixed gradient integrand gvg_v satisfying ∥gv(τ)∥≤G\|g_v(\tau)\|\leq G, define

gˉmix=(1−α)EPv[gv(τ)]+αEPrep[gv(τ)],\bar g_{\mathrm{mix}}=(1-\alpha)\mathbb E_{P_v}[g_v(\tau)]+\alpha\mathbb E_{P_{\mathrm{rep}}}[g_v(\tau)],

where α\alpha is the Replay mixture fraction. Then

∥gˉmix−EPv[gv(τ)]∥≤2αG TV⁡(Prep,Pv).\left\|\bar g_{\mathrm{mix}}-\mathbb E_{P_v}[g_v(\tau)]\right\|\leq2\alpha G\,\operatorname{TV}(P_{\mathrm{rep}},P_v).

This bound identifies Replay exposure and Replay/current-policy divergence as relevant factors, motivating QCAR’s coordinated control of reuse eligibility, policy age, and fresh exploration.

Relation to allocation and prefix-exploration methods

The key attribution question is which mechanisms account for the improvement. Our added experiments provide complementary evidence: dynamic sampling and fixed-pool filtering evaluate alternative acquisition strategies; the factorial experiment examines acquisition–credit interaction; the Replay ablation tests reuse within QCAR; and the graph switch evaluates the utility multiplier.

These controls address mechanisms central to the methods you identify. Knapsack RL focuses on rollout-budget assignment; InfoTree combines tree selection, expansion, and hierarchical advantages; Prefix Sampling changes continuation starting points and copied-prefix credit. We will clarify these distinctions in the revised related-work discussion.

The revised experimental description will specify the allocation unit, retained actor batch, generated and discarded trajectories, environment restoration, and credit objective. Inclusive timing accounts for QCAR’s execution and bookkeeping costs, while the factorial and Replay controls clarify how acquisition, policy credit, and reuse contribute to policy learning.

Reviewer_Ta8POpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes QCAR on top of GRPO as a dynamic sampling approach to increase the likelihood of each rollout group for a given prompt to be mixed with successful & unsuccessful attempts. QCAR treats agent's tool executions in environment as a stateful graph, and use branching to control stateful exploration in model rollouts. And QCAR samples dynamic number of rollouts for each prompt so that the chance of getting at least 1 successful or 1 unsuccessful sample. The authors how that on ALFWorld, Natural Questions, HotpotQA, Webshop, QCAR can outperform GRPO under similar training compute (same steps, same # of rollouts per optimization step).

Critical strengths

S1. The paper touches on the right problem and have a sound motivation. i.e. in GRPO, there's no mechanism for making sure we sample "balanced" groups of rollouts with both successful and unsuccessful attempts, which would lead to low signal to noise ratio in some training steps and make it hard to control the variance of gradients across training steps.
S2. QCAR uses similar training compute budget compared to GRPO but aim to balance the # of samples across instances.

Critical weaknesses

W1. Lack of critical comparison / discussion against other methods dealing with similar problems: I think QCAR's competetive advantage will be much better justified if authors can compare it to (1) existing approaches with similar motivations (2) and a few simpler baselines.

  • Existing approaches: e.g. DAPO [1], which has become a popular & empirically validated way of solving the unbalanced number of mixed (success, failure) sampling groups across training batches using dynamic sampling methods. Although DAPO's dynamic sampling strategy uses more training time compute compared to QCAR, but from the variance reduction perspective of policy gradient (see above bullet point), DAPO seems to make more sense than the proposed QCAR here at least on paper. So in any case, given how similar the two methods are, it'd be helpful to do a baseline comparison between the two. This will also be a helpful ablation study for understanding how the stateful graph + branching contributes to performance vs. dynamic sampling.
  • Simpler baseline: Ideally there are simpler baselines we can compare to in order to justify the design of QCAR. e.g. (1) Dropping all success / failure batches (which DAPO also does), or (2) Do a simple curriculum / batch scheduling [2]; either move the unsuccessful batches to later steps, etc. (3) Rejection sampling fine-tuning [3]

W2: In current form, QCAR would cause task/prompt level variance shift in its policy gradient estimation: This is my biggest concern on QCAR when I look at how QCAR uses different group sizes and design the "task weights". e.g. From eq 8 and eq 9, assuming we have tasks weights which sum to 1, effectively this prevents "tasks/prompts" with more rollout receives more total weight. But the problem is you get imbalanced variance / covariance control on individual task level by doing this. e.g. a simple way to think about this would be, let's say you have two groups, one group of 4 rollouts and one group of 8 rollouts, suppose each group has 1 success rollout, then the gradient contribution of the successful rollout in group of 4 would have a larger coefficient than the group of 8 even with the task weights, making the gradient optimization more subject to task/prompt-level variance. e.g. In DAPO [1], the variance control is much cleaner because the group sizes are consistent.

My concern here is that QCAR introduces this unbalanced group size + weighting strategy without properly describing its effect on gradient covariance. I would suggest the authors add more theoretical justification here, and having empirical comparisons against established baselines could also help.

[1] Yu, Q., Zhang, Z., Zhu, R., Yuan, Y., Zuo, X., Yue, Y., Dai, W., Fan, T., Liu, G., Liu, L. and Liu, X., 2026. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38, pp.113222-113244.

[2] Shi, T., Wu, Y., Song, L., Zhou, T., & Zhao, J., 2026. Efficient Reinforcement Finetuning via Adaptive Curriculum Learning. Transactions on Machine Learning Research

[3] Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chowdhery, A., Pan, C., & Zhang, S., 2023. RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment. Transactions on Machine Learning Research.

Questions for authors

See weaknesses above.

2 comments
AuthorsOpenReview ↗

Rebuttal (Part 1/2)

Dynamic sampling and simpler acquisition baselines

Thank you for suggesting more direct acquisition baselines. We added dynamic mixed-group sampling, fixed Root/Fork, and fixed-pool filtering controls, keeping the actor-training batch at 16 tasks and 128 trajectories.

The main result is better use of an expensive interaction budget. Within the same 12,537-second allocation, QCAR reaches 109/134, compared with 103/134 for GRPO, 90/134 for Fixed Root/Fork, and 65/134 for dynamic mixed-group sampling. QCAR exceeds the fixed branching control by 19 tasks. Compared with dynamic sampling, QCAR avoids repeated generation solely to fill a mixed-only training batch and completes more policy updates within the same budget.

Primary comparison: policy quality at the same wall-clock budget.

MethodTraining dataSuccesses at equal wall time
QCARALFWorld109/134
GRPOALFWorld103/134
Fixed Root/ForkALFWorld90/134
Dynamic mixed-group samplingALFWorld65/134

All methods receive 12,537 seconds, approximately 3.482 hours, on eight reserved H800 GPUs. Training tasks are drawn from the 3,553-task ALFWorld pool, and evaluation uses 134 unseen tasks. Each score is the latest completed validation within this budget. The inclusive measurement covers initialization, generation, environment interaction/restoration, predictor fitting, rescoring, optimization, and evaluation.

Matched-update comparison: fully mixed training batches require substantially more generation. Dynamic sampling retains 160/160 mixed training groups, compared with 81/160 for QCAR. Achieving this fully mixed batch requires generating 4,736 candidate trajectories, retaining 1,280 and discarding 3,456. Its complete candidate pool contains 176/592 mixed groups; the 160/160 figure describes the retained training groups after filtering.

During experiments, dynamic sampling and QCAR achieve similar mean validation success counts at matched updates: 38.40/134 versus 38.00/134. Dynamic sampling nevertheless consumes 3.52× the fresh generated tokens and 2.17× the inclusive time. Its higher retained mixed-group count therefore comes with substantial acquisition overhead, while the corresponding validation performance remains close.

MethodTraining dataMean successes at matched updatesMixed training groupsFresh generated tokensInclusive time
GRPOALFWorld28.0049/160643,7191.634 h
QCARALFWorld38.0081/160627,5101.605 h
Dynamic mixed-group samplingALFWorld38.40160/1602,208,3793.482 h

Mean successes are averaged across the same scheduled evaluation checkpoints within the matched-update comparison, with each evaluation using the same 134 held-out tasks. Inclusive time covers all computation from launch through the final evaluation included in this comparison, using separate eight-H800 allocations. The equal-wall-clock comparison reports each method’s latest completed evaluation within the common time budget.

AuthorsOpenReview ↗

Rebuttal (Part 2/2)

Fixed-pool filtering and related approaches

To test whether simple oversampling and homogeneous-group filtering is sufficient, we added a fixed-pool filtering control. It generates 256 candidate trajectories and retains 128 per actor update, prioritizing mixed groups. During experiments, QCAR and filtering achieve similar mean validation success counts across their shared evaluation checkpoints: 23.00/134 and 22.00/134, respectively. QCAR produces 48/96 mixed training groups, compared with 40/96 for filtering, corresponding to 20% more mixed groups. Filtering generates 1,536 trajectories, retaining 768 and discarding 768. Thus, despite generating twice as many candidate trajectories as it retains, fixed-pool filtering yields fewer mixed training groups than QCAR, while their validation performance remains close.

Together, these comparisons distinguish the number of mixed groups retained for optimization from the resources required to acquire them. Dynamic sampling obtains fully mixed batches through substantial additional generation, while fixed-pool filtering limits candidate generation to 256 trajectories per update. QCAR produces more mixed groups than fixed-pool filtering and achieves comparable matched-update validation performance to both acquisition alternatives. Under the same wall-clock budget, QCAR achieves higher checkpoint accuracy than GRPO, Fixed Root/Fork, and dynamic sampling.

We will label the dynamic control as the DAPO-style dynamic-sampling component, distinguishing it from the complete DAPO optimizer recipe. Complete DAPO also changes clipping, loss aggregation, and long-output treatment. The added control evaluates its dynamic-acquisition component within our training setup; the equal-wall-clock comparison evaluates the resulting training schemes end to end.

We will also sharpen the comparison with AdaRFT, which schedules task difficulty, and RAFT, which reward-ranks samples for supervised fine-tuning. These distinctions clarify how the alternatives operate through experience acquisition, task scheduling, and optimization objectives.

Task normalization, unequal group sizes, and covariance

Thank you for identifying the distinction between task mass and gradient covariance. Task normalization prevents a task from receiving more explicit objective weight merely because the allocator acquired more trajectories for it. Gradient covariance additionally depends on individual gradient magnitudes, group-relative advantage normalization, within-task dependence, and shared-prefix cross terms. We will make these roles explicit in the revision.

Your four-versus-eight example illustrates this distinction. With one success, binary rewards, and sample-standard-deviation GRPO, the successful trajectory’s coefficient in a task mean is

cn=n−1nn,c_n=\frac{n-1}{n\sqrt n},

giving 0.375 for n=4n=4 and approximately 0.3094 for n=8n=8, before numerical epsilon. Equal task mass therefore coexists with group-size-dependent coefficients for individual trajectories.

For an acquisition plan AA and task gradient estimator Gx=∑jwj(A)gjG_x=\sum_j w_j(A)g_j, we will explicitly include both the conditional cross-covariances and the contribution from random acquisition:

Cov⁡(Gx∣A)=∑j,kwj(A)wk(A)Cov⁡(gj,gk∣A),\operatorname{Cov}(G_x\mid A)=\sum_{j,k}w_j(A)w_k(A)\operatorname{Cov}(g_j,g_k\mid A),

Cov⁡(Gx)=E[Cov⁡(Gx∣A)]+Cov⁡(E[Gx∣A]).\operatorname{Cov}(G_x)=\mathbb E\left[\operatorname{Cov}(G_x\mid A)\right]+\operatorname{Cov}\left(\mathbb E[G_x\mid A]\right).

These expressions distinguish three effects: task/type weights determine explicit objective contribution; group size and advantage normalization affect individual coefficients; and acquisition dependence introduces covariance terms. Masking reused Fork prefixes avoids duplicate loss on those tokens, while correlations among the remaining gradient contributions still enter the covariance.

Accordingly, we will describe Balanced-TreeRPO as a mechanism for controlling explicit task/type weighting and assigning credit according to exact trajectory lineage. We will explicitly distinguish task-weight normalization from gradient-covariance behavior and revise the corresponding claims. The added experiments evaluate the end-to-end performance and acquisition cost of the resulting training scheme.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.