- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces QCAR, a quality-constrained framework for allocating rollouts in stateful agentic reinforcement learning. Using initial probe trajectories and graph-conditioned predictions, QCAR distributes the training budget among fresh trajectory generation, prefix branching, and replay. Quality constraints aim to prevent ineffective reuse, while the policy update procedure accounts for differences in group size and shared trajectory prefixes. Experiments exhibits an average accuracy gain of 2.32% over GRPO and a reduction in fresh-token usage of up to 8.35%.
Critical strengths
-
Efficient exploration and rollout allocation is an important problem to be studied in agentic RL, where expensive interactions do not always yield useful learning signals. Improving the use of collected trajectories therefore has clear practical value.
-
The experiments on outcome prediction, stalled continuations, and reward contrast help motivate the proposed approach. They provide empirical support for considering graph-derived information when allocating rollout resources.
Critical weaknesses
-
(critical) AUROC on seen tasks provides only partial evidence of the predictor’s usefulness. It does not guarantee whether predictions remain reliable on unfamiliar task structures or under the changing policy distribution encountered during training.
-
QCAR relies on predicted probabilities to allocate rollouts, yet the paper does not evaluate their calibration. This matters because inaccurate estimates may distort resource allocation. In particular, states whose potential is underestimated may receive less exploration, reducing opportunities to correct those estimates and potentially reinforcing the initial errors.
-
(critical) Prior work, including [1], has already investigated how to estimate the utility of exploration. Although QCAR addresses rollout allocation during training, the paper does not clearly clarify which aspects of its formulation or mechanism constitute a substantive advance over related exploration-aware approaches.
-
The evaluation includes GRPO but ignores relevant agentic RL methods, such as GiGPO [2], as well as alternative rollout-allocation and trajectory-reuse strategies. These comparisons are needed to examine the benefit of QCAR’s proposed allocation mechanism.
[1] Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization [2] Group-in-Group Policy Optimization for LLM Agent Training
Questions for authors
How does the proposed value estimator perform on unseen task structures, and how stable are its predictions as the policy distribution changes during training? Can the authors provide direct evidence of its reliability in online allocation decisions?
2 comments
Rebuttal (Part 2/2)
Controlled acquisition and policy-learning results
We added acquisition baselines and an acquisition-by-credit factorial experiment to evaluate how stateful acquisition contributes to policy learning.
Under the same wall-clock budget, QCAR achieves the highest validation accuracy among the compared methods.
| Method | Training data | Successes within shared wall time |
|---|---|---|
| QCAR | ALFWorld | 109/134 |
| GRPO | ALFWorld | 103/134 |
| Fixed Root/Fork | ALFWorld | 90/134 |
| Dynamic mixed-group sampling | ALFWorld | 65/134 |
Each run receives 12,537 seconds, approximately 3.482 hours, on eight reserved H800 GPUs. Scores are the latest completed evaluations within that allocation on the same 134 held-out tasks. Inclusive time covers initialization, generation, environment interaction/restoration, predictor fitting, Replay rescoring, optimization, and evaluation.
QCAR exceeds Fixed Root/Fork by 19 successful tasks, supporting the complete adaptive acquisition-and-learning scheme over fixed branching in this setting.
Fully mixed batches require substantial additional generation. During matched-update experiments, dynamic sampling and QCAR achieve similar mean validation successes: 38.40/134 versus 38.00/134. Dynamic sampling consumes 3.52× the fresh generated tokens and 2.17× the inclusive time. It generates 4,736 trajectories, retains 1,280, and discards 3,456 to obtain 160/160 mixed training groups. QCAR obtains 81/160 mixed groups with substantially less generation. Mean validation successes are averaged across shared scheduled evaluation checkpoints.
We also added fixed-pool filtering, which generates 256 candidates and retains 128 per actor update. QCAR produces 48/96 mixed groups versus 40/96, a 20% increase, while mean validation successes remain close at 23.00/134 versus 22.00/134. Filtering generates 1,536 trajectories and discards half.
Acquisition and policy credit work together. The factorial control crosses fixed/adaptive Root-Fork acquisition with ordinary/balanced+sibling credit:
| Acquisition | Credit | Mean validation successes | Mixed groups |
|---|---|---|---|
| Fixed Root/Fork | Ordinary | 33.00 | 45/160 |
| Fixed Root/Fork | Balanced + sibling | 32.60 | 44/160 |
| Adaptive Root/Fork | Ordinary | 32.40 | 47/160 |
| Adaptive Root/Fork | Balanced + sibling | 34.40 | 47/160 |
All arms use the same Qwen3-8B initialization, actor-training budget, KL coefficient, and policy-GPU topology. Under fixed acquisition, the credit schemes yield similar validation performance. The combination of adaptive acquisition and balanced+sibling credit achieves the highest mean. Within adaptive acquisition, balanced+sibling credit improves the mean from 32.40/134 to 34.40/134, while both configurations produce 47/160 mixed groups, indicating a benefit in how acquired contrast is used during optimization.
Replay also contributes within QCAR.
| Setting | Mean validation successes (/134) | Endpoint successes (/134) | Mixed groups |
|---|---|---|---|
| QCAR w/o Replay | 102.80 | 124 | 175/1600 |
| QCAR w/ Replay | 106.50 | 127 | 191/1600 |
Replay occupies part of the fixed actor budget. Its inclusion improves mean and endpoint validation performance while increasing mixed groups by 9.1%.
Contribution relative to exploration-aware optimization
We will make the connection to Learning to Explore/EAPO explicit. EAPO uses information-value rewards and exploration/completion-aware optimization. QCAR contributes a training-time experience-acquisition layer that allocates a retained budget among fresh generation, verified-prefix Fork, and bounded-age Replay, with quality feedback and provenance-aware credit.
This layer coordinates how much experience to acquire, where to continue, when to reuse, and how reused tokens contribute to the update. The observable graph organizes acquisition opportunities; exact lineage enables verified restoration and suffix credit.
We will organize related work by mechanism: GiGPO for policy credit, Knapsack RL for rollout budgets, and InfoTree/Prefix Sampling for tree or prefix exploration. The online prediction audit, executed-Fork accounting, acquisition controls, credit factorial, and Replay ablation jointly evaluate these aspects of quality-constrained, stateful experience acquisition.
Rebuttal (Part 1/2)
Online prediction on new instructions and changing policies
Thank you for highlighting predictor reliability under instruction novelty and policy evolution. We added an audit of the actual online neural allocator, using probabilities recorded before expansion and before fitting the current update’s outcomes.
The new-instruction cohort contains 736 distinct instructions absent from all earlier predictor updates and 1,482 bare-root outcomes. Within-policy-update AUROC is 0.827 on this cohort and 0.802 across all post-initialization roots, with a pooled AUROC of 0.805. These results support task discrimination on newly encountered instructions and under the policy making the allocation decision.
The complete post-initialization root cohort contains 3,168 outcomes from 1,584 previously unselected game files. Instruction novelty is determined against the predictor’s earlier online fitting history, and all audited predictions precede their corresponding outcomes.
We also evaluate forward transfer in time: a monotone affine-logit calibrator is fitted on a preceding policy window, frozen, and evaluated on the following window. Across four future windows on first-instruction encounters, ECE improves from 0.06801 to 0.04847, and Brier score improves from 0.12580 to 0.12368. The forward test cohort contains 994 outcomes from 493 instructions. This evaluates probability adaptation on subsequent allocation decisions as the policy changes.
Calibration and protection against under-exploration
We expanded probability evaluation to include Brier score, ECE, a recent-success-rate baseline, and forward temporal testing. The recent-rate baseline measures global policy progress, while within-update AUROC evaluates task-specific ranking.
Fresh exploration is maintained independently of the predicted value. Every candidate task receives at least four fresh trajectories, including two reward-blind probes. Tasks with low predicted success therefore continue to supply new observations for policy learning and predictor recalibration. Allocation combines predicted utility with uncertainty, coverage, and an exploration bonus.
For prefixes, we record candidate probabilities, selected anchors, restoration outcomes, and exact lineage. These records connect predicted opportunities to executed continuations and their observed rewards.
Executed Fork decisions and graph utility
We added complete pre-selection and execution-trace auditing. All 160 selected Fork requests match the actual restored parent/boundary lineage. These executions produce 35 parent–Fork reward contrasts and convert 14 otherwise homogeneous task groups into mixed groups, relative to the same batches’ fresh trajectories. These measurements verify both faithful execution of the selected continuations and their contribution to additional reward-contrasting training groups.
We further isolate the graph utility multiplier while retaining text prediction, auxiliary-head fitting, credit, and actor budget:
Training tasks are drawn from the 3,553-task ALFWorld pool, and evaluation uses 134 unseen tasks. Means average the same scheduled evaluation checkpoints. The learned graph utility multiplier improves both mean validation performance and mixed-group yield, supporting its contribution to informative acquisition and subsequent policy learning.
We will explain the implementation roles explicitly: the text model supplies continuation-success probabilities, the graph head modulates acquisition utility, and exact token lineage verifies prefix reuse and determines suffix credit.