acceptodds
Under review as a conference paper at ICLR 2027

Beyond Fixed Groups: Adaptive Rollouts for Coding-Agent Reinforcement Learning

Abstract

Long-horizon coding-agent rollouts have highly variable latency: an agent may search a repository, edit files, and run several test suites before receiving one verifiable reward. Fixed-size GRPO allocates the same number of rollouts to every query, while round-based collection delays resampling and fresh work behind stragglers. We present ARCA, an event-driven realization of Adaptive Rollouts for Coding Agents. It processes each query when its current sampling stage completes, closes mixed groups, resamples all-wrong groups within a bounded budget, and refills fresh queries. The policy stays fixed during collection; the learner updates after collection and draining. In single-run experiments with Qwen3.6-27B on SWE-bench Verified, the schedule at fresh-query width records rollout steps/h versus for native GRPO (). Observed peak pass@1 on the 51-problem evaluation slice is versus , a gain of percentage points. At , reaches peak pass@1, while records the highest throughput, steps/h. These throughput comparisons use different mixed-group targets and do not measure total compute savings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic reinforcement learning, rollout scheduling, adaptive sampling, group-relative policy optimization, verifiable rewards.

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 2 reviews, above 0% of the others.
Reviewer_1KLKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ARCA, an event-driven adaptive rollout collection strategy for coding-agent RL. It dynamically allocates additional rollouts to all-failure groups, stops sampling once a group contains both successful and failed trajectories, and asynchronously refills completed queries. Experiments on SWE-bench Verified report improved training throughput and pass@1 over fixed-size GRPO.

Critical strengths
  1. Addresses a practical efficiency problem in long-horizon coding-agent RL.

  2. Evaluates on real coding-agent tasks, with additional OOD results on SWE-Gym and Terminal-Bench.

Critical weaknesses
  1. Limited novelty. The core ideas substantially overlap with existing work. NGU already investigates asynchronous resampling of unsuccessful queries; AR3PO and AERO study adaptive rollout allocation; Reinforce-Ada and SARA provide more principled formulations of adaptive sampling and stopping decisions. The paper does not clearly establish what is fundamentally new beyond combining existing techniques in a coding-agent setting.

  2. Inadequate baselines. Despite extensive related work, the experiments primarily compare against fixed-size GRPO. There are no empirical comparisons with closely related adaptive sampling methods such as NGU, AR3PO, AERO, or even DAPO-style dynamic sampling. Outperforming vanilla GRPO is insufficient to establish the effectiveness of the proposed approach relative to existing methods.

  3. Insufficient experimental rigor. The main experiments lack multi-seed evaluations and report peak performance on a small 51-problem evaluation set. More importantly, most GRPO baselines exhibit severe training collapse. It is unclear whether the reported performance gains reflect the effectiveness of ARCA or the instability of the chosen baselines.

  4. Unclear presentation and inconsistent experimental evidence. The paper's overall narrative is difficult to follow, and the appendix contains substantial historical results and defensive explanations that obscure the final method and its supporting evidence. A particularly concerning example is the hard-task analysis in Section 5 (Figure 6): Appendix A.5 explicitly states that these results use a historical sampler that resamples both all-correct and all-wrong groups, rather than the proposed drop-all-correct variant. Since these variants have different stopping rules and sampling distributions, the analysis does not directly support the claimed hard-task improvements of the final method.

References

  • NGU: Noukhovitch, M., et al. (2026). Learning to Solve Hard Problems in RL for LLMs by Never Giving Up. arXiv:2609.13443

  • Reinforce-Ada: Xiong, W., et al. (2025). Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives. arXiv:2510.04996

  • SARA: Nomand, P., et al. (2026). Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR. arXiv:2607.26253

  • AR3PO: Zhang, Y., et al. (2025). Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse. arXiv:2509.25808

  • AERO: Zhang, Z., et al. (2026). Train Less, Learn More: Adaptive Efficient Rollout Optimization for Group-Based Reinforcement Learning. arXiv:2602.14338

  • DAPO: Yu, Q., et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. NeurIPS 2025. arXiv:2503.14476

Questions for authors
  1. What is the key algorithmic contribution of ARCA beyond NGU, AR3PO, and AERO? Why are these methods not included as baselines?

  2. Why do most GRPO baselines collapse so severely? Do the improvements persist against stable, properly tuned baselines and across multiple seeds?

  3. Can the authors reproduce the hard-task analysis in Figure 6 using the final drop-all-correct algorithm rather than the historical variant?

Reviewer_TTzROpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ARCA, an event-driven realization of adaptive rollouts for coding agents.

Critical strengths

Introduction is well written and clear. Figures are clearly presented. The problem is well-defined and timely.

Critical weaknesses
  • The writing over-indexes on a few experiments and makes broad generalisations from here. How do we know that these findings generalize? For example, there are no error bars in Table 1. Was only one seed used for experiments?
  • SWE-Bench-verified is a good dataset to use, but it's quite small relative to others. Why were other comparable coding agent evaluations not also used, such as Terminal-Bench?
  • The only baseline used is vanilla GRPO -- why were other key baselines excluded?
  • Figures 7 & 8 are difficult to follow, with text overlap and unclear captions/axes.
Questions for authors
  • Why are there no other baselines to compare to?
  • Why are there no multi-seed experiments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.