acceptodds
Under review as a conference paper at ICLR 2027

WHEN QUEUE POLICY BECOMES LEARNING POLICY: AUDITING BATCH PATHS IN AGENTIC RL

Abstract

An asynchronous rollout queue can change learning even when every trajectory is eventually consumed: it determines which complete prompt groups share an optimizer step and in what order. We audit this batch-path effect using immutable recorded experience and persistent-Adam GRPO. In a prospective six-seed Qwen3-0.6B experiment, reversing two task-family blocks changes final product-minus-shared success by -23.6 percentage points [-30.5, -16.3], with experience, tokens, initialization, and update count matched. Each order favors its first family relative to balanced batches. An independent six-seed intervention clears only Adam’s first moment at the family switch, attenuating the contrast by +21.88 points [+16.54, +27.21]; all 168 checkpoint-and-optimizer control pairs are exact. The reset mainly removes the first-family advantage and reduces aggregate accuracy. We make the batch-composition choice explicit through a per-update contract for required coverage and histogram total variation, with earliest-feasible complete-group admission. Simple per-family queues exactly match the general allocator in the tested two-family case. Measured multi-turn traces expose composition violations and an explicit waiting cost. A separate 3B learning result remains unconfirmed, and a public-code pilot fails validity and its numerical screens. The evidence supports a controlled queue-path diagnosis and a composition guarantee, without establishing broad learning generality or time-to-quality improvement.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
agentic reinforcement learning, asynchronous learning systems, batch composition, order-sensitive learning, systems contracts, optimizer memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_c64gOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In asynchronous GRPO-style post-training the queue fixes the order of optimizer batches, and persistent Adam state makes that order matter. The paper formalizes this "batch path", proposes an admission contract with an exact allocator, and audits it with a pre-registered six-seed 0.6B experiment over eight updates. Reversing the two tool-family blocks shifts product-minus-shared success by −23.6 pp; a first-moment reset attenuates this but lowers aggregate success, and a larger FIFO-versus-contract study is null.

Critical strengths
  • Clean isolation of the batch-path treatment. Recording all trajectories before assigning availability and byte-identical fixed-batch checkpoints (Appendix B.2–B.3) make "same experience, different path" an estimand, and pre-registration lets the −23.6 pp effect be taken at face value.
  • Transparent reporting of failed and unconfirmed extensions. Appendix D and Table 9 document every stopped, invalid or inconclusive study with its frozen decision rule.
Critical weaknesses
  • Headline effect is Adam's prediction at T=8; no longer horizon tested. With T=8 below Adam's time constant 1/(1−β₁)=10, idealised constant-gradient accounting gives the first family ≈6.2 update-equivalents and the second ≈1.8, so the −23.6 pp reversal in Table 3 is, in my reading, the expected momentum effect. Absent a β₁=0/SGD ablation or horizon sweep, this confirms only Sweeney (2026)'s momentum-favors-the-first-comer result.
  • The reset's "capability cost" follows from preserving the Adam clock. Zeroing m before step 5 leaves the second family at ≈1.8 and removes only first-family coasting, which qualitatively predicts Table 5 (first family −13.80 pp, second −1.37 pp); the claimed "material first-moment dependence" is, in my reading, a design artifact, not a queue property.
  • Real queues never linked to learning; contract never shown to help. The closest thing to a natural queue (3B FIFO-versus-contract) is null under the pre-registered rule (+2.08 pp [−1.11, +5.34], Appendix D.1), and the learner-free τ²-bench traces fail the paper's joint rule, leaving "agentic RL" supported by trace timing alone.
Questions for authors
  • Report absolute update-8 held-out success for source, balanced and blocked paths; how far do eight steps move the model?
  • Do saved Adam diagnostics match the accounting above (≈6.2 vs 1.8; 4.0 vs 1.8 after reset), and did you try a full reset or β₁=0?
  • Run a horizon sweep (T up to 512) under measured service-trace imbalance plus a β₁=0/SGD control to see where the interaction decays.
Reviewer_itCMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether an asynchronous rollout queue can change learning even when every complete prompt group is eventually consumed. Using Qwen3-0.6B with LoRA on two synthetic arithmetic tool-call tasks, the authors fix the experience and compare balanced.

Critical strengths

The methodology is clear and the reporting is honest. Failed and inconclusive extensions are disclosed thoroughly, and the claims are mostly appropriately modest.

Critical weaknesses
  • The underlying phenomenon is general order sensitivity under a stateful optimizer, which would likely appear in SFT as well and has already been studied. The only confirmed learning experiment is single-step calculator calls, so "agentic RL" reads as a motivating application rather than the object of study.
  • All confirmed effects come from one 0.6B model. Larger-model attempts are all negative or invalid. Whether the effect persists at realistic scale is unknown.
  • Industry still relies on asynchrony for throughput, and the paper offers no guidance that would change how such training is done.
Questions for authors

The related issues will be elaborated in the "Weakness" section.

Reviewer_zAx7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how queue scheduling affects learning outcomes in asynchronous agent-based reinforcement learning. Through controlled experiments that hold training data constant while varying batch composition and update order, the authors analyze changes in task performance and investigate the role of Adam’s first-order momentum through reset interventions. The paper also proposes a scheduling mechanism that constructs batches subject to task coverage and proportion constraints, evaluating its composition guarantees and associated waiting costs.

Critical strengths
  1. This paper addresses an intriguing and worthy research question: even when the training data used is identical, differences in batching and update ordering, which arise from asynchronous queuing, can still influence the model’s learning outcomes.

  2. The controlled experimental design is rigorous; by keeping the training data, initialization, and number of updates constant while varying the training trajectory, the authors analyze the "ordering effect" and its potential underlying mechanisms, which are further supported by control experiments with fixed batches and targeted interventions on the first-order momentum of the Adam optimizer.

Critical weaknesses
  1. Limited connection to real asynchronous training. The strongest results rely on artificially blocked task sequences and rollouts generated by a fixed policy. It remains unclear whether realistic queues produce comparable effects when the policy is continuously updated. A direct comparison of learning outcomes against random batching is also missing.

  2. Unverified practical benefits. Matching batch composition does not necessarily improve learning quality or reduce the time needed to reach a target performance level. In the tested two-family setting, simple per-family queues exactly match the controller’s behavior, leaving its added value and associated waiting costs insufficiently justified.

  3. Limited experimental scale. The demonstrated learning effects are based on Qwen3-0.6B, two synthetic task templates, 64 rollouts, and eight updates. Validation across models, optimizers, and longer training runs is lacking, and the extensions to larger models and other tasks do not establish generality.

  4. Writing and organization need improvement. The abstract includes too many numerical results, control checks, and details about experimental validity, obscuring the core research question, key findings, and broader implications. The main text, meanwhile, is overly condensed. The paper would benefit from fewer implementation details in the abstract and a more systematic presentation of the experimental design and the conclusions drawn from the results in the main text.

Questions for authors

Please refer to the weaknesses part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.