- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper identifies the issue of scheduling environment tasks for agentic RL, where the way of the schedule affects how the agent behaves under finite compute budgets. To that end, the author proposes to use a graph-based data structure to model cross-task and within-task dependencies and use exploitation and coverage (exploration) through a specified reward signal on selecting the schedule of tasks. Experiment results demonstrate that the proposed approach EnvSched outperformed random baselines and ablation studies show each component helps performance.
== post rebuttal
Thank you for the thorough response and adding the UCB baseline. However, it seems that the margin between the proposed method and UCB is not as large as compared to the random baseline. For that reason, I choose to maintain my score.
Critical strengths
The paper does identify a new setting where exploration and exploitation could matter, which is environment task scheduling in agentic RL. It also combines a graph data structure with variance as the reward to select a good schedule of the tasks. The paper is easy to read and follow.
Critical weaknesses
The only baseline of the paper is a random baseline by selecting tasks uniformly at random, which is the weakest baseline possible. Although the ablation studies show that each design choice in EnvSched gives performance improvements, the baseline comparison still raises questions about whether EnvSched overall is a strong method. One stronger baseline I can think of is using bandit algorithms on each task, such as the upper-confidence bound algorithm, so that a more informed choice of tasks based on history is made.
Questions for authors
The exploration and exploitation strategy of EnvSched follows an epsilon-greedy strategy with a fixed probability for exploration. I wonder if the authors considered more advanced search heuristics such as bandit algorithms, which would probably give a greater improvement?
Response to Reviewer zdGb
Thank you for your review and your positive assessment of the problem formulation and writing. Your suggestion to include a bandit baseline was helpful.
W1 and Q1: Baselines, the distinction from ε-greedy, and the UCB comparison
Baselines. In addition to Random, Table 1 and Section 3.1 of the original manuscript compare two recent online data schedulers: DPS (Mao et al., 2026) and LEEPS (Liang et al., 2026). LEEPS itself is an explore–exploit portfolio strategy based on historical outcomes. For Qwen3-4B at 200 steps, EnvSched exceeds the stronger DPS/LEEPS score on each benchmark by 1.50 / 2.22 / 1.33 points. We will make these comparisons more prominent in the revision.
ε-greedy. EnvSched's exploration does not select tasks randomly. Coverage prioritizes environments not yet covered in the batch and with lower cumulative exposure, then less-observed tasks within them, and finally ranks by predicted value and within-batch redundancy (Section 2.4). This increases environments covered per batch from 36.53 under Random to 45.96 (Table 2).
Bandits. Thank you for this suggestion. It is a natural approach that we also considered during design. We did not directly adopt a per-task bandit mainly because of three characteristics:
Task value can be estimated before the first attempt. Per-task UCB requires repeated pulls to learn an arm's value. Yet each task is selected only about five times on average over 200 steps, making per-task convergence difficult. In agent tasks, reward differentiation depends largely on structure, such as a key lookup that sometimes succeeds and sometimes fails. Similar tasks share this structure.
Task value changes during training and often declines precisely because the task has been trained on. Pulling an arm leaves its reward distribution unchanged in a bandit, while training teaches the policy to solve a task, reducing its informativeness. Much of a task's value may be consumed before repeated pulls establish it. Sliding-window UCB can handle drift but further reduces the already limited observations. EnvSched predicts value before selection and uses long/short-window comparisons and exposure discounting to deprioritize tasks whose value is declining. Without warm-up, the exploitation hit rate rises from 57.2% at step 1 to 92.1% at step 7, with the fifth percentile across 100 runs reaching 84.4% (Table 4).
Coverage is itself a training objective, not merely a cost of exploration. In bandits, exploration improves value estimation and should decrease over time. In Agent RL, diverse environments, tools, and verification procedures directly benefit generalization. In Table 3, exploitation-only maximizes reward variance, the signal pursued by the bandit, but covers fewer environments than Random and underperforms full EnvSched on all three benchmarks.
To test these observations, we followed your suggestion and added a UCB baseline on Qwen3-4B for 200 steps. Each task is an arm; each pull returns the variance of eight rollout rewards, EnvSched's same target quantity, divided by its upper bound of 0.25 to normalize it to [0, 1]. The empirical payoff μ^i averages all historical group payoffs for that task. We use the standard UCB1 score:
si=μ^i+cni2lnN
Here, ni counts selections of task i, N counts all pulls, and c=0.05. Tasks never selected receive a score of +∞. Ties use random keys seeded by (seed, step). No other mechanisms, such as a similarity graph or coverage quotas, are used.
EnvSched outperforms UCB on all three benchmarks. Nonstationary, contextual, or graph-structured bandits are closer to our approach and are promising directions for further exploration.
Paper Update: Qwen3-30B-A3B
To test effectiveness on larger models, we ran Qwen3-30B-A3B experiments and added the results to Table 1 of the revised PDF. We invite you to review these results. With the same task pool, rollout budget, and scheduler hyperparameters, EnvSched at 200 steps achieves 37.79 / 54.75 / 70.67 on BFCL-v3 / τ-Bench / ACEBench-Agent, versus 33.88 / 51.72 / 68.00 for Random at 200 steps. After only 40 steps, EnvSched (35.00 / 53.54 / 70.00) exceeds Random at 200 steps on all three benchmarks.
Thank you again for your comments and suggestions. The additional UCB comparison makes our argument more complete. We hope these responses address your concerns. If you find that our responses resolve them, we kindly ask you to consider reassessing the paper. We would also be happy to discuss any further questions.