acceptodds
Under review as a conference paper at ICLR 2027

EnvSched: Environment-Aware Scheduling for Rollout-Efficient Agent Reinforcement Learning

48 views

Abstract

As executable environments for language agents scale, generating and verifying their rollouts becomes increasingly expensive. Under finite rollout budgets, this raises an important question: can data management at the environment level make agentic reinforcement learning (RL) more efficient by organizing, evaluating, and scheduling task instances together with the environments that determine their tools, states, transitions, and verification? To address this question, we introduce EnvSched (**Env**ironment-aware **Sched**uling), an online scheduler that learns from rollout history. EnvSched assigns scheduling value through two complementary modeling choices: exploitation prioritizes tasks likely to provide informative reward differences, while coverage spreads each batch across underobserved environments and interaction patterns. EnvSched organizes environment–task pairs as nodes in a dual-relation graph, with within-environment edges capturing task semantics and shared tool, state, and verifier structure, and cross-environment edges capturing abstract execution and reward topology. By propagating historical rollout statistics along both edge types, it predicts within-task reward variance. These predictions guide a scheduler that allocates each batch to tasks with high predicted reward variance and underobserved environments and tasks. Environment quotas and a similarity penalty further limit concentration and redundancy, yielding informative and diverse batches without additional pilot rollouts. In RL on Qwen3-8B, EnvSched achieves an average score of 48.81 across BFCL-v3, -Bench, and ACEBench-Agent after 40 training steps, exceeding the highest score attained by random task selection after 120 steps. It also improves downstream performance across model sizes at the same training budget. Statistical analyses and ablation studies highlight the complementary roles of within-task reward differentiation and cross-environment coverage in these improvements.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, large language model agents, training data management, rollout-efficient learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zdGbOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies the issue of scheduling environment tasks for agentic RL, where the way of the schedule affects how the agent behaves under finite compute budgets. To that end, the author proposes to use a graph-based data structure to model cross-task and within-task dependencies and use exploitation and coverage (exploration) through a specified reward signal on selecting the schedule of tasks. Experiment results demonstrate that the proposed approach EnvSched outperformed random baselines and ablation studies show each component helps performance.

== post rebuttal

Thank you for the thorough response and adding the UCB baseline. However, it seems that the margin between the proposed method and UCB is not as large as compared to the random baseline. For that reason, I choose to maintain my score.

Critical strengths

The paper does identify a new setting where exploration and exploitation could matter, which is environment task scheduling in agentic RL. It also combines a graph data structure with variance as the reward to select a good schedule of the tasks. The paper is easy to read and follow.

Critical weaknesses

The only baseline of the paper is a random baseline by selecting tasks uniformly at random, which is the weakest baseline possible. Although the ablation studies show that each design choice in EnvSched gives performance improvements, the baseline comparison still raises questions about whether EnvSched overall is a strong method. One stronger baseline I can think of is using bandit algorithms on each task, such as the upper-confidence bound algorithm, so that a more informed choice of tasks based on history is made.

Questions for authors

The exploration and exploitation strategy of EnvSched follows an epsilon-greedy strategy with a fixed probability for exploration. I wonder if the authors considered more advanced search heuristics such as bandit algorithms, which would probably give a greater improvement?

1 comment
AuthorsOpenReview ↗

Response to Reviewer zdGb

Thank you for your review and your positive assessment of the problem formulation and writing. Your suggestion to include a bandit baseline was helpful.

W1 and Q1: Baselines, the distinction from ε-greedy, and the UCB comparison

Baselines. In addition to Random, Table 1 and Section 3.1 of the original manuscript compare two recent online data schedulers: DPS (Mao et al., 2026) and LEEPS (Liang et al., 2026). LEEPS itself is an explore–exploit portfolio strategy based on historical outcomes. For Qwen3-4B at 200 steps, EnvSched exceeds the stronger DPS/LEEPS score on each benchmark by 1.50 / 2.22 / 1.33 points. We will make these comparisons more prominent in the revision.

ε-greedy. EnvSched's exploration does not select tasks randomly. Coverage prioritizes environments not yet covered in the batch and with lower cumulative exposure, then less-observed tasks within them, and finally ranks by predicted value and within-batch redundancy (Section 2.4). This increases environments covered per batch from 36.53 under Random to 45.96 (Table 2).

Bandits. Thank you for this suggestion. It is a natural approach that we also considered during design. We did not directly adopt a per-task bandit mainly because of three characteristics:

  1. Task value can be estimated before the first attempt. Per-task UCB requires repeated pulls to learn an arm's value. Yet each task is selected only about five times on average over 200 steps, making per-task convergence difficult. In agent tasks, reward differentiation depends largely on structure, such as a key lookup that sometimes succeeds and sometimes fails. Similar tasks share this structure.

  2. Task value changes during training and often declines precisely because the task has been trained on. Pulling an arm leaves its reward distribution unchanged in a bandit, while training teaches the policy to solve a task, reducing its informativeness. Much of a task's value may be consumed before repeated pulls establish it. Sliding-window UCB can handle drift but further reduces the already limited observations. EnvSched predicts value before selection and uses long/short-window comparisons and exposure discounting to deprioritize tasks whose value is declining. Without warm-up, the exploitation hit rate rises from 57.2% at step 1 to 92.1% at step 7, with the fifth percentile across 100 runs reaching 84.4% (Table 4).

  3. Coverage is itself a training objective, not merely a cost of exploration. In bandits, exploration improves value estimation and should decrease over time. In Agent RL, diverse environments, tools, and verification procedures directly benefit generalization. In Table 3, exploitation-only maximizes reward variance, the signal pursued by the bandit, but covers fewer environments than Random and underperforms full EnvSched on all three benchmarks.

To test these observations, we followed your suggestion and added a UCB baseline on Qwen3-4B for 200 steps. Each task is an arm; each pull returns the variance of eight rollout rewards, EnvSched's same target quantity, divided by its upper bound of 0.25 to normalize it to [0, 1]. The empirical payoff μ^i\hat{\mu}_i averages all historical group payoffs for that task. We use the standard UCB1 score:

si=μ^i+c2ln⁡Nnis_i = \hat{\mu}_i + c\sqrt{\frac{2\ln N}{n_i}}

Here, nin_i counts selections of task ii, NN counts all pulls, and c=0.05c = 0.05. Tasks never selected receive a score of +∞+\infty. Ties use random keys seeded by (seed, step). No other mechanisms, such as a similarity graph or coverage quotas, are used.

RL MethodBFCL-v3 Overallτ-Bench OverallACEBench-Agent Overall
Random28.8337.7856.00
UCB-based scheduler30.4238.7957.33
EnvSched31.8841.0158.00

EnvSched outperforms UCB on all three benchmarks. Nonstationary, contextual, or graph-structured bandits are closer to our approach and are promising directions for further exploration.

Paper Update: Qwen3-30B-A3B

To test effectiveness on larger models, we ran Qwen3-30B-A3B experiments and added the results to Table 1 of the revised PDF. We invite you to review these results. With the same task pool, rollout budget, and scheduler hyperparameters, EnvSched at 200 steps achieves 37.79 / 54.75 / 70.67 on BFCL-v3 / τ-Bench / ACEBench-Agent, versus 33.88 / 51.72 / 68.00 for Random at 200 steps. After only 40 steps, EnvSched (35.00 / 53.54 / 70.00) exceeds Random at 200 steps on all three benchmarks.

Thank you again for your comments and suggestions. The additional UCB comparison makes our argument more complete. We hope these responses address your concerns. If you find that our responses resolve them, we kindly ask you to consider reassessing the paper. We would also be happy to discuss any further questions.

Reviewer_N5dhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes ENVSCHED, a scheduler designed to improve rollout efficiency when training tool-using language agents with reinforcement learning. The main idea is to schedule environment-task pairs rather than individual prompts, since the environment determines the available tools, underlying state, transition rules, and verifier. ENVSCHED represents environment-task pairs using a dual-relation graph. Within-environment edges connect tasks based on similarities in their descriptions, tools, states, and verifiers, while cross-environment edges link tasks with similar execution and reward structures. Historical rollout statistics are propagated over the graph to estimate each task’s reward variance. At each training step, half of the batch is allocated to underexplored environments and tasks, and the other half to tasks with high predicted variance. Environment quotas and a similarity penalty are used to avoid concentration and redundancy. The authors train Qwen3-Thinking models at three scales (1.7B, 4B, and 8B) with GRPO on 2,550 tasks from 51 EnvScaler environments. They compare ENVSCHED with random scheduling, DPS, and LEEPS on BFCL-v3 Multi-Turn, τ-Bench, and ACEBench-Agent. The evaluation also includes ablations and analyses of the graph design, initialization, scheduling cost, and individual scheduling decisions.

Critical strengths

A key strength is that ENVSCHED treats the environment-task pair, rather than the prompt alone, as the scheduling unit. This is reasonable for agentic RL because selecting a task also determines the tools, states, transition rules, and verifier seen during training. The use of neighboring tasks is also helpful for tasks with little or no rollout history, while combining recent and longer-term statistics allows the scheduler to adapt as the policy changes.

The paper provides enough implementation detail to make the scheduler reasonably reproducible. The appendices include the graph-construction prompt, propagation equations, scheduling rules, and constants. The qualitative cases are also useful because they show how individual scheduling decisions are made. The experiments cover three model sizes and include several ablations of the graph and batch-construction choices.

Critical weaknesses
  1. The current evaluation does not establish practical post-training value or a fair rollout-efficiency advantage

All experiments apply GRPO directly to the Qwen3-Thinking checkpoints using the EnvScaler training pool, without an intermediate SFT stage on these data. This setup does not cause degradation across all benchmarks: Random-200 improves BFCL-v3 and τ-Bench overall, but reduces ACEBench-Agent performance in some settings. On ACEBench-Agent, Random-200 lowers Qwen3-1.7B from a Base score of 30.67 to 28.67 and lowers Qwen3-8B from 58.67 to 55.33. For Qwen3-8B, ENVSCHED-40 reaches 58.00 and ENVSCHED-200 reaches 58.67. The gains of +2.67 and +3.34 over Random on this benchmark therefore mainly recover the performance of the corresponding pre-RL checkpoint, reported as “Base” in Table 1, rather than improve beyond it. The concern is thus benchmark-specific negative transfer, not a general failure of direct RL. The training regime also limits practical relevance. The EnvScaler source paper reports that direct-RL gains are more limited than SFT+RL and states that SFT initialization remains important for policy quality. A scheduler may behave differently after SFT because task success rates, within-task variance, exploration needs, and the distribution of learnable failures have changed. Without an SFT-to-RL comparison, it is unclear whether ENVSCHED is useful for the stronger post-training regime or mainly helps during early exploration from a weaker base policy. The rollout-efficiency comparison is also incomplete. Table 1 reports a 40-step row only for ENVSCHED. Random, DPS, and LEEPS are reported only at 200 steps. Figure 1 provides a Random curve only for Qwen3-8B, obtained by averaging scores across the three benchmarks, without per-benchmark curves or uncertainty. ENVSCHED's performance is itself non-monotonic: for Qwen3-8B, τ-Bench decreases from 50.71 at 40 steps to 50.30 at 200 steps, while several 4B results do not change between the two checkpoints. The paper does not provide a checkpoint-selection rule or show that step 40 was selected independently of the test results.

  1. The central mechanism remains insufficiently validated

ENVSCHED contains many fixed design choices, including the weights used to combine the two graph relations, the history windows, the coverage–exploitation split, and several penalties and filters. Appendix D lists the values, but does not explain how they were chosen or what validation signal was used. The ablations cover the two relations, the coverage extremes, and neighborhood size, leaving several other choices untested. Table 4 provides useful evidence that the complete scheduler learns to select tasks with high realized reward variance: the exploitation hit rate rises from 57.2% at step 1 to 92.1% at step 7. However, this experiment does not separate graph propagation from the use of task history and other parts of the scheduler. A history-only or graph-free predictor under the same batch construction would make the contribution of the graph much clearer. More importantly, high reward variance is not necessarily the same as high training value. In Table 3, the exploitation-only variant has the highest mean variance (0.0442) but performs worse than the full method on all three benchmarks. This is consistent with the authors’ claim that variance and coverage are complementary, but it does not show whether variance-based ranking helps once coverage is held fixed. A useful ablation would keep the coverage slots, quotas, filters, and rollout budget unchanged and replace only the exploitation score with random selection or a history-only estimate.

Questions for authors
  1. Performance after SFT initialization

The experiments apply GRPO directly to Qwen3-Thinking checkpoints using the EnvScaler training pool, and Random-200 reduces ACEBench-Agent scores for the 1.7B and 8B models. Does ENVSCHED retain its advantage when all schedulers start from the same EnvScaler SFT checkpoint? Please report the change from that checkpoint under matched rollout and optimization budgets.

  1. Comparison at common checkpoints

Table 1 reports ENVSCHED at 40 steps, while Random, DPS, and LEEPS are reported only at 200 steps. How was step 40 selected? Please compare all four methods at the same checkpoints on each benchmark, without averaging across benchmarks, and report whether ENVSCHED improves over the corresponding pre-RL or SFT checkpoint.

  1. Contribution of variance ranking and graph propagation

Table 4 shows that ENVSCHED increasingly selects tasks with high realized reward variance, but Table 3 shows that exploitation-only scheduling has the highest mean variance and still underperforms the full method. Could the authors keep coverage and batch construction fixed while replacing only the exploitation score with random selection or a history-only, graph-free estimate? This would clarify both whether variance-based ranking improves training and whether graph propagation adds value beyond task history.

  1. Parameter selection and robustness

How were the main relation weights, history windows, coverage fraction, and filtering rules selected? Please describe the validation protocol and report sensitivity results for the choices that most affect task ranking. Were any of the downstream benchmark results used during tuning?

  1. Evaluation on maintained benchmark versions

The paper evaluates BFCL-v3 and the original τ-Bench tasks. Could the authors report results on maintained versions of these benchmarks, or explain why the newer tracks are not applicable to the models or interaction setting? Otherwise, the empirical claims should be limited to the evaluated benchmark versions.

7 comments
AuthorsOpenReview ↗

Response to Reviewer N5dh (1 of 3)

Thank you for your careful review. We are pleased that you recognize the motivation for using environment–task pairs as the scheduling unit, as well as the implementation details and case studies in the appendix. Your specific questions helped us add several experiments.

W1, first part, & Q1: Starting RL from an SFT checkpoint

Thank you for raising this point. Starting RL directly from a Thinking checkpoint is also a common post-training paradigm, and all experiments in the original manuscript use this setting. As you noted, however, SFT changes task success rates and variance distributions, so the scheduler may not remain equally effective. To also cover SFT→RL, we first conducted experiments on EnvScaler-Qwen3-4B-SFT under the time constraints: Random and EnvSched start from the same SFT checkpoint, with identical task pools, per-step rollout counts, and optimization configurations. We will add other model sizes in subsequent revisions.

RL MethodStepsBFCL-v3 MTτ-BenchACEBench-Agent
SFT checkpoint033.3340.2064.67
SFT + Random20037.50 (+4.17)42.83 (+2.63)70.00 (+5.33)
SFT + EnvSched4039.00 (+5.67)45.66 (+5.46)69.33 (+4.66)
SFT + EnvSched20040.83 (+7.50)46.26 (+6.06)70.67 (+6.00)

The results from the SFT checkpoint can be summarized in three points:

  1. Both methods are effective. Random and EnvSched exceed the SFT checkpoint on all three benchmarks.

  2. EnvSched performs substantially better on BFCL-v3 and τ-Bench. After only 40 steps, it already exceeds Random's 200-step results.

  3. Our method is effective on ACEBench-Agent. This benchmark contains only 50 questions, so one or two questions can amplify fluctuations. Although our 40-step result does not exceed Random at 200 steps, our method performs better at 200 steps.

Thus, EnvSched remains effective under the SFT→RL paradigm.

Regarding ACEBench-Agent, your observation is accurate:

  • Direct RL: On 8B, EnvSched matches the pre-RL model, with its benefit coming from avoiding the decline caused by Random. On 1.7B, 4B, and 30B-A3B, EnvSched exceeds the pre-RL model.
  • RL after SFT: Random no longer degrades performance, and both methods exceed the checkpoint by more than four points. This indicates that the decline on ACEBench-Agent reflects negative transfer under direct RL, consistent with your assessment. We will revise the corresponding wording accordingly.

W1, second part, & Q2: Why 40 steps, and comparisons at matched checkpoints

Thank you for pointing this out. We saved checkpoints every 40 steps, so step 40 was the first saved checkpoint and was not selected based on test results. The original manuscript did not explain this, and we will add the explanation in the revision.

We additionally evaluated all checkpoints at steps 40 / 80 / 120 / 160 / 200 for the four methods on Qwen3-4B, reporting each benchmark separately:

RL MethodStepsBFCL-v3 Overallτ-Bench OverallACEBench-Agent Overall
Random4027.8837.5856.00
Random8028.8838.3854.00
Random12028.4636.5756.00
Random16028.7937.3759.33
Random20028.8337.7856.00
DPS4029.4636.7757.33
DPS8029.2137.3753.33
DPS12029.6336.1654.67
DPS16030.5836.5756.67
DPS20030.3836.7756.67
LEEPS4029.3338.1854.00
LEEPS8029.8839.6058.00
LEEPS12029.3337.7854.67
LEEPS16030.7937.9858.67
LEEPS20030.0038.7956.67
EnvSched4030.1741.0158.00
EnvSched8030.9242.0257.33
EnvSched12032.5841.6256.67
EnvSched16031.3340.6156.67
EnvSched20031.8841.0158.00

On BFCL-v3 and τ-Bench, EnvSched ranks first at all five checkpoints. On τ-Bench, even its lowest checkpoint score exceeds the best score achieved by any baseline at any checkpoint.

ACEBench-Agent fluctuates more across checkpoints. EnvSched ranks first at steps 40, 120, and 200, falls slightly below LEEPS at step 80, and falls below Random and LEEPS while tying DPS at step 160. Its average across the five checkpoints is still the highest.

Reporting step 40 does not inflate EnvSched's results: it is the lowest-scoring EnvSched checkpoint on BFCL-v3, while step 120 is the highest. On τ-Bench and ACEBench-Agent, the step-40 and step-200 results are identical. EnvSched also has the highest average across all five checkpoints on every benchmark.

The pre-RL Qwen3-4B scores are 25.38 / 36.36 / 53.33. Every EnvSched checkpoint exceeds this starting point on all three benchmarks. Most gains emerge within the first 40 steps. τ-Bench and ACEBench-Agent then remain largely stable, fluctuating by no more than 1.5 points, while BFCL-v3 continues improving through step 120 before stabilizing.

AuthorsOpenReview ↗

Response to Reviewer N5dh (2 of 3)

W2, first part, & Q4: Hyperparameter selection and sensitivity

Most hyperparameters were chosen heuristically at moderate values. All model sizes use the same configuration, without model-specific adjustments, and downstream benchmark results were not used to choose these values.

To examine the impact of these choices, we varied one parameter at a time on Qwen3-1.7B while keeping all other settings unchanged, and compared BFCL-v3 Overall after 40 training steps:

HyperparameterSettingBFCL-v3 Overall
DefaultDefault configuration in Appendix D (EnvSched-40 in Table 1)11.75
Relation fusion weights (within : cross), default 0.5 : 0.50.75 : 0.25 (favoring within-environment relations)11.50
0.25 : 0.75 (favoring cross-environment relations)11.67
History half-lives (h_long / h_short), default 8 / 2 steps4 / 1 steps (both halved)11.50
16 / 4 steps (both doubled)12.04
Coverage fraction ρ, default 0.5ρ = 0.25 (16 coverage + 48 exploitation)11.50
ρ = 0.75 (48 coverage + 16 exploitation)11.25
Recent-decline coefficient η, default 0.5η = 0 (no correction; long-window estimate only)12.25
η = 1 (full correction; minimum of the long- and short-window estimates)11.42
Similarity penalty λₛᵢₘ, default 0.08λₛᵢₘ = 0.04 (halved)11.46
λₛᵢₘ = 0.16 (doubled)11.25
Exposure discount scale c₀, default 16c₀ = 8 (stronger discount)11.33
c₀ = 32 (weaker discount)10.96
Environment/task caps (C₁ / C₃ / Cₜₐₛₖ), default 4 / 11 / 22 / 5 / 1 (tighter)11.33
8 / 23 / 4 (looser)11.79
Progress filter, enabled by default (triggered at ≥3 scorable transitions)Disabled12.00
Earlier activation (≥2 scorable transitions)11.50
Random scheduling (reference)Uniform random task selection; 40 training steps10.38

Across these settings, BFCL-v3 varies by no more than 1.29 points. Together with the full 200-step experiments in Table 3(b), where halving or doubling the neighbor counts still outperforms Random, these results indicate that the method is not sensitive to these parameters.

W2, second part, and Q3: Fixing coverage and changing only exploitation scoring

This comparison separates variance-based ranking from graph propagation. We compare three Qwen3-4B settings with identical coverage slots and their internal ranking rules, environment quotas, candidate filtering, and batch construction. Only the selection rule for the 32 exploitation slots changes; the other 32 slots remain allocated to coverage:

  • EnvSched: the complete score combining the dual-relation graph and each task's own history;
  • Coverage + history-only: current variance is estimated only from the task's own historical reward variance, without neighbors;
  • Coverage + random: random selection from the same candidate set.
RL MethodMean reward varianceEnvironments covered per batchBFCL-v3 MT Overall
EnvSched0.0429045.9631.88
Coverage + history-only exploitation0.0394545.2530.42
Coverage + random exploitation0.0337746.1129.71

Coverage differs by less than one environment per batch, with random exploitation slightly higher. The differences therefore come from which tasks exploitation selects, rather than coverage. Both realized variance and BFCL-v3 follow the same order: EnvSched > history-only > random exploitation.

First, variance-based ranking improves training when coverage is fixed. Both history-only scoring and full EnvSched improve realized variance and BFCL-v3 over random exploitation. This also explains Table 3: exploitation-only has the highest variance yet performs worse because it reduces coverage to 33.95 environments per batch. With coverage fixed, higher realized variance corresponds to better downstream performance.

Second, graph propagation adds value beyond each task's own history. With the same ranking principle, the graph-based estimator further improves both realized variance and BFCL-v3, accounting for about two-thirds of the BFCL-v3 gain from exploitation scoring. A task's own history is sparse and prone to becoming outdated: each task is selected only about five times on average over 200 steps. A previously high-variance task may already have been learned by its next selection, and many early candidates have no history at all. Neighbors supply more extensive, more recent evidence, bringing estimates closer to the variance under the current policy. Selecting high-variance tasks is effective; graph propagation makes that selection more accurate.

AuthorsOpenReview ↗

Response to Reviewer N5dh (3 of 3)

Q5: Results on updated benchmark versions

Thank you for this suggestion. We additionally evaluated the 200-step Qwen3-4B checkpoints from Random and EnvSched on BFCL-v4 (Multi-Turn, Web Search, and Memory) and τ²-Bench (Retail, Airline, and Telecom). BFCL-v4 Multi-Turn and τ²-Bench Retail/Airline contain the same types of tasks as our original experiments, with corrections made by the benchmark maintainers. BFCL-v4 Multi-Turn retains the same scenario composition and evaluation procedure as the original version: 800 instances across four subcategories, with the same state/response checks. The maintainers corrected some task descriptions, initial states, reference answers, and backend logic. τ²-Bench retains the original τ-Bench Retail/Airline domains and tasks, with reviewed task instructions and evaluation criteria. These three evaluations therefore test whether the original conclusions continue to hold on the corrected versions. Web Search, Memory, and Telecom add new capability dimensions, testing performance on capabilities not covered by the original experiments.

RL MethodBFCL-v4 MTBFCL-v4 WSBFCL-v4 Memτ² Retailτ² Airlineτ² Telecom
Random36.257.523.1545.0333.3316.67
EnvSched39.049.023.1549.1238.6721.64

EnvSched outperforms Random on five of the six evaluations and ties on Memory. Its gains on BFCL-v4 Multi-Turn and τ² Retail/Airline are broadly consistent with the original versions, while Web Search and Telecom also improve.

Paper Update

To examine whether the method remains effective for larger models, we additionally ran experiments on Qwen3-30B-A3B and included the results in Table 1 of the revised PDF. We invite you to review these results. With the task pool, rollout budget, and scheduler hyperparameters unchanged, EnvSched at 200 steps achieves 37.79 / 54.75 / 70.67 on BFCL-v3 / τ-Bench / ACEBench-Agent, compared with 33.88 / 51.72 / 68.00 for Random at 200 steps. After only 40 training steps, EnvSched (35.00 / 53.54 / 70.00) already exceeds Random at 200 steps on all three benchmarks.

Thank you again for your careful review. The comparisons you suggested have been valuable in strengthening the paper. We hope these responses address your concerns. If you find that the additional experiments resolve them, we kindly ask you to consider reassessing the paper. We would also be happy to discuss any further questions.

Reviewer_PgnjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies task scheduling for Agent RL under a limited rollout budget. EnvSched treats environment–task pairs as the scheduling unit. Before training, it constructs within-environment and cross-environment similarity relations using information about tasks, tools, states, and verifiers. During training, it combines historical reward statistics from each task and its neighbors to estimate within-task reward variance, and selects the next batch by combining environment coverage, exposure discounting, and within-batch redundancy reduction. The authors train Qwen3 models at three scales on an EnvScaler training pool containing 51 environments and 2,550 tasks, and evaluate the method using three tool-use benchmarks, scheduling baselines, and ablation studies.

Critical strengths
  1. The research problem has clear practical value, and the environment-aware modeling is well suited to Agent RL: Agent rollouts involve multi-turn generation, tool execution, and environment-based verification, making task selection directly relevant to resource allocation under a limited budget. Incorporating tools, states, dependencies, and verifiers into scheduling provides a useful additional perspective on the value of training data. The method reuses existing training trajectories without sampling additional pilot trajectories, giving it potential practical value for Agent RL training.
  2. The main experiments and ablations provide reasonably consistent support for the overall scheduling gains: With matched per-step rollout counts, EnvSched outperforms random scheduling on the Overall scores of all three benchmarks at all three model scales.
Critical weaknesses
  1. The authors already extract structure and fix the similarities before training, and Table 3 includes ablations retaining only within-environment edges or only cross-environment edges, as well as ablations of the coverage ratio. These experiments support the complementarity of the selected components. However, the method is not compared with a simple predictor based on “offline structural categorization + online historical reward statistics” while keeping the same coverage, quotas, and filtering rules. It therefore remains difficult to determine how much additional benefit fine-grained dual-relation propagation provides over simpler statistics aggregated by environment or operation category.
  2. The connection between high reward variance and the current group-relative update is clear. However, Table 2 and Figure 4 primarily report aggregate statistics for selected tasks, without directly tracking how much subsequent training exposure and reward improvement are obtained by tasks whose early rollouts mostly failed and exhibited little within-task reward variation. I understand that coverage preserves opportunities to revisit these tasks, and that the progress filter only temporarily excludes them from the exploitation candidate set. Further evidence is needed to show whether these revisits translate into reward improvements during training. A stratified analysis of these tasks would help determine whether the overall budget-efficiency gains also benefit tasks with initially low rewards and little reward variation.
Questions for authors
  1. With the same coverage, environment quotas, candidate filtering, and rollout budget, how does replacing the dual-relation neighbor estimator with online historical statistics aggregated by environment or static operation category affect performance? If a full training comparison is unavailable, could the authors compare the two estimators’ ranking quality against subsequently observed within-task reward variances?
  2. Could the authors report the standard deviations of the main results across the three random seeds and clarify whether the gains over the strongest baseline are consistent across seeds?
  3. Under the same rollout budget, how do the methods differ in the total number of generated tokens and end-to-end training time?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Pgnj (1 of 3)

Thank you for your positive assessment of our work, particularly your recognition that environment-aware modeling is well suited to Agentic RL and that our method reuses existing trajectories without additional pilot rollouts. Your questions have helped us improve our experimental analysis.

W1 & Q1: What does the dual-relation graph add over environment-mean estimation?

We implemented a simpler estimator that replaces dual-relation neighbor estimation with the environment mean: the historical reward statistics of observed tasks within the same environment are averaged to estimate the tasks in that environment.

Fusion with each task's own history, the coverage component, environment quotas, candidate filtering, and the rollout budget remain the same as in EnvSched; only the estimator is replaced. The results for Qwen3-4B after 200 training steps are:

RL MethodStepsBFCL-v3 Overallτ-Bench OverallACEBench-Agent Overall
Random20028.8337.7856.00
Environment mean20030.0838.3856.67
EnvSched20031.8841.0158.00
RL MethodMean reward varianceMedian reward varianceEnvironments covered per batch
Random0.032080.0146836.53
Environment mean0.037950.0179745.12
EnvSched0.042900.0210845.96

Environment-mean estimation yields both lower downstream scores and lower realized variance among selected tasks than EnvSched. We attribute this to two factors.

First, tasks within the same environment can have substantially different variances, and averaging obscures these differences. In Appendix F.1, the six neighbors within the same environment have variances ranging from 0 to 0.186, whereas EnvSched weights each neighbor individually according to similarity.

Second, the environment mean uses only evidence from the same environment and cannot use structurally similar tasks in other environments. Appendix F.2 illustrates this case: the target task's within-environment neighbors all have near-zero variance, so the environment mean would directly classify it as low-value. The different signal comes from tasks in other domains with the same execution structure. The target task's realized variance is 0.066, above the batch mean of 0.038 at that step.

This proves that dual-relation neighbor estimation is necessary.

AuthorsOpenReview ↗

Response to Reviewer Pgnj (2 of 3)

W2: Are initially low-reward, low-variance tasks revisited later in training?

These tasks have three routes back into training:

  1. The coverage phase prioritizes environments not yet covered in the current batch, then environments with lower cumulative exposure, and then tasks with fewer observations. Tasks observed only once or twice early in training therefore continue to be revisited.

  2. The progress filter applies only to exploitation, and a task must have at least three scorable transitions before it can be filtered (Appendix C). Tasks observed only once early in training are therefore not excluded by this filter.

  3. A task's predicted variance also depends on its neighbors. As the policy improves on similar tasks, the neighbors' statistics change, updating the target task's score and giving it an opportunity to reenter exploitation.

To verify this directly, we analyzed the Qwen3-4B training process. We identified the tasks you described: tasks whose early rollouts mostly failed and whose within-group rewards showed little variation. Specifically, these tasks were selected within the first 10 steps and had a mean reward of ≤ 0.2 and a within-group variance of ≤ 0.01 across their eight rollouts at that time. This yielded 59 tasks for Random and 44 for EnvSched. We tracked how often they were selected again later in training, their within-group reward variance when revisited, and changes in their rewards.

Of EnvSched's 64 task slots per step, 32 coverage slots prioritize environments and tasks that have received less training, largely independently of their current variance; the other 32 exploitation slots go to tasks with the highest predicted variance (Section 2.4). Because the two phases use different selection criteria, we report them separately. The table below shows the mean number of revisits per task, with the within-group reward variance upon revisiting in parentheses:

Training stepsRandomEnvSched (total)Of which: coverageOf which: exploitation
Steps 11–400.63 (0.024)0.20 (0.018)00.20 (0.018)
Steps 41–2003.63 (0.029)4.52 (0.049)1.73 (0.023)2.80 (0.065)
Entire period4.25 (0.028)4.73 (0.048)1.73 (0.023)3.00 (0.062)

The changes in these tasks' rewards are:

RL MethodMean reward when selected in the first 10 stepsMean reward at the last selectionFraction of tasks with reward ≥ 0.5 at the last selection
Random≈ 0.080.3732%
EnvSched≈ 0.080.3830%

EnvSched does not abandon these tasks. They receive at least as many revisits as under Random, and every task is revisited at least once. A single low-variance observation therefore does not cause the scheduler to discard a task.

The difference lies in when the tasks are revisited. During the first 40 steps, their variance upon revisiting is well below the mean across all training groups over the same period: 0.040 for Random and 0.047 for EnvSched. Most of these tasks are still difficult to learn from, and EnvSched selects them less often. During steps 41–200, exploitation revisits become more frequent, with much higher within-group variance than under Random. EnvSched thus defers training while these tasks offer little learning signal and concentrates on them once they become informative.

Deferring their training does not leave them behind. Both methods improve their mean reward, and similar proportions of tasks reach rewards of at least 0.5.

The same conclusions hold for tasks appearing in the first four steps: 31 for Random and 22 for EnvSched. All 22 EnvSched tasks were revisited, averaging 4.68 revisits per task versus 4.48 for Random. Their variance when revisited through exploitation remains much higher (0.086 versus 0.030), and both methods improve their mean reward from below 0.1 to above 0.35.

For reference, the downstream performance of the two methods at step 200 is:

RL MethodBFCL-v3 Overallτ-Bench OverallACEBench-Agent Overall
Random28.8337.7856
EnvSched31.8841.0158

Taken together, EnvSched's overall gains do not come at the expense of these initially difficult tasks.

Q2: Standard deviations across three seeds

Thank you for the reminder. Table 1 in the original manuscript reports means over three seeds; standard deviations were omitted for space. Appendix G of the revised manuscript includes the complete Table 1 with standard deviations. EnvSched's improvement over the strongest baseline holds for 3 seeds.

AuthorsOpenReview ↗

Response to Reviewer Pgnj (3 of 3)

Q3: Generated tokens and end-to-end training time

Thank you for this suggestion. The Qwen3-4B results under the same eight-GPU configuration are:

RL MethodStepsTotal generated tokensTraining time (h)GPU-hours
Random200693 million83.4667
EnvSched200613 million78.4627
EnvSched40133 million19.6157

At 200 steps, EnvSched uses fewer generated tokens and less training time. Scheduling, including graph propagation, takes 34.93 seconds per step, approximately 2.3% of the step time.

We focus on the total cost of reaching the same performance level. EnvSched at 40 steps already exceeds Random at 200 steps on all three benchmarks (30.17 / 41.01 / 58.00 versus 28.83 / 37.78 / 56.00). It uses 20,480 trajectories instead of 102,400, reducing training time by 76.5% and generated tokens by 81%.

Paper Update: Qwen3-30B-A3B

To examine whether the method remains effective for larger models, we additionally ran experiments on Qwen3-30B-A3B and included the results in Table 1 of the revised PDF. We invite you to review these results. With the task pool, rollout budget, and scheduler hyperparameters unchanged, EnvSched at 200 steps achieves 37.79 / 54.75 / 70.67 on BFCL-v3 / τ-Bench / ACEBench-Agent, compared with 33.88 / 51.72 / 68.00 for Random at 200 steps. After only 40 training steps, EnvSched (35.00 / 53.54 / 70.00) already exceeds Random at 200 steps on all three benchmarks.

Thank you again for your comments. These additional experiments strengthen the paper's argument. We hope the responses address your questions and welcome further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.