- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes COPE, which augments GRPO with an auxiliary action-forecasting objective. Using successful on-policy trajectories, a shared policy backbone learns to predict the current action and several future actions without the intervening observations. Experiments on five agent benchmarks show improvements over the authors’ GRPO baseline, although performance decreases on the Airline domain of τ²-bench.
Critical strengths
- COPE reuses collected trajectories without requiring expert demonstrations, a separate world model, or additional inference-time computation.
- Success rates improve over GRPO from 64.5% to 77.0% on WebShop, from 64.5% to 72.5% on ScienceWorld, and from 41.67% to 55.36% on AppWorld normal (Tables 1 and 3). Improvements with Qwen3-8B provide additional evidence beyond one backbone.
- The experiments cover household tasks, scientific experiments, shopping, customer service, and API-based coding. Improvements on AppWorld’s scenario-held-out challenge split and with Qwen3-8B provide evidence of applicability beyond a single task distribution or backbone
Critical weaknesses
-
Variation across independent training runs and confidence intervals are not reported. This limits confidence in smaller gains, particularly the 1-point improvement over ECHO on ALFWorld and the approximately 1.2-point overall improvement over GRPO on τ²-bench.
-
The training configurations are incomplete. Table 9 does not include COPE, and complete settings for τ²-bench and AppWorld are not provided. More detail on the update schedule, loss reduction, and checkpoint selection would help assess reproducibility.
-
Table 5 provides useful controls, but increasing also increases the number of supervised targets and may change the effective auxiliary-loss weight. Clarifying normalization and comparing with a comparably tuned baseline would better identify the benefit of future-action prediction.
Questions for authors
- Does COPE retain its advantage over the baseline when auxiliary supervision budgets are matched and loss weights are tuned comparably?
- How stable are the gains across independent training seeds, particularly on ALFWorld and τ²-bench? Can you provide complete COPE configurations and held-out learning curves against cumulative environment interactions or wall-clock time?
- Does the Airline regression persist when tasks requiring no substantive tool actions are evaluated separately or excluded from forecasting supervision?
2 comments
[Training configuration details] The complete configurations are provided in the revised manuscript (marked in blue in Appendix C.2). Specifically:
- CoPE settings. CoPE uses the same settings as GRPO (Table 9 in the original submission, now Table 10), with forecasting horizon and forecasting coefficient in all environments; this is now stated in the table caption.
- τ²-bench and AppWorld. Their complete settings are listed in Tables 11 and 12, shared by GRPO (ours) and CoPE.
- Update schedule. At every training step, the GRPO and forecasting losses are computed on the same rollouts collected in that step and minimized jointly as ; all other settings, including PPO epochs and mini-batch sizes, follow GRPO.
- Loss reduction. The forecasting loss is averaged over its target tokens and over the successful rollouts of each group, so its scale changes neither with nor with the number of successes.
- Checkpoint selection. For all methods, we evaluate the final checkpoint after the training steps listed in the tables.
[K=0 baseline and loss normalization] For each constructed sample of CoPE, the loss is normalized on the token scale. Therefore, the effective auxiliary loss weight is the same across different horizon length. The same clarification is added in our revised version.
As stated in Sec. 8 (Q2), "augmenting GRPO with a success-filtered SFT loss is therefore approximately equivalent to CoPE with ", and Table 6 confirms this: without an invalid-action filter, (66.5) performs on par with GRPO + success-filtered SFT (65.5), and both stay close to GRPO (64.5). narrows the gap only after we add an invalid-action filter (69.5; Fig. 5), a task-specific heuristic that requires knowing which actions are invalid, and even then it remains below CoPE (72.5), which needs no such filtering. Further tuning the baseline with such heuristics would therefore not be a like-for-like comparison, and the remaining gap reflects the benefit of future-action prediction. If the reviewer considers it necessary, we are happy to add a baseline with a separately tuned .
[Analysis of the Airline regression on τ²-bench] We traced the drop to Airline’s task design. 38% of Airline tasks (19/50; 13 of the 30 training tasks) must be refused and leave the database unchanged, compared with only 4.4% in Retail. We therefore evaluated the Airline test set by task type (6 refusal and 14 action tasks, 4 trials each):
| Method | Refusal tasks | Action tasks |
|---|---|---|
| GRPO | 18/24 | 3/56 |
| CoPE | 13/24 | 0/56 |
Both methods rarely solve action tasks, so the gap comes mainly from refusal tasks. This reveals a limitation of action forecasting. In a successful refusal, the agent looks up the reservation, checks the flight, finds that the policy forbids the request, and only then declines. Most forecasting targets in such a trajectory are lookup steps, while the single declining step carries little weight in the loss. As a result, forecasting teaches the lookup sequence rather than restraint and, together with the action-dominated trajectories, reinforces an “investigate, then act” pattern: on refusal tasks, the agent reliably reaches the point of acting, and then acts.
We have not yet run the variant that excludes refusal tasks from forecasting supervision. Given the explanation above, we expect it to help only partially, since the same pattern is still learned from action tasks; supervising explicit refusals is a more direct remedy, which we leave to future work. We revised Sec. 8 (Q3) accordingly and added the per-type evaluation and a case study with look-alike task pairs to App. C.5 (Table 13).
We sincerely thank the reviewer for the careful reading and the constructive questions. We respond to each weakness and question below. We have highlighted all revisions in blue in the updated manuscript, and we would be glad to address any remaining concerns during the discussion period.
[Statistical reliability] The results in Tables 1–3 of the original paper are from a single run. We have now repeated the main experiments with independent seeds and report mean ± std and 95% confidence intervals (Student’s t) below. Due to limited time and API budget, the additional runs cover ALFWorld, ScienceWorld, and the retail and telecom domains of τ²-bench (40 training steps).
CoPE has the highest mean on every benchmark. On ScienceWorld, the gain over GRPO is clearly separated (CoPE’s lower bound of 68.72 exceeds GRPO’s upper bound of 66.23). On ALFWorld, CoPE outperforms ECHO by 2.3 points on average with lower variance (95.5 ± 0.8 vs. 93.3 ± 1.7). The gap is necessarily small, as ALFWorld is near saturation: GPT-5.5 already reaches 93.5% without training, and CoPE fails only 9 of the 200 test tasks. On τ²-bench the intervals are wide and overlap because of the small number of runs, so we regard these gains (+11.1 on retail, +10.8 on telecom) as consistent in direction but not yet statistically conclusive. Airline is analyzed separately below.
[Sample efficiency under matched budgets] We added Table 4 (Lines 378–396) in the revised version, which reports held-out test success on ScienceWorld at matched cumulative environment interactions, generated tokens, and GPU (wall-clock) time, averaged over two seeds. CoPE leads GRPO throughout training under all three budgets: by +13 to +24 points at matched environment interactions, +11 to +30 at matched generated tokens, and +7 to +17 at matched GPU hours. Under matched training tokens or FLOPs, CoPE leads only in early training, since the forecasting pass trains on every turn of each successful trajectory; this pass accounts for about 70% of CoPE’s FLOPs but only about 40% of its GPU time.
[Analysis on forecasting horizon] We present the test performance of different horizons K on SciWorld. Among different horizons, performance peaks at an intermediate value, with K=2 achieving the highest score. This reflects a trade-off between foresight and target reliability: too short a horizon provides little multi-step supervision and reaches a slightly lower final score, whereas too long a horizon adds distant targets that are increasingly unpredictable from the current history, which slows early learning without improving the final score. The analysis is also added to the ablation study in the revised manuscript. We apply this setting across all environments for simplicity.