- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SAGE, which generates supervision from local interactions with the environment. At student-visited states, the current student explores alternative actions, and an EMA teacher uses the resulting observations to make better-informed decisions. These decisions are distilled into a student that acts without lookahead. Experiments on ALFWorld and WebShop demonstrate improvement without task rewards, expert demonstrations, or an external stronger teacher.
Critical strengths
-
A useful source of supervision. The method uses observed action consequences to guide learning, reducing dependence on rewards and external teachers. Transferring this additional interaction information into a zero-lookahead student is practically valuable.
-
Strong evidence of self-improvement. On ALFWorld Unseen, the student improves from 28.4% to 90.3%, exceeding the initial higher-budget policy’s 76.9%. Ablations of lookahead budget, EMA updates, and branch generation, together with qualitative examples, provide support for the design.
Critical weaknesses
-
The environment-access assumptions limit applicability. Constructing alternative branches requires restoring the same original state. This is feasible in the evaluated benchmarks but may be difficult for irreversible actions or dynamic environments. The motivation emphasizes open-ended tasks without reliable supervision, while the experiments cover only two resettable environments.
-
The comparisons and experimental scope could be strengthened. Several baseline results are taken from prior papers, and direct comparisons under matched interaction budgets are missing. These would help separate the benefits of additional environment information from those of iterative teacher improvement. Experiments also use only a 4B model, leaving scalability to larger models unclear.
Questions for authors
-
I find this work valuable. The paper states that the code and models will be released; could the authors provide the implementation, training configurations, and model checkpoints during rebuttal to support verification and reproduction?
-
Could the authors add controlled comparisons with recent related methods and evaluate larger models and additional tasks? These results would help establish the method’s scalability and broader applicability.
If the authors convincingly address my concerns, I would be happy to raise my rating.
3 comments
Thank you for the detailed rebuttal. The additional experiments strengthen the paper, but they do not provide sufficient evidence for me to change my recommendation.
We thank the reviewer for acknowledging that our additional experiments strengthen the paper and for carefully reconsidering our work. We would like to share one more experimental result. Among the RL baselines we compared, GRPO has the lowest environment interaction cost. We therefore chose GRPO as the baseline for evaluating interaction efficiency.
To control the interaction budget, we reduced the number of sampled tasks per step in SAGE from 128 to 32, and the number of states used for lookahead analysis and distillation from 512 to 64. The results are shown below:
| Method | ALFWorld Seen | Interactions | Reward-Free |
|---|---|---|---|
| GRPO | 70.6 | 266,521 | No |
| SAGE | 85.1 | 250,087 | Yes |
Notably, SAGE achieves a 14.5 percentage-point higher success rate on ALFWorld seen while using 6.2% fewer environment interactions than GRPO. Unlike GRPO, SAGE also does not require task-level reward signals during training. This result further shows that SAGE's performance gains do not simply come from more environment interactions.
We sincerely appreciate the reviewer's earlier feedback. It encouraged us to further investigate the interaction efficiency of SAGE and led to this positive result. We would be grateful if you could reconsider your score in light of this new result and the additional evidence provided earlier.
We would appreciate it if you could let us know which aspects of our work still need improvement. We would be happy to address any remaining concerns.
Rebuttal for Reviewer 3gP5
We sincerely thank the reviewer 3gP5 for the positive assessment of our work and for recognizing the value of SAGE in learning from environment interactions without external supervision. We also appreciate the constructive comments and suggestions. We address each point below.
We thank the reviewer for pointing out this limitation. First, we would like to clarify an important distinction: SAGE does not require the environment to be resettable, but only forkable. Specifically, we do not need to undo executed actions or roll back the environment along its original trajectory. Instead, we only need to create independent copies of the current decision state, allowing different candidate actions to be executed from the same state. This substantially reduces the requirements for action reversibility.
This assumption applies to a broad range of environments. Many recent agent benchmarks, including Terminal-Bench, DeepSWE, and OSWorld used in the GPT-6 evaluation, as well as WebArena, can support branch execution through sandbox isolation or environment cloning. Such mechanisms can be implemented through state snapshots and restoration, deterministic trajectory replay , or isolated simulation. Importantly, this requirement only applies to supervision generation during training. After distillation, the Student does not need lookahead branches during execution.
We also thank the reviewer for raising this valuable point. For tasks that cannot be reproduced in a simulated environment and require real online interactions, constructing branches from identical states may indeed be difficult. Inspired by this suggestion, we see possible ways to extend SAGE. For example, we could retrieve similar states and their environment feedback from historical interactions, or use a World Model to simulate the outcomes of different candidate actions. These approaches could approximate real environment branches. Although they require further validation, we believe they offer promising directions for extending SAGE to more open and complex real-world tasks.
1. Fairness of Baseline Comparisons.
SAGE requires no external rewards, expert demonstrations, or additional strong teachers, while existing training methods often rely on different forms of supervision. For example, IWM and WKM rely on expert trajectories, ITP-I requires an additionally trained world model, ITP-R also relies on reward optimization, CoMAP uses expert trajectories for initialization, and OPD methods use Qwen3-30B-A3B as an external teacher.
These methods differ substantially in their supervision requirements and environment interactions. Forcing identical training budgets or removing additional supervision would change the original methods and make the comparison unfair to the baselines.
To ensure a fair comparison as much as possible, we follow the evaluation settings of published baselines, using the same data splits, environment observations, admissible action interfaces, execution step limits, evaluation metrics, and policy backbone. We use the results reported in published papers to preserve their original training settings and avoid performance differences caused by reimplementation.
2. Model Scalability and Task Generalization.
We conducted additional experiments with Qwen3-8B on ALFWorld, using the same 4×2 to 0×0 lookahead distillation setting as Qwen3-4B.
On ALFWorld Unseen, Qwen3-8B achieves a success rate of 83.5%, outperforming Qwen3-4B by 8.9 percentage points.
We also added experiments on Minesweeper, a task different from ALFWorld and WebShop. The model must infer mine locations from gradually revealed numerical clues and make sequential decisions. We used Qwen3-4B with the 4×2 to 0×0 lookahead distillation setting.
SAGE achieves a success rate of 57.81%, outperforming the strongest baseline, DAPO, by 11.81 percentage points.
We have provided the complete training and evaluation code for SAGE on ALFWorld in the Supplementary Material for reviewers to examine and verify. Due to the large size of the trained model checkpoints, it is difficult to release them anonymously during the current double-blind review process. Therefore, we have not provided the model weights at this stage. We plan to release the corresponding model checkpoints and trained weights through an official GitHub repository when the final version of the paper is published.