acceptodds
Under review as a conference paper at ICLR 2027

Self-Generated Supervision for Self-Evolving Agents

Abstract

Long-horizon interactive agents often rely on task rewards, expert trajectories, or stronger teacher models to improve their performance. However, many complex, open-ended tasks lack verifiable outcomes and reliable external supervision, making it difficult for agents to improve through these approaches. Yet agents can actively interact with their environments, and the feedback from these interactions provides a natural source of learning signals. Building on this observation, we propose a simple framework for online agent self-improvement through on-policy self-distillation. At selected student-visited states, the current student explores alternative actions under a larger interaction budget, while an EMA teacher uses the resulting environment feedback to make a better-informed decision that is distilled into the zero-lookahead student. Our method generates supervision at individual decision steps, without relying on task rewards, explicit value estimates, or the outcomes of complete trajectories. As the student policy is updated, it generates new higher-budget teacher supervision, allowing both the student and teacher to improve over time. Experiments on ALFWorld show that, without using task rewards or expert supervision during training, the student's task success rate increases from 28.4% to 90.3%, while that of the higher-budget teacher increases from 76.9% to 96.3%. The trained student also surpasses the initial higher-budget teacher, demonstrating that agents can achieve sustained self-improvement through their own interactions with the environment. Our code and models will be publicly released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-Evolving Agents, On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3gP5OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SAGE, which generates supervision from local interactions with the environment. At student-visited states, the current student explores alternative actions, and an EMA teacher uses the resulting observations to make better-informed decisions. These decisions are distilled into a student that acts without lookahead. Experiments on ALFWorld and WebShop demonstrate improvement without task rewards, expert demonstrations, or an external stronger teacher.

Critical strengths
  1. A useful source of supervision. The method uses observed action consequences to guide learning, reducing dependence on rewards and external teachers. Transferring this additional interaction information into a zero-lookahead student is practically valuable.

  2. Strong evidence of self-improvement. On ALFWorld Unseen, the student improves from 28.4% to 90.3%, exceeding the initial higher-budget policy’s 76.9%. Ablations of lookahead budget, EMA updates, and branch generation, together with qualitative examples, provide support for the design.

Critical weaknesses
  1. The environment-access assumptions limit applicability. Constructing alternative branches requires restoring the same original state. This is feasible in the evaluated benchmarks but may be difficult for irreversible actions or dynamic environments. The motivation emphasizes open-ended tasks without reliable supervision, while the experiments cover only two resettable environments.

  2. The comparisons and experimental scope could be strengthened. Several baseline results are taken from prior papers, and direct comparisons under matched interaction budgets are missing. These would help separate the benefits of additional environment information from those of iterative teacher improvement. Experiments also use only a 4B model, leaving scalability to larger models unclear.

Questions for authors
  1. I find this work valuable. The paper states that the code and models will be released; could the authors provide the implementation, training configurations, and model checkpoints during rebuttal to support verification and reproduction?

  2. Could the authors add controlled comparisons with recent related methods and evaluate larger models and additional tasks? These results would help establish the method’s scalability and broader applicability.

If the authors convincingly address my concerns, I would be happy to raise my rating.

3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer 3gP5

We sincerely thank the reviewer 3gP5 for the positive assessment of our work and for recognizing the value of SAGE in learning from environment interactions without external supervision. We also appreciate the constructive comments and suggestions. We address each point below.

W1: The environment-access assumptions limit applicability. [...]

We thank the reviewer for pointing out this limitation. First, we would like to clarify an important distinction: SAGE does not require the environment to be resettable, but only forkable. Specifically, we do not need to undo executed actions or roll back the environment along its original trajectory. Instead, we only need to create independent copies of the current decision state, allowing different candidate actions to be executed from the same state. This substantially reduces the requirements for action reversibility.

This assumption applies to a broad range of environments. Many recent agent benchmarks, including Terminal-Bench, DeepSWE, and OSWorld used in the GPT-6 evaluation, as well as WebArena, can support branch execution through sandbox isolation or environment cloning. Such mechanisms can be implemented through state snapshots and restoration, deterministic trajectory replay , or isolated simulation. Importantly, this requirement only applies to supervision generation during training. After distillation, the Student does not need lookahead branches during execution.

We also thank the reviewer for raising this valuable point. For tasks that cannot be reproduced in a simulated environment and require real online interactions, constructing branches from identical states may indeed be difficult. Inspired by this suggestion, we see possible ways to extend SAGE. For example, we could retrieve similar states and their environment feedback from historical interactions, or use a World Model to simulate the outcomes of different candidate actions. These approaches could approximate real environment branches. Although they require further validation, we believe they offer promising directions for extending SAGE to more open and complex real-world tasks.

W2 & Q2: The comparisons and experimental scope could be strengthened. [...]

1. Fairness of Baseline Comparisons.

SAGE requires no external rewards, expert demonstrations, or additional strong teachers, while existing training methods often rely on different forms of supervision. For example, IWM and WKM rely on expert trajectories, ITP-I requires an additionally trained world model, ITP-R also relies on reward optimization, CoMAP uses expert trajectories for initialization, and OPD methods use Qwen3-30B-A3B as an external teacher.

These methods differ substantially in their supervision requirements and environment interactions. Forcing identical training budgets or removing additional supervision would change the original methods and make the comparison unfair to the baselines.

To ensure a fair comparison as much as possible, we follow the evaluation settings of published baselines, using the same data splits, environment observations, admissible action interfaces, execution step limits, evaluation metrics, and policy backbone. We use the results reported in published papers to preserve their original training settings and avoid performance differences caused by reimplementation.

2. Model Scalability and Task Generalization.

We conducted additional experiments with Qwen3-8B on ALFWorld, using the same 4×2 to 0×0 lookahead distillation setting as Qwen3-4B.

MetricQwen3-4BQwen3-8B
Seen SR (%)70.087.4
Unseen SR (%)74.683.5

On ALFWorld Unseen, Qwen3-8B achieves a success rate of 83.5%, outperforming Qwen3-4B by 8.9 percentage points.

We also added experiments on Minesweeper, a task different from ALFWorld and WebShop. The model must infer mine locations from gradually revealed numerical clues and make sequential decisions. We used Qwen3-4B with the 4×2 to 0×0 lookahead distillation setting.

MethodReActGRPODAPOGiGPOSAGE
Success Rate (%)3.1239.146.042.257.8

SAGE achieves a success rate of 57.81%, outperforming the strongest baseline, DAPO, by 11.81 percentage points.

Q1: I find this work valuable. The paper states that the code and models will be released; [...]

We have provided the complete training and evaluation code for SAGE on ALFWorld in the Supplementary Material for reviewers to examine and verify. Due to the large size of the trained model checkpoints, it is difficult to release them anonymously during the current double-blind review process. Therefore, we have not provided the model weights at this stage. We plan to release the corresponding model checkpoints and trained weights through an official GitHub repository when the final version of the paper is published.

Thank you for the detailed rebuttal. The additional experiments strengthen the paper, but they do not provide sufficient evidence for me to change my recommendation.

We thank the reviewer for acknowledging that our additional experiments strengthen the paper and for carefully reconsidering our work. We would like to share one more experimental result. Among the RL baselines we compared, GRPO has the lowest environment interaction cost. We therefore chose GRPO as the baseline for evaluating interaction efficiency.

To control the interaction budget, we reduced the number of sampled tasks per step in SAGE from 128 to 32, and the number of states used for lookahead analysis and distillation from 512 to 64. The results are shown below:

MethodALFWorld SeenInteractionsReward-Free
GRPO70.6266,521No
SAGE85.1250,087Yes

Notably, SAGE achieves a 14.5 percentage-point higher success rate on ALFWorld seen while using 6.2% fewer environment interactions than GRPO. Unlike GRPO, SAGE also does not require task-level reward signals during training. This result further shows that SAGE's performance gains do not simply come from more environment interactions.

We sincerely appreciate the reviewer's earlier feedback. It encouraged us to further investigate the interaction efficiency of SAGE and led to this positive result. We would be grateful if you could reconsider your score in light of this new result and the additional evidence provided earlier.

We would appreciate it if you could let us know which aspects of our work still need improvement. We would be happy to address any remaining concerns.

Reviewer_rKNcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SAGE distills local environment lookahead into a zero-lookahead agent. The current student explores alternative actions from restored states, and an EMA teacher uses the resulting observations to supervise decisions at student-visited histories through forward KL. Experiments on ALFWorld and WebShop report zero-lookahead success rates of 90.3% and 61.6%, respectively, with ablations of lookahead budget and teacher updates.

Critical strengths
  1. Using short counterfactual interactions to supervise individual decisions is a useful alternative to waiting for successful trajectories. Querying the teacher at student-visited states also makes failed exploration usable for training. This is particularly appealing when outcome rewards are unavailable but the environment supports reliable restoration.

  2. The reported improvement survives removal of lookahead at deployment: the ALFWorld Unseen student reaches 90.3%, exceeding the initial high-budget policy’s 76.9%. This is a meaningful demonstration of amortizing extra training interactions into policy parameters. The WebShop result provides a second setting with substantially different decision structure.

Critical weaknesses
  1. The benchmark interface leaves the headline comparisons unresolved. Appendix B supplies exact admissible-action menus and requires copying an action verbatim, but does not explain how parameterized WebShop search queries enter that menu. Sections 3.2–3.3 import baseline results without establishing equivalence of these interfaces. Appendix A.5 also does not specify the training and evaluation task splits. Action enumeration and query construction can remove substantial decision difficulty, so a common backbone alone does not establish that the large gains come from SAGE. These details are necessary to interpret both the comparisons and generalization claims.

  2. The evidence does not isolate the proposed improvement in supervision across rounds. Figure 2 evaluates the current student with and without lookahead, rather than measuring the EMA teacher that supplies training targets. Figure 7 tests immediate teacher copying and EMA-generated branches, but neither a fixed initial teacher with fresh student branches nor a fixed branch generator. Surpassing the initial high-budget policy therefore does not establish that improving teacher supervision drives the gains; online imitation with a fixed teacher on newly visited states remains an untested explanation. This distinction matters because the evolving supervision loop is a central contribution.

Questions for authors
  1. What task splits, observation fields, and action-construction rules are used, particularly for WebShop search queries? Please provide a comparison with a strong baseline under the identical interface and execution-step limit.

  2. How much improvement remains when teacher weights and the branch generator are frozen separately? On a common set of held-out histories, does the actual EMA teacher provide increasingly effective decisions when branch information and subsequent continuation policy are held fixed?

3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer rKNc

We sincerely thank the reviewer rKNc for the thoughtful evaluation of our work and for recognizing the value of using counterfactual interactions for supervision and improving student performance without lookahead at deployment. We also appreciate the detailed comments and suggestions, which have helped us improve our experimental evaluation. We address each point below.

W1 & Q1: The benchmark interface leaves the headline comparisons unresolved. [...]

We thank the reviewer for the comments on our experimental settings and baseline comparisons. We would like to clarify the following points.

1. Environment Observations and Action Interface. SAGE strictly follows the standard benchmark interaction protocols. During both training and testing, the zero-lookahead policy uses the same observation fields, action/observation preprocessing, and action execution rules as existing baselines. We do not provide additional environment information, expert hints, or action filtering or ranking.

For WebShop lookahead branches, we apply rule-based Python post-processing to compress and organize lengthy environment observations, as part of SAGE's lookahead design. This processing relies solely on real environment feedback, without introducing additional information, ground-truth answers, rewards, or success scores.

In WebShop, search is a parameterized action. The model generates search queries based on the task goal and interaction history, and executes them through search[query]. This is fully consistent with existing baselines, and we do not provide any additional candidate queries.

2. Data Splits. ALFWorld uses the official Train / Valid Seen / Valid Unseen splits, while WebShop uses the official Train / Development / Test splits. We follow the standard evaluation protocols used in previous work.

3. Fairness of Baseline Comparisons. SAGE requires no external rewards, expert demonstrations, or additional strong teachers, while existing training methods often rely on different forms of supervision. For example, IWM and WKM rely on expert trajectories, ITP-I requires an additional trained world model, ITP-R also relies on reward optimization, CoMAP uses expert trajectories for initialization, and OPD methods use Qwen3-30B-A3B as an external teacher.

These methods differ substantially in their supervision requirements and environment interactions. Forcing identical training budgets or removing additional supervision would change the original methods and make the comparison unfair to the baselines. To ensure a fair comparison as much as possible, we follow the evaluation settings of published baselines, using the same data splits, environment observations, admissible action interfaces, execution step limits, evaluation metrics, and policy backbone. We use the results reported in the published papers to preserve their original training settings and avoid performance differences caused by reimplementation.

4. Attribution of Performance Gains. In Table 2, Qwen3-4B ReAct evaluated using our prompt achieves performance consistent with existing baselines. This indicates that our interaction settings do not artificially reduce task difficulty.

W2 & Q2: The evidence does not isolate the proposed improvement in supervision across rounds. [...]

We thank the reviewer for raising this important question. We conducted additional experiments on ALFWorld Unseen.

First, we separately froze the initial EMA Teacher and the branch generator. The results are shown below.

SettingFrozen BranchEvolving Branch
Frozen EMA Teacher64.1781.34
Evolving EMA Teacher71.3890.29

As shown in the table, when the branch generator is fixed, updating the EMA Teacher improves the success rate from 64.17% to 71.38%. This shows that updating the EMA Teacher makes an independent contribution. Similarly, when the EMA Teacher is frozen, updating the branch generator improves the success rate from 64.17% to 81.34%, showing that the evolving branch generator is also important.

We further randomly sampled 300 decision states from the ALFWorld Unseen test set and generated 8×4 lookahead branches for each state. We then used the teacher models at different training stages (from Step 0 to the final step) to select actions, while keeping the decision states and lookahead branches fixed. We also used GPT-6.1 Sol to generate reference actions under the same conditions and measured action agreement to evaluate the decision quality of the models.

MetricStep 0Step 25Step 50Step 75Final
Action Agreement (%)65.6769.3370.6772.0076.00

With the same states and lookahead branches, action agreement increases steadily from 65.67% to 76.00%. This provides evidence that the decision capability of the evolving EMA Teacher improves during training. Together with the freezing experiments, these results further support the effectiveness of the joint evolution of the EMA Teacher and branch generator in SAGE.

Thank you for the thoughtful response and additional experiments. The separate freezing ablations substantially address my concern about the contributions of the teacher and branch generator, and the interface clarifications are helpful. I am raising my rating to 3.

Thank you for recognizing our additional experiments. Your valuable feedback has helped us further improve the paper. We also appreciate your time and effort.

Reviewer_jXs3OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes improving self-distillation without accessing task rewards or privileged teacher, by 1) enabling higher budget lookahead rollouts alongside training, and 2) using an EMA teacher access this info and guide the student policy. The argument is that having more number of visible environment interactions will provide better guidance to the zero-budget student, and overtime this reinforces a better student leading to improved success rate. The key idea is almost surprisingly effective, and unlocks a new avenue of self-distillation which might have broader impact beyond RL training.

Critical strengths
  1. The paper promotes the idea of self-distillation without access to auxiliary rewards or additional supervision, and the idea itself is quite simple and novel. This is an important contribution since access to task-specific rewards is restrictive in several domains, and this technique can help improve RL model in those scenarios.

  2. The algorithm is shown to be complementary to actual RL training, indicating potential for further improvement.

  3. The ablations presented provide sufficient insight into the effectiveness of the hypothesis, with strongest being the potential for sustained gains with increased look-ahead budgets.

Critical weaknesses
  1. The core idea of the paper is slightly counter-intuitive (or non-trivial), and it is not clear how just providing visibility into more environment interactions (with no notion of success or failure) would lead to better policy outputs for the teacher. Specifically, an underlying assumption is that the teacher policy πθˉr\pi_{\bar{\theta}_r} is a better guide when presented with Bt(bh)\mathcal{B}_t(b_h) - but it is not entirely clear why this should be true.

  2. The authors should also comment on the possible domains this assumption might fail, where despite having access to several rollouts, this need not guarantee an improved guidance. This is slightly apparent with WebShop which has lesser overall gains than ALFWorld, but this needs to be studied more.

Questions for authors
  1. Essentially, this looks like distilling information from pass@K (K being the look-ahead budget) to improve pass@1 (no-look student) - in which case the authors should comment on the possible relations between these.

Minor:

  1. The notation for B\mathcal{B} is inconsistent in Eq 3 and elsewhere, with subscript bhb_h dropped several times.
2 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer jXs3 Weaknesses

We sincerely thank the reviewer jXs3 for the thoughtful evaluation of our work and for recognizing the novelty of SAGE, its ability to improve agents without additional supervision, and its potential to complement reinforcement learning. We also appreciate the constructive comments and questions, which have helped us better explain our motivation and strengthen our experimental evaluation. We address each point below. We have also carefully reviewed and corrected the notation issues pointed out by the reviewer. We address each point below.

W1: The core idea of the paper is slightly counter-intuitive (or non-trivial) [...]

We thank the reviewer for raising this important question. The key insight of SAGE is that trial and error is an important way to learn and improve decisions. Pretrained language models already have some ability to understand tasks and reason about actions. They can therefore learn from the actual outcomes of different actions without explicit rewards or success/failure labels.

In long horizon tasks, a model may understand the task goal but struggle to predict how its current actions will affect future states. For example, two actions may seem equally reasonable at first, but after several steps, one may reveal information needed to complete the task, while the other may block further progress. Through lookahead in the real environment, SAGE allows the EMA Teacher to evaluate the outcomes of different branches using its existing language understanding and reasoning abilities, and select actions that better serve the task goal.

We conducted two additional experiments to examine whether environment lookahead improves decision quality.

First, we fixed the initial model parameters and evaluated complete task success rates on ALFWorld Unseen using lookahead budgets of 0×0, 4×2, and 8×4.

Second, we randomly sampled 300 decision states from ALFWorld Unseen. For each lookahead budget, we used GPT-6.1 Sol to generate reference actions under the same conditions and measured their agreement with the initial model's actions.

Metric0×04×28×4
Success Rate (%)26.154.476.9
Action Agreement (%)25.3353.3365.67

As the lookahead budget increases, the task success rate improves from 26.1% to 76.9%, while action agreement increases from 25.33% to 65.67%. These results show that lookahead can help the same model make better decisions and significantly improve task performance without additional rewards or success labels.

W2: The authors should also comment on the possible domains this assumption might fail [...]

As discussed in the previous response, SAGE uses the actual outcomes of different actions to help the EMA Teacher evaluate which choices better serve the task goal. This is based on an observation: trial and error is an important way to improve decisions, and pretrained language models already have some ability to understand and reason about their environment. They can therefore identify problems through environment interactions without explicit success or failure labels. However, if additional lookahead cannot help the Teacher make better action choices, the resulting supervision may also fail to improve. This is a possible limitation of SAGE.

We further added experiments on Minesweeper, a task different from ALFWorld and WebShop. The model must infer mine locations from gradually revealed numerical clues and make sequential decisions. SAGE achieves a success rate of 57.81%, outperforming the strongest baseline, DAPO, by 11.81 percentage points.

MethodReActGRPODAPOGiGPOSAGE
Success Rate (%)3.1239.146.042.257.8

We also thank the reviewer for raising this interesting question. Identifying tasks where lookahead cannot improve decisions is worth further investigation and provides a useful direction for our future work.

AuthorsOpenReview ↗

Rebuttal for Reviewer jXs3 Questions

Q1: Essentially, this looks like distilling information from pass@K [...]

We thank the reviewer for this insightful question. First, we compared Pass@K with environment lookahead under approximately matched computational costs. Pass@8 corresponds to 4×2 lookahead, while Pass@32 corresponds to 8×4 lookahead. The results are shown below.

SettingPass@K (%)SAGE Lookahead (%)
Pass@8 vs. 4×242.456.4
Pass@32 vs. 8×450.278.6

Under similar computational costs, lookahead outperforms Pass@K by 14.0 and 28.4 percentage points, respectively. This is because Pass@K independently samples trajectories from a fixed policy, while lookahead allows the model to use real environment feedback from different actions to improve its decisions during exploration. In this sense, SAGE is closer to active learning, where the model explores its own weaknesses and learns from the outcomes, potentially improving its policy beyond what can be achieved by simply sampling more trajectories from the original policy.

We further examined whether SAGE only improves efficiency under a limited sampling budget or can improve the policy itself. Given the large decision space of language models, we increased the sampling budget to Pass@100 and compared it with the trained models.

MetricPass@100GRPOSAGE
Success Rate (%)56.070.689.5

The results show that the initial policy achieves only 56.0% success even with 100 independent samples, where a task is counted as successful if at least one sample succeeds. In contrast, the SAGE student achieves 89.5% success with a single rollout and no lookahead after training.

Therefore, we believe that SAGE turns the decision experience gained through exploration into transferable policy capabilities, while Pass@K mainly relies on increasing the sampling budget at inference time to improve the probability of success.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear AC and Reviewers,

We sincerely thank the AC and all reviewers for their time and effort throughout the review process. We have carefully addressed the questions raised by the reviewers and conducted additional experiments and analyses.

As this rebuttal comes to a close, we would like to briefly summarize the main concerns raised during the review process and the corresponding improvements.

Reviewer jXs3 acknowledged the novelty of SAGE and its potential to complement reinforcement learning. To address the concerns about lookahead decision quality, the relationship with Pass@K, and the applicability of our method, we added decision quality analyses, Pass@K comparison experiments under the same budget, and further evaluations on Minesweeper.

Reviewer rKNc acknowledged the value of using environment interactions to construct decision supervision signals, as well as the fact that the Student does not require lookahead at deployment. To address the concerns about experimental interface fairness, data splits, and whether the Teacher and Branch Generator truly achieve co-evolution, we further clarified the experimental settings and added controlled experiments to separately verify the contributions of EMA Teacher and Branch Generator updates.

Reviewer 3gP5 acknowledged the value and novelty of SAGE in achieving self-improvement through environment interactions without external rewards, expert demonstrations, or an additional teacher. To address the concerns about environment interaction assumptions, experimental scope, model scale, and interaction efficiency, we added experiments with Qwen3-8B and Minesweeper, provided the complete training and evaluation code, and further added a controlled comparison of interaction budgets with GRPO.

Through the extensive discussions and additional experiments in this rebuttal, we believe that we have fully addressed and resolved all the main concerns raised by the reviewers. We sincerely thank all reviewers for their careful reviews and valuable suggestions, which have helped us further improve the experimental analysis and arguments in our paper. We also extend our special thanks to the AC for the time and effort devoted throughout the review process.

Warm regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.