- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies deployment-time self-evolution of LLM agents under limited user feedback. It introduces DEAL, a benchmark consisting of related workplace and AI research tasks where agents receive only one external feedback per task. To address the exploration bottleneck of single-rollout RL, the authors propose Self-Rewarded Exploration, which generates additional rollouts from saved environment snapshots, uses the original feedback to construct task-specific reward rubrics, and trains the model with GRPO using self-rewards. Experiments show improved online learning and held-out performance, particularly on AI research tasks, although OOD generalization and general capability preservation remain challenging.
Critical strengths
- The paper addresses an important challenge of enabling LLM agents to continuously improve from limited user feedback during real-world deployment.
- DEAL provides a dedicated testbed for evaluating continual agent adaptation, covering both AI research and workplace assistance scenarios.
- The paper provides a useful theoretical perspective by decomposing the regret of self-rewarded exploration into coverage gap and selection error, offering insights into the trade-off between exploration benefits and imperfect reward estimation.
- SRE offers a simple yet effective framework that combines offline rollout expansion, feedback-conditioned self-reward, and GRPO, making it easy to implement and integrate into existing agent training pipelines.
Critical weaknesses
- Although the paper motivates SRE as a solution to the exploration bottleneck, the external feedback is used only to construct a surrogate reward function rather than to guide the exploration process itself. Additional trajectories are sampled from the original policy and subsequently optimized using standard GRPO. I'm not sure what distinguishes SRE from conventional RL with a feedback-conditioned reward model and additional rollout sampling. Is the main contribution a new exploration mechanism, or simply an alternative way of constructing reward signals under limited external feedback?
- SRE achieves substantial improvements on AI research tasks, but its gains are much smaller on workplace tasks, with notable performance degradation on workplace OOD benchmarks. The paper does not sufficiently investigate the underlying reasons for these discrepancies. This makes it difficult to understand under what conditions SRE is effective and whether its benefits generalize across different deployment scenarios.
Questions for authors
While the paper provides useful analyses of rollout count and feedback-conditioned reward estimation, their effects on the final policy performance remain insufficiently explored. I suggest the authors (1) evaluate how different rollout counts affect final performance, (2) examine the contribution of the feedback anchor to final performance, rather than only reward ranking accuracy, and (3) include compute-matched RL baselines, such as REINFORCE++ with additional rollout sampling or more training updates under the same computational budget as SRE. This would help determine whether SRE's improvements come from its feedback-conditioned reward mechanism or simply from using more rollouts and computational resources.
6 comments
Response by the authors (2/3)
W1 Exploration mechanism or reward construction?
- Although the paper motivates SRE as a solution to the exploration bottleneck, the external feedback is used only to construct a surrogate reward function rather than to guide the exploration process itself. Additional trajectories are sampled from the original policy and subsequently optimized using standard GRPO. I'm not sure what distinguishes SRE from conventional RL with a feedback-conditioned reward model and additional rollout sampling. Is the main contribution a new exploration mechanism, or simply an alternative way of constructing reward signals under limited external feedback?
Thanks for your comments. You are right that the user feedback does not directly guide the sampling of the current task; extra trajectories are sampled from the current policy, and the feedback is used to construct their rewards.
Firstly, we would like to clarify the overall contribution of our paper: we introduce a complete framework for deployment-time self-evolution by: 1) formulating the single-feedback deployment protocol, 2) building the executable testbed DEAL across workspace and research workflows, and 3) systematically comparing training-free and training-based methods.
Within parameter-based learning, SRE is not a new sampling algorithm, but a simple and practical coupling of offline rollout expansion and feedback-conditioned reward construction under single-feedback constraints. Under this protocol, the two parts depend on each other: 1) with only one live rollout, even an exact user reward leaves better behaviors unexplored; 2) with extra offline rollouts, the user cannot evaluate them again, so we need a feedback-anchored surrogate reward to turn them into useful learning signals. Although its algorithmic form resembles RL with a feedback-conditioned reward model and extra rollout sampling, to the best of our knowledge, we are the first to identify and address this bottleneck in deployment-time learning.
In addition, following Reviewer EcEp's suggestion, we also test another way of using user feedback, feedback-conditioned self-distillation (SDPO [2]), which puts the user feedback into the teacher's context to compute token-level advantages. Experimental results on DEAL-RS can be found in the following table.
| Benchmark | Base | R++ | SDPO [2] | SRE (Ours) |
|---|---|---|---|---|
| DEAL-RS Held-out | 21.30 | 36.95 | 26.10 | 41.22 |
| AARRI-Bench | 35.38 | 37.01 | 35.37 | 38.60 |
Here SRE clearly outperforms SDPO on both benchmarks. This shows that although SRE is simple, turning single feedback into an outcome-level rubric for GRPO is much more stable and effective in long-horizon agentic tasks.
For future work, it is also promising to further improve the method, such as exploring feedback-guided sampling or introducing process reward models for finer-grained credit assignment.
We have revised Section 1 and Section 4 to state this positioning more clearly, and added the SDPO comparison to Section 5.3 (Figure 5a), Section 6, and Appendix E.4 (Table 8). Thanks again for raising this insightful point!
Q1 Effect of rollout counts on final policy performance
While the paper provides useful analyses of rollout count and feedback-conditioned reward estimation, their effects on the final policy performance remain insufficiently explored. I suggest the authors (1) evaluate how different rollout counts affect final performance ...
Thanks for your constructive suggestion! Following your advice, we train SRE end-to-end on DEAL-RS with different rollout counts (we start from since GRPO advantage normalization requires estimating the group std). Experimental results in the following table show that final policy performance improves steadily on both benchmarks as increases. At a small rollout count (), estimating the group mean and standard deviation from very few rollouts introduces noisy advantage estimates; larger both expands behavior coverage and stabilizes group-relative advantages.
| Benchmark | Base | SRE () | SRE () | SRE () |
|---|---|---|---|---|
| DEAL-RS Held-out | 21.30 | 33.92 | 36.51 | 41.22 |
| AARRI-Bench | 35.38 | 36.61 | 37.52 | 38.60 |
We have included this rollout-count ablation in Section 5.3 (Figure 5b) and Appendix E.4 (Table 8). Thanks for your helpful advice!
Response by the authors (3/3)
Q2 Contribution of the feedback anchor to final policy performance
... (2) examine the contribution of the feedback anchor to final performance, rather than only reward ranking accuracy ...
Thanks for your suggestion. Following your advice, we train SRE w/o anchor end-to-end on DEAL-RS: it keeps the exact same additional rollouts and GRPO update, and directly uses the model's own self-reward without user feedback (building the rubric from the task description alone). Experimental results can be found in the following table.
| Method (DEAL-RS) | Held-out Avg. | AARRI-Bench |
|---|---|---|
| Base | 21.30 | 35.38 |
| SRE w/o anchor (7 rollouts, task-only rubric) | 37.39 | 37.25 |
| SRE (7 rollouts, feedback-anchored rubric) | 41.22 | 38.60 |
SRE clearly outperforms SRE w/o anchor on both benchmarks under the exact same rollout budget, confirming that the higher reward ranking accuracy brought by the feedback anchor translates into clear gains on the final policy.
This end-to-end comparison is now included in Section 5.3 (Figure 5a) and Appendix E.4 (Table 8). Thank you very much.
Q3 Compute-matched RL baselines
... and (3) include compute-matched RL baselines, such as REINFORCE++ with additional rollout sampling or more training updates under the same computational budget as SRE. This would help determine whether SRE's improvements come from its feedback-conditioned reward mechanism or simply from using more rollouts and computational resources.
Thanks for your helpful suggestion! We first examine the GPU hours of REINFORCE++ and SRE. On an 8$\times$H200 node with asynchronous rollout and training, SRE takes 24.06h (192.5 GPU-hours) vs. 18.02h (144.2 GPU-hours) for REINFORCE++ (~). Following your suggestion, we compare SRE on DEAL-RS with two baselines under a matched or comparable compute budget: 1) R++ (Matched), where we update REINFORCE++ twice per rollout step on the live trajectories to match SRE's GPU hours; and 2) SDPO [2] (), a feedback-conditioned self-distillation baseline over the same 7 offline rollouts, which has a comparable compute cost to SRE. Experimental results can be found in the following table.
| Benchmark | Base | R++ | R++ (Matched) | SDPO () | SRE () |
|---|---|---|---|---|---|
| DEAL-RS Held-out | 21.30 | 36.95 | 38.40 | 26.10 | 41.22 |
| AARRI-Bench | 35.38 | 37.01 | 37.43 | 35.37 | 38.60 |
Here we observe that neither R++ Matched nor SDPO matches SRE. Together with Q2, this confirms that SRE's gains come from its feedback-conditioned reward mechanism on expanded rollouts rather than extra compute alone.
We have included these results in Section 5.3 (Figure 5a), Appendix D.3 (Table 4), and Appendix E.4 (Table 8). Thank you again for your constructive suggestions!
References
[1] MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement. https://arxiv.org/abs/2610.11959
[2] Reinforcement Learning via Self-Distillation. ICML 2026.
We hope our response has addressed your concerns. If we have missed anything, please let us know.
Response by the authors (1/3)
Thanks for your encouraging and insightful comments. Below please find our responses. We will start with W2 because we fixed an issue in our training reward implementation during the rebuttal period, which updates the main results on both domains.
W2 Different gains on research and workspace tasks
Thank you for pointing this out! After double checking our submitted implementation, we found that the discrepancy on workspace tasks came from an issue in our training reward system. Sometimes the agent fails to submit deliverables in a valid format, and instead outputs the content in other ways. For example, when asked to submit a Word document (
.docx), the agent may write raw Markdown bytes directly into the file instead of a valid Word binary. Although the file itself is invalid, the user agent may still give partial credit when scoring with the rubric if the text content is right. During training, the agent gradually hacks this loophole.To fix it, we add a deliverable check: if the agent fails to produce valid artifacts, it directly gets 0 reward. This acts as a multiplicative gate on the reward, similar to the agentic RL reward design in MiMo-V2.6 [1]. As shown in https://anonymous.4open.science/r/SRE-rebuttal-figs/ws_reward_gate_before_after.pdf (Gemma4-E4B-IT with REINFORCE++ on DEAL-WS), before the fix, the reward drops sharply after step 40 and the entropy blows up. After the fix, the entropy stays stable and the reward keeps improving.
This reward fix only affects the training-based methods (REINFORCE++ and SRE), while Base and the training-free baselines remain unchanged. We re-run REINFORCE++ and SRE on both domains. Experimental results can be found in the following table.
Here we observe that once the reward-hacking issue is resolved, SRE achieves strong gains on workspace tasks as well, obtaining the best online gain, held-out average, and OOD scores. This shows that the benefits of SRE generalize across both research and workspace deployment scenarios.
We have updated Section 3.3, Section 5.2 (Table 1 and Figure 4), Section 5.3 (Figure 6 for Qwen3.5-9B on both domains), Appendix D.2 (Figure 9), and Appendix E.3 (Table 7) in our paper accordingly. Thank you very much!