acceptodds
Under review as a conference paper at ICLR 2027

Learning from Single-Feedback Deployment Experience

Abstract

The success of LLM agents has attracted growing interest in deploying small-size, self-hosted models to serve as user assistants for everyday work. After deployment, users would like these assistants to improve as they work. In practice, however, tasks arrive sequentially, and users cannot be expected to repeatedly evaluate different attempts at the same task. A common approach is to adapt memories or skills in the agent harness while keeping model parameters fixed. Parameter-based learning provides another promising direction, yet how to effectively train models from such single-feedback deployment experience remains under-explored. In this paper, we study this problem and introduce DEAL, an executable testbed with related task streams in general workplace assistance and AI research. Using DEAL, we find that single-rollout reinforcement learning suffers from an exploration bottleneck, where one evaluated trajectory may leave better behaviors unexplored. Based on this observation, we propose Self-Rewarded Exploration (SRE), which expands additional trajectories from isolated task snapshots and uses the live feedback as an anchor to assign self-rewards. We further provide a theoretical analysis of when the benefit of rollout expansion can outweigh self-reward errors. Experiments on Gemma4-E4B-IT demonstrate the effectiveness of SRE in improving deployment-time performance, generalizing to unseen tasks, and preserving general capabilities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Deployment-time Learning, Agent Self-evolution

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fcDZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies deployment-time self-evolution of LLM agents under limited user feedback. It introduces DEAL, a benchmark consisting of related workplace and AI research tasks where agents receive only one external feedback per task. To address the exploration bottleneck of single-rollout RL, the authors propose Self-Rewarded Exploration, which generates additional rollouts from saved environment snapshots, uses the original feedback to construct task-specific reward rubrics, and trains the model with GRPO using self-rewards. Experiments show improved online learning and held-out performance, particularly on AI research tasks, although OOD generalization and general capability preservation remain challenging.

Critical strengths
  1. The paper addresses an important challenge of enabling LLM agents to continuously improve from limited user feedback during real-world deployment.
  2. DEAL provides a dedicated testbed for evaluating continual agent adaptation, covering both AI research and workplace assistance scenarios.
  3. The paper provides a useful theoretical perspective by decomposing the regret of self-rewarded exploration into coverage gap and selection error, offering insights into the trade-off between exploration benefits and imperfect reward estimation.
  4. SRE offers a simple yet effective framework that combines offline rollout expansion, feedback-conditioned self-reward, and GRPO, making it easy to implement and integrate into existing agent training pipelines.
Critical weaknesses
  1. Although the paper motivates SRE as a solution to the exploration bottleneck, the external feedback is used only to construct a surrogate reward function rather than to guide the exploration process itself. Additional trajectories are sampled from the original policy and subsequently optimized using standard GRPO. I'm not sure what distinguishes SRE from conventional RL with a feedback-conditioned reward model and additional rollout sampling. Is the main contribution a new exploration mechanism, or simply an alternative way of constructing reward signals under limited external feedback?
  2. SRE achieves substantial improvements on AI research tasks, but its gains are much smaller on workplace tasks, with notable performance degradation on workplace OOD benchmarks. The paper does not sufficiently investigate the underlying reasons for these discrepancies. This makes it difficult to understand under what conditions SRE is effective and whether its benefits generalize across different deployment scenarios.
Questions for authors

While the paper provides useful analyses of rollout count and feedback-conditioned reward estimation, their effects on the final policy performance remain insufficiently explored. I suggest the authors (1) evaluate how different rollout counts affect final performance, (2) examine the contribution of the feedback anchor to final performance, rather than only reward ranking accuracy, and (3) include compute-matched RL baselines, such as REINFORCE++ with additional rollout sampling or more training updates under the same computational budget as SRE. This would help determine whether SRE's improvements come from its feedback-conditioned reward mechanism or simply from using more rollouts and computational resources.

6 comments
AuthorsOpenReview ↗

Response by the authors (1/3)

Thanks for your encouraging and insightful comments. Below please find our responses. We will start with W2 because we fixed an issue in our training reward implementation during the rebuttal period, which updates the main results on both domains.

W2 Different gains on research and workspace tasks

  1. SRE achieves substantial improvements on AI research tasks, but its gains are much smaller on workplace tasks, with notable performance degradation on workplace OOD benchmarks. The paper does not sufficiently investigate the underlying reasons for these discrepancies. This makes it difficult to understand under what conditions SRE is effective and whether its benefits generalize across different deployment scenarios.

Thank you for pointing this out! After double checking our submitted implementation, we found that the discrepancy on workspace tasks came from an issue in our training reward system. Sometimes the agent fails to submit deliverables in a valid format, and instead outputs the content in other ways. For example, when asked to submit a Word document (.docx), the agent may write raw Markdown bytes directly into the file instead of a valid Word binary. Although the file itself is invalid, the user agent may still give partial credit when scoring with the rubric if the text content is right. During training, the agent gradually hacks this loophole.

To fix it, we add a deliverable check: if the agent fails to produce valid artifacts, it directly gets 0 reward. This acts as a multiplicative gate on the reward, similar to the agentic RL reward design in MiMo-V2.6 [1]. As shown in https://anonymous.4open.science/r/SRE-rebuttal-figs/ws_reward_gate_before_after.pdf (Gemma4-E4B-IT with REINFORCE++ on DEAL-WS), before the fix, the reward drops sharply after step 40 and the entropy blows up. After the fix, the entropy stays stable and the reward keeps improving.

This reward fix only affects the training-based methods (REINFORCE++ and SRE), while Base and the training-free baselines remain unchanged. We re-run REINFORCE++ and SRE on both domains. Experimental results can be found in the following table.

DEAL-RSOnline GainComp.Align.Verify.WorkflowAuditAvg.AARRI-Bench (OOD)General Cap.
Base0.0016.6827.9833.8614.0918.3921.3035.3852.91
EvoSkill-1.0726.0429.9034.6119.7124.7026.9030.88–
CASCADE+2.4925.5223.9337.3016.3819.5924.9335.25–
REINFORCE+++5.6331.3443.4055.2629.6829.6436.9537.0152.97
SRE (Ours)+10.6636.9147.8955.9328.6541.6341.2238.6055.16
DEAL-WSOnline GainOps.Log.Res.Dev.Prod.Avg.PinchBenchClawGym-BenchGeneral Cap.
Base0.0048.5244.7838.3437.0232.5540.2455.1146.6152.91
EvoSkill-6.5047.5849.2235.3538.6133.7440.9050.8746.86–
CASCADE+0.9252.5051.0832.7054.8436.8445.5956.1846.26–
REINFORCE++-0.4442.3244.2741.3661.3133.4244.5457.9947.4452.98
SRE (Ours)+4.5449.2251.6745.6150.6936.9046.8159.5450.2752.87

Here we observe that once the reward-hacking issue is resolved, SRE achieves strong gains on workspace tasks as well, obtaining the best online gain, held-out average, and OOD scores. This shows that the benefits of SRE generalize across both research and workspace deployment scenarios.

We have updated Section 3.3, Section 5.2 (Table 1 and Figure 4), Section 5.3 (Figure 6 for Qwen3.5-9B on both domains), Appendix D.2 (Figure 9), and Appendix E.3 (Table 7) in our paper accordingly. Thank you very much!

AuthorsOpenReview ↗

Response by the authors (2/3)

W1 Exploration mechanism or reward construction?

  1. Although the paper motivates SRE as a solution to the exploration bottleneck, the external feedback is used only to construct a surrogate reward function rather than to guide the exploration process itself. Additional trajectories are sampled from the original policy and subsequently optimized using standard GRPO. I'm not sure what distinguishes SRE from conventional RL with a feedback-conditioned reward model and additional rollout sampling. Is the main contribution a new exploration mechanism, or simply an alternative way of constructing reward signals under limited external feedback?

Thanks for your comments. You are right that the user feedback does not directly guide the sampling of the current task; extra trajectories are sampled from the current policy, and the feedback is used to construct their rewards.

Firstly, we would like to clarify the overall contribution of our paper: we introduce a complete framework for deployment-time self-evolution by: 1) formulating the single-feedback deployment protocol, 2) building the executable testbed DEAL across workspace and research workflows, and 3) systematically comparing training-free and training-based methods.

Within parameter-based learning, SRE is not a new sampling algorithm, but a simple and practical coupling of offline rollout expansion and feedback-conditioned reward construction under single-feedback constraints. Under this protocol, the two parts depend on each other: 1) with only one live rollout, even an exact user reward leaves better behaviors unexplored; 2) with extra offline rollouts, the user cannot evaluate them again, so we need a feedback-anchored surrogate reward to turn them into useful learning signals. Although its algorithmic form resembles RL with a feedback-conditioned reward model and extra rollout sampling, to the best of our knowledge, we are the first to identify and address this bottleneck in deployment-time learning.

In addition, following Reviewer EcEp's suggestion, we also test another way of using user feedback, feedback-conditioned self-distillation (SDPO [2]), which puts the user feedback into the teacher's context to compute token-level advantages. Experimental results on DEAL-RS can be found in the following table.

BenchmarkBaseR++SDPO [2]SRE (Ours)
DEAL-RS Held-out21.3036.9526.1041.22
AARRI-Bench35.3837.0135.3738.60

Here SRE clearly outperforms SDPO on both benchmarks. This shows that although SRE is simple, turning single feedback into an outcome-level rubric for GRPO is much more stable and effective in long-horizon agentic tasks.

For future work, it is also promising to further improve the method, such as exploring feedback-guided sampling or introducing process reward models for finer-grained credit assignment.

We have revised Section 1 and Section 4 to state this positioning more clearly, and added the SDPO comparison to Section 5.3 (Figure 5a), Section 6, and Appendix E.4 (Table 8). Thanks again for raising this insightful point!

Q1 Effect of rollout counts on final policy performance

While the paper provides useful analyses of rollout count and feedback-conditioned reward estimation, their effects on the final policy performance remain insufficiently explored. I suggest the authors (1) evaluate how different rollout counts affect final performance ...

Thanks for your constructive suggestion! Following your advice, we train SRE end-to-end on DEAL-RS with different rollout counts K∈3,5,7K\in\\{3, 5, 7\\} (we start from K=3K=3 since GRPO advantage normalization requires estimating the group std). Experimental results in the following table show that final policy performance improves steadily on both benchmarks as KK increases. At a small rollout count (K=3K=3), estimating the group mean and standard deviation from very few rollouts introduces noisy advantage estimates; larger KK both expands behavior coverage and stabilizes group-relative advantages.

BenchmarkBaseSRE (K=3K=3)SRE (K=5K=5)SRE (K=7K=7)
DEAL-RS Held-out21.3033.9236.5141.22
AARRI-Bench35.3836.6137.5238.60

We have included this rollout-count ablation in Section 5.3 (Figure 5b) and Appendix E.4 (Table 8). Thanks for your helpful advice!

AuthorsOpenReview ↗

Response by the authors (3/3)

Q2 Contribution of the feedback anchor to final policy performance

... (2) examine the contribution of the feedback anchor to final performance, rather than only reward ranking accuracy ...

Thanks for your suggestion. Following your advice, we train SRE w/o anchor end-to-end on DEAL-RS: it keeps the exact same K=7K=7 additional rollouts and GRPO update, and directly uses the model's own self-reward without user feedback (building the rubric from the task description alone). Experimental results can be found in the following table.

Method (DEAL-RS)Held-out Avg.AARRI-Bench
Base21.3035.38
SRE w/o anchor (7 rollouts, task-only rubric)37.3937.25
SRE (7 rollouts, feedback-anchored rubric)41.2238.60

SRE clearly outperforms SRE w/o anchor on both benchmarks under the exact same rollout budget, confirming that the higher reward ranking accuracy brought by the feedback anchor translates into clear gains on the final policy.

This end-to-end comparison is now included in Section 5.3 (Figure 5a) and Appendix E.4 (Table 8). Thank you very much.

Q3 Compute-matched RL baselines

... and (3) include compute-matched RL baselines, such as REINFORCE++ with additional rollout sampling or more training updates under the same computational budget as SRE. This would help determine whether SRE's improvements come from its feedback-conditioned reward mechanism or simply from using more rollouts and computational resources.

Thanks for your helpful suggestion! We first examine the GPU hours of REINFORCE++ and SRE. On an 8$\times$H200 node with asynchronous rollout and training, SRE takes 24.06h (192.5 GPU-hours) vs. 18.02h (144.2 GPU-hours) for REINFORCE++ (~1.33×1.33\times). Following your suggestion, we compare SRE on DEAL-RS with two baselines under a matched or comparable compute budget: 1) R++ (Matched), where we update REINFORCE++ twice per rollout step on the live trajectories to match SRE's GPU hours; and 2) SDPO [2] (K=7K=7), a feedback-conditioned self-distillation baseline over the same 7 offline rollouts, which has a comparable compute cost to SRE. Experimental results can be found in the following table.

BenchmarkBaseR++R++ (Matched)SDPO (K=7K=7)SRE (K=7K=7)
DEAL-RS Held-out21.3036.9538.4026.1041.22
AARRI-Bench35.3837.0137.4335.3738.60

Here we observe that neither R++ Matched nor SDPO matches SRE. Together with Q2, this confirms that SRE's gains come from its feedback-conditioned reward mechanism on expanded rollouts rather than extra compute alone.

We have included these results in Section 5.3 (Figure 5a), Appendix D.3 (Table 4), and Appendix E.4 (Table 8). Thank you again for your constructive suggestions!

References

[1] MiMo-V2.6: Scaling Reinforcement Learning Towards Self-Improvement. https://arxiv.org/abs/2610.11959

[2] Reinforcement Learning via Self-Distillation. ICML 2026.


We hope our response has addressed your concerns. If we have missed anything, please let us know.

Reviewer_EcEpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how LLM agents can learn during deployment when users provide a score and textual feedback only once per task. The authors introduce DEAL, a testbed covering workplace and AI research tasks, and propose Self-Rewarded Exploration (SRE). SRE builds a task-specific rubric from the agent’s original execution and the feedback it receives. It then samples additional trajectories in copies of the initial task environment, scores them with the rubric, and uses these scores for GRPO-style training. This lets the agent learn from further attempts without asking users for more feedback. The paper also provides an analysis of exploration gains and scoring errors, and evaluates performance during deployment, on held-out tasks, and on general capability benchmarks.

Critical strengths
  1. The paper addresses a practical problem: how agents can learn during deployment when users provide feedback only once per task.
  2. SRE reuses feedback on one trajectory to evaluate additional trajectories through a shared rubric. This provides more training signals without asking users to assess further attempts.
Critical weaknesses
  1. Limited methodological novelty. SRE explores by sampling more trajectories from the same task and initial environment; the feedback does not guide this sampling. The main new element seems to be the feedback-conditioned rubric, but the paper gives few details about how it is constructed or how it differs from existing feedback-based learning methods.
  2. Strong assumptions about user feedback. Every task receives a score and textual feedback from a simulated evaluator that can access the hidden rubric. Real users may provide feedback only occasionally, and their comments may be vague or incorrect. The paper does not test these conditions or show how much feedback is needed before SRE becomes useful.
  3. Insufficient baseline comparisons. SRE uses both textual feedback and seven additional rollouts per task, whereas REINFORCE++ trains on live trajectories with scalar rewards. It is therefore difficult to tell whether the gains come from rubric construction, richer feedback, or extra computation. Comparisons with feedback-conditioned distillation and multi-rollout learning under comparable compute budgets would help answer this question.
Questions for authors
  1. SRE generates seven additional trajectories per task and performs rubric generation and repeated scoring, whereas REINFORCE++ learns from the live trajectory alone. Can you report these additional costs and compare SRE with alternatives under matched rollout or token budgets? In particular, using the same seven candidates for a no-anchor rubric baseline would isolate the contribution of feedback anchoring, while a feedback-conditioned distillation baseline would test whether rubric-based scoring is a better use of the available feedback.
  2. Could you clarify the rubric agent’s configuration, including its model, prompt, and output format? Table 2 lists Gemma4-31B-IT as the “reward judge,” but it is unclear whether this model generates rubrics, scores internal trajectories, or provides external feedback. Please also specify whether the rubric generator and self-scorer remain fixed throughout training.
  3. How much feedback is needed before SRE provides useful gains? For example, 50, 100, and 500 feedback events on the same held-out set. It would also be useful to keep the task stream fixed and provide feedback on only a subset of tasks, or replace detailed feedback with brief comments or scalar scores, to test whether the gains persist under more realistic feedback conditions.
  4. Are the main gains robust across training seeds or task orders? Please also clarify the following discrepancies: Section 3.2 reports 2,000 online workplace tasks, whereas Appendix C.2 reports 1,000; the base model’s general-capability average is 57.04 in Table 1 but 57.86 in Table 6
11 comments
AuthorsOpenReview ↗

Response by the authors (1/6)

Thanks for your helpful and detailed comments. Below please find our responses.

W1-1 & W3-1 & Q1-1 Overall contributions and comparison with feedback-conditioned distillation methods

  1. Limited methodological novelty. SRE explores by sampling more trajectories from the same task and initial environment; the feedback does not guide this sampling. The main new element seems to be the feedback-conditioned rubric, ... or how it differs from existing feedback-based learning methods.

  2. ... Comparisons with feedback-conditioned distillation ... would help answer this question.

  3. ... while a feedback-conditioned distillation baseline would test whether rubric-based scoring is a better use of the available feedback.

Thanks for your helpful comments! We would like to first clarify the overall scope of our paper. Our core contribution is a complete framework designed for deployment-time self-evolution: we formulate the single-feedback deployment protocol, build an executable testbed DEAL across workspace and AI research workflows, and systematically compare both training-free and training-based methods under this realistic setting.

Within parameter-based learning, we examine the most direct baseline, single-rollout RL, and point out its exploration bottleneck: with only one live trajectory evaluated per task, better behaviors remain unexplored. To address this, we propose SRE as a simple and direct improvement. Instead of guiding current-task sampling, SRE samples extra rollouts offline from the saved snapshot. It then uses the live attempt and feedback to build a shared rubric and score these rollouts, so the agent can learn from them without asking the user again.

We agree that this is not the only way to use feedback. Following your suggestion, we also compare SRE with a feedback-conditioned distillation method, SDPO [1]. We keep the same K=7K=7 additional offline rollouts (the anchor itself is not used for updates), and only change how advantages are computed. Specifically, the teacher receives the user feedback from the live trajectory as privileged information, while the student only sees the task context. For each token yty_t in the 7 rollouts, we compute the token-level advantage At=log⁡π0(yt∣context,F)−log⁡πstudent(yt∣context)A_t = \log \pi_0(y_t \mid \text{context}, F) - \log \pi_{\text{student}}(y_t \mid \text{context}) to update the student directly, without GRPO outcome advantages. Experimental results on DEAL-RS can be found in the following table.

Method (DEAL-RS)DEAL-RS Held-out Avg.AARRI-Bench
Base21.3035.38
REINFORCE++36.9537.01
SDPO [1] (feedback-conditioned distillation)26.1035.37
SRE (Ours)41.2238.60

We can observe that both SRE and REINFORCE++ clearly outperform SDPO, though SDPO also improves the agent. A possible reason is that such distillation-based methods may suffer from training instability in long-horizon agentic RL, which is also observed in recent work [2].

We have included this comparison and discussion in Section 5.3 (Figure 5a), Section 6, and Appendix E.4 (Table 8) of the revised paper. Thank you again for your constructive feedback!

AuthorsOpenReview ↗

Response by the authors (2/6)

W1-2 & Q2 Rubric construction details and role configurations

  1. ... the paper gives few details about how it is constructed ...

  2. Could you clarify the rubric agent's configuration, including its model, prompt, and output format? Table 2 lists Gemma4-31B-IT as the "reward judge," but it is unclear whether this model generates rubrics, scores internal trajectories, or provides external feedback. Please also specify whether the rubric generator and self-scorer remain fixed throughout training.

Thank you for pointing these out. Our wording "reward judge" in Table 2 is indeed unclear, and we are sorry for not explicitly stating the detailed settings in our submitted version. Below we clarify the three roles and connect them to the corresponding sections in our paper:

  • External user simulator: Uses Gemma4-31B-IT ("Reward judge" in Table 2). As described in Section 3.3, it accesses the hidden ground-truth rubric to evaluate the single live response and return feedback. It is only part of the testbed environment and never scores offline rollouts.
  • Self-rubric agent: Uses the frozen base model (Gemma4-E4B-IT or Qwen3.5-9B in our experiments). As outlined in Section 4.2 and Algorithm 1 (Appendix A), it takes the task description, the live execution/deliverables, and the user feedback as input to build a shared task-specific rubric QtQ_t.
  • Self-scoring agent: Also uses the frozen base model. For each offline rollout, it evaluates the candidate trace and deliverables against QtQ_t (without seeing the user feedback).

Following recent work [3-4], we keep both the self-rubric agent and the self-scoring agent as the frozen base model throughout training, instead of updating them with the actor, to stabilize the RL training.

For aggregation and error handling, as noted in Appendix D.1, the self-scoring agent runs 3 evaluations for each candidate rollout and takes the median score of the valid responses as the self-reward. If all 3 scoring responses are invalid or if the generated rubric / user-agent output is invalid, we mask out that trajectory so it neither participates in advantage calculation nor updates the policy.

We have revised Table 2 (Appendix D.1) to distinguish these roles, updated the agent role settings in Section 4.2 and Appendix D.1, and included the core prompt templates and output schemas in Appendix G for clarity. Thank you again for pointing this out!

AuthorsOpenReview ↗

Response by the authors (3/6)

W2-1 & Q3-1 Imperfect user feedback (incorrect, occasional, and scalar-only)

  1. Strong assumptions about user feedback. Every task receives a score and textual feedback from a simulated evaluator that can access the hidden rubric. Real users may provide feedback only occasionally, and their comments may be vague or incorrect. ...

  2. ... It would also be useful to keep the task stream fixed and provide feedback on only a subset of tasks, or replace detailed feedback with brief comments or scalar scores, to test whether the gains persist under more realistic feedback conditions.

Thanks for your constructive suggestions! We agree that in real-world deployment, users may make mistakes, skip feedback on some tasks, or simply give a score without comments. Following your suggestions, we test these three conditions on DEAL-RS (trained for 20 steps due to limited time and compute during the rebuttal):

  • Incorrect feedback: With probability p∈0.2,0.5,0.8p\in\\{0.2,0.5,0.8\\}, we add a system prompt instructing the user agent to simulate a real user giving inaccurate feedback (prompt in Appendix G.3), while keeping held-out evaluation clean.
  • Occasional feedback: Keeping the task stream fixed, we provide feedback on 5050\\% randomly selected tasks; for tasks without feedback, SRE directly falls back to our w/o anchor update (constructing the rubric from the task alone).
  • Scalar-only feedback: We remove textual comments and only give the agent the scalar score.

As shown in the following table, SRE still learns effectively across all these imperfect feedback settings and clearly improves over the base policy. Even under 8080\\% wrong feedback or scalar-only scores, SRE remains above the w/o anchor baseline.

Feedback settingDEAL-RS Held-out Avg.
Base21.30
SRE w/o anchor28.23
SRE, 50% missing feedback27.51
SRE, scalar-only feedback29.10
SRE, 80% incorrect feedback28.95
SRE, 50% incorrect feedback29.34
SRE, 20% incorrect feedback30.27
SRE, full feedback33.53

Upon closer inspection, SRE remains robust under imperfect feedback for three reasons:

  1. Our rubric-and-scoring pipeline can naturally defend against noise from user feedback by grounding criteria in the original task description and local files (with a case study in Appendix F.3);
  2. Even if the feedback is wrong, the anchor also includes the complete live trajectory and its deliverables, implying for the self-rubric agent to know which files and deliverables should be inspected;
  3. We only pass the anchor to the self-rubric agent (not the self-scoring agent), so if a flawed rubric criterion acts like a shared linear group-level reward shift bxb_x, it can be cancelled out in GRPO group-centered advantages (Proposition 1) without flipping the relative ranking of candidates.

Thus, SRE is quite robust even with imperfect user feedback, showing great potential in real-world usage. These new results and discussions have been added to Section 5.3 (Figure 7b), Appendix E.4, and Appendix F.3. Thank you for these helpful suggestions!

W2-2 & Q3-2 Feedback efficiency across early checkpoints

  1. ... The paper does not ... show how much feedback is needed before SRE becomes useful.

  2. How much feedback is needed before SRE provides useful gains? For example, 50, 100, and 500 feedback events on the same held-out set.

Thanks for your good question! Following your suggestion, we evaluate early checkpoints of SRE on the DEAL-RS held-out set from 0 to 500 feedbacks, as shown in the table below (and plotted in Figure 7a of Section 5.3). After a short warm-up in the first 100 feedbacks, SRE starts to improve quickly during 100–300 feedbacks. At 500 feedbacks, SRE already gets a clear gain over Base and comes close to the final performance of REINFORCE++.

Feedback eventsPolicyDEAL-RS Held-out Avg.
0Base21.30
50SRE22.29
100SRE21.90
200SRE28.12
300SRE32.56
400SRE33.78
500SRE34.55
1,800REINFORCE++ (final)36.95
1,800SRE (final)41.22

We have included this discussion in Section 5.3 (Figure 7a) and Appendix E.4. Thanks again for suggesting this analysis!

Reviewer_TGdZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies how agents can learn during deployment when each task receives only one external feedback signal. It introduces DEAL, a testbed with related workplace and research task streams, and proposes Self-Rewarded Exploration. Starting from the feedback on a live execution, SRE generates additional trajectories in isolated task snapshots, scores them with a feedback-conditioned LLM judge, and updates the model through GRPO.

Critical strengths
  1. The paper addresses an interesting and practical problem of learning from deployment trajectories when users provide feedback only once per task.

  2. The paper is well written and clearly structured.

Critical weaknesses
  1. Although the motivation is reasonable, saving task snapshots and allowing repeated offline execution makes a straightforward baseline possible. One could generate multiple rollouts, construct task-specific rubrics from the task alone, use a strong LLM judge to score the final results, and train with GRPO. The paper does not compare against this baseline, making it difficult to establish whether the gains come from the proposed method or from the richer trajectories provided by additional rollouts.

  2. The method starts from a single user feedback signal, but that feedback is not necessarily correct. Could using it to assign self-rewards across multiple rollouts amplify an incorrect judgment? The paper would benefit from examining how SRE behaves when the original feedback is inaccurate.

  3. The generalizability of the method remains unclear. SRE shows relatively clear gains on research tasks, but its benefits on workplace tasks are limited, and its General Capabilities scores also decrease after adaptation.

  4. The realism of the environments and tasks remains uncertain. Tasks are generated automatically, and a user agent with access to hidden rubrics evaluates the results and simulates user feedback. It is unclear how well this setup reflects natural interactions with real users.

Questions for authors

See Weaknesses.

7 comments
AuthorsOpenReview ↗

Response by the authors (1/4)

Thanks for your valuable comments. Below please find our responses. We will start with W3 because we fixed an issue in our training reward implementation during the rebuttal period, which updates the main results.

W3 Updated workspace results and general capabilities

  1. The generalizability of the method remains unclear. SRE shows relatively clear gains on research tasks, but its benefits on workplace tasks are limited, and its General Capabilities scores also decrease after adaptation.

Thank you for pointing this out. After double checking our submitted implementation, we found an issue in the training reward system. Sometimes the agent fails to submit deliverables in a valid format, and instead outputs the content in other ways. For example, when asked to submit a Word document (.docx), the agent may write raw Markdown bytes directly into the file instead of a valid Word binary. Although the file itself is invalid, the user agent may still give partial credit when scoring with the rubric if the text content is right. During training, the agent gradually hacks this loophole.

To fix it, we add a deliverable check: if the agent fails to produce valid artifacts, it directly gets 0 reward. This acts as a multiplicative gate on the reward, similar to the agentic RL reward design in MiMo-V2.6 [1]. As shown in https://anonymous.4open.science/r/SRE-rebuttal-figs/ws_reward_gate_before_after.pdf (Gemma4-E4B-IT with REINFORCE++ on DEAL-WS), before the fix, the reward drops sharply after step 40 and the entropy blows up. After the fix, the entropy stays stable and the reward keeps improving.

This reward fix only affects the training-based methods (REINFORCE++ and SRE), while Base and the training-free baselines remain unchanged. We re-run REINFORCE++ and SRE on both domains. Experimental results can be found in the following table.

DEAL-RSOnline GainComp.Align.Verify.WorkflowAuditAvg.AARRI-Bench (OOD)General Cap.
Base0.0016.6827.9833.8614.0918.3921.3035.3852.91
EvoSkill-1.0726.0429.9034.6119.7124.7026.9030.88–
CASCADE+2.4925.5223.9337.3016.3819.5924.9335.25–
REINFORCE+++5.6331.3443.4055.2629.6829.6436.9537.0152.97
SRE (Ours)+10.6636.9147.8955.9328.6541.6341.2238.6055.16
DEAL-WSOnline GainOps.Log.Res.Dev.Prod.Avg.PinchBenchClawGym-BenchGeneral Cap.
Base0.0048.5244.7838.3437.0232.5540.2455.1146.6152.91
EvoSkill-6.5047.5849.2235.3538.6133.7440.9050.8746.86–
CASCADE+0.9252.5051.0832.7054.8436.8445.5956.1846.26–
REINFORCE++-0.4442.3244.2741.3661.3133.4244.5457.9947.4452.98
SRE (Ours)+4.5449.2251.6745.6150.6936.9046.8159.5450.2752.87

Here we observe that SRE achieves the best online gain, held-out average, and OOD scores on both DEAL-RS and DEAL-WS (and similarly on Qwen3.5-9B; Figure 6 in Section 5.3). For general capabilities, under the standardized weight-only evaluation across all seven benchmarks (Table 7 in Appendix E.3), SRE preserves general capabilities without forgetting, improving over the base model on DEAL-RS and remaining comparable against the base model on DEAL-WS.

We have updated Section 3.3, Section 5.2 (Table 1 and Figure 4), Appendix D.2 (Figure 9), and Appendix E.3 (Table 7) in our revised paper to make these results clear. Thank you very much!

AuthorsOpenReview ↗

Response by the authors (2/4)

W1 Comparison with a task-only rubric baseline

  1. Although the motivation is reasonable, saving task snapshots and allowing repeated offline execution makes a straightforward baseline possible. One could generate multiple rollouts, construct task-specific rubrics from the task alone, use a strong LLM judge to score the final results, and train with GRPO. The paper does not compare against this baseline, making it difficult to establish whether the gains come from the proposed method or from the richer trajectories provided by additional rollouts.

Thanks for your insightful suggestion! We sincerely agree that this is an important baseline. However, in our deployment-time setting, calling a strong external LLM judge does not fit our motivation: if users can afford a strong LLM, they would not need to deploy the small self-hosted model, and it would also incur high API costs. Therefore, we implement this baseline by directly using our model's own self-reward without user feedback. Specifically, we construct the rubric from the task alone while keeping the same K=7 additional rollouts and GRPO update (i.e., SRE w/o anchor).

In our initial submission, Figure 6 has already compared them on fixed candidates. Following your suggestion, we train this baseline end-to-end on DEAL-RS. Experimental results can be found in the following table.

Method (DEAL-RS)DEAL-RS Held-outAARRI-Bench
Base21.3035.38
REINFORCE++36.9537.01
SRE w/o anchor (task-only rubric)37.3937.25
SRE (Ours)41.2238.60

Here we observe that SRE outperforms w/o anchor on both held-out tasks and AARRI-Bench. Since both methods use the same additional rollouts, this gain comes from the feedback anchor.

As also shown in Figure 6 of our initial submission and https://anonymous.4open.science/r/SRE-rebuttal-figs/anchor_rank_signflip_2cond.pdf, on 100 randomly sampled task groups, adding the anchor improves within-group Spearman correlation from 0.6770.677 to 0.8160.816 and reduces the advantage sign-flip rate from 20.2520.25\\% to 14.1214.12\\%, showing the necessity of ground-truth user feedback.

We have included this baseline and analysis in Section 5.3 (Figure 5a), Appendix E.1 (Figure 10), and Appendix E.4 (Table 8) to strengthen our discussion. Thank you again for your valuable suggestion.

AuthorsOpenReview ↗

Response by the authors (3/4)

W2 Robustness to incorrect feedback

  1. The method starts from a single user feedback signal, but that feedback is not necessarily correct. Could using it to assign self-rewards across multiple rollouts amplify an incorrect judgment? The paper would benefit from examining how SRE behaves when the original feedback is inaccurate.

Thanks for your good question. In real-world deployment, users can indeed give inaccurate or noisy feedback (e.g., due to misjudgment or oversight), and an incorrect anchor may affect the self-rewards of multiple rollouts. Thus, we agree that it is worth validating how the agent learns with incorrect judgment. To examine this, we evaluate both training performance and reward quality under incorrect feedback.

Firstly, with probability p∈0.2,0.5,0.8p\in\\{0.2,0.5,0.8\\}, we add a system prompt instructing the user agent to simulate a real user giving inaccurate feedback (prompt in Appendix G.3), while keeping held-out evaluation clean. Due to limited rebuttal time and computational resources, we train for 20 steps on DEAL-RS. Experimental results can be found in the following table.

Setting (DEAL-RS, 20 steps)DEAL-RS Held-out Avg.
SRE w/o anchor (task-only rubric)28.23
SRE, p=0.8p=0.8 incorrect feedback28.95
SRE, p=0.5p=0.5 incorrect feedback29.34
SRE, p=0.2p=0.2 incorrect feedback30.27
SRE, clean feedback33.53

We find that performance drops smoothly as pp grows, but stays above w/o anchor even at p=0.8p=0.8. Besides, as shown in https://anonymous.4open.science/r/SRE-rebuttal-figs/anchor_rank_signflip_3cond.pdf, on the 100 randomly sampled groups, incorrect feedback gives a within-group Spearman ρ=0.672\rho=0.672 and sign-flip rate 21.0221.02\\%, which are worse than SRE w/ anchor.

Overall, although SRE under wrong feedback does not match the clean-feedback version, it can still train robustly and consistently beat the w/o anchor baseline. Upon closer inspection, the reasons are three-fold:

  • Robust rubric-and-scoring pipeline: the self-rubric agent still reads the original task description, so basic task requirements are kept in the rubric even when the feedback is noisy.
  • Complete live attempt in the anchor: besides the user comment, the anchor also contains the live trajectory and its deliverables. Seeing this attempt helps the self-rubric agent know which files and deliverables to check when designing the rubric, which explains why it still beats w/o anchor.
  • Shared reward bias can be cancelled in GRPO: we only pass the anchor to the self-rubric agent, not to the self-scoring agent. As shown in our theoretical analysis (Proposition 1 in Section 4.3), a shared group-level reward shift bxb_x does not change the GRPO group-centered advantages, so the relative ranking among rollouts stays largely intact.

We have incorporated these experiments and discussions into Section 5.3 (Figure 7b), Appendix E.1 (Figure 10), Appendix E.4, and Appendix F.3 of our revised paper. Thank you very much.

Comments on the paper

2 comments

Dear Reviewers,

The authors have posted their responses and a revised manuscript addressing your questions and concerns. Please review their rebuttal carefully, let them know if any concerns remain, and update your review accordingly.

Best regards,

Area Chair

AuthorsOpenReview ↗

General Response

Dear Reviewers and Area Chair,

Thank you for your careful reviews and constructive discussions. In the revised PDF, initial revisions are highlighted in red and follow-up updates in blue.

Deliverable format gate and updated main results (TGdZ, fcDZ, EcEp). We identified and fixed a training-time format reward-hacking issue by adding a deterministic deliverable check before rubric scoring, stabilizing policy entropy and reward growth (Section 3.3, Appendix D.2, Figure 9). Re-evaluating across both domains and both model families (Gemma4-E4B-IT and Qwen3.5-9B), SRE achieves the best online utility gain, held-out ID average, and external OOD performance across all benchmarks (Sections 5.2-5.3, Table 1, Figures 4 and 6), while preserving General Capabilities without catastrophic forgetting (Appendix E.3, Table 7).

Controlled baselines and feedback anchoring (EcEp, fcDZ, TGdZ). On DEAL-RS under the same offline rollout budget, end-to-end SRE clearly outperforms the task-only rubric baseline (SRE w/o anchor), feedback-conditioned distillation (SDPO), and compute-matched REINFORCE++ on both held-out and OOD benchmarks (Section 5.3, Figure 5a, Appendix E.4, Table 8). Diagnostics on randomly sampled task groups further confirm that the feedback anchor improves within-group reward rank correlation and reduces advantage sign flips (Appendix E.1, Figure 10).

Rollout budget, training cost, and seed robustness (EcEp, fcDZ). End-to-end performance improves steadily as the offline rollout count increases, consistent with our offline coverage-gain and gradient-alignment analyses (Section 5.3, Figure 5b, Appendix E.2, Figure 11). Additional runs with an independent random seed and shuffled online task order confirm consistent gains with low variance (Appendix E.4, Table 8).

Feedback efficiency and imperfect-feedback robustness (EcEp, TGdZ, fcDZ). Evaluating early deployment checkpoints on DEAL-RS shows that SRE improves rapidly after a brief warm-up and approaches the final performance of REINFORCE++ in early stage (Section 5.3, Figure 7a, Appendix E.4). Under missing feedback, scalar-only scores, and varying levels of inaccurate textual feedback, SRE degrades lightly without amplifying wrong judgments and remains effective, supported by qualitative rubric-grounding case studies (Section 5.3, Figure 7b, Appendices E.4 and F.3).

Other revisions. Section 4.2 and Appendix D.1 (Table 2) distinguish the external user simulator from the frozen base-model self-rubric and self-scoring agents, and Appendix G adds the prompt templates, schemas, and scoring aggregation rules. Appendix C.2 (Figure 8d) fixes the workspace task-count typo. Sections 1, 3.1, 4, and 6 sharpen the positioning of our overall framework and related work, Sections 3.3 and 7 discuss selective feedback filtering and lightweight parametric memory for future work.

Thank you again for your insightful comments, which have helped us substantially improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.