acceptodds
Under review as a conference paper at ICLR 2027

When Non-Adversarial Noise Hurts Learning: Benchmarking Skill Self-Evolution in Agents

Abstract

Large language model agents increasingly improve by distilling experience into reusable skills, but errors in that experience can persist in learned rules and affect later tasks. Existing evaluations largely examine clean learning or perturbations encountered after skills are frozen, leaving robustness during self-evolution insufficiently understood. We introduce RSE-Bench, to our knowledge the first benchmark for evaluating skill self-evolution under ordinary, non-adversarial noise. It spans four domains and six benchmarks, with controlled perturbations to instructions, environmental evidence, execution records, and outcome–record pairings that preserve the underlying tasks and native scoring rules. The protocol separates noise during learning from input-side noise at evaluation, testing frozen skill stores on held-out tasks. We also introduce RSEClaw, a method-agnostic framework supporting consistent noise injection and process diagnostics of actual exposure and skill updates. Across three methods and two backbones, noisy learning retains only – of the aggregate gain achieved by clean self-evolution, depending on the noise type. Perturbations confined to execution records and outcome–record pairings cause average losses comparable to input-side noise, despite leaving the original executions and evaluations intact. Diagnostics further identify differences in candidate skill text that coexist with small changes in mean held-out performance.

Primary area:
datasets and benchmarks
Keywords:
Skill Self-evolution, Benchmark, Agent, LLM, MLLM

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zoDZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how ordinary, non-adversarial noise introduced during LLM agent skill self-evolution affects learned skills and subsequent task performance. The authors introduce RSE-Bench, covering six benchmarks across four domains, with noise injected into instructions, environmental evidence, execution records, and outcome–record pairings. The protocol separates learning-time noise from evaluation-time noise and evaluates frozen skill stores on held-out tasks. The authors also introduce RSEClaw, a unified framework that records noise exposure and skill updates. Experiments with three methods and two backbones show that clean self-evolution improves macro-average performance by 2.6–6.5 percentage points, while the noisy-learning conditions retain only 25.5%–42.1% of the aggregate improvement. Record and pairing noise produce average losses comparable to those caused by input-side noise.

Critical strengths
  1. The evaluation design explicitly distinguishes learning-time effects from execution-time effects. Comparing noisy learning followed by clean evaluation with clean learning followed by noisy evaluation allows the paper to examine both the downstream utility of skills learned under noise and the execution robustness of frozen skills. Matched task identities, a separate final-test split, and frozen skill stores provide a useful evaluation structure for studying noise in skill self-evolution.

  2. The experiments cover multiple skill-use settings and self-evolution methods. The six benchmarks span spreadsheet manipulation, document understanding, interactive decision-making, and sequential skill reuse. The main experiments compare three methods and two backbones and report means and standard deviations over three runs of the complete pipeline. This coverage helps reveal similarities and differences across methods and tasks, while record and pairing noise extend evaluation beyond input perturbations.

Critical weaknesses
  1. The description of how N-Rec reaches RethinkSkill’s learner contains a substantive inconsistency that undermines the interpretability of a central experiment. Appendix D.3 describes RethinkSkill’s learning evidence as an evaluation ledger containing success labels, scores, and evaluator-provided reasons. Appendices D.4 and F.1 specify that N-Rec deletes only assistant text and tool outputs while preserving evaluator feedback. Appendix D.5 explicitly notes that a method reading experience only through its evaluation ledger would not be exposed to trajectory perturbations. Nevertheless, Table 9 reports 100% input exposure for RethinkSkill under N-Rec, and Table 10 reports corresponding performance changes. The paper does not explain how the deleted content enters the actual optimizer input. Consequently, it is currently unclear whether this condition operates as defined and whether the observed changes can be attributed to record noise. This directly affects confidence in the cross-method N-Rec findings and the framework’s method adaptations.

  2. The benchmark’s central task-preservation guarantee lacks sufficiently verifiable validation results. Section 3.2 defines N-Inst as background information and non-binding suggestions that leave the effective request unchanged. However, Appendices F.1–F.2 explicitly permit unrelated chore requests and do not treat their presence alone as grounds for rejection. The paper does not provide a clear operational criterion for ensuring that these requests introduce no additional task objectives; preserving the original question, answer, and evaluator does not establish this by itself. Appendix F.3 describes a human audit of 30 instances per benchmark and noise type but reports no observed pass rates, failure categories, or annotator agreement. The available evidence therefore does not sufficiently establish that these instances consistently preserve the original goals and deliverables, which affects the validity of interpreting the performance losses as effects of task-preserving noise.

Questions for authors
  1. Which fields in RethinkSkill’s actual optimizer input are modified by N-Rec, and how do these fields correspond to the evaluation ledger described in Appendix D.3?
  2. How is the IER in Table 9 computed from optimizer-request logs? What specific prompt differences were observed in the paired replay of RethinkSkill under N-Rec?
  3. How do the generation and review procedures ensure that the unrelated chore requests permitted under N-Inst remain non-binding and introduce no additional task objectives?
  4. What are the human-audit pass rates, failure categories, and annotator agreement for each benchmark and noise type in Appendix F.3?
Reviewer_vbQdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RSE-Bench, a benchmark that tests for skill self-evolution under non-adversarial noise. RSE-Bench is constructed by injecting non-adversarial perturbations to 6 public agent benchmarks across 4 domains (spreadsheet manipulation, document understanding, interactive decision-making, sequential skill reuse). Noise might be bad in real-life settings as agents might learn the wrong thing and propagate it to future tasks.

The authors test noise injection at several entry points (task instructions, execution-time evidence, record to learner, outcome-reward pairings), and separate learning vs. evaluation noise. They also introduce RSEClaw, a diagnostic framework that conducts noise injection and tests exposure updates. Predictably, injecting noise degrades self-evolution performance on two models they test (DeepSeek V4.1 Flash and GPT 5.2). But a more interesting claim is that noise acquisition can affect future performance, even when the evaluation is clean.

Critical strengths
  • I think the idea is interesting and practical. In real-world scenarios, some (innocuous) noise is inevitable to many agentic tasks.
Critical weaknesses
  • Novelty: It’s not entirely unsurprising that (even irrelevant) noise in context can damage end task performance, and also persistent skills acquired through self-evolution. Section 5.4. (the Analysis section) feels like a bullet-point list of observations to me, and I think the paper would benefit more from a more detailed analysis of why different noise types affect persistent skill acquisition, beyond just documenting the extent of performance degradation. Personally, I would feel more comfortable vouching for the paper if it gives us a better understanding beyond what is obvious.
  • Presentation: The writing is pretty hard to follow throughout the draft. Concretely, I think the entire Section 3.2 (Task Taxonomy) is very jargon-dense and could be rewritten in simpler, more concise language, and the results and analyses sections as well.
  • Model benchmarking: I feel that 2 model backbones is insufficient for a benchmark paper…not sure why you didn’t test a broader range of open and closed models?
Questions for authors
  • Sometimes you choose specific values (e.g., record corruption removes 25% of characters from execution traces) without justification or ablation. Could you explain why you choose these specific values?
  • Are your benchmark results statistically significant across multiple seeds? I noticed you used a fixed seed (42) in several places….
Reviewer_BLAVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RSE-Bench and the RSECLAW framework to evaluate the robustness of LLM agents' skill self-evolution under non-adversarial noise. By decoupling learning-side noise from evaluationary noise and testing frozen skill stores on different tasks, the study demonstrates that noisy learning severely erodes self-evolution gains.

Critical strengths
  1. The motivation is clear and easy to understand.

  2. The ablation study is complete.

Critical weaknesses
  1. The evaluated methods rely heavily on LLMs for reflection and curation. Are the observed performance drops caused by structural flaws in the self-evolution loop?

  2. When performance drops under noise, is it due to active negative learning or validation gates rolling back the entire round to the baseline?

  3. The computational and financial cost across multiple passes, backbones, and noise conditions should be reported.

  4. Since the selection partition also undergoes noise perturbation, is there a risk that the validation gate overfits to noise characteristics rather than the generalization?

  5. The paper utilizes specific noise parameters. How sensitive are the main conclusions to variations in noise intensity?

  6. Why does document understanding suffer so little from noise compared to multi-step tasks?

  7. In sequential skill reuse (e.g., SkillFlow), minor errors learned from tasks might be enhanced in subsequent steps.

  8. RSECLAW uses the same model as both the actor and the optimizer. Using a stronger model for reflection and a weaker one for execution, do the noise robustness conclusions change?

  9. The noise in RSE-Bench is synthetically injected using rules and a model. Do these controlled synthetic perturbations truly represent the complex and long-tail noise of real-world environments?

Questions for authors

See the Weaknesses part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.