- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how ordinary, non-adversarial noise introduced during LLM agent skill self-evolution affects learned skills and subsequent task performance. The authors introduce RSE-Bench, covering six benchmarks across four domains, with noise injected into instructions, environmental evidence, execution records, and outcome–record pairings. The protocol separates learning-time noise from evaluation-time noise and evaluates frozen skill stores on held-out tasks. The authors also introduce RSEClaw, a unified framework that records noise exposure and skill updates. Experiments with three methods and two backbones show that clean self-evolution improves macro-average performance by 2.6–6.5 percentage points, while the noisy-learning conditions retain only 25.5%–42.1% of the aggregate improvement. Record and pairing noise produce average losses comparable to those caused by input-side noise.
Critical strengths
-
The evaluation design explicitly distinguishes learning-time effects from execution-time effects. Comparing noisy learning followed by clean evaluation with clean learning followed by noisy evaluation allows the paper to examine both the downstream utility of skills learned under noise and the execution robustness of frozen skills. Matched task identities, a separate final-test split, and frozen skill stores provide a useful evaluation structure for studying noise in skill self-evolution.
-
The experiments cover multiple skill-use settings and self-evolution methods. The six benchmarks span spreadsheet manipulation, document understanding, interactive decision-making, and sequential skill reuse. The main experiments compare three methods and two backbones and report means and standard deviations over three runs of the complete pipeline. This coverage helps reveal similarities and differences across methods and tasks, while record and pairing noise extend evaluation beyond input perturbations.
Critical weaknesses
-
The description of how N-Rec reaches RethinkSkill’s learner contains a substantive inconsistency that undermines the interpretability of a central experiment. Appendix D.3 describes RethinkSkill’s learning evidence as an evaluation ledger containing success labels, scores, and evaluator-provided reasons. Appendices D.4 and F.1 specify that N-Rec deletes only assistant text and tool outputs while preserving evaluator feedback. Appendix D.5 explicitly notes that a method reading experience only through its evaluation ledger would not be exposed to trajectory perturbations. Nevertheless, Table 9 reports 100% input exposure for RethinkSkill under N-Rec, and Table 10 reports corresponding performance changes. The paper does not explain how the deleted content enters the actual optimizer input. Consequently, it is currently unclear whether this condition operates as defined and whether the observed changes can be attributed to record noise. This directly affects confidence in the cross-method N-Rec findings and the framework’s method adaptations.
-
The benchmark’s central task-preservation guarantee lacks sufficiently verifiable validation results. Section 3.2 defines N-Inst as background information and non-binding suggestions that leave the effective request unchanged. However, Appendices F.1–F.2 explicitly permit unrelated chore requests and do not treat their presence alone as grounds for rejection. The paper does not provide a clear operational criterion for ensuring that these requests introduce no additional task objectives; preserving the original question, answer, and evaluator does not establish this by itself. Appendix F.3 describes a human audit of 30 instances per benchmark and noise type but reports no observed pass rates, failure categories, or annotator agreement. The available evidence therefore does not sufficiently establish that these instances consistently preserve the original goals and deliverables, which affects the validity of interpreting the performance losses as effects of task-preserving noise.
Questions for authors
- Which fields in RethinkSkill’s actual optimizer input are modified by N-Rec, and how do these fields correspond to the evaluation ledger described in Appendix D.3?
- How is the IER in Table 9 computed from optimizer-request logs? What specific prompt differences were observed in the paired replay of RethinkSkill under N-Rec?
- How do the generation and review procedures ensure that the unrelated chore requests permitted under N-Inst remain non-binding and introduce no additional task objectives?
- What are the human-audit pass rates, failure categories, and annotator agreement for each benchmark and noise type in Appendix F.3?