- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper aims to investigate the failure modes of self-evolving agents while they repeatedly revise their own harness/workflow or their own weights during online interaction with environments with different styles of feedback. The paper's goal is also to test whether self-revising agents can retain information needed to find, reverse, or repair their own mistakes. The paper tests a few settings. The first one is a workflow revision based on the EvoAgentX implementation of AFlow on the tau^2 Bench. They tested the model revising its own harness while receiving feedback from the environment. They show that the agent can make significant failures when revising their own workflow, causing catastrophic degradation in the agent's real functionality, but the validation score remains unchanged since doing nothing passes some of the validation tasks. This happens only under the accept-all modification workflow, not the stock archive-based parent selection workflow. The second setting is on prompt engineering inspired by IFEval and compares different types of feedback. The paper tries to show that knowing what is wrong does not mean the model knows how to fix the requirement. The third task is HTTP 503 failures assigned with some probability to a tool used by the agent. The authors show that with traces, the models can identify the problems, but without traces the models fail to correct that. The last task is using STaR-style self-training with LoRA. The paper shows that with repeated training, the Qwen 3 0.6B model would converge to selecting almost all "A" on MedMCQA.
Critical strengths
Self-evolving agents may fail even when validation scores indicate similar levels. This is the main point the paper tries to show and experiments are built around this point.
Critical weaknesses
Although the paper presents examples of degradation that monitored scores fail to expose, the experiments do not sufficiently establish how broadly these failures occur or whether they reflect a common mechanism of impaired recovery following self-modification.
- The workflow experiments do not adequately isolate loss of recoverability caused by self-modification from limitations of the feedback and harness design. My concern here is, for instance, if the agent can assess a history of its past revisions and failure reports, then it should be granted this skill otherwise, it feels more of a harness problem to me. In the tool-outage experiment, providing error traces enables the optimizers to identify the fault, whereas score-only feedback does not. Successful diagnosis does not guarantee repair, but this comparison primarily establishes the importance of making diagnostic information available. As acknowledged in Appendix D, it does not demonstrate a previously functioning diagnostic channel being destroyed by self-modification: fault detection is the same for the original and modified checkpoints under each feedback condition. Furthermore, the headline workflow collapse occurs after replacing AFlow’s archive-based selection with an accept-all policy, while the stock selected workflow remains functional. This means that validating before applying fixes is important but does not mean the model cannot fix the issues with good context management or feedback. These results leave unclear how much the observed failure generalizes to systems retaining standard selection and diagnostic mechanisms.
- The parameter-update experiments provide limited evidence about recovery from self-induced damage. The STaR-style experiments demonstrate that correctness-filtered training can cause Qwen3-0.6B to converge toward a constant answer while accuracy remains near the majority-answer baseline. However, they primarily examine the emergence and prevention of collapse, rather than whether an already-collapsed model can recover under different corrective interventions. What if we consider a simple workflow: we do checkpointing and prevent the model from collapsing by reverting the training if we find the validation scores on answer choices uniformly distributed validation questions drop or the answer converges to a single choice? Their connection to the paper’s central recovery question is therefore less direct than that of the prompt experiments. Moreover, Llama-3.2-3B does not collapse under the tested training intensity, suggesting that the phenomenon depends on the model’s initial competence and its distance from the majority-answer baseline. The manuscript should clarify these boundary conditions and explain which conclusions about recoverability the parameter experiments independently support.
Questions for authors
- What are the evidences you find most convincing that shows self-modification or workflow revision reduces subsequent recoverability with a better harness design, if any?
- What conclusion about recovery is supported by the STaR self-training experiment?
These are questions that I have for the authors. However, I find it might be hard for me to further improve the scores unless new experiments or insights are added.