- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AgentStream, a framework for evaluating self-evolving LLM agents under continuous task streams. It defines three scenarios—Isolated, Sequential, and Interleaved—and evaluates five self-evolving methods across three foundation models and six agent benchmarks. The results suggest that self-evolution is not uniformly beneficial: its effectiveness depends on model capability, method architecture, and stream composition. Context-integrated methods generally favor isolated streams, whereas retrieval-based methods appear more robust under interleaved streams.
Critical strengths
-
Studies an important limitation of conventional independent evaluation for self-evolving agents.
-
Covers a broad combination of models, methods, benchmarks, and streaming scenarios.
-
Reports negative evolution effects and cost increases rather than focusing only on successful configurations.
-
Provides per-seed results, cumulative performance curves, cost analysis, state-evolution analysis, and complete prompts.
-
Offers useful empirical observations about the interaction between method architecture and stream composition.
Critical weaknesses
-
Sequential evaluation uses only one fixed benchmark order, so its results may be specific to that ordering.
-
Many reported gains are small relative to the observed variance, but no paired confidence intervals or significance tests are provided.
-
The capability-gating and non-monotonicity claims are based on only three models and an average over heterogeneous benchmark metrics.
-
Isolated states contain at most 50 tasks, while shared states may accumulate 300 tasks. This confounds domain composition with state size, age, and possible context truncation.
-
Comparisons may be affected by method adaptations, the custom Harness implementation, and different model reasoning configurations.
Questions for authors
-
Why is Sequential evaluated with only one benchmark ordering? Do the conclusions hold under alternative orders?
-
What are the 45 statistical units in Table 2, and does the reported ± denote standard deviation, standard error, or a confidence interval?
-
Were the vanilla baselines repeated with the same random seeds and decoding settings as the evolving methods?
-
How do the authors justify capability gating and non-monotonicity from only three model observations?
-
Are state or token budgets normalized across scenarios? How are truncation and retrieval capacity handled as the state grows?
-
Does the cost analysis include reflection, state updates, embedding, retrieval, and skill or memory maintenance? Why is Claude omitted?
2 comments
Thank you for appreciating our work and for raising your score accordingly. We also value the effort you put into reviewing our submission.
Thanks for the additional results and response. I will raise my score.