- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EvoRelay, an online experience-learning framework for long-horizon agents. It organizes experience around functional stages and asynchronously extracts and propagates stage-level memories, allowing later tasks to use them before the source task finishes. Experiments on SWE-bench Verified and AppWorld with four backbone models show moderate task-success improvements and large reductions in end-to-end stream time.
Critical strengths
- The paper identifies a real systems bottleneck: newly acquired experience may remain unavailable until the source task finishes.
- The evaluation is reasonably broad, covering two interactive benchmarks, four backbone models, multiple memory baselines, task success, runtime, ablations, and stage-wise behavior.
Critical weaknesses
- Limited methodological novelty. Category filtering, stage-level experience extraction, and online updates substantially overlap with the existing approach, so stage-aligned memory itself does not appear to be a new underlying principle. The more distinctive part is asynchronous visibility and pipeline scheduling, but this feels primarily like a systems-level execution optimization rather than a new experience-learning mechanism.
- The paper does not sufficiently establish that the asynchronous scheduling result is robust beyond the reported setting.
Questions for authors
- Please clarify how SASM was implemented and whether the reported baseline faithfully follows the original method. Could the authors provide a controlled comparison that keeps SASM’s stage-level memory mechanism fixed and changes only the experience-availability rule from task-completion-based to stage-update-based?
- How stable is the reported 2.9× runtime improvement across backbone models, task orders, numbers of workers, and experience-extraction delays? Please report repeated-run means, variance, and significance tests.
- Stage declarations are generated dynamically, but the experiments also rely on predefined stage categories. It would be better to report stage-declaration errors and test sensitivity to alternative stage-category definitions.
- Under the append-only memory design, how are duplicate, conflicting, or incorrect experiences handled as the task stream grows?
I appreciate the authors' efforts in reporting the new statistics. I am inclined to keep my original score.