- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces SCLATE, an execution framework for evaluating and post-training continual-learning agents across long, multi-session interactions. It integrates existing benchmarks and agent harnesses through lightweight adapters and a shared event scheduler, while a hybrid simulated clock skips idle periods to accelerate evaluation. The authors validate scheduling consistency with GAIA-2, integrate seven benchmarks, and evaluate ten harness–memory configurations across ten models. Results reveal substantial model-dependent variation, with no universally optimal configuration and no consistent benefit from adding external memory. Finally, the paper demonstrates post-training Qwen3.5-4B using existing agent harnesses without modification.
Critical strengths
-
This paper addresses an important gap. SCLATE brings benchmark events and agent-side events onto a shared timeline without changing the agent harness, it makes long-running evaluations much faster.
-
The framework supports both evaluation and training. SCLATE records model outputs and log probabilities during evaluation, allowing the same unmodified agent harnesses to be used for post-training. The experiments show that training can improve both agent behavior and how agents manage memory.
Critical weaknesses
Key empirical findings rely on single runs. Each cell of Table 1 is a single run, and no variance is reported. The MetaClaw rounds have memory across days, so an early divergence can affect the whole run, so the result might be noisy. I discover that some of the cells where added memory underperforms native memory differ by only 0.3 pp, which makes the claim that added memory “does not reliably beat” native memory hard to assess. It would be helpful for the authors to report results over multiple runs with variance estimates.
Questions for authors
- Can you report repeated runs for at least a subtable of Table 1? Similarly, how stable are the post-training results across training seeds?
- Because simulated time flows with wall-clock time during generation, how sensitive are GAIA-2 Time and Adaptability results to inference latency? Does the same model give different scores when served from different providers? I am asking this question because it will impact reproducabiliy.
6 comments
(2/4) Response to Reviewer Yq1b
Q1. Key findings rely on single runs. Some cells where added memory underperforms native memory differ by only 0.3 pp. Can you report repeated runs for at least a subtable of Table 1?
A1. Thank you for the suggestion. Yes, every cell, not only a subtable: each cell of Table 1 is now the mean of three independent 30-workday arcs (Appendix D.3). As you anticipated, memory configurations diverge more (median spread 5.2 pp, against 1.7 pp for No-Mem). The claims hold against the arc spread (Welch's t-test, 95%; see also fKrT Q2), and we restate the claim in Section 4.2:
| Claim | Against the arc spread |
|---|---|
| Every memory configuration beats its harness's No-Mem | higher in 70/70 cells, significantly in 58/70; every arc above every No-Mem arc in 66/70 |
| An added memory system does not reliably beat native memory | significantly better in 15/50, never significantly worse; largest shortfall 3.3 pp |
| No memory system is best for every model | the leader changes across models, and is separable from the runner-up in 2 of 20 model-harness pairs |
(3/4) Response to Reviewer Yq1b
Q2. How stable are the post-training results across training seeds?
A2. Thank you for asking. Across four training runs and four evaluations of the base, Codex SWE-bench Verified Pass@1 rises from 27.7% ± 1.2 to 35.7% ± 2.4, a gain of +8.0 pp. We trained the Codex SWE-Gym configuration three more times with different seeds. Each run's Pass@1 is below, beside an evaluation of the base; Appendix E.2 plots their rollout reward (see also fKrT Q4).
| Evaluation | Base Pass@1 | Trained Pass@1 |
|---|---|---|
| Reported run | 28.8% | 33.6% |
| Repeat 1 | 27.6% | 39.0% (seed 2) |
| Repeat 2 | 26.0% | 36.0% (seed 3) |
| Repeat 3 | 28.4% | 34.2% (seed 4) |
| Mean ± SD | 27.7% ± 1.2 | 35.7% ± 2.4 |
The repeats share one evaluation setup, on which the three new runs gain +9.1 pp, and each beats every base evaluation.
On MetaClaw, three more training runs each of Hermes + Hindsight and Claude Code + Mem0 all improve on the base on the held-out days, by 6.1 to 11.5 pp (Appendix E.4; see also fKrT Q4):
| Held-out days 21–30, epoch 5 | Base | Reported | Repeat 1 | Repeat 2 | Repeat 3 | Mean |
|---|---|---|---|---|---|---|
| Hermes + Hindsight | 10.1% | 20.2% | 18.8% | 18.8% | 16.2% | 17.9% |
| Claude Code + Mem0 | 10.1% | 21.8% | 21.6% | 21.0% | 21.6% | 21.4% |
(1/4) Response to Reviewer Yq1b
Dear Reviewer Yq1b,
We thank you for the thorough review, and we are glad you found that SCLATE addresses an important gap, bringing benchmark and agent-side events onto one timeline without changing the harness, for both evaluation and post-training. We respond to each point below; changes in the revised paper appear in blue.
SCLATE introduces a new paradigm for evaluating and training continual-learning agents such as Muse, Dots and Grokbot [1]: the benchmark and the agent are peers on one open scheduler with a shared clock, so the agent's own events (session stop and start, crons, memory consolidation) run alongside the benchmark's, as deployed. Each benchmark and each agent is integrated once, at O(m+n) in place of O(m×n) for m benchmarks and n agents, after which any heterogeneous agent, with its unmodified harness and memory system, runs on any integrated benchmark, and the same runs serve evaluation and training.
What SCLATE brings. To our knowledge, SCLATE is the first framework in which a continual-learning agent's own events are first-class:
What this enables. Together, these let researchers compare heterogeneous continual-learning agents, each as a whole system, on the same long-horizon, multi-session benchmarks, and train a model through the harness and memory system it is deployed with, neither of which a prior framework supports; we show both with seven ported benchmarks, a head-to-head comparison of ten harness and memory configurations on ten models (Table 1), and post-training through unmodified harnesses and memory (Section 4.3).
Summary. Every cell of Table 1 now has three independent arcs (Q1); we added three more training runs on SWE-Gym and on MetaClaw (Q2); and slowing every agent model call shows that GAIA-2 Time depends strongly on response time when its timing checks are enforced, by GAIA-2's design, and little when they are off (Q3).