acceptodds
Under review as a conference paper at ICLR 2027

SCLATE: A SUBSTRATE FOR CONTINUAL-LEARNING AGENT TRAINING AND EVALUATION

Abstract

Continual-learning agents are systems of models, harnesses, and memory operating over long multi-session horizons. Evaluating and training them requires interleaving tasks with agent-side events such as session stop and start, crons, and memory consolidation. Yet existing benchmarks and training frameworks schedule only the benchmark's own events, leaving each benchmark and agent pair to build a custom scheduling loop. We present SCLATE, an execution substrate where benchmarks and unmodified agents each add their events to one open event scheduler through an adapter. A hybrid simulated clock runs these events on a shared timeline, flowing in real time while the agent works and skipping idle gaps, which compresses a month-long scenario into hours. SCLATE also serves as a rollout engine that runs any agent's harness and memory unmodified, recording the tokens and log probabilities of every model call through an in-container proxy. We port seven benchmarks to SCLATE and compare ten unmodified harness and memory configurations head to head on ten models. The comparison shows that an added memory system does not reliably beat the harness's native memory and that models differ widely in how they use the same harness and memory. We then post-train Qwen3.5-4B through unmodified harnesses and memory systems. The model learns to use both, reading 6.8× fewer file lines with a 16.7-point higher SWE-bench Verified pass rate, and writing richer memory records, while its held-out MetaClaw accuracy rises by up to 11.8 points.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Continual learning, Agentic evaluation, Post-training framework, harness agnostic

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Yq1bOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces SCLATE, an execution framework for evaluating and post-training continual-learning agents across long, multi-session interactions. It integrates existing benchmarks and agent harnesses through lightweight adapters and a shared event scheduler, while a hybrid simulated clock skips idle periods to accelerate evaluation. The authors validate scheduling consistency with GAIA-2, integrate seven benchmarks, and evaluate ten harness–memory configurations across ten models. Results reveal substantial model-dependent variation, with no universally optimal configuration and no consistent benefit from adding external memory. Finally, the paper demonstrates post-training Qwen3.5-4B using existing agent harnesses without modification.

Critical strengths
  1. This paper addresses an important gap. SCLATE brings benchmark events and agent-side events onto a shared timeline without changing the agent harness, it makes long-running evaluations much faster.

  2. The framework supports both evaluation and training. SCLATE records model outputs and log probabilities during evaluation, allowing the same unmodified agent harnesses to be used for post-training. The experiments show that training can improve both agent behavior and how agents manage memory.

Critical weaknesses

Key empirical findings rely on single runs. Each cell of Table 1 is a single run, and no variance is reported. The MetaClaw rounds have memory across days, so an early divergence can affect the whole run, so the result might be noisy. I discover that some of the cells where added memory underperforms native memory differ by only 0.3 pp, which makes the claim that added memory “does not reliably beat” native memory hard to assess. It would be helpful for the authors to report results over multiple runs with variance estimates.

Questions for authors
  1. Can you report repeated runs for at least a subtable of Table 1? Similarly, how stable are the post-training results across training seeds?
  2. Because simulated time flows with wall-clock time during generation, how sensitive are GAIA-2 Time and Adaptability results to inference latency? Does the same model give different scores when served from different providers? I am asking this question because it will impact reproducabiliy.
6 comments
AuthorsOpenReview ↗

(1/4) Response to Reviewer Yq1b

Dear Reviewer Yq1b,

We thank you for the thorough review, and we are glad you found that SCLATE addresses an important gap, bringing benchmark and agent-side events onto one timeline without changing the harness, for both evaluation and post-training. We respond to each point below; changes in the revised paper appear in blue.

SCLATE introduces a new paradigm for evaluating and training continual-learning agents such as Muse, Dots and Grokbot [1]: the benchmark and the agent are peers on one open scheduler with a shared clock, so the agent's own events (session stop and start, crons, memory consolidation) run alongside the benchmark's, as deployed. Each benchmark and each agent is integrated once, at O(m+n) in place of O(m×n) for m benchmarks and n agents, after which any heterogeneous agent, with its unmodified harness and memory system, runs on any integrated benchmark, and the same runs serve evaluation and training.

What SCLATE brings. To our knowledge, SCLATE is the first framework in which a continual-learning agent's own events are first-class:

  • An open event scheduler, the necessary component for this: benchmarks and agents both register events on it, ordered on one shared clock that compresses month-long scenarios into hours while preserving event order and timing.
  • Adapters for benchmarks and whole agents, with which any heterogeneous continual-learning agent runs as a whole system, its model, unmodified harness and memory system together, on any integrated long-horizon, multi-session benchmark, at O(m+n) in place of O(m×n).
  • One substrate for evaluation and training, whose runs serve both, with every model call attributed to its component.

What this enables. Together, these let researchers compare heterogeneous continual-learning agents, each as a whole system, on the same long-horizon, multi-session benchmarks, and train a model through the harness and memory system it is deployed with, neither of which a prior framework supports; we show both with seven ported benchmarks, a head-to-head comparison of ten harness and memory configurations on ten models (Table 1), and post-training through unmodified harnesses and memory (Section 4.3).

Summary. Every cell of Table 1 now has three independent arcs (Q1); we added three more training runs on SWE-Gym and on MetaClaw (Q2); and slowing every agent model call shows that GAIA-2 Time depends strongly on response time when its timing checks are enforced, by GAIA-2's design, and little when they are off (Q3).

AuthorsOpenReview ↗

(2/4) Response to Reviewer Yq1b

Q1. Key findings rely on single runs. Some cells where added memory underperforms native memory differ by only 0.3 pp. Can you report repeated runs for at least a subtable of Table 1?

A1. Thank you for the suggestion. Yes, every cell, not only a subtable: each cell of Table 1 is now the mean of three independent 30-workday arcs (Appendix D.3). As you anticipated, memory configurations diverge more (median spread 5.2 pp, against 1.7 pp for No-Mem). The claims hold against the arc spread (Welch's t-test, 95%; see also fKrT Q2), and we restate the claim in Section 4.2:

ClaimAgainst the arc spread
Every memory configuration beats its harness's No-Memhigher in 70/70 cells, significantly in 58/70; every arc above every No-Mem arc in 66/70
An added memory system does not reliably beat native memorysignificantly better in 15/50, never significantly worse; largest shortfall 3.3 pp
No memory system is best for every modelthe leader changes across models, and is separable from the runner-up in 2 of 20 model-harness pairs
AuthorsOpenReview ↗

(3/4) Response to Reviewer Yq1b

Q2. How stable are the post-training results across training seeds?

A2. Thank you for asking. Across four training runs and four evaluations of the base, Codex SWE-bench Verified Pass@1 rises from 27.7% ± 1.2 to 35.7% ± 2.4, a gain of +8.0 pp. We trained the Codex SWE-Gym configuration three more times with different seeds. Each run's Pass@1 is below, beside an evaluation of the base; Appendix E.2 plots their rollout reward (see also fKrT Q4).

EvaluationBase Pass@1Trained Pass@1
Reported run28.8%33.6%
Repeat 127.6%39.0% (seed 2)
Repeat 226.0%36.0% (seed 3)
Repeat 328.4%34.2% (seed 4)
Mean ± SD27.7% ± 1.235.7% ± 2.4

The repeats share one evaluation setup, on which the three new runs gain +9.1 pp, and each beats every base evaluation.

On MetaClaw, three more training runs each of Hermes + Hindsight and Claude Code + Mem0 all improve on the base on the held-out days, by 6.1 to 11.5 pp (Appendix E.4; see also fKrT Q4):

Held-out days 21–30, epoch 5BaseReportedRepeat 1Repeat 2Repeat 3Mean
Hermes + Hindsight10.1%20.2%18.8%18.8%16.2%17.9%
Claude Code + Mem010.1%21.8%21.6%21.0%21.6%21.4%
Reviewer_fKrTOpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

The paper presents SCLATE, a system for testing and training AI agents. They can work across many sessions over days/weeks and store memories. The authors point out that existing benchmarks control their own timeline and don't give the agent an opportunity for its own events (restarting between sessions, running scheduled jobs, compacting its memory overnight). SCLATE provides a remedy to this by thus letting both the benchmark and the agent add their events to a shared scheduler. Then, a simulated clock runs at normal speed while the agent works and skips ahead while nothing is happening. A month-long scenario could only take hours. A proxy inside each container records every model call, so the same runs can also be used to train the model. For evaluations, the authors incorporate 7 benchmarks. They check that their version of GAIA-2 gives the same results as the original. They then compare three agent harnesses (both with and without extra memory systems) across ten models. The authors find that adding a memory system is not necessarily better than the harness’s built-in memory. Also, different models end up using the same setup in rather different ways.

Critical strengths

S1. SCLATE is demonstrated to be compatible with 3 agent harnesses, 7 benchmarks, and 10 models. No agent code changes necessary. The authors do a good job of verifying their version of GAIA-2. Every scheduled event is delivered almost exactly on time. The results are shown to not differ from the original by more than two repeat runs. The simulated clock in SCLATE also reduces a month-long scenario into about 6 hours.

S2. SCLATE can train a model while it runs inside an unchanged agent and memory system. A proxy inside each container records every call the model makes. It then notes which part of the system made it (the agent itself or the memory system). This means that the model is able to later learn from its own runs. The authors show that this works with two different training methods (on coding tasks and on the 30-day MetaClaw scenario).

Critical weaknesses

W1. The core motivation, as I understand, lies in the fact that existing benchmarks cannot put agent-side events (crons, session restarts, memory consolidation) on the same clock as the benchmark. Yet I did not see any experiment that tests whether doing so changes any result. What the experiments show is that 1) the GAIA-2 port matches the original, 2) the clock speeds up long runs, and 3) the substrate can run a large grid. The authors never compare SCLATE with the old approach. Two simple ablations should be enough to do this. One would run MetaClaw with prompt-injected dates against SCLATE’s simulated clock. The other could then run GBrain with and without its nightly consolidation on the shared clock. It's also important to note that GBrain is the only memory system that schedules events between sessions, and the SWE-Gym experiment is single-session, so most of the evaluation never really makes use of the scheduler’s distinctive feature.

W2. Many results are reported based on single runs. A big portion of the findings come from Table 1, but it is unreliable to draw conclusions (“no configuration is best for every model”) from only one evaluation, along with no confidence intervals. Especially since in Figure 3d we see that two runs of the same setup disagree about 20% of the time. Many of the scores behind the conclusions barely differ, so they may as well be noise. Also, the GAIA-2 scores partly depend on latency from the API, since model generation uses up simulated time. The post-training results do have significance tests, but each configuration is again trained only once, which does not show variation across seeds. Also, for MetaClaw, only 3/6 improvements on the held-out days are significant. The large SWE-bench gain there mostly comes from the base model reaching the context limit on almost every task, not necessarily from a better use of the harness.

Questions for authors

Q1. How much would the GAIA-2 scores change if the same model were to run on a slower or faster endpoint?
Q2. The No-Mem baseline keeps session search over earlier transcripts. Gemini-3.7F seems to use this quite a bit. Is No-Mem then really a memory-free baseline?
Q3. If Hindsight or Mem0 is still extracting facts in its own process after the agent’s turn ends, can the clock jump before the write finishes? Could this make the memory state differ between runs?

11 comments
AuthorsOpenReview ↗

(1/7) Response to Reviewer fKrT

Dear Reviewer fKrT,

We thank you for the thorough review, and we are glad you found that SCLATE runs 3 harnesses, 7 benchmarks and 10 models with no agent code changes, that its GAIA-2 port delivers every scheduled event almost exactly on time, and that the same unmodified agent and memory system can be trained through the capture proxy. Following your comments, we add the GBrain ablation, three arcs for every Table 1 cell, the latency experiment, a stricter No-Mem baseline and three more training runs on SWE-Gym and on MetaClaw. We respond to each point below; changes in the revised paper appear in blue.

SCLATE introduces a new paradigm for evaluating and training continual-learning agents such as Muse, Dots, and Grokbot [1]: the benchmark and the agent are peers on one open scheduler with a shared clock, so the agent's own events (session stop and start, crons, memory consolidation) run alongside the benchmark's, as deployed. Each benchmark and each agent is integrated once, at O(m+n) in place of O(m×n) for m benchmarks and n agents, after which any heterogeneous agent, with its unmodified harness and memory system, runs on any integrated benchmark, and the same runs serve evaluation and training.

Summary. In short:

  • The shared clock leaves the benchmark as it is. SCLATE runs each benchmark's events as the benchmark defines them, and on GAIA-2 it delivers every event in order and on time, with outcomes matching upstream within run-to-run variation (Figure 3).
  • The shared clock is not an addition but the necessary component that makes the agent's own events first-class. It gives them a place on the same timeline, without which they fall out of step with the benchmark or never fire. What can be switched off, we ablated as you suggest: GBrain's nightly consolidation adds 3.1 pp (Q1).
  • Single runs. Every Table 1 cell now has three arcs, and the claims hold against their spread (Q2).
  • Latency. Yes, GAIA-2 Time depends on response time, and that is upstream GAIA-2's own design: with its timing checks enforced, Time falls from 32% to 4% with 5 s added per call, and with them off only from 62% to 58%; every score is reported with its serving latency (Q3).
  • Training. Three more SWE-Gym training runs and repeated base evaluations, and the context-limit split: for Codex and Hermes the gain holds without the limit, and for Claude Code working within its harness's context is what training teaches (Q4).
  • No-Mem. You are right: with earlier transcripts readable, Claude Code's No-Mem is not memory-free. We add a memory-free run with them hidden: Gemini-3.7F, which read them, drops 5.4 pp, while Haiku-4.5 and Sonnet-5 stay within the arc spread. This shows one benefit of SCLATE: any component of a continual-learning agent, its model, harness or memory system, can be ablated end to end, here with one adapter setting (Q5).
  • Memory writes. The clock does not jump while any component of the agent or its memory system is busy, including Mem0's and Hindsight's background writes (Q6).
AuthorsOpenReview ↗

(2/7) Response to Reviewer fKrT

Q1 (W1). No experiment tests whether putting agent-side events on the benchmark's clock changes a result. Suggested ablations: MetaClaw with prompt-injected dates versus SCLATE's simulated clock, and GBrain with and without its nightly consolidation. GBrain is the only memory system that schedules events between sessions, and SWE-Gym is single-session.

A1. Thank you for suggesting these ablations. We ran the GBrain ablation you suggest: Claude Code + GBrain on Gemini-3.7F scores 81.3% with its nightly consolidation and 78.2% without it, three arcs each, a gain of 3.1 pp. The shared clock itself has no "off" setting to ablate, because it is not a component added on top of a benchmark run. SCLATE implements each benchmark as is, in a shared event contract that the agent also uses, and this contract is what lets the agent's own events (session stop and start, crons, memory consolidation) run alongside the benchmark's at all, without a custom scheduling loop for each benchmark-agent pair. The benchmark's own behavior is unchanged: on GAIA-2, SCLATE delivers every scheduled event in the same order as upstream, within 0.9 s at the median, and a SCLATE run and an upstream run disagree no more often than two runs of the same stack (Figure 3). Taking it away does not leave the same run on a different clock; it leaves a run in which the agent's routines never fire (0 of 29 nightly consolidations in a 40-day arc).

Why the shared clock is needed. Consolidating memory between sessions, often called dreaming, is an important part of agent memory [2], and such an event can be ordered with the benchmark's events only when both sides define what happens and when on the same clock. In MetaClaw, the benchmark's "workday 2 starts the next morning" and GBrain's "consolidate every night at 02:00" must be ordered so that the consolidation falls between the two workdays, as deployed. If the agent's schedule runs on a clock of its own, nothing orders the two: the consolidation can land in the middle of a workday, or not at all. In SCLATE both sides define their events in one contract on one clock, and one scheduler orders them, so each night's 02:00 falls between the workdays.

The evaluation uses it beyond GBrain.

  • Session stop and start are agent-side events in every configuration, and the memory systems' own session hooks run on them. For example, Hindsight's plugin recalls at session start (in one Claude Code + Hindsight segment it injected 9 to 11 memories from day 3 on) and makes a final retain at session end.
  • Over the 300 MetaClaw arcs of Table 1, the scheduler dispatched 37,740 agent-side events alongside 234,600 benchmark-side events, in every configuration. LongMemEval and PersonaMem are multi-session as well.
  • SWE-Gym is indeed single-session; the multi-session case is the MetaClaw post-training in Section 4.3.

On the suggested ablations.

  • The ablation of the shared clock itself is the wall-clock count above: without it the agent's routines do not happen at all, so there is no outcome to compare.
  • GBrain with and without its nightly consolidation: we ran it, 3 arcs each with Claude Code and Gemini-3.7F, with the consolidation event left unregistered through one setting of SCLATE's GBrain adapter and nothing else changed: 278 / 285 / 281 with it on against 271 / 265 / 276 off, of 346 (29 consolidations per arc on, none off; Appendix D.5). The consolidation runs only because SCLATE dispatches it at 02:00 on the shared clock; whether it helps is a property of the agent, which SCLATE makes measurable for an unmodified agent.
  • Prompt-injected dates leave every process on the host clock (date, file times, crons, consolidation), while MetaClaw's grader checks the simulated day in what the agent writes (for example created_at: 2026-03-30T...+08:00). Such a run would measure that mismatch. SCLATE gives every process the simulated time (Appendix C.2).

[1] Meta. Bringing Specialized Models to Muse and Meta AI. November 2026. https://research.meta.ai/blog/specialized-models-for-muse-and-metaai

[2] L. Martin. The components of agent memory. Anthropic, November 2026. https://www.anthropic.com/research/components-of-agent-memory

AuthorsOpenReview ↗

(3/7) Response to Reviewer fKrT

Q2 (W2). Many results are single runs (Table 1), with no confidence intervals, while two runs of one setup disagree about 20% of the time.

A2. Thank you for raising this. We updated Table 1 with repeated runs, and the claims hold against them. The disagreement you note in Figure 3d (about 20% in the submission, 23–24% on the revised, larger sample) is the model's own run-to-run variation, which two runs of upstream GAIA-2 show just as much, not a disagreement that SCLATE introduces. Every cell is now the mean of three independent 30-workday arcs, each listed in Appendix D.3; the 95% confidence interval of a cell mean has a median half-width of 6.5 pp for memory configurations and 2.2 pp for No-Mem. The three claims of Section 4.2 hold against the spread of the arcs (Welch's t-test, 95%):

  • Memory helps. In all 70 cells, the memory configuration scores above its harness's No-Mem, significantly in 58; in 66 of them every one of its arcs is above every No-Mem arc.
  • An added memory system does not reliably beat native memory. It is significantly better in 15 of 50 cells and never significantly worse; where it scores lower, the gap is at most 3.3 pp.
  • No memory system is best for every model. The best configuration changes from model to model, and it is significantly ahead of the second best in only 2 of 20 model-harness pairs.

Figure 3d is a parity check: two runs of the same stack, upstream GAIA-2 included, disagree on 23–24% of scenarios, and a SCLATE run and a GAIA-2 run on 24%. The variation comes from the model's capability: scenarios near the edge of what the model can do are solved in some runs and not others (on upstream GAIA-2 Time, pass@1 21% against pass@3 42%), not from the substrate or latency: within a scenario the faster repeat does not pass more often (Q3). It is why one run per cell cannot separate close configurations, hence the three arcs above.

Appendix D.3 now lists all three arcs of every cell, and Section 4.2 states the claims against their spread.

Reviewer_6sb5OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

SCLATE is an execution substrate for evaluating and post-training agents across multiple sessions. Benchmark events and agent-side events share an open scheduler, a hybrid clock advances in real time during activity and skips idle periods, and an API proxy records model trajectories through unmodified harnesses. The evaluation ports seven benchmarks, compares ten harness/memory configurations on ten models, and demonstrates post-training with Qwen3.5-4B.

Critical strengths

SCLATE addresses several interacting requirements within a unified architecture: coordinating benchmark events with agent-side lifecycle activities, preserving persistent state across sessions, advancing simulated time without interrupting active execution, and capturing model calls for training. The separation between benchmark adapters, agent adapters, scheduling, and rollout orchestration is well motivated. In particular, treating memory maintenance and session transitions as part of the execution lifecycle makes the framework suitable for evaluating complete agent systems.

The paper goes beyond demonstrating that the integrations run. Its comparison of upstream GAIA-2, flow-only execution, and flow-plus-jump execution examines event delivery, timing, task outcomes, and run-to-run disagreement. These checks provide meaningful evidence that time compression preserves evaluation behavior within the tested configurations, strengthening the credibility of the systems contribution.

The same execution infrastructure supports both head-to-head evaluation and trajectory collection through existing harnesses and memory systems. Component-level attribution of model calls, explicit state persistence, and separation of agent-accessible resources from grading data reflect attention to important implementation details. The breadth of integrations and the post-training demonstrations provide substantive evidence that the proposed abstractions are useful beyond a single benchmark or agent configuration.

Critical weaknesses

Several major components of SCLATE have close precedents: Agent Lightning and POLAR support proxy-based trajectory capture and training through existing agent harnesses; Harbor provides reusable infrastructure for agent evaluation and rollout generation; and ARE/GAIA-2 already supports dynamic, asynchronous environments. SCLATE’s most distinctive contribution appears to be the coordination of benchmark-side and agent-side lifecycle events on a shared hybrid clock, together with cross-session state management. While this is a useful integration, the paper does not yet convincingly establish how much new functionality or reduced engineering effort it provides over extending existing frameworks. Appendix A offers a helpful feature comparison, but the experiments primarily demonstrate execution fidelity and downstream utility rather than isolate this incremental contribution. A comparison against an existing framework augmented with session hooks and persistent state, together with targeted tests of agent-side event scheduling and quantitative integration costs, would clarify whether SCLATE introduces a substantive systems advance beyond a well-engineered consolidation of established mechanisms.

Questions for authors

will the framework open sourcce?

6 comments
AuthorsOpenReview ↗

(1/5) Response to Reviewer 6sb5

Dear Reviewer 6sb5,

We thank you for the thorough review, and we are glad you found that the separation of benchmark adapters, agent adapters, scheduling and rollout orchestration is well motivated, that treating memory maintenance and session transitions as part of the execution lifecycle suits the evaluation of complete agent systems, and that the GAIA-2 parity checks give meaningful evidence that time compression preserves evaluation behavior. Following your comments, we explain why existing frameworks cannot be extended, report targeted checks of agent-side scheduling and quantify integration costs. We respond to each point below; changes in the revised paper appear in blue.

SCLATE introduces a new paradigm for evaluating and training continual-learning agents such as Muse, Dots and Grokbot [1]: the benchmark and the agent are peers on one open scheduler with a shared clock, so the agent's own events (session stop and start, crons, memory consolidation) run alongside the benchmark's, as deployed. Each benchmark and each agent is integrated once, at O(m+n) in place of O(m×n) for m benchmarks and n agents, after which any heterogeneous agent, with its unmodified harness and memory system, runs on any integrated benchmark, and the same runs serve evaluation and training.

Summary. Your main concern is whether SCLATE is more than a well-engineered consolidation of existing systems. It addresses a problem none of them poses: evaluating and training a continual-learning agent as a whole system (model, harness and memory) whose own events interleave with the benchmark's on one timeline. Existing frameworks put event scheduling on the benchmark side (Harbor's trial, ARE/GAIA-2's scenario scheduler) and treat the agent as a passive responder, while SCLATE makes the benchmark and the agent peers on one open scheduler, so heterogeneous continual-learning agents, each with events of its own, run across various benchmarks. In short:

  • The shared scheduler is a precondition, not an addition. A benchmark's events and an agent's own events each come with their own timing, so running them together needs one scheduling loop in which both live on a shared clock, with the benchmark and the agent each defining what happens and when (Q1).
  • No existing system has it, and none can add it without changing its core design. Harbor contributes reusable evaluation infrastructure, with adapters that connect many harnesses to many benchmarks, and ARE/GAIA-2 contributes asynchronous environment events, with adapters that connect agents to its benchmarks. Both, however, schedule only the benchmark's events and call the agent only to reply, so their infrastructure cannot be reused for continual-learning agents, which have events of their own (session stop and start, crons, memory consolidation); even Harbor's port of GAIA-2's timed events falls short (Q2). Each benchmark-agent pair therefore needs its own scheduling loop, an O(m×n) effort for m benchmarks and n agents; SCLATE defines each benchmark and each agent once, at O(m+n) (Q1).
  • It makes a new class of benchmarks runnable with any agent. Long-horizon continual-learning benchmarks such as MetaClaw now run with any unmodified harness and memory system, which no existing framework supports (Q1, Q2).
  • It shows what only whole-system evaluation can. The same harness feature acts as memory for one model and not for others: hiding earlier transcripts costs Gemini-3.7F 5.4 pp but leaves Haiku-4.5 and Sonnet-5 unchanged (Appendix D.4), and an added memory system does not reliably beat native memory across ten models (Table 1). Neither finding is visible when model, harness and memory are evaluated apart.
AuthorsOpenReview ↗

(2/5) Response to Reviewer 6sb5

Q1. Several components have close precedents (Agent Lightning and POLAR for proxy-based capture, Harbor for evaluation infrastructure, ARE/GAIA-2 for dynamic environments). How much new functionality or reduced engineering effort does SCLATE provide over extending existing frameworks?

A1. Thank you for this question. SCLATE provides one scheduler on which a benchmark's events and an agent's own events run in order on a shared clock, which no existing framework has or can add without changing its core design. Concretely, this gives:

  • New functionality. The agent's own events now happen. On the host's wall clock, a 40-day MetaClaw arc runs in about 7 hours and GBrain's nightly 02:00 consolidation fires zero times instead of 29; on SCLATE's shared clock it fires every night between the workdays, as deployed. In addition, SCLATE simulates long horizons fast yet realistically from the agent's view: the clock jumps over idle time only when nothing is busy, so a 40-day arc takes hours while the harness, its shell and the memory systems all see the simulated date (Appendices C.1, C.2). Long-horizon benchmarks such as MetaClaw run through unmodified harnesses and memory systems, which no shared framework could do before.
  • Reduced engineering effort. Without a shared scheduler every benchmark-agent pair needs its own scheduling loop, as our answer to Q2 shows for MetaClaw and for Harbor's GAIA-2 adapter. With SCLATE the scheduler, clock and capture proxy are written once, and 70 benchmark-agent pairs are served by 13 adapters (Q2).

Running a continual-learning agent faithfully has two requirements: a long horizon (many sessions, stopped and restarted with their state, on a simulated clock that skips idle time) and faithful execution (the benchmark's and the agent's events on one clock, with a way for the agent side to place its events on it). As Appendix Table A1 shows, no existing system has both:

SystemHarnessMemoryMulti-sessionTime accel.Shared clockAgent-side events
Harbor✓✗✗✗△✗
GAIA-2 / ARE△✗✗△△✗
MetaClaw△✗△✗✗△
POLAR✓✗✗✗△✗
OpenForgeRL✓✗✗✗△✗
Agent Lightning✓✗✗✗△✗
rLLM✗✗✗✗△✗
SCLATE (ours)✓✓✓✓✓✓

△ partial; a partial shared clock is wall-clock time.

Why can't they be extended? Harbor and ARE/GAIA-2 define an adapter abstraction that lets an agent connect to many benchmarks without coding each connection, unifying how agents and benchmarks plug in. Both assume the agent has no events of its own, and their adapters cannot be extended to carry them.

[Continued]

AuthorsOpenReview ↗

(3/5) Response to Reviewer 6sb5

(Q1, continued)

  • Harbor: Harbor has no scheduler and no clock of its own, and treats the agent as a passive responder with no events of its own: its adapter maps each benchmark item to a task that one fixed trial runs on real time (set up, run the agent until it replies, grade). A benchmark's asynchronous events therefore have no place in the trial, as Harbor's own GAIA-2 port shows (Q2; [2]), and neither do the agent's own (crons, memory consolidation, session stop and start). Supporting them would require redesigning Harbor's adapters so that agents register their own events, and, so that any benchmark runs without reimplementing a custom scheduling loop, defining a benchmark adapter and an event scheduler that carry the benchmark's events as well, with the agent stopped, started, saved and restored mid-run. That is a new abstraction, which SCLATE builds, not an extension of the trial. Harbor therefore cannot be extended to run continual-learning agents against its benchmarks: every such agent-benchmark pair needs its own custom scheduling loop, an O(m×n) integration cost.
  • ARE/GAIA-2: the scenario owns the scheduler and the clock, in ARE's own runner and in gaia2-cli's event daemon alike, and its agent adapters assume a passive responder that only answers the scenario's notifications. Only the scenario can put events on the queue, so the agent's own events (crons, memory consolidation) cannot be scheduled; a run is a single session that nothing can stop, start, save or restore; and its per-harness adapters serve GAIA-2 only, without extending to other benchmarks. Supporting continual-learning agents means redesigning its core: a shared event contract in which both the benchmark and a heterogeneous agent declare their events, a scheduler that orders both on one clock, and agent lifecycle operations (stop, start, save, restore) that the scheduler can call.
  • POLAR, Agent Lightning: these are proxy-capture tools, not frameworks for running agents on benchmarks; they record the model calls of one episode, and scheduling is outside their scope. In continual learning the proxy has to be designed together with the substrate: calls come from the agent, its memory system, a judge or a simulated user, across sessions, so the proxy must tag each call with its caller, which decides what a training trajectory keeps and what each evaluation metric counts. SCLATE therefore builds the capture proxy into the substrate rather than adding it on.
  • SCLATE: its adapters carry heterogeneous events from both sides on one scheduler: the benchmark's turns, grades, world events and session boundaries, and the continual-learning agent's crons, memory consolidation and session stop and start. A Harbor trial is the special case of one instruction and one grade, and a GAIA-2 run the special case in which only the benchmark schedules events (SCLATE runs Harbor's terminal tasks through its Harbor adapter and matches upstream GAIA-2 in Figure 3). An agent's events are defined once and it runs on every benchmark, and a benchmark is defined once and runs with every agent (Q2).

SCLATE is not a combination of Harbor, POLAR and Agent Lightning; its integrated proxy serves evaluation and training from the same runs, with component-level attribution.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear reviewers and AC,

We deeply appreciate your time and effort in reviewing our work.

Continual-learning agents such as Muse, Dots and Grokbot are appearing, and evaluating and training them faithfully has become important [1]. These agents have events of their own (session stop and start, crons, memory consolidation) that existing frameworks do not schedule. Each benchmark-agent pair has therefore run through its own custom scheduling loop, an O(m×n) effort for m benchmarks and n agents, with no shared place to compare or train them.

What SCLATE brings. To our knowledge, SCLATE is the first framework in which a continual-learning agent's own events are first-class:

  • An open event scheduler, the necessary component for this: benchmarks and agents both register events on it, ordered on one shared clock that compresses month-long scenarios into hours while preserving event order and timing.
  • Adapters for benchmarks and whole agents, with which any heterogeneous continual-learning agent runs as a whole system, its model, unmodified harness and memory system together, on any integrated long-horizon, multi-session benchmark, at O(m+n) in place of O(m×n).
  • One substrate for evaluation and training, whose runs serve both, with every model call attributed to its component.

What this enables. Together, these let researchers compare heterogeneous continual-learning agents, each as a whole system, on the same long-horizon, multi-session benchmarks, and train a model through the harness and memory system it is deployed with, neither of which a prior framework supports; we show both with seven ported benchmarks, a head-to-head comparison of ten harness and memory configurations on ten models (Table 1), and post-training through unmodified harnesses and memory (Section 4.3).

Reviewer labels below: R1 is 6sb5, R2 is fKrT, R3 is Yq1b. What the reviewers valued:

  • S1: The shared scheduler addresses an important gap. Benchmark and agent-side events run on one timeline without changing the harness, with memory maintenance and session transitions part of the lifecycle, and the clock compresses month-long scenarios into hours (R1, R2, R3).
  • S2: Head-to-head evaluation and training of unmodified harnesses and memory. 3 harnesses with their memory systems on 7 benchmarks and 10 models with no agent code changes, and the same runs train the model through them, with every call attributed to its component (R1, R2, R3).
  • S3: Evidence that the shared scheduler preserves evaluation behavior. GAIA-2 events on time, outcomes within run-to-run variation (R1, R2).
  • S4: Useful to practitioners, given the usual difficulty of running unmodified harnesses and memory systems over long multi-session horizons and of training models through them (R2).

What we changed, and what each change shows:

  • [R1, R2] To show why existing frameworks cannot run continual-learning agents, we added Appendix A: Harbor's fixed trial and ARE/GAIA-2's scenario-owned scheduler leave no place for the agent's own events.
  • [R1, R2] To check that the agent's own events run in order, we added Appendices C.1 and C.3: all 9,000 Table 1 sessions stopped and started in order, and memory work holds the clock.
  • [R1] To quantify integration cost, we added Appendix B: 13 adapters serve 70 benchmark-agent pairs.
  • [R2] To test an agent-side event directly, we added the GBrain ablation in Appendix D.5: its nightly consolidation, which runs only on the shared clock, adds 3.1 pp (281.3 against 270.7 of 346).
  • [R2, R3] For reliability, every Table 1 cell now has three independent arcs (Appendix D.3), and the paper's claims hold against their spread.
  • [R2, R3] To strengthen the parity study, GAIA-2 parity now uses 200 scenarios with 3 repeats on every stack, run concurrently (Figure 3): a SCLATE run and a GAIA-2 run disagree about as often as two runs of one stack.
  • [R2, R3] To measure sensitivity to serving latency, we added Appendix C.5: with timing checks enforced, Time falls from 32% to 4% with 5 s added per model call, and with them off from 62% to 58%.
  • [R2] Claude Code's No-Mem left earlier transcripts readable, so we added a memory-free No-Mem in Appendix D.4: hiding them lowers Gemini-3.7F, which read them, by 5.4 pp, while Haiku-4.5 and Sonnet-5 stay within the spread of their arcs.
  • [R2, R3] To check post-training stability, we added three more training runs each on SWE-Gym (Appendix E.2: Codex Pass@1 from 27.7% ± 1.2 to 35.7% ± 2.4) and on MetaClaw for two configurations (Appendix E.4: every run gains 6.1 to 11.5 pp on the held-out days).
  • [R1] SCLATE will be open-sourced upon publication (reproducibility statement).

Changes in the revised paper appear in blue.

Sincerely,

The Authors

[1] Meta. Bringing Specialized Models to Muse and Meta AI. November 2026. https://research.meta.ai/blog/specialized-models-for-muse-and-metaai

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.