acceptodds
Under review as a conference paper at ICLR 2027

ReLiveGym: Evaluating Long-Lived Agents over Weeks of Replayed Reality

Abstract

As large language model (LLM) agents become widely adopted, they are increasingly deployed for tasks that require persistent monitoring or recurring actions (e.g., market analysis). These agents are expected to operate unattended for days or weeks, act at the right timing, and adapt to the dynamic environment over time. These challenges are not fully captured in the existing long-horizon agent work, as they often consider a static environment that is not temporally changing. We introduce ReLiveGym, a diagnostic evaluation environment of long-lived tasks in which agents act sparsely over simulated weeks of chronologically replayed real-world news, market, and social-media streams. The tasks span diverse levels of time sensitivity, reasoning intensity, and recurrence. Across eight base language models, we investigate how model choice and harness design affect agent performance on such long-lived tasks. Our results show that how agents determine when to act arises as an important harness-design axis for long-lived tasks; and that the optimal design varies across tasks and sometimes model choices as well. We also evaluate how continuous learning from hindsight feedback affect performance and address failure modes observed in these long-lived tasks. These findings indicate model choice, action timing mechanism, and use of feedback as important considerations in the design of long-lived agents.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM agents, long-lived agents, agent harness, test-time learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HbcEOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ReLiveGym, an evaluation environment for long-lived LLM agents that act intermittently over weeks of chronologically replayed real-world data. Its eight tasks cover forecasting, event detection, and recurring automation, with evaluation based on both correctness and action timing under limited budgets. The authors evaluate eight models across three triggering mechanisms—sleep, programmatic watchers, and scheduled jobs—and investigate hindsight feedback through outcome memory and best-of-N selection with pruning. The experiments highlight task-dependent effects of model and harness choices, persistent failures such as instruction drift and poor initial strategies, and the benefits and limitations of feedback-based improvement.

Critical strengths
  1. The authors introduce a new category of long-horizon tasks that closely aligns with my understanding of agent research. Beyond tasks such as coding and workspace activities that involve many interaction turns, I believe another important category consists of tasks that span long periods of time. Such tasks are common in real-world applications but have often been overlooked. This paper systematically characterizes representative task types and concrete tasks, establishing a valuable evaluation direction that could meaningfully contribute to the future development of agents.
  2. The work also makes a substantial experimental contribution. The authors build a relatively comprehensive simulation environment that recreates these scenarios through historical data replay. This is a clever design that requires considerable engineering effort and can facilitate and accelerate future evaluations.
Critical weaknesses
  1. The paper does not sufficiently examine the impact of mature agent harnesses used in practice. For example, how effectively and efficiently would agents perform these tasks within well-developed harnesses such as Claude Code or Codex? Could many of these tasks already be handled well by such systems? I believe this question is important for assessing how well the findings generalize to real-world deployments.
  2. Since monitoring fundamentally involves achieving timely responses with limited resources, the paper should systematically evaluate the relationship between budget, polling frequency, and performance, rather than simply impose a budget cap. It should also distinguish the costs of model calls from those of monitoring the environment.
Questions for authors

Could the authors provide additional experimental results using Claude Code or Codex as the agent harness?

5 comments

We thank the reviewer for the thoughtful comments and for recognizing the contributions of our paper. We address the reviewer's questions below.

Q1. The paper does not sufficiently examine the impact of mature agent harnesses used in practice. For example, how effectively and efficiently would agents perform these tasks within well-developed harnesses such as Claude Code or Codex? Could many of these tasks already be handled well by such systems?

We agree that evaluating widely used agent harnesses is important. ReLiveGym supports harnesses such as Claude Code and Codex, and we have added experiments with the following model–harness combinations:

ModelHarness
GPT-5.6-luna (in paper)Codex 0.160.0
Claude Haiku 4.5 (in paper)Claude Code 2.1.319
Claude Sonnet 5.5Claude Code 2.1.319

Experiment setup

  • Environment: Agents run in a Docker container with the cron service installed.
  • Prompt: Agents are instructed to choose appropriate waiting or scheduling mechanisms from those available in the environment.
  • Infrastructure: Experiments use a development version of the upcoming ReLiveGym release. It intercepts the standard C library's clock, sleep, and timer calls so that programs throughout the container—including agents, the cron service, and Python scripts—read and wait on the same simulated clock.

Empirically, we observe agents select a range of triggering mechanisms, summarized below.

MechanismWhat the agent set up
in-run waitThe agent process stays alive and wakes itself using mechanisms native to the harness (e.g., clock.sleep in Codex, ScheduleWakeup or Monitor in Claude Code). These mechanisms often have harness-enforced time limits (e.g., 1 hour for ScheduleWakeup).
cron, fresh (cron without resume)A crontab line that starts the CLI in a new session at each firing (codex exec "…", claude -p "…"). Information persists between runs only through files.
cron + resumeA crontab entry that starts the agent CLI and resumes the previous session (claude -p --continue, codex exec resume)
cron → scriptA crontab entry that runs an agent-authored Python or shell program, which may invoke the agent CLI.
program → CLIAn agent-authored program that runs continuously without cron, monitors the environment, and starts the agent CLI when a decision is needed.
programThe same kind of long-running program, but one that does the work without starting the agent CLI.
CronCreate onlyThe agent relies solely on Claude Code's built-in CronCreate, which is unsuitable for scheduling in headless mode.
Ended without schedulingThe agent exited without scheduling anything or launching a persistent program.

Finding 1. Agents using mainstream harnesses adopt diverse scheduling strategies, often relying on cron.

Most runs use variants of cron to handle long-lived tasks. Codex with GPT-5.6-luna and Claude Code with Sonnet 5.5 select relatively consistent triggering mechanisms across repeated runs, whereas Claude Code with Haiku 4.5 shows more variation.

The table below summarizes the main triggering mechanisms selected across repeated runs and episodes.

TaskCodex + GPT-5.6-lunaClaude Code + Haiku 4.5Claude Code + Sonnet 5.5
Breakout newscron, fresh ×9cron, fresh ×7; cron + resume ×2cron, fresh ×9
Resolution detectioncron, fresh ×9cron + resume ×6; cron → script ×2; CronCreate onlycron, fresh ×9
Forecast portfoliocron, fresh ×3CronCreate only; cron + resume; cron, freshcron, fresh ×3
Popularity predictioncron, fresh ×3cron + resume ×2; cron → scriptcron → script ×3
Daily digestcron, fresh ×2; cron → scriptcron, fresh ×2; cron → scriptcron → script ×3
Market price checkcron, fresh ×2; ended without scheduling ×1cron, fresh ×2; cron → scriptcron → script ×3
Web filing trackingcron, fresh ×3in-run wait; cron, fresh; cron + resumeprogram ×2; program → CLI
Web broker opscron, fresh ×3cron + resume ×2; cron, freshprogram → CLI ×3

The selected mechanism also depends on the model–harness combination. For example, for popularity prediction, all three Codex + GPT-5.6-luna runs use fresh cron sessions, while all three Claude Code + Sonnet 5.5 runs use cron-triggered scripts.

Agents also combine cron for long-term scheduling with in-run mechanisms for shorter waits, as illustrated below.

AgentTask, seedMain triggerNative in-run wait addedWhat happened
Codex + GPT-5.6-lunaPopularity prediction, seed 1Hourly cron runsclock.sleep, 42 times, up to 22 minWhen all 10 recommendation slots were occupied, the agent checked when the oldest slot would become available, slept until then, and used it. ("The first slot frees at 13:06:26 UTC. I'll stay active through it.")
Codex + GPT-5.6-lunaWeb filing tracking, seed 3Hourly cron runsclock.sleep, 10 minBefore the 09:30 ET market open (a filing deadline), it slept 10 minutes inside the run for "one final fresh mirror check shortly before the deadline".
Claude Code + Haiku 4.5Breakout news, episode 3, seed 1Hourly cron runsScheduleWakeup every 15 minIt added 15-minute in-session checks alongside cron, but one session mistakenly remained active for 40 h and exhausted the budget.

Finding 2. To answer the reviewer's question about the performance of well-developed harnesses, we compare Codex and Claude Code with hand-rolled harness with three triggering mechanisms reported in the paper, and summarize main findings:

  • With GPT-5.6-luna, Codex often performs close to the best of the three triggering mechanisms ({\sleep, watcher, cron}) evaluated in the paper, e.g., on breakout news, reslution detection, and popularity prediction tasks. There are substantial gaps on certain tasks such as forecast portfolio, where Codex triggers too frequently and get distracted by unimportant news.
  • With Haiku 4.5, Claude Code generally performs worse than the best custom-harness configuration. One possible explanation is a mismatch between Haiku 4.5 and the latest Claude Code harness.
  • We also report Sonnet 5.5 results as an additional reference for performance with the latest Claude Code harness.
Taskluna, paper: bestluna, paper: worstluna, E1 (Codex)haiku, paper: besthaiku, paper: worsthaiku, E1 (Claude Code)sonnet, E1 (Claude Code)
Breakout news15.5 ± 0.8 (watcher)11.2 ± 0.7 (sleep)15.3 ± 0.48.6 ± 2.5 (sleep)5.2 ± 2.3 (watcher)3.5 ± 0.45.4 ± 3.1
Resolution detection20.0 ± 3.9 (sleep)17.1 ± 4.6 (cron)20.5 ± 0.55.2 ± 1.8 (sleep)2.6 ± 0.6 (cron)0.9 ± 1.59.7 ± 6.8
Popularity prediction43.3 ± 11.3 (sleep)38.2 ± 15.6 (watcher)42.1 ± 9.69.2 ± 4.5 (sleep)4.2 ± 3.0 (cron)8.7 ± 13.170.2 ± 5.1
Forecast portfolio31.7 ± 12.2 (watcher)7.0 ± 24.1 (cron)-6.9 ± 14.4-4.3 ± 4.8 (watcher)-14.1 ± 19.1 (sleep)-8.6 ± 12.730.2 ± 7.2
Daily digest100.0 ± 0.0 (cron)98.9 ± 2.0 (sleep, watcher)98.6 ± 2.495.6 ± 3.5 (cron)42.4 ± 23.6 (sleep)66.6 ± 53.8100.0 ± 0.0
Market price check99.4 ± 0.4 (cron)68.2 ± 42.9 (sleep)65.9 ± 57.196.7 ± 1.7 (cron)35.2 ± 35.3 (sleep)99.3 ± 1.099.7 ± 0.5
Web filing tracking76.2 ± 13.7 (watcher)10.0 ± 10.3 (sleep)65.1 ± 53.621.2 ± 22.9 (watcher)1.6 ± 1.6 (cron)16.4 ± 11.532.8 ± 56.8
Web broker ops97.9 ± 3.7 (cron)0.0 ± 0.0 (watcher)96.8 ± 3.290.3 ± 16.8 (cron)11.8 ± 3.7 (sleep)40.9 ± 37.7100.0 ± 0.0
Reviewer_mxEHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ReLiveGym, an evaluation framework for long-lived LLM agents operating over weeks of chronologically replayed real-world data. ReLiveGym focuses on agents that act intermittently in independently evolving environments, requiring timely decisions and potential adaptation using historical feedback. This differs from conventional long-horizon benchmarks.

The experiment includes eight tasks spanning forecasting, event detection, etc. The authors evaluate 8 LLMs under 3 triggering mechanisms (sleep, watcher, and cron). The experiments show that triggering mechanisms substantially affect performance and that their effectiveness varies across models and tasks. The paper also examines whether memory and trajectory selection can mitigate the failures.

Critical strengths
  1. The paper addresses an important gap in existing agent benchmarks by considering long-lived agents operating in independently evolving environments. The paper provides a modular simulation framework, which is a useful contribution for studying agents deployed over extended periods.
  2. The experiments across multiple models and triggering mechanisms reveal substantial performance variation, which is an interesting observation.
Critical weaknesses

The main concern is that it is unclear what fundamentally new agent capabilities ReLiveGym can evaluate that existing benchmarks cannot. While ReLiveGym extends agent evaluation to weeks of chronologically replayed real-world data, its key components have been explored in prior benchmarks such as GAIA2. The main distinction appears to be the combination of longer time horizons, diverse tasks, and configurable triggering mechanisms. This confuses me about whether the benchmark effectively measures the distinctive capabilities of long-lived agents rather than general automation reliability. The authors should clarify this distinction in the rebuttal.

The paper emphasizes temporal planning, persistent execution reliability, and continuous adaptation as important capabilities of long-lived agents, but the experiments primarily report aggregate task performance under different models and harness configurations. These comparisons do not sufficiently disentangle the individual capabilities being evaluated. For example, the cron harness includes predefined task-specific schedules, making it difficult to distinguish autonomous temporal planning from scheduling priors.

Moreover, most tasks are evaluated on only one to three fixed historical episodes, leaving the generalizability of the findings across different temporal windows and environments insufficiently established.

Questions for authors
  1. Compared with FutureSim, GAIA2, and SentinelBench, what unique capabilities or failure modes does ReLiveGym enable researchers to evaluate? What is the key advantage of the proposed framework?
  2. Could the authors compare the existing fixed-cron implementation (tm: C) with agent-managed cron, and provide evidence that feedback leads to actual strategy adaptation rather than trajectory selection?
  3. Given that most tasks use only one to three historical episodes, how robust are the reported findings across different time windows, event distributions, and environments? Could the authors provide additional evidence or analyses to support the generality of the observed model and harness effects?
6 comments

We thank the reviewer for the thoughtful comments and for recognizing that our paper fills an important gap. We address the reviewer's questions below.

Q1: Compared with FutureSim, GAIA2, and SentinelBench, what unique capabilities or failure modes does ReLiveGym enable researchers to evaluate? (Weakness: The main concern is that it is unclear what fundamentally new agent capabilities ReLiveGym can evaluate that existing benchmarks cannot / whether the benchmark effectively measures the distinctive capabilities of long-lived agents rather than general automation reliability)

We thank the reviewer for highlighting the need to clarify our framing. ReLiveGym evaluates the interaction between timing decisions and challenging commitment decisions over long horizons. Here, timing decisions determine when to observe or act, while commitment decisions determine what user-facing action to take given the available observations.

GAIA2 and SentinelBench evaluate timing through tasks with explicit timing rules (e.g., send a notification after 3 minutes) or bounded monitoring windows (e.g., watch for an incoming email for 3 minutes). ReLiveGym extends this setting in two ways:

  • Longer episodes introduce additional timing and commitment challenges. Recurring scheduling mechanisms such as cron become relevant over long horizons. Agents must also sustain reliable timing and commitment decisions throughout an episode (the automation reliability the reviewer mentioned), and delayed outcome feedback creates opportunities for test-time improvement (as stated in our introduction).
  • Commitment decisions are often challenging even when observation times are fixed. In GAIA2 and SentinelBench, these decisions are often straightforward once an observation is available, such as detecting whether a new email has arrived. ReLiveGym includes established, challenging NLP/ML tasks such as forecasting, resolution detection (fact checking), and popularity prediction.
  • FutureSim and other work cited in our introduction study challenging commitment decisions, such as forecasting, but run agents on fixed schedules and therefore do not evaluate autonomous timing decisions.

Why evaluate timing decision and commitment decisions jointly? It is because of the interaction - Timing decisions determine which observations an agent receives, and those observations shape its commitments. Our experiments show that performance on reasoning-intensive tasks, such as breakout news detection, depends strongly on the triggering mechanism (sleep, watcher, or cron). A poorly chosen triggering mechanism can therefore undermine an otherwise capable agent. Evaluating timing and commitment decisions separately as in prior line of works would miss this interaction. Therefore, ReLiveGym thus complements GAIA2, SentinelBench, and FutureSim by evaluating both capabilities together over long horizons.

We will revise our framing in introduction in the next revision of the paper.

Q2: Could the authors compare the existing fixed-cron implementation (tm: C) with agent-managed cron? (Weakness: the cron harness includes predefined task-specific schedules, making it difficult to distinguish autonomous temporal planning from scheduling priors.)

We thank the reviewer for this suggestion. We compare the following five cron variants using GPT-5.6-luna across the eight ReLiveGym tasks, with three runs per evaluated configuration:

  • Editable cron, with default schedules (the cron-trigger in the paper)
  • Editable cron, without default schedules
  • Fixed cron, faster than the default schedule
  • Fixed cron, at the default schedule
  • Fixed cron, slower than the default schedule
TaskDefault scheduleEditable cron, with default (paper)Editable cron, without defaultFixed cron, fasterFixed cron, default scheduleFixed cron, slower
Breakout news detectionevery 6 h14.7 ± 1.215.0 ± 2.112.6 ± 1.2 (1 h)15.3 ± 3.811.8 ± 2.3 (24 h)
Resolution detectiondaily17.1 ± 4.616.1 ± 8.114.8 ± 1.9 (6 h)12.9 ± 4.17.9 ± 1.7 (72 h)
Popularity predictionhourly41.9 ± 5.357.6 ± 9.119.8 ± 13.0 (15 min)48.2 ± 9.049.7 ± 2.0 (6 h)
Forecast portfoliodaily7.0 ± 24.15.2 ± 6.75.7 ± 10.0 (6 h)5.1 ± 22.020.0 ± 9.1 (weekly)
Web broker opsevery 2 h97.9 ± 3.768.8 ± 14.982.8 ± 27.1 (30 min)79.6 ± 30.057.0 ± 4.9 (4 h)
Daily digestdaily100.0 ± 0.0100.0 ± 0.0–––
Market price checkdaily99.4 ± 0.499.7 ± 0.0–––
Web filing trackingdaily61.4 ± 23.180.2 ± 14.6–––

Scores are the primary metric ×100, reported as mean ± standard deviation over three runs with GPT-5.6-luna. Breakout news detection and resolution detection pool three episodes per run, as in Table 4; the other tasks use the episode evaluated in the paper.

We summarize the findings below.

Finding 1. Fixed-cron performance depends on the schedule. On breakout news detection, triggering every 6 hours performs better than triggering every hour or every 24 hours. On forecast portfolio, weekly triggering performs better, consistent with the benefit of avoiding decision reversals in response to unimportant news.

Finding 2. Agent-managed cron without default schedules is competitive with the best fixed schedule on three of the five tasks with fixed-cron comparisons. Its mean score is comparable to or higher than that of the best fixed schedule on breakout news detection, resolution detection, and popularity prediction. It trails the best fixed schedule on forecast portfolio and web broker ops.

We agree that comparing agent-managed cron without default schedules against fixed schedules provides a clearer evaluation of autonomous temporal planning. We are extending the no-default-schedule evaluation across models and tasks and will include these results in subsequent revisions.

Q3: Evidence that feedback leads to actual strategy adaptation rather than trajectory selection

Part 1. Outcome memory: evidence of strategy adaptation

We thank the reviewer for this question. Outcome memory and Best-of-N + Prune use feedback differently: memory supports strategy adaptation within a run, whereas Best-of-N + Prune selects among runs. To examine adaptation, we analyzed every recommendation in the popularity prediction task (six seeds per cell). For each recommendation, we reconstructed the evidence available to the agent: the post's comment count at that moment, based on replayed comment timestamps, and its age.

No learning → outcome memory on days 2–30, mean over 6 seeds. Precision and time-weighted (TW) recall ×100.

ModelTriggerComments at recommendationPost age (h)PrecisionTW recall
GPT-5.6-lunaSleep40 → 512.0 → 3.168.5 → 62.233.5 → 29.3
Watcher16 → 310.9 → 1.450.8 → 71.233.7 → 43.2
Cron15 → 350.8 → 1.547.7 → 69.038.4 → 43.2
Qwen-3.7-plusSleep126 → 598.3 → 3.796.4 → 83.813.9 → 25.4
Watcher32 → 302.8 → 1.962.5 → 73.019.4 → 18.8
Cron36 → 412.1 → 1.964.5 → 75.529.6 → 44.7
  • Feedback changes how much evidence the agent waits for, with the direction depending on its initial behavior. Without memory, GPT-5.6-luna recommends early under watcher and cron (15–16 comments). With memory, it waits for roughly twice as many comments (31–35; 5 of 6 watcher seeds and all 6 cron seeds exceed the corresponding no-learning median), and precision rises by about 20 percentage points. In contrast, Qwen-3.7-plus under sleep starts at the other extreme: it waits about 8 h before recommending posts with high precision. With memory, it recommends after 3.7 h (59 comments, p = 0.01), trading some precision for almost twice the time-weighted recall.
  • The combined test of precision shifts across the six cells is significant (Fisher-combined p = 0.030). Together with the changes in comment count and post age at recommendation, these results support strategy adaptation rather than trajectory selection. Adaptation does not improve performance in every configuration, as the table shows.

Part 2. Best-of-N + Prune: selection to mitigate early failures

Best-of-N + Prune uses selection by design to address the early-failure mode of long-lived tasks (Sec. 5.2). Agents often adopt a strategy, such as a sleep interval, at the outset and retain it throughout the episode. A poor initial choice can therefore impair performance for weeks. Outcomes revealed during the first few days help identify these runs so that they can be pruned early. On popularity prediction, Best-of-N + Prune improves on a single run in every cell in the paper, and in all 9 trigger × month cells on the new episodes (+0.3 to +7.0 tw-F1). We will make the distinction between adaptation and selection explicit in the revision.

Reviewer_RkSZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ReLiveGym, an environment for evaluating agents that act intermittently over weeks of replayed news, market, financial filing, and social media data. Its eight tasks cover forecasting, event detection, and recurring operations. The study compares eight language models under three timing mechanisms, then tests outcome memory and selection among parallel agents using feedback available during execution. The main finding is that performance depends on how the harness schedules actions, with substantial variation between repeated runs. The benchmark addresses a useful distinction between taking many reasoning steps and remaining reliable as the outside world changes.

Critical strengths
  • The evaluation setting captures a practical problem that ordinary task completion benchmarks often miss. An agent can understand a task and still fail by acting too late, sleeping too long, or stopping routine work. ReLiveGym makes these failures measurable through timed commitments and recurring obligations. The task definitions in Table 2 help explain which aspects of timing each task tests.

  • The study examines interactions between models and harnesses instead of attributing every result to model capability. The differences across sleep, watcher, and cron configurations show that a model's apparent reliability can depend heavily on its scheduling support. Reporting repeated runs and their spread is valuable in this setting.

  • The trajectory examples make several failures concrete. Tables 7 through 10 show how wake times affect the opportunity to respond before a market move. Section 5.2 also distinguishes a poor strategy chosen early from a later loss of adherence to the task. These are useful observations for designing persistent agents.

Critical weaknesses
  • Figure 4 on page 6 reports a best Web Broker Ops score of 100.0 for Qwen3.7-plus. Table 4 on page 18 gives 90.0, 65.5, and 71.7 for its three timing mechanisms. GPT-5.6-luna similarly has 97.9 in Figure 4, while its largest entry in Table 4 is 76.2. The figure says that it selects the best configuration from the full results, so these values should agree. The discrepancy affects the claim that some models achieve perfect performance on this task and prevents a reliable interpretation of the model comparison.

  • The paper uses one to three episodes per task and three to six runs per configuration. Figure 6 attributes a large share of variation to differences between repeated runs, yet Figure 4 selects each model's best harness using those same results. The limitations section appropriately describes the work as a diagnostic study. Claims about stable model rankings and the relative importance of design choices should follow that scope. Results separated by episode would help show whether the findings persist across different events and time windows.

Questions for authors

Plz see the weakness section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer RkSZ (1/2)

We thank the reviewer for the thoughtful comments and for finding our evaluation setting practical and our analyses useful. We address the two weaknesses below.

W1: Web Broker Ops scores in Figure 4 and Table 4 (now Table 5) disagree.

We thank the reviewer for catching this and we apologize for this incorrectness. The appendix Table 5 (Table 4 in the previous version) contained a stale version of Web Broker Ops column from an earlier build of the task. We have replaced Web Broker Ops columns in appendix Table 5 in the revised version. The values and discussions in the main text have been correct (Qwen-3.7-plus scores 100.0 ± 0.0 under both watcher and cron, and GPT-5.6-luna scores 97.9 under cron), and Web Broker Ops are the only affected columns in Table 5.

W2: Few episodes, best-harness selection on noisy results, and the scope of the claims; whether the findings persist across different events and time windows.

We thank the reviewer for the suggestion and agree that our claims should reflect the scope of a diagnostic study. To show whether the findings persist across different events and time windows, we evaluated GPT-5.6-luna and GPT-5.6-terra on additional episodes. We compare results across triggering mechanisms and approaches to test-time learning from hindsight feedback.

TaskEpisodes
Breakout news3 -> 6
Popularity prediction1 -> 3
Forecast portfolio1 -> 3
Web filing tracking1 -> 3

Finding 1. The best-performing triggering mechanism remains unchanged in 7 of 8 task–model settings when results are pooled across the expanded episode set.

Scores ×100. Bold marks the best of the three triggers in each row.

TaskModelEpisodesSleepWatcherCron
Breakout news (cov F1)lunapaper: w10, w13, w1711.2 ± 0.715.5 ± 0.814.7 ± 1.2
lunaall six: + w15, w19, w2311.8 ± 1.814.0 ± 1.113.5 ± 0.8
terra ᵃpaper: w10, w13, w17, first 7 days20.9 ± 3.227.7 ± 6.833.7 ± 3.7
terra ᵃall six: + w15, w19, w23, first 7 days17.8 ± 1.521.3 ± 4.125.9 ± 3.5
Popularity (tw-F1)lunapaper: Apr43.3 ± 11.338.2 ± 15.641.9 ± 5.3
lunaall: Apr, May, Jun37.6 ± 3.942.3 ± 4.747.4 ± 2.5
terrapaper: Apr37.6 ± 13.665.1 ± 17.918.8 ± 7.2
terraall: Apr, May, Jun41.9 ± 4.859.2 ± 2.922.3 ± 1.7
Forecast portfolio (BSS)lunapaper: Mar15.0 ± 8.231.7 ± 12.27.0 ± 24.1
lunaall: Mar, May, Jun5.1 ± 5.29.0 ± 6.5−4.0 ± 4.2
terrapaper: Mar16.6 ± 7.834.2 ± 10.0−5.4 ± 9.8
terraall: Mar, May, Jun9.6 ± 1.218.0 ± 4.23.7 ± 4.1
Web filing (on-time accuracy)lunapaper: Apr10.0 ± 10.376.2 ± 13.761.4 ± 23.1
lunaall: Apr, Jan, Jul8.1 ± 5.270.9 ± 12.451.4 ± 13.7
terrapaper: Apr23.3 ± 13.970.9 ± 18.183.1 ± 6.0
terraall: Apr, Jan, Jul27.8 ± 3.974.1 ± 2.681.4 ± 5.0

ᵃ For GPT-5.6-terra, performance on breakout news depends on how events are distributed over time within an episode. Terra's cron runs exhaust the $20 LLM budget between days 7.7 and 13.5 and cannot act afterward, so their scores depend on how many breakpoints occur early. When evaluation is restricted to the first 7 days, before any run exhausts its budget, the rankings are consistent between the original and expanded episode sets. See Finding 2 for details.

AuthorsOpenReview ↗

Response to Reviewer RkSZ (2/2)

Finding 2. Whether the optimal triggering mechanism is consistent across different episodes depends on the task.

Below, each cell gives the pairwise trigger ordering pooled over all episodes (S = sleep, W = watcher, C = cron), followed by the number of episodes whose mean scores over seeds show the same ordering.

Case 1. On popularity prediction and web filing tracking, the comparison of triggering mechanisms is relatively stable across episodes. For each pair of triggers, the order within a single episode matches the order pooled over all episodes in 32 of 36 comparisons (89%).

TaskModelSleep vs watcherSleep vs cronWatcher vs cron
Popularity (3 months)lunaW > S: 2/3C > S: 2/3C > W: 3/3
terraW > S: 3/3S > C: 3/3W > C: 3/3
Web filing (3 windows)lunaW > S: 3/3C > S: 3/3W > C: 2/3
terraW > S: 3/3C > S: 3/3C > W: 2/3

Each cell gives the order pooled over all episodes (S = sleep, W = watcher, C = cron), then the number of episodes that show the same order.

Case 2. On breakout news, the ranking is less consistent: 22 of 36 per-episode comparisons (61%) agree with the pooled order. A likely factor is that breakout news events are distributed more heterogeneously across episodes.

For GPT-5.6-terra, performance also depends on when events occur within an episode. When evaluation is restricted to the first 7 days, before any run exhausts its budget, cron outperforms watcher in 5 of the 6 episodes.

TaskModelSleep vs watcherSleep vs cronWatcher vs cron
Breakout news (6 episodes)lunaW > S: 5/6C > S: 4/6W > C: 3/6
terraW > S: 5/6C > S: 3/6W > C: 2/6
terra, first 7 daysW > S: 2/6C > S: 4/6C > W: 5/6

Finding 3. Across the expanded episode set, Best-of-N + Prune continues to effectively mitigate the prevalent "early failure" mode in popularity prediction analyzed in Sec. 5.2. The small advantage of the memory approach on breakout news detection does not persist across the expanded episode set. This is consistent with our discussion in Sec. 5.3 that memory alone may be insufficient to redirect an agent after it adopts a poor strategy. We will update the table to include the new evaluation results.

TaskTriggerNo learningMemoryBest-of-N + Prune
Breakout news (cov F1)Sleep11.8 ± 1.812.3 ± 0.712.6 ± 0.8
Watcher14.0 ± 1.114.1 ± 0.513.8 ± 0.4
Cron13.5 ± 0.812.2 ± 0.811.6 ± 0.6
Average13.112.812.7
Popularity (tw-F1)Sleep37.6 ± 3.937.4 ± 5.341.0 ± 3.6
Watcher42.3 ± 4.742.7 ± 10.048.5 ± 5.3
Cron47.4 ± 2.550.9 ± 11.948.5 ± 1.4
Average42.443.746.0

We thank the reviewer again for the thoughtful comments and are happy to answer follow-up questions. We would appreciate it if the reviewer could reconsider the score in light of these additional results.

Comments on the paper

1 comment

We thank all the reviewers for their thoughtful comments and suggestions. We have updated the paper draft to incorporate the comments.

  • Robustness across historical episodes [RkSZ, mxEH]. We evaluated trigger mechanisms and use of hindsight feedback on new episodes on 4 tasks. We added Appendix G and added pointers in the Experiment Results section 5.1.
  • Distinction from GAIA2, SentinelBench, and FutureSim, and the capabilities evaluated by ReLiveGym [mxEH]. We slightly revised the introduction and related works. ReLiveGym evaluates persistence reliability and use of hindsight feedback in longer runs; and how observation timing affects performance of diverse tasks beyond monitoring.
  • Evaluation of established harnesses [HbcE]. We added Codex results to Table 2 and a new discussion in Sec. 5.4. Appendix E includes Claude Code results, scheduling behaviors, and failure cases, showing how ReLiveGym can expose gaps in established harnesses and support their development for long-lived tasks.
  • Stale Appendix table fix [RkSZ]. We replaced the stale Web Broker Ops results in the original Table 4 (now Table 5).
  • Autonomous scheduling and cron variants [mxEH]. Appendix F compares fixed cron cadences and agent-managed schedules with and without an initial schedule.

We would appreciate it if the reviewers could reconsider their scores in light of the additional evidence and revisions. As the response period continues, we are happy to answer any follow-up questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.