- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ReLiveGym, an evaluation environment for long-lived LLM agents that act intermittently over weeks of chronologically replayed real-world data. Its eight tasks cover forecasting, event detection, and recurring automation, with evaluation based on both correctness and action timing under limited budgets. The authors evaluate eight models across three triggering mechanisms—sleep, programmatic watchers, and scheduled jobs—and investigate hindsight feedback through outcome memory and best-of-N selection with pruning. The experiments highlight task-dependent effects of model and harness choices, persistent failures such as instruction drift and poor initial strategies, and the benefits and limitations of feedback-based improvement.
Critical strengths
- The authors introduce a new category of long-horizon tasks that closely aligns with my understanding of agent research. Beyond tasks such as coding and workspace activities that involve many interaction turns, I believe another important category consists of tasks that span long periods of time. Such tasks are common in real-world applications but have often been overlooked. This paper systematically characterizes representative task types and concrete tasks, establishing a valuable evaluation direction that could meaningfully contribute to the future development of agents.
- The work also makes a substantial experimental contribution. The authors build a relatively comprehensive simulation environment that recreates these scenarios through historical data replay. This is a clever design that requires considerable engineering effort and can facilitate and accelerate future evaluations.
Critical weaknesses
- The paper does not sufficiently examine the impact of mature agent harnesses used in practice. For example, how effectively and efficiently would agents perform these tasks within well-developed harnesses such as Claude Code or Codex? Could many of these tasks already be handled well by such systems? I believe this question is important for assessing how well the findings generalize to real-world deployments.
- Since monitoring fundamentally involves achieving timely responses with limited resources, the paper should systematically evaluate the relationship between budget, polling frequency, and performance, rather than simply impose a budget cap. It should also distinguish the costs of model calls from those of monitoring the environment.
Questions for authors
Could the authors provide additional experimental results using Claude Code or Codex as the agent harness?
5 comments
Finding 1. Agents using mainstream harnesses adopt diverse scheduling strategies, often relying on cron.
Most runs use variants of cron to handle long-lived tasks. Codex with GPT-5.6-luna and Claude Code with Sonnet 5.5 select relatively consistent triggering mechanisms across repeated runs, whereas Claude Code with Haiku 4.5 shows more variation.
The table below summarizes the main triggering mechanisms selected across repeated runs and episodes.
| Task | Codex + GPT-5.6-luna | Claude Code + Haiku 4.5 | Claude Code + Sonnet 5.5 |
|---|---|---|---|
| Breakout news | cron, fresh ×9 | cron, fresh ×7; cron + resume ×2 | cron, fresh ×9 |
| Resolution detection | cron, fresh ×9 | cron + resume ×6; cron → script ×2; CronCreate only | cron, fresh ×9 |
| Forecast portfolio | cron, fresh ×3 | CronCreate only; cron + resume; cron, fresh | cron, fresh ×3 |
| Popularity prediction | cron, fresh ×3 | cron + resume ×2; cron → script | cron → script ×3 |
| Daily digest | cron, fresh ×2; cron → script | cron, fresh ×2; cron → script | cron → script ×3 |
| Market price check | cron, fresh ×2; ended without scheduling ×1 | cron, fresh ×2; cron → script | cron → script ×3 |
| Web filing tracking | cron, fresh ×3 | in-run wait; cron, fresh; cron + resume | program ×2; program → CLI |
| Web broker ops | cron, fresh ×3 | cron + resume ×2; cron, fresh | program → CLI ×3 |
The selected mechanism also depends on the model–harness combination. For example, for popularity prediction, all three Codex + GPT-5.6-luna runs use fresh cron sessions, while all three Claude Code + Sonnet 5.5 runs use cron-triggered scripts.
Agents also combine cron for long-term scheduling with in-run mechanisms for shorter waits, as illustrated below.
| Agent | Task, seed | Main trigger | Native in-run wait added | What happened |
|---|---|---|---|---|
| Codex + GPT-5.6-luna | Popularity prediction, seed 1 | Hourly cron runs | clock.sleep, 42 times, up to 22 min | When all 10 recommendation slots were occupied, the agent checked when the oldest slot would become available, slept until then, and used it. ("The first slot frees at 13:06:26 UTC. I'll stay active through it.") |
| Codex + GPT-5.6-luna | Web filing tracking, seed 3 | Hourly cron runs | clock.sleep, 10 min | Before the 09:30 ET market open (a filing deadline), it slept 10 minutes inside the run for "one final fresh mirror check shortly before the deadline". |
| Claude Code + Haiku 4.5 | Breakout news, episode 3, seed 1 | Hourly cron runs | ScheduleWakeup every 15 min | It added 15-minute in-session checks alongside cron, but one session mistakenly remained active for 40 h and exhausted the budget. |
Finding 2. To answer the reviewer's question about the performance of well-developed harnesses, we compare Codex and Claude Code with hand-rolled harness with three triggering mechanisms reported in the paper, and summarize main findings:
- With GPT-5.6-luna, Codex often performs close to the best of the three triggering mechanisms ({\sleep, watcher, cron}) evaluated in the paper, e.g., on breakout news, reslution detection, and popularity prediction tasks. There are substantial gaps on certain tasks such as forecast portfolio, where Codex triggers too frequently and get distracted by unimportant news.
- With Haiku 4.5, Claude Code generally performs worse than the best custom-harness configuration. One possible explanation is a mismatch between Haiku 4.5 and the latest Claude Code harness.
- We also report Sonnet 5.5 results as an additional reference for performance with the latest Claude Code harness.
| Task | luna, paper: best | luna, paper: worst | luna, E1 (Codex) | haiku, paper: best | haiku, paper: worst | haiku, E1 (Claude Code) | sonnet, E1 (Claude Code) |
|---|---|---|---|---|---|---|---|
| Breakout news | 15.5 ± 0.8 (watcher) | 11.2 ± 0.7 (sleep) | 15.3 ± 0.4 | 8.6 ± 2.5 (sleep) | 5.2 ± 2.3 (watcher) | 3.5 ± 0.4 | 5.4 ± 3.1 |
| Resolution detection | 20.0 ± 3.9 (sleep) | 17.1 ± 4.6 (cron) | 20.5 ± 0.5 | 5.2 ± 1.8 (sleep) | 2.6 ± 0.6 (cron) | 0.9 ± 1.5 | 9.7 ± 6.8 |
| Popularity prediction | 43.3 ± 11.3 (sleep) | 38.2 ± 15.6 (watcher) | 42.1 ± 9.6 | 9.2 ± 4.5 (sleep) | 4.2 ± 3.0 (cron) | 8.7 ± 13.1 | 70.2 ± 5.1 |
| Forecast portfolio | 31.7 ± 12.2 (watcher) | 7.0 ± 24.1 (cron) | -6.9 ± 14.4 | -4.3 ± 4.8 (watcher) | -14.1 ± 19.1 (sleep) | -8.6 ± 12.7 | 30.2 ± 7.2 |
| Daily digest | 100.0 ± 0.0 (cron) | 98.9 ± 2.0 (sleep, watcher) | 98.6 ± 2.4 | 95.6 ± 3.5 (cron) | 42.4 ± 23.6 (sleep) | 66.6 ± 53.8 | 100.0 ± 0.0 |
| Market price check | 99.4 ± 0.4 (cron) | 68.2 ± 42.9 (sleep) | 65.9 ± 57.1 | 96.7 ± 1.7 (cron) | 35.2 ± 35.3 (sleep) | 99.3 ± 1.0 | 99.7 ± 0.5 |
| Web filing tracking | 76.2 ± 13.7 (watcher) | 10.0 ± 10.3 (sleep) | 65.1 ± 53.6 | 21.2 ± 22.9 (watcher) | 1.6 ± 1.6 (cron) | 16.4 ± 11.5 | 32.8 ± 56.8 |
| Web broker ops | 97.9 ± 3.7 (cron) | 0.0 ± 0.0 (watcher) | 96.8 ± 3.2 | 90.3 ± 16.8 (cron) | 11.8 ± 3.7 (sleep) | 40.9 ± 37.7 | 100.0 ± 0.0 |
We thank the reviewer for the thoughtful comments and for recognizing the contributions of our paper. We address the reviewer's questions below.
Q1. The paper does not sufficiently examine the impact of mature agent harnesses used in practice. For example, how effectively and efficiently would agents perform these tasks within well-developed harnesses such as Claude Code or Codex? Could many of these tasks already be handled well by such systems?
We agree that evaluating widely used agent harnesses is important. ReLiveGym supports harnesses such as Claude Code and Codex, and we have added experiments with the following model–harness combinations:
Experiment setup
Empirically, we observe agents select a range of triggering mechanisms, summarized below.
clock.sleepin Codex,ScheduleWakeuporMonitorin Claude Code). These mechanisms often have harness-enforced time limits (e.g., 1 hour forScheduleWakeup).codex exec "…",claude -p "…"). Information persists between runs only through files.claude -p --continue,codex exec resume)CronCreate, which is unsuitable for scheduling in headless mode.