acceptodds
Under review as a conference paper at ICLR 2027

Thaw: Benchmarking LLM Controllers Under Inference Delay

Abstract

An agent’s command can arrive too late when the environment continues to change during inference. Thaw is a benchmark for studying this timing problem in a deterministic outpost task: a large language model (LLM) directs simulated scientists to deliver supplies before shortages cause failure. We measure survival as the share of a fixed-duration episode completed before failure. To show that the task can be completed, we replay commands found by a planning program or fixed-rule controller. Every replay keeps the outpost operating for the entire fixed duration without any LLM calls. We compare a clock that pauses during inference with one that advances proportionally to each model response’s measured duration. In a task variant with brief gathering, processing and travel times, each evaluated LLM has lower survival when response duration advances the simulation. As the simulation advances more per second of inference, one LLM goes from completing the episode with the clock stopped to surviving less than half of it on average. In a separate timing test, each response advances the simulation by a delay sampled from that model’s earlier response times, independently of the current response’s duration. As we increase how far each sampled delay advances the simulation, survival falls for every model. Benchmarks that pause the environment during inference can miss this source of control failure.

Primary area:
datasets and benchmarks
Keywords:
llms, benchmarks, realtime

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zVzjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Thaw is a deterministic resource-maintenance benchmark for evaluating LLM controllers when the environment advances during inference. A controller directs two workers to gather, process, and deposit resources to maintain three continuously draining buffers. Seven configurations vary the economy, routes, capacity, and maintenance timescale. The paper compares a stopped clock, a clock advanced according to measured response latency, and independently assigned delays sampled from earlier traces. Successful scripted replays establish task feasibility. Experiments with three LLMs show decision-quality differences without inference delay and substantial survival degradation under increased delay in the Short-loop condition. Command-gap and trajectory diagnostics help characterize these failures.

Critical strengths

The authors replay command sequences produced by a planning program or a fixed-rule controller, without LLM inference, and these sequences complete each evaluated task configuration. This shows that the underlying resource-maintenance tasks admit successful execution.

Sampling delays independently of the current response duration reduces the coupling between a decision's content and its measured generation time. The assigned-delay results use the same running-clock tool interface and corroborate the Short-loop degradation observed with measured latency.

Critical weaknesses
  1. The main finding—that inference delay can hurt performance in changing environments—overlaps with Kang et al., Win Fast or Lose Slow and Wen et al., Real-Time Reasoning Agents in Evolving Environments. Thaw cites both works. Its new contribution is mainly the controlled benchmark and diagnostic design. The evidence is also limited to three models in one synthetic environment, with the clearest delay effects in Short loop. It is worth exploring the performance from stronger LLMs like Claude-Opus-5.5.

  2. The Nemotron delay pool predates a runner correction (Appendix B). Please explain whether the correction could change the sampled delays; if so, check the result with a corrected pool. Some assigned-delay groups contain only two or five episodes. The paper should explain how the sample sizes were chosen and which episodes were retained.

Questions for authors
  1. Were all completed episodes meeting the evaluation protocol included? Please clarify “retained,” explain the unequal sample sizes, and report any exclusion criteria and counts. What bootstrap confidence level and resampling procedure were used?

  2. What was the runner correction, and could it affect the Nemotron delay pool? If so, does the result hold with a corrected pool? Please also provide the provider, model version, reasoning effort, and generation settings.

2 comments

And also thank you for the careful review.

Re W1 (overlap with Kang et al. and Wen et al.; model coverage).

We agree that both works show delay effects and that our contribution is the controlled benchmark.

What we changed: Kang et al. vary model size and quantization, which also changes the commands. Wen et al. mainly advance the game by a per-step token budget that limits reasoning, plus a wall-clock test. Thaw compares stopped, own-latency and assigned-latency clocks on the same task, varying delay independently of the current turn with model and token budget fixed (Section 4, lines 456-471). We added Claude Opus 5.5 as the fourth model on Short loop (Section 3.1, lines 248-252) at lower reasoning effort (Appendix B, lines 750-758); adding it was a runner configuration change. All 5 stopped-clock episodes reach the horizon, survival 1.000, but survival falls with own latency: 0.561 at rate 8 (median gap 19 ticks), 0.366 at rate 16 (35.8 ticks), n 10 each (Section 3.3, lines 312-316; Table 3, lines 325-341). On the shared 1.5 s pool at rate 8, it survives 0.992 (median gap 13 ticks), above GPT-5.6 Terra 0.937, GPT-5.6 Luna 0.520 and Nemotron 3 Super 0.382 (Table 3, lines 325-341; Section 3.3, lines 356-358). Results cover one synthetic task family (Section 5, lines 486-489).

Re W2 / Q2 (runner correction, Nemotron pool and settings).

Yes, the correction could change the pool: it stopped charging provider timeouts as thinking time, which previously let timeouts enter as long delays.

What we changed: The old pool, now dropped, had 175 response times from 4 episodes. We rebuilt Nemotron 3 Super's pool from all 538 times in the 10 current-runner own-latency episodes (Appendix B, lines 764-768), then reran and replaced its assigned-latency results. Short loop survival is 0.585, 0.427 and 0.327 at rates 2, 4 and 8, with median gaps 11, 22.5 and 31.5 ticks, n 10 each (Appendix Table 8, lines 1508-1510; Section 3.3, lines 322-323 and 344-346). Survival still falls with delay, as on the old pool (0.513 at rate 2, 0.392 at rate 4). Settings are Amazon Bedrock, region us-east-2, identifiers openai.gpt-5.6-terra, openai.gpt-5.6-luna, nvidia.nemotron-super-3-120b and global.anthropic.claude-opus-5-5. Running clocks use effort low for the GPT models and Claude Opus 5.5 and reasoning off for Nemotron 3 Super; only the effort comparison also uses the higher setting (Appendix B, lines 750-758). Temperature is the provider default, and output caps are 1024 tokens per turn for GPT models and 4096 for the others (Section 3.1, lines 253-257; Appendix B, lines 750-758).

Re W2 / Q1 (sample sizes, retention and bootstrap).

We agree that sample sizes and retention need explanation.

What we changed: Given each episode costs between 5 and 25 USD, we decided to use 10 episodes at the reference charge rates, 5 at the rate below the scripted tolerance, and 5 for the rate 2 control. We unintentionally ran only two rate 2 episodes per group, which we have corrected here. The rate 4 groups now have 10 episodes each. Terra wins 10 of 10, surviving 1.000 (gap 8), and Luna wins 9 of 10, surviving 0.986 (gap 10.25). A win is an episode that reaches the horizon. The rows are in Appendix Table 8: Terra at rates 2 and 4, lines 1490-1491; Luna at rate 4, line 1502. Nemotron 3 Super's rebuilt groups have 10 at every rate (lines 1508-1510). The smallest assigned-latency group is GPT-5.6 Terra's one-tick-per-call control, with all 3 episodes reaching the horizon (Table 3, lines 325-341; Section 5, lines 486-489).

We retained episode any attempt outside the exclusions (Section 3.1, lines 265-269). Of 563 attempts, 542 are retained across 68 groups. Exclusions are seed-0 protocol probes (8), provider failures (8), provider timeouts charged as thinking time (3), and action-budget truncations re-run uncapped on the same seed (2) (Appendix Table 6, lines 1455-1456). None is assigned-latency. Separately, 147 earlier-runner episodes, including 143 from Nemotron 3 Super, are outside the 563 attempts (Appendix B, lines 759-762).

Mean-survival intervals use 95 percent percentile bootstrap with 2000 episode resamples within each group; groups below 4 episodes show their observed range, and identical-survival groups have no interval (Section 3.1, lines 269-272). The analysis code fixes each group's bootstrap seed.

Reviewer_zVzjOpenReview ↗

Response to Rebuttal

Thanks to the authors for the clarifications and for addressing some of my concerns. My main concern remains the limited novelty of the findings: prior work has already demonstrated the latency–performance trade-off, and the new evidence remains concentrated in the Short-loop setting of one synthetic task family. I decide to keep my score unchanged.

Reviewer_HmjjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces Thaw, a benchmark for evaluating LLM-based controllers under continuously evolving environments, where action lag arises from inference delay. The task is formulated as a deterministic resource supply-chain simulation: two scientists must collect, process, and transport resources to sustain multiple subsystems of an outpost. Three time protocols are designed in this paper:

  1. Stopped clock: the environment remains static during model inference
  2. Own-latency clock: the environment advances according to the model’s actual response time
  3. Assigned-latency clock: environment advancement duration is sampled from a pre-recorded latency pool. The authors report survival rate, command interval, observation ratio, and resource carry capacity across a set of models, task configurations, and inference intensities. They conclude that inference latency substantially degrades the task success rate of LLM controllers.
Critical strengths
  1. The proposed issue is practical and addresses a frequently overlooked dimension in existing LLM agent benchmarks: agents do not perform reasoning within static environments, and response latency can alter environmental states.
  2. The three proposed time protocols are technically resonable. The work distinguishes controller performance under environment pausing, latency induced by the model’s real response time, and exogenous latency independent of the current action. Beyond end-task success rate, the paper examines control behaviors and analyzes behavioral variations across diverse task conditions.
Critical weaknesses
  1. The core experiments fail to sufficiently validate the central claim that inference delay reduces the survival rate of LLM controllers. The current experimental design contains multiple factors, such as inference duration, number of model invocations, tool-use behaviors, and agent scheduling order. Therefore, the performance degradation might arises from model behavior, response latency, tool-use strategies, etc., instead of the claimed inference delay.
  2. This work is incomplete with limited LLM coverage. Only three LLMs are tested. It remains unclear how other model families including Claude, Gemini, Qwen, GLM and Kimi would perform on this benchmark.
  3. This paper requires improvements in writing. The figures and tables are difficult to interpret. Furthermore, some details are missing, such as the total
Questions for authors
  1. The core experiments cannot sufficiently validate your central claim that inference delay reduces the survival rate of LLM controllers. Your experimental design involves multiple confounding factors including inference duration, the number of model invocations, tool-use behaviors and agent scheduling order. Could you clarify how you rule out the possibility that the performance degradation comes from model behaviors, tool-use strategies or other confounders rather than inference delay itself?
  2. The evaluation only covers three LLMs, leading to limited model coverage and incomplete results. Could you discuss how other model families such as Claude, Gemini, Qwen, GLM and Kimi are expected to perform on your Thaw benchmark? Are there any plans to extend the evaluation to these models?
  3. We find this work is incomplete and difficult to follow, as many figures and tables are hard to interpret, many details are missing, such as the statistical information benchmark, what types of tasks are included in this benchmark?
1 comment

Thank you also for the careful review.

Re W1 / Q1 (confounders).

We agree in part: model behavior matters, since the models already differ with the stopped clock, but under the assigned-latency clock only the delay changes, so shifts in calls, tool choices and scientist choice are effects of delay, not separate causes.

What we changed: Each turn returns exactly one tool call (Section 2.4, lines 184-186), and the model picks the scientist it commands. We fix model, seven tools, prompt, condition and seeds; each episode-seeded draw is independent of the turn, and identical seeds give identical delays (Section 2.4, lines 197-209). In a new control, every model draws the same delays, averaging 0.75 s or 1.5 s (Section 2.4, lines 199-201), on Short loop at 8 Hz, seeds 201 to 210, n 10 per group. Increasing the mean lowers survival from 1.000 to 0.937 for GPT-5.6 Terra, 0.968 to 0.520 for GPT-5.6 Luna, and 0.509 to 0.382 for Nemotron 3 Super (Table 3; Section 3.3, lines 329-358). Survival is the mean fraction of the horizon survived (Table 3 caption, lines 326-327). At one tick per call with unchanged tools and prompt, GPT-5.6 Luna survives 1.000 in 5 of 5 episodes and GPT-5.6 Terra in 3 of 3, matching the stopped clock. Nemotron 3 Super survives 0.636 versus 0.719 with the stopped clock, so its play accounts for part of its shortfall (Section 3.3, lines 312-314 and 349-351). Timing includes request preparation, network and provider queueing, and generation, all of which a deployed controller waits for (Section 2.4, lines 192-196). These controls cover Short loop, where the delay effect is clearest.

Re W2 / Q2 (model coverage).

We agree and added a Claude model.

What we changed: Claude Opus 5.5 runs at the lower reasoning effort on Short loop only (Section 3.1, lines 248-252; Table 3 caption, line 325). Stopped clock survival is 1.000 in all 5 episodes; own-latency clock survival is 0.561 at 8 Hz (gap 19.0 ticks) and 0.366 at 16 Hz, n 10 each (Section 3.3, lines 312-316). Shared 1.5 s delays yield 0.992 (gap 13.0 ticks, n 10), above every other model on that pool, GPT-5.6 Terra's 0.937 included (Table 3, lines 329-341; Section 3.3, lines 356-358). On the 1.5 s pool it survives best, yet under its own measured latency it loses much of its survival. Adding another family, such as Gemini, Qwen, GLM or Kimi, is a configuration change in the code we will open source (Reproducibility Statement, line 518).

Re W3 / Q3 (presentation, statistics and task types).

We agree: missing statistics and unclear figures and tables hindered reading.

What we changed: A new Table 2 reports 7 conditions: five long-loop with an 18000-tick horizon and two short-loop with a 3500-tick horizon. It covers 3 clocks, 4 models, 3 scripted controllers, 68 run groups and 542 counted of 563 attempted episodes (Section 2.4, lines 234-241). Thaw is one task: two scientists gather, process and deliver resources to keep three outpost subsystems supplied. Table 1 defines condition changes in demand, supply, routes, loop duration and buffer capacity (Section 2.4, lines 217-231). Every condition has a successful command trace. Appendix Table 6 (lines 1405-1457) gives exclusion counts by reason for every run group. Intervals use 95 percent episode-level percentile bootstrap, 2000 resamples (Section 3.1, lines 263-272). Figure 4's caption explains lines joining model groups in charge-rate order, stopped-clock stars and interval bars, and defines command gap and scripted tolerance (Section 3.4, lines 400-406). The new Table 3 collects the delay controls and intervals (Section 3.3, lines 325-341).

Reviewer_iBzPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Thaw, a benchmark for evaluating LLM controllers when the environment continues to evolve during model inference. In a deterministic outpost-management simulation, an LLM controls two scientists who gather, process, and deliver resources to maintain three subsystems. The benchmark compares three timing protocols: a stopped clock, an own-latency clock driven by measured response time, and an assigned-latency clock using delays sampled from previously recorded model responses. The results show that survival can decrease substantially when inference delay advances the environment, especially in configurations with short action loops.

Critical strengths

1.Important benchmark dimension. Treating inference delay as part of the environment dynamics is a valuable contribution for the Datasets and Benchmarks track. The stopped-, own-, and assigned-latency protocols provide a useful framework for separating decision quality from timing sensitivity.

2.Careful task construction and documentation. Every evaluated condition has a successful command trace, reducing the risk of attributing an inherently unsolvable task to model failure. The paper also gives unusually detailed descriptions of the simulator, controller interface, metrics, and reproduction procedure.

Critical weaknesses

1.Imperfectly controlled timing comparisons. The stopped-clock controller receives two additional clock-management tools, whereas the running-clock controllers do not. Consequently, stopped-clock versus running-clock performance differs in both inference timing and available actions.

2.Limitations of the assigned-delay intervention. The delay pools come from truncated source episodes, are not matched by episode progress or context length, and vary substantially in size across models. One pool also predates a runner correction. This makes the intervention less controlled than an externally shared or progress-matched delay distribution.

Questions for authors

1.Tool-set confound: Do the authors have results for a stopped-clock or zero-latency condition using exactly the same seven-tool interface as the running-clock conditions? This would clarify how much of the observed difference is attributable to latency rather than to the two additional clock tools.

2.Episode retention: How many episodes were attempted but not retained in each model/protocol group, and what were the exclusion reasons? In particular, were API failures, malformed tool calls, timeouts, or context-summary failures correlated with the model or timing condition?

1 comment

Thank you for the careful review.

Re W1 / Q1 (tool confound).

We agree that stopped-versus-running comparisons confound delay with the two extra clock tools, free-inference prompt and per-command tick cost.

What we changed: Section 2.4, lines 208-215 now states the three differences, and we added the control you asked for. In the paper, GPT-5.6 Terra and GPT-5.6 Luna are the larger and smaller frontier LLMs, Nemotron 3 Super the open-weight LLM and Claude Opus 5.5 the fourth LLM (Section 3.1, lines 248-250). The control uses the seven-tool running-clock interface and prompt at 8 Hz on Short loop, advancing exactly one tick per call. Luna reached the horizon in 5 of 5 episodes and Terra in 3 of 3, survival 1.000 (Table 3 and Section 3.3, lines 347-351). Stopped-clock survival was 1.000 for Terra (n 5) and Luna (n 10; Appendix Table 8, lines 1484-1486 and 1498-1500). So the running-clock tools and prompt alone do not lower Terra's or Luna's survival; delay does. Nemotron survived 0.636, near 0.719 stopped and above 0.322 with its own latency at 8 Hz (Table 3; Appendix Table 8, lines 1504-1507).

Re W2 (delay pools).

We agree that recorded pools are a weaker control than a shared distribution.

What we changed: Shared pools are uniform from 0.5625 to 0.9375 s or 1.125 to 1.875 s (means 0.75 s and 1.5 s). An episode-seeded generator gives every model the same delays for the same seed, whatever its own latency (Section 2.4, lines 197-201). On Short loop at 8 Hz (10 episodes per group), survival on the shorter and longer pools was 1.000 and 0.937 for Terra, 0.968 and 0.520 for Luna, and 0.509 and 0.382 for Nemotron. Opus 5.5 had survival 0.992 on the longer pool only (Table 3 and Section 3.3, lines 351-358). Survival fell as shared delay rose, and Terra and Luna outlasted Nemotron on both pools. We rebuilt Nemotron's recorded pool from 10 current-runner episodes and reran its assigned-latency groups, replacing the old pool. It holds 538 durations (Appendix Table 11, lines 1558-1564); survival is 0.585, 0.427 and 0.327 at charge rates 2, 4 and 8 Hz (Appendix Table 8, lines 1508-1510). For Terra at 8 Hz and the same median command gap of 13.0 ticks, survival is 0.937 on the longer shared pool versus 0.772 on its recorded pool (Section 3.3, lines 355-357). Terra's recorded pool reaches 6.2 s and the longer shared pool stops at 1.875 s, so the median gap alone does not set survival. Recorded pools come from source episodes that end before the horizon, so they may underrepresent later parts of long episodes (Section 2.4, lines 205-207).

Re Q2 (retention).

We agree that the submission should have reported counts and defined retained episodes.

What we changed: Section 3.1, lines 263-269 defines retention, with 542 of 563 attempts retained. Appendix Table 6, lines 1405-1457 lists all 21 exclusions by group. We excluded the 8 seed-0 probes that open each batch to check the protocol (Luna and Terra, stopped and own-latency clocks). All 8 provider failures were Nemotron's: 7 stopped-clock, 1 own-latency. The 3 timeout exclusions were own-latency episodes (Luna 2, Terra 1), where provider timeouts are charged as thinking time. Terra's 2 own-latency action-budget episodes were rerun uncapped on the same seed. No assigned-latency episodes were excluded. Failed context summaries are logged without advancing the clock, not excluded (Section 2.4, lines 195-196). Separately, the data validity rule dropped whole batches run on an earlier runner version, 147 episodes (143 of them Nemotron's), which are not among the 563 attempts (Appendix B, lines 759-761).

Comments on the paper

1 comment
Area_Chair_RKyXOpenReview ↗

Reminder: Please respond to the author rebuttal

Dear Reviewers, the authors have responded to your reviews with a revised paper and new control experiments, and the author-reviewer discussion ends soon. Please read their response, post a brief reply on whether your concerns are addressed, and update your scores if your assessment has changed. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.