- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Thaw is a deterministic resource-maintenance benchmark for evaluating LLM controllers when the environment advances during inference. A controller directs two workers to gather, process, and deposit resources to maintain three continuously draining buffers. Seven configurations vary the economy, routes, capacity, and maintenance timescale. The paper compares a stopped clock, a clock advanced according to measured response latency, and independently assigned delays sampled from earlier traces. Successful scripted replays establish task feasibility. Experiments with three LLMs show decision-quality differences without inference delay and substantial survival degradation under increased delay in the Short-loop condition. Command-gap and trajectory diagnostics help characterize these failures.
Critical strengths
The authors replay command sequences produced by a planning program or a fixed-rule controller, without LLM inference, and these sequences complete each evaluated task configuration. This shows that the underlying resource-maintenance tasks admit successful execution.
Sampling delays independently of the current response duration reduces the coupling between a decision's content and its measured generation time. The assigned-delay results use the same running-clock tool interface and corroborate the Short-loop degradation observed with measured latency.
Critical weaknesses
-
The main finding—that inference delay can hurt performance in changing environments—overlaps with Kang et al., Win Fast or Lose Slow and Wen et al., Real-Time Reasoning Agents in Evolving Environments. Thaw cites both works. Its new contribution is mainly the controlled benchmark and diagnostic design. The evidence is also limited to three models in one synthetic environment, with the clearest delay effects in Short loop. It is worth exploring the performance from stronger LLMs like Claude-Opus-5.5.
-
The Nemotron delay pool predates a runner correction (Appendix B). Please explain whether the correction could change the sampled delays; if so, check the result with a corrected pool. Some assigned-delay groups contain only two or five episodes. The paper should explain how the sample sizes were chosen and which episodes were retained.
Questions for authors
-
Were all completed episodes meeting the evaluation protocol included? Please clarify “retained,” explain the unequal sample sizes, and report any exclusion criteria and counts. What bootstrap confidence level and resampling procedure were used?
-
What was the runner correction, and could it affect the Nemotron delay pool? If so, does the result hold with a corrected pool? Please also provide the provider, model version, reasoning effort, and generation settings.
2 comments
Response to Rebuttal
Thanks to the authors for the clarifications and for addressing some of my concerns. My main concern remains the limited novelty of the findings: prior work has already demonstrated the latency–performance trade-off, and the new evidence remains concentrated in the Short-loop setting of one synthetic task family. I decide to keep my score unchanged.
And also thank you for the careful review.
Re W1 (overlap with Kang et al. and Wen et al.; model coverage).
We agree that both works show delay effects and that our contribution is the controlled benchmark.
What we changed: Kang et al. vary model size and quantization, which also changes the commands. Wen et al. mainly advance the game by a per-step token budget that limits reasoning, plus a wall-clock test. Thaw compares stopped, own-latency and assigned-latency clocks on the same task, varying delay independently of the current turn with model and token budget fixed (Section 4, lines 456-471). We added Claude Opus 5.5 as the fourth model on Short loop (Section 3.1, lines 248-252) at lower reasoning effort (Appendix B, lines 750-758); adding it was a runner configuration change. All 5 stopped-clock episodes reach the horizon, survival 1.000, but survival falls with own latency: 0.561 at rate 8 (median gap 19 ticks), 0.366 at rate 16 (35.8 ticks), n 10 each (Section 3.3, lines 312-316; Table 3, lines 325-341). On the shared 1.5 s pool at rate 8, it survives 0.992 (median gap 13 ticks), above GPT-5.6 Terra 0.937, GPT-5.6 Luna 0.520 and Nemotron 3 Super 0.382 (Table 3, lines 325-341; Section 3.3, lines 356-358). Results cover one synthetic task family (Section 5, lines 486-489).
Re W2 / Q2 (runner correction, Nemotron pool and settings).
Yes, the correction could change the pool: it stopped charging provider timeouts as thinking time, which previously let timeouts enter as long delays.
What we changed: The old pool, now dropped, had 175 response times from 4 episodes. We rebuilt Nemotron 3 Super's pool from all 538 times in the 10 current-runner own-latency episodes (Appendix B, lines 764-768), then reran and replaced its assigned-latency results. Short loop survival is 0.585, 0.427 and 0.327 at rates 2, 4 and 8, with median gaps 11, 22.5 and 31.5 ticks, n 10 each (Appendix Table 8, lines 1508-1510; Section 3.3, lines 322-323 and 344-346). Survival still falls with delay, as on the old pool (0.513 at rate 2, 0.392 at rate 4). Settings are Amazon Bedrock, region us-east-2, identifiers openai.gpt-5.6-terra, openai.gpt-5.6-luna, nvidia.nemotron-super-3-120b and global.anthropic.claude-opus-5-5. Running clocks use effort low for the GPT models and Claude Opus 5.5 and reasoning off for Nemotron 3 Super; only the effort comparison also uses the higher setting (Appendix B, lines 750-758). Temperature is the provider default, and output caps are 1024 tokens per turn for GPT models and 4096 for the others (Section 3.1, lines 253-257; Appendix B, lines 750-758).
Re W2 / Q1 (sample sizes, retention and bootstrap).
We agree that sample sizes and retention need explanation.
What we changed: Given each episode costs between 5 and 25 USD, we decided to use 10 episodes at the reference charge rates, 5 at the rate below the scripted tolerance, and 5 for the rate 2 control. We unintentionally ran only two rate 2 episodes per group, which we have corrected here. The rate 4 groups now have 10 episodes each. Terra wins 10 of 10, surviving 1.000 (gap 8), and Luna wins 9 of 10, surviving 0.986 (gap 10.25). A win is an episode that reaches the horizon. The rows are in Appendix Table 8: Terra at rates 2 and 4, lines 1490-1491; Luna at rate 4, line 1502. Nemotron 3 Super's rebuilt groups have 10 at every rate (lines 1508-1510). The smallest assigned-latency group is GPT-5.6 Terra's one-tick-per-call control, with all 3 episodes reaching the horizon (Table 3, lines 325-341; Section 5, lines 486-489).
We retained episode any attempt outside the exclusions (Section 3.1, lines 265-269). Of 563 attempts, 542 are retained across 68 groups. Exclusions are seed-0 protocol probes (8), provider failures (8), provider timeouts charged as thinking time (3), and action-budget truncations re-run uncapped on the same seed (2) (Appendix Table 6, lines 1455-1456). None is assigned-latency. Separately, 147 earlier-runner episodes, including 143 from Nemotron 3 Super, are outside the 563 attempts (Appendix B, lines 759-762).
Mean-survival intervals use 95 percent percentile bootstrap with 2000 episode resamples within each group; groups below 4 episodes show their observed range, and identical-survival groups have no interval (Section 3.1, lines 269-272). The analysis code fixes each group's bootstrap seed.