acceptodds
Under review as a conference paper at ICLR 2027

Does the World Wait While Agents Think? A Temporal Fidelity Audit of Interactive-Agent Benchmarks

Abstract

Interactive agents often spend seconds reasoning before they act, yet benchmarks disagree on what should happen in that interval: some freeze the task, others keep wall, episode, or simulated time running. Calling both settings “dynamic” hides whether state progresses, whether the evaluator charges the interval, and what elapsed time can change afterward. We introduce the Temporal Contract Specification (TCS) to record these three aspects in plain terms. Across a frozen set of eleven pinned interactive-agent suites, public descriptions fully determined the audited contract in 3 cases, 5 required implementation-level qualification, and 3 remained semantically underspecified. Holding action, state, legality, and success fixed, a MiniWoB++ contract swap shows that changing only timer accounting moves the official score by 0.185 at a two-second delay (95% CI [0.165, 0.200]). Controlled probes on RealtimeGym Freeway and a project AndroidWorld lifetime protocol further show that progression and hard expiry can change viability or success under matched actions. As inference latency grows, time spent reasoning is part of the evaluation contract and should be reported explicitly.

Primary area:
datasets and benchmarks
Keywords:
interactive agents, benchmarks, evaluation, temporal fidelity, latency, decision interval, Temporal Contract Specification, real-time evaluation, GUI agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZxddOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that labels like "dynamic" or "real-time" hide three separate design choices about what a benchmark does while an agent is deliberating. It proposes the Temporal Contract Specification (TCS), which records these choices on three multi-hot axes (Sec. 3, Eq. 1, Table 2):

  • state progression (Φ): does task state change during the decision interval?
  • decision-time accounting (Ψ): does the evaluator charge for that interval?
  • opportunity consequence (Γ): what can elapsed time change afterwards?

Using pinned commits, the authors audit 11 interactive-agent suites. Public descriptions fully determine the audited contract in 3 suites, 5 need implementation-level qualification, and 3 remain semantically underspecified (Table 3).

Three fixed-action probes then show that contract differences change measured outcomes even when the chosen action is held fixed:

  • MiniWoB++ timer swap (Sec. 5.1): native vs. compensated timer accounting changes official reward by 0.185 at a 2 s delay (95% CI [0.165, 0.200]).
  • Freeway stale-action probe (Sec. 5.2): 32.5% of initially safe actions become unsafe after one Stay step, vs. ~7% for a null baseline.
  • AndroidWorld call-expiry protocol (Sec. 5.3, Table 4): the delay × lifetime interaction is E2 = 0.633 [0.540, 0.727], pooled over 5 VLMs and 1,800 episodes.
Critical strengths
  1. A precise, checkable vocabulary for an increasingly important evaluation issue. Test-time scaling makes multi-second deliberation routine. Separating whether the world moves (Φ), whether the interval is charged (Ψ), and what time can change (Γ), with scoring-only timers assigned to Ψ rather than Φ (Sec. 3, "How to read the axes"), gives benchmark authors a concrete way to state assumptions that are currently buried in harness code.
  2. Unusually strong reproducibility practice for an audit. The paper provides full commit SHAs for all 11 suites, protocol hashes, inclusion criteria fixed before the audit, explicit HIGH/MEDIUM evidence tiers per row, and retained failure cases (e.g., the 3 of 40 failed scripted MiniWoB clicks are kept rather than excluded).
Critical weaknesses
  1. No evidence that contract differences change the conclusions people draw from these benchmarks. The probes isolate mechanisms, but each effect largely follows from construction:

    • The MiniWoB gap tracks the documented decay formula raw · max(0, 1 − dt/EPmax), which the authors acknowledge (Sec. 5.1).
    • The Android B1+Delay arm is calibrated to B2 latencies of ~4.7–7.8 s against an 8 s lifetime, so collapse at D = 8 is expected. For InternVL3.5-8B, B1+Delay goes from 0.000 at D = 8 to 1.000 at D = 12 (per-model matrix).
    • The Freeway Stay-step injection is explicitly "not native think-time world advance" (appendix).
  2. The central axis is not yet reliably codable. The second coder agrees exactly on only 54.5% of Q1 (state progression) labels and 63.6% of Q3 labels (Sec. 4.3; appendix "Audit reliability"). κ = 1.0 holds only for the coarse claim-to-implementation category, and the second coder labelled prepared evidence packets, not raw repositories. In addition, 6 of the 11 stock rows are MEDIUM tier (Table 3), meaning no full runtime P1 probe was executed. These are exactly the live browser/VM/emulator suites where Q1 is most ambiguous.

Questions for authors
  1. The supplementary disagreement ledger shows Q1 disagreements on six suites (mostly the Mixed qualifier), and the protocol states that coder 2 did not see coder 1's labels. Was coder 2 also blind to the paper's hypotheses, and who adjudicated? Please also reconcile the agreement rates: reliability/analysis.md reports Q1 45.5% and Q3 81.8%, but the paper reports 54.5% and 63.6%. Which definitions does each use?
  2. For the MEDIUM rows (WebArena, VisualWebArena, OSWorld), what prevented a lightweight Think-Freeze probe, e.g. a DOM/VM fingerprint before and after a sleep? Running it would directly address the reliability concern.
  3. Can you show even one closed-loop comparison in which two real agent configurations (e.g. a fast vs. a slow reasoning model) change rank between the native and the compensated or progressing contract on MiniWoB++ or Freeway?
Reviewer_Aar6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks what interactive-agent benchmarks do while an agent is deliberating. It introduces the Temporal Contract Specification (TCS), which records three things:

  • whether task state changes during thinking;
  • whether the evaluator charges the interval;
  • what elapsed time can change, such as score, evidence or whether an action is still valid.

The authors use TCS to audit eleven pinned benchmarks. Public descriptions fully determine the temporal contract in only 3 cases. Fixed-action probes on MiniWoB++, RealtimeGym Freeway and an AndroidWorld protocol then show that changing only the temporal treatment can change measured outcomes.

Critical strengths
  1. It identifies an important and neglected issue. Time has largely been ignored by agent benchmarks, yet inference latency is now routinely several seconds. The paper asks precisely what happens during that interval. Its three questions (p.1) are well defined, and TCS is more precise than labels such as "dynamic" or "real-time". This gives benchmark designers a usable vocabulary and makes clear that results under different contracts aren't comparable. The issue matters even more for settings like voice agents, where timing is central to the interaction.
  2. The fixed-action probe design is elegant. Holding the start state and the action fixed while varying only the temporal contract (p.6, p.15) isolates the effect of timing from differences in the agent's decisions. That makes the outcome changes attributable to the contract itself.
Critical weaknesses
  1. The paper stops at "report the contract" without showing what follows. The conclusion (p.9) only restates the abstract, and ranking reversals are explicitly out of scope (p.9).

    • The outcome probes use scripted actions or fixed policies, mostly under contracts the authors constructed rather than native benchmark modes.
    • None tests whether agent rankings change. The authors' own per-model Android results suggest they do: InternVL3.5-8B is best at D = 15 s (1.000) but scores 0.000 at D = 8 s, where Qwen2.5-VL-3B leads (p.17).

    Since the contribution is mostly conceptual, this missing analysis is what limits the paper's impact.

  2. TCS misses agent awareness, and the audit lacks justification.

    • Agent awareness. Whether the agent is told about deadlines or about the world progressing appears only as checklist item (9) (p.18). It isn't one of the axes and isn't reported anywhere. An informed agent can trade deliberation for speed, so this should be a fourth axis.
    • Options and selection. The options on each axis aren't derived (Table 2), and the choice of the eleven suites is explained in a single sentence (p.5).
    • Labelling. Benchmarks that make no real-time claims are labelled "Underspecified".
    • Coding validation. It is thin: the second coder worked from prepared packets, with 54.5% and 63.6% exact agreement on two of the three axes (p.16).
Questions for authors
  1. (Clarification) How were the eleven suites selected? What was the candidate pool, and why were others excluded?
  2. (Clarification) How were the "typical options" for each axis derived, and how do you know they are complete?
  3. (Clarification) Who performed the coding, and with what expertise? Were the Q1/Q2/Q3 definitions revised after the first agreement check (p.16 reports agreement under "revised" vs. "prior" wording)?
  4. (Clarification) Which probes use modes the benchmarks already support, and which are reimplemented or overlaid?
  5. (Clarification) Were the models in the Android protocol told about the call lifetime or any time constraint?
  6. (Future work) Could TCS be extended with an agent-awareness axis, e.g. none / deadline disclosed / elapsed time observable / progression observable, and reported for the eleven suites?
  7. (Future work) Do agent rankings change when the same models are evaluated closed-loop under the native contract vs. an alternative one on at least one suite? Can you analyse the apparent reversal in your per-model Android results (p.17)?
Reviewer_ZrKSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the problem of thinking time for LLM agents (before actions) and how it should be accounted for or penalized for. The authors propose TCS (temporal contract spcification) that provides an answer to 3 questions for a benchmark: (1) does the world change while thinking, (2) does the benchmark score account for that time, and (3) what does a delay actually cost beyond a lower score? Then, the authors checked 11 benchmarks and found that most don’t document the TCS questions clearly. By holding the agent’s action fixed and changing only the timing rules, they showed that these three rules can change the scores.

Critical strengths
  • The methodology used to analyze benchmarks is very thorough and the results presentation quite honest.

  • The paper defines clear terms that separate behaviors like "time isn't charged" and "delay has no consequences" which are not clearly defined in various benchmarks.

Critical weaknesses
  • It would be nice to have one example where a TCS difference flips the ranking of models in a benchmark.

  • The contribution is unclear, especially over related work that have an explicit type of contract (Gaia2, gameworld-rt, etc)

  • Many of the probe results are self-evident: if you penalize time delay, then delays will hurt the benchmark score.

  • The paper is difficult to read, at least for the non expert on the topic. The ideas seem simple and the paper could be improved.

Questions for authors
  • Can you show a benchmark case where a TCS difference changes the ranking for the benchmark?

  • Are there some recommendations about types of TCS contracts that benchmarks should pick by default depending on the domain? Or what other ways do you expect the community to use your observations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.