- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that labels like "dynamic" or "real-time" hide three separate design choices about what a benchmark does while an agent is deliberating. It proposes the Temporal Contract Specification (TCS), which records these choices on three multi-hot axes (Sec. 3, Eq. 1, Table 2):
- state progression (Φ): does task state change during the decision interval?
- decision-time accounting (Ψ): does the evaluator charge for that interval?
- opportunity consequence (Γ): what can elapsed time change afterwards?
Using pinned commits, the authors audit 11 interactive-agent suites. Public descriptions fully determine the audited contract in 3 suites, 5 need implementation-level qualification, and 3 remain semantically underspecified (Table 3).
Three fixed-action probes then show that contract differences change measured outcomes even when the chosen action is held fixed:
- MiniWoB++ timer swap (Sec. 5.1): native vs. compensated timer accounting changes official reward by 0.185 at a 2 s delay (95% CI [0.165, 0.200]).
- Freeway stale-action probe (Sec. 5.2): 32.5% of initially safe actions become unsafe after one Stay step, vs. ~7% for a null baseline.
- AndroidWorld call-expiry protocol (Sec. 5.3, Table 4): the delay × lifetime interaction is E2 = 0.633 [0.540, 0.727], pooled over 5 VLMs and 1,800 episodes.
Critical strengths
- A precise, checkable vocabulary for an increasingly important evaluation issue. Test-time scaling makes multi-second deliberation routine. Separating whether the world moves (Φ), whether the interval is charged (Ψ), and what time can change (Γ), with scoring-only timers assigned to Ψ rather than Φ (Sec. 3, "How to read the axes"), gives benchmark authors a concrete way to state assumptions that are currently buried in harness code.
- Unusually strong reproducibility practice for an audit. The paper provides full commit SHAs for all 11 suites, protocol hashes, inclusion criteria fixed before the audit, explicit HIGH/MEDIUM evidence tiers per row, and retained failure cases (e.g., the 3 of 40 failed scripted MiniWoB clicks are kept rather than excluded).
Critical weaknesses
-
No evidence that contract differences change the conclusions people draw from these benchmarks. The probes isolate mechanisms, but each effect largely follows from construction:
- The MiniWoB gap tracks the documented decay formula
raw · max(0, 1 − dt/EPmax), which the authors acknowledge (Sec. 5.1). - The Android B1+Delay arm is calibrated to B2 latencies of ~4.7–7.8 s against an 8 s lifetime, so collapse at D = 8 is expected. For InternVL3.5-8B, B1+Delay goes from 0.000 at D = 8 to 1.000 at D = 12 (per-model matrix).
- The Freeway Stay-step injection is explicitly "not native think-time world advance" (appendix).
- The MiniWoB gap tracks the documented decay formula
-
The central axis is not yet reliably codable. The second coder agrees exactly on only 54.5% of Q1 (state progression) labels and 63.6% of Q3 labels (Sec. 4.3; appendix "Audit reliability"). κ = 1.0 holds only for the coarse claim-to-implementation category, and the second coder labelled prepared evidence packets, not raw repositories. In addition, 6 of the 11 stock rows are MEDIUM tier (Table 3), meaning no full runtime P1 probe was executed. These are exactly the live browser/VM/emulator suites where Q1 is most ambiguous.
Questions for authors
- The supplementary disagreement ledger shows Q1 disagreements on six suites (mostly the Mixed qualifier), and the protocol states that coder 2 did not see coder 1's labels. Was coder 2 also blind to the paper's hypotheses, and who adjudicated? Please also reconcile the agreement rates:
reliability/analysis.mdreports Q1 45.5% and Q3 81.8%, but the paper reports 54.5% and 63.6%. Which definitions does each use? - For the MEDIUM rows (WebArena, VisualWebArena, OSWorld), what prevented a lightweight Think-Freeze probe, e.g. a DOM/VM fingerprint before and after a sleep? Running it would directly address the reliability concern.
- Can you show even one closed-loop comparison in which two real agent configurations (e.g. a fast vs. a slow reasoning model) change rank between the native and the compensated or progressing contract on MiniWoB++ or Freeway?