- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces WorkDay Simulator and WorkDay-Bench to evaluate agents performing occupational tasks while the environment continues to evolve during deliberation. The benchmark contains 900 simulated workdays across six occupations, incorporating persistent processes, resource constraints, asynchronous events, deadlines, and safety consequences through a textual action–observation interface. Six LLM-based agents are evaluated on completion, timeliness, safety, and economic value. The results reveal a substantial gap between eventual completion and timely execution, with the best aggregate On-Time Rate reaching 59.7%. The paper also presents interaction-protocol ablations, a small human decision study, and Just-in-Time Context Interaction (JIT-C), which adjusts dialogue-history length according to the event triggering a decision and improves timeliness on selected chef scenarios.
Critical strengths
-
Relevant evaluation problem. The paper highlights an important limitation of evaluations that effectively pause the environment during agent deliberation. Concurrent processes, deadlines, and delayed consequences provide a useful setting for studying sustained decision-making under time pressure.
-
Structured benchmark and informative metrics. The benchmark covers six occupations and distinguishes task completion from timeliness, safety, and economic outcomes. Seeded scenario generation, a shared interaction protocol, and execution logs support controlled comparisons and analysis of how failures accumulate.
-
Useful initial empirical analysis. The evaluation goes beyond aggregate performance by examining safety-related responses, occupational differences, and interaction-protocol ablations. JIT-C also provides a concrete example of how context management can affect execution outcomes under the proposed timing model.
Critical weaknesses
-
The setting is overly abstract, and its relevance to realistic embodied agents remains insufficiently validated. WorkDay-Bench abstracts perception and motor control into textual observations and discrete actions, with occupational processes governed by predefined simulation rules. This is reasonable for isolating executive decision-making, but it removes substantial challenges arising from visual uncertainty, navigation, manipulation, and physical execution failures. No validation in a visually grounded, physics-based embodied simulator or on real robotic systems is reported. Consequently, it is unclear whether the observed model rankings, safety trade-offs, and context-management benefits would persist in realistic embodied environments. The study with four annotators on 13 chef decision scenarios offers limited support for this broader correspondence. The current evidence more directly establishes a benchmark for abstract occupational workflow management than for practical embodied competence.
-
The timing conclusions depend heavily on an insufficiently validated token-to-time model. Deliberation advances simulated time according to fixed coefficients: 16,384 input tokens or 1,024 output tokens correspond to one 10-second tick, while actual provider latency is used only for auditing. This offers a standardized accounting convention, but its calibration and sensitivity are not adequately established. In particular, the negative correlation between token usage and timeliness is partly induced by the evaluation protocol itself. Removing deliberation cost demonstrates that the mechanism affects outcomes, but does not establish that the chosen coefficients reflect realistic deployment conditions. Sensitivity analysis is needed to determine whether the main findings remain stable under different timing assumptions.
-
JIT-C lacks sufficiently broad validation and comparison with simpler alternatives. JIT-C is evaluated on only two models and two chef scenario families, despite the benchmark covering six occupations. Comparisons against fixed short-history windows, periodic context refresh, or summary-based context management are missing, making it difficult to attribute the gains specifically to routing based on the event triggering a decision rather than simply reducing context length. Moreover, improved timeliness is accompanied by increased safety violations for Doubao. Broader evaluation, matched-token-budget comparisons, and uncertainty estimates would help establish the generality and practical value of the intervention.
Questions for authors
-
Can representative WorkDay-Bench tasks be validated in a physics-based embodied simulator or on a real robotic system? Which conclusions are expected to remain valid when perception errors, uncertain action durations, and manipulation failures are introduced?
-
How were the token-to-time coefficients selected? How sensitive are model rankings, safety outcomes, and JIT-C improvements to these coefficients or to a timing model based on measured inference latency?