acceptodds
Under review as a conference paper at ICLR 2027

WorkDay-Bench: Can Embodied Agents Keep Up with a Busy Workday?

Abstract

Real work does not pause while an agent thinks: food keeps cooking, patients' conditions change, and cold-chain items expire. What happens when an embodied agent must sustain competent performance across a workday rather than an isolated task? Existing benchmarks offer little evidence: most emphasize short-horizon perception and control or use fixed observe–act turns that reduce continuously evolving work to static snapshots. To isolate long-horizon execution, we factor out raw perception and motor control through a textual action–observation interface, while preserving situated actors, movement, stateful resources, persistent processes, and delayed consequences. We introduce the WorkDay Simulator, where occupational environments evolve continuously on a shared clock: actions consume time and resources, external events arrive asynchronously, and deliberation itself advances the world. Using this simulator, we construct WorkDay-Bench with 900 workdays across six occupations and evaluate six embodied agents under a shared interaction scaffold on completion, timeliness, safety, and economic value. Results expose a sharp gap between task closure and real-time execution: even the best On-Time Rate is only 59.7%. Safety further exposes distinct priority tendencies, with Gemini favoring deadlines and Doubao favoring safety-critical events. To reduce interaction cost under EP dynamics, we introduce Just-in-Time Context Interaction (JIT-C), which improves timeliness and economic value. Code and artifacts: https://github.com/anonymous-workday/Workday-Bench.

Primary area:
datasets and benchmarks
Keywords:
Embodied Agents, Long-Horizon Agent Evaluation, Real-Time Decision Making, Dynamic Simulation Environments, Occupational Benchmarking, Language-Model Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_przPOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces WorkDay Simulator and WorkDay-Bench to evaluate agents performing occupational tasks while the environment continues to evolve during deliberation. The benchmark contains 900 simulated workdays across six occupations, incorporating persistent processes, resource constraints, asynchronous events, deadlines, and safety consequences through a textual action–observation interface. Six LLM-based agents are evaluated on completion, timeliness, safety, and economic value. The results reveal a substantial gap between eventual completion and timely execution, with the best aggregate On-Time Rate reaching 59.7%. The paper also presents interaction-protocol ablations, a small human decision study, and Just-in-Time Context Interaction (JIT-C), which adjusts dialogue-history length according to the event triggering a decision and improves timeliness on selected chef scenarios.

Critical strengths
  • Relevant evaluation problem. The paper highlights an important limitation of evaluations that effectively pause the environment during agent deliberation. Concurrent processes, deadlines, and delayed consequences provide a useful setting for studying sustained decision-making under time pressure.

  • Structured benchmark and informative metrics. The benchmark covers six occupations and distinguishes task completion from timeliness, safety, and economic outcomes. Seeded scenario generation, a shared interaction protocol, and execution logs support controlled comparisons and analysis of how failures accumulate.

  • Useful initial empirical analysis. The evaluation goes beyond aggregate performance by examining safety-related responses, occupational differences, and interaction-protocol ablations. JIT-C also provides a concrete example of how context management can affect execution outcomes under the proposed timing model.

Critical weaknesses
  • The setting is overly abstract, and its relevance to realistic embodied agents remains insufficiently validated. WorkDay-Bench abstracts perception and motor control into textual observations and discrete actions, with occupational processes governed by predefined simulation rules. This is reasonable for isolating executive decision-making, but it removes substantial challenges arising from visual uncertainty, navigation, manipulation, and physical execution failures. No validation in a visually grounded, physics-based embodied simulator or on real robotic systems is reported. Consequently, it is unclear whether the observed model rankings, safety trade-offs, and context-management benefits would persist in realistic embodied environments. The study with four annotators on 13 chef decision scenarios offers limited support for this broader correspondence. The current evidence more directly establishes a benchmark for abstract occupational workflow management than for practical embodied competence.

  • The timing conclusions depend heavily on an insufficiently validated token-to-time model. Deliberation advances simulated time according to fixed coefficients: 16,384 input tokens or 1,024 output tokens correspond to one 10-second tick, while actual provider latency is used only for auditing. This offers a standardized accounting convention, but its calibration and sensitivity are not adequately established. In particular, the negative correlation between token usage and timeliness is partly induced by the evaluation protocol itself. Removing deliberation cost demonstrates that the mechanism affects outcomes, but does not establish that the chosen coefficients reflect realistic deployment conditions. Sensitivity analysis is needed to determine whether the main findings remain stable under different timing assumptions.

  • JIT-C lacks sufficiently broad validation and comparison with simpler alternatives. JIT-C is evaluated on only two models and two chef scenario families, despite the benchmark covering six occupations. Comparisons against fixed short-history windows, periodic context refresh, or summary-based context management are missing, making it difficult to attribute the gains specifically to routing based on the event triggering a decision rather than simply reducing context length. Moreover, improved timeliness is accompanied by increased safety violations for Doubao. Broader evaluation, matched-token-budget comparisons, and uncertainty estimates would help establish the generality and practical value of the intervention.

Questions for authors
  • Can representative WorkDay-Bench tasks be validated in a physics-based embodied simulator or on a real robotic system? Which conclusions are expected to remain valid when perception errors, uncertain action durations, and manipulation failures are introduced?

  • How were the token-to-time coefficients selected? How sensitive are model rankings, safety outcomes, and JIT-C improvements to these coefficients or to a timing model based on measured inference latency?

Reviewer_W3HXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WorkDay Simulator and WorkDay-Bench to evaluate agents executing occupational workflows while the environment continues to evolve during deliberation. The benchmark contains 900 simulated workdays across six occupations, incorporating persistent processes, resource constraints, task dependencies, asynchronous events, and delayed consequences. Agents interact through a textual action–observation interface, and their performance is evaluated in terms of completion, timeliness, safety, and economic value. The paper also proposes Just-in-Time Context Interaction (JIT-C), which selects the dialogue-history window based on the trigger for each decision. Experiments demonstrate a substantial gap between eventual completion and timely execution, as well as improvements from JIT-C on selected chef scenarios.

Critical strengths
  1. The paper addresses a meaningful limitation of conventional agent evaluation. Evaluating whether agents complete tasks on time, while handling competing responsibilities and safety constraints, provides useful information beyond eventual task completion. Modeling deliberation as consuming time is a reasonable way to investigate this problem.
  2. The simulator integrates several useful mechanisms into a coherent evaluation environment. Persistent processes, resource contention, costly information acquisition, dispatch-time feasibility checks, and replayable event logs support detailed analysis of execution failures. The occupational diversity and protocol ablations provide useful initial evidence, and the paper appropriately acknowledges that JIT-C does not consistently improve safety.
Critical weaknesses
  1. The benchmark's validity and the robustness of its main conclusions are insufficiently established.

Section 3.2 converts token counts into simulated time using fixed throughput coefficients, directly penalizing longer generations. Consequently, the reported correlation between token cost and timeliness does not adequately separate model competence, reasoning configuration, and the imposed accounting rule. Table 4 illustrates the sensitivity: removing deliberation costs increases Qwen's On-Time Rate from 11.9% to 82.8% on the evaluated subset. While this confirms that time costs matter, sensitivity analyses across plausible nonzero costs and within-model reasoning budgets are needed to establish the robustness of the conclusions. Moreover, structural validation of generated scenarios does not establish attainable performance under their combined constraints. The absence of reported scheduling baselines or broader human evaluation makes low scores difficult to interpret. The human study, involving four annotators and thirteen local chef scenarios, provides preliminary evidence of priority conflicts but does not validate full-workday execution across occupations.

  1. The intervention and ablation experiments do not sufficiently isolate the contributions of the proposed design choices.

JIT-C shortens context and therefore directly reduces charged deliberation time under the benchmark's accounting rule. Without fixed short-window, periodic-refresh, or approximately token-matched baselines, the experiments cannot determine whether the gains require event-conditioned routing or merely context reduction. Its evaluation is also restricted to two models and two chef scenario families, without reported uncertainty estimates, and Doubao's safety score worsens despite improved timeliness. Similarly, the “w/o Asynchrony” ablation simultaneously changes query scheduling, interruption behavior, and interaction frequency, increasing Doubao's calls from 279 to 821. These experiments demonstrate that the overall interaction configuration matters, but do not cleanly establish the specific benefits of JIT-C’s routing mechanism or asynchronous interaction. Broader evaluation and better-controlled comparisons are needed.

Questions for authors
  1. How stable are the model rankings and main conclusions when the input / output token-throughput coefficients are varied? Do the conclusions persist when comparing different reasoning budgets within the same model?
  2. What performance can rule-based or scheduling baselines achieve under explicitly matched observation and deliberation constraints? Can such results help distinguish agent failures from limitations imposed by the scenarios?
  3. Does JIT-C outperform fixed-window and periodic-refresh strategies with comparable token costs? How much of its advantage remains when deliberation costs are removed?
  4. Can the protocol ablations control interaction frequency and technical timeout handling more closely, and can the key intervention results be extended to additional occupations with paired confidence intervals?
Reviewer_fbQFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce Workday-Bench, which measures how well LLM agents can perform in scenarios where there are changes to environment/task state that can happen while the model thinks. The benchmark is text-based simulation, where time advances in 10 second ticks, actions (and thinking) use up time/etc resources, events arrive on a schedule, and actions from the agent are checked for feasibility at the time they are provided. Time advances by one tick for every 1024 output tokens or 16384 input tokens, following the formalism put forth in the engagement process prior work. WD-Bench considers 900 workdays from 30 templates covering 6 occupations (chef, courier, doctor, etc.) across 5 scenarios. The authors evaluate 6 recent LLMs (flash or similar versions), and on the chef scenarios there is also a small human study.

Critical strengths

The extensions here to the original EP work seem to make sense, code is released and the evaluation seems reasonable over the six models chosen. The scenarios cover a range of plausible application domains, and could be a useful test bed.

Critical weaknesses

The main weakness is the formulation that is somewhat inherited - that number of tokens somehow anchors timeliness across all models (especially when some models may be able to generate those much faster due to architectural or other innovations). This then propagates throughout the entire analysis, since many of these models and rankings can be substantially revised due to this. It also impacts the long-term efficacy of this benchmark, since either this benchmark will not match actual real-world deployments, and/or become highly game-able.

Questions for authors

Please address the core weakness identified above, thanks!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.