- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces Interactive Visual Gym, a closed-loop testbed in which a simulated user performs a physical task in an executable environment while a visual assistant observes the user’s view and guides them verbally. Three domains are covered: cooking, household activities, and mobile-interface use. The authors build a persona-conditioned user simulator and a library of scheduled out-of-plan events that let them probe specific assistant capabilities when such events occur. The resulting IVE benchmark tests a list of models and even frontier models fall short on these interactive, visual assistance tasks.
Critical strengths
- Fills a real gap. The community lacks an interactive benchmark for embodied assistance across diverse tasks. Compared with text-only or digital agents, where executable environments and simulated users are relatively easier to obtain, high-quality physical simulators paired with credible user simulators remain scarce. To my knowledge this is the first work that puts a physical simulated environment, a behaviorally rich user simulator, and a visual assistant into a single closed-loop testbed.
- Thoughtful, comprehensive design. The scheduled out-of-plan events and the five persona axes cover many natural edge cases with impressive care, whereas prior work typically relies on three or four simplistic variants on both ends. Simulation time and wall-clock latency are properly separated, so streaming models can be tested in their native mode; ASR can be connected in, which makes the testbed even more realistic.
Critical weaknesses
The motivation for a closed-loop, simulator-driven setup is to move beyond fixed trajectories: the assistant should influence user behavior, the user alters the environment and observations, and those changes feed back to the assistant. Basically, user, assistant, and environment co-evolve. However, the paper lacks explicit analysis or evaluation on these dynamics once an out-of-plan event is triggered, nor how different models differ in handling the evolving interaction. The reported metrics capture some of it, but they do not characterize, e.g., how recovery unfolds after an error commits, how guidance quality changes under persona push-back across turns, or how an assistant’s behavior shifts as the user diverges from the plan. Evaluating these dynamic shifts is precisely where the core challenge lies, and the HCI literature on real-user interaction offers established paradigms the authors could draw on. This may be a reason to lower my score if this is not addressed to some extent, at minimum with analyses on existing traces.
Other weaknesses (not critical)
- In-time success entangles timeliness with correctness. “Any success” appears only in the user-model sensitivity table (Tab. 5) and not for the main body of the paper. Table 1 shows e.g., Gemini 3.8 Flash at 30.6% vs. Qwen3-Omni-30B at 26.2% on CookSim, while one cannot tell whether a model gives worse guidance or simply cannot finish within budget because it spends longer reasoning. It is recommended to report any-success (and ideally time-to-completion) alongside in-time success for all assistants in all three environments.
- LLM-as-a-judge reliability Several recent research has revealed that LLM-as-a-judge may not provide reliable, fine-grained judgement. It is recommended to have a small-scale human annotation of the rubric items and see whether they are aligned well. Also, binary scoring might not be appropriate, since there might be middle ground in between.
Comments
- Mobile interface. While mobile-interface use is important, ScreenSim sits loosely alongside cooking and household tasks: it has discrete gestures, no latency charging, and seems relatively simpler. I would consider dropping it in favor of deepening or extending the two embodied domains, though this is the authors' call.
- Scene-graph ablation. Since the ground-truth scene graph is available, an ablation giving the assistant partial or full environmental state would disentangle whether perception or task understanding is the bottleneck.
Questions for authors
- What was the reason for building CookSim from scratch rather than using existing platforms such as BEHAVIOR or CookBench? How extensible your platform is, not only for adding tasks or scenarios, but for supporting richer evaluation of interaction dynamics?
- What is the rationale of using Human-communicable action space instead of physical actions such as move, go to, pick up, etc. Can the verbal action always successfully translated into low-level physical actions?
Muzhen Cai et al. CookBench: A Long-Horizon Embodied Planning Benchmark for Complex Cooking Scenarios. arxiv. 2025
2 comments
Q3. How reliable are the judge scores, and why use binary rubrics?
The judge scores align with direct human ratings, beyond the preference-ranking agreement reported in the paper. Participants rated individual episodes for helpfulness and correctness; we compare those ratings with the corresponding judge scores below.
| Judge score / human rating | Episodes | Spearman ρ [95% CI] |
|---|---|---|
| Interaction quality / helpfulness | 189 | 0.50 [0.37, 0.61] |
| Factual grounding / correctness | 189 | 0.64 [0.53, 0.74] |
| Actionable guidance / helpfulness | 187 | 0.39 [0.26, 0.50] |
We use binary rubric items because they express specific requirements that can be checked against the observed world state and interaction history. This follows CheckEval’s finding that decomposing evaluation into explicit criteria improves reliability and agreement with human judgments (Lee et al., 2025). Aggregating these judgments preserves differences between responses that satisfy different requirements, as in rubric-based evaluations such as HealthBench (Arora et al., 2025). We average across dimensions to avoid weighting them by their number of rubric items, and require factual grounding so that incorrect guidance cannot be compensated by presentation quality.
Q4. Why include ScreenSim alongside the embodied environments?
Thank you for the suggestion. In the camera-ready version, we will increase the difficulty of ScreenSim tasks to better match the other two simulators and improve setting consistency across different simulators.
Q5. Would access to scene-graph state clarify the bottleneck?
Access to explicit world state improves interaction quality, but does not improve task success in this ablation. Thank you for suggesting this comparison. We evaluate two assistants on ten matched CookSim cases with visual input alone or visual input plus a current-state readout, including object contents, preparation states, and cooking progress (40 episodes).
| Assistant | In-time success | Interaction quality | Factual grounding |
|---|---|---|---|
| GPT-6 Astra | 60.0 → 60.0 | 77.8 → 80.7 | 85.2 → 85.5 |
| Gemini 3.8 Flash | 50.0 → 50.0 | 70.9 → 77.5 | 78.9 → 88.6 |
Each entry reports visual input → visual input plus state. We saw that explicit state improves factual grounding, particularly for Gemini 3.8 Flash, while task-level failures persist.
Q6. Why build CookSim from scratch, and how extensible is the simulator?
We built CookSim to provide a lightweight cooking simulator that is easy to deploy and modify. BEHAVIOR requires the OmniGibson/Isaac Sim stack and separately licensed assets, while CookBench depends on the commercial Cooking Simulator game. CookSim instead uses a Python grid-based engine and Three.js renderer, without a robotics physics stack or game license. Recipes and layouts reuse existing components; new tools or cooking processes extend object/station logic and corresponding visuals. These extensions can introduce new interaction situations, such as equipment failures requiring replanning, beyond adding task instances.
Q7. Why use human-communicable actions, and can they always be executed?
We use an abstract action space so the simulated user can plan and reason at the task level, reproducing human-like physical behavior as well as conversation. With this design, our human study observes similar task-execution patterns between simulated and real users. In contrast, prior work reports that most of VLMs struggle with long-horizon action sequencing over low-level action space (EmbodiedBench; VLM-TAMP). And yes in our case the action can always be executed when satisfying explicit syntax and preconditions. which is embedded as part of the user simulation prompt: for example, chopping requires empty hands and a raw ingredient on the facing board. When the pre-condition is satisfied, the engine executes the command through a rule-based controller.
Q1. How do recovery and guidance evolve after an out-of-plan event?
Thank you for highlighting this missing analysis. We now examine error recovery and multi-turn responses to user push-back, following the HCI emphasis on supporting correction and adapting to feedback (Amershi et al., 2019).
Models that prevent many mistakes can still struggle to repair committed errors. Qwen3-Omni and Muse Glimmer achieve relatively high prevention rates, but substantially lower error fix rates than Astra and Gemini 3.8 Flash.
The same assistant faces different interaction dynamics across user personas. For every model, objections persist more frequently with the expert, low-trust persona than with the novice, high-trust persona.
Q2. Can task correctness be separated from completion time?
We agree that correctness and timeliness should be reported separately. The table below adds any-success for every assistant: correct completion anywhere within the recorded rollout, including after the deadline. In-time success additionally requires completion within the budget.