- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces InteractiveWorld, a human-centric closed-loop simulation environment for evaluating MLLM-based assistants that guide humans through physical tasks. To reduce the cost and limited scalability of real human participation, the framework uses an MLLM-driven Human agent that interprets natural-language instructions from an Assistant agent and translates them into executable primitive actions within a simulated environment. This enables multi-turn interactions in which the Assistant's guidance influences subsequent human actions and environment states. Based on this framework, the authors construct InteractiveBench, comprising 120 tasks across six categories with state-verifiable evaluation metrics. Experiments involving 12 frontier MLLMs evaluate task completion, action efficiency, and interaction capabilities, revealing substantial performance limitations in long-horizon assistance. The paper further investigates failure patterns, memory augmentation, and the influence of different Human agents on evaluation outcomes.
Critical strengths
-
Meaningful problem formulation and practical evaluation framework. The paper addresses an important problem of evaluating MLLM-based assistants in closed-loop human–AI interactions. By introducing an MLLM-driven Human agent and executable primitive actions, the framework reduces the cost of involving real human participants while enabling scalable, state-verifiable evaluation. This provides a potentially useful platform for studying interactive assistance beyond static egocentric benchmarks.
-
Substantial system implementation and extensive empirical evaluation. The authors construct a comprehensive simulation environment and a benchmark comprising 120 tasks across six categories. Experiments with 12 frontier MLLMs, together with analyses of task success, action efficiency, failure modes, and Human-agent variations, demonstrate the platform's functionality and potential utility for evaluating interactive MLLM capabilities.
Critical weaknesses
-
Limited methodological novelty beyond existing interactive simulation frameworks. While the paper presents a meaningful application scenario, human-centric closed-loop interaction is not fundamentally new. Prior works such as TEACh, FreeAskWorld, and HumanCLAW have explored related aspects of language-guided interaction, human-centric simulation, and embodied action execution. The primary contribution of InteractiveWorld appears to be the integration of existing components into an Assistant–Human–Environment interaction framework, rather than a substantially new simulation or interaction mechanism. Furthermore, the paper does not demonstrate a concrete downstream learning application that would further establish the practical advantages of the proposed platform.
-
Insufficient validation of the simulated Human agent and limited attribution of evaluation failures. The reliability of the proposed benchmark critically depends on whether the MLLM-driven Human agent can meaningfully approximate real human interaction patterns. However, the Human agent is governed by restrictive prompts that substantially constrain its autonomous reasoning, planning, and interpretation of instructions. Although the authors provide preliminary comparisons with human participants, these experiments focus primarily on task completion rather than behavioral-level agreement in instruction interpretation, action selection, and communication. This also raises concerns about the validity of the reported communication capability scores. Moreover, the conclusion that current MLLMs are unreliable assistants may be overstated, as task failures can arise from the Human-agent design, prompting strategies, action primitives, and environment constraints rather than the Assistant's capabilities alone. A direct-action baseline, where the same MLLM directly predicts executable actions, would help isolate the performance limitations introduced by the Assistant–Human–Action interaction pipeline.
Questions for authors
Q1. Validity of the simulated Human proxy (Clarification and validation). Were the real human participants subject to the same behavioral restrictions as the model-based Human agent? How were task repetition, execution order, and participant familiarity controlled in the human study? Beyond final task success, is there evidence that simulated and real humans exhibit similar interaction patterns when handling ambiguous instructions, clarification requests, and error recovery?
Q2. Added evaluation value and failure attribution (Additional analysis). On a representative subset of tasks, how does the proposed Human agent differ from a simpler language-to-action adapter under comparable observations and task conditions? In addition, could the authors investigate whether the identified perception and memory bottlenecks persist when providing oracle perception or history summaries derived strictly from observable evidence? These comparisons would help establish the added value of the Human-agent design and better isolate the sources of failure within the interaction pipeline.
Q3. Evaluation reliability and stability (Clarification and additional evidence). Which tasks and evaluation components rely on the frozen semantic verifier rather than direct environment-state checks, and how has the verifier's reliability been validated? Could the authors report paired uncertainty estimates for the main results, stability across repeated evaluations, and the proportion of VOID or rerun episodes? Such evidence would help establish whether the reported model comparisons and failure patterns are robust to task sampling, evaluator behavior, and execution variability.
7 comments
Response to Reviewer 7Vuy [Part 1]
We thank the reviewer for the detailed and constructive review, and for recognizing that our paper presents a meaningful application scenario. Below we answer each point.
W1: The novelty beyond existing interactive simulators is limited.
Response: Thanks for your concern. Beblow we explain our novelty.
Core difference. Prior works test whether an AI can complete a task by itself; we test whether an AI can help a human complete a task. This leads to three key differences.
(1) A full Assistant–Human–Environment loop. Prior works study parts of this setting, such as language-guided interaction, human motion simulation, or embodied execution, usually with limited scenes and assets. None of them closes the loop. To our knowledge, InteractiveWorld is the first to do so, and it supports both training and evaluating recent MLLMs.
(2) The human is a necessary role, which enables testing new abilities. Acting in the environment mainly tests perception and memory. Helping a human also requires proactivity and communication, which matter in real-world assistance but cannot be trained or evaluated in prior environments. By prompting the human agent, InteractiveWorld can also simulate different users.
(3) Broad and extensible task coverage. Limited by scene and object assets, prior works cover only one or a few task types (e.g., household chores in TEACh, navigation in FreeAskWorld). InteractiveBench covers 6 types of assistance across 50+ scenes, and every task is manually verified as a real-world task that needs an assistant. With rich assets and support for scanning real-world scenes, new tasks can be added easily.
Please refer to tables for a direct comparison with the works mentioned, and Table 4 in the main paper for a broader overview.
Table. Comparison with the works mentioned.
| TEACh | FreeAskWorld | HumanCLAW | InteractiveWorld (ours) | |
|---|---|---|---|---|
| Role of the tested AI | Usually the Follower; does the actions | A navigating robot; does the actions | Directly controls a human body; does the actions | The assistant; never touches the environment |
| What the guide knows | The Commander has the answer, object locations, and the map | The virtual human knows the route | No guide | Only the human's egocentric view; it must see, remember, and reason by itself |
| Role of the "human" | A real-human dataset | A passer-by who gives information | Used by the AI as a body | An autonomous executor who must understand and carry out the AI's words |
| Tasks | Household chores | Navigation | Finding objects | 6 types of assistance |
Table. Comparison with related works. ◐ means partial.
| Method | Primary Focus | Platform | Agent Output | Ego. View | Human Actor | Guided Human | NL Dialog | Closed Loop | Verif. Success |
|---|---|---|---|---|---|---|---|---|---|
| VirtualHome (Puig et al., 2018) | Activity programs | Unity | Action scripts | ✓ | ✓ | ✗ | ✗ | ✓ | ✓ |
| TEACh (Padmakumar et al., 2022) | Dialogue-guided tasks | AI2-THOR | Follower actions | ◐ | ✗ | ◐ | ✓ | ✓ | ✓ |
| Habitat 3.0 (Puig et al., 2023) | Human–robot collab. | Habitat | Robot actions | ✗ | ◐ | ✗ | ✗ | ✓ | ✓ |
| GRUtopia (Wang et al., 2024) | City-scale society | Isaac Sim | Robot actions | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ |
| FreeAskWorld (Peng et al., 2026) | Direction inquiry | Unity | Nav. + queries | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ |
| HumanCLAW (Li et al., 2026) | Acting via a body | Habitat | Body skills | ✓ | ✓ | ✗ | ✗ | ✓ | ✓ |
| InteractiveWorld (ours) | Human–assistant collab. | Isaac Sim | NL guidance | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Response to Reviewer 7Vuy [Part 2]
W2-1: The human agent is not well validated, and failures may come from the pipeline rather than the assistant; a direct-action baseline would help.
Response: Thanks for your concern. We answer this in three parts: (1) how our setup isolates the assistant, (2) a new direct-action baseline, as suggested by the reviewer, and (3) a new experiment with a different human personality. We discuss the behavior-level similarity between the human agent and real humans in Q1.
Part 1: How our setup isolates the assistant.
We take three measures when evaluating models on InteractiveBench.
(1) Every episode is checked by hand (Appendix D.1, L1914–1917). After each run, we review the whole interaction and confirm that the success or failure is caused by the assistant's instructions. If it is caused by the human (e.g., the human does not follow the assistant), we run the model on that task again, until the outcome is caused by the assistant. So all results in the main paper reflect the assistant's ability.
(2) The human agent is compliant (L241–243). We ask the human agent to follow the assistant as closely as possible, so it adds no extra variables. We randomly check 100 failed episodes: 96 failures are caused by the assistant and only 4 by the human (these 4 are removed and run again). So the human almost always acts on the assistant's instructions.
(3) Oracle and no-assistant checks (Table 2 caption, L272–273). On all 120 tasks, with the same Gemini-3.1-Pro human agent, an oracle assistant (i.e., one that gives the correct steps from the ground truth) reaches a 100% pass rate, while the human alone reaches 0%. So every task is solvable, and the outcome is decided by the assistant.
In addition, Figure 5 (left) in the paper uses different models as the human agent, and the final results change very little, which shows that our setup can reliably measure the assistant. Appendix D.5 compares model-played humans with real human participants: when both follow the assistant, the choice of human has little effect on most tasks.
Part 2: New experiment: direct-action baseline.
The human agent's main job is to turn the assistant's words into commands that can run in the environment, just as a person hears an instruction and then acts. So the human agent is a necessary part of the interaction. To fully remove the effect of its personality, we follow the reviewer's suggestion and add a direct-action baseline.
Setup. At each step, besides its normal message, the assistant also outputs the human's next action in natural language (e.g., <walk to the table in the living room>). The human agent then only maps this action to an executable command (e.g., <walk> table2), without its own analysis. So it cannot add errors such as misunderstanding the instruction or doing extra or missing actions. We test Claude Opus 5.5, GPT-5.5, and Gemini-3.1-Pro (a strong, a medium, and a weak model) on all 120 tasks.
Results. The result leads to two findings.
(1) Task success barely changes. For every model, the average pass rate changes by at most 1.7 points. So our original human agent already follows the assistant well, and its personality does not add a large bias.
(2) The model ranking stays the same. In both settings, Claude Opus 5.5 > GPT-5.5 > Gemini-3.1-Pro. So our results reflect the models' real interactive ability.
Table. Pass rate / mean completion rate (%) when the assistant outputs the human's next action.
| Model | Setting | Guidance | Planning | Accessibility | Decision | Prevention | Finding | Avg. |
|---|---|---|---|---|---|---|---|---|
| Claude Opus 5.5 | Human agent | 15.0/33.7 | 25.0/47.3 | 25.0/56.5 | 70.0/86.4 | 30.0/30.0 | 35.0/35.0 | 33.3/48.1 |
| Direct action | 15.0/38.1 | 20.0/38.2 | 35.0/62.2 | 75.0/89.2 | 30.0/35.8 | 35.0/35.0 | 35.0/49.8 | |
| GPT-5.5 | Human agent | 10.0/20.3 | 15.0/32.8 | 45.0/64.2 | 65.0/82.1 | 25.0/25.0 | 20.0/22.7 | 30.0/41.2 |
| Direct action | 15.0/19.6 | 5.0/20.0 | 50.0/70.2 | 65.0/80.1 | 20.0/20.8 | 20.0/25.1 | 29.2/39.3 | |
| Gemini-3.1-Pro | Human agent | 10.0/22.7 | 10.0/37.5 | 35.0/65.9 | 50.0/79.1 | 10.0/10.0 | 10.0/11.8 | 20.8/37.8 |
| Direct action | 5.0/15.2 | 10.0/38.2 | 35.0/70.2 | 50.0/80.1 | 10.0/12.6 | 10.0/11.8 | 20.0/38.0 |
These results show that the human agent does not change the task success rate, and that InteractiveBench gives a fair evaluation of the assistant's own ability.
Response to Reviewer 7Vuy
We thank the reviewer for the time and the valuable comments. We are still preparing a detailed response and will post it soon.