acceptodds
Under review as a conference paper at ICLR 2027

InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants

Abstract

People have long dreamed of an assistant that is always there to guide them and reminder what they forget, from new dishes to lost keys. Powered by multimodal large language models (MLLMs), egocentric AI assistants on wearable devices are bringing this dream within reach: they see what the user sees and give instructions for user's physical work when necessary. However, testing such assistance is challenging and usually requires real people. The expensive and rare real interaction data leaves a key question unanswered:***Can today's MLLMs truly help people in physical world?*** To answer this question, we propose **InteractiveWorld**, the first verifiable environment for closed-loop interaction between human and AI assistant. Unlike existing robot-centric simulators, InteractiveWorld lets the assistant guide a human agent who observes the scene, understands instructions, and acts independently. It thus realistically simulates how AI assistant collaborates with human: the assistant instructs, the human acts, and the environment executes actions and updates what they sees next. Building on it, we introduce **InteractiveBench**, the first dynamic benchmark for egocentric AI assistants, with 120 multi-turn tasks across six categories. In each task, the assistant guides the human in real time, and every instruction shapes the outcome, which is scored by verifiable, state-based rewards. Experiments with 12 frontier MLLMs show they are still far from reliable assistants. Even the strongest model, Claude Fable 5.1, achieves at most 20% and 25% on long-horizon Guidance and Planning tasks, respectively. Overall, our work offers a verifiable platform and a rigorous evaluation set, laying a solid foundation for research on interactive MLLM assistant that truly help people in physical world.

Primary area:
datasets and benchmarks
Keywords:
Visual Interaction Model, Daily Assistant, VLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7VuyOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces InteractiveWorld, a human-centric closed-loop simulation environment for evaluating MLLM-based assistants that guide humans through physical tasks. To reduce the cost and limited scalability of real human participation, the framework uses an MLLM-driven Human agent that interprets natural-language instructions from an Assistant agent and translates them into executable primitive actions within a simulated environment. This enables multi-turn interactions in which the Assistant's guidance influences subsequent human actions and environment states. Based on this framework, the authors construct InteractiveBench, comprising 120 tasks across six categories with state-verifiable evaluation metrics. Experiments involving 12 frontier MLLMs evaluate task completion, action efficiency, and interaction capabilities, revealing substantial performance limitations in long-horizon assistance. The paper further investigates failure patterns, memory augmentation, and the influence of different Human agents on evaluation outcomes.

Critical strengths
  1. Meaningful problem formulation and practical evaluation framework. The paper addresses an important problem of evaluating MLLM-based assistants in closed-loop human–AI interactions. By introducing an MLLM-driven Human agent and executable primitive actions, the framework reduces the cost of involving real human participants while enabling scalable, state-verifiable evaluation. This provides a potentially useful platform for studying interactive assistance beyond static egocentric benchmarks.

  2. Substantial system implementation and extensive empirical evaluation. The authors construct a comprehensive simulation environment and a benchmark comprising 120 tasks across six categories. Experiments with 12 frontier MLLMs, together with analyses of task success, action efficiency, failure modes, and Human-agent variations, demonstrate the platform's functionality and potential utility for evaluating interactive MLLM capabilities.

Critical weaknesses
  1. Limited methodological novelty beyond existing interactive simulation frameworks. While the paper presents a meaningful application scenario, human-centric closed-loop interaction is not fundamentally new. Prior works such as TEACh, FreeAskWorld, and HumanCLAW have explored related aspects of language-guided interaction, human-centric simulation, and embodied action execution. The primary contribution of InteractiveWorld appears to be the integration of existing components into an Assistant–Human–Environment interaction framework, rather than a substantially new simulation or interaction mechanism. Furthermore, the paper does not demonstrate a concrete downstream learning application that would further establish the practical advantages of the proposed platform.

  2. Insufficient validation of the simulated Human agent and limited attribution of evaluation failures. The reliability of the proposed benchmark critically depends on whether the MLLM-driven Human agent can meaningfully approximate real human interaction patterns. However, the Human agent is governed by restrictive prompts that substantially constrain its autonomous reasoning, planning, and interpretation of instructions. Although the authors provide preliminary comparisons with human participants, these experiments focus primarily on task completion rather than behavioral-level agreement in instruction interpretation, action selection, and communication. This also raises concerns about the validity of the reported communication capability scores. Moreover, the conclusion that current MLLMs are unreliable assistants may be overstated, as task failures can arise from the Human-agent design, prompting strategies, action primitives, and environment constraints rather than the Assistant's capabilities alone. A direct-action baseline, where the same MLLM directly predicts executable actions, would help isolate the performance limitations introduced by the Assistant–Human–Action interaction pipeline.

Questions for authors

Q1. Validity of the simulated Human proxy (Clarification and validation). Were the real human participants subject to the same behavioral restrictions as the model-based Human agent? How were task repetition, execution order, and participant familiarity controlled in the human study? Beyond final task success, is there evidence that simulated and real humans exhibit similar interaction patterns when handling ambiguous instructions, clarification requests, and error recovery?

Q2. Added evaluation value and failure attribution (Additional analysis). On a representative subset of tasks, how does the proposed Human agent differ from a simpler language-to-action adapter under comparable observations and task conditions? In addition, could the authors investigate whether the identified perception and memory bottlenecks persist when providing oracle perception or history summaries derived strictly from observable evidence? These comparisons would help establish the added value of the Human-agent design and better isolate the sources of failure within the interaction pipeline.

Q3. Evaluation reliability and stability (Clarification and additional evidence). Which tasks and evaluation components rely on the frozen semantic verifier rather than direct environment-state checks, and how has the verifier's reliability been validated? Could the authors report paired uncertainty estimates for the main results, stability across repeated evaluations, and the proportion of VOID or rerun episodes? Such evidence would help establish whether the reported model comparisons and failure patterns are robust to task sampling, evaluator behavior, and execution variability.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 7Vuy

We thank the reviewer for the time and the valuable comments. We are still preparing a detailed response and will post it soon.

AuthorsOpenReview ↗

Response to Reviewer 7Vuy [Part 1]

We thank the reviewer for the detailed and constructive review, and for recognizing that our paper presents a meaningful application scenario. Below we answer each point.

W1: The novelty beyond existing interactive simulators is limited.

Response: Thanks for your concern. Beblow we explain our novelty.

Core difference. Prior works test whether an AI can complete a task by itself; we test whether an AI can help a human complete a task. This leads to three key differences.

(1) A full Assistant–Human–Environment loop. Prior works study parts of this setting, such as language-guided interaction, human motion simulation, or embodied execution, usually with limited scenes and assets. None of them closes the loop. To our knowledge, InteractiveWorld is the first to do so, and it supports both training and evaluating recent MLLMs.

(2) The human is a necessary role, which enables testing new abilities. Acting in the environment mainly tests perception and memory. Helping a human also requires proactivity and communication, which matter in real-world assistance but cannot be trained or evaluated in prior environments. By prompting the human agent, InteractiveWorld can also simulate different users.

(3) Broad and extensible task coverage. Limited by scene and object assets, prior works cover only one or a few task types (e.g., household chores in TEACh, navigation in FreeAskWorld). InteractiveBench covers 6 types of assistance across 50+ scenes, and every task is manually verified as a real-world task that needs an assistant. With rich assets and support for scanning real-world scenes, new tasks can be added easily.

Please refer to tables for a direct comparison with the works mentioned, and Table 4 in the main paper for a broader overview.

Table. Comparison with the works mentioned.

TEAChFreeAskWorldHumanCLAWInteractiveWorld (ours)
Role of the tested AIUsually the Follower; does the actionsA navigating robot; does the actionsDirectly controls a human body; does the actionsThe assistant; never touches the environment
What the guide knowsThe Commander has the answer, object locations, and the mapThe virtual human knows the routeNo guideOnly the human's egocentric view; it must see, remember, and reason by itself
Role of the "human"A real-human datasetA passer-by who gives informationUsed by the AI as a bodyAn autonomous executor who must understand and carry out the AI's words
TasksHousehold choresNavigationFinding objects6 types of assistance

Table. Comparison with related works. ◐ means partial.

MethodPrimary FocusPlatformAgent OutputEgo. ViewHuman ActorGuided HumanNL DialogClosed LoopVerif. Success
VirtualHome (Puig et al., 2018)Activity programsUnityAction scripts✓✓✗✗✓✓
TEACh (Padmakumar et al., 2022)Dialogue-guided tasksAI2-THORFollower actions◐✗◐✓✓✓
Habitat 3.0 (Puig et al., 2023)Human–robot collab.HabitatRobot actions✗◐✗✗✓✓
GRUtopia (Wang et al., 2024)City-scale societyIsaac SimRobot actions✗✗✗✓✓✓
FreeAskWorld (Peng et al., 2026)Direction inquiryUnityNav. + queries✗✗✗✓✓✓
HumanCLAW (Li et al., 2026)Acting via a bodyHabitatBody skills✓✓✗✗✓✓
InteractiveWorld (ours)Human–assistant collab.Isaac SimNL guidance✓✓✓✓✓✓
AuthorsOpenReview ↗

Response to Reviewer 7Vuy [Part 2]

W2-1: The human agent is not well validated, and failures may come from the pipeline rather than the assistant; a direct-action baseline would help.

Response: Thanks for your concern. We answer this in three parts: (1) how our setup isolates the assistant, (2) a new direct-action baseline, as suggested by the reviewer, and (3) a new experiment with a different human personality. We discuss the behavior-level similarity between the human agent and real humans in Q1.

Part 1: How our setup isolates the assistant.

We take three measures when evaluating models on InteractiveBench.

(1) Every episode is checked by hand (Appendix D.1, L1914–1917). After each run, we review the whole interaction and confirm that the success or failure is caused by the assistant's instructions. If it is caused by the human (e.g., the human does not follow the assistant), we run the model on that task again, until the outcome is caused by the assistant. So all results in the main paper reflect the assistant's ability.

(2) The human agent is compliant (L241–243). We ask the human agent to follow the assistant as closely as possible, so it adds no extra variables. We randomly check 100 failed episodes: 96 failures are caused by the assistant and only 4 by the human (these 4 are removed and run again). So the human almost always acts on the assistant's instructions.

(3) Oracle and no-assistant checks (Table 2 caption, L272–273). On all 120 tasks, with the same Gemini-3.1-Pro human agent, an oracle assistant (i.e., one that gives the correct steps from the ground truth) reaches a 100% pass rate, while the human alone reaches 0%. So every task is solvable, and the outcome is decided by the assistant.

In addition, Figure 5 (left) in the paper uses different models as the human agent, and the final results change very little, which shows that our setup can reliably measure the assistant. Appendix D.5 compares model-played humans with real human participants: when both follow the assistant, the choice of human has little effect on most tasks.

Part 2: New experiment: direct-action baseline.

The human agent's main job is to turn the assistant's words into commands that can run in the environment, just as a person hears an instruction and then acts. So the human agent is a necessary part of the interaction. To fully remove the effect of its personality, we follow the reviewer's suggestion and add a direct-action baseline.

Setup. At each step, besides its normal message, the assistant also outputs the human's next action in natural language (e.g., <walk to the table in the living room>). The human agent then only maps this action to an executable command (e.g., <walk> table2), without its own analysis. So it cannot add errors such as misunderstanding the instruction or doing extra or missing actions. We test Claude Opus 5.5, GPT-5.5, and Gemini-3.1-Pro (a strong, a medium, and a weak model) on all 120 tasks.

Results. The result leads to two findings.

(1) Task success barely changes. For every model, the average pass rate changes by at most 1.7 points. So our original human agent already follows the assistant well, and its personality does not add a large bias.

(2) The model ranking stays the same. In both settings, Claude Opus 5.5 > GPT-5.5 > Gemini-3.1-Pro. So our results reflect the models' real interactive ability.

Table. Pass rate / mean completion rate (%) when the assistant outputs the human's next action.

ModelSettingGuidancePlanningAccessibilityDecisionPreventionFindingAvg.
Claude Opus 5.5Human agent15.0/33.725.0/47.325.0/56.570.0/86.430.0/30.035.0/35.033.3/48.1
Direct action15.0/38.120.0/38.235.0/62.275.0/89.230.0/35.835.0/35.035.0/49.8
GPT-5.5Human agent10.0/20.315.0/32.845.0/64.265.0/82.125.0/25.020.0/22.730.0/41.2
Direct action15.0/19.65.0/20.050.0/70.265.0/80.120.0/20.820.0/25.129.2/39.3
Gemini-3.1-ProHuman agent10.0/22.710.0/37.535.0/65.950.0/79.110.0/10.010.0/11.820.8/37.8
Direct action5.0/15.210.0/38.235.0/70.250.0/80.110.0/12.610.0/11.820.0/38.0

These results show that the human agent does not change the task success rate, and that InteractiveBench gives a fair evaluation of the assistant's own ability.

Reviewer_M8j6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes InteractiveWorld and InteractiveBench, a closed-loop simulation environment and benchmark built on Isaac Sim to evaluate how egocentric AI assistants guide humans. The motivation of this work is novel and the manuscript is well-structured. However, the current evaluation framework suffers from significant methodological flaws, as detailed in Weaknesses.

Critical strengths
  1. The task definition is novel. The paper correctly points out that existing benchmarks are largely offline, static, and reliant on answer-matching.
Critical weaknesses
  1. I know that the main goal of this paper is to test if an assistant is actually helpful. But the final success rate is generated by a mix of the assistant's guidance and the human executor's ability. The authors show that 70.4%–85.7% of failures come from perception and memory errors. However, there is no clarification on whether the mistakes come from the assistant's guidance or the human execution. It is recommended to separate the results caused by these two factors since it can prove the effectiveness of the proposed work.
Questions for authors

See weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer M8j6 [Part 1]

We thank the reviewer for the careful review, and for finding our task definition novel and our paper well-structured. We address the main concern below.

W1. The success rate mixes the assistant's guidance and the human executor's ability; please separate the two.

Response: We agree that the contributions of the human and the assistant should be looked at separately. During the rebuttal, we clarify our evaluation setting (Part 1) and add a new experiment (Part 2).

Part 1. Our evaluation setting isolates the assistant

To evaluate only the assistant and remove the human as a variable, we take three checks when evaluating models on InteractiveBench:

(1) We check that each result is caused by the assistant (Appendix D.1, L1914–1917). After every evaluation run, we review the whole interaction and confirm that the success or failure is driven by what the assistant said. If the outcome is caused by a human error (e.g., the human does not follow the assistant), we rerun the model on that task until the outcome is caused by the assistant. So all results in the main paper are driven by the assistant.

(2) The human agent is compliant (L241–243). We ask the human agent to follow the assistant as closely as possible, so that it does not add extra variables. During evaluation, we randomly checked 100 failed trajectories: 96 failures were caused by the assistant and only 4 by the human (these 4 were removed and rerun for final results). This shows that, in almost all cases, the human acts by following the assistant.

(3) Oracle and no-assistant checks (Table 2 caption, L272–273). To make sure that the results come from the assistant and that every task is solvable, we tested all 120 tasks with the same Gemini-3.1-Pro human agent. With an oracle assistant that gives the correct steps from the reference solution, the pass rate is 100%. Without any assistant, the human alone passes 0%.

In addition, Figure 5 (left) uses different models as the human agent, and the final results change little. This shows that our setting can reliably evaluate the assistant. Appendix D.5 also compares model-based human agents with real people: for most tasks, as long as the human follows the assistant, who plays the human has little effect on whether the task succeeds.

AuthorsOpenReview ↗

Response to Reviewer M8j6 [Part 2]

Part 2. New experiment: removing the human agent's own interpretation

In our loop, the main job of the human agent is to turn the assistant's natural-language instructions into commands that can run in the environment, just like a person who hears an instruction and then acts. So the human agent is a necessary part of the interaction.

To fully remove the effect of the human agent's own behavior, we add an experiment in which the assistant directly outputs the human's next action. At each step, besides its original message, the assistant also describes the action the human should take in natural language (e.g., <walk to the table in the living room.>). The human agent then only maps it to an executable command (e.g., <walk>table2), without any analysis of its own. Since the human agent only translates commands here, it cannot introduce errors such as misunderstanding the instruction or doing extra or missing actions.

We re-evaluate Claude Opus 5.5, GPT-5.5, and Gemini-3.1-Pro (a strong, a mid-level, and a weaker model) on all 120 tasks, and compare the original setting (human agent) with the new setting (direct action). Each cell shows Pass Rate / Mean Completion Rate (%).

Model (setting)GuidancePlanningAccessibilityDecisionPreventionFindingAvg.
Claude Opus 5.5 (direct action)15.0 / 38.120.0 / 38.235.0 / 62.275.0 / 89.230.0 / 35.835.0 / 35.035.0 / 49.8
Claude Opus 5.5 (human agent)15.0 / 33.725.0 / 47.325.0 / 56.570.0 / 86.430.0 / 30.035.0 / 35.033.3 / 48.1
GPT-5.5 (direct action)15.0 / 19.65.0 / 20.050.0 / 70.265.0 / 80.120.0 / 20.820.0 / 25.129.2 / 39.3
GPT-5.5 (human agent)10.0 / 20.315.0 / 32.845.0 / 64.265.0 / 82.125.0 / 25.020.0 / 22.730.0 / 41.2
Gemini-3.1-Pro (direct action)5.0 / 15.210.0 / 38.235.0 / 70.250.0 / 80.110.0 / 12.610.0 / 11.820.0 / 38.0
Gemini-3.1-Pro (human agent)10.0 / 22.710.0 / 37.535.0 / 65.950.0 / 79.110.0 / 10.010.0 / 11.820.8 / 37.8

We find two things:

(1) The final results barely change. This shows that our original human agent follows the assistant well and does not add a large bias of its own.

(2) The ranking of models does not change. With the same human, the three models keep the same order, so our results reflect the models' real interactive ability.

In summary, the human agent barely affects the task success rate. With result checking and controlled settings, InteractiveBench fairly evaluates the assistant itself. We add these results and clarifications to Appendix D.6 and C.2.4 of the revised paper. Thank you again for the suggestion.

We hope our responses address your concerns, and we are happy to answer any further questions.

Dear Reviewer M8j6,

Thank you very much for the time and effort you have devoted to reviewing our paper. During the rebuttal period, we ran a substantial number of additional experiments. We summarize them below for your convenience.

1. Analysis of the human agent

  • Task success rate with the effect of the human agent’s personality isolated (see Response to Reviewer M8j6, W2-1);
  • Impact of different human personalities on the final results (see Response to Reviewer YtJp, W1-1);
  • Impact of replacing the human agent with a language-to-action adapter (see Response to Reviewer 7Vuy, Q2-1).

Overall, the human agent is an indispensable part of InteractiveWorld, which supports human agents with diverse personalities. Under our design, the human agent follows the assistant’s guidance, and task success is driven mainly by the assistant.

2. Additional performance analyses and ablation studies

  • Stability analysis across multiple runs (see Response to Reviewer YtJp, W2-2);
  • Comparison of the effectiveness of different models on the same set of tasks (see Response to Reviewer YtJp, Q2);
  • Assistant performance under oracle (ideal) perception and memory (see Response to Reviewer 7Vuy, Q2-2).

These results show that evaluation on InteractiveBench is stable and reproducible, with consistent model rankings across settings, and that perception, memory, and communication remain the key bottlenecks during interaction.

Because these additional experiments took considerable time, our response comes later than we had hoped, and we sincerely apologize for any inconvenience. We would be very grateful if you could let us know whether our responses have addressed your concerns, or whether you have any further questions. We will do our best to address them promptly.

Best regards,

The Authors

Reviewer_YtJpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces InteractiveWorld, a simulation environment for evaluating multimodal assistants that guide humans through everyday tasks using egocentric observations and natural-language instructions. An Assistant model interacts with a model-based human executor in a closed loop, with task outcomes assessed through simulator states and execution traces. The accompanying InteractiveBench contains 120 tasks across six categories. Experiments with 12 MLLMs reveal substantial challenges in perception, memory, and sustained task execution, with the best model achieving 41.7% overall success.

Critical strengths
  1. The focus on language-guided task completion is well motivated. The closed-loop setup jointly exercises visual understanding, multi-step planning, and timely intervention, capturing how an assistant’s instructions affect subsequent observations and outcomes. This makes the benchmark useful for studying practical interactive assistance.
  2. The environment provides explicit state- and trajectory-based success criteria, supporting concrete assessment of task completion and execution constraints. The diagnostic analyses, executor ablations, and small human-participant study provide useful insights into model failures and preliminary support for the evaluation setup.
Critical weaknesses
  1. The human executor’s tightly constrained behavior only partially captures real users’ habits and initiative, while the representativeness of scene layouts and object distributions remains insufficiently established. The small human study is still conducted within the simulator, leaving the applicability of benchmark findings to real-world assistance unverified.
  2. The benchmark contains 120 tasks, with only 20 per category, limiting coverage of diverse assistance scenarios. A five-percentage-point improvement within a category corresponds to just one task. The main comparisons and memory ablations lack repeated-run variability or corresponding uncertainty estimates, making small performance differences difficult to assess.
Questions for authors
  1. The static and dynamic evaluations differ in task scope and success criteria: static evaluation often requires a single answer or next-step instruction, whereas dynamic evaluation requires full task completion. These differences complicate the interpretation of the reported performance gap.
  2. Efficiency is computed over each model’s own successful tasks, introducing potential selection bias into cross-model comparisons. The overlap between these subsets and the consistency of efficiency conclusions on shared successful tasks should be clarified using the existing evaluation logs.
5 comments
AuthorsOpenReview ↗

Response to Reviewer YtJp [Part 1]

We thank the reviewer for the constructive review, and for finding our closed-loop setting well motivated, our success criteria concrete, and our analyses useful. We address each point below.

W1-1. The tightly constrained human executor only partly captures real users' habits and initiative.

Response: Below we explain our human setting and report a new experiment.

Why the human follows the assistant in the main evaluation. We want to remove human personality as a variable. With the same human, we can fairly compare how well different assistants interact with the world and the human. If humans with different personalities were used, an assistant could get a higher pass rate by changing its wording or communication style to please the human. This would not mean the assistant is more capable, and we want to avoid it. Our response to Reviewer M8j6 also explains how we remove the human's influence in evaluation.

That said, InteractiveWorld does support simulating different humans. During the rebuttal, we add an experiment on how human personality affects the results.

New experiment: a proactive human. We change the human agent's prompt to create a proactive human. This human predicts the next step by itself and, when it is confident, does not follow the assistant. In other words, it has its own will. All other settings, including the inputs and outputs of the human agent and the assistant, stay the same. We re-evaluate Claude Opus 5.5, GPT-5.5, and Gemini-3.1-Pro (a strong, a mid-level, and a weaker model) with both types of human on all 120 tasks. Each cell shows the pass rate (%).

Model (human)GuidancePlanningAccessibilityDecisionPreventionFindingAvg.
Claude Opus 5.5 (original)15.020.035.075.030.035.035.0
Claude Opus 5.5 (proactive)25.025.035.085.030.045.040.8
GPT-5.5 (original)15.05.050.065.020.020.029.2
GPT-5.5 (proactive)20.020.055.080.020.035.038.3
Gemini-3.1-Pro (original)5.010.035.050.010.010.020.0
Gemini-3.1-Pro (proactive)15.010.040.070.010.015.026.7

We find two things:

(1) A proactive human clearly raises the success rate. The human no longer has to follow the assistant, so it sometimes steps in when the assistant is wrong and acts on its own judgment. The gain is especially large on long-horizon tasks such as Guidance and Planning. This is also why we do not use a proactive human in the main evaluation: part of the success would then come from the human, not the assistant.

(2) The ranking of assistants stays the same. For each type of human, the three models keep the same order, so stronger assistants do well with different humans. Personality changes the absolute scores, but not the relative standing of the assistants.

We add these results to Appendix D.6 of the revised paper. Thank you for the suggestion.

W1-2. Scene representativeness and real-world applicability are not verified.

Representativeness of scenes and objects. InteractiveWorld is built on the scenes and object assets of BEHAVIOR-1K, which supports 1,000 everyday activities. These activities were selected from about 2,000 candidates through a survey in which 1,461 people rated and ranked them. So the distribution of scenes and objects reflects real human daily activities, not environments set freely by researchers. In this sense, the scenes in InteractiveWorld are grounded in the real world and reflect its distribution.

Transfer to the real world. Although the tasks run in a simulator, we believe results on InteractiveBench can, to some extent, reflect an assistant's ability to help people in real life, for two reasons:

(1) Every task is human-reviewed. We make sure each task could happen in the real world and truly needs an egocentric AI assistant, so the tasks match real-world assistance scenarios.

(2) The tested abilities are general. To complete these tasks, an egocentric assistant needs memory, proactivity, communication, and other abilities. These are the same abilities it needs to truly help people in real life.

In future work, we plan to deploy assistants trained and tested in InteractiveWorld in the real world, and further study how far InteractiveWorld results carry over to real life. We add this discussion to Appendix C.5 and Appendix E of the revised paper. Thank you for the comment.

AuthorsOpenReview ↗

Response to Reviewer YtJp [Part 2]

W2-1. 120 tasks give limited coverage.

Response: Thanks for your concern. Although InteractiveBench has only 120 tasks, we believe it can reflect an assistant's ability to help, because every task has three properties:

(1) Many turns. Every task is long-horizon and multi-turn. Counted by the oracle's turns, the 120 tasks contain 1,383 turns in total, or 11.5 turns per task on average. So the final result reflects a model's average performance over more than 1,000 turns of interaction, which makes it statistically meaningful.

(2) Multiple state checkpoints. Success is not judged by a single final state. Each task has several checkpoints for completion: 480 in total, or 4 per task on average. Besides the pass rate, we also report the Mean Completion Rate, which is computed over these 480 checkpoints and is finer-grained and more informative than the pass rate.

(3) Multi-dimensional evaluation. Besides the 6 task categories, we also evaluate 9 basic abilities used during the interaction (Table 3), each with many checkpoints: object recognition (222), spatial localization (238), text recognition (455), long-term memory (554), planning (562), knowledge reasoning (289), error detection (76), timing of proactive intervention (56), and communication (696).

So, although there are only 120 tasks, InteractiveBench gives a comprehensive evaluation of interactive ability, with fine-grained results. We add these statistics to Appendix C.5 and Appendix E of the revised paper.

W2-2. The main comparisons lack repeated-run variance.

Response: Thank you for the comment. During the rebuttal, we add repeated runs with different seeds.

New experiment: repeated runs. We run Claude Opus 5.5, GPT-5.5, and Gemini-3.1-Pro on all 120 tasks 10 times each, and report the mean ± standard deviation of the pass rate (%).

ModelGuidancePlanningAccessibilityDecisionPreventionFindingAvg.
Claude Opus 5.515.0 ± 0.023.7 ± 1.831.7 ± 1.975.7 ± 2.930.0 ± 1.031.7 ± 2.834.6 ± 0.8
GPT-5.515.0 ± 2.85.0 ± 1.950.0 ± 0.065.0 ± 5.621.7 ± 1.925.0 ± 2.830.3 ± 1.2
Gemini-3.1-Pro5.0 ± 2.811.7 ± 0.833.3 ± 1.953.3 ± 4.38.3 ± 0.93.3 ± 1.919.2 ± 1.0

Most per-category standard deviations are within 3 points. The number of passed tasks in a category changes by about one task or less across runs. The standard deviation of the overall average is about 1 point, which is clearly smaller than the gaps between models. This shows that the overall ranking of models is stable over different runs.

We add these results to Appendix D.8 of the revised paper. Thank you for the comment.

AuthorsOpenReview ↗

Response to Reviewer YtJp [Part 3]

Q1. Static and dynamic evaluations differ in scope and success criteria, which makes the performance gap hard to interpret.

Response: Thank you for the comment. We agree that, because static and dynamic evaluations use different methods, the performance gap comes from many factors. Our main point is the difference between the two evaluation forms, not the size of the gap: because the two forms are very different, models behave differently on them (e.g., Figure 8 in the Appendix). InteractiveWorld supports dynamic evaluation, which is more complex and closer to how humans and assistants interact in the real world. In the revised paper, we will tone down the interpretation of the performance gap, and focus more on the difference between the two evaluation forms and on InteractiveWorld's support for dynamic evaluation.

Q2. Efficiency is computed on each model's own successful tasks, which may cause selection bias; please check it on shared successful tasks.

Response: Thank you for the suggestion. During the rebuttal, we add this analysis.

New analysis: efficiency on shared successful tasks. We first look at the tasks solved by all models. Only 4 tasks are solved by all 12 models, and these 4 are fairly easy, so they are not very informative. We therefore choose three groups of models that share more solved tasks, and compare their efficiency on these shared tasks with the same step-based metric as in the paper (a lower ratio means fewer steps). "Fewest steps" counts the shared tasks on which a model uses the fewest steps in its group (ties included).

Group A (15 shared tasks)

RankModelMean ratio ↓Median ratio ↓Fewest steps
1GLM 5.3 Flash1.43×1.00×11/15
2GPT-5.51.59×1.08×9/15
3Claude Opus 52.07×1.67×4/15
4Claude Fable 5.12.16×1.20×7/15
5GPT-5.6 Sol2.31×1.33×6/15

Group B (13 shared tasks)

RankModelMean ratio ↓Median ratio ↓Fewest steps
1Kimi K31.28×1.08×9/13
2GPT-5.51.64×1.30×7/13
3Muse Spark 1.11.67×1.85×6/13
4Claude Opus 52.04×1.57×4/13
5Claude Fable 5.12.25×1.10×7/13
6GPT-5.6 Sol2.32×1.33×7/13

Group C (13 shared tasks)

RankModelMean ratio ↓Median ratio ↓Fewest steps
1Gemini-3.1-Pro1.82×1.33×5/13
2Claude Opus 52.16×1.75×4/13
3Muse Spark 1.12.30×1.75×7/13
4Claude Fable 5.12.31×1.10×7/13
5GPT-5.6 Sol2.47×1.50×5/13

The efficiency conclusions mostly match the main paper:

(1) GLM 5.3 Flash and Kimi K3 use the fewest steps in their groups.

(2) The three top-performing models (Claude Fable 5.1, Claude Opus 5, and GPT-5.6 Sol) show no clear efficiency difference among themselves, and none of them is especially efficient.

Overall, the efficiency ranking is close to the one in the paper. We add these results to Appendix D.9 of the revised paper. Thank you for the suggestion.

We hope our responses address your concerns, and we are happy to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair and Reviewers,

We sincerely thank you for the time and effort you have devoted to reviewing our paper; your comments have substantially strengthened our work. As the AC–reviewer discussion is about to begin, we provide below a concise Rebuttal Summary for your convenience, outlining the key clarifications and additional experiments made in response to the reviewers' main concerns.

1. Reviewer YtJp – Final Score: 3 (Confidence: 3)

  • Main Concern: Concerned that, because the human agent is constrained to follow the assistant's instructions, it can represent only a small fraction of real humans; also questioned whether 120 tasks are sufficient for a comprehensive evaluation of model capabilities.
  • Response & Changes:
    • New Experiments with Diverse Human Personalities: We clarified that the human agent is set to be compliant in our experiments so that it does not dominate task outcomes, while InteractiveWorld itself supports human agents with diverse personalities. We further added experiments with a proactive human agent. Results show that while different personalities lead to different task performance, the ranking of assistant models remains nearly unchanged, confirming the stability of our evaluation.
    • Clarification on Task Complexity: We clarified that the 120 tasks still enable a comprehensive evaluation of the assistant: (1) tasks are multi-turn and long-horizon, averaging 11.5 turns per task; (2) each task contains 4 scoring criteria on average (480 in total); (3) the tasks evaluate 9 core capabilities, including object recognition (222 checkpoints), spatial localization (238), text recognition (455), long-term memory (554), planning (562), knowledge reasoning (289), and error detection (76). The evaluation is thus multi-dimensional and fine-grained, and the results are statistically meaningful.

2. Reviewer M8j6 – Final Score: 3 (Confidence: 4)

  • Main Concern: Concerned that the final evaluation results do not disentangle the respective contributions of the human agent and the assistant agent to task failures.
  • Response & Changes: We clarified that our setting ensures task outcomes are driven by the assistant rather than the human agent:
    • Outcome Attribution Check: We checked every evaluation result to confirm that it was attributable to the assistant, and re-ran the task whenever the outcome was caused by the human agent.
    • Compliant Human Agent by Design: The human agent is set to follow the assistant's guidance; our statistics show that only 4% of final outcomes are caused by the human agent.

3. Reviewer 7Vuy – Final Score: 2.5 (Confidence: 2)

  • Main Concern: Concerned about the novelty of the work and the insufficient validation of failures caused by the human agent.
  • Response & Changes:
    • Distinction from Prior Work: We clarified that prior work tests whether an AI can complete a task by itself, whereas we test whether an AI can help a human complete a task. Accordingly, InteractiveWorld (1) is the first environment to realize a closed Assistant–Human–Environment loop; (2) extends model–environment interaction to simultaneous interaction with both the environment and a human, more closely reflecting the real world; and (3) offers rich, extensible scenes and objects, supporting both the evaluation and training of state-of-the-art MLLMs.
    • New Analyses of the Human Agent: During the rebuttal, we added three analyses: (1) isolating the effect of human personality on task success rates; (2) measuring how different human-agent personalities affect final results; and (3) replacing the human agent with a language-to-action adapter to assess its impact on final results. Overall, these analyses show that the human agent is an indispensable component of InteractiveWorld, which supports human agents with diverse personalities. Under our design, the human agent follows the assistant's guidance, so task success is driven primarily by the assistant.

Thank you again for your efforts. Your feedback has made our paper more complete and rigorous.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.