- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
UniETP is a benchmark and training resource that unifies visual observations, semantic actions, and trajectory-based goal evaluation across AI2-THOR, VirtualHome, Habitat, and BEHAVIOR. Its template-based pipeline produces 536 evaluation tasks and approximately 16,000 training instances. Experiments evaluate VLMs under three difficulty settings and examine transfer to held-out simulators, task templates, and external benchmarks.
Critical strengths
-
Substantive resource integration. Shared action semantics, scene-graph trajectory evaluation, and task generation connect four household simulators. The detailed backend adaptations go beyond a thin API wrapper and provide practical infrastructure for training and evaluating the same agent across environments.
-
Demonstrated data utility. Table 6 shows improvements on the excluded AI2-THOR backend and held-out task templates. Training also improves EB-ALFRED success from 28% to 46%. These results establish useful transfer beyond the training instances.
Critical weaknesses
-
The additional benchmark value is limited. EAI already provides cross-environment object-centric interfaces and temporal goal evaluation; ET-Plan-Bench generates executable tasks and reference plans; and EmbodiedBench evaluates visual planning through simulator execution. The more recent SpatialWorld also unifies visual observations and actions across simulators, including household environments.
UniETP's distinct contribution is a common scene-graph evaluator and task-generation pipeline across four household backends. This integration has practical value, but the paper establishes limited additional diagnostic capability or empirical understanding. The resource improvements and reported findings together do not establish a sufficiently substantial advance over the existing benchmark landscape.
-
The analyses overstate the evidence for simulator-diversity benefits and capability bottlenecks. Table 7 compares the full training corpus with smaller single-simulator subsets without matching supervision or optimization budgets. The additional gains therefore cannot be attributed specifically to simulator diversity, as suggested in Section 4.4.
M1–M3 are valid difficulty settings, but jointly change scene priors, navigation access, and action granularity; their score differences cannot isolate individual bottlenecks. Figure 5 also groups episodes by realized rollout length, which failed searches and repeated actions can increase. It therefore cannot establish that intrinsically longer tasks cause lower reliability. These limitations weaken the paper's explanations of why agents fail and why mixed training helps.
Questions for authors
- Which concrete evaluation capability or finding best demonstrates the additional research value of the shared scene-graph and task-generation design relative to the cited frameworks?
- What are the training-sample and update counts for Table 7's full and single-simulator conditions? Does the advantage persist when these budgets are matched?
5 comments
Author Response to Reviewer YYrK (part 2)
- Controlled Comparison of Cross-Simulator Training (Weakness 2, Question 2) (cont.)
We have the following conclusions from the controlled comparison.
(1) Transferability of cross-simulator data: (b) vs. (c). Under matched numbers of training tasks and optimization steps, multi-simulator training achieves comparable or slightly better performance than single-simulator training on both EB-ALFRED (39.3% vs. 38.0%) and EB-Habitat (40.3% vs. 39.7%). Notably, the proportion of same-simulator tasks is substantially reduced in (c). This suggests that training data from other simulators can provide effective transferable supervision, compensating for the lack of same-simulator data without degrading overall performance on the target benchmark.
(2) Benefits of data scaling across simulators: (b) vs. (d). When incorporating additional training tasks from other simulators while keeping the optimization steps unchanged, performance improves from 38% to 45% on EB-ALFRED and from 40% to 42% on EB-Habitat. The improvements demonstrate that incorporating additional data from other simulators can further improve performance on the target benchmark under a fixed optimization budget.
(3) Importance of the unified environment interface: (e) vs. (f). To further examine whether the improvements can be achieved by simply combining data from multiple simulators, we introduce setting (f), which uses exactly the same training tasks, samples, and optimization steps as (e), but retains simulator-specific observation and action representations instead of UniETP's unified interface. Specifically, we use AI2-THOR's original Controller actions, VirtualHome's script commands, PARTNR-style skill calls in Habitat, and OmniGibson's semantic action primitives in BEHAVIOR. We also retain the original scene-graph observations in VirtualHome and BDDL-style object references in BEHAVIOR. The results show that removing interface unification reduces performance from 46% to 38% on EB-ALFRED and from 42% to 35% on EB-Habitat. This indicates that simply aggregating heterogeneous training data is insufficient to obtain the gains in (1) and (2), highlighting the importance of UniETP's unified observation and action interface in facilitating effective cross-simulator training.
As stated in Lines 058-063 of the paper, UniETP aims to expand both the scale and diversity of training data by integrating otherwise isolated simulators. We do not claim that simulator diversity alone necessarily outperforms single-simulator training under the same data scale. Rather, the goal of UniETP is to make complementary data from heterogeneous simulators jointly usable for training. The experiments above demonstrate that, through interface unification, UniETP's cross-simulator data can provide effective supervision comparable to in-domain data, while incorporating additional data from other simulators further improves performance. These findings highlight the value of UniETP in enabling effective and scalable cross-simulator training.
Author Response to Reviewer YYrK (part 3)
- Fine-Grained Analysis of Evaluation Factors (Weakness 2)
We agree that the performance differences between M1-M3 reflect multiple factors and should not be directly attributed to individual capability bottlenecks. To provide a more fine-grained analysis, we conduct additional experiments by varying the observation space, action space, and step budget.
(1) Scene priors and teleportation. Starting from M1, we progressively remove (i) the movable object list, (ii) the static object list, and (iii) the local bounding box annotations. Accordingly, in (ii) and (iii), teleportation to objects is banned since the agent has no access to the global object list. In (iii), the object-targeted actions need to be parameterized with object bounding boxes instead of object IDs, since the agent has no access to object IDs. The ablation settings share a large step budget of 150 steps to reduce the influence of the step limit. As shown in the table below, each removal of observation leads to a clear performance drop, indicating that partial observability in unknown environments and the lack of prior knowledge about object locations remain major bottlenecks for current agents. Notably, (i) causes the largest drop for both models (despite removing only a subset of objects from the scene information), suggesting that the agents still struggle to leverage relationships among objects for environment exploration and target search.
| Model | Setting | At. | Co. | Lo. | In. | Li. | Avg. |
|---|---|---|---|---|---|---|---|
| Qwen3.5-4B | M1 | 67 | 46 | 41 | 24 | 43 | 41 |
| (i) − Movable-object list | 33 | 8 | 13 | 20 | 17 | 16 | |
| (ii) − All global object lists | 19 | 8 | 6 | 7 | 11 | 9 | |
| (iii) − Local bbox annotations | 7 | 0 | 1 | 0 | 3 | 2 | |
| Gemini-3.1-pro | M1 | 98 | 79 | 80 | 46 | 74 | 73 |
| (i) − Movable-object list | 62 | 37 | 48 | 48 | 46 | 46 | |
| (ii) − All global object lists | 62 | 37 | 32 | 35 | 33 | 36 | |
| (iii) − Local bbox annotations | 48 | 13 | 18 | 15 | 19 | 19 |
(2) Manipulation action granularity. We separately examine the effect of removing composite actions in the following experiment. The results show that restricting the action space indeed reduces agent performance, particularly on the Compositional split (Co.), which involves longer sequences of manipulation actions. However, compared with the results in the table above, removing composite manipulation actions has a smaller impact than restricting the observation scope (excluding the movable-object list), suggesting that environment exploration poses a greater challenge than action reasoning for current agents.
| Model | Setting | At. | Co. | Lo. | In. | Li. | Avg. |
|---|---|---|---|---|---|---|---|
| Qwen3.5-4B | M1 | 67 | 46 | 41 | 24 | 43 | 41 |
| - composite actions | 62 | 23 | 38 | 20 | 40 | 33 | |
| Gemini-3.1-pro | M1 | 98 | 79 | 80 | 46 | 74 | 73 |
| - composite actions | 83 | 59 | 66 | 47 | 67 | 62 |
(3) Step budget. We evaluate M2 with a maximum of 150 steps and 30 failed actions, and report the success rates within 50, 100, and 150 steps. We observe that the agent is relatively insensitive to larger step budgets, particularly for the small model. This suggests that the performance is limited by misinterpretations of task instructions and action effects, as well as inefficient or even stalled environment exploration, which cannot be easily addressed by allowing more interaction steps.
| Model | M2 Budget | At. | Co. | Lo. | In. | Li. | Avg. |
|---|---|---|---|---|---|---|---|
| Qwen3.5-4B | 50 | 29 | 6 | 10 | 13 | 13 | 12 |
| 100 | 33 | 6 | 10 | 14 | 13 | 13 | |
| 150 | 33 | 6 | 10 | 14 | 13 | 13 | |
| Gemini-3.1-pro | 50 | 55 | 31 | 35 | 41 | 32 | 36 |
| 100 | 60 | 36 | 37 | 42 | 38 | 40 | |
| 150 | 60 | 37 | 39 | 42 | 41 | 41 |
Finally, we would like to clarify that M1, M2, and M3 used in the paper are three representative evaluation modes rather than full configurations designed to isolate all factors. As explained in Lines 262-284, they represent typical combinations of scene knowledge, action abstraction, and grounding requirements, each corresponding to a meaningful or practical embodied planning scenario. Importantly, UniETP is not restricted to these predefined modes, and its observation space, action space, and interaction budget can be easily, separately configured. The additional experiments above leverage this flexibility to provide a more fine-grained analysis of how scene priors, action abstraction, and interaction budgets affect model performance. Such flexible and fine-grained evaluation is also a contribution of UniETP (Lines 085-087).
Author Response to Reviewer YYrK (part 1)
We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate the opportunity to clarify our work and respond to the concerns point by point.
We thank the reviewer for highlighting these relevant benchmarks, and we would like to clarify two key distinctions of UniETP. (1) Scalable, effective cross-simulator training data. EAI, EmbodiedBench, and SpatialWorld do not involve a training split or the automatic generation of training tasks. ET-Plan-Bench supports task generation, but it focuses primarily on VirtualHome (Habitat is only used for navigation tasks) and text-based environment. Our UniETP supports the automatic generation of various training tasks on different simulator backends, and builds training trajectories ready for SFT. It shows greater diversity in task types, scenes, and actions compared with prior works like ET-Plan-Bench. We also validate the effectiveness of the data and the potential to transfer across simulators and benchmarks are validated through extensive experiments (Table 6, Table 7, and the additional results in point 2 below). (2) Unified and fine-grained evaluation modes. Although some previous works also combine multiple simulators in one suite, our UniETP develops a unified environment interface and makes it finely configurable. We design a clear, multi-level definition of the observation space and the action space. By varying the information exposed to the agent and the allowed action formats, we can analyze the factors limiting agent performance, as shown by the experiments in Table 3 and point 3 below. In summary, we believe that the automated approach for constructing scalable multi-simulator training data, the demonstrated cross-simulator transferability and generalizability, and the fine-grained control over evaluation modes to analyze model bottlenecks together form the key advantages of UniETP over the cited works.
We conduct additional controlled experiments on EmbodiedBench (EB) to examine the benefits of cross-simulator training. The tables below show the results on EB-ALFRED and EB-Habitat, respectively. Here, the training settings (a), (b), and (e) reproduce Table 7 of the original paper. (b) and (e) use the same base model, the same number of tasks from the target benchmark's simulator (AI2-THOR for EB-ALFRED and Habitat for EB-Habitat), the same global batch size and learning rate, and the same number of training epochs. We further introduce setting (d), which follows the design of (e) but uses the same number of optimization steps as (b). Setting (c) applies uniform sampling to the training data in (d), ensuring the same number of tasks and a comparable number of training samples as (b). Setting (f) ablates the unified interface, as detailed below.