acceptodds
Under review as a conference paper at ICLR 2027

UniETP: Unifying Environments for Generalizable Embodied Task Planning

Abstract

This paper focuses on the problem of Embodied Task Planning, where an agent is required to execute a sequence of atomic actions within an interactive environment to complete a user-specified task. Though a variety of simulators and datasets have previously been built for this task, these efforts are largely isolated, with each using its own observation format, action type, and task domain. This fragmentation complicates comprehensive model evaluation and hinders the scalability of training data. As an effort towards generalizable embodied planning, we propose UniETP, a unified interface integrating four commonly used simulators (AI2-THOR, VirtualHome, Habitat, BEHAVIOR). UniETP is characterized by both standardization and diversity. On one hand, it formalizes all the simulators into a consistent observation and action space, and builds an evaluation system to support complicated task goals. On the other hand, it enhances task diversity and complexity across dimensions such as task logic, instance grounding, and instruction understanding, constructing a new dataset with varied levels of difficulty automatically. Extensive experiments on the proposed benchmark are conducted to evaluate the embodied planning capabilities of recent models and validate the utility of our framework.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Task Planning, Vision-Language Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YYrKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UniETP is a benchmark and training resource that unifies visual observations, semantic actions, and trajectory-based goal evaluation across AI2-THOR, VirtualHome, Habitat, and BEHAVIOR. Its template-based pipeline produces 536 evaluation tasks and approximately 16,000 training instances. Experiments evaluate VLMs under three difficulty settings and examine transfer to held-out simulators, task templates, and external benchmarks.

Critical strengths
  • Substantive resource integration. Shared action semantics, scene-graph trajectory evaluation, and task generation connect four household simulators. The detailed backend adaptations go beyond a thin API wrapper and provide practical infrastructure for training and evaluating the same agent across environments.

  • Demonstrated data utility. Table 6 shows improvements on the excluded AI2-THOR backend and held-out task templates. Training also improves EB-ALFRED success from 28% to 46%. These results establish useful transfer beyond the training instances.

Critical weaknesses
  1. The additional benchmark value is limited. EAI already provides cross-environment object-centric interfaces and temporal goal evaluation; ET-Plan-Bench generates executable tasks and reference plans; and EmbodiedBench evaluates visual planning through simulator execution. The more recent SpatialWorld also unifies visual observations and actions across simulators, including household environments.

    UniETP's distinct contribution is a common scene-graph evaluator and task-generation pipeline across four household backends. This integration has practical value, but the paper establishes limited additional diagnostic capability or empirical understanding. The resource improvements and reported findings together do not establish a sufficiently substantial advance over the existing benchmark landscape.

  2. The analyses overstate the evidence for simulator-diversity benefits and capability bottlenecks. Table 7 compares the full training corpus with smaller single-simulator subsets without matching supervision or optimization budgets. The additional gains therefore cannot be attributed specifically to simulator diversity, as suggested in Section 4.4.

    M1–M3 are valid difficulty settings, but jointly change scene priors, navigation access, and action granularity; their score differences cannot isolate individual bottlenecks. Figure 5 also groups episodes by realized rollout length, which failed searches and repeated actions can increase. It therefore cannot establish that intrinsically longer tasks cause lower reliability. These limitations weaken the paper's explanations of why agents fail and why mixed training helps.

Questions for authors
  1. Which concrete evaluation capability or finding best demonstrates the additional research value of the shared scene-graph and task-generation design relative to the cited frameworks?
  2. What are the training-sample and update counts for Table 7's full and single-simulator conditions? Does the advantage persist when these budgets are matched?
5 comments
AuthorsOpenReview ↗

Author Response to Reviewer YYrK (part 1)

We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate the opportunity to clarify our work and respond to the concerns point by point.

  1. Distinct Contributions and Value of UniETP (Weakness 1, Question 1)

We thank the reviewer for highlighting these relevant benchmarks, and we would like to clarify two key distinctions of UniETP. (1) Scalable, effective cross-simulator training data. EAI, EmbodiedBench, and SpatialWorld do not involve a training split or the automatic generation of training tasks. ET-Plan-Bench supports task generation, but it focuses primarily on VirtualHome (Habitat is only used for navigation tasks) and text-based environment. Our UniETP supports the automatic generation of various training tasks on different simulator backends, and builds training trajectories ready for SFT. It shows greater diversity in task types, scenes, and actions compared with prior works like ET-Plan-Bench. We also validate the effectiveness of the data and the potential to transfer across simulators and benchmarks are validated through extensive experiments (Table 6, Table 7, and the additional results in point 2 below). (2) Unified and fine-grained evaluation modes. Although some previous works also combine multiple simulators in one suite, our UniETP develops a unified environment interface and makes it finely configurable. We design a clear, multi-level definition of the observation space and the action space. By varying the information exposed to the agent and the allowed action formats, we can analyze the factors limiting agent performance, as shown by the experiments in Table 3 and point 3 below. In summary, we believe that the automated approach for constructing scalable multi-simulator training data, the demonstrated cross-simulator transferability and generalizability, and the fine-grained control over evaluation modes to analyze model bottlenecks together form the key advantages of UniETP over the cited works.

  1. Controlled Comparison of Cross-Simulator Training (Weakness 2, Question 2)

We conduct additional controlled experiments on EmbodiedBench (EB) to examine the benefits of cross-simulator training. The tables below show the results on EB-ALFRED and EB-Habitat, respectively. Here, the training settings (a), (b), and (e) reproduce Table 7 of the original paper. (b) and (e) use the same base model, the same number of tasks from the target benchmark's simulator (AI2-THOR for EB-ALFRED and Habitat for EB-Habitat), the same global batch size and learning rate, and the same number of training epochs. We further introduce setting (d), which follows the design of (e) but uses the same number of optimization steps as (b). Setting (c) applies uniform sampling to the training data in (d), ensuring the same number of tasks and a comparable number of training samples as (b). Setting (f) ablates the unified interface, as detailed below.

Name# Tasks# THOR Tasks# Samples# Training StepsBaseCSCIVASRLHAvg.
(a) Qwen3.5-4B (Raw)————28303234281828
(b) UniETP (THOR only)7k7k67k4k46344844322438
(c) UniETP (Sampled)7k3k63k4k52385434342439
(d) UniETP (Full)16k7k151k4k54465646323645
(e) UniETP (Full)16k7k151k9k48445248364846
(f) UniETP (Full, w.o. unification)16k7k151k9k42364838244238
Name# Tasks# Habitat Tasks# Samples# Training StepsBaseCSCIVASRLHAvg.
(a) Qwen3.5-4B (Raw)————66162830321231
(b) UniETP (Habitat only)3k3k35k2k58444632362240
(c) UniETP (Sampled)3k0.6k29k2k56444032403040
(d) UniETP (Full)16k3k151k2k58305246402442
(e) UniETP (Full)16k3k151k9k66445228382442
(f) UniETP (Full, w.o. unification)16k3k151k9k64284816262635
AuthorsOpenReview ↗

Author Response to Reviewer YYrK (part 2)

  1. Controlled Comparison of Cross-Simulator Training (Weakness 2, Question 2) (cont.)

We have the following conclusions from the controlled comparison.

(1) Transferability of cross-simulator data: (b) vs. (c). Under matched numbers of training tasks and optimization steps, multi-simulator training achieves comparable or slightly better performance than single-simulator training on both EB-ALFRED (39.3% vs. 38.0%) and EB-Habitat (40.3% vs. 39.7%). Notably, the proportion of same-simulator tasks is substantially reduced in (c). This suggests that training data from other simulators can provide effective transferable supervision, compensating for the lack of same-simulator data without degrading overall performance on the target benchmark.

(2) Benefits of data scaling across simulators: (b) vs. (d). When incorporating additional training tasks from other simulators while keeping the optimization steps unchanged, performance improves from 38% to 45% on EB-ALFRED and from 40% to 42% on EB-Habitat. The improvements demonstrate that incorporating additional data from other simulators can further improve performance on the target benchmark under a fixed optimization budget.

(3) Importance of the unified environment interface: (e) vs. (f). To further examine whether the improvements can be achieved by simply combining data from multiple simulators, we introduce setting (f), which uses exactly the same training tasks, samples, and optimization steps as (e), but retains simulator-specific observation and action representations instead of UniETP's unified interface. Specifically, we use AI2-THOR's original Controller actions, VirtualHome's script commands, PARTNR-style skill calls in Habitat, and OmniGibson's semantic action primitives in BEHAVIOR. We also retain the original scene-graph observations in VirtualHome and BDDL-style object references in BEHAVIOR. The results show that removing interface unification reduces performance from 46% to 38% on EB-ALFRED and from 42% to 35% on EB-Habitat. This indicates that simply aggregating heterogeneous training data is insufficient to obtain the gains in (1) and (2), highlighting the importance of UniETP's unified observation and action interface in facilitating effective cross-simulator training.

As stated in Lines 058-063 of the paper, UniETP aims to expand both the scale and diversity of training data by integrating otherwise isolated simulators. We do not claim that simulator diversity alone necessarily outperforms single-simulator training under the same data scale. Rather, the goal of UniETP is to make complementary data from heterogeneous simulators jointly usable for training. The experiments above demonstrate that, through interface unification, UniETP's cross-simulator data can provide effective supervision comparable to in-domain data, while incorporating additional data from other simulators further improves performance. These findings highlight the value of UniETP in enabling effective and scalable cross-simulator training.

AuthorsOpenReview ↗

Author Response to Reviewer YYrK (part 3)

  1. Fine-Grained Analysis of Evaluation Factors (Weakness 2)

We agree that the performance differences between M1-M3 reflect multiple factors and should not be directly attributed to individual capability bottlenecks. To provide a more fine-grained analysis, we conduct additional experiments by varying the observation space, action space, and step budget.

(1) Scene priors and teleportation. Starting from M1, we progressively remove (i) the movable object list, (ii) the static object list, and (iii) the local bounding box annotations. Accordingly, in (ii) and (iii), teleportation to objects is banned since the agent has no access to the global object list. In (iii), the object-targeted actions need to be parameterized with object bounding boxes instead of object IDs, since the agent has no access to object IDs. The ablation settings share a large step budget of 150 steps to reduce the influence of the step limit. As shown in the table below, each removal of observation leads to a clear performance drop, indicating that partial observability in unknown environments and the lack of prior knowledge about object locations remain major bottlenecks for current agents. Notably, (i) causes the largest drop for both models (despite removing only a subset of objects from the scene information), suggesting that the agents still struggle to leverage relationships among objects for environment exploration and target search.

ModelSettingAt.Co.Lo.In.Li.Avg.
Qwen3.5-4BM1674641244341
(i) − Movable-object list33813201716
(ii) − All global object lists19867119
(iii) − Local bbox annotations701032
Gemini-3.1-proM1987980467473
(i) − Movable-object list623748484646
(ii) − All global object lists623732353336
(iii) − Local bbox annotations481318151919

(2) Manipulation action granularity. We separately examine the effect of removing composite actions in the following experiment. The results show that restricting the action space indeed reduces agent performance, particularly on the Compositional split (Co.), which involves longer sequences of manipulation actions. However, compared with the results in the table above, removing composite manipulation actions has a smaller impact than restricting the observation scope (excluding the movable-object list), suggesting that environment exploration poses a greater challenge than action reasoning for current agents.

ModelSettingAt.Co.Lo.In.Li.Avg.
Qwen3.5-4BM1674641244341
- composite actions622338204033
Gemini-3.1-proM1987980467473
- composite actions835966476762

(3) Step budget. We evaluate M2 with a maximum of 150 steps and 30 failed actions, and report the success rates within 50, 100, and 150 steps. We observe that the agent is relatively insensitive to larger step budgets, particularly for the small model. This suggests that the performance is limited by misinterpretations of task instructions and action effects, as well as inefficient or even stalled environment exploration, which cannot be easily addressed by allowing more interaction steps.

ModelM2 BudgetAt.Co.Lo.In.Li.Avg.
Qwen3.5-4B5029610131312
10033610141313
15033610141313
Gemini-3.1-pro50553135413236
100603637423840
150603739424141

Finally, we would like to clarify that M1, M2, and M3 used in the paper are three representative evaluation modes rather than full configurations designed to isolate all factors. As explained in Lines 262-284, they represent typical combinations of scene knowledge, action abstraction, and grounding requirements, each corresponding to a meaningful or practical embodied planning scenario. Importantly, UniETP is not restricted to these predefined modes, and its observation space, action space, and interaction budget can be easily, separately configured. The additional experiments above leverage this flexibility to provide a more fine-grained analysis of how scene priors, action abstraction, and interaction budgets affect model performance. Such flexible and fine-grained evaluation is also a contribution of UniETP (Lines 085-087).

Reviewer_VUFMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents UniETP, a framework for unified embodied task planning across AI2-THOR, VirtualHome, Habitat, and BEHAVIOR. It standardizes observation and action interfaces, introduces a unified scene graph for logical goal evaluation, and defines three evaluation modes with different information availability and action granularity. A task generation pipeline produces 536 benchmark tasks and 16k training tasks, covering compositional planning, logical constraints, instruction understanding, and instance grounding. The paper evaluates a broad range of vision-language models and conducts supervised fine-tuning experiments, reporting improvements on UniETP, held-out environments and task types, and external benchmarks including EmbodiedBench and ALFWorld.

Critical strengths
  • The paper addresses a meaningful problem in embodied task planning. Differences in observation formats, action interfaces, and evaluation protocols make it difficult to compare and train agents across simulators. Unifying four complementary environments is a useful contribution that can reduce integration effort and support more consistent evaluation across diverse scenes and interaction capabilities.

  • The benchmark covers several important dimensions of planning difficulty. Beyond basic object manipulation, UniETP includes temporal and numerical constraints, complex language instructions, and instance-specific grounding. The combination of formal goal evaluation, scripted feasibility checks, and manual instruction review strengthens the dataset construction process. The accompanying training set also makes the framework useful for developing agents.

  • The experiments provide encouraging evidence of practical utility. The evaluation includes proprietary models, open general-purpose VLMs, and embodied foundation models, revealing substantial challenges in exploration and instance grounding. Fine-tuning improves Qwen3.5-4B’s reported average success rate from 41% to 68%, while experiments on held-out tasks and external benchmarks provide additional evidence that the generated training data are useful.

Critical weaknesses
  • Validation under more realistic embodied conditions remains limited. Although the framework covers four simulators, substantial execution difficulty is abstracted away. For example, the BEHAVIOR implementation uses symbolic state changes and kinematic actions without executing low-level motion trajectories. These abstractions are reasonable for studying high-level planning, but they limit conclusions about complete embodied systems. Moreover, M3, which requires more independent exploration and visual grounding, is evaluated only on 42 Atomic tasks using three proprietary models. Its coverage of complex planning is therefore incomplete.

  • The generalization experiments do not fully isolate the benefit of environment diversity. The training study uses only Qwen3.5-4B, and the experiment that excludes an entire simulator considers only AI2-THOR. Furthermore, the comparison between training on one simulator and the full dataset does not clearly match training data volume and optimization budget. The reported improvements are promising, but it remains unclear how much comes specifically from combining environments versus simply using more training data.

  • The evaluation modes change multiple factors simultaneously, making the performance gaps difficult to interpret. Moving from M1 to M2 reduces scene information, restricts navigation, and removes composite manipulation actions. Nevertheless, both modes use the same 50-step limit, despite the paper reporting that many M2 episodes exhaust this budget. Consequently, the performance drop reflects a mixture of exploration difficulty, finer action granularity, and budget constraints. More controlled comparisons would strengthen the analysis of which capabilities are actually limiting performance.

Questions for authors

1, Can the authors extend M3 evaluation to compositional and instance-grounding tasks, including the fine-tuned model, to assess whether the reported improvements persist with less privileged information?

2, Does multi-simulator training retain its advantage when the number of training examples and optimization steps are matched? Would the conclusions hold when each of the other simulators is excluded from training in turn?

3, How much of the M1–M2 performance gap remains when scene information and action granularity are varied independently? Can the authors report sensitivity to the step budget and the fraction of failures caused by reaching the limit?

6 comments
AuthorsOpenReview ↗

Author Response to Reviewer VUFM (part 1)

We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate the opportunity to clarify our work and respond to the concerns point by point.

  1. Validation under More Realistic Embodied Conditions (Weakness 1, Question 1)

Following the reviewer's suggestion, we extend the M3 evaluation from the Atomic split to all five task splits, including Compositional and Instance Grounding, and evaluate two open-source models and one proprietary model. The results reveal substantial challenges across all task categories, particularly for compositional planning. Even Gemini-3.1-pro achieves only 10% on Compositional tasks and 15% overall, while the two open-source models achieve only 2-3% overall. Compared with M2, the average success rate of Gemini-3.1-pro drops from 35% to 15%, despite the larger step budget (150 vs. 50). These findings highlight the difficulty of embodied planning under limited scene priors and visual grounding requirements.

We also construct training data for M3 and perform SFT on Qwen3.5-4B. Due to time and resource constraints, we generate expert trajectories for a subset of 8k training tasks in UniETP, yielding 300k step-level training samples. We fine-tune the model for one epoch (~10k optimization steps). The results show that the fine-tuned model achieves substantial improvements over the base model and clearly outperforms the larger 9B model. It demonstrates that UniETP's training data remain effective under more challenging and realistic settings. Nevertheless, the overall performance of current models still leaves substantial room for improvement, opening up opportunities for future research on enhancing agent memory, improving agent harnesses, and further scaling training data.

ModelAt.Co.Lo.In.Li.Avg.
Gemini-3.1-pro401011151715
Qwen3.5-9B1702153
Qwen3.5-4B701152
Qwen3.5-4B + SFT3347588

As for realistic execution, we acknowledge that UniETP abstracts away low-level motion control, as explicitly described in Appendix A and discussed in the Limitations section. This follows a common practice in the field of embodied task planning, where planners operate through semantic action primitives rather than directly controlling robot motions. This abstraction allows us to focus on task understanding, planning, exploration, and grounding, while reducing the influence of simulator-specific controllers and motion execution. We believe that modeling the randomness and failure of low-level control will be an important direction for future work, but it is beyond the scope of this paper.

  1. More Experiments on Training (Weakness 2, Question 2)

To further validate the generalizability of UniETP-generated training data, we extend the experiments in Table 6 along two dimensions: additional held-out simulators and additional base model.

(1) Beyond the original experiment excluding AI2-THOR, we additionally exclude VirtualHome, Habitat, and BEHAVIOR from training, respectively. The evaluation results below show similar generalization patterns to the original held-out-AI2-THOR setting: excluding any individual simulator from training still yields clear improvements on that simulator.

ModelAI2-THORVirtualHomeHabitatBEHAVIORAvg.
Qwen3.5-4B (Raw)2657275641
SFT (w.o. AI2-THOR)4272817262
SFT (w.o. VirtualHome)6065747367
SFT (w.o. Habitat)6071617266
SFT (w.o. BEHAVIOR)5569805862
SFT (Full)5772837368

(2) To examine whether the training benefits extend beyond Qwen3.5-4B, we further conduct SFT experiments on a larger model, Qwen3.5-9B. Similar to Table 6, the results below demonstrate the performance gains brought by UniETP's training set, as well as its cross-simulator and cross-task transfer capabilities.

ModelA. SeenA. UnseenV. SeenV. UnseenH. SeenH. UnseenB. SeenB. UnseenAvg.
Qwen3.5-9B (Raw)414370535776727259
SFT (w.o. AI2-THOR)474781747776837567
SFT (w.o. unseen tasks)595276637776837369
SFT (Full)645985847488827574

Together, these experiments provide a broader assessment of the effectiveness and transferability of UniETP's training tasks and supervision across different model scales and held-out simulation environments.

AuthorsOpenReview ↗

Author Response to Reviewer VUFM (part 2)

  1. Controlled Comparison of Cross-Simulator Training (Weakness 2, Question 2)

We conduct additional controlled experiments on EmbodiedBench (EB) to examine the benefits of cross-simulator training. The tables below show the results on EB-ALFRED and EB-Habitat, respectively. Here, the training settings (a), (b), and (e) reproduce Table 7 of the original paper. (b) and (e) use the same base model, the same number of tasks from the target benchmark's simulator (AI2-THOR for EB-ALFRED and Habitat for EB-Habitat), the same global batch size and learning rate, and the same number of training epochs. We further introduce setting (d), which follows the design of (e) but uses the same number of optimization steps as (b). Setting (c) applies uniform sampling to the training data in (d), ensuring the same number of tasks and a comparable number of training samples as (b). Setting (f) ablates the unified interface, as detailed below.

Name# Tasks# THOR Tasks# Samples# Training StepsBaseCSCIVASRLHAvg.
(a) Qwen3.5-4B (Raw)————28303234281828
(b) UniETP (THOR only)7k7k67k4k46344844322438
(c) UniETP (Sampled)7k3k63k4k52385434342439
(d) UniETP (Full)16k7k151k4k54465646323645
(e) UniETP (Full)16k7k151k9k48445248364846
(f) UniETP (Full, w.o. unification)16k7k151k9k42364838244238
Name# Tasks# Habitat Tasks# Samples# Training StepsBaseCSCIVASRLHAvg.
(a) Qwen3.5-4B (Raw)————66162830321231
(b) UniETP (Habitat only)3k3k35k2k58444632362240
(c) UniETP (Sampled)3k0.6k29k2k56444032403040
(d) UniETP (Full)16k3k151k2k58305246402442
(e) UniETP (Full)16k3k151k9k66445228382442
(f) UniETP (Full, w.o. unification)16k3k151k9k64284816262635

We have the following conclusions from the controlled comparison.

(1) Transferability of cross-simulator data: (b) vs. (c). Under matched numbers of training tasks and optimization steps, multi-simulator training achieves comparable or slightly better performance than single-simulator training on both EB-ALFRED (39.3% vs. 38.0%) and EB-Habitat (40.3% vs. 39.7%). Notably, the proportion of same-simulator tasks is substantially reduced in (c). This suggests that training data from other simulators can provide effective transferable supervision, compensating for the lack of same-simulator data without degrading overall performance on the target benchmark.

(2) Benefits of data scaling across simulators: (b) vs. (d). When incorporating additional training tasks from other simulators while keeping the optimization steps unchanged, performance improves from 38% to 45% on EB-ALFRED and from 40% to 42% on EB-Habitat. The improvements demonstrate that incorporating additional data from other simulators can further improve performance on the target benchmark under a fixed optimization budget.

(3) Importance of the unified environment interface: (e) vs. (f). To further examine whether the improvements can be achieved by simply combining data from multiple simulators, we introduce setting (f), which uses exactly the same training tasks, samples, and optimization steps as (e), but retains simulator-specific observation and action representations instead of UniETP's unified interface. Specifically, we use AI2-THOR's original Controller actions, VirtualHome's script commands, PARTNR-style skill calls in Habitat, and OmniGibson's semantic action primitives in BEHAVIOR. We also retain the original scene-graph observations in VirtualHome and BDDL-style object references in BEHAVIOR. The results show that removing interface unification reduces performance from 46% to 38% on EB-ALFRED and from 42% to 35% on EB-Habitat. This indicates that simply aggregating heterogeneous training data is insufficient to obtain the gains in (1) and (2), highlighting the importance of UniETP's unified observation and action interface in facilitating effective cross-simulator training.

AuthorsOpenReview ↗

Author Response to Reviewer VUFM (part 3)

  1. Controlled Comparison of Cross-Simulator Training (Weakness 2, Question 2) (cont.)

As stated in Lines 058-063 of the paper, UniETP aims to expand both the scale and diversity of training data by integrating otherwise isolated simulators. We do not claim that simulator diversity alone necessarily outperforms single-simulator training under the same data scale. Rather, the goal of UniETP is to make complementary data from heterogeneous simulators jointly usable for training. The experiments above demonstrate that, through interface unification, UniETP's cross-simulator data can provide effective supervision comparable to in-domain data, while incorporating additional data from other simulators further improves performance. These findings highlight the value of UniETP in enabling effective and scalable cross-simulator training.

  1. Fine-Grained Evaluation Analysis and Step-Budget Sensitivity (Weakness 3, Question 3)

We agree that the performance gap between M1 and M2 reflects multiple factors. To provide a more fine-grained analysis, we conduct additional experiments by varying the observation space, action space, and step budget.

(1) Scene information and teleportation. Starting from M1, we progressively remove (i) the movable object list, (ii) the static object list, and (iii) the local bounding box annotations. Accordingly, in (ii) and (iii), teleportation to objects is banned since the agent has no access to the global object list. In (iii), the object-targeted actions need to be parameterized with object bounding boxes instead of object IDs, since the agent has no access to object IDs. The ablation settings share a large step budget of 150 steps to reduce the influence of the step limit. As shown in the table below, each removal of observation leads to a clear performance drop, indicating that partial observability in unknown environments and the lack of prior knowledge about object locations remain major bottlenecks for current agents. Notably, (i) causes the largest drop for both models (despite removing only a subset of objects from the scene information), suggesting that the agents still struggle to leverage relationships among objects for environment exploration and target search.

ModelSettingAt.Co.Lo.In.Li.Avg.
Qwen3.5-4BM1674641244341
(i) − Movable-object list33813201716
(ii) − All global object lists19867119
(iii) − Local bbox annotations701032
Gemini-3.1-proM1987980467473
(i) − Movable-object list623748484646
(ii) − All global object lists623732353336
(iii) − Local bbox annotations481318151919

(2) Manipulation action granularity. We separately examine the effect of removing composite actions in the following experiment. The results show that restricting the action space indeed reduces agent performance, particularly on the Compositional split (Co.), which involves longer sequences of manipulation actions. However, compared with the results in the table above, removing composite manipulation actions has a smaller impact than restricting the observation scope (excluding the movable-object list), suggesting that environment exploration poses a greater challenge than action reasoning for current agents.

ModelSettingAt.Co.Lo.In.Li.Avg.
Qwen3.5-4BM1674641244341
- composite actions622338204033
Gemini-3.1-proM1987980467473
- composite actions835966476762
Reviewer_B7fFOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces UniETP, a framework for embodied task planning that integrates AI2-THOR, VirtualHome, Habitat, and BEHAVIOR through a shared observation and action interface. A unified scene graph supports formal evaluation of goals involving object states, spatial relations, quantification, counting, and temporal constraints. Three evaluation modes vary the available scene information, action abstraction, and grounding requirements. The authors construct 138 task templates, 536 benchmark instances, and a 16k-task training set through a pipeline combining affordance knowledge bases, scene initialization, scripted feasibility validation, and language annotation. Experiments evaluate general-purpose and embodied models, reveal substantial difficulties with instance grounding and exploration, and demonstrate improvements from supervised fine-tuning, including transfer to held-out tasks, a held-out simulator, and external benchmarks.

Critical strengths

A useful integration of heterogeneous environments with explicit evaluation assumptions. UniETP standardizes observations, action parameterization, and goal evaluation across four complementary simulators. Its explicit treatment of scene priors and action abstraction addresses an important source of ambiguity in comparing embodied planners. This infrastructure could support more systematic evaluation and broader training data collection.

Executable task specifications coupled with a validated generation pipeline. The combination of formal trajectory-level goals, instance-specific references, and expert feasibility checks provides a practical basis for constructing diverse tasks beyond simple final-state objectives. The training experiments provide encouraging evidence of utility: Qwen3.5-4B improves from 41% to 68% overall in Table 6, while the held-out-simulator experiment also shows gains without AI2-THOR training episodes.

Critical weaknesses

The experiments do not fully isolate the source of the claimed cross-simulator generalization benefits. In Table 7, training on the full dataset improves over training on the corresponding simulator subset, but the comparison does not establish whether this comes from simulator diversity, additional training data, or differences in task coverage. Appendix F.3 also describes benchmark-specific adaptations to expert generation and training-data formatting, so these results require a more qualified interpretation than direct transfer of an unchanged policy. The held-out-AI2-THOR experiment is valuable, but covers only one held-out backend. These limitations weaken the evidence for a central motivation of the work.

The evaluation modes provide limited separation of the capabilities being diagnosed. M1 and M2 simultaneously change movable-object information, teleportation access, and composite manipulation, while retaining the same 50-step budget. The large success-rate drop therefore cannot isolate exploration, primitive planning, or budget effects. Furthermore, feasibility validation is performed with a privileged expert in M1 (Appendix C.4), leaving feasibility within the M2/M3 interfaces and budgets insufficiently established. M3 evaluation is restricted to atomic tasks and three models, limiting validation of the most demanding setting.

Questions for authors

I am worried about the actual performance, due to lack of video.

Controlled comparison: Can you compare single-simulator and mixed-simulator training while matching the number of training samples, optimization budget, and task-template coverage? Does the mixed-simulator advantage persist?

Transfer protocol clarification: For Tables 7 and 13, are separate checkpoints trained using the benchmark-specific adaptations described in Appendix F.3? Which data-generation and formatting changes are made for each benchmark, and is any target-benchmark task data used?

Feasibility and difficulty clarification: What fraction of benchmark tasks can an expert complete under the actual M2/M3 interfaces and step limits? Can you separately measure the effects of removing movable-object information, restricting teleportation, and removing composite actions?

7 comments
AuthorsOpenReview ↗

Author Response to Reviewer B7fF (part 1)

We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate the opportunity to clarify our work and respond to the concerns point by point.

  1. Benefits of Cross-Simulator Training (Weakness 1, Question 2)

We conduct additional controlled experiments on EmbodiedBench (EB) to examine the benefits of cross-simulator training. The tables below show the results on EB-ALFRED and EB-Habitat, respectively. Here, the training settings (a), (b), and (e) reproduce Table 7 of the original paper. (b) and (e) use the same base model, the same number of tasks from the target benchmark's simulator (AI2-THOR for EB-ALFRED and Habitat for EB-Habitat), the same global batch size and learning rate, and the same number of training epochs. We further introduce setting (d), which follows the design of (e) but uses the same number of optimization steps as (b). Setting (c) applies uniform sampling to the training data in (d), ensuring the same number of tasks and a comparable number of training samples as (b). Setting (f) ablates the unified interface, as detailed below.

Name# Tasks# THOR Tasks# Samples# Training StepsBaseCSCIVASRLHAvg.
(a) Qwen3.5-4B (Raw)————28303234281828
(b) UniETP (THOR only)7k7k67k4k46344844322438
(c) UniETP (Sampled)7k3k63k4k52385434342439
(d) UniETP (Full)16k7k151k4k54465646323645
(e) UniETP (Full)16k7k151k9k48445248364846
(f) UniETP (Full, w.o. unification)16k7k151k9k42364838244238
Name# Tasks# Habitat Tasks# Samples# Training StepsBaseCSCIVASRLHAvg.
(a) Qwen3.5-4B (Raw)————66162830321231
(b) UniETP (Habitat only)3k3k35k2k58444632362240
(c) UniETP (Sampled)3k0.6k29k2k56444032403040
(d) UniETP (Full)16k3k151k2k58305246402442
(e) UniETP (Full)16k3k151k9k66445228382442
(f) UniETP (Full, w.o. unification)16k3k151k9k64284816262635

We have the following conclusions from the controlled comparison.

(1) Transferability of cross-simulator data: (b) vs. (c). Under matched numbers of training tasks and optimization steps, multi-simulator training achieves comparable or slightly better performance than single-simulator training on both EB-ALFRED (39.3% vs. 38.0%) and EB-Habitat (40.3% vs. 39.7%). Notably, the proportion of same-simulator tasks is substantially reduced in (c). This suggests that training data from other simulators can provide effective transferable supervision, compensating for the lack of same-simulator data without degrading overall performance on the target benchmark.

(2) Benefits of data scaling across simulators: (b) vs. (d). When incorporating additional training tasks from other simulators while keeping the optimization steps unchanged, performance improves from 38% to 45% on EB-ALFRED and from 40% to 42% on EB-Habitat. The improvements demonstrate that incorporating additional data from other simulators can further improve performance on the target benchmark under a fixed optimization budget.

(3) Importance of the unified environment interface: (e) vs. (f). To further examine whether the improvements can be achieved by simply combining data from multiple simulators, we introduce setting (f), which uses exactly the same training tasks, samples, and optimization steps as (e), but retains simulator-specific observation and action representations instead of UniETP's unified interface. Specifically, we use AI2-THOR's original Controller actions, VirtualHome's script commands, PARTNR-style skill calls in Habitat, and OmniGibson's semantic action primitives in BEHAVIOR. We also retain the original scene-graph observations in VirtualHome and BDDL-style object references in BEHAVIOR. The results show that removing interface unification reduces performance from 46% to 38% on EB-ALFRED and from 42% to 35% on EB-Habitat. This indicates that simply aggregating heterogeneous training data is insufficient to obtain the gains in (1) and (2), highlighting the importance of UniETP's unified observation and action interface in facilitating effective cross-simulator training.

AuthorsOpenReview ↗

Author Response to Reviewer B7fF (part 2)

  1. Benefits of Cross-Simulator Training (Weakness 1, Question 2) (cont.)

As stated in Lines 058-063 of the paper, UniETP aims to expand both the scale and diversity of training data by integrating otherwise isolated simulators. We do not claim that simulator diversity alone necessarily outperforms single-simulator training under the same data scale. Rather, the goal of UniETP is to make complementary data from heterogeneous simulators jointly usable for training. The experiments above demonstrate that, through interface unification, UniETP's cross-simulator data can provide effective supervision comparable to in-domain data, while incorporating additional data from other simulators further improves performance. These findings highlight the value of UniETP in enabling effective and scalable cross-simulator training.

Finally, we note that it is difficult to diversify the simulators while controlling for both the number of training samples and task-template coverage, since each simulator supports only a subset of task templates and it is natural to include more task templates when more simulators are involved. For example, the "Habitat only" split covers only 38 task templates, mainly involving pick-and-place tasks. If we were to restrict the mixed-simulator (containing 130+ templates) corpus to the same template set, a large portion of the state-changing tasks in AI2-THOR and BEHAVIOR would be wasted, which undermines our motivation for integrating these simulators. We therefore regard increased task-template diversity as a natural and intended benefit of simulator diversification, rather than a factor that should be isolated.

  1. Clarification of Cross-Benchmark Transfer Protocols (Weakness 1, Question 3)

We would like to clarify that no training tasks or expert trajectories from EB or ALFWorld (AW) are used in the experiments. All SFT data are generated exclusively from UniETP training tasks. For Table 7, the full-data model is a single checkpoint shared across EB-ALFRED and EB-Habitat, while Table 19 (Table 13 in the paper before revision) uses a separately trained checkpoint for AW.

The "benchmark-specific adaptations" align the generated training trajectories and input/output formats with the target benchmarks' interfaces. The adaptation procedures are described in the Appendix (Lines 2090-2094), and we briefly reiterate them here for clarity. For EB, we follow the M1 expert policy but decompose composite manipulation actions into atomic actions. For AW, we use the M2 expert policy with additional random exploration. For both cases, we also remove bounding-box lists from the training inputs and randomly omit unused action descriptions to simulate diverse action spaces, since EB and AW do not provide bounding-box lists and use a smaller action space than ours. These adaptations do not rely on training data from the target benchmarks.

  1. Additional Held-Out-Simulator Experiments (Weakness 1)

To further validate cross-simulator generalization, we extend the leave-one-simulator-out experiment in Table 6 by additionally excluding VirtualHome, Habitat, and BEHAVIOR from the training data, respectively.

ModelAI2-THORVirtualHomeHabitatBEHAVIORAvg.
Qwen3.5-4B (Raw)2657275641
SFT (w.o. AI2-THOR)4272817262
SFT (w.o. VirtualHome)6065747367
SFT (w.o. Habitat)6071617266
SFT (w.o. BEHAVIOR)5569805862
SFT (Full)5772837368

The additional experiments show similar generalization patterns to the original held-out-AI2-THOR setting: excluding any individual simulator from training still yields improvements on that simulator. These results provide further evidence that the transfer benefits of UniETP training are not specific to AI2-THOR, but can be commonly found on different simulation environments.

AuthorsOpenReview ↗

Author Response to Reviewer B7fF (part 3)

  1. Separation of Evaluation Factors (Weakness 2, Question 4)

To provide a more fine-grained analysis of the performance gap between evaluation modes M1 and M2, we conduct additional experiments by separately varying the observation space, action space, and step budget.

(1) Observation space and teleportation. Starting from M1, we progressively remove (i) the movable object list, (ii) the static object list, and (iii) the local bounding box annotations. Accordingly, in (ii) and (iii), teleportation to objects is banned since the agent has no access to the global object list. In (iii), the object-targeted actions need to be parameterized with object bounding boxes instead of object IDs, since the agent has no access to object IDs. The ablation settings share a large step budget of 150 steps to reduce the influence of the step limit. As shown in the table below, each removal of observation leads to a clear performance drop, indicating that partial observability in unknown environments and the lack of prior knowledge about object locations remain major bottlenecks for current agents. Notably, (i) causes the largest drop for both models (despite removing only a subset of objects from the scene information), suggesting that the agents still struggle to leverage relationships among objects for environment exploration and target search.

ModelSettingAt.Co.Lo.In.Li.Avg.
Qwen3.5-4BM1674641244341
(i) − Movable-object list33813201716
(ii) − All global object lists19867119
(iii) − Local bbox annotations701032
Gemini-3.1-proM1987980467473
(i) − Movable-object list623748484646
(ii) − All global object lists623732353336
(iii) − Local bbox annotations481318151919

(2) Manipulation action space. We separately examine the effect of removing composite actions in the following experiment. The results show that restricting the action space indeed reduces agent performance, particularly on the Compositional split (Co.), which involves longer sequences of manipulation actions. However, compared with the results in the table above, removing composite manipulation actions has a smaller impact than restricting the observation scope (excluding the movable-object list), suggesting that environment exploration poses a greater challenge than action reasoning for current agents.

ModelSettingAt.Co.Lo.In.Li.Avg.
Qwen3.5-4BM1674641244341
- composite actions622338204033
Gemini-3.1-proM1987980467473
- composite actions835966476762

(3) Step budget. We evaluate M2 with a maximum of 150 steps and 30 failed actions, and report the success rates within 50, 100, and 150 steps. We observe that the agent is relatively insensitive to larger step budgets, particularly for the small model. This suggests that the performance is limited by misinterpretations of task instructions and action effects, as well as inefficient or stalled environment exploration, which cannot be easily addressed by allowing more interaction steps.

ModelM2 BudgetAt.Co.Lo.In.Li.Avg.
Qwen3.5-4B5029610131312
10033610141313
15033610141313
Gemini-3.1-pro50553135413236
100603637423840
150603739424141

Finally, we would like to clarify that M1, M2, and M3 used in the paper are three representative evaluation modes rather than full configurations designed to isolate all factors. As explained in Lines 262-284, they represent typical combinations of scene knowledge, action abstraction, and grounding requirements, each corresponding to a meaningful or practical embodied planning scenario. Importantly, UniETP is not restricted to these predefined modes, and its observation space, action space, and interaction budget can be easily, separately configured. The additional experiments above leverage this flexibility to provide a more fine-grained analysis of how scene priors, action abstraction, and interaction budgets affect model performance. Such flexible and fine-grained evaluation is also a contribution of UniETP (Lines 085-087).

Comments on the paper

1 comment

We would like to thank all the reviewers for their thoughtful and constructive feedback. We appreciate the reviewers' recognition that UniETP provides a useful integration and standardization of heterogeneous environments (B7fF, VUFM, YYrK), enables systematic evaluation of embodied planning (B7fF) while covering several important dimensions of planning difficulty (VUFM), and provides training data whose utility is empirically demonstrated on held-out tasks, held-out simulators, and external benchmarks (B7fF, VUFM, YYrK). We have responded to each reviewer in a point-by-point manner, and here we summarize some common issues below.

(1) Controlled comparison on EmbodiedBench. In Table 7 of the paper, we demonstrate the benefits of UniETP's training data on another benchmark, EmbodiedBench (EB). The two training configurations use the same number of training tasks originating from the same simulator as the target benchmark, as well as the same number of training epochs. Following the suggestions of the reviewers, we further conduct controlled experiments that account for the total number of training tasks and optimization steps. We would like to clarify that we do not claim that simply replacing data from the target simulator with multi-simulator data leads to substantial performance improvements. Rather, the primary purpose of this experiment is to demonstrate that UniETP facilitates scaling the training data by unifying multiple simulators (Lines 058-063), enabling data from other simulators to effectively benefit performance on the target benchmark. We also emphasize that all training configurations exclusively use tasks constructed within UniETP, without using training tasks from the target benchmark.

(2) Separate analysis on evaluation factors. The evaluation modes M1, M2, and M3 are defined as three representative combinations of observation and action spaces, reflecting meaningful and practical embodied planning scenarios (Lines 262-284). Meanwhile, UniETP also supports independently ablating individual factors that affect agent performance, enabling a fine-grained analysis of performance bottlenecks. In the rebuttal, we further provide separate analysis on the effects of observation availability, composite actions, and step budgets.

(3) Additional training experiments. Building upon Table 6 of the paper, we conduct additional experiments using a different base model and ablating different simulators from the training data, obtaining results consistent with the original findings across a broader range of settings. We also extend training from mode M1 to M3, further demonstrating the effectiveness of UniETP's training data under more challenging conditions.

(4) Feasibility of the benchmark tasks. We originally validate the feasibility of the benchmark tasks under M1 using a privileged expert policy. We have now extended the expert policy to M2 and M3 and supplemented it with human teleoperation through a web UI, further validating that the benchmark tasks are solvable across all evaluation modes and that the chosen step budgets are reasonable.

(5) Comparison with other benchmarks. We emphasize UniETP's distinct value of (i) automatically generating cross-simulator training data that show the potential of transferring among simulators and benchmarks, and (ii) unifying simulator interfaces and implementing multi-level, controllable evaluation modes that enable fine-grained analysis of the agent's performance.

All revisions to the paper are highlighted in blue. The main changes include: (1) In Section 4.3, we add a new figure for the distribution of successful/failed episodes with respect to the length of expert trajectory, showing the relation between task difficulty and task length. (2) In Section 4.4 and Table 7, we add an ablation experiment demonstrating that simply increasing data scale is insufficient to improve performance, highlighting the importance of UniETP's unified environment interface. (3) In Appendix C.4, we add details on the design of expert policies for M2 and M3, along with feasibility validation through expert policies and human teleoperation. (4) In Appendix F, we include additional experimental results, including more comprehensive evaluations under M3, fine-grained analysis of multiple evaluation factors, and training and generalization experiments with different base models and held-out simulators. In addition, we have uploaded two execution videos to the supplementary materials to serve as qualitative visualizations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.