acceptodds
Withdrawn from ICLR 2027

RoboState: State-Aware Planning for Continual Embodied Instruction Followinging Open Dynamic Environment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Continual embodied instruction following (CEIF) is prevalent in long-term human-agent cohabitation tasks such as home service, elderly care, and autonomous inspection. CEIF is particularly challenging in open dynamic environments, which are affected by human activity between consecutive observations, invalidating the agent’s previous recognition of environment state. Existing methods make decisions primarily based on surface-level environmental information, which causes planning to depend on stale or misaligned information and involve unavailable objects, ultimately leading to failures. We propose RoboState, a state-aware framework for CEIF in open dynamic environments, which operates through a three-stage pipeline of (1) parsing an instruction into goal states, (2) aligning and tracking the current states of the relevant objects, and (3) constructing an executable plan from the current states to the goal states. We evaluate RoboState on a new benchmark containing 120 CEIF instructions as well as on the existing ExRAP benchmark. The results show that RoboState remains robust across different dynamic difficulty levels, instruction scales, instruction types, and LLM backbones, and achieves higher task success rates compared with the state-of-the-art methods.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Continual embodied instruction following

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_u4sHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the task of continual embodied instruction following for long-term human-agent cohabitation. The task can have downstream applications such as home service, elderly care and autonomous inspection. At the core of the task, a key challenge is that in the open environment, the environment state can change dynamically by humans, so the agents have to always update their recognition of the environment. Therefore, the authors propose RoboState, a framework that parses an instruction into goal states, aligns and tracks the current states of the relevant objects, and constructs an executable plan from the current states to the goal states. The experiment results at different settings show that the proposed method outperform previous state-of-the-art methods.

Critical strengths

The problem setting the paper studies is important. A key challenge in long-term human-agent cohabitation is that the agent has to constantly update its understanding and recognition of the open environment, as the states can change because of human activities. It is good to have the paper focus on this setting.

Critical weaknesses
  1. It would be better if the authors can include more qualitative examples and analysis in the paper to show how their method helps improve the task of continual embodied instruction following for long-term human-agent cohabitation. In this way, the audience can learn more insights from the study and get more intuitive insights on actually how the proposed framework works.

  2. The presentation of the paper needs to be improved. In Section 4 Experiment, there are multiple issues: (a) ExRAP benchmark (?); (b) Appendix ?? provides; (c) Instruction scale. columns 1-3 of (should be Columns). These indicate that the paper had not been carefully checked and proofread at the time of submission. I do not think the current version with several obvious presentation problems is ready for ICLR. In addition, the authors have not highlighted the best result in the experiment tables. It would be much better if the authors can do so and then it would easier for the audience to understand how much improvement the proposed method actually brings.

Questions for authors

As in Weakness 1, could you please give more examples on how actually the proposed framework help improve the long-term continual embodied instruction following task?

I would suggest the authors to address my concerns as in the weakness section during the rebuttal.

Reviewer_DDWGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes RoboState for continual embodied instruction following in dynamic environments. The problem is relevant, but the submission appears insufficiently finalized, with unresolved presentation issues and important gaps in the description of its core method.

Critical strengths

none

Critical weaknesses

1.The manuscript contains obvious unfinished editing, including “ExRAP benchmark (?)”, “Appendix ??”, and “textbfHalluc.” on page 7.

2.Table 5 refers to “Class G-(2)” without defining this label in the manuscript or appendix, making it unclear which evaluation subset the reported results correspond to.

Questions for authors

The submission requires substantial revision to complete the technical description and resolve these inconsistencies. I recommend rejection in its current form.

Reviewer_UL9ZOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses open dynamic environment for robot action planning (or replanning by updating memory). In particular, the method consists of three components; (1) parsing an instruction to goal statement, (2) aligning and tracking the states of the relevant objects, (3) making an executable plan. The paper also propose a new benchmark and use ExRAP benchmark. On the benchmarks, the propose method achieves state of the art success rate.

Critical strengths
  • It is an important issue to reflect changing environment in generating action plans
  • Using the notion of 'state' seems reasonable
  • Proposing a new benchmark and achieve good success rate (with large gain against previous arts) on both new and existing benchmarks
Critical weaknesses
  • Novelty is limited: there are a number of efforts to revise actions for changing environments, for example
    • Ahmed et al., ProAct-VLM: Pre-Failure Vision-Language Task Replanning with Continuous Perception Feedback, arXiv 2609
    • Yu et al., Scene Graph-Guided Proactive Replanning for Failure-Resilient Embodied Agents, RSS 2025
    • Kim et al., Pre-emptive Action Revision by Environmental Feedback for Embodied Instruction Following Agents, CoRL 2024
    • Especially, the proposed method is quite similar to ExRAP.
  • Details of the proposed method are missing (see my comments in 'questions for authors')
  • Details of the proposed benchmark is missing. (how many episodes, what are the types of environmental changes, what are the instruction complexities?)
Questions for authors
  • Articulate the difference of the proposed method against ExRAP

  • (detail) When does the state is updated? Every time the robot sees the object again?

  • (detail) How to obtain the "current state"? From simulator? or by perception (vision)?

  • Minor comments

    • Missing reference of ExRAP benchmark in Section 4.
    • No line number (out of given template)
    • The fonts are out of given template.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.