acceptodds
Withdrawn from ICLR 2027

Preemptive Instruction Following With Re-Sumption

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent advances in embodied AI have enabled agents to follow natural languageinstructions for navigation. However, real-world interactions often involve dy-namic user behaviors such as issuing instructions before the agent is ready orinterrupting and later resuming tasks, which existing models struggle to handle.To bridge this gap, we introduce Preemptive Instruction Following with Resumption (PIFR), a novel paradigm that requires agents to interpret instructions givenat arbitrary times and seamlessly resume previously interrupted goals. To sup-port research in this direction, we construct a new multi-task dataset coveringobject-goal navigation and vision-language navigation, augmented with realisticpreemptive and resumption scenarios. We conduct comprehensive evaluations of awide range of monolithic and modular methods, analyzing their capabilities in in-struction buffering, maintaining goal context under interruption, and generalizingacross task types and environments. Our benchmark reveals significant limita-tions in current approaches and provides a foundation for developing more robust,human-aligned interactive navigation systems.

Primary area:
applications to robotics, autonomy, planning
Keywords:
navigation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pQMqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces PIFR, which composes navigation tasks from SG3D, GOAT-Bench, and HM3D-OVON into three interruption structures: single, repeated, and nested. Simulator state is preserved throughout execution, and the original sequence resumes after the inserted sequence is completed. The benchmark contains 80,417 composite episodes. PIFR-Rank optimizes the goal-grounding scorer using candidate-level binary cross-entropy (BCE), a margin ranking loss between the highest-scoring positive and negative candidates, and increased training weights for resumption decisions. At inference time, it retains the original candidate-generation and navigation procedures.

Critical strengths
  1. The problem is practically relevant. Embodied agents may need to handle newly introduced, higher-priority goals and resume their original tasks from a different location. Evaluating navigation with persistent state is therefore a worthwhile research direction.
  2. The benchmark covers diverse goal types and interruption structures. The three source benchmarks span instance descriptions, category goals, language goals, and image goals. Repeated and nested interruptions provide broader coverage than a single insertion.
  3. The training objective is reasonably clear. The paper explicitly handles padding, multiple positive candidates, and cases without negative candidates. It also clarifies that the ranking loss constrains only the highest-scoring positive and negative candidates, and that training labels and reference scene information are unavailable at inference time. There is no basis for treating privileged supervision during training as evidence of test-time information leakage.
Critical weaknesses

W1. It has not been established whether the benchmark measures a distinct interruption-and-resumption capability. Does the agent or the evaluator maintain the task stack, suspended progress, and correct resumption order? If the evaluator handles these operations and issues the current goals sequentially, a nested sequence such as A→B→C→B→A may appear to the agent simply as a series of navigation goals. Given identical goal ordering, initial state, and policy-state management, merely labeling the sequence as “nested interruption” rather than “ordinary sequential execution” may not change the problem faced by the agent.

W2. The metrics do not directly measure completion of entire tasks or full sequences.

W3. Key implementation details and information needed to assess comparison fairness are missing. The paper specifies the optimizer, learning rate, number of epochs, ranking margin, and resumption weight, but does not sufficiently clarify PIFR-Rank’s backbone, initialization, trainable parameters, or the identity of the “Baseline” in Table 4. It also does not clearly report whether all methods receive equivalent PIFR fine-tuning and supervision budgets. Other details affecting reproducibility include the rule for selecting a “suitable frontier” as a positive training candidate, the proportion of decisions without positive candidates, and how internal state is managed across task switches. No accessible code or dataset materials are provided for verification.

Questions for authors
  1. Does the agent or the evaluator maintain the task stack, saved progress, and resumption order? Given the same sequence of atomic goals, how do the online inputs received by the agent differ substantively between PIFR and ordinary continuous multi-goal navigation?
  2. How are failures, timeouts, and unexecuted segments handled? What are the exact denominators for Interrupt SR and Resume SR? Could you additionally report success rates for complete inserted sequences and full episodes?
  3. The equal-weight average of Resume SR across the nine “Ours” cells in Table 3 is 43.11. Why does Table 4 report 45.04?
  4. Please clarify the backbone, the identity of the “Baseline,” and the adaptation budgets for the baselines. Please also provide a BCE+RW comparison and uncertainty estimates for the key improvements to establish the additional contribution of CR.
Reviewer_mYZcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PIFR, a benchmark for evaluating interruption and resumption in long-horizon embodied navigation. It constructs single, repeated, and nested interruption scenarios using three existing datasets: SG3D, GOAT-Bench, and HM3D-OVON. The paper also proposes Interrupt SR and Resume SR to evaluate navigation performance at different stages, along with PIFR-Rank, which combines candidate-level supervision, ranking loss, and resumption weighting. Experiments compare several navigation baselines across the three benchmark variants and interruption settings.

Critical strengths

The proposed task setup is well-motivated. Introducing interruption and resumption into long-horizon embodied navigation is reasonable and practically relevant. The single, repeated, and nested scenarios extend existing sequential navigation benchmarks to a more flexible task execution setting.

Critical weaknesses
  • The benchmark assumes task switching occurs only at atomic-task boundaries, which limits the range of interruption scenarios considered. Moreover, it is unclear how and when interruption instructions are delivered, and whether the agent itself manages task switching and resumption. Since the next unfinished instruction is reissued upon resumption, the benchmark may largely reduce to sequential navigation with externally managed task transitions. This raises concerns about whether it actually evaluates the intended interruption and resumption capabilities.
  • Interrupt SR and Resume SR in Eq. 6 are essentially conventional success rates computed over different subsets of navigation segments. I do not believe that they adequately justify why these metrics capture interruption and resumption capabilities beyond ordinary navigation success. In particular, they do not directly measure whether the agent correctly handles task switching or resumes the intended unfinished sequence. Since evaluating these capabilities is a central contribution, this limitation weakens the experimental evidence supporting the proposed benchmark.
Questions for authors
  • When exactly is the interruption instruction provided to the agent? Does it arrive during an atomic task or only after its completion? How does the agent handle this instruction, and why is task switching restricted to atomic-task boundaries?
  • Could the authors provide a concrete example of the instructions received by the agent throughout an interruption/resumption episode? In particular, which parts of task switching and resumption are handled by the agent versus the benchmark?
  • What motivates the definitions of Interrupt SR and Resume SR? How are they different from applying conventional SR to selected navigation segments, and why are existing metrics insufficient?
  • How do the proposed metrics distinguish interruption recognition, correct task switching, and resumption from ordinary navigation success? Could the authors report metrics that directly assess these capabilities, along with full-sequence completion and efficiency?
  • How exactly is the candidate set constructed, and what information is included in the navigation context?
  • Which component of PIFR-Rank specifically addresses interruption and resumption, beyond improving general navigation target grounding?
  • Are the baselines trained or fine-tuned using the same data and candidate-level supervision as PIFR-Rank? If not, could the authors provide comparisons under matched training conditions to isolate the contribution of the proposed method, particularly given its underperformance in several settings?
  • Could the authors provide independent ablations of BCE, ranking loss, and resumption weighting? Given that most improvements come from BCE supervision, what is the individual contribution of the other components?
Reviewer_j1juOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PIFR, a benchmark for embodied navigation with single, repeated, and nested interruptions of task sequences. Composite episodes are constructed from SG3D, GOAT-Bench, and HM3D-OVON, yielding 80,417 episodes. Execution preserves the simulator state, and suspended sequences continue at the next unfinished atomic goal after an inserted sequence finishes. The paper also proposes PIFR-Rank, a lightweight baseline combining candidate-level binary supervision, a margin-ranking loss, and increased training weight for resumption decisions. Experiments compare navigation baselines across interruption structures and goal splits, with aggregate navigation metrics, transition-entry success rates, stage-wise analyses, and component ablations.

Critical strengths

PIFR provides a sizeable, structured evaluation resource spanning three source benchmarks and single, repeated, and nested task insertions. This breadth could support systematic comparison of navigation behavior across different goal specifications and execution patterns.

The stage-wise analyses expose differences hidden by aggregate navigation success. For example, on PIFR-OVON Single, PIFR-Rank improves overall SR over MTU3D but reduces resumption-entry SR. Reporting such nonuniform behavior provides useful diagnostic information.

Critical weaknesses

PIFR’s distinction from continuous multi-goal navigation remains unclear. Switching occurs at atomic-task boundaries, and the next unfinished instruction is re-issued on resumption (Sec. 3.1). Under the described online interface, executing the same flattened goal sequence may present the same problem to the policy. Without clarifying which recovery decisions the agent makes rather than the evaluator, the benchmark’s additional interruption-specific capability is not established.

The evaluation does not substantiate the full-sequence completion claim: SR averages atomic segments, while Int./Res. SR measure only transition-entry segments. Failure and timeout handling are also unspecified. Moreover, Table 3’s nine full-method Res. SR values average 43.11%, whereas Table 4 reports 45.04% under its stated equal-cell averaging rule. These gaps limit interpretation of the recovery results.

Questions for authors
  1. Which component maintains suspended-task progress and selects the next instruction? What information or decision distinguishes an interruption/resumption from the corresponding flattened goal sequence at the policy interface? Please provide the execution protocol and clarify the distinction from persistent multi-goal navigation such as GOAT-Bench; a matched control would be helpful if the distinction depends on the composed task distribution.

  2. Please reconcile the 43.11% Res. SR average derived from Table 3 with the 45.04% reported for the full method in Table 4. Do these results use different checkpoints, runs, or aggregation procedures?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.