- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces PIFR, which composes navigation tasks from SG3D, GOAT-Bench, and HM3D-OVON into three interruption structures: single, repeated, and nested. Simulator state is preserved throughout execution, and the original sequence resumes after the inserted sequence is completed. The benchmark contains 80,417 composite episodes. PIFR-Rank optimizes the goal-grounding scorer using candidate-level binary cross-entropy (BCE), a margin ranking loss between the highest-scoring positive and negative candidates, and increased training weights for resumption decisions. At inference time, it retains the original candidate-generation and navigation procedures.
Critical strengths
- The problem is practically relevant. Embodied agents may need to handle newly introduced, higher-priority goals and resume their original tasks from a different location. Evaluating navigation with persistent state is therefore a worthwhile research direction.
- The benchmark covers diverse goal types and interruption structures. The three source benchmarks span instance descriptions, category goals, language goals, and image goals. Repeated and nested interruptions provide broader coverage than a single insertion.
- The training objective is reasonably clear. The paper explicitly handles padding, multiple positive candidates, and cases without negative candidates. It also clarifies that the ranking loss constrains only the highest-scoring positive and negative candidates, and that training labels and reference scene information are unavailable at inference time. There is no basis for treating privileged supervision during training as evidence of test-time information leakage.
Critical weaknesses
W1. It has not been established whether the benchmark measures a distinct interruption-and-resumption capability. Does the agent or the evaluator maintain the task stack, suspended progress, and correct resumption order? If the evaluator handles these operations and issues the current goals sequentially, a nested sequence such as A→B→C→B→A may appear to the agent simply as a series of navigation goals. Given identical goal ordering, initial state, and policy-state management, merely labeling the sequence as “nested interruption” rather than “ordinary sequential execution” may not change the problem faced by the agent.
W2. The metrics do not directly measure completion of entire tasks or full sequences.
W3. Key implementation details and information needed to assess comparison fairness are missing. The paper specifies the optimizer, learning rate, number of epochs, ranking margin, and resumption weight, but does not sufficiently clarify PIFR-Rank’s backbone, initialization, trainable parameters, or the identity of the “Baseline” in Table 4. It also does not clearly report whether all methods receive equivalent PIFR fine-tuning and supervision budgets. Other details affecting reproducibility include the rule for selecting a “suitable frontier” as a positive training candidate, the proportion of decisions without positive candidates, and how internal state is managed across task switches. No accessible code or dataset materials are provided for verification.
Questions for authors
- Does the agent or the evaluator maintain the task stack, saved progress, and resumption order? Given the same sequence of atomic goals, how do the online inputs received by the agent differ substantively between PIFR and ordinary continuous multi-goal navigation?
- How are failures, timeouts, and unexecuted segments handled? What are the exact denominators for Interrupt SR and Resume SR? Could you additionally report success rates for complete inserted sequences and full episodes?
- The equal-weight average of Resume SR across the nine “Ours” cells in Table 3 is 43.11. Why does Table 4 report 45.04?
- Please clarify the backbone, the identity of the “Baseline,” and the adaptation budgets for the baselines. Please also provide a BCE+RW comparison and uncertainty estimates for the key improvements to establish the additional contribution of CR.