- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses how a robot can accommodate new requests during ongoing execution while preserving existing commitments, reusing planned actions, and respecting skill interruption boundaries. The approach is clearly presented, and the real-robot deployment represents a substantial implementation effort. However, the current experiments mainly support the conclusion that explicit constraints are more reliable than weakly constrained generative replanning within the authors’ defined setting. Evidence remains insufficient to establish broader continual instruction reconciliation capabilities, advantages over strong planning baselines, or safety during real-world execution.
Critical strengths
- Practically relevant problem. The paper addresses how robots can accommodate new instructions during ongoing execution while preserving existing commitments, reusing planned actions, and respecting skill interruption boundaries.
- Clear and interpretable design. Separating rule-constrained candidate generation from LLM-based semantic arbitration makes the framework easy to understand. The structural constraints explicitly preserve existing subtasks and their relative order.
Critical weaknesses
- The evaluation criteria closely align with the proposed scheduling policy. For example, Appendix A.4 defines the target insertion position as the frontmost admissible position within the earliest matched visit, closely matching CIRRA’s strategy. The dataset collection protocol and evaluation criteria are insufficiently specified, and the method has not been validated on a broader range of datasets, limiting meaningful comparison with other algorithms.
- Output expressiveness and baseline adaptations are difficult to interpret. Appendix B acknowledges that flat-output systems cannot express certain same-visit task structures and are therefore systematically classified as queueing in the corresponding fusion cases. Several baselines also undergo substantial adaptations, including reduced refinement rounds. These choices make it difficult to attribute the performance gap specifically to CIRRA’s reconciliation mechanism. The paper lacks a strong baseline with the same plan representation, execution constraints, and computational budget.
- The physical evaluation requires clearer attribution and reporting. Grasp-failure trials are excluded without reporting exclusion counts or clarifying whether all methods follow the same exclusion protocol. Moreover, CIRRA explicitly enforces interruption boundaries, whereas the baselines do not account for skill interruptibility, making the IC comparison insufficient to isolate planning improvements. The definitions and denominators of the reported metrics also require clarification, particularly because TA and EA are initially defined only for episodes whose ground-truth label is fusion, yet are also reported for preemption and queueing combinations.
Questions for authors
- Can the authors evaluate plans independently of their output representation and distinguish task feasibility from agreement with a single canonical insertion position? How much of the performance gap remains under this evaluation?
- How does CIRRA compare with a constrained replanning baseline that uses the same compilation pipeline, equally expressive output format, commitment-preservation constraints, interruption-boundary wrapper, and computational budget?
- How does the algorithm preserve the ordering of incoming subtasks when their required location order conflicts with the existing visit order? What is the fallback when no valid insertion position exists, and under what conditions is reusing existing actions to avoid duplicate execution permitted?
- Can the authors provide per-combination success counts, denominators, exclusion counts, and aggregation rules for the robot experiments?
Overall, my concerns primarily focus on the design of the experimental metrics and the evaluation methodology. If the authors adequately address these issues, I would consider raising my score.