- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Sequential Planning via Anchored Robotic Keypoints (SPARK), which separates a symbolic manipulation plan from the current perceptual grounding of that plan. The paper represents a manipulation task as a symbolic behavior-tree while grounding its object references to current 3D observations only when the action is executed, and introduces a scripted displacement protocol to test whether execution can recover when objects move during a task, which provides controlled evidence for recovery without replanning. The proposed method is evaluated on LIBERO-PRO, CaP-Bench, and three physical robot platforms.
Critical strengths
-
The central insight of this research is clearly stated: some scene changes invalidate only the spatial grounding of an otherwise valid plan, not the plan itself.
-
The object-displacement experiments directly support the central claim of the paper. With identical scripted plans, verification and adaptation preserve 30/45 successes at 10 cm displacement, which shows significant performance gain compared to 1/45 when both are disabled.
-
The experiments span two simulation benchmarks (LIBERO-PRO and CaP-Bench) and three distinct physical robot platforms (UR10e, Franka FR3, and a bimanual Franka platform), thereby demonstrating the effectiveness of the method.
Critical weaknesses
-
The proposed method is primarily an integration of existing ideas: LLM-generated behavior trees, open-vocabulary segmentation, keypoint grounding, post-condition checks, and a fixed skill library. The paper’s conceptual separation is useful, but the technical novelty is limited.
-
The fixed library contains more than thirty registered skills, which is a substantially engineered prior. It remains unclear to what extent the observed performance gains benefit from this prior. Therefore, the claim that SPARK does not rely on task-specific model training should not obscure the amount of benchmark-informed skill and controller design involved.
-
The paper lacks sufficient comparison between local recovery and immediate replanning under the same failure conditions. Consequently, although the results demonstrate that recovery without replanning is feasible, it remains unclear whether this approach is preferable to immediate replanning.
-
It seems that the key displacement study changes verification and adaptation together. Therefore, the respective contributions of the two factors cannot be disentangled, which is a confounding that limits understanding of the distinct contributions of these two critical factors.
Questions for authors
-
Which registered skills are generic, and which were designed around the evaluated benchmarks? And, to what extent the observed performance gains benefit from benchmark-informed skill engineering?
-
In Table 2, post-condition checking and re-grounding are always enabled or disabled together. Could they be evaluated separately? If so, what are their respective contributions?
-
Some experiments rely on scripted plans. How were these plans obtained? If they were manually designed, the claim for recovery without replanning has not yet been validated in a fully end-to-end system. When such plans are generated by an LLM, are the verification and adaptation mechanisms equally effective? Is there preliminary evidence that LLM-generated plans perform comparably to the current scripted plans?
4 comments
We again thank Reviewer BK58 for the review, and would like to check whether we have addressed W2-4 and Q1-3 with the experiments included in the revised PDF. We measure verification and adaptation separately, with scripted and planner-generated plans, in Table 2. We compare local recovery to immediate replanning on the same displaced trials in Table 18. Using only the 5 base primitives gives 424/600 successes vs 464/600 with the full skill library in Table 11. We are willing to run any additional experiments before the discussion ends, and are happy to clarify any points.
Local recovery against replanning, and what is new (W1, W3)
Thank you again Reviewer BK58. We are following up on W1 and W3 with a comparison to the closest prior systems.
In W3, you noted it was unclear whether recovery without replanning is preferable to immediate replanning. We found no work before our submission that compares the two with the plan source, skills and perception held fixed. The LM-based systems that we identified (Inner Monologue, REFLECT, DoReMi, Recover, Code-as-Monitor, CaP-Agent0, RATS) invoke the LM after a failure, and ReKep recovers via local repair but has no replanning tier. This comparison is hence new in our paper: on the same displaced trials, trying local repair before a new plan succeeds more often (85/150 and 91/150 at 5 and 10 cm) than requesting a new plan after each failure (70/150 and 72/150), while making fewer planner calls. We are able to make this comparison because each step has an explicit typed check, so the arms differ by one switch. The same structure serves a system operator: the plan is a short tree of named steps that can be read and edited before it runs, and every step and check is logged with its outcome, so a failure can be traced to the step where it happened without reading model reasoning or generated code. This is how we found that 159 of the 767 failures in the strict Table 1 end with the wrong object held. We would be glad to know whether this changes your view of W1.
Thank you again Reviewer BK58. As the discussion closes in a few hours, we would like to ask whether our responses and the revised PDF address your concerns (W1-W4, Q1-Q3). We would be grateful if you would consider updating your assessment if so, and are glad to clarify anything before the close.
Response to BK58: Verification and adaptation measured separately, ladder vs replanning, LLM-written plans, and the skill prior
Thank you reviewer BK58 for your close review of our work and for noting the experimental breadth and method insights. We address your feedback below with new experiments and discussion that we hope will clarify our technical contributions.
In each comparison, we fix the tasks, initial states, displacements, skills, executor and perception, varying only the named factor.
Defining the mechanisms.
W1: Novelty. We acknowledge and cite additional prior work underlying SPARK's components. As detailed in “Additions” in our general response, our contributions include a controlled evaluation of when local recovery suffices without replanning and labeled execution traces, as well as real world deployment efficiencies.
W2, Q1: Skill prior.
Benchmark-informed skills contribute 6.7 percentage points to SPARK's success rate. Additionally, only 4 of the >30 registered skills were developed with reference to LIBERO scenes. We quantify their contribution through an ablation: using only 5 base primitives, SPARK achieves 70.7% success, outperforming RATs by 26.9 points without task-specific model training or practice on test tasks.
open_drawer,push_object, and knob turning viascrew/turn_knob). We acknowledge this skill engineering in our revised Limitations section.grasp_se3, and the remaining skills support physical tasks and CaP-Bench (wipe, pour, sweep, fold, etc.).open_drawer(400),push_object(200),screw(171),tilt_wrist(11),turn_knob(6).We ablate the skill library with the same planner and executor (LIBERO-PRO states 0–9, RATs protocol, 200 trials per suite). "5 base" denotes
move_to_keypoint,move_relative,grasp,release,wait. "Generic + 2 checks" addsgrasp_se3and the two verify conditions.The performance loss is concentrated in the goal suite. On pick & place tasks, the 5 base primitives achieve a success rate of 423/520 vs 415/520 for the full library.
W3: Ladder vs. immediate replanning. We compare both methods on the same displaced trials using planner-generated plans (150 trials across 10 tasks per condition; a separate matched run from the W4 table, hence 91 vs 90 for the ladder at 10 cm):
The ladder improves success by +10.0 pts [+2.0, +20.0] at 5 cm and +12.7 pts [0.0, +32.0] at 10 cm, while reducing planner calls by 0.42 [0.34, 0.51] per displaced trial.
W4, Q2: Verification and adaptation, separated. We isolate verification (scene diff) and adaptation (ladder) in a four-arm ablation, using both scripted and planner-generated plans (150 trials per condition):
At 10 cm, the ladder alone improves scripted-plan success by +56.7 pts [+32.7, +78.7], versus +3.3 pts [+0.7, +6.7] from scene diff alone. Thus, adaptation drives most of the improvement for scripted plans.
Q3: LLM-written plans. The planner-generated plans are written independently for each trial and executed end-to-end with the four ablation arms. With both mechanisms enabled, planner-generated plans achieve 123/150 successes at 0 cm and 107/150 at 10 cm, compared with 123/150 and 90/150 for scripted plans. The ladder's incremental benefit is smaller for planner-plans (+12.7 pts [+4.7, +22.0] at 10 cm), as their grasp and place checks already recover some failures. Adding scene diff to the ladder further improves planner-plan success by +11.3 pts [+2.0, +21.3], compared with +1.3 pts [-5.3, +8.0] for scripted plans. We also evaluated planner-generated plans on a physical UR10e by manually displacing objects during execution (see general response).