acceptodds
Under review as a conference paper at ICLR 2027

Sequential Planning via Anchored Robotic Keypoints

Abstract

Changes in a robot’s observations do not automatically imply changes in the task it is solving, but language-conditioned manipulation systems often fail to separate the two. If an object moves, for example, the locations at which actions should be executed may change while the underlying manipulation plan remains valid. We introduce Sequential Planning via Anchored Robotic Keypoints (SPARK), which separates a task's symbolic plan from its execution-time perceptual grounding, and we experimentally isolate whether a fixed symbolic plan can survive substantial changes to the scene through execution-time re-grounding. SPARK represents the task as a symbolic behavior tree whose object references are grounded to current 3D observations during execution, allowing the robot to update where actions are performed while preserving the plan. Runtime verification then enables local recovery and renewed grounding, only resorting to a new plan if necessary. To test the separation between plan and perceptual-grounding, we perform a set of manipulation experiments with a fixed plan while moving objects during execution. Under a 10 cm displacement, verification and adaptation maintain 66.7% task success, compared with 2.2% when disabled, demonstrating the importance of the underlying principle. Without task-specific model training, SPARK also achieves 72.6% mean success across six LIBERO-PRO settings and 68% across eleven physical task–embodiment settings on three robot platforms. Together, these results show that robustness does not require re-planning the task and for an important class of failures, it is sufficient to preserve the plan and update the system's perception of a changing world. Code and videos can be found at https://longjohnsilvers22.github.io/spark_page/

Primary area:
applications to robotics, autonomy, planning
Keywords:
robotics foundations, general purpose-knowledge system, end-to-end pipeline, training-free manipulation, Code-as-Policies

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BK58OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Sequential Planning via Anchored Robotic Keypoints (SPARK), which separates a symbolic manipulation plan from the current perceptual grounding of that plan. The paper represents a manipulation task as a symbolic behavior-tree while grounding its object references to current 3D observations only when the action is executed, and introduces a scripted displacement protocol to test whether execution can recover when objects move during a task, which provides controlled evidence for recovery without replanning. The proposed method is evaluated on LIBERO-PRO, CaP-Bench, and three physical robot platforms.

Critical strengths
  1. The central insight of this research is clearly stated: some scene changes invalidate only the spatial grounding of an otherwise valid plan, not the plan itself.

  2. The object-displacement experiments directly support the central claim of the paper. With identical scripted plans, verification and adaptation preserve 30/45 successes at 10 cm displacement, which shows significant performance gain compared to 1/45 when both are disabled.

  3. The experiments span two simulation benchmarks (LIBERO-PRO and CaP-Bench) and three distinct physical robot platforms (UR10e, Franka FR3, and a bimanual Franka platform), thereby demonstrating the effectiveness of the method.

Critical weaknesses
  1. The proposed method is primarily an integration of existing ideas: LLM-generated behavior trees, open-vocabulary segmentation, keypoint grounding, post-condition checks, and a fixed skill library. The paper’s conceptual separation is useful, but the technical novelty is limited.

  2. The fixed library contains more than thirty registered skills, which is a substantially engineered prior. It remains unclear to what extent the observed performance gains benefit from this prior. Therefore, the claim that SPARK does not rely on task-specific model training should not obscure the amount of benchmark-informed skill and controller design involved.

  3. The paper lacks sufficient comparison between local recovery and immediate replanning under the same failure conditions. Consequently, although the results demonstrate that recovery without replanning is feasible, it remains unclear whether this approach is preferable to immediate replanning.

  4. It seems that the key displacement study changes verification and adaptation together. Therefore, the respective contributions of the two factors cannot be disentangled, which is a confounding that limits understanding of the distinct contributions of these two critical factors.

Questions for authors
  1. Which registered skills are generic, and which were designed around the evaluated benchmarks? And, to what extent the observed performance gains benefit from benchmark-informed skill engineering?

  2. In Table 2, post-condition checking and re-grounding are always enabled or disabled together. Could they be evaluated separately? If so, what are their respective contributions?

  3. Some experiments rely on scripted plans. How were these plans obtained? If they were manually designed, the claim for recovery without replanning has not yet been validated in a fully end-to-end system. When such plans are generated by an LLM, are the verification and adaptation mechanisms equally effective? Is there preliminary evidence that LLM-generated plans perform comparably to the current scripted plans?

4 comments
AuthorsOpenReview ↗

Response to BK58: Verification and adaptation measured separately, ladder vs replanning, LLM-written plans, and the skill prior

Thank you reviewer BK58 for your close review of our work and for noting the experimental breadth and method insights. We address your feedback below with new experiments and discussion that we hope will clarify our technical contributions.

In each comparison, we fix the tasks, initial states, displacements, skills, executor and perception, varying only the named factor.

Defining the mechanisms.

  • Scene diff (verification): Compares the latest detection to the original binding (the detection a plan label points to) before and during each keypoint move. It retargets if the object moves <10 cm and ends the attempt if it is >10 cm.
  • Ladder (adaptation): After a failed attempt, it rebinds labels to new detections, then retries the action and replans if necessary (up to 3 attempts).
  • Replan: After failure, it retracts, redetects the scene, and requests a new plan (up to 2 times).

W1: Novelty. We acknowledge and cite additional prior work underlying SPARK's components. As detailed in “Additions” in our general response, our contributions include a controlled evaluation of when local recovery suffices without replanning and labeled execution traces, as well as real world deployment efficiencies.

W2, Q1: Skill prior.

Benchmark-informed skills contribute 6.7 percentage points to SPARK's success rate. Additionally, only 4 of the >30 registered skills were developed with reference to LIBERO scenes. We quantify their contribution through an ablation: using only 5 base primitives, SPARK achieves 70.7% success, outperforming RATs by 26.9 points without task-specific model training or practice on test tasks.

  • LIBERO-informed: 4 skills (open_drawer, push_object, and knob turning via screw / turn_knob). We acknowledge this skill engineering in our revised Limitations section.
  • Benchmark-independent: The 5 base primitives, grasp_se3, and the remaining skills support physical tasks and CaP-Bench (wipe, pour, sweep, fold, etc.).
  • Usage: 86.9% of initial plans in the updated Table 1 use no specialized skill. The remaining 788/6,000 occur exclusively in the goal suite: open_drawer (400), push_object (200), screw (171), tilt_wrist (11), turn_knob (6).

We ablate the skill library with the same planner and executor (LIBERO-PRO states 0–9, RATs protocol, 200 trials per suite). "5 base" denotes move_to_keypoint, move_relative, grasp, release, wait. "Generic + 2 checks" adds grasp_se3 and the two verify conditions.

LibraryObjectGoalSpatialAll [vs full, 95% CI]
Full182127155464/600
Generic + 2 checks18180160421 [-7.2: -13.8, -1.5]
5 base18285157424 [-6.7: -13.5, -1.0]
5 base, mask-width grasp18581161427 [-6.2: -13.2, -0.2]

The performance loss is concentrated in the goal suite. On pick & place tasks, the 5 base primitives achieve a success rate of 423/520 vs 415/520 for the full library.

W3: Ladder vs. immediate replanning. We compare both methods on the same displaced trials using planner-generated plans (150 trials across 10 tasks per condition; a separate matched run from the W4 table, hence 91 vs 90 for the ladder at 10 cm):

Arm0 cm5 cm10 cmMean planner calls per displaced trial
Replan12070721.56
Ladder12085911.14

The ladder improves success by +10.0 pts [+2.0, +20.0] at 5 cm and +12.7 pts [0.0, +32.0] at 10 cm, while reducing planner calls by 0.42 [0.34, 0.51] per displaced trial.

W4, Q2: Verification and adaptation, separated. We isolate verification (scene diff) and adaptation (ladder) in a four-arm ablation, using both scripted and planner-generated plans (150 trials per condition):

ArmScripted, 0 cmScripted, 10 cmPlanner, 0 cmPlanner, 10 cm
Neither117312071
Scene diff105812373
Ladder1208812090
Both12390123107

At 10 cm, the ladder alone improves scripted-plan success by +56.7 pts [+32.7, +78.7], versus +3.3 pts [+0.7, +6.7] from scene diff alone. Thus, adaptation drives most of the improvement for scripted plans.

Q3: LLM-written plans. The planner-generated plans are written independently for each trial and executed end-to-end with the four ablation arms. With both mechanisms enabled, planner-generated plans achieve 123/150 successes at 0 cm and 107/150 at 10 cm, compared with 123/150 and 90/150 for scripted plans. The ladder's incremental benefit is smaller for planner-plans (+12.7 pts [+4.7, +22.0] at 10 cm), as their grasp and place checks already recover some failures. Adding scene diff to the ladder further improves planner-plan success by +11.3 pts [+2.0, +21.3], compared with +1.3 pts [-5.3, +8.0] for scripted plans. We also evaluated planner-generated plans on a physical UR10e by manually displacing objects during execution (see general response).

We again thank Reviewer BK58 for the review, and would like to check whether we have addressed W2-4 and Q1-3 with the experiments included in the revised PDF. We measure verification and adaptation separately, with scripted and planner-generated plans, in Table 2. We compare local recovery to immediate replanning on the same displaced trials in Table 18. Using only the 5 base primitives gives 424/600 successes vs 464/600 with the full skill library in Table 11. We are willing to run any additional experiments before the discussion ends, and are happy to clarify any points.

AuthorsOpenReview ↗

Local recovery against replanning, and what is new (W1, W3)

Thank you again Reviewer BK58. We are following up on W1 and W3 with a comparison to the closest prior systems.

In W3, you noted it was unclear whether recovery without replanning is preferable to immediate replanning. We found no work before our submission that compares the two with the plan source, skills and perception held fixed. The LM-based systems that we identified (Inner Monologue, REFLECT, DoReMi, Recover, Code-as-Monitor, CaP-Agent0, RATS) invoke the LM after a failure, and ReKep recovers via local repair but has no replanning tier. This comparison is hence new in our paper: on the same displaced trials, trying local repair before a new plan succeeds more often (85/150 and 91/150 at 5 and 10 cm) than requesting a new plan after each failure (70/150 and 72/150), while making fewer planner calls. We are able to make this comparison because each step has an explicit typed check, so the arms differ by one switch. The same structure serves a system operator: the plan is a short tree of named steps that can be read and edited before it runs, and every step and check is logged with its outcome, so a failure can be traced to the step where it happened without reading model reasoning or generated code. This is how we found that 159 of the 767 failures in the strict Table 1 end with the wrong object held. We would be glad to know whether this changes your view of W1.

Thank you again Reviewer BK58. As the discussion closes in a few hours, we would like to ask whether our responses and the revised PDF address your concerns (W1-W4, Q1-Q3). We would be grateful if you would consider updating your assessment if so, and are glad to clarify anything before the close.

Reviewer_pdMTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SPARK, a robotic manipulation framework that separates symbolic behavior trees from execution-time object grounding. A language model generates a behavior tree over a fixed skill library. The system obtains 3D object locations through text-prompted image segmentation and depth information, and binds object labels to current observations during execution. Checks after skill execution can trigger local corrections or perceptual re-grounding; replanning occurs after the execution attempt is exhausted. The authors report a success rate of 66.7% with verification and adaptation enabled, versus 2.2% with both disabled, in a fixed-script experiment involving a continuous 10 cm displacement. They also report a mean success rate of 72.6% across six LIBERO-PRO settings and 150 successes out of 220 physical trials across eleven task-platform combinations on three robot platforms.

Critical strengths
  1. The fixed-script displacement experiment removes differences in language-model plan generation, providing direct evidence that execution-level adaptation can recover task completion while leaving the task plan unchanged.
  2. Experiments on three robot platforms demonstrate implementations of the interface on different hardware, covering both single-arm and bimanual manipulation.
Critical weaknesses
  1. The incremental contribution and informativeness remain insufficiently clear. After understanding the concrete method, I remain uncertain about the technical or conceptual contributions beyond combining existing modules. Retaining an action sequence, re-localizing objects, and performing local recovery are relatively straightforward design choices for this problem. The paper demonstrates that this design works and reports the effects of additional perception descriptions and candidate plans in different settings, but does not sufficiently explain what new, generalizable insights these findings provide relative to existing systems. I therefore remain unconvinced of the incremental technical or conceptual contribution. The paper should clarify the specific new mechanisms and explain how its experimental findings can guide the design of other robotic manipulation frameworks.

  2. The current experiments do not sufficiently attribute the performance gains to the proposed recovery framework. The dynamic displacement experiment enables or disables verification and adaptation together. This demonstrates the benefit of their combination relative to the disabled configuration, but does not separate their individual contributions or compare alternative retry, re-localization, or replanning strategies. Although the main benchmark includes CaP-Agent0 and RATs, these systems differ in their skill libraries, language-model configurations, and control implementations. The results therefore reflect differences between complete systems and do not sufficiently attribute the gains to the proposed recovery framework. Different system configurations do not invalidate comparisons of overall performance, but they limit conclusions about the advantages of the specific mechanisms.

Questions for authors
  1. What specific mechanisms does SPARK introduce beyond existing systems with execution feedback, re-localization, and retry capabilities? Which experiments distinguish the contribution of these mechanisms from that of the skill library, perception, and control implementation?
  2. Which recovery and target-update mechanisms are enabled in each condition of the dynamic displacement experiment? Could the authors provide a matched comparison that shares the skill library and executor while varying only the recovery strategy, or other existing evidence, to establish the benefits relative to alternative recovery methods?
5 comments
AuthorsOpenReview ↗

Attributing the gains (W2, Q2)

Thank you reviewer pdMT for the detailed review and noting the evidence and experiments behind our proposed method. To address your concerns regarding the attribution of gains and the novelty of our method, we extend the fixed-script experiment to a matched comparison between different recovery methods.

Q2: Mechanism by arm. Across all arms, the skills, executor, controller, perception, displacement schedule and plan source were identical. Furthermore, each arm queries perception to refresh object detections if they are >0.5 s old (50ms in the scene-diff arms) prior to executing a keypoint move, and follows the strict protocol (both protocols are defined in the general response). The arms differ in:

ArmScene diffRecovery after a failed attempt
offnonone
scene diffyesnone
laddernorebind to new detections, local retry, then replan, up to 3 attempts
ladder + scene diffyesas ladder
replannoretract, redetect, new plan, up to 2 times
  • Scene diff compares the latest detection to the original binding (the detection a plan label points to) before and during each keypoint move. It retargets if the object moves <10 cm and ends the attempt if it is >10 cm.
  • Attempt failure is defined as failing due to scene diff, failing a check within the plan, or having the last grasp be empty, slipped or not reached.
  • Planner-generated plans include grasp/place checks and retries in all arms (including off), while those produced via scripts do not. For scripts, the ladder stops after local retry.

Strict results over 10 object tasks (successes out of 150 at displacements of 0 / 2 / 5 / 10 cm). Most of the lift comes from the ladder when using scripts. With plans generated by the planner, the ladder contributes roughly half of the +24.0 pts improvement (ladder + scene diff vs. off) at 10 cm.

ArmScripted plansPlanner-generated plans
off117 / 87 / 14 / 3120 / 104 / 72 / 71
scene diff105 / 82 / 43 / 8123 / 101 / 80 / 73
ladder120 / 108 / 79 / 88120 / 109 / 85 / 90
ladder + scene diff123 / 101 / 84 / 90123 / 109 / 99 / 107

Differences in paired trials at 10 cm (tasks resampled):

  • Ladder vs off: +56.7 [+32.7, +78.7] scripted, +12.7 [+4.7, +22.0] planner.
  • Scene diff vs off: +3.3 [+0.7, +6.7] scripted, +1.3 [−11.3, +14.7] planner.
  • Ladder + scene diff vs ladder: +1.3 [−5.3, +8.0] scripted, +11.3 [+2.0, +21.3] for planner.

Ladder vs replan, same displaced trials, planner-generated plans: we ran both arms with and without scene diff so that each pair is matched (successes of 150 at 0 / 2 / 5 / 10 cm):

No scene diffWith scene diff
ladder120 / 111 / 85 / 91123 / 114 / 95 / 105
replan120 / 104 / 70 / 72123 / 108 / 84 / 89

Ladder vs replan with no scene-diff is +10.0 [+2.0, +20.0] at 5 cm and +12.7 [0.0, +32.0] at 10 cm. With scene diff: +7.3 [−0.7, +20.0] and +10.7 [0.0, +28.7].

Cost and hardware. Besides its higher success rate, the ladder is also cheaper than replan. Per displaced trial, it makes 1.14 planner calls against 1.56 for replan (−0.42 [−0.51, −0.34]) since repairing a binding needs no new planner query. With scene diff on both, it makes 1.17 calls versus 1.42 for replan. The same regrounding runs on a physical UR10e in the live-displacement trials of the general response, which have no replan arm.

LIBERO-PRO (no displacement). When nothing moves, the ladder and replan achieve comparable success rates: 457 vs 464 of 600 paired trials under the RATs protocol (−1.2 [−3.3, +0.8]) and 441 vs 443 (−0.3 [−1.7, +1.0]) under the strict protocol. The ladder gets there with fewer LLM calls per trial: 2.71 vs 3.15 under the RATs protocol and 2.68 vs 2.77 under the strict protocol. Recovery remains important even without displacement, as without it, success drops 6.3 points.

System comparisons: We acknowledge that the CaP-Agent0 and RATs rows are system-level comparisons, and we have indicated this in the revision. Our mechanistic claims are supported by the matched arms above.

AuthorsOpenReview ↗

What is new and how the findings guide other systems (W1, Q1)

W1, Q1: What SPARK adds. We outline what SPARK contributes under "Additions" in the general response: a typed plan step coupled with a check of its outcome, which leads to the two contributions listed there. We additionally discuss within the paper the idea of SPARK being used as a data flywheel for training future foundation models.We concur that keeping the plan, relocalizing objects, and retrying are straightforward options, and we cite prior sources of these ideas in the revised manuscript (perceptual anchoring from Coradeschi & Saffiotti; retry before replanning in PLANEX and RAP). What SPARK adds is a systematic evaluation of when local recovery is enough without high-level replanning. While systems such as REFLECT, DoReMi, Recover, and CaP-Agent0 rely on language-model intervention for failure recovery, we isolate the effect of recovery by holding the plan, skills, and perception fixed. Under object displacement, regrounding and local retry substantially outperform replanning (91 vs. 72 successes out of 150 at 10 cm), while requiring fewer planner calls.

Insights on design of concurrent and future frameworks.

  1. Put the effort into retry after a failed check. Rebinding detection and retrying the failed action provides most of the improvement under displacement with scripted plans (about half for planner-generated plans). An object-motion check alone confers little improvement, but helps in combination with retry for planner-generated plans.
  2. Try the cheapest repair first. Local repair beats replanning under displacement and ties it without, all while querying the planner less.
  3. Check which object is held. A grasp check that only tests if an object is grasped misses the 159/767 failures in our strict Table 1 (by our trace labels) that end with the incorrect object held. Checking object identities could address these failures; increasing the number of retries would not.

Q1: How are these factors individually varied?

  • Recovery: We vary only the recovery method and keep the plan, skills, executor and perception fixed, under displacement and without in both real and sim (response 1/2).
  • Skills: We compare 4 skill libraries, from the full library down to the 5 base primitives, with the same planner and executor (see response to BK58).
  • Perception: We compare one object description with alternative descriptions, and we add faults to one perception input at a time (depth noise, mask errors, camera pose errors, or no wrist camera) (see response 2/2 to uFEq).
  • Control: We do not vary control; each robot uses the same controller and IK settings in every experiment.

Please refer to our general response for the current values of the 72.6%, 66.7%, and 2.2% success rates listed.

We again thank Reviewer pdMT for the review, and would like to check whether our responses addressed W1-2 and Q1-2. We have now completed the matched comparison with only the recovery method varied while sharing skills, executor and perception. We obtain 91/150 successes for local recovery at 10 cm displacement vs 72/150 for replanning, with fewer planner calls (see Table 2 and Table 18 in the revised PDF). Our second response to W1/Q1 summarises three lessons that we believe carry over to other robotic systems. We are willing to run any additional experiments or comparisons before the discussion ends.

Reviewer_uFEqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the problem of adaptive manipulation strategies in sequential manipulation tasks. The authors first identify two potential issues when a policy fails on a manipulation task: symbolic plan grounding or perception grounding. This paper thus proposes SPARK, which uses a behavior tree to generate the symbolic plan while using execution-time 3D observations for perception grounding. During evaluation, the proposed approach runs verification and triggers local recovery for small object-location changes, and triggers expensive replanning only when necessary. The proposed approach achieves a 72.6% success rate on the LIBERO-PRO benchmark and is validated on 11 real-world tasks.

Critical strengths

1: I like the motivation for separating symbolic plan grounding and perception grounding. The proposed approach will only trigger expensive replanning when necessary.

2: The proposed approach achieves strong results in the LIBERO-PRO benchmark while outperforming several baselines.

3: The proposed approach is also tested on 11 real-world tasks spanning Tool Use, Folding, and Pick & Place.

Critical weaknesses

1: The perception grounding could be expensive if the proposed approach triggers it frequently. How often does the proposed approach trigger perception grounding?

2: The paper assumes perfect multi-camera perception, which can be a strong assumption in open-world manipulations. Do you have any thoughts on how to operate your system directly from partial-view observations?

3: Because of the multi-camera perception assumption, it’s a bit unfair to directly compare to VLA baselines. VLA can directly operate on RGB images without perfect calibration for external cameras.

4: Failure detection and recovery is significantly studied in the robotics literature. I suggest the authors review the literature more carefully. Some references: https://arxiv.org/abs/2410.04640, https://arxiv.org/abs/2407.08735v1, and https://arxiv.org/abs/2509.01746.

Questions for authors

1: What’s the deployment frequency of your approach in the real world?

2: How would the proposed approach perform given noisy perception? For example, noisy depth, segmentation, and calibration. Would your approach be robust to these noisy observations?

3: What are the failure cases of your approach in the real world?

4: How are the parameterized skills implemented in the paper? How do you plan to generalize to more diverse parameterized skills?

4 comments
AuthorsOpenReview ↗

Cost of grounding and real-world deployment (W1, Q1)

Thank you reviewer uFEq for your constructive feedback, notes on experimental design, concerns about real world deployment and questions. We address your comments below with additional information and experiments that we hope will improve the clarity and wholeness of our work. These updates are also reflected in the revised manuscript.

W1: Grounding Frequency and Cost

SPARK performs perception grounding repeatedly during task execution to maintain up-to-date object locations, while still incurring only a modest runtime overhead. Across 4 manipulation tasks on the UR10e (the live-displacement trials of the general response), we report the frequency of detection calls, their latency, the proportion of trial time spent on perception, and the peak GPU memory usage. Perception ran on the robot PC (RTX 4090) while the 27B planner ran remotely on a separate consumer-grade GPU. The results are summarized below.

UR10e, per episodeUtensils in trayStack blocksPlushie in bowlUtens. in bowl
Success Rate10/1010/1010/108/10
Median trial duration (s)54.437.443.555.7
Mean detection calls per trial17.68.49.79.3
Mean detection calls after failure per trial0.1000.700.70
Median time per detection call (s)0.520.480.500.52
Median share of trial time in perception*15.6%10.1%10.0%5.4%
Peak GPU memory, alloc./reserved (GB)4.8/8.64.9/8.04.7/5.54.6/8.0

*Perception time is the total time of all detection calls (SAM3 on both scene cameras plus 3D localization).

Rows 3-4 show that only 3.3% of detection calls result from a failed action; the remainder are made proactively throughout execution to refresh stale detections before keypoint moves. Paired with the low latency of a single call (∼0.51s\sim 0.51s), the overall time spent on our proposed failure recovery process is marginal relative to the total length of the trial (<1% in every task), and the total account of perception (proactive + after-failure calls) remains small (5.4-15.6%). Importantly, peak GPU memory does not increase with repeated perception calls, so more frequent grounding increases cumulative inference time without increasing the memory footprint.

Q1: Real-World Deployment

Section 5.4 and Appendix H of the original submission detail SPARK's initial physical experiments: 220 randomized trials split evenly across 11 tasks on 3 robot embodiments. We ran the perception pipeline on a local RTX 4090 GPU (24 GB VRAM) and used a hosted Gemini planner accessed via API. The recent physical experiments detailed in W1 use an additional consumer GPU (RTX 5090, 32GB VRAM) to host the open-weight planner Qwen3.8. Below, we report the measured GPU memory footprint and host RAM for each PC on a sample of Stack block trials. The median planner latency is 7.9s7.9 s per planning request (1 per trial) over a median total time of 32.2s32.2 s per trial. When the cameras cannot confirm the final state, the planner also checks the images (∼1.5s\sim1.5 s).

Planner (RTX 5090)Robot PC (RTX 4090)
GPU memory24.5 GB4.9 GB
Host RAM5.9 GB8.3 GB

The planner is invoked once at the start of each trial to generate the behavior-tree plan, and subsequently for failure cases where replanning is required. Perception is similarly event-driven (see W1), triggered when execution requires updated grounding. Crucially, neither component runs continuously at the robot's control frequency.

The physical experiments run across local and hosted planners show SPARK's flexibility across different computational configurations and robot embodiments.

AuthorsOpenReview ↗

Perception faults, VLA comparison, literature, real-world failures, skills (W2, W3, W4, Q2, Q3, Q4)

W2, Q2: Perception perturbation. The 220 original + 120 new physical experiments use real sensors. These include two Time-of-Flight scene cameras and a wrist stereo camera for the single arms, and a ZED Mini and one wrist camera per arm for the Franka bimanual. Each setup requires its own calibration.

To measure the effects of additional perception noise, we ran 300 rollouts across all 6 LIBERO-PRO settings, independently introducing 4 types of visual faults. Results are shown as percentage point differences to the clean baseline (76.7%).

Fault (mild / medium / severe)MildMediumSevere
Depth noise, σ=kz\sigma = kz, kk = 0.01 / 0.02 / 0.05−3.3 [−8.3, +1.0]−9.3−47.7
Mask erosion or dilation, 2 / 4 / 8 px−8.0 [−12.3, −3.7]−12.3−24.7
Extrinsic offset*, 0.5 / 1 / 2 cm−3.7 [−7.3, 0.0]−6.0−29.7
Scene camera only (wrist off)−3.0 [−9.0, +2.7]

*Also adds 0.5$^\circ$ of rotation error per cm.

SPARK shows a tolerance for medium perturbations in depth noise and extrinsic offset, while segmentation errors and severe perturbations more drastically reduce performance. The wrist camera ablation (−3.0-3.0) suggests that SPARK performs decent without grasp refinement and a third view. Operating the system directly from partial-view observations may only be a challenge when occlusions cover the target objects themselves. For obstructed line-of-sight cases related to the position of the arm itself, a natural extension is to perform an active recovery-like step that clear the camera's view and enables more reliable regrounding. Thank you for identifying a future area of work.

W3: VLA comparison. We acknowledge that the input modalities are different, and we now explicitly state it in each relevant section of the paper. Yet, we do not necessarily believe it detracts from the value of the comparison. VLAs don't rely on scene calibration like SPARK does, but they also benefit from task-specific demonstration training that SPARK forgoes. Instead of comparing the specific competitive advantages of each architecture, we choose to use VLAs as a practical system-level benchmark against which to evaluate our work.

W4: Literature. We appreciate the additional references and have included them in the revision. We have also added a paragraph to the related work about detecting failures (Agia et al., Sinha et al.) and learning skills to recover from them (Huang et al.). Still, SPARK remains distinct for its planned post-condition checks and frequent perception regrounding that allow it to recover from failure without specialized model training or high-level replanning.

Q3: Real-world failures. We expand upon the 70 robot failures in the original submission by breaking them down by task & cause:

  • Folding a t-shirt, 24: Unable to reach the bottom hem with the two arm configuration; SAM3 gives inconsistent hem labels when the fabric is dark.
  • Using utensils, 15: SAM3 labels the fork/spoon handle as a knife with high confidence; robot tries to regrasp utensils that fall.
  • Sweeping, 9: Robot grasps too high on the brush.
  • Sorting silverware (bimanual), 8: One arm reaches for an object in the other arm's space.
  • Pouring from a mug, 7: Depth sensor misreads the thin handle on the mug; planner uses the wrong rotation to pour.
  • Stacking blocks, 7: Blocks are too wide for the gripper.

A major limitation arises when an object is systematically misidentified; in this case, no recovery mechanism will help, since detecting the object again will only yield the same incorrect label.

Q4: How skills are implemented and added. We provide each skill as a Python function, along with a specification of input args/types and the post-condition required for success. The executor simply invokes the appropriate function at the corresponding node of the tree. New skills can thus be added by registering a new Python function, which will then be available to the planner with no training needed. Ablations using base primitives only succeeded in block stacking (10/10) but not wiping (0/20). Adding the wipe skill back yielded regular results (12/20).

We again thank Reviewer uFEq for the review, and would like to check whether our responses addressed your questions. We report grounding frequency and cost on the UR10e, with 8.4-17.6 detection calls per trial at about 0.5 s each. We include the sweep over depth noise, mask error and calibration error in Table 17 of the revised PDF, and updated our related work. We are willing to run any additional experiments before the discussion ends, and happy to address any remaining concerns.

Thank you again Reviewer uFEq. As the discussion closes in a few hours, we would like to ask whether our responses address your questions: grounding frequency and deployment cost on the UR10e (W1, Q1), robustness to depth, mask and calibration faults (W2, Q2; Table 17 of the revised PDF), the VLA comparison (W3), the added related work (W4), and the real-world failures and skill implementation (Q3, Q4). We would be grateful if you would consider updating your assessment if so, and are glad to clarify anything before the close.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: What is new, four added studies, and updated results for Tables 1 and 2

We thank the reviewers for their thoughtful feedback and time.

Additions (pdMT W1, Q1; BK58 W1). Some aspects of SPARK (failure detection and recovery, execution monitoring, and perceptual anchoring) exist in prior work, which we cite. Our key novel contributions are the following:

  • Question & answer: What must be recomputed after a failure: (i) reassociate labels with detections, (ii) retry the plan step, or (iii) invoke the planner to construct a new plan? With scripted plans, typically (i)+(ii). These improve the rate of successful completions under 10 cm displacement from 3/150 to 88/150. Further, reexecution within the existing plan achieves 91/150 successes compared to 72/150 with a new plan, while also requiring fewer queries to the planner (same without displacement). Table 1 Strict yielded 159/767 failures where the robot was holding the incorrect object. Increasing retry attempts alone will not solve this; however, validating object identities can. See our response to pdMT (2/2).
  • Labeled plan execution traces: All plan nodes in SPARK correspond to a typed primitive whose result is verified at run time, thus we save the labeled sequence comprising: i) BT plan, ii) individual node logs (target detection, time taken and success/failure flag), iii) robot trajectory with step labels, iv) video, v) success/failure. This produces 6,000 trials with a success label, 7,696 plans executed, 13,186 checks and 14,081 grasp/place actions with a recorded outcome in our Table 1 runs. In physical rollouts, plans that succeed are stored and can be given to the planner as context, but were not used for experiment results. We will share the labeled rollouts with the code upon acceptance.

Added studies:

  • Matched recovery (pdMT W2, Q2; BK58 W3, W4, Q2, Q3). 10 arms (recovery method ×\times plan source: script / planner) across 10 tasks and 4 displacement levels. See pdMT (1/2).
  • Skill ablations (BK58 W2, Q1). 4 sets of skills with the executor and planner held constant.
  • Perception sweep (uFEq W2, W3, Q2). Depth noise, mask error, and camera miscalibration across 3 levels; 1 camera setup mod. See uFEq (2/2).
  • Cost and rates (uFEq W1, Q1). Time taken and #s of queries to LM and scene regroundings per trial on the UR10e and 2 consumer-grade GPUs. See uFEq (1/2).

New Table 1 on LIBERO-PRO: 6 position/task settings on all 50 official initial states (3,000 trials per protocol). The planner is Qwen3.8-27B at 4-bit running on a local machine. We show results under two protocols:

  • Strict: The robot's own checks decide whether to retry; success is checked once, at the end of the episode.
  • RATs protocol: Success is checked after each attempt and a failure is retried, as in RATs. | Protocol | Obj Pos | Obj Task | Goal Pos | Goal Task | Spat Pos | Spat Task | Mean | |---|---|---|---|---|---|---|---| | Strict | 84.0 | 94.6 | 62.2 | 44.2 | 78.8 | 82.8 | 74.4 [67.5, 80.9] | | RATs protocol | 87.8 | 96.0 | 68.6 | 52.2 | 78.0 | 84.2 | 77.8 [71.8, 83.4] |

SPARK outperforms RATs (43.8%) on their protocol by 34.0 pp and 30.6 pp under the strict protocol. This was achieved with no practice on the tasks and no object pose information from simulation. After a failed attempt, both rows replan (at most twice). On CaP-Bench our open planner gets 65.1% vs CaP-Agent0's 68.3% (a 3-model ensemble).

These results replace the submitted 72.6% and 73.5% rows. We changed the planner to an open-weight model to reduce API cost and improve pipeline execution. We also included detection with alternative phrases, wrist-camera checks of target objects, grasp and placement checks added to plans, and a reach check before the gripper closes.

New Table 2 (displacement). Scripted plans with scene diff (an object-motion check) + ladder (rebind, then retry) vs neither, at 10 cm. Over 10 tasks: 111/150 vs 2/150 under the submitted protocol (the success check after an attempt can also start a recovery), and 90/150 vs 3/150 under the strict protocol of New Table 2. On the three submitted tasks under the original protocol: 35/45 vs. 1/45 (previously 30/45 vs. 1/45). See pdMT (1/2) for full statistics.

Physical, UR10e, with the planner of Table 1. Static: 20 trials per task. Live displacement: once the plan has been established, the operator displaces the target/destination object by up to 70 cm. The robot redetects each object prior to the step that names it; 36/40 trials ran on a single plan.

TaskStaticDisplacement
Utensils in tray20/2010/10
Stack blocks19/2010/10
Plushie in bowl20/2010/10
Utensils in bowl18/208/10
The Franka and bimanual rows keep their submitted values with the planner named but they will be run with the open-weight planner for the camera-ready version.
AuthorsOpenReview ↗

Summary of completed experiments

We would like to thank Reviewers BK58, pdMT and uFEq, as well as the Area Chair, for reviewing this work. All requested analyses are in the revised PDF: verification and adaptation measured separately, with scripted and planner-written plans (BK58 W4 and Q2-3; pdMT Q2; Table 2); local recovery versus replanning on the same displaced trials (BK58 W3; pdMT W2; Table 18); the skill-library ablation (BK58 W2 and Q1; Table 11); and the perception-fault sweep over fault type and severity (uFEq W2 and Q2; Table 17). Grounding frequency and cost on the UR10e (uFEq W1 and Q1) are in our November 14 response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.