acceptodds
Under review as a conference paper at ICLR 2027

SPACE: Subtask Planning with Adaptive Computational Effort

Abstract

Hierarchical robot policies enable complex, long-horizon tasks by combining multimodal reasoning for high-level planning with low-level control. However, querying an expensive planner at every planning step incurs unnecessary latency, while relying solely on a lightweight planner can lead to incorrect subtask decisions when stronger reasoning is required. We introduce Subtask Planning with Adaptive Computational Effort (SPACE), a two-step framework that pairs a lightweight local planner with a compute-heavy frontier planner to decide when to update a subtask and when stronger reasoning is needed. The local planner first predicts a binary transition token; if it indicates retention, decoding stops and the robot continues executing the current subtask. Otherwise, the local planner predicts the next subtask, and a confidence-based router either accepts that prediction or invokes the frontier planner. To make that choice, the router considers token probabilities and how much plausible alternative predictions alter the proposed subtask’s meaning or numerical details. Compared with a hierarchical baseline that predicts a full subtask at every planning step, SPACE raises task success from 40.6% to 50.3% on RoboMME and from 28.6% to 38.5% on RoboDojo, while reducing high-level planning time by 60.7% and 31.6%, respectively. On three challenging real-world manipulation tasks, SPACE also outperforms the hierarchical baseline in task success, highlighting its effectiveness across diverse reasoning demands.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Hierarchical Robot Planning, Adaptive Reasoning, Long-Horizon Manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AoQBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SPACE, a framework for improving the efficiency of hierarchical robot planning through adaptive allocation of computational effort. SPACE combines a lightweight Local Planner with a more capable Frontier Planner. The Local Planner first predicts whether to retain the current subtask or generate a new one. When a new subtask is needed, a learned Reliability Router evaluates its correctness using token-level confidence and semantic or numeric differences among candidate predictions, invoking the Frontier Planner when necessary. Experiments on RoboMME, RoboDojo, and three real-world manipulation tasks demonstrate improvements in task success and substantial reductions in high-level planning latency compared with the local-planning baseline.

Critical strengths
  1. The paper addresses an important practical problem in hierarchical robot planning: unnecessary repeated reasoning and the high inference cost of powerful planners. The proposed framework reduces high-level planning latency while improving task success, demonstrating some value for long-horizon robotic tasks.

  2. The authors evaluate SPACE on two simulation benchmarks and three real-world manipulation tasks. The component ablations further demonstrate that transition-aware planning and reliability-based routing contribute to planning efficiency and task performance, respectively.

Critical weaknesses
  1. Limited methodological novelty and insufficient analysis of the key decision mechanisms. The proposed framework primarily combines existing concepts, including early-exit decoding, binary transition prediction, confidence-based model routing, and semantic uncertainty estimation. Although their integration is practically useful, the methodological novelty relative to existing adaptive computation and hierarchical planning approaches appears limited. Furthermore, the Hold mechanism is central to the efficiency improvements. Yet, the paper does not provide a direct evaluation of its classification reliability or analyze the consequences of incorrect Hold decisions. Since the Reliability Router is activated only after Advance predictions, incorrect Hold decisions cannot be corrected by the stronger planner.

  2. Concerns regarding experimental comparability and reported results. According to Appendix D, the Frontier-Only baseline is evaluated on a 192-episode subset of RoboMME, whereas the main SPACE and MemER results use 800 episodes. These results are compared directly in Table 1, making the claimed performance comparability difficult to establish without matched-subset results. Furthermore, Appendix C.1 reports 25 evaluation episodes per RoboDojo task, while Table 7 includes success rates such as 37.0%, 43.6%, and 97.6%. These values are not consistent with binary success rates computed from exactly 25 episodes unless additional averaging or evaluation procedures were used. These issues require clarification because they directly affect the reliability and interpretation of the experimental comparisons.

Questions for authors
  1. Evaluation protocol and statistical consistency. Could the authors clarify how the RoboDojo success rates in Table 7 were computed? Given the stated evaluation of 25 episodes per task, binary success rates should change in increments of 4 percentage points.

  2. Fairness of baseline comparisons. Could the authors report SPACE, MemER, and Frontier-Only results on the same 192-episode RoboMME subset, ideally with uncertainty estimates? Additionally, the Frontier Planner receives task-specific prompt information and a more complete visual history. How do these differences in available information affect the comparison?

  3. Effectiveness and cost of the routing mechanism. How does the learned Reliability Router compare with simpler routing strategies, such as random routing or entropy-based thresholding, under the same Frontier Planner budget? Could the authors also report planning latency averaged over all real-world episodes, including failed rollouts, rather than only successful ones?

4 comments

Dear reviewer AoQB,

We sincerely appreciate your efforts and insightful comments to improve the manuscript.


[W1-1] Concerns on limited methodological novelty.

We would like to clarify that SPACE is the first framework that selectively allocates computational effort when using frontier high-level planners for robot planning. Its methodological novelty lies in allocating computational effort based on subtask transitions and the reliability of subtask predictions. Figure 4 and Table 3 in the paper show that first-token termination reduces planning time compared with full generation, and using semantic differences improves task success over confidence-only routing. Our additional experiments show higher task success with fewer Frontier calls than entropy-based routing. These results demonstrate that these design choices are crucial for SPACE to reduce computational effort while maintaining task performance.

[W1-2] Reliability of Hold predictions and consequences of incorrect decisions.

Following your suggestion, we conducted two additional experiments on RoboMME to evaluate the reliability of Hold/Advance predictions and the effect of transition errors on task success.

First, we compared SPACE’s predictions with simulator-provided ground-truth history context in 192 rollouts. We report transition recall, the percentage of ground-truth transitions detected. When the current subtask was supposed to continue, the Local Planner correctly predicted Hold on 97.6% of calls. As shown in the table below, the Local Planner detected most transitions it had missed on the first call within the next three calls. By the end of this window, its detection rate was close to that of MemER [1], which generates a full subtask at every call.

\def\detrow#1#2{\begin{array}{cc}\detcell{#1}&\detcell{#2}\end{array}} \begin{array}{lc} \hline \begin{array}{c}\vphantom{\text{Transition recall (\\%)}}\newline\text{Method}\end{array} & \begin{array}{c}\text{Transition recall (\\%)}\newline\hline\detrow{\text{At transition}}{\text{Through next 3 calls}}\end{array} \newline \hline \text{SPACE} & \detrow{19.1}{\mathbf{84.0}} \newline \text{MemER} & \detrow{23.2}{\mathbf{86.0}} \newline \hline \end{array}$$ Second, we tested whether correcting transition decisions improves task success. In the same episodes, we replaced incorrect Hold/Advance tokens with the simulator’s ground-truth transition tokens. After Advance, the Local Planner still decoded its own subtask and the original Router was applied. As shown in the table below, correcting Hold/Advance predictions improved task success from 52.08% to 55.73%. $$\begin{array}{lc} \hline \text{Method} & \text{Task success (\\%)} \\\\ \hline \text{SPACE} & 52.08 \\\\ \text{SPACE + oracle transition} & 55.73 \\\\ \hline \end{array}$$ Although the Router cannot correct an incorrect Hold directly, the Local Planner often detects the missed transition within the next planning timesteps. The oracle results show that correcting Hold/Advance errors can further improve task success. We will include both evaluations in the revision. [1] Sridhar et al. Scaling up Memory for Robotic Control via Experience Retrieval. ICLR, 2026.

[W2-1,Q2-1] Additional Results on matched size.

Thank you for highlighting the difference in evaluation size. The original results used 800 episodes for SPACE and MemER [1] and 192 episodes for Frontier-only. We now report all three methods on the same 192 RoboMME episodes, retaining the original Frontier-only rollouts.

\def\srrow#1#2#3#4#5{\begin{array}{ccccc}\srcell{#1}&\srcell{#2}&\srcell{#3}&\srcell{#4}&\srcell{#5}\end{array}} \begin{array}{lcc} \hline \begin{array}{c}\vphantom{\text{Task success rate}}\newline\text{Method}\end{array} & \begin{array}{c}\vphantom{\text{Task success rate}}\newline\text{Time (s)}\end{array} & \begin{array}{c}\text{Task success rate (\\%)}\ \pm\ \text{bootstrap SE}\newline\hline\srrow{\text{Count.}}{\text{Perm.}}{\text{Refer.}}{\text{Imit.}}{{\textbf{Avg.}}}\end{array} \newline \hline \pi_{0.5} & \text{—} & \srrow{29.17\pm5.04}{22.92\pm5.24}{27.08\pm5.50}{6.25\pm2.40}{21.35\pm2.36} \newline \text{w/ Local Planner (MemER)} & 62.90 & \srrow{43.75\pm5.03}{45.83\pm5.20}{45.83\pm5.80}{27.08\pm3.81}{40.63\pm2.51} \newline \text{w/ Frontier Planner} & 422.58 & \srrow{33.33\pm5.15}{52.08\pm3.34}{66.67\pm4.58}{45.83\pm5.15}{49.48\pm2.31} \newline \text{\textbf{w/ SPACE (Ours)}} & \mathbf{40.51} & \srrow{35.42\pm3.41}{72.92\pm3.51}{70.83\pm4.41}{29.17\pm5.34}{\mathbf{52.08}\pm2.12} \newline \hline \end{array}$$ On the same 192 episodes, SPACE achieves a higher success rate than Frontier-only while substantially reducing planning time. We will include this comparison and explicitly distinguish the evaluation sizes in the main text and appendix. [1] Sridhar et al. Scaling up Memory for Robotic Control via Experience Retrieval. ICLR, 2026. --- **[W2-2,Q1] RoboDojo scores and the expected increments of 4 percentage points.** We would like to clarify that the RoboDojo results follow the official partial-credit Score metric rather than binary Success Rate. We report the mean Score over 25 episodes per task and will correct the metric label from Success Rate to Score in the revision. --- **[Q2-2] Effects of differences in prompt information and visual history on the comparison.** We conducted an additional experiment to assess how input context affects the comparison. We gave the Local Planner the same task-specific prompt information and more complete visual history sampled at 2 fps as the Frontier Planner, and compared this setting with SPACE on the same 192 RoboMME episodes. $$\begin{array}{lccc} \hline \text{Method} & \begin{array}{c}\text{Task success} \\\\ (\\%)\end{array} & \begin{array}{c}\text{Planning latency} \\\\ \text{(s)}\end{array} & \begin{array}{c}\text{Frontier usage} \\\\ (\\%)\end{array} \\\\ \hline \text{SPACE + Local Planner with full context} & 8.85 & 58.39 & 43.6 \\\\ \text{SPACE (Ours)} & \mathbf{52.08} & \mathbf{40.51} & 48.1 \\\\ \hline \end{array}$$ When full visual history and task-specific knowledge were added to the Local Planner, task success dropped and planning latency increased, as shown in the table. These results show that providing more information does not necessarily improve performance and that the current Local Planner struggles to effectively use the expanded context. --- **[Q3-1] Additional evaluation on random routing and entropy-based thresholding.** We appreciate your suggestion to compare the Reliability Router with random routing and entropy-based thresholding. We conducted additional experiments comparing the three strategies on the same 192 RoboMME episodes. SPACE and entropy-based routing are calibrated to the same Frontier invocation rate, although actual usage differs during rollouts. As shown in the table below, SPACE achieved higher task success than both random routing and entropy-based thresholding. $$\begin{array}{lcc} \hline \text{Method} & \begin{array}{c}\text{Task} \\\\ \text{success (\\%)}\end{array} & \begin{array}{c}\text{Frontier} \\\\ \text{usage (\\%)}\end{array} \\\\ \hline \text{Local Planner (MemER)} & 40.6 & 0.0 \\\\ \text{Random routing} & 43.2 & 44.7 \\\\ \text{Entropy-based thresholding} & 46.4 & 51.5 \\\\ \text{\textbf{SPACE}} & \mathbf{52.1} & 48.1 \\\\ \hline \end{array}$$ **[Q3-2] Planning latency including failed real-world rollouts.** We report planning latency for successful real-world episodes to show the total high-level planning inference time needed to complete each task. Failed rollouts ended at timeout, and their planning time was excluded from the reported average. We will clarify this measurement scope in the revised appendix. --- We sincerely appreciate the time and effort you took to engage with our work. If there are any remaining concerns beyond those already discussed, we would be grateful if you could share them so that we may address them more thoroughly during the remainder of the rebuttal period.

Thank you for the detailed rebuttal and additional experiments. I appreciate the efforts to address my concerns. The matched 192-episode evaluation, clarification of the RoboDojo Score metric, and comparisons with random and entropy-based routing substantially improve the experimental evidence. However, several concerns remain:

  1. The additional experiments reveal that SPACE detects only 19.1% of ground-truth transitions at the actual transition step, although recall increases to 84.0% within the next three calls. Since incorrect Hold predictions bypass the Reliability Router, could the authors further clarify the distribution of transition delays and their consequences for task execution? The oracle experiment is useful, but it does not fully characterize the severity of delayed transitions.

  2. My original question specifically requested average planning latency across all real-world episodes, including failures. The rebuttal explains why failed episodes were excluded but does not provide the requested results. Could the authors report the accumulated planning time averaged over all attempted episodes under the same evaluation protocol?

Dear Reviewer AoQB,

We are glad that our responses have substantially addressed your concerns regarding the experimental evidence. Here, we address your remaining questions below.


[Q1] Transition delays and their effects on task execution.

Following your suggestion, we analyzed the distribution of transition delays on RoboMME. We also evaluated task success on RoboMME by providing the high-level planner with ground-truth transition calls at varying delays.

Specifically, we first compute the cumulative percentage of ground-truth subtask transitions that SPACE detected by each planning call. As shown in the table below, most delayed detections occurred one or two planning calls after the ground-truth transition (0.8–1.6 seconds).

\def\rerow#1#2#3#4#5#6{\begin{array}{cccccc}\recell{#1}&\recell{#2}&\recell{#3}&\recell{#4}&\recell{#5}&\recell{#6}\end{array}} \begin{array}{lc} \hline \begin{array}{c}\vphantom{\text{Transition recall}}\newline\text{Method}\end{array} & \begin{array}{c} \text{Transition recall (\\%) (increase in percentage points)}\newline \hline \rerow{\text{At transition}}{\text{By +1 call}}{\text{By +2 calls}}{\text{By +3 calls}}{\text{By +4 calls}}{\text{Full window}} \end{array} \newline \hline \text{SPACE} & \rerow{19.1}{50.5(+31.4)}{77.7(+27.2)}{84.0(+6.3)}{85.6(+1.6)}{88.7(+3.1)} \newline \hline \end{array}$$ Next, to assess how these transition delays affect task execution, we additionally evaluated task success on 192 RoboMME episodes, varying the allowed delay after ground-truth transition. The Local Planner’s Advance prediction is used if it occurs by the specified call. Otherwise, the oracle forces Advance at that call. In these settings, the VLA continued executing the preceding subtask for up to 0.8, 1.6, and 2.4 seconds, respectively. The table below shows similar task success across the oracle transition correction settings except ground-truth transition call (“At transition”). Forcing Advance at the “At transition” yielded the lowest task success. In this setting, the Frontier Planner did not update the preceding subtask in 27 of 103 cases, compared with 9–17 of 103 under delay settings. This suggests that the planner may recognize subtask transition from current visual observation later than the state based oracle transition. $$\def\orcell#1{\mathop{#1}\limits^{\textstyle\hphantom{\text{At transition}}}} \def\orrow#1#2#3#4{\begin{array}{cccc}\orcell{#1}&\orcell{#2}&\orcell{#3}&\orcell{#4}\end{array}} \begin{array}{lcc} \hline \begin{array}{c}\vphantom{\text{Oracle transition correction}}\newline\text{Metric}\end{array} & \begin{array}{c}\vphantom{\text{Oracle transition correction}}\newline\text{SPACE}\end{array} & \begin{array}{c} \text{Oracle transition correction}\newline \hline \orrow{\text{At transition}}{\text{By +1 call}}{\text{By +2 calls}}{\text{By +3 calls}} \end{array} \newline \hline \text{Task success (\\%)} & 52.08 & \orrow{48.44}{56.77}{57.81}{55.73} \newline \hline \end{array}$$ --- **[Q2] Planning time including failed real-world rollouts.** Thank you for your clarification. Following your suggestion, we additionally report the mean accumulated high-level planning time per episode across all attempts, including failures. For failed episodes, we measured the time spent on high-level planning up to the timeout, which was 60 seconds for Normal episodes and 120 seconds for Hard episodes. As shown in the table below, SPACE had lower mean planning latency than MemER on all three tasks at both difficulty levels. $$\def\taskcell#1{\mathop{#1}\limits^{\textstyle\hphantom{\mathbf{00.00\pm00.00}}}} \def\taskrow#1#2{\begin{array}{cc}\taskcell{#1}&\taskcell{#2}\end{array}} \def\taskhead#1{\begin{array}{c}\text{#1}\newline\hline\taskrow{\text{Normal}}{\text{Hard}}\end{array}} \begin{array}{lcccc} \hline \begin{array}{c}\vphantom{\text{Tower of London}}\newline\text{Method}\end{array} & \taskhead{Tower of London} & \taskhead{Solve Equations} & \taskhead{Shell Game} & \begin{array}{c}\vphantom{\text{Tower of London}}\newline\text{Overall}\end{array} \newline \hline \text{MemER} & \taskrow{27.99\pm0.71}{59.63\pm0.99} & \taskrow{29.16\pm0.81}{60.20\pm1.82} & \taskrow{37.62\pm0.75}{38.69\pm0.62} & 41.63\pm0.30 \newline \text{SPACE} & \taskrow{\mathbf{9.19\pm0.03}}{\mathbf{46.40\pm6.50}} & \taskrow{\mathbf{14.18\pm0.98}}{\mathbf{43.01\pm16.04}} & \taskrow{\mathbf{29.51\pm0.71}}{\mathbf{34.86\pm1.63}} & \mathbf{29.91\pm2.25} \newline \hline \end{array}$$ --- If you have any further comments or suggestions, please let us know. We are committed to improving the quality of our work, and we value your feedback. Thank you very much, Authors
Reviewer_UVk9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework called SPACE that reduces the cost of high-level robot planning through two decisions. First, a local planner predicts whether to keep the current subtask or generate a new one. If a new subtask is needed, a learned router decides whether to accept the local prediction or call a stronger frontier model.

Critical strengths
  1. The problem is practical and the design is sensible and I personally think this insight is interesting . Deciding whether a new plan is needed is different from deciding which model should produce it. Separating these decisions can avoid expensive planning calls when the robot should simply continue its current subtask. Also this is much practical in the policy deployment setting.

  2. The paper includes two simulation benchmarks and real-world tasks involving rearrangement, arithmetic, and tracking through occlusion.

Critical weaknesses
  1. The RoboDojo success rates do not match the stated evaluation size. The paper states that each task is evaluated on 25 episodes. For binary success metric, the per-task success rate should therefore change in steps of 4 percentage points. However, the paper reports values such as 37.0%? From the Line 844, "We run 25 episodes per task, yielding 200 closed-loop episodes in to- tal. "
Questions for authors
  1. At the same frontier-call budget, how does SPACE compare with random routing? How many local errors are corrected, and how many correct predictions are harmed?

  2. Which components are included in the timing measurements? Can you provide a latency breakdown and results that include failed episodes?

  3. Can you clarify the real-world dataset sizes and clarify the executor and training protocols?

3 comments

Dear reviewer UVk9,

We sincerely appreciate your efforts and insightful comments to improve the manuscript.


[W1] RoboDojo scores and the expected increments of 4 percentage points.

We would like to clarify that the RoboDojo results follow the official partial-credit Score metric rather than binary Success Rate. We report the mean Score over 25 episodes per task and will correct the metric label from Success Rate to Score in the revision. We have revised the manuscript to clarify the scoring metric.


[Q1] Comparison of SPACE with random routing.

Following your suggestion, we additionally conducted experiments to compare random routing with SPACE. Specifically, we compared local planner, random router, and reliability router (SPACE) on the same 192 RoboMME episodes in the table below.

We calibrate SPACE’s routing threshold to target 40% Frontier usage. Random routing uses a fixed invocation probability of 48.1%, matching SPACE’s observed usage, although actual Frontier invocation rates differ. For routed predictions, we compare Local and Frontier outputs against ground-truth annotations to count corrections and newly introduced errors.

As shown in the table, SPACE achieved 52.1% task success, compared with 43.2% for random routing. Frontier Planner corrected 62 Local Planner errors with SPACE and 28 with random routing. It turned 37 correct predictions into errors under each strategy. This result shows the advantage of the reliability router of SPACE over random routing.

\hline \text{Method} & \begin{array}{c}\text{Task} \\\\ \text{success (\\%)}\end{array} & \begin{array}{c}\text{Frontier} \\\\ \text{usage (\\%)}\end{array} & \begin{array}{c}\text{Corrected} \\\\ \text{Local errors}\end{array} & \begin{array}{c}\text{Harmed} \\\\ \text{correct predictions}\end{array} \\\\ \hline \text{Local Planner (MemER)} & 40.6 & 0.0 & \text{—} & \text{—} \\\\ \text{Random routing} & 43.2 & 44.7 & 28/95 & 37/143 \\\\ \text{\textbf{SPACE}} & \mathbf{52.1} & 48.1 & 62/179 & 37/99 \\\\ \hline \end{array}$$ --- **[Q2] Planning latency breakdown including failed episodes.** We measure end-to-end planning latency, including communication and model inference for both Local and Frontier Planners. For SPACE, Local Planner latency also includes Reliability Router inference. The planning latency reported in *Table 1* of the paper already includes both successful and failed episodes. We additionally analyzed the planning latency of the Local Planner baseline (MemER [1]), SPACE, and Frontier-only on the same 192 RoboMME episodes to report a latency breakdown, including separate results for successful and failed episodes. The table reports mean latency in seconds per episode, with the Local Planner running on NVIDIA A100 80GB GPUs. As shown in the table below, SPACE had the lowest mean planning latency when both successful and failed episodes were included, averaging 40.51 seconds per episode compared with 62.90 for MemER and 323.97 for Frontier-only. $$\def\latcell#1{\mathop{#1}\limits^{\textstyle\hphantom{\text{Frontier}}}} \def\latrow#1#2#3{\begin{array}{ccc}\latcell{#1}&\latcell{#2}&\latcell{#3}\end{array}} \def\lathead#1{\begin{array}{c}\text{#1}\newline\hline\latrow{\text{Local}}{\text{Frontier}}{\text{Total}}\end{array}} \begin{array}{lccc} \hline \begin{array}{c}\vphantom{\text{All (s)}}\newline\begin{array}{c}\latcell{\text{Method}}\end{array}\end{array} & \lathead{All (s)} & \lathead{Success (s)} & \lathead{Fail (s)} \newline \hline \text{MemER} & \latrow{\phantom{0}62.90}{\text{—}}{{\phantom{0}62.90}} & \latrow{\phantom{0}51.40}{\text{—}}{{\phantom{0}51.40}} & \latrow{\phantom{0}70.77}{\text{—}}{{\phantom{0}70.77}} \newline \text{Frontier-only} & \latrow{\text{—}}{323.97}{{323.97}} & \latrow{\text{—}}{117.03}{{117.03}} & \latrow{\text{—}}{537.58}{{537.58}} \newline \mathbf{SPACE} & \latrow{\phantom{0}22.65}{\phantom{0}17.86}{\mathbf{\phantom{0}40.51}} & \latrow{\phantom{0}16.38}{\phantom{0}12.86}{\mathbf{\phantom{0}29.24}} & \latrow{\phantom{0}29.46}{\phantom{0}23.30}{\mathbf{\phantom{0}52.76}} \newline \hline \end{array}$$ [1] Sridhar et al. Scaling up Memory for Robotic Control via Experience Retrieval. ICLR, 2026.

[Q3] Real-world dataset sizes, executor, and training protocols.

We would like to clarify that we collected 165 teleoperated demonstrations and evaluated each method on the same 70 separate test episodes. The table below gives the demonstration counts by task and difficulty.

\def\eprow#1#2#3{\begin{array}{ccc}\epcell{#1}&\epcell{#2}&\epcell{#3}\end{array}} \begin{array}{lc} \hline \begin{array}{c}\vphantom{\text{Number of episodes}}\newline\begin{array}{c}\epcell{\text{Task}}\end{array}\end{array} & \begin{array}{c}\text{Number of episodes}\newline\hline\eprow{\text{Normal}}{\text{Hard}}{\mathbf{Total}}\end{array} \newline \hline \text{Tower of London} & \eprow{20}{40}{\mathbf{60}} \newline \text{Solve Equations} & \eprow{30}{30}{\mathbf{60}} \newline \text{Shell Game} & \eprow{13}{32}{\mathbf{45}} \newline \hline \end{array}$$ For the executor, we use $\pi_{0.5}$ to generate robot actions on all three tasks. The baseline ($\pi_{0.5}$) uses the global task instruction during training and evaluation. For hierarchical methods, we train $\pi_{0.5}$ with subtask annotations. At evaluation time, it generates action based on the subtask selected by the high-level planner. For training, we used 132 demonstrations for Local Planner fine-tuning, 24 for Reliability Router training, and 9 for threshold calibration. Both the SPACE Local Planner and MemER [1] were fine-tuned from Qwen3-VL-4B-Instruct for 10 epochs with LoRA rank 16 and a global batch size of 32. Across all three tasks, we fine-tuned $\pi_{0.5}$ from $\pi_{0.5}$-base for 10,000 steps with a global batch size of 64. We will include these details in the revised manuscript. [1] Sridhar et al. Scaling up Memory for Robotic Control via Experience Retrieval. ICLR, 2026. --- We sincerely appreciate the time and effort you took to engage with our work. If there are any remaining concerns beyond those already discussed, we would be grateful if you could share them so that we may address them more thoroughly during the remainder of the rebuttal period.

Dear Reviewer UVk9,

Thank you again for your detailed feedback. We would like to follow up on our rebuttal, where we clarified the RoboDojo Score metric and provided additional results on routing and latency, along with details of the real-world experiments.

We would appreciate your thoughts on whether these responses address your concerns and whether any questions remain. We would be happy to provide further clarification.

Thank you for your time and consideration.

Reviewer_JwyHOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work addressed the topic of accelerating a modular architecutre of existing robotic models, which combinnes both high-level planner and low-level maniplator. The core idea of this work is to develop a cascaded scheme that adaptively chooses between a normal-sized planner (a finetuned version of Qwen-VL) and a more accurate yet computationally expensive GPT. To this end, authors design a route to switch between local planner and frontier planner. Extensive experiments are conducted on both simulation and real robotic data.

Critical strengths

In general I think that this work is making advance towards an important direction in current field of robotics. After the release of GPT-6 astra, it becomes more accepted insight that general foundation models (originally for text and multi-modality data) can be extended to server as effective high-level robotic planner. However, these model suffers from high computations. Therefore, it is a promising and urgent directiont to develop new schemes that more efficiently routes between VLA and a combination of varying-sized high-level planner with diffreren parameer numbers. The work is thus well motivated.

The experiments sufficiently support the major claim regarding accuracy-reasong cost tradeoff, as shown in Table 1 in the experimental sections. It is noted that while the high success rate of GPT-style planner is sustained, the cost is largely reduced in terms of time. Similar observations for the real-world experiments.

The writing is clear for the readers grasping the key ideas and the proposed model. There are also multiple figures to better illustrate the computational pipeline as in Figure 2.

The proposed model as described in Secton 3 is a reasonable one. It consists of a sequential decision pipeline, leaving most cases to staying with VLA.

Figure 5 is quite information and consistent to the belief that many hold: the heavy use of GPT-style planner does not guarantee a continuous performance improvement since it is not sufficiently trained for lower level decisions and actions.

Critical weaknesses

There are multiple weaknesses in this paper. First, the technical contribution is somewhat limited. The proposed model in Section 3 is a direct cascade of three components, with a reliability router for scheduling. The generalization to more unseen robotic manipulation tasks, particularly high-precision manipulaiton, is unclear. It is noted that the proposed model is post-trained with data for all tasks in the experiments. More discussion and empirical studies on unseen tasks will be appreciated.

The local planner uses a token of "advance" or "hold" for choosing between advancing with VLA or not. Essentially, this requires an accurate estimation of sub-goal's progress. However, this is not explicitly addressed or learned in the proposed model. I have concern regarding its generatlization to progress estimation in out-of-domain scenarios. In the experiments, it is also help to reveal the detailed ratios of different stages in the whole pipeline in practice.

There are several designs and details that need clarification. It is revealed in Section 3.3 that part of the data is excluded for later training of the local planner. The local planner also predicts frame indices that will be stored in the context. However, it is only superficially mentioned in main text (with reference to the appendix) and no further validation is provided in the experiments. The hyper-parameter tao_B is used as a threshold for q_t. A discussion or ablation study is missing. For other hyper-parameters (such as 10 pixels or 2 fps used in the implementation), justification is mostly missing.

The performance improvement with respect to Frontier planner is only marginal. As seen in Table 1, for example, it is improved from a success rate of 36% upto 38.48% on RoboDojo. Considering further fitting to the benchmark is done in SPACE, the improvement is not too low to be significant enough. The choice of GPT-5.6 rather than GPT-6 is a bit weird, since the latter is known to be far more better on 3D understanding and embodied planning. Some clarification is desired for this choice. The real robot experiments choose three tasks, yet most of them are not touch-rich, thus the claim is not convincing enough. More experiments on more diverse real robot tasks are encouraged.

Questions for authors

See my comments in the weakness section.

5 comments

Dear reviewer JwyH,

We sincerely appreciate your efforts and insightful comments to improve the manuscript.


[W1-1] Technical contribution of SPACE.

We would like to clarify that SPACE is the first framework that selectively allocates computational effort for efficient use of frontier high-level planners in robot planning. The contribution lies in determining when additional computation improves planning, with architectural choices supported by experiments. Figure 4 in the paper demonstrates the benefit of first-token termination, and Table 3(b) shows the benefit of using semantic differences for routing. Our additional experiments further show that SPACE’s semantic router achieves higher task success rate with fewer Frontier calls than entropy-based routing. These results show that effective computational effort allocation in SPACE depends on the specific design of its components, not simply on cascading them.

[W1-2] Generalization to unseen tasks including high-precision manipulation.

Following your constructive suggestion, we conducted additional experiments on RoboMME by constructing 3 different task splits, each excluding 2 of the 16 tasks from training. For each split, we evaluated each Local Planner and Router configuration on both the 14 seen and 2 unseen tasks. As shown in the table below, with the Local Planner fixed, excluding the evaluated tasks from Reliability Router training yields no observed decline in task success at similar Frontier usage. Also, when the Local Planner was trained without the evaluated tasks, SPACE showed a smaller drop in success rate than MemER [1] under the same unseen task setting.

Regarding high-precision manipulation, our RoboDojo evaluation includes Fill Pen Holder, Fill Egg Holder, and Make Kong, which involve precise object placement and bimanual coordination. Using the same low-level policy across planning methods allows controlled comparison of high-level planning on these manipulation tasks.

\def\ratebox#1{\fixcell{#1}{\mathbf{00.0}}} \def\sruse#1#2{\ratebox{#1}\text{ (}\ratebox{#2}\text{)}} \def\trainrow#1#2{\begin{array}{cc}\fixcell{#1}{\text{Reliability}}&\fixcell{#2}{\text{Reliability}}\end{array}} \def\resultrow#1#2#3#4{\begin{array}{cccc}\fixcell{#1}{\text{VideoUnmaskSwap}}&\fixcell{#2}{\text{VideoUnmaskSwap}}&\fixcell{#3}{\text{VideoUnmaskSwap}}&\fixcell{#4}{\sruse{00.0}{00.0}}\end{array}} \begin{array}{lcc} \hline \text{Method} & \begin{array}{c} \text{Training on evaluated tasks}\newline \hline \trainrow{\begin{array}{c}\text{Local}\newline\text{Planner}\end{array}}{\begin{array}{c}\text{Reliability}\newline\text{Router}\end{array}} \end{array} & \begin{array}{c} \text{Task success (Frontier usage), \\%}\newline \hline \resultrow{\begin{array}{c}\text{VideoUnmask +}\newline\text{VideoUnmaskSwap}\end{array}}{\begin{array}{c}\text{RouteStick +}\newline\text{PickHighlight}\end{array}}{\begin{array}{c}\text{BinFill +}\newline\text{VideoRepick}\end{array}}{\mathbf{Avg.}} \end{array}\newline \hline \begin{array}{l}\text{Local Planner}\newline\text{(MemER)}\end{array} & \begin{array}{c} \trainrow{\text{Seen}}{\text{—}}\newline \trainrow{\text{Unseen}}{\text{—}} \end{array} & \begin{array}{c} \resultrow{\sruse{58.3}{\text{N/A}}}{\sruse{50.0}{\text{N/A}}}{\sruse{33.3}{\text{N/A}}}{\sruse{\mathbf{47.2}}{\text{N/A}}}\newline \resultrow{\sruse{50.0}{\text{N/A}}}{\sruse{12.5}{\text{N/A}}}{\sruse{25.0}{\text{N/A}}}{\sruse{\mathbf{29.2}}{\text{N/A}}} \end{array}\newline \hline \mathbf{SPACE} & \begin{array}{c} \trainrow{\text{Seen}}{\text{Seen}}\newline \trainrow{\text{Seen}}{\text{Unseen}}\newline \trainrow{\text{Unseen}}{\text{Unseen}} \end{array} & \begin{array}{c} \resultrow{\sruse{91.7}{69.2}}{\sruse{29.2}{56.0}}{\sruse{50.0}{42.9}}{\sruse{\mathbf{56.9}}{\mathbf{56.0}}}\newline \resultrow{\sruse{95.8}{70.7}}{\sruse{33.3}{56.0}}{\sruse{58.3}{39.0}}{\sruse{\mathbf{62.5}}{\mathbf{55.2}}}\newline \resultrow{\sruse{95.8}{95.1}}{\sruse{25.0}{70.0}}{\sruse{12.5}{79.2}}{\sruse{\mathbf{44.4}}{\mathbf{81.5}}} \end{array}\newline \hline \end{array}$$ [1] Sridhar et al. Scaling up Memory for Robotic Control via Experience Retrieval. ICLR, 2026.

[W2-1] Training the Local Planner to estimate subtask progress.

We would like to clarify that progress estimation is explicitly learned during Local Planner fine-tuning, as described in Section 3.2 and Appendix A.1. Specifically, we derive labels from ground-truth subtask annotations and train the first output token with Advance at subtask transitions and Hold otherwise.

[W2-2] Generalization to progress estimation in out-of-domain scenarios.

To address your concern about generalization of progress estimation, we conducted additional offline evaluations on RoboMME. We excluded two of the 16 tasks from Local Planner fine-tuning and repeated the evaluation with three different task combinations. Both models received the ground-truth planning context. In this setting, we evaluated transition recall (detected transitions / all ground-truth transitions) across three task combinations covering 1,045 ground-truth transitions. Detection requires an Advance prediction at the ground-truth transition timestep or within the following three planning timesteps.

On the same evaluated tasks, mean recall was 88.7±6.988.7 \pm 6.9\\% when these tasks were included in fine-tuning and 73.9±13.973.9 \pm 13.9\\% when they were excluded, with standard deviations across the three combinations. Although recall decreased, the model still detected most transitions on tasks excluded from fine-tuning, supporting the generalization of learned progress estimation under this offline evaluation.

[W2-3] Details on ratio of different pipeline stages.

Thank you for the suggestion. On RoboMME, the Router selects Hold, Advance with the Local Planner, and Advance with the Frontier Planner in 89.7%, 5.5%, and 4.8% of decisions, respectively.

[W3-1] Data excluded from Local Planner training.

We would like to clarify that the episodes excluded from Local Planner fine-tuning are reserved for Reliability Router training and threshold calibration, as described in Section 3.3. The Router therefore learns to assess the correctness of the Local Planner’s subtask predictions on episodes the Local Planner has not trained on. The table below summarizes the episode counts for Local Planner fine-tuning, Router training, and threshold calibration.

\def\epcal#1{\smash[t]{\overset{\textstyle\hphantom{\text{Calibration Ep.}}}{#1}}} \def\routerrow#1#2{\begin{array}{cc}\eptrain{#1}&\epcal{#2}\end{array}} \begin{array}{lccc} \hline \begin{array}{l}\vphantom{\text{Router}} \\\\ \text{Benchmark}\end{array} & \begin{array}{c}\text{Local Planner} \\\\ \hline \text{Training Ep.}\end{array} & \begin{array}{c} \text{Router} \\\\ \hline \begin{array}{cc}\text{Training Ep.}&\text{Calibration Ep.}\end{array} \end{array} & \begin{array}{c}\vphantom{\text{Router}} \\\\ \text{Total Ep.}\end{array} \\\\ \hline \text{RoboMME} & 1{,}600 & \routerrow{560}{240} & 2{,}400 \\\\ \text{RoboDojo} & 640 & \routerrow{112}{48} & 800 \\\\ \text{Real-world} & 132 & \routerrow{24}{9} & 165 \\\\ \hline \end{array}$$ **[W3-2] Clarification of frame indices prediction.** The Local Planner predicts frame indices as an auxiliary function for preserving task history in long-horizon tasks, following the keyframe memory mechanism of MemER [1]. As described in *Section 3.2* and *Appendix A.1*, the Local Planner predicts frame indices to retain as memory keyframes along with a new subtask when it predicts Advance, and the selected frames are added to visual memory. This preserves task-relevant history beyond the recent observation window. Since we follow MemER’s keyframe memory mechanism directly [1], a separate evaluation was outside the primary scope of this study. [1] Sridhar et al. Scaling up Memory for Robotic Control via Experience Retrieval. ICLR, 2026. **[W3-3] Clarification of hyper-parameter $\tau_B$.** We would like to clarify that Figure 5 in the main paper already provides an ablation on the routing threshold by varying the target Frontier invocation rate $B$. We select $\tau_B$ on the calibration set to match $B$ among Advance predictions and keep the threshold fixed during evaluation. The results show that greater Frontier usage increases planning time without necessarily improving task success. **[W3-4] Justification of hyper-parameters.** Following your suggestion, we additionally conducted sensitivity experiments on the same 192 RoboMME episodes. We evaluated different Frontier history sampling rates and the coordinate tolerances, while keeping other parameters at their default values. As shown in the table below, success rate and Frontier usage remain similar across these settings, indicating limited sensitivity to the parameter values tested. We will include the hyper-parameter choice and sensitivity results in the revision. $$\def\setcell#1#2{\mathop{#1}\limits^{\textstyle\hphantom{#2}}} \def\setrow#1#2{\begin{array}{cc}\setcell{#1}{\text{2 (Default)}}&\setcell{#2}{\text{Coordinate error (px)}}\end{array}} \begin{array}{ccc} \hline \begin{array}{c}\text{Setting}\newline\hline\setrow{\text{FPS}}{\text{Coordinate error (px)}}\end{array} & \begin{array}{c}\vphantom{\text{Setting}}\newline\text{Task success (\\%)}\end{array} & \begin{array}{c}\vphantom{\text{Setting}}\newline\text{Frontier usage (\\%)}\end{array} \newline \hline \setrow{\mathbf{1}}{10} & 46.9 & 48.9 \newline \setrow{\mathbf{3}}{10} & 51.6 & 47.1 \newline \setrow{2}{\mathbf{5}} & 53.1 & 44.7 \newline \setrow{2}{\mathbf{20}} & 53.6 & 46.7 \newline \hline \setrow{\text{2 (Default)}}{\text{10 (Default)}} & 52.1 & 48.1 \newline \hline \end{array}$$

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.