acceptodds
Under review as a conference paper at ICLR 2027

CRACKING Mobile Manipulation: From Failure Diagnosis to Agentic Intervention

Abstract

Recent vision-language-action (VLA) models have shown strong performance on local robot manipulation, yet their success does not readily translate to long-horizon mobile manipulation. In our experiments, a pretrained subtask VLA achieves 80% success on atomic manipulation tasks but only 19% when these skills are composed into mobile manipulation. We argue that this gap stems from failures across the execution pipeline rather than from local manipulation capability alone. To uncover these bottlenecks, we systematically diagnose failures in long-horizon mobile manipulation and identify three dominant regimes: task planning, navigation and base positioning, and local manipulation. Based on this diagnosis, we develop a failure-guided agentic framework that augments a fixed subtask VLA with demonstration-based planning guidance, adaptive visual memory, accessibility-aware base adjustment, and local end-effector adjustment. On RoboCasa365, these interventions progressively improve task success from 19% to 47%, with the finalized system reaching 49% on held-out episode seeds, compared with 18% for the baseline. The same design improves performance from 14% to 37% on BEHAVIOR-1K and from 5/20 to 13/20 on real-robot tasks. These results suggest that reliable mobile manipulation can be advanced by diagnosing and intervening on the current bottleneck, rather than uniformly scaling the underlying policy.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Mobile Manipulation, Agent, Embodied AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_91XXOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents an agentic framework for long-horizon mobile manipulation. The authors first analyze failures in task planning, navigation, and manipulation. Based on this analysis, they build a framework that combines demonstration-based planning, visual memory, base adjustment, and end-effector correction with a fixed VLA policy. The method improves task success rates from 19% to 47% on RoboCasa365 and from 14% to 37% on five BEHAVIOR-1K tasks. Real-robot experiments also show improvements.

Critical strengths
  1. Practical and well-designed system framework.

The paper presents a comprehensive agentic framework specifically designed for mobile manipulation. The authors carefully divide the long-horizon task into different stages and design corresponding tools to address their challenges. This requires considerable engineering effort and provides a practical solution for improving mobile manipulation reliability.

  1. Clear improvements in task success rates.

The proposed framework achieves substantial improvements on RoboCasa365, BEHAVIOR-1K, and real-robot tasks. These results demonstrate the practical value of combining task planning, memory, and geometric adjustment with an existing VLA policy.

Critical weaknesses
  1. Limited methodological novelty and simplified reachability analysis.

The proposed framework mainly integrates existing and commonly used methods. Its contribution appears to be more about system integration than new algorithmic ideas. In particular, the reachability check, which is important for the transition from navigation to manipulation, relies mainly on the distance between the robot base and the target. However, manipulation reachability also depends on target visibility, available arm workspace, and collision risks. These factors are not sufficiently considered, making the current analysis rather simplified.

  1. Limited connection between failure diagnosis and online intervention, with insufficient evaluation.

Although failure diagnosis is a central motivation of the paper, the collected failure trajectories are mainly used to identify problems and guide system design. During execution, the framework supports observation-based adjustments and retries, but it does not provide an explicit online failure diagnosis mechanism. This weakens the connection between the stated motivation and the proposed method.

In addition, the evaluation lacks analysis of system latency and computational cost. The experiments on BEHAVIOR-1K are also limited, and the generalization ability of the framework to unseen instructions and tasks remains unclear.

Questions for authors

1.Reachability analysis: Why does the proposed reachability check mainly rely on target distance? Could the authors clarify how factors such as visibility, available arm workspace, and collision risks are handled?

2.Failure diagnosis: Does the proposed framework explicitly diagnose failure types during online execution, or is failure analysis mainly used during system design? How does the agent determine which intervention is needed when a failure occurs?

3.System efficiency: What additional inference latency and computational cost are introduced by the agentic framework? Could the authors provide a comparison of task execution time and computational overhead against conventional VLA methods?

4.Evaluation and generalization: Could the authors provide more extensive experiments on BEHAVIOR-1K? In particular, how well does the framework generalize to unseen instructions in familiar environments or completely unseen tasks? Such results would help establish whether the proposed framework improves general mobile manipulation capabilities rather than performance on a fixed set of tasks.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 91XX

W1-1. Methodological novelty. The framework mainly integrates existing, commonly used methods. Its contribution appears to be system integration rather than new algorithmic ideas.

Please see our response to Reviewer jnBh’s W1 & Q1, where we clarify how our approach and failure analysis extend prior work. More importantly, given that GPT-6-Astra already demonstrates strong capability in embodied tasks, similar agentic frameworks would be promising in the future. Our work provides empirical diagnosis results for this new paradigm which was not offered by prior works. And we hope that these results would guide future design about agentic framework for mobile manipulation.


W1-2 & Q1. Simplified reachability analysis. Why does the reachability check rely mainly on base-to-target distance, and how does it account for target visibility, arm workspace, and collision risk?

Thank you for suggesting a direct comparison with reachability-based and learned base placement. Originally, we used the 0.75/1.00 m distance thresholds as a low-cost way to test whether adjusting the robot’s stance before manipulation could help the same VLA executor. For the follow-up comparison, we evaluated both alternatives on the RoboCasa365 development set used for failure diagnosis, with the same episode seeds as in the original distance-threshold run. The simple inverse kinematics (IK) method checks whether the arm can reach the estimated interaction target from a candidate base pose. The learned method ranks candidate poses using the next subtask and target-relative RGB-D features. Across these matched episodes, five failures were attributed to navigation and base positioning with the original distance thresholds, two with simple IK, and one with learned base placement.


W2-1 & Q2. Does the framework explicitly classify failures during execution or mainly use failure analysis for system design, and how does the agent choose an intervention?

We assign the three failure categories offline by reviewing completed rollouts and identifying the first unrecovered error. This analysis guided the design of the intervention tools. During execution, the VLM planner uses current observations, tool feedback, and episode history to track which subtasks are complete and what remains to be done. When an action fails or the task state changes, the planner revises its plan online. Based on the observed outcome, it can select a recovery tool and retry the current subtask or change what it does next.


W2-2 & Q3. How do inference latency, task execution time, and computational cost compare with conventional VLA methods

On the held-out RoboCasa365 episodes, our full system with GPT-6.1 Sol high achieved 52% success with our original subtask VLA and 56% with Xiaomi Robotics 1 fine-tuned to follow subtask instructions.

We compared three Xiaomi Robotics 1 configurations on the matched episodes. Harness VLA and our full system used the same subtask-adapted checkpoint, the same GPT-6.1 Sol high planner, and a limit of 100 online VLM calls per episode. The end-to-end baseline ran the released checkpoint directly, with zero calls to a separate online VLM.

ModelHeld-out successMean wall-clock time per episodeMaximum online VLM calls allowed for per episodeMean online VLM calls observed per episodePlanner API cost per episode (USD)
End-to-end VLA (Xiaomi Robotics 1)45%5.65 mins———
Harness VLA (GPT-6.1 Sol high) + Xiaomi Robotics 121%21.42 mins100712.08
Our full system (GPT-6.1 Sol high) + Xiaomi Robotics 156%18.33 mins100491.78

With the same planner and executor, our full system improved success by 35 percentage points over Harness VLA, reduced mean wall-clock time by 14.4%, and made 31.0% fewer online VLM calls. Compared with the released end-to-end policy, the full system achieved 11 percentage points higher success and took 3.24 times as long per episode.

These results validate that (1) our design works better than Harness VLA, (2) there is still room for optimization to achieve a better success-efficiency trade-off between end-to-end VLA and our system. We believe the latter is promissing as a new paradigm for embodied AI in the near future, in the light of efficient VLM while powerful as GPT-6 models.


We believe that our work explores a promising avenue for embodied AI and deserves an opportunity so that we can convey these messages to the community. If the responses above have addressed your concerns, we would be grateful if you could consider raising your score. Thank you for your time and constructive feedback!

AuthorsOpenReview ↗

Response to Reviewer 91XX

W2-3 & Q4. BEHAVIOR-1K coverage and generalization. Can the evaluation be expanded to cover more BEHAVIOR-1K tasks? How well does the framework generalize to unseen instructions in familiar environments and to entirely unseen tasks

Thank you for suggesting a broader evaluation. We added five BEHAVIOR-1K tasks, bringing the total from five to ten. Our full system outperformed both baselines on each of the five added tasks.

Added BEHAVIOR-1K taskEnd-to-end VLA (π0.5)Harness VLAOur full system
Putting Dishes Away After Cleaning10%20%40%
Loading the Car5%15%40%
Clean Boxing Gloves10%25%45%
Tidying Bathroom0%5%35%
Sorting Books on Shelf5%10%50%
Mean over the five added tasks6%15%42%

The full system outperformed both baselines on all five additional BEHAVIOR-1K tasks and in the real-robot experiments, as reported in our response to Reviewer oetU's W2 & Q2. This provides evidence that our agentic design transfers beyond the five tasks in the original evaluation, using low-level policies trained on BEHAVIOR-1K data. Limited GPU availability constrained how many additional tasks we could evaluate during the rebuttal period. We would consider adding more tasks in the future.

We believe that our work explores a promising avenue for embodied AI and deserves an opportunity so that we can convey these messages to the community. If the responses above have addressed your concerns, we would be grateful if you could consider raising your score. Thank you for your time and constructive feedback!

I appreciate the author's responses in the rebuttal. Howerver, I still think that the work lacks sufficient novelty. The proposed approach relies on traditional or exsiting methods, with limited new insights or fundamental contributions. I do think agentic system is a certain promising field, and an efficient system with generalizations ability is the goal. So based on this consideration, I keep the score.

Reviewer_oetUOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates why a subtask VLA with 80% atomic-task success achieves only 19% in long-horizon mobile manipulation. It identifies three failure regimes that contains task planning, navigation/base positioning, and manipulation. It augments the fixed system with demonstration-based guidance, adaptive visual memory, and base/end-effector adjustments. On RoboCasa365, success increases from 19% to 47%. Additional evaluations report improvements from 14% to 37% on BEHAVIOR-1K and from 5/20 to 13/20 on real-robot tasks. These results suggest that diagnosing and targeting execution bottlenecks can improve mobile-manipulation reliability without modifying the underlying policy.

Critical strengths
  • The paper has a clear motivation: understanding why strong subtask-level capabilities do not translate into reliable long-horizon mobile manipulation, and the failure analysis provides a concrete rationale for system-level interventions targeting planning, navigation/base positioning, and local manipulation.
  • The framework is evaluated on RoboCasa365, BEHAVIOR-1K, and real-robot tasks, showing consistent performance improvements across these settings. The progressive integration and ablations help clarify the observed contributions of the skill library, adaptive visual memory, and base and end-effector adjustments.
Critical weaknesses
  • The additional computational overhead is not adequately analyzed. Building on the Harness VLA framework, the authors introduce Gemini 3.5 Flash for skill card and Qwen3.8-27B for visual-memory retrieval and summarization. However, the paper does not quantify the api costs or the additional inference time and budget required during system execution. Despite substantial improvements in task success over the baseline, the absence of this analysis makes it difficult to assess the system’s practical efficiency and cost-effectiveness.
  • Limited real-robot evaluation. The real-robot evaluation covers only two tasks, with 10 trials per method per task. This small number of trials makes the reported success rates sensitive to random variation, limiting the persuasiveness of the real-world results.

Although the system design is motivated by failure analysis, the paper is ultimately a study of robotic mobile manipulation system. In my view, the detailed failure analysis should primarily motivate and justify the system design, rather than be presented as a core contribution in its own right. The manuscript should place greater emphasis on the concrete system design. If accepted, I recommend revising its organization accordingly.

Questions for authors
  • Could the authors add experiments comparing the runtime and computational/api costs of the full system with the Harness VLA baseline?
  • Could the authors evaluate a broader range of real-robot tasks and increase the number of trials to provide more robust empirical support?
1 comment
AuthorsOpenReview ↗

Response to Reviewer oetU

W1 & Q1. With the added skill-card and visual-memory modules, how does the full system compare with Harness VLA in runtime, compute, and API costs?

On the held-out RoboCasa365 episodes, our full system with GPT-6.1 Sol high achieved 52% success with our original subtask VLA and 56% with Xiaomi Robotics 1 fine-tuned to follow subtask instructions.

We compared three Xiaomi Robotics 1 configurations on the matched episodes. Harness VLA and our full system used the same subtask-adapted checkpoint, the same GPT-6.1 Sol high planner, and a limit of 100 online VLM calls per episode. The end-to-end baseline ran the released checkpoint directly, with zero calls to a separate online VLM.

ModelHeld-out successMean wall-clock time per episodeMaximum online VLM calls allowed for per episodeMean online VLM calls observed per episodePlanner API cost per episode (USD)
End-to-end VLA (Xiaomi Robotics 1)45%5.65 mins———
Harness VLA (GPT-6.1 Sol high) + Xiaomi Robotics 121%21.42 mins100712.08
Our full system (GPT-6.1 Sol high) + Xiaomi Robotics 156%18.33 mins100491.78

With the same planner and executor, our full system improved success by 35 percentage points over Harness VLA, reduced mean wall-clock time by 14.4%, and made 31.0% fewer online VLM calls. Compared with the released end-to-end policy, the full system achieved 11 percentage points higher success and took 3.24 times as long per episode.

These results validate that (1) our design works better than Harness VLA, (2) there is still room for optimization to achieve a better success-efficiency trade-off between end-to-end VLA and our system. We believe the latter is promissing as a new paradigm for embodied AI in the near future, in the light of efficient VLM while powerful as GPT-6 models.


W2 & Q2. Can the real-robot evaluation cover more than two tasks and increase the number of trials beyond 10 per method per task?

Despite the limited rebuttal period, we made every effort to address the reviewer's request by conducting two additional experiments: Plate Retrieval and Table Cleaning, with 10 trials per method on each task. This expands the evaluation in Table 18 from two to four tasks and from 20 to 40 trials per method.

MethodPlate RetrievalTable CleaningTotal on the new tasks
End-to-end VLA (π0.5)3/102/105/20
Agentic baseline (Harness VLA)4/104/108/20
Our full system9/108/1017/20

Across all four tasks, the full system succeeded in 30/40 trials (75%), compared with 13/40 (32.5%) for the agentic baseline and 9/40 (22.5%) for end-to-end π0.5. The full system outperformed both baselines on each task. With 10 trials per method per task, the individual success rates remain uncertain.


We believe that our work explores a promising avenue for embodied AI and deserves an opportunity so that we can convey these messages to the community. If the responses above have addressed your concerns, we would be grateful if you could consider raising your score. Thank you for your time and constructive feedback!

Reviewer_jnBhOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper claims two contributions: a failure diagnosis of long-horizon mobile manipulation, and a framework of interventions guided by that diagnosis. It concludes that diagnosing the current bottleneck is a better route to reliable mobile manipulation than scaling the policy. A subtask VLA with 80% atomic-task success reaches only 19% on the 20 mobile-manipulation tasks of RoboCasa365 when driven by an LLM planner. Labeling the first unrecovered error in 81 failed episodes gives three regimes: planning, navigation/base pose, and manipulation. The authors add one tool per regime: same-task demonstrations summarized as skill cards, VLM-queried visual memory, a distance-threshold reachability check with base repositioning, and an RGB-D end-effector correction. With the VLA frozen, success rises from 19% to 47% (held-out seeds: 18% to 49%), from 14% to 37% on five BEHAVIOR-1K tasks, and from 5/20 to 13/20 on two real-robot tasks.

Critical strengths
  1. The evaluation protocol is stronger than usual for agentic robot systems. The design was frozen before the held-out seeds were run, the tests are paired, and episode-level transition tables (Tables 7 and 15) show which failures each component fixes. The held-out gain (18% to 49%, p<0.001, Table 14) is credible.
  2. The gains hold beyond the development benchmark without retraining the VLA. The same frozen-VLA system improves BEHAVIOR-1K (14% to 37%, Table 11) and two real-robot tasks (5/20 to 13/20). This makes it a strong, reproducible systems baseline for VLA-based mobile manipulation, even though the individual findings are not new.
Critical weaknesses

In short: the paper's findings are already established in the mobile-manipulation literature. Its interventions are standard modular components, and the full system only ties a scaled end-to-end policy. It does not show what is new.

  1. The central findings restate known results, and the paper does not engage with the work that established them. The paper presents three findings as its contribution: strong atomic skills fail when composed, the navigation-to-manipulation base pose is a major bottleneck, and failures split across planning, navigation and manipulation. All three are documented.

    • Composition and hand-off: Habitat 2.0 (Szot et al., NeurIPS 2021) found that hierarchies of independently trained skills suffer "hand-off problems." Adversarial skill chaining (Lee et al., CoRL 2021) attributes chained-skill failure to terminal states falling outside the next skill's initiation set.
    • Base pose: Gu et al. (ICLR 2023, "Multi-skill Mobile Manipulation for Object Rearrangement") identify the same failure in the same pipeline structure: chaining stationary skills with point-goal navigation compounds errors, with the robot "navigating to a bad location where a stationary manipulation skill can not reach its target." Their fix, region-goal navigation plus mobile skills, addresses what Sec. 4.2 addresses. Base placement for manipulation is itself an established problem: Action-Related Places (Stulp et al., JAIR 2012) and inverse reachability maps (Vahrenkamp et al., ICRA 2013). The proposed check is a 0.75/1.00 m distance threshold, which the paper says ignores IK, joint limits and collisions (Sec. 4.2).
    • Module-level failure breakdowns: HomeRobot and OK-Robot, both cited, already report them for modular mobile manipulation. The paper cites these as systems, not as prior findings.

    None of Habitat 2.0, Lee et al., Gu et al., ARPlace or reachability maps is cited, so the paper never states what it adds beyond confirming them with a VLA executor. Needed: Position the diagnosis against this literature, and identify a finding specific to VLA-based executors that the earlier modular and RL work did not show. Compare the base-adjustment component against a reachability-based or learned base-placement method (or M3-style region-goal navigation) rather than against no adjustment.

  2. Because the components are standard, the contribution has to be the diagnose-then-intervene method, and that method is not tested beyond one system. The full system (47/49%) only ties Xiaomi Robotics 1 with no agentic layer (44/45%, Table 1), so the paper's closing claim that diagnosis beats scaling is not shown. Two of four components are not statistically supported: memory has p=0.267 on held-out leave-one-out, and the skill library and EEF adjustment reach p<0.05 only once across Tables 13–14. The skill card is also retrieved from a training demonstration of the same task, including its ordered subtask plan (Sec. 4.1, H.1). The drop in planning failures (17 to 1) may therefore come from supplying the answer. Needed: Apply the same loop to a stronger executor (e.g., Xiaomi Robotics 1 or π0.5) and show that the gains add to scaling. Retrieve skill cards only from other tasks.

Questions for authors
  1. What does this paper show about composing VLA skills that Gu et al. (ICLR 2023) and Habitat 2.0 did not already show for composing RL or modular skills?
  2. How does the 0.75/1.00 m distance check compare with a reachability-map or learned base-placement method on the same 81 failure episodes?
  3. How does the full system do when the skill card is retrieved only from a different RoboCasa365 task?
  4. Does the framework improve a stronger executor (Xiaomi Robotics 1 as the subtask policy), and with what wall-clock time and planner/VLM call budget relative to the end-to-end baselines?
3 comments
AuthorsOpenReview ↗

Response to Reviewer jnBh

W1 & Q1. Module-level failure breakdowns: HomeRobot and OK-Robot already report module-level failure breakdowns for modular mobile manipulation. The paper cites them as systems rather than as prior findings, leaving unclear what is added beyond confirming known results with a VLA executor. What does the failure analysis reveal about VLA skill composition beyond prior work on RL and modular skills, including Gu et al., Habitat 2.0, and adversarial skill chaining?

Thank you for pointing us to these papers. Habitat 2.0 [1] and Lee et al. [2] showed that chaining skills would fail, and M3 [3] showed how a poor stopping position would make the next manipulation harder. We will add these citations and discuss the failure analyses in HomeRobot [4] and OK-Robot [5].

However, the experiments in Habitat 2.0 [1] and M3 [3] provide the subtask order in advance for three HAB task families. This is fundamentally different from what we have done. In our system, a VLM agent decides what to do as the task unfolds. It leverages what the robot sees at current stage and the results of previous actions, so as to choose a target, pick the next step, or retry. It can also recall earlier views to check progress. We evaluate 20 RoboCasa365 [6] tasks with 2–15 subtasks. In the task of PackIdenticalLunches, for example, the agent has to keep track of which container still lacks each ingredient.

Moreover, our analysis covers the agent’s planning decisions as well as the robot’s actions. For each failed run, we identify the first error that contributed to the failure and was never corrected. Such analysis lets us distinguish planning errors, such as choosing the wrong container or doing steps in the wrong order, from navigation or base-positioning errors and failures to carry out the intended manipulation. These contents are also different from previous works.

  1. Habitat 2.0: Training Home Assistants to Rearrange their Habitat. NeurIPS, 2021.
  2. Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization. CoRL, 2022.
  3. Multi-skill Mobile Manipulation for Object Rearrangement. ICLR, 2023.
  4. HomeRobot: Open-Vocabulary Mobile Manipulation. CoRL, 2023.
  5. Demonstrating OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics. RSS, 2024
  6. RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots. ICLR, 2026.

W2 & Q3. How does the full system perform when skill cards come only from other RoboCasa365 tasks, so the cards cannot supply a same-task subtask plan?

The demonstration library and the evaluation set have disjoint RoboCasa365 task IDs. We use the evaluation instruction to retrieve a similar demonstration from the training sets. We will make this task-level split explicit in Sec. 4.1 and App. H.1. We compared two demonstration sources on the same 100 development episodes. The system achieved 47% with cards from the RoboCasa365 training library and 40% with cards from HIW-500, a dataset of real-world teleoperated humanoid demonstrations (Tables 2 and 16).


W2 & Q4-1. With Xiaomi Robotics 1 as the subtask policy, does the framework improve success beyond using Xiaomi Robotics 1 alone?

As requested by the reviewer, we performed additional experiments. Specifically, we evaluated our agentic framework with an additional subtask VLA on held-out RoboCasa365 episodes. Both configurations used GPT‑6.1 Sol with high reasoning effort as the agent planner. The framework achieved 52% success rate with our original subtask VLA and 56% with Xiaomi Robotics 1 fine-tuned to follow subtask instructions. These results suggest that a stronger VLA can benefit our overall agentic framework.

But this is not our focus to obtain a strong VLA. Instead, our focus lies at that the diagnosis can sucessfully guide the design of an agentic framework, even when each module only achieves moderate performance.


Q2. On the same 81 failure episodes, how does the 0.75/1.00 m distance-only check compare with reachability-map or learned base placement?

Thank you for suggesting a direct comparison with reachability-based and learned base placement. Originally, we used the 0.75/1.00 m distance thresholds as a low-cost way to test whether adjusting the robot’s stance before manipulation could help the same VLA executor. For the follow-up comparison, we evaluated both alternatives on the RoboCasa365 development set used for failure diagnosis, with the same episode seeds as in the original distance-threshold run. The simple inverse kinematics (IK) method checks whether the arm can reach the estimated interaction target from a candidate base pose. The learned method ranks candidate poses using the next subtask and target-relative RGB-D features. Across these matched episodes, five failures were attributed to navigation and base positioning with the original distance thresholds, two with simple IK, and one with learned base placement.

AuthorsOpenReview ↗

Response to Reviewer jnBh

Q4-2. How do wall-clock time and planner/VLM call budgets with the stronger executor compare with the end-to-end baselines?

On the held-out RoboCasa365 episodes, our full system with GPT-6.1 Sol high achieved 52% success with our original subtask VLA and 56% with Xiaomi Robotics 1 fine-tuned to follow subtask instructions.

We compared three Xiaomi Robotics 1 configurations on the matched episodes. Harness VLA and our full system used the same subtask-adapted checkpoint, the same GPT-6.1 Sol high planner, and a limit of 100 online VLM calls per episode. The end-to-end baseline ran the released checkpoint directly, with zero calls to a separate online VLM.

ModelHeld-out successMean wall-clock time per episodeMaximum online VLM calls allowed for per episodeMean online VLM calls observed per episodePlanner API cost per episode (USD)
End-to-end VLA (Xiaomi Robotics 1)45%5.65 mins———
Harness VLA (GPT-6.1 Sol high) + Xiaomi Robotics 121%21.42 mins100712.08
Our full system (GPT-6.1 Sol high) + Xiaomi Robotics 156%18.33 mins100491.78

With the same planner and executor, our full system improved success by 35 percentage points over Harness VLA, reduced mean wall-clock time by 14.4%, and made 31.0% fewer online VLM calls. Compared with the released end-to-end policy, the full system achieved 11 percentage points higher success and took 3.24 times as long per episode.

These results validate that (1) our design works better than Harness VLA, (2) there is still room for optimization to achieve a better success-efficiency trade-off between end-to-end VLA and our system. We believe the latter is promissing as a new paradigm for embodied AI in the near future, in the light of efficient VLM while powerful as GPT-6 models.


We believe that our work explores a promising avenue for embodied AI and deserves an opportunity so that we can convey these messages to the community. If the responses above have addressed your concerns, we would be grateful if you could consider raising your score. Thank you for your time and constructive feedback!

Thank you for the rebuttal and for agreeing to cite Habitat 2.0, Lee et al., and M3 and to discuss the failure analyses in HomeRobot and OK-Robot. This addresses the missing citations, but my central concern from the review remains. The paper does not identify a finding specific to VLA-based executors beyond what this prior work established.

The distinction drawn in the rebuttal, that a VLM agent decides the subtask sequence online instead of receiving it in advance, is a real difference in setup. However, it is weakened by the method itself, since the drop in planning failures comes from skill cards that supply an ordered subtask plan from a demonstration. On base placement, the new comparison shows that a simple IK check and learned placement both outperform the distance threshold. This is consistent with my original point that this component is a simpler version of established base-placement methods (ARPlace, inverse reachability maps).

More broadly, skill composition, base placement, and module-level failure analysis in mobile manipulation have a long history in robotics beyond the papers I listed. I'd encourage the authors to frame the related work with that in mind, and to state clearly what the VLA setting adds.

I appreciate the additional experiments, but because the novelty concern is the basis of my score, I am keeping it.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the Rebuttal.

We sincerely thank Reviewers jnBh, oetU, and 91XX for their careful reviews and constructive suggestions. We appreciate jnBh’s recognition of our rigorous evaluation protocol, oetU’s acknowledgment of the clear motivation and results across evaluation settings, and 91XX’s recognition of the practical system design and consistent performance improvements.

We would like to clarify that our core contribution is methodological: diagnosing bottlenecks in modern VLM–VLA systems under a unified benchmarking protocol, and demonstrating that targeted agentic interventions can effectively compensate for these shortcomings. This connects systematic analysis to concrete design choices and provides empirical guidance for future agentic mobile manipulation.

The long-horizon tasks we study require more than executing a prescribed subtask sequence. The robot must remember earlier observations, track partially completed goals, and revise its plan when actions fail or outcomes change. Consequently, the current observation alone may be insufficient to determine the next action. Our framework brings together the reasoning and tool-use capabilities of a VLM with the sensorimotor capabilities of a VLA, coordinating memory, planning, and corrective actions through feedback. While memory and replanning have established precedents, our study examines how these capabilities should be organized around current foundation models and which interventions actually improve task completion under controlled conditions.

The value lies in demonstrably mitigating system bottlenecks, without requiring every module to be individually state of the art. The lightweight base-adjustment module illustrates this principle: a simple geometric intervention can be directly incorporated into the agent and deliver meaningful gains with modest computational overhead. More sophisticated modules can further improve the framework, as our additional comparisons indicate. Developing and exhaustively optimizing every component would require substantial additional research and computation beyond this study’s scope.

Our detailed responses address the reviewers’ concerns:

  • Reviewer jnBh: We acknowledge insufficient coverage of prior work on skill chaining, base placement, and modular failure analysis, provide the relevant references in our response, and will expand the manuscript’s discussion accordingly. We also clarify demonstration retrieval and add comparisons using a stronger executor and alternative base-placement methods.
  • Reviewer oetU: We add runtime and API-cost analyses and broaden the real-robot evaluation.
  • Reviewer 91XX: We clarify how offline diagnosis guides system design and how the agent selects interventions online, examine the limitations of the reachability heuristic, and expand the BEHAVIOR-1K evaluation.

We thank the reviewers again and respectfully ask them to reassess the paper in light of this clarified methodological contribution and the additional evidence. We believe that our work explores a promising avenue for embodied AI and deserves an opportunity so that we can convey these messages to the community. If the responses above have addressed your concerns, we would be grateful if you could consider raising your score. Thank you for your time and constructive feedback!

Best,
The Authors

Dear Reviewers,

Thank you for your time and thoughtful feedback. We would appreciate it if you could take a look at our responses and the updates to the manuscript when you have a chance. We would be very happy to discuss any remaining concerns and provide further clarification. Thank you again for your consideration

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.