acceptodds
Under review as a conference paper at ICLR 2027

Decoupling Planning and Execution: A Cerebrum-Cerebellum Framework for Robust Visual-Language-Action Models

Abstract

Visual-Language-Action (VLA) models have demonstrated remarkable capabilities in end-to-end robotic manipulation by integrating multimodal information. However, these models frequently degenerate into static vision-action mappings, resulting in a critical failure to adhere to language instructions under complex constraints or target changes. We identify that this limitation stems from an inherent modality imbalance where physical actions correlate more strongly with visual states than with abstract linguistic descriptions. Consequently, models spontaneously develop a "vision-dominated bias" that marginalizes linguistic guidance during training. To address this, we propose a bio-inspired "Cerebrum-Cerebellum" architecture to structurally decouple task planning from execution. The language-dominant Cerebrum Module functions as a high-level planner to parse instructions into structured sub-tasks and tokens with visual information, while the vision-dominant Cerebellum Module executes these plans with real-time feedback. Across LIBERO-Plus and RoboTwin 2.0, our method improves instruction-following robustness, outperforming by 5.28 and 9.20 percentage points, respectively.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Visual-Language-Action Model, Embodied AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aL7QOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a hierarchical VLA framework to improve instruction following. A Qwen-VL planner produces structured manipulation plans augmented with continuous segmentation and depth tokens, which condition a π0.5-based executor. Auxiliary reconstruction encourages the executor to retain the plan’s geometric information, while a progress predictor triggers replanning. The paper reports improvements on LIBERO-Plus, RoboTwin 2.0, target-replacement evaluation, and a small real-world benchmark.

Critical strengths
  1. The paper addresses an important limitation of current VLAs: successful manipulation does not necessarily imply sensitivity to language instructions. The target-replacement evaluation is particularly relevant, with a reported improvement from 55% to 74% over π0.5.
  2. The proposed combination is technically plausible and reports improvements across several settings, including LIBERO-Plus, bimanual manipulation, and real-world distribution shifts. Training the executor with generated rather than ground-truth plans is also a sensible design choice.
Critical weaknesses
  1. The contribution of the plan interface is unclear. The system adds a planner, geometric supervision, reconstruction, and replanning, but Table 4 does not separate their effects well. Reconstruction also does not guarantee that the action head uses the plan.
  2. Important training and evaluation details are missing. The paper needs to explain the supervision sources, data splits, and training budgets. The target-replacement protocol and Table 1’s aggregation are also unclear.
Questions for authors
  1. How are structured plans and segmentation/depth labels generated for simulation and real-world data? Please specify the datasets, quantities, splits, and training budgets for both stages, including whether replacement-target instructions appear in planner supervision.
  2. How does the full model compare with the same planner and executor using a text-only plan interface, while retaining reconstruction and replanning? Results on target replacement, together with interventions that change or remove plan information while holding observations fixed, would help establish which components actually drive instruction following.
  3. Please provide the target-replacement task/instruction pairs, success criteria, rollout counts, and seeds. Please also explain Table 1’s aggregation: using the task counts published in LIBERO-Plus Appendix C.2, Table 7, I obtain approximately 81.96 for π0 and 89.85 for the proposed method, rather than 82.94 and 89.72. Were different task subsets or aggregation rules used?
2 comments
AuthorsOpenReview ↗

Response to Reviewer aL7Q

Response to Question 1: In simulation, ground-truth masks are fetched directly from the simulator API, and structured plans are annotated manually. For real-world data, masks and depth labels are annotated using DepthAnything, GroundingDINO and Segment Anything. Simulation and real-world models are trained independently on their respective datasets with no mixed training.

Response to Question 2: We considered the proposed text-only baseline while keeping reconstruction and replanning unchanged. However, this comparison is not meaningful. When execution stalls at a sub-task, only object positions and visual tokens change, while the text plan content stays the same. Replanning cannot yield useful updates in this case. We will also clarify our novelty as the structured geometric interface that binds planner semantics to geometric masks for valid replanning, and remove the unsupported “mathematically enforces” statement in Section 3.3.

Response to Question 3: For target-replacement, an example instruction pair is “put the bowl on the plate” and “put the wine bottle on the plate”. We follow the official LIBERO success criteria: an episode is successful if all goal predicates are satisfied, verified by the simulator internal state. We evaluate 10 tasks with 10 rollouts per task. Regarding Table 1 aggregation, this was indeed our mistake. After checking our code, we found that we computed the simple arithmetic mean. We will fix this, and this issue does not affect our core results.

Thanks for the clarifications. Some of my concerns have been addressed, but I still think the text-only baseline is worth testing. Since the executor receives updated visual observations, an unchanged text plan does not necessarily prevent it from adapting. Without this comparison, it is hard to tell how much the geometric interface actually contributes.I will keep my rating unchanged as 2 (Weak Rejection).

Reviewer_ZCjqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper pairs a Qwen-VL planner with a pi0.5 executor to improve instruction following. The planner produces manipulation primitives and visual tokens for segmentation and depth. The executor is trained with an auxiliary reconstruction loss, and a progress head triggers replanning. Reported success rates exceed pi0.5 by 5.28 percentage points on LIBERO-Plus and 9.20 on RoboTwin 2.0.

Critical strengths
  1. Changing target instructions while keeping the scene fixed tests whether the policy follows language. Success increases from 55% for pi0.5 to 74%.
  2. Gains appear on both simulation benchmarks and in the small UR5E study, where success reaches 80% versus 58.3% for pi0.5.
  3. Decoding masks and depth from the visual tokens makes the planner's object choices and spatial estimates inspectable.
  4. The executor reuses plans while receiving fresh observations at each control step, reducing the frequency of calls to the large planner.
Critical weaknesses
  1. Section 3.3 overstates what reconstruction guarantees. The fused features can retain plan information for the reconstruction head while the action head relies on vision alone. The reconstruction loss permits this solution and therefore cannot provide the claimed mathematical guarantee that actions follow the plan.
  2. Equation (3) trains the progress head to predict normalized time in successful demonstrations. Replanning requires a drop greater than 0.3 within 50 frames. If a grasp repeatedly fails while predicted progress stays at 0.5, the rule will never trigger. The paper does not explain how such stalls are handled or why demonstration timing is a reliable failure signal.
  3. Equation (2) does not specify the constraints on Hungarian matching. If assignment is unrestricted, exchanging the predicted pick-target and place-target masks leaves the optimal segmentation loss unchanged. A low loss would then allow a visual token to encode the wrong object for its role, even when the plan text is correct.
  4. The reconstruction head recovers masks and depth from the latest planning frame, while actions use current observations. Successful grasping and transport change object positions without necessarily triggering replanning. How the cached masks and depth are associated with objects at their new positions is not explained.
Questions for authors
  1. With observations fixed, how do actions change when the plan or target tokens change? What evidence links this response to the reconstruction loss?
  2. What triggers recovery when execution stalls but predicted progress stays constant? How is normalized demonstration time validated as a failure signal?
  3. Does Hungarian matching restrict assignments by semantic role or object identity? Please give the matching cost and allowed assignments.
  4. Between replanning calls, how does the executor relate cached geometry to current object positions while reconstructing masks and depth from the planning frame? 5.My main concerns are whether reconstruction makes the action head use the plan and whether the progress signal can detect execution failures
2 comments
AuthorsOpenReview ↗

Response to Reviewer ZCjq

Response to Weakness 1 & Question 1: We acknowledge that our original wording “mathematically enforcing that the execution policy is conditioned on the plan” was overstated. The reconstruction auxiliary loss regularizes the fused representation hth_t to encode geometric information from the Cerebrum plan y. Our ablation study shows that removing this reconstruction loss leads to a notable drop in task success rate, which supports that this representation alignment objective helps the policy utilize plan information. We will revise Section 3.3 to remove the overclaim.

Response to Weakness 2 & Question 2: Our progress head is trained using normalized timestamps from successful demonstrations as a dense surrogate of task progress. During training, we filter out zero-action frames from demonstrations, so the learned policy rarely produces persistent stationary behavior on its own; algorithmic stagnation is uncommon in simulation rollouts. For repeated grasp failures, each failed attempt alters the observed scene, including object pose, contact state and occlusion. These scene variations are captured by the visual stream and fused representation hth_t, gradually reducing the predicted progress. Therefore, repeated failed grasps will eventually produce a detectable progress drop that satisfies our replanning rule (progress drop >0.3>0.3 within a 50-frame window). Nearly all policy-relevant failures in our LIBERO-Plus evaluations, such as object dropping, collision-induced drift and target misalignment, induce task regression and reduce predicted progress, making this replanning condition reliable for dominant failure cases.

Response to Weakness 3 & Question 3: The Hungarian matching in Eq. (2) is semantic-role-aware, and assignments are not unrestricted. We predefine two disjoint role slots: pick-target and place-target. The matching cost matrix only allows a predicted mask to be matched to a ground-truth mask with the same semantic role.

Response to Weakness 4 & Question 4: The cached masks and depth decoded from the planning frame remain fixed until replanning is invoked. During normal task execution, once the object is grasped, it moves synchronously with the gripper, and its cached mask contributes little to action generation. The place-target mask serves as the dominant geometric cue guiding placement. When the grasped object no longer moves together with the gripper, this usually indicates task failure, which triggers replanning to refresh the masks.

Thank you for the clarifications on reconstruction and role-constrained matching. However, failure detection and the use of cached geometry remain insufficiently supported. I maintain my original rating.

Reviewer_FDjbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Cerebrum-Cerebellum framework that separates high-level task planning from low-level action execution in VLA models. The Cerebrum module generates structured task plans with visual tokens, while the Cerebellum module leverages a pretrained π0.5\pi_{0.5} backbone to generate robot actions. The framework also incorporates a value-based replanning mechanism to improve execution robustness. Experiments on LIBERO, LIBERO-Plus, and RoboTwin 2.0 demonstrate promising improvements over existing VLA baselines.

Critical strengths
  1. The paper addresses the vision-dominated bias in VLA models by decoupling language-driven planning from vision-based action execution.

  2. The proposed method demonstrates promising performance across multiple benchmarks, supported by ablation studies and real-world robot experiments.

Critical weaknesses
  1. The proposed Cerebrum-Cerebellum framework adopts a hierarchical VLA architecture. However, the paper primarily compares against end-to-end VLA models. Comparisons with existing hierarchical approaches, such as Hi Robot, RT-H, and CoT-VLA, would better demonstrate the advantages of the proposed framework.

  2. The ablation in Section 4.3 is insufficient to demonstrate whether the performance gains primarily come from decoupling high-level planning and low-level execution or the additional visual supervision and replanning mechanisms. The performance differences across different settings are relatively modest.

Questions for authors
  1. The framework incorporates a 7B Qwen model for high-level planning, which may introduce computational overhead. Could the authors provide an efficiency analysis, including inference latency and computational cost?

  2. I am confused about why the value head in Section 3.3 uses predicted task progress to trigger replanning. A decrease in progress does not necessarily indicate execution failure. For example, if the robot gets stuck during grasping, the predicted progress might remain unchanged rather than decrease. Could the authors clarify how this mechanism reliably detects execution failures?

1 comment
AuthorsOpenReview ↗

Response to Reviewer FDjb

Response to Weakness 1: Thank you for pointing out the important hierarchical VLA baselines, including Hi‑Robot, RT‑H and CoT‑VLA. We follow your suggestion and clarify comparisons below. For Hi‑Robot and RT‑H: Both are representative hierarchical VLA systems, yet they do not report results on public benchmarks, which makes it difficult for us to compare them on public benchmarks. We provide a qualitative comparison between these two methods and our framework here. Hi‑Robot and RT‑H adopt pure natural‑language as their only hierarchical intermediate interface, without explicit geometric outputs such as segmentation masks or depth maps. For CoT‑VLA‑7B, which supports the original LIBERO simulation suite, the quantitative comparison on the original LIBERO benchmark is shown in the table below. CoT‑VLA‑7B obtains 81.13±0.681.13\pm0.6% average success rate. Our cerebrum‑cerebellum framework achieves 97.8% average success rate, outperforming CoT‑VLA‑7B by 16.7% overall. The most prominent gain appears in long‑horizon tasks, with an absolute improvement of 26.5%. Unlike CoT‑VLA, which does not explicitly separate planning from execution or adopt closed‑loop replanning, our two‑module design decouples language‑dominant geometric planning (producing interpretable segmentation‑depth‑aware visual tokens) from high‑frequency low‑level execution. The value‑head‑driven re‑planning loop mitigates compounding execution errors, which yields huge improvements for long‑horizon manipulation. We note CoT‑VLA achieves competitive object‑level perception performance. Nevertheless, our system demonstrates stronger overall performance across all four LIBERO suites, especially for spatial reasoning and long‑horizon task execution.

MethodSpatialObjectGoalLongAvg
CoT-VLA-7B87.5±1.487.5\pm1.491.6±0.591.6\pm0.587.6±0.687.6\pm0.669.0±0.869.0\pm0.881.13±0.681.13\pm0.6
Ours98.898.798.295.597.8
Table 1: Success rate comparison on LIBERO benchmark (%).

Response to Weakness 2: The ablation results in Table 4 are reported on LIBERO-Plus. From Table 4, each component targets distinct failure modes and brings tangible gains for particular task categories:

  1. Decomposition (planning‑execution decoupling) mainly benefits goal‑oriented tasks, lifting the Goal subset from 81.52% to 84.49%. It helps parse ambiguous language instructions into structured subtasks, yet brings limited improvement for long‑horizon scenarios.
  2. Reconstruction Head (auxiliary geometric supervision) provides considerable gains on Goal (+4.10%) and Spatial (+2.59%) tasks. By regularizing the executor to preserve plan‑derived geometric cues, it alleviates vision‑dominated shortcut and strengthens spatial grounding, while offering marginal benefit for long‑horizon drift.
  3. Value‑Head & Replanning is most effective for long‑horizon tasks, bringing an absolute improvement of 3.07% (80.23% to 83.30%). The replanning mechanism corrects compounding execution drift, but cannot fix poor initial language‑geometry grounding for goal‑centric tasks. No single component can resolve all failure modes alone. The substantial overall performance jump emerges from strong synergy when all modules work together: Goal rises from 81.52% to 90.43%, Spatial from 90.09% to 94.42%, Long‑horizon from 80.23% to 84.04%, and Object from 85.90% to 89.99%.

Response to Question 1: A key architectural property of our framework is sparse invocation of the Cerebrum. The Cerebrum is executed only once at episode initialization and re‑invoked solely when replanning conditions are satisfied by the value‑head signal. It is never executed at every high‑frequency control step. The heavy‑weight 7B reasoning cost is therefore amortized over the full episode, rather than added to every action step. As shown in the appendix, the high control throughput in Figure 3 supports this argument.

Response to Question 2: We filter out zero-action frames from our training demonstrations. As a result, the learned policy itself rarely produces persistent stationary behavior; algorithm‑level policy stagnation is uncommon in simulation rollouts. Hardware‑level robot jamming, by contrast, originates from mechanical faults rather than policy mis‑execution. Such physical hardware failures cannot be recovered by our software policy loop and require manual environment reset, which is out‑of‑scope for our replanning mechanism. In our LIBERO‑Plus simulation evaluations, nearly all policy‑relevant failures arise from events such as object dropping, positional drift after collision, and target misalignment. These events cause observable task regression, which leads to a sharp drop in the predicted progress value. Under these dominant failure modes, our replanning condition (progress drop >0.3 within 50‑frame window) can fire reliably.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.