acceptodds
Withdrawn from ICLR 2027

SCULPT-VLA: Learning Structured Control through Staged Action Grounding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language-action (VLA) policies increasingly incorporate structured intermediate supervision beyond action labels. Yet specifying what an intermediate representation should encode leaves open how action prediction learns to depend on it. We introduce SCULPT-VLA, a policy that learns structured control through staged action grounding. Its action-conditioning state comprises complementary factors for task progression, scene dynamics, and spatial grounding. Training first forms these factors with teacher scaffolds, then grounds coarse action prediction through their composition as scaffold inputs are withdrawn. Direct perceptual access is subsequently restored for continuous refinement, combining the learned state with perceptual detail. The curriculum separates learning to condition actions on structure from refining continuous control. Deployment requires neither teachers nor discrete-action autoregression. SCULPT-VLA achieves higher average success than shared-backbone baselines on LIBERO, SimplerEnv-WidowX, and RoboTwin 2.0 Full. On SimplerEnv-WidowX, final success is 83.5%, versus 71.3% when Stage-II action learning directly accesses vision and language. Across four physical robot tasks, average success under the tested distribution shifts reaches 58.1%, compared with 45.6% for π0.5. Training ablations and factor-wise interventions support the staged design and show that the learned state continues to contribute to control after direct perceptual access is restored.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Robot Learning, Structured Control, Action Grounding, Robot Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XrwXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks how structured intermediate states can become useful for robot control when an action decoder can bypass them. SCULPT-VLA learns task-progression, scene-dynamics, and spatial-grounding factors from frozen teachers. A three-stage curriculum first aligns the factors, then trains coarse actions through them while restricting direct perceptual access. It finally restores perception for continuous control. The main strength is this explicit transition from learning the factors to using them. Deployment requires neither the teachers nor autoregressive discrete-action decoding. However, exact training schedules and the semantic accuracy of generated annotations remain insufficiently documented. The overall performance is ok. On SimplerEnv-WidowX, success reaches 83.5%, compared with 71.3% when direct perceptual access remains available during the middle stage. Evaluation also covers LIBERO, RoboTwin, and real robots. The results support the staged design, although reproducing it requires more complete implementation details.

Critical strengths

• A direct test of the central idea. The perceptual-access ablation shows that the middle-stage restriction improves final control even after perceptual access is restored.

• Strong representation controls. Generic-pool and shared-computation variants test the organized factors beyond removing supervision. The generic-pool control matches teacher targets, token count, and training schedule.

• Broad and complementary evaluation. Multiple simulation settings and physical robots test practical value. Open-loop action agreement and closed-loop success provide consistent supporting evidence. The evaluation scope is ok.

Critical weaknesses
  1. Key training settings are missing. Actual scaffold-annealing steps, numerical loss schedules, LoRA rank, and target modules are unspecified. These settings are important for reproducing the curriculum. It is the most important thing that I concern.

  2. Annotation accuracy remains unquantified. A manual audit was performed, but human-verified instruction and boundary error rates are not reported. Parsing success and model confidence do not establish semantic correctness.

  3. Derived data counts are incomplete. The exact Bridge release and retained segment totals are unclear. Source trajectory counts alone cannot reconstruct the data after segmentation and filtering.

Questions for authors

I still have three questions:

  1. What are the exact annealing schedule, loss weights, LoRA rank, and target modules?

  2. What instruction and boundary error rates were found in the existing manual audit?

  3. Which Bridge release was used, and how many segments remained in each derived dataset?

Reviewer_yi2cOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

SCULPT-VLA gives a Qwen3-VL policy typed slots for task progression, scene dynamics and spatial grounding, trained in three stages that align them to frozen teachers, learn FAST actions from them alone as teacher inputs are annealed out, and refine with flow matching once vision–language access returns alongside a DINOv2 branch. It reports 98.5% on LIBERO, 83.5% on SimplerEnv-WidowX, 92.2% on RoboTwin 2.0 and 58.1% average OOD success on four UR5e tasks against 45.6% for π0.5, and backs these results with SimplerEnv ablations and a study that replaces each factor with its mean.

Critical strengths

Treating how the action head comes to rely on intermediate structure as something training can shape is a fresh angle, and it adds no cost at deployment.

The improvements carry over from three simulation suites to real robots, most clearly on SimplerEnv, where success rises to 83.5% from 65.3% for the strongest shared-backbone head.

Critical weaknesses
  1. Each slot is regressed onto the stop-gradient of its own projected teacher feature, which Eqs. 1 and 3 and Appendix A.1 show is also its Stage-I input, so the objective can be met by passing that input through the residual stream or by collapsing PinP_{in}. With the target branch detached, only the Stage-II FAST loss keeps the projection informative, and since the dynamics scaffold is a UniVLA latent action computed from the current frame and one about a second ahead, the formulation alone does not ensure that the slots carry the intended factors.

  2. DreamVLA, cited only as a predictive policy, already combines dynamic, depth and semantic queries with block-wise attention isolation and a DiT action head, while Spatial Forcing, which reaches 98.5% on LIBERO through REPA-style alignment to VGGT features, and Knowledge Insulation, the FAST-then-action-expert recipe behind π0.5, go uncited. The paper should state precisely what the staged visibility schedule adds beyond these designs and describe it in concrete mechanistic terms instead of appeals to "functional role" and "composition".

  3. The shared-backbone baseline numbers are identical to those in the StarVLA report for Qwen3-VL-4B, namely 31.6, 64.6, 60.9 and 65.3 on SimplerEnv, and the Stage-III-only control at 61.0% is on par with Qwen3-VL-PI at 60.9%, so the headline gain cannot be separated from the extra Stage-I and Stage-II training budget. The statistics are thin as well, because the ablations come from single training runs, the Stage-II effect is absent on LIBERO-Spatial at 99.1% against 98.7%, and all three SimplerEnv seeds give exactly 167 of 200 successes with per-task spreads within one episode, odds near 1.7×10⁻⁵ for independent episodes.

Questions for authors
  1. Were the Qwen3-VL baselines retrained under your pipeline, and could you add a single-stage co-training control with the same backbone, teachers, DINOv2 branch and total number of updates as the three-stage schedule? If SCULPT-VLA keeps a clear margin under matched compute, I would raise my score.

  2. Does anything such as a normalization, a variance term or a fixed projector stop the Stage-I objective from being met by passing the input straight through or by collapsing PinP_{in}, and does early Stage-II FAST learning exploit the future-frame information in the dynamics scaffold before ρ\rho reaches zero? If these cannot be ruled out, how should the slot semantics be interpreted once the scaffold inputs are removed?

  3. What are the backbone size, global batch size and total GPU-hours of the three stages, including teacher-feature extraction and the Qwen3-VL-32B subtask annotation? How much inference latency does the DINOv2 branch add relative to Qwen3-VL-PI, given that the paper reports only a query rate of about 7.8 Hz on the UR5e?

Reviewer_YpQGOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SCULPT-VLA adds typed latent slots to a Qwen3-VL backbone (4 subtask, 4 dynamics and 8 spatial slots, plus action-readout slots), each type with its own residual FFN, and aligns them during training to three frozen teachers: Qwen3-Embedding applied to VLM-generated subtask annotations, UniVLA latent actions, and Depth Anything V2. Training has three stages with stage-specific attention masks. The factors are first aligned to the teachers; the teacher inputs are then annealed away while action prediction (FAST tokens) can read context only through the factor slots; finally direct perceptual access is restored and a DiT flow-matching head with an additional DINOv2 branch is trained. The paper reports 98.5% on LIBERO, 83.5% on SimplerEnv-WidowX (vs. 65.3% for the best shared-backbone baseline), 92.2% on RoboTwin 2.0 Full, and 58.1% vs. 45.6% average OOD success over π0.5\pi_{0.5} on four UR5e tasks, together with ablations and factor-replacement interventions.

Critical strengths

The core idea, first forcing action prediction through a structured bottleneck and only later restoring direct perceptual access, is simple and could transfer to other architectures. Table 4 indicates that this constraint is critical for performance on SimplerEnv: granting direct vision-language access during Stage II drops success from 83.5 to 71.3, whereas LIBERO-Spatial barely changes (Appendix C.3.1).

Critical weaknesses
  1. The real-robot results, which are the main evidence outside saturated or noisy simulation benchmarks, are difficult to accept as reported. First, the real-world baselines (OpenVLA-OFT, π0\pi_0, π0.5\pi_{0.5}) differ from SCULPT-VLA in both backbone architecture and pretraining compute. Because no shared-backbone baseline is evaluated on the physical robot, the comparison fails to isolate the impact of the proposed design. Second, the per-batch counts in Table 14 exhibit far lower variance than expected for independent physical rollouts. At most of the observed success rates, a batch of 30 trials should fluctuate by 2 to 3 successes (binomial standard deviation), but the reported batches mostly differ by only one or two. Aggregated across all 32 cells, the between-batch variance is only ~26% of the binomial expectation (an index of dispersion of 16.8 on 64 degrees of freedom, yielding a lower-tail p≈4×10−10p \approx 4 \times 10^{-10}; a parametric bootstrap with 200,000 replicates failed to produce a single dataset this homogeneous). This anomaly persists across all four evaluated methods, including the baselines. For instance, on the deformable Cloth Folding ID task, the reported 3-batch counts are 8/8/7, 11/11/10, 13/12/12, and 18/17/16, whereas the expected batch-level standard deviation is 2.4 to 2.7 successes. In contrast, the intervention rows of Table 16, which use the same robot setup, task, and 3×30 protocol, do not show this homogeneity. While replaying identical initial conditions across batches could reduce variance, the text describes the batches as independent, and reproducing such tight counts would require near-deterministic outcomes across initializations. In either case, the reported standard deviations fail to reflect actual evaluation uncertainty, making raw per-rollout logs essential before these findings can support the paper's claims.

  2. The simulation evidence for the design is also weaker than it looks. The ablation variants appear to be single training runs, so the reported ± ranges reflect evaluation noise rather than training variance; additionally, stage-removal ablations alter the total training compute budget (Appendix B.4). The overall training scale is very small (batch size 4–8, max 45k steps, LoRA fine-tuning on the VLM backbone; Tables 6 and 7), so single runs say little, and in the comparison with the shared-backbone baselines, SCULPT-VLA also gets supervision they do not (VLM-generated subtask labels and three frozen teacher models). LIBERO is near saturation, and a major portion of the margin on SimplerEnv stems from a single task (Stack: 75.3 vs. ≤30.2\le 30.2 for shared-backbone baselines). The ablations point in the right direction, but with single runs and changing budgets it is hard to tell how robust the effect of staged grounding is. Together with the real-robot issues above, the current evidence does not support the paper's central claim.

Questions for authors

Clarifications Requested:

  1. In the real-robot evaluation, did the three batches share an identical set of 30 initial scene configurations? How were these configurations specified and reset, and what was the execution order across methods and batches?
  2. Please provide detailed per-rollout logs for all real-robot experiments reported in Tables 14 and 16 (including method, task, test condition, batch index, configuration ID, timestamp, outcome, and failure mode), along with corresponding rollout video recordings.
  3. What accounts for the significantly lower between-batch variability observed in Table 14 compared to Table 16 under the same evaluation protocol?
  4. Were the shared-backbone baselines retrained using the identical dataset (including subtask annotations), compute budget, and random seeds, or were their metrics directly imported from the StarVLA release?

Key Experiments Needed:

  1. Report training-seed variance for both the full SCULPT-VLA model and key ablations (e.g., removing Stage II restrictions, omitting Stage I) on SimplerEnv, accompanied by a real-robot evaluation against at least one shared-backbone baseline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.