acceptodds
Withdrawn from ICLR 2027

Learning to Act through Sparse 4D Queries

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Successful manipulation requires a policy to understand where things are and how they will move, yet action demonstrations leave this spatiotemporal structure implicit. Existing methods supply it through dense supervision: world action models predict every pixel of future frames, and 3D-aware vision-language-action (VLA) models align every token with teacher features, both at substantial training cost. We ask whether sparse, explicit supervision suffices. We introduce SparQ4D, which casts depth, 2D/3D point motion, multi-view correspondence, and hand-eye relations as a single D4RT-style question (where is a selected point, at a specified time, in a specified view?), which is answered by a lightweight query decoder attached to a pretrained VLA. Each training step samples only a small query budget, and the decoder is discarded after training, leaving the VLA's architecture and inference cost unchanged. Across two VLA backbones in simulation and the real world, SparQ4D consistently improves manipulation: it raises the success rate of from 77.6% to 92.6% on RoboTwin 2.0, reducing over 60% failures, and outperforms X-WAM with 20% of its training GPU-hours and faster inference. Probing further shows that the trained VLA itself learns recoverable 4D structure.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-language-action Models, Spatiotemporal Learning, Sparse Supervision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gTv4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SparQ4D, a framework that enhances Vision-Language-Action (VLA) policies through sparse 4D query supervision. It unifies depth estimation, point motion prediction, multi-view correspondence, and hand-eye relations into a shared 3D coordinate prediction task. A lightweight query decoder extracts geometric information from intermediate VLA representations and is removed after training, introducing no additional inference overhead. Experiments on RoboTwin 2.0 and real-world manipulation tasks demonstrate consistent improvements over the original VLA backbones, alongside favorable computational efficiency compared with world action models.

Critical strengths
  1. The proposed framework integrates multiple spatial and temporal supervision objectives through a shared query decoder, avoiding dense visual reconstruction. Removing the decoder after training preserves the original VLA inference architecture and computational requirements, providing a practical approach to improving geometric representations.

  2. The method demonstrates substantial performance gains across two VLA backbones, RoboTwin simulation benchmarks, and real-world robotic manipulation tasks. The additional evaluation on 50 RoboTwin tasks further supports the effectiveness of the proposed supervision strategy across diverse manipulation scenarios.

Critical weaknesses
  1. The ablation results in Table 6 indicate that individual geometric supervision objectives already recover a substantial portion of the overall performance improvement. Although combining all four query types achieves the best results, it remains unclear whether the unified 4D query formulation provides meaningful advantages over simpler alternatives, such as conventional depth regression or separate geometric prediction heads. Comparisons under matched supervision and computational budgets would better isolate the contribution of the proposed design.

  2. The RoboTwin experiments use demonstrations from both Clean and Randomized settings, leaving the effectiveness of the learned geometric representations under genuinely unseen perturbations insufficiently examined. A more informative evaluation would train the baseline and SparQ4D exclusively on Clean demonstrations and evaluate them zero-shot under Randomized or Hard settings. This would help establish whether the benefits of 4D supervision extend beyond the training distribution.

  3. The reported training and inference efficiency advantages over X-WAM and Fast-WAM involve substantially different backbone architectures and parameter scales. Consequently, the observed differences cannot be attributed solely to sparse query supervision. Furthermore, the accounting of offline preprocessing costs, including point tracking and geometric target generation, should be clarified to support a more comprehensive and fair comparison of computational efficiency.

Questions for authors

See weaknesses.

Reviewer_9AtWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SparQ4D, an auxiliary training objective for vision-language-action policies. It frames depth, future point motion, multi-view correspondence, and hand–eye relations as sparse 4D queries answered by a lightweight decoder attached to the policy backbone. The decoder is removed at inference. The authors report improved manipulation success on RoboTwin 2.0 and five real-world tasks, along with lower training cost than the compared world action models.

Critical strengths
  1. The unified query formulation is simple and flexible. Several geometric and motion signals can supervise the same policy through one decoder, without adding inference-time computation.

  2. The empirical results are strong across two VLA backbones, simulation, and real-world tasks. The ablations on query types, query budget, and decoder placement help identify which design choices contribute to the gains.

Critical weaknesses
  1. Intuitively, the proposed method seems to bring the D4RT strategy directly into the VLA setting. Encouragingly, the reported results are strong. However, I am concerned that the potential of this D4RT-style strategy has not been fully explored. In line 191, delta_t spans the action horizon and thus refers to future observations. Yet the authors do not implement future-observation prediction, as a world model would. This raises questions about the method’s temporal modeling capability. Beyond world modeling, another possible direction would be a VLA with memory, with delta_t extended to past time steps. Have the authors considered or tried either approach?

  2. Table 5 should report the compute cost of the baseline, PI0.5, to facilitate comparison. Also, Spatial Forcing and SparQ4D both retain only the VLA backbone at inference. Why do they have different inference latencies?

Questions for authors

See Weakness.

Reviewer_4qZTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces SparQ4D, a training framework that teaches vision-language-action (VLA) policies geometry and motion through sparse, unified 4D queries. Each query asks where a selected point will be at a specified time and in a specified camera/coordinate frame. By varying the point, time, and reference frame, the same interface supervises depth, 2D/3D point motion, multi-view correspondence, and hand-eye relations. A lightweight query decoder is attached to intermediate VLA representations and jointly trained with the action objective; it is discarded at inference, so the policy architecture and inference cost remain unchanged. The authors evaluate SparQ4D on seven RoboTwin 2.0 tasks and five real-world tasks. They report average success-rate gains and better efficiency than X-WAM.

Critical strengths

(1) The core idea is simple, general, and deployment-efficient. SparQ4D unifies multiple geometric and motion supervision signals through one sparse-query interface, and the decoder is removed after training. This makes the method easy to attach to existing VLAs without adding inference cost.

(2) The empirical validation spans simulation and real-world manipulation across two backbones. Gains are shown on both pi0 and pi0.5. This provides reasonable evidence that sparse explicit supervision can capture part of the benefit of dense world-action modeling.

Critical weaknesses
  1. The empirical validation is narrow and not statistically or fairly established. The evaluation covers only seven simulated tasks and five real-world tasks, with 25 real-robot trials per task and no variance, confidence intervals, or significance tests.

  2. In the provided manuscript, several tables are malformed or incomplete, and it is not fully established that baselines received comparable tuning, data, and compute.

  3. The supervision depends on noisy offline labels without sensitivity analysis. The probing results show that geometry becomes decodable after joint training, but they do not establish that this geometry is causally responsible for the policy gains.

  4. Real-world depth targets rely on MoGe-3 estimates and motion targets rely on CoTracker3 tracks, yet sensitivity to tracker/depth noise is not analyzed.

  5. Ablations on query budget, decoder layer, loss weight, and training-cost breakdown are insufficiently reported.

Questions for authors

Please refer to paper weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.