acceptodds
Withdrawn from ICLR 2027

Closing the Act-Predict-Evaluate Loop with a Shared Video-Pretrained Representation for Robotic Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video-pretrained representations provide rich visual and temporal structure for embodied learning, but acting, prediction, and evaluation are still typically developed in separate representation spaces. This separation limits model-based policy improvement because predicted future states cannot be directly consumed by the policy and value model. We introduce Pivot, a framework that instead uses a frozen video-pretrained representation as the shared state space of the entire Act-Predict-Evaluate loop. The policy reads this state, action-conditioned dynamics predicts its successor, and a value model evaluates the predicted state under the instruction, enabling recursive latent imagination without observation reconstruction or representation translation. To make this generic video representation effective for manipulation, we introduce task-aware supervision at two levels. First, we train task-relevant latent dynamics using a semantic affordance prior derived from image-instruction attributions, emphasizing the interaction-critical changes. Second, we learn a task-regularized value function with temporal-difference learning and dense image-instruction supervision, providing progress estimates for imagined states. These estimates allow latent rollouts to directly guide policy improvement. On CALVIN, Pivot improves the imitation policy's average successful sequence length from to . On a five-stage real-robot task, it increases the average number of completed stages from to , with controlled ablations showing that both forms of task-aware supervision contribute to the improvement.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied AI, Representation Learning, Robot Learning, World Models, JEPA

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_s7wMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Pivot, a framework where a policy, dynamics model, and value function operate in a shared representation space from a frozen V-JEPA 2 encoder. The authors use instruction-dependent spatial weighting for dynamics prediction and combine TD learning with LIV-based semantic supervision for value estimation. They then use imagined rollouts and value-based binary labels to refine a flow-matching policy. The method improves average sequence length on CALVIN from 3.946 to 4.294, and on a five-stage real-robot task from 1.6 to 4.3.

Critical strengths
  1. The paper shows meaningful improvements over its own imitation baseline, particularly on the real-robot task, where average completed stages increase from 1.6 to 4.3. While the evaluation is limited to a single task with 10 trials, the improvement is encouraging.
  2. The value-learning ablations are useful. In particular, combining TD and semantic supervision improves ranking accuracy on imagined successors from 66.9% with TD alone to 80.3%. This provides some evidence that the proposed supervision helps evaluate predicted action outcomes.
Critical weaknesses
  1. My main concern is the choice of representation. The paper motivates using V-JEPA as a shared state space, but it is unclear how much of the improvement comes from the pretrained representation itself versus the task-aware supervision that is not sufficiently aligned with manipulation. All ablations keep the encoder fixed. I also find the related work on latent-action and motion-centric representations underdeveloped. Works like UniVLA [1] and ViPRA [2] explicitly learn task-relevant motion representations from video, with ViPRA using optical-flow supervision for fine-grained motion. These are not necessarily direct replacements for a full visual state, but are relevant alternatives worth discussing. Some comparison with other visual or motion-centric representations would help establish the contribution.
  2. I have some concerns with the experimental comparisons. The reported VPP baseline (4.261) is below its originally published 4.33 [3]. The authors do mention re-evaluating the checkpoint, but the difference matters given Pivot's small margin over VPP. DreamVLA [4], which reports 4.44 on CALVIN, is also missing from the comparison despite being cited. More importantly, the adapted WMPO baseline only uses a static camera while Pivot uses both static and wrist views, making this comparison difficult to interpret. This seems particularly relevant since VPP [3] (Appendix C) reports a substantial performance drop when using only the static camera (4.33 to 3.58). Multi-view π0\pi_0 and OpenVLA-OFT results on CALVIN have already been reported [5].
  3. I would also like to see more evidence for the recursive imagination component. The dynamics model is trained on demonstrations with relatively short rollout horizons, but is later used to evaluate policy-generated actions in imagined states. The paper does not analyze how prediction errors or value-ranking accuracy change with imagination depth, which seems important for the main claim.
Questions for authors
  1. Have the authors considered comparing different pretrained representations while keeping the proposed task-aware losses fixed? This would help clarify whether V-JEPA itself is important to the results.
  2. What is the imagination rollout length used during policy refinement? How does dynamics accuracy and value-ranking performance change with rollout depth, especially for actions outside the demonstration distribution?
  3. Why is WMPO evaluated with only a static camera? Could the authors provide a comparison with matched visual inputs? Also, what explains the difference between the re-evaluated VPP checkpoint and its originally reported performance?

References

[1] Bu et al., UniVLA: Learning to Act Anywhere with Task-centric Latent Actions, RSS 2025.

[2] Routray et al., ViPRA: Video Prediction for Robot Actions, ICLR 2026.

[3] Hu et al., Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations, ICML 2025.

[4] Zhang et al., DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge, NeurIPS 2025.

[5] Lin et al., HiF-VLA: Hindsight, Insight and Foresight through Motion Representation for Vision-Language-Action Models, CVPR 2026.

Reviewer_DsDzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Pivot, a model-based policy improvement framework that unifies policy, dynamics prediction, and value evaluation within a shared video-pretrained representation space. By leveraging a frozen V-JEPA 2 encoder, the framework enables closed-loop latent imagination without observation reconstruction or representation translation.

The method further introduces a semantic affordance prior to emphasize task-relevant regions during dynamics learning and combines temporal-difference learning with semantic supervision for value estimation. The learned dynamics and value models are then used to generate and evaluate imagined trajectories, which guide policy improvement through advantage-conditioned flow matching.

Experiments on CALVIN and a five-stage real-robot manipulation task demonstrate improvements over the initial imitation policy.

Critical strengths
  1. The paper presents a well-structured framework that integrates policy, dynamics, and value learning in a shared latent space. This design enables recursive latent imagination without reconstructing visual observations, providing a practical approach to model-based policy improvement.

  2. The method demonstrates consistent improvements over its imitation initialization on both CALVIN and real-robot experiments. In particular, the real-robot average completed stages increase from 1.6 to 4.3. The ablation studies also provide evidence supporting the effectiveness of the task-aware dynamics and value supervision.

Critical weaknesses
  1. Insufficient validation of the core representation design.

The paper emphasizes the benefits of a shared + video-pretrained representation, but the experiments do not sufficiently isolate its contribution.

  1. The improvement over the imitation baseline demonstrates the effectiveness of the overall pipeline rather than the shared representation itself. Although WMPO is included as a baseline, it differs in multiple aspects beyond representation design, making the comparison less conclusive.

  2. Furthermore, the paper does not compare V-JEPA 2 with image-pretrained encoders, leaving the specific benefit of video pretraining unclear.

2. Limited benchmark diversity and experimental validation.

  1. The paper evaluates Pivot on only one standardized simulation benchmark (CALVIN), without exploring its applicability to different robot embodiments or higher-dimensional action spaces, such as dual-arm manipulation. An additional benchmark would strengthen the evidence for the general applicability of the proposed framework.

  2. Furthermore, the real-robot experiments only compare Pivot against its own imitation initialization, without additional competitive imitation learning baselines. The evaluation is also limited to a single five-stage task with 10 trials per policy. Details about the training data, evaluations under varied conditions, and failure analysis are insufficient, limiting the assessment of the method's competitiveness, robustness, and generalization.

Questions for authors

Q1. Contribution of shared representations.

Could the authors provide a controlled ablation comparing the proposed shared representation with separate representations for acting, prediction, and evaluation, while keeping other components consistent? This would help clarify the independent contribution of the shared representation.

Q2. Necessity of video-pretrained representations.

Since Appendix A.1 indicates that the current frame is duplicated rather than using consecutive video frames, what is the specific benefit of video pretraining in this framework? Could the authors compare V-JEPA 2 with image-pretrained encoders such as DINO or SigLIP? If V-JEPA 2 significantly outperforms these image-pretrained alternatives under comparable settings, it would provide stronger empirical justification for emphasizing video-pretrained representations as a key component of the proposed framework.

Q3. Real-robot baselines.

Could the authors provide comparisons with other competitive imitation learning methods, such as pi05, under the same real-robot setup and training data? This would help establish the effectiveness of Pivot beyond improvements over its own imitation initialization.

Q4. Real-robot evaluation and generalization.

Could the authors clarify the amount and sources of demonstration and failure data used in the real-robot experiments? Additionally, could you provide evaluations under varied initial conditions or object configurations, along with execution videos and failure-case analysis, to better demonstrate the robustness and generalization of the proposed method?

Q5. Benchmark diversity and applicability to different robot embodiments.

The paper only evaluates Pivot on CALVIN as a standardized simulation benchmark. Could the authors provide additional evaluations on other benchmarks, particularly dual-arm manipulation benchmarks? This would help verify whether the proposed framework remains effective across different robot embodiments, higher-dimensional action spaces, and more complex manipulation tasks.

(Bt the way, compared to the questions mention above, question 5 is just a small question, you can pay more attention to the question 1-4, question 5 will additionally improve your paper quality. I will raise my score if you solve most of these questions.)

Reviewer_aENaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Pivot, a model-based policy improvement framework for language-conditioned robotic manipulation. A frozen V-JEPA 2 encoder provides a shared video-pretrained representation for the policy, dynamics model, and value model, enabling recursive latent imagination without observation reconstruction or representation translation. The authors introduce a semantic affordance prior derived from LIV image-instruction attributions to emphasize instruction-relevant regions in dynamics learning. They also combine temporal-difference learning with dense image-instruction supervision to learn a task-regularized value function. Predicted value gains from latent rollouts provide supervision for policy refinement. Experiments on CALVIN and a five-stage real-robot task demonstrate improvements over the imitation policy, with ablation studies supporting the contributions of both forms of task-aware supervision.

Critical strengths
  1. Model-based policy improvement enhances long-horizon robotic manipulation performance, with effectiveness demonstrated in real-robot experiments.
  2. The effectiveness of task-aware supervision for dynamics and task-regularized value learning is supported by ablation studies and action-ranking evaluation.
Critical weaknesses
  1. Although the paper distinguishes Pivot from RISE and V-JEPA 2-AC, the novelty of its specific methodological components is not sufficiently discussed.
  2. Recursive latent imagination may suffer from accumulated prediction errors. However, the paper lacks systematic analysis across different imagination horizons, leaving their impact on value estimation and policy improvement unclear.
  3. Although Table 3 supports the effectiveness of task-aware supervision, the paper lacks comparisons for the shared representation and direct comparisons between single-chunk and recursive latent imagination, making the independent contributions of these central designs unclear.
Questions for authors
  1. The paper discusses the differences between Pivot and RISE and V-JEPA 2-AC. Could the authors further clarify the methodological novelty of task-aware supervision for dynamics and task-regularized value learning beyond existing approaches?
  2. Although Table 4 reports action-ranking accuracy on imagined successors, it remains unclear how prediction errors affect value estimation over longer imagination horizons. An analysis of latent prediction errors and action-ranking accuracy across different horizons would help clarify their impact on predicted value gains and policy improvement.
  3. Given the differences in model backbones and visual inputs, how can the benefits of the shared representation be distinguished from other factors?
  4. Does recursive latent imagination provide additional performance gains over imagination with a single action chunk?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.