acceptodds
Under review as a conference paper at ICLR 2027

Scaling Vision-Language-Action Models with Video Generative Priors

Abstract

Scaling vision-language-action (VLA) models requires substantial robot data and computation. Pretrained video generators provide visual and temporal representations, but rendering future frames adds iterative sampling and re-encoding. We introduce VID-Policy, a video-generation-model-enhanced policy that transfers history-conditioned representations from a frozen video generator into a pretrained VLA. A projector maps these features to policy-side keys and values; gated additive attention combines them with native context while preserving the action generator. We assess source precision, history content, and closed-loop control in two policy families. In a fixed-seed, 24-task, five-episode-per-task DIAL development screen, BF16 video features yield 80.8% success, 4.2 percentage points above our reproduced Native DIAL reference. On the full 24-task, 50-episode-per-task RoboCasa-GR1-Tabletop benchmark, WALA-based VID-Policy reaches 72.33% success, improving over our WALA re-implementation by about 1.83 percentage points. The results show that video representations can guide action without sampling future frames.

Primary area:
applications to robotics, autonomy, planning
Keywords:
embodied model, video generation model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XfddOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VID-Policy, which projects historical observation features from a frozen video generative model, MiniMax-H3, into keys and values for a pretrained vision-language-action policy. The added attention branch and the original branch are normalized separately and combined through a learnable gate. Training updates the projectors, gates, and optional LoRA parameters while retaining the base policy’s action-generation architecture and training objective.

Experiments use DIAL and WALA as base policies. In the preliminary DIAL development evaluation covering 24 tasks with five trials per task, the BF16 configuration achieves 97/120 successes, compared with 92/120 for Native DIAL, an improvement of approximately 4.17 percentage points. On the full RoboCasa-GR1-Tabletop benchmark, WALA-based VID-Policy achieves 868/1,200 successes (72.33%), compared with 70.50% for the local WALA baseline, an improvement of 1.83 percentage points. The paper also reports history perturbation experiments, quantization results, and computational costs.

Critical strengths
  1. The interface provides a practical design for reusing existing policies. It allows a policy to incorporate external video-model representations while retaining its action-generation architecture. Separate softmax operations prevent the added video tokens from competing with the original tokens within a shared softmax, and zero-initialized gates make the added branch contribute zero at initialization. The DIAL and WALA implementations demonstrate compatibility with different action-generation architectures. (Section 3; Appendix H.)

  2. The paper includes full closed-loop evaluation and reports limitations transparently. The main comparison covers all 24 tasks with 50 trials per task, using the same evaluation seeds and scene settings, and reports improvements on 12 tasks, regressions on seven, and ties on five. The authors distinguish Native policies from the OFF condition obtained by disabling the branch after joint training. They also report history-perturbation results that were not statistically significant and deployment differences that affect runtime comparisons. (Sections 4.4–4.5; Appendices A, B, and F.)

Critical weaknesses
  1. The main comparison does not isolate the contribution of the video branch beyond policy adaptation. The configuration in Table 5 trains approximately 66.06M LoRA parameters and 15.34M projector/gate parameters, but the paper does not report an independently trained rank-32 LoRA-only baseline matched for initialization, adapter scope, training data, training steps, and model-selection protocol. The adapters in the OFF condition were learned jointly with the video branch, so this condition does not estimate what adaptation alone could achieve. The earlier LoRA-only experiments in Appendix C use a single task, few trials, and different implementation settings. A matched control is needed to establish the incremental benefit of the video branch beyond LoRA adaptation. The reported improvement corresponds to a net gain of 22 successful episodes across 1,200 evaluations. Reporting the numbers of paired episodes in which only one method succeeds, together with a confidence interval for the success-rate difference, would help quantify uncertainty. Please clarify whether the evaluation seeds were also used for checkpoint or configuration selection. (Table 5; Appendices A–D and F.)

  2. The experiments do not establish the benefit of choosing video-generative features as the history representation. The paper emphasizes video generative priors, but the results leave open whether the gains reflect access to historical observations, an additional pretrained encoder, or advantages of the chosen video-generative features. The history-perturbation controls test responses to altered inputs; shuffled histories, repeated current frames, and equal-RMS noise do not substitute for meaningful alternative representation sources. A comparison with a non-generative image or video encoder, using the same observation history and a comparable policy-adaptation protocol, would help assess the value of the chosen feature source. This gap limits the evidence for the advantage of video-generative features, rather than the feasibility of the proposed interface. (Sections 4.3–4.5; Table 4.)

Additional comments. The paper reports source-precision timing comparisons and acknowledges deployment differences, but the incremental deployment cost relative to the base policy remains unclear. Comparable end-to-end latency and memory measurements would help interpret the practical significance of the gains.

Appendix G appropriately notes the receiver’s permutation invariance. Please clarify how this limitation applies to the implemented frame-wise extraction pipeline and the interpretation of the shuffle results in Section 4.4.

The paper already distinguishes its integration into an existing policy from VPP and acknowledges the influence of Flamingo. A brief discussion of its relationship to IP-Adapter’s decoupled attention would further clarify the interface’s technical contribution.

Questions for authors

The first two questions are central to my assessment; the third concerns uncertainty in the reported improvement.

  1. Can you provide an independently trained rank-32 LoRA-only baseline matched to Table 5 in initialization, adapter scope, training data, training steps, and model-selection protocol?

  2. Can you compare the chosen features with an alternative representation of the same observation history, such as a non-generative image or video encoder, under a comparable policy-adaptation protocol? This would help establish whether the benefit depends on the chosen video-generative feature source.

  3. For the full-benchmark comparison, can you report the paired discordant outcome counts and a confidence interval for the success-rate difference? Please also clarify whether the evaluation seeds were used for checkpoint or configuration selection.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Xfdd (Part I)

W1 / Q1. A matched, independently trained rank-32 LoRA-only baseline is needed.

Response: Thank you for this suggestion. We have added a further experiment to train original WALA with only Rank-32 LoRA. The experimental result is shown below and in Table 4 of Section 4.3. We can see that the method (B) with only LoRA shows a very marginal improvement, indicating that LoRA is not the principle attribute to performance improvement.

WALA configuration# Adaptation parametersSR (%)
(A): Native WALA0.0 M70.50
(B): A + Rank-32 LoRA66.06 M70.75
(C): A + Rank-32 LoRA + Projector + video branch OFF81.40 M69.00
(D): A + Rank-32 LoRA + Projector + video branch ON (VID-policy)81.40 M73.33

W2 / Q2. Compare with a non-generative encoder using the same history.

Response: Thanks for your comments. We have added further ablation studies, which utilize InternVideo (training for video recognition but not generation) as the visual encoder to extract latent from the same history. The experimental results are shown in the table below. We can see that H3 indeed contains more useful knowledge for the embodied control than the non-generative encoder.

Video feature sourceSR (%)
Native WALA (trained with DINO-V3)70.50
WALA w/ InternVideo2 (video-encoder)70.08
WALA w/ H3 (Ours)73.33

Q3. Paired uncertainty and use of evaluation seeds for model selection.

Response: Across the 1,200 matched task-seed episode pairs, Native WALA succeeds in 846 episodes, and VID-Policy succeeds in 880, giving a gain of 2.83 percentage points. The 95% paired-bootstrap confidence interval is [+0.5,+5.2][+0.5,+5.2] percentage points.

For the full-benchmark comparison, we evaluate WALA-based VID-Policy against Native WALA on the same 24 tasks and evaluation seeds, with 50 episodes per task. We fixed the model configuration and training budget in advance and evaluated the final training checkpoint. Evaluation outcomes were not used to select checkpoints or configurations. This configuration was trained in one adaptation run.

We also report VID-Policy results on three five-seed subsets, each covering 24 tasks with five episodes per task. The success rates are 73.33%, 74.17%, and 75.00%, compared with 73.33% on the full benchmark.

Additional comment 1. Comparable end-to-end latency and memory measurements would help interpret the practical significance of the gains.

Response: INT8 reduces the recorded H3 weight-file size from 61.73 to 31.70 GiB, a 48.6% storage reduction. In the archived BF16/INT8 timing comparison, four-frame H3 extraction takes 10.028/0.952 seconds (10.53 times), policy requests take 9.168/1.547 seconds (5.93 times), and optimizer updates take 402.675/75.269 seconds (5.35 times). The training measurements use different device placements and servers. We additionally use the Qwen router to select between Native WALA and H3-assisted predictions; Appendix F reports the implementation and deployment costs.

The updated inference costs are shown below. VID-policy-S uses Qwen at every action chunk (K1), while VID-policy-S-Parallel runs WALA and H3 on separate NPUs. Latency includes routing, H3 feature extraction, communication, and action prediction; loading and warmup are excluded. Throughput is measured over the full request loop. Memory sums per-process peak Torch allocations and is an upper bound on simultaneous allocation.

MethodEnd-to-end latency (s/request)Throughput (requests/s)Peak allocation sum (GiB)
Native WALA0.1785.59110.44
VID-policy1.6280.61348.10
VID-policy-S1.5900.62749.79
VID-policy-S-Parallel1.3960.71650.02

We also compare three routing strategies using the same evaluation seeds. K1 queries Qwen at every action chunk and selects the higher-scoring branch; K3 queries it once every three chunks and holds the decision between queries. Budget uses the same Qwen selector at every chunk, with a threshold updated from recent H3 preference scores to target approximately 35% H3 invocation. H3 runs when its score reaches the threshold; otherwise WALA runs without H3. Appendix F gives the threshold and initialization details.

Routing strategySR (%)Latency (s/request)Throughput (requests/s)Peak allocation sum (GiB)
Qwen K174.171.5900.62749.79
Qwen K375.001.5070.66149.79
Qwen-score budget75.831.0570.94249.79

K3 reduces Qwen calls but can retain H3 decisions for longer. In the timing measurements, H3 invocation rates are 75% for K1, 87.5% for K3, and 37.5% for Budget routing. The budgeted strategy achieves 75.83% SR with an average latency of 1.057 seconds per request.

AuthorsOpenReview ↗

Response to Reviewer Xfdd (Part II)

Additional comment 2. frame-wise extraction pipeline and the interpretation of the shuffle results in Section 4.4.

Response: Thank you for pointing out the distinction between input-frame order and feature-token order. The permutation-invariance result in Appendix G concerns jointly permuting an already extracted set of keys and values. In our shuffle control, the observed RGB frames are reordered before video-feature extraction, after which the selected hidden states are concatenated and projected into policy-side K/V. These are different operations: any sensitivity to input-frame order must arise in the feature-extraction pipeline, since concatenation followed by the receiver’s unmasked attention does not itself introduce order sensitivity.

In the six-task, 30-episode screen, correct history achieves 76.7% success and shuffled history achieves 70.0%. We report this comparison as an exploratory history-content control. The observed difference does not by itself establish temporal-order reasoning or identify the component responsible for the difference.

Additional comment 3. Relationship to IP-Adapter.

Response: Thank you for pointing out this connection. Compared with IP-Adapter [R1], the proposed VID-policy generally has the following differences.

(1) From the functional perspective, VID-policy is designed to adapt the video information into the VLA latents. However, IP-Adapter is to transfer knowledge from a reference image to a diffusion model.

(2) From the attention architecture level, VID-policy hierarchically extracts features from different levels. However, the IP-Adapter only utilizes the feature from final layers.

(3) From the adaptation level, IP-Adapter does the reference image feature adaptation. However, the proposed VID-policy adapts the target VLA layers.

Based on the aforementioned analysis, we can find that although the IP adapter shares some similarities with the proposed VID-policy. However, the VID-policy indeed has significant differences with the IP-Adapter.

[R1] Ye, Hu, et al. "IP-Adapter: Text compatible image prompt adapter for text-to-image diffusion models." arXiv preprint arXiv:2308.06721 (2023).

AuthorsOpenReview ↗

Thanks for your review.

Dear Reviewer Xfdd,

I wanted to kindly follow up on the review for our submission. I would greatly appreciate any update at your earliest convenience. Thank you for your time!

Reviewer_gmw8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VID-Policy, a method that integrates spatiotemporal representations from a pretrained video generative model into vision-language-action models through feature projection and gated attention fusion. By leveraging historical video features without generating future frames during inference, VID-Policy improves the success rate on RoboCasa-GR1-Tabletop from 70.50% to 72.33%.

Critical strengths
  1. This paper proposes a new approach to integrating video features from a world model into a Vision-Language-Action (VLA) model, providing the VLA with feature information regarding future video and thereby enhancing its capabilities.

  2. This paper is logically coherent and features numerous visualizations, making it clear and easy to understand.

Critical weaknesses
  1. In Section 4.2 and Appendix E, two results are actually opposite, H3’s prediction metrics perform worse than simply replicating the current frame. Could you analyze why this occurs? Furthermore, what impact does using H3-generated futures have on the final action execution results in these scenarios? Is it possible that H3 is not the most suitable video generation model for this task? Would it be feasible to conduct an ablation study using a different model?

  2. In Section 4.5 and Figure S1 of the Appendix, it can be observed that VID-Policy underperforms on several of the 24 tasks within the RoboCasa-GR1-Tabletop benchmark. Could you analyze the specific reasons for these failures and provide visualizations of the relevant data to shed more light on the issue?

  3. For the future features of video generation models, not all features are necessarily useful for the execution of the final action. This paper does not explore which specific features are actually beneficial. It might be worth imposing dynamic constraints on the video features to extract the relevant dynamic information from them.

Questions for authors
  1. Related to the study that compares BF16 and INT8 precision for video features, however, since the number of denoising steps in a video generation model can also affect the precision of these features, shouldn't an ablation study be conducted to examine this as well?

  2. What is the specific contribution of the key/value pairs from this video feature within the gated attention mechanism? Are there any quantitative metrics available to assess this?

  3. Could you try using pi0.5[1] as the backbone for VLA?

[1] Intelligence, Physical, et al. "$\pi_ {0.5} $: a Vision-Language-Action Model with Open-World Generalization." arXiv preprint arXiv:2504.16054 (2025).

5 comments
AuthorsOpenReview ↗

Response to Reviewer gmw8 (Part I)

W1. H3 prediction quality differs between Section 4.2 and Appendix E; would another video model work better?

Response: Thanks for the comment. Here, we detailedly discuss the following 3 sub-problems.

(1) Differences between Sec. 4.2 and Appendix E. Section 4.2 shows one four-history-frame example, while Appendix E also reports a separate four-case test conditioned on a single frame.

(2) Effect of H3-generated features. We have conducted ablation studies to illustrate the effect of H3-generated features. For your convenience, we reproduce Table 4 from Section 4.3 below.

WALA configuration# Adaptation parametersSR (%)
(A): Native WALA0.0 M70.50
(B): A + rank-32 LoRA66.06 M70.75
(C): A + Rank-32 LoRA + Projector + video branch OFF81.40 M69.00
(D): A + Rank-32 LoRA + Projector + video branch ON (VID-policy)81.40 M73.33

For the results (C) and (D) or (B) and (D), we can easily find that the core of the VID policy is to construct an effective connection between the VLA and video generative model latents.

(3) Different video generation networks. To solve your concern, we have trained the VID-policy with WAN and SVD, with the same setting and protocols. From the below table. We can see that H3 achieves the highest success rate among the evaluated feature sources under the reported adaptation protocol.

WALA configurationEncoder SizeSR (%)
(A): Native WALA4B70.50
(B): A + VID-policy + WAN5.1B71.92
(C): A + VID-policy + SVD2.2B70.83
(D): A + VID-policy + H3 (Ours)33.1B73.33

W2. Why does VID-policy underperform on some tasks? Please provide failure visualizations.

Response: Thanks for the comment. Actually, in the original manuscript, the VID policy has been adapted with only 20% of training data. However, the original WALA is trained for more than 4 epochs. So there are nearly 20 times differences between the number of training iterations with WALA and our VID-policy. Then, the underperformance may be caused by insufficient training. Note that currently, we have extended the training process of VID-policy to 50% of the training data. Its success rate has been increased to 73.33%. Moreover, we have made a further statistic of the success rate of different tasks. Please refer to our supplementary Appendix A, Fig. S1. Our VID policy has surpassed WALA on most tasks.

We have added a failure visualization in Appendix H.2, Figure S6. In this Plate-to-Pan episode, the policy moves another object into the pan before transferring the target pear; both objects remain in the pan when the episode ends unsuccessfully.

Furthermore, another possible explanation is historical features may provide useful context in some states but introduce redundant or task-irrelevant information in others.

W3. Which video features are useful, and could dynamic constraints help select them?

Response: Yes, this is indeed an interesting point to further polish the model. We have conducted a further experiment to train a router, which is initialized by a tiny VLM, Qwen-0.8b. We train it to make a binary choice between utilizing the original WALA or our VID policy. Please refer to Appendix F for detailed implementation details. Integrating this router into the training process can accelerate the inference process. Moreover, we have found that for the subtasks of PlateToBowl, PlacematToBowl, PlacematToPlate, TrayToPot, and TrayToPlate (86 to 97% of action chunks), the router turns to select more video-policy predictions, indicating that under those cases VID-policy helps more cases.

Q1. The effect of the number of denoising steps.

Response: We want to note that one key point of our method is that it does not run iterative future-video sampling during policy inference. It still runs the frozen video network to extract hidden states from observed frames. The number of denoising steps in the standalone image-generation examples is therefore not a parameter of the deployed feature branch.

AuthorsOpenReview ↗

Response to Reviewer gmw8 (Part II)

Q2. What is the specific contribution of the key/value pairs from this video feature within the gated attention mechanism? Are there any quantitative metrics available to assess this?

Response: We have a further visual comparison of the attention map pattern in the appendix.~H.1, Fig. S5. For the figure, we can find that the additional KV cache from the video diffusion model can make the model focus more on the key elements, e.g., grasp target or actuator. Since the branches have separate softmax operations, their raw attention probabilities cannot be compared as if they shared one probability budget.

We have also measured closed-loop success with controlled K/V interventions at the same checkpoint (Section 4.3, Table 4):

Video branch conditionSR (%)
VID-policy w/ K/V from H373.33
VID-policy w/o K/V from H369.00

From the results, we can see that the k/V set can introduce more than a 4.3% success rate improvement compared with the one without K/V set, indicating the critical role of K/V cache from the video generative model.

Q3. Could you try using pi0.5 [R1] as the backbone for VLA?

Response: Thanks for your comment. To address your concern, we have trained an additional adapter using pi0.5 as the backbone. For this experiment, we use SVD as the video feature source and train the adapter to incorporate its features into pi0.5. Both variants use the same benchmark and observation/action setup. The results are reported below.

Policy configurationTrainable parametersSR (%)
pi0.50.0 M34.17
pi0.5 + VID-policy (SVD)41.48 M37.50

This comparison is also reported in Appendix I of the revised manuscript.

[R1] Physical Intelligence et al. pi0.5: a Vision-Language-Action Model with Open-World Generalization. [arXiv:2504.16054].

AuthorsOpenReview ↗

Thanks for your review.

Dear Reviewer gmw8,

I wanted to kindly follow up on the review for our submission. I would greatly appreciate any update at your earliest convenience. Thank you for your time!

Reviewer_RofbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents VID-Policy, a framework for augmenting pretrained Vision-Language-Action (VLA) models with representations extracted from a frozen video generation model. Instead of explicitly generating future video frames, the proposed approach extracts hidden representations from MiniMax-H3 and projects them into key-value representations that can be integrated into existing VLA architectures through gated additive attention. The framework preserves the original action generation head while optionally adapting the VLA backbone using LoRA.

The authors evaluate the approach on two VLA architectures, DIAL and WALA, using the RoboCasa-GR1-Tabletop benchmark. The results show a 4.2 percentage point improvement over Native DIAL in a small development evaluation and a 1.83 percentage point improvement over the authors' WALA re-implementation on the full 24-task benchmark.

Overall, the paper explores a meaningful direction for leveraging pretrained video models in robotic control. However, I have several concerns regarding the novelty of the proposed approach, the attribution of its performance improvements, and the strength of the experimental evidence.

Critical strengths

S1. Meaningful problem and practical motivation.

The paper addresses an important question: how to leverage knowledge acquired by large-scale video generation models to improve existing robotic policies without requiring expensive video generation or extensive retraining of the action model. This is a relevant direction given the cost of collecting robot demonstrations and training large VLA models.

S2. Modular and reasonably motivated architecture.

The proposed framework provides a relatively straightforward way to integrate pretrained video representations into existing VLA architectures. In particular, the separate attention normalization and learnable gating mechanism allow video-derived features to be incorporated while preserving native attention computation. Retaining the original action generation architecture is also a practical design choice.

S3. Evaluation across two policy architectures and multiple diagnostic settings.

The authors evaluate their approach using DIAL and WALA and provide several additional analyses, including source precision comparisons, adaptation configurations, history-content perturbations, and task-level success rates. The supplementary material also includes detailed implementation configurations and records of additional experiments, which improve the transparency of the work .

Critical weaknesses

W1. The contribution of video generative priors is not sufficiently isolated from other sources of improvement.

A central motivation of the paper is that representations learned by video generation models contain useful temporal and physical knowledge that can improve robotic control. However, the current experiments do not convincingly establish that the improvements specifically originate from these video generative priors.

In particular, the paper does not compare H3 representations against alternative pretrained visual representations, such as features from DINO-style encoders or predictive video representation models, under a matched adaptation setup. It is therefore unclear whether the observed improvements come from the generative and temporal knowledge of H3, or simply from introducing additional pretrained visual features and model capacity.

Although the history-content interventions in Table 4 provide some evidence that the injected information matters, they do not establish which properties of the video representations are responsible for the improvement.

I believe this is an important gap in the paper's contribution story. A controlled comparison with simpler visual feature extractors or lightweight history encoders would substantially strengthen the central claim.

W2. Key ablations are missing to disentangle the contributions of the video branch and additional policy adaptation.

The strongest full-benchmark result uses the rank-32 VLM-LoRA configuration, which introduces approximately 81.40M trainable parameters, including 66.06M LoRA parameters (Appendix F, Table S5).

However, the paper does not provide a matched LoRA-only baseline trained without H3 features under the final configuration. Consequently, it remains unclear how much of the improvement can be attributed to the injected video representations rather than the additional adaptation of the pretrained VLA.

The paper reports ON/OFF comparisons in smaller development evaluations, but these do not fully resolve the issue. Moreover, disabling the video branch after joint training is not equivalent to independently training a LoRA-only policy, since the jointly adapted parameters may have become dependent on the video features.

I would encourage the authors to include:

A LoRA-only baseline with a matched training budget and no video features.

A full-benchmark ON/OFF comparison using the final rank-32 checkpoint.

Ablations comparing the proposed gated attention mechanism against simpler feature fusion strategies.

These comparisons are important for establishing the necessity of the proposed components.

W3. The main performance improvement is relatively small, and its statistical reliability is unclear.

On the full RoboCasa-GR1-Tabletop benchmark, VID-Policy achieves 72.33% success compared with 70.50% for the authors' WALA re-implementation, corresponding to 22 additional successful episodes out of 1,200 (Table 5, Appendix A).

While this is a positive result, the paper does not provide sufficient statistical analysis to determine the reliability of the improvement. Since both methods are evaluated on matched task-seed pairs, reporting paired success/failure outcomes and confidence intervals would be particularly helpful. Results across independent training seeds would also improve confidence in the reported gain.

Furthermore, the improvements are unevenly distributed across tasks. Of the 24 tasks, 12 improve, seven deteriorate, and five remain unchanged. The three largest improvements together account for 22 additional successes, equal to the overall net gain across the benchmark. This suggests that the average improvement is substantially influenced by a small number of tasks.

A more detailed analysis of the task characteristics associated with improvements and failures would help establish when and why the additional video context is beneficial.

W4. The methodological novelty relative to existing video-representation-based policies needs to be better established.

Prior work, particularly Video Prediction Policy (VPP; Hu et al., 2024/2025), has already explored leveraging internal representations from video prediction models for robotic action generation without explicitly generating future frames.

The main distinction of the present work appears to lie in transferring frozen video-model representations into an existing VLA through projected key-value features and gated additive attention, while retaining the original action generation architecture.

This is a meaningful architectural difference. However, the individual techniques used in the interface, including projected attention features, gated cross-attention, and LoRA adaptation, are closely related to established methods.

Therefore, I find the novelty to be primarily in the particular integration of these components rather than in the general idea of using video-model internal representations for control.

The authors could strengthen this contribution by more clearly articulating the advantages of their interface over existing alternatives and by experimentally demonstrating which design choices are essential.

W5. The computational trade-off and practical benefits are not sufficiently demonstrated.

Although the proposed approach requires relatively few trainable parameters, it still relies on a frozen video generation backbone with approximately 33B parameters. This introduces substantial additional inference and memory requirements compared with the receiving policy alone.

Appendix F reports feature extraction times and precision-dependent efficiency measurements. However, the paper does not provide a sufficiently controlled end-to-end comparison of inference latency, throughput, and memory consumption against the native WALA policy under matched hardware conditions.

Given that the full-benchmark improvement is only 1.83 percentage points, the additional computational cost is an important consideration when assessing the practical value of the approach.

A more comprehensive performance-versus-compute analysis would make the proposed method's advantages and limitations clearer.

W6. Some implementation details and mechanistic explanations could be clarified.

The exact process by which history-conditioned representations are extracted from H3 would benefit from further clarification, particularly regarding temporal conditioning and feature construction (Sections 3.1–3.2, Appendix H).

In addition, the final rank-32 configuration differs from the illustrated visual-attention setup in its injection locations and omission of capped native-RMS rescaling. Although these differences are disclosed in the appendix, explicitly distinguishing the configurations in the main experimental discussion would improve clarity.

Finally, the history-content experiments are conducted on only 30 episodes per condition, and the reported exact McNemar tests do not establish statistically reliable differences. The cross-task intervention also changes both the video history and its prompt, complicating the interpretation of the results (Section 4.4, Table 4).

Questions for authors
  1. What is the unique contribution of video generative representations? Have the authors considered replacing H3 with a simpler pretrained visual or predictive video encoder while keeping the projection and adaptation budgets fixed? Such a comparison seems important for validating the central motivation.

  2. How much of the final performance improvement is attributable to the video branch rather than VLM-LoRA? Could the authors provide a LoRA-only baseline and a full-benchmark ON/OFF evaluation for the final rank-32 configuration?

  3. How statistically reliable is the reported 1.83 percentage point gain? Could the authors report paired task-seed outcomes, confidence intervals, and, if available, results across independent training seeds?

  4. Why do some tasks benefit substantially while others deteriorate? Is there a consistent relationship between the effectiveness of the video representations and task properties such as temporal dependency, occlusion, or object motion?

  5. What is the end-to-end computational overhead of VID-Policy relative to native WALA? In particular, what are the latency, throughput, and memory costs under matched deployment conditions?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Rofb (Part I)

W1 & Q1. Further experiments to illustrate the real advantages of generative priors. Comparison with simple history encoders and feature extractors.

Response: Thank you for your suggestion. First, we want to note that both baselines, DIAL and WALA, already use pretrained visual representations, especially that WALA has already utilized DINO-V3 [R1] to encode the visual information for training. Thus, the advantage of the proposed strategy actually demonstrates the superiority of the introduced video generative model.

To address your concern, we have compared our VID-policy with a different encoder, e.g., InternVideo2 [R2], which is trained to understand content but not predict the next frames, as alternative video feature sources, and add non-generative image/video encoders. All variants receive the same four observed frames, with the same protocols, e.g., receiving policy, training data, training steps, and LoRA scope. We report the success rate for each variant as below.

Video feature sourceSR (%)
Native WALA (trained with DINO-V3)70.50
WALA w/ InternVideo2 (video-encoder)70.08
WALA w/ H3 (Ours)73.33

The advancement of the method with the H3 feature to correspond using InternVideo2 (70.08%) indicates the necessity of introducing the generative prior.

W2 & Q2. Key ablations are missing to disentangle the contributions of the video branch and additional policy adaptation.

Response: Thanks for your question. To solve your concern, we have added further ablation studies.

LoRA-only baseline & full benchmark ON-OFF. We evaluate both an independently trained LoRA-only baseline and the jointly trained policy with its video branch enabled or disabled (Section 4.3, Table 4). We match the initialization, LoRA rank and locations, training data, training steps, and checkpoint-selection protocol. This differs from switching the branch OFF after joint training, as you pointed out.

WALA configuration# Adaptation parametersSR (%)
(A): Native WALA0.0 M70.50
(B): A + rank-32 LoRA66.06 M70.75
(C): A + Rank-32 LoRA + Projector + video branch OFF81.40 M69.00
(D): A + Rank-32 LoRA + Projector + video branch ON (VID-policy)81.40 M73.33

The LoRA-only model has the same 66.06M policy (VLA) adapters; VID-policy additionally trains 15.34M projector/gate parameters. Through comparing (D) and (B), VID-Policy exceeds the matched LoRA-only baseline by 2.58 percentage points, demonstrating a benefit beyond LoRA-only adaptation. Moreover, (C) and (D) are tested with the same VID-policy models with ON/OFF of the video branch. Through comparing them, we can get a clear illustration that the video generative feature indeed benefits the VLA.

Gated attention mechanism. We have also compared the simpler fusion choice while keeping the H3 features (Section 4.3, Table 5). In the following table, we directly concatenate the H3 and VLA features and utilize a standard self-attention to extract the feature, and the remaining other settings are the same.

Fusion strategySR (%)
(A): WALA Baseline70.50
(B): A + VID-policy w/ Shared-softmax fusion69.67
(C): A + VID-policy w/ Gated additive fusion73.33

The table clearly indicates that the proposed additive gated attention indeed effectively blends the features from the video generation model and VLAs.

W3 & Q3. The main performance improvement is relatively small, and its statistical reliability is unclear.

Response: Thanks for the comments. Note that, in the original paper, our adaptation only uses 20% of the training data. However, the baseline method WALA utilizes all training data and is trained by 4 epochs, nearly 20\times of our VID policy. Actually, we have extended training to 50% training data and reported the resulting success rate, which is 73.33%, with 2.83% improvement. It's a considerable improvement.

Across the 1,200 matched task-seed episode pairs, Native WALA succeeds in 846 episodes and VID-Policy succeeds in 880, giving a gain of 2.83 percentage points. The 95% paired-bootstrap confidence interval is [+0.5,+5.2][+0.5,+5.2] percentage points. We also report VID-Policy results on three random seed subsets, each covering 24 tasks with five episodes per task. The success rates are 73.33%, 74.17%, and 75.00%, compared with 73.33% on the full benchmark.

AuthorsOpenReview ↗

Response to Reviewer Rofb (Part II)

W4. The methodological novelty relative to existing video-representation-based policies needs to be better established.

Response: Thank you for pointing this out. I think there is a considerable difference between the VPP [R3] and our method. Actually, our contribution is the modeling of the interaction between the feature from a pretrained video generative model and the VLA model. And then to achieve better performance on the embodied work. However, VPP[R3] focuses solely on utilizing the feature from the video generative model. If you apply the VPP, you need to retrain a new head for generation, which may consume a large number of computational resources.

Moreover, we have proposed to apply quantification and selective feature fusion (please refer to the response to Q5) to reduce the amount of computing. We think these differences make our method significantly different from VPP.

Q4. Why do some tasks benefit substantially while others deteriorate?

Response: Thanks for your question. Actually, this may be caused by that; in the original submission, we only trained about 20% of the original training data. Increasing the adaptation data (from 20% to 50% of total training data) raises overall success from 72.33% to 73.33%, with improvements on 16 of the 24 tasks. Moreover, we can see from the updated subtask performance in Fig. S1 of the updated manuscript that the proposed method can outperform the original WALA on more cases.

Furthermore, another possible explanation is historical features may provide useful context in some states but introduce redundant or task-irrelevant information in others.

W5 & Q5. The computational trade-off and practical benefits are not sufficiently demonstrated.

Response: INT8 reduces the recorded H3 weight file size from 61.73 to 31.70 GiB, a 48.6% storage reduction. In the archived BF16/INT8 timing comparison, four-frame H3 extraction takes 10.028/0.952 seconds (10.53 times), policy requests take 9.168/1.547 seconds (5.93 times), and optimizer updates take 402.675/75.269 seconds (5.35 times). The training measurements use different device placements and servers. We additionally use the Qwen router to select between native WALA and H3-assisted predictions; Appendix F reports the implementation and deployment costs.

The updated inference costs are shown below. VID-policy-S uses Qwen at every action chunk (K1), while VID-policy-S-Parallel runs WALA and H3 on separate NPUs. Latency includes routing, H3 feature extraction, communication, and action prediction; loading and warmup are excluded. Throughput is measured over the full request loop. Memory sums per-process peak Torch allocations and is an upper bound on simultaneous allocation.

MethodEnd-to-end latency (s/request)Throughput (requests/s)Peak allocation sum (GiB)
Native WALA0.1785.59110.44
VID-policy1.6280.61348.10
VID-policy-S1.5900.62749.79
VID-policy-S-Parallel1.3960.71650.02

We also compare three routing strategies using the same evaluation seeds. K1 queries Qwen at every action chunk and selects the higher-scoring branch; K3 queries it once every three chunks and holds the decision between queries. Budget uses the same Qwen selector at every chunk, with a threshold updated from recent H3 preference scores to target approximately 35% H3 invocation. H3 runs when its score reaches the threshold; otherwise WALA runs without H3. Appendix F gives the threshold and initialization details.

Routing strategySR (%)Latency (s/request)Throughput (requests/s)Peak allocation sum (GiB)
Qwen K174.171.5900.62749.79
Qwen K375.001.5070.66149.79
Qwen-score budget75.831.0570.94249.79

K3 reduces Qwen calls but can retain H3 decisions for longer. In the timing measurements, H3 invocation rates are 75% for K1, 87.5% for K3, and 37.5% for Budget routing. The budgeted strategy achieves 75.83% SR with an average latency of 1.057 seconds per request.

AuthorsOpenReview ↗

Response to Reviewer Rofb (Part III)

W6. Some implementation details and mechanistic explanations could be clarified.

Response: Thanks for the comment. We explain each point in the following content.

(1) history-conditioned representation extraction. The history consists of four observed frames at offsets (6, 4, 2, 0). In the implemented frame-wise route, we extract hidden states from these frames with the task prompt, concatenate them along the context-token axis, and project the selected source-block features into policy-side K/V. We do not generate future RGB frames for the policy. We have specified the feature-extraction timestep setting (Appendix H) and any temporal-position or cross-frame conditioning (Appendix H), since these details determine what temporal information the features can contain.

(2) Rank-32 configuration. We have also distinguished the two WALA placements more clearly. Schemes A/B inject into visual-attention layers 12--23 and use capped native-RMS rescaling. The final rank-32 configuration injects into language-attention layers 12, 14, ..., 34, uses receiver-normalized queries/keys, and omits that rescaling. Its gates/projectors and VLM-LoRA account for 15.34M and 66.06M trainable parameters, respectively (Appendices F and H).

[R1] Siméoni, O., Vo, H.V., Seitzer, M., Baldassarre, F., Oquab, M., Jose, C., Khalidov, V., Szafraniec, M., Yi, S., Ramamonjisoa, M. and Massa, F., 2025. Dinov3. arXiv preprint arXiv:2508.10104.

[R2] Wang, Yi, et al. "Internvideo2: Scaling foundation models for multimodal video understanding." European conference on computer vision. Cham: Springer Nature Switzerland, 2024.

[R3] Hu, Yucheng, et al. "Video prediction policy: A generalist robot policy with predictive visual representations." arXiv preprint arXiv:2412.14803 (2024).

Comments on the paper

5 comments
AuthorsOpenReview ↗

General Response (Part I)

We thank the reviewers for their careful assessment and constructive suggestions. We have carefully revised the manuscript and note the major revision using the red color. We summarize our responses in two parts: clarifications of the method and its interpretation, followed by the additional experiments and their results.

I. Key Clarifications

(1) Contribution and relationship to prior work. Our contribution is an interface for transferring frozen video-model representations into an existing pretrained VLA while retaining its action generator. VPP learns an action policy conditioned on video-predictor representations, whereas VID-Policy adapts an existing VLA through layer-specific K/V projections and gated attention. We acknowledge the relationship to established gated and decoupled attention mechanisms, including Flamingo and IP-Adapter, and position our contribution in their integration with pretrained VLA policies. These discussions appear in Section 2 (Figure 1) and Section 3.3.

(2) Attribution of the improvement. An independently trained LoRA-only baseline and a video-branch ON/OFF comparison answer different questions. The former measures what policy adaptation alone can achieve; the latter measures the contribution of the video branch to the jointly trained policy. We therefore include both controls. Comparisons with alternative feature sources further assess the choice of H3 under the evaluated adaptation setup. The controls are reported in Section 4.3 (Table 4), and the feature-source comparisons are in Appendix B (Tables S2 and S3).

(3) Generated-image quality versus policy features. The deployed policy uses hidden representations extracted from observed frames through the frozen video network, without iterative future-video sampling. The number of sampling steps used for standalone RGB generation is therefore distinct from the deployed feature-extraction procedure. The four-history-frame example in Section 4.2 (Figure 3 and Table 2) and the single-frame examples in Appendix E (Figure S4 and Table S6) also use different conditioning settings. We distinguish these image-generation probes from the closed-loop experiments that evaluate whether the extracted features improve control. The feature-extraction procedure is described in Section 3.1 and Appendix H, under "Feature extraction."

(4) Adaptation budget and deployment cost. The reported trainable-parameter budget covers the projectors, gates, and policy adapters. Deployment also requires executing the frozen video backbone. We therefore assess practical cost through end-to-end latency, throughput, and memory measurements, and separately report selective and parallel execution variants. Section 4.5 summarizes deployment cost; Appendix F provides parameter accounting (Table S7), precision-dependent costs (Table S8), and deployment and routing measurements (Tables S9 and S10).

AuthorsOpenReview ↗

General Response (Part II)

II. Additional Experiments and Results

(1) [Rofb: W3, Q3, Q4; Xfdd: W1, Q3; gmw8: W2] Extended adaptation and paired evaluation. Additional adaptation improves the full-benchmark success rate from 72.33% to 73.33%. Across 24 tasks and 1,200 episodes per method, VID-Policy succeeds in 880 episodes versus 846 for Native WALA (70.50%), giving 34 additional successes and a gain of 2.83 percentage points. Updated task-level results show improvements on 16 tasks, regressions on six, and ties on two. We have reported these results in Section 4.5 (Table 7), Appendix A, and Figure S1.

(2) [Rofb: W2, Q2; Xfdd: W1, Q1; gmw8: Q2] Independent LoRA and video-branch controls. The independently trained rank-32 LoRA-only baseline achieves 70.75% success, compared with 73.33% for VID-Policy. At the same jointly trained checkpoint, disabling and enabling the video branch yields 69.00% and 73.33%, respectively. These results distinguish the benefit beyond LoRA-only adaptation from the jointly trained policy's reliance on video features. These comparisons appear in Section 4.3 (Table 4).

(3) [Rofb: W2; gmw8: Q2] Fusion ablation. We added a shared-attention baseline that places native and H3 keys/values under one softmax. It achieves 69.67% success, compared with 73.33% for the separately normalized, gated additive interface, supporting the proposed fusion configuration in this comparison. We have reported these results in Section 4.3 (Table 5).

(4) [Rofb: W1, Q1; Xfdd: W2, Q2; gmw8: W1] Alternative feature sources. We evaluated InternVideo2 as a non-generative video encoder and SVD and Wan as alternative video generators. With WALA as the receiving policy, these variants achieve 70.08%, 70.83%, and 71.92% success, respectively. H3 achieves 73.33%, the highest success rate among the tested sources. These comparisons appear in Appendix B (Tables S2 and S3).

(5) [gmw8: Q3] Additional VLA backbone. We extended the evaluation to pi0.5. The baseline achieves 34.17% success, while pi0.5 with VID-Policy achieves 37.50%, an improvement of 3.33 percentage points. We have reported these results in Appendix I (Table S12) of the revised manuscript.

(6) [Rofb: W5, Q5; Xfdd: Additional comment on deployment cost] Deployment measurements and acceleration. VID-Policy takes 1.628 seconds per request. Qwen selection and parallel execution reduce this to 1.590 and 1.396 seconds, respectively. We also compare routing at every action chunk, routing once every three chunks, and score-budgeted routing using the same evaluation seeds. The measurements and routing implementation are reported in Appendix F (Tables S9 and S10).

(7) [gmw8: W3, Q2] Attention analysis. We added spatial attention visualizations and object/hand region-of-interest measurements. In the two illustrated observations, the manipulated-object attention share increases from 0.64% to 0.76% and from 1.82% to 2.00% when comparing Native WALA with additive attention. These examples provide descriptive evidence of changes in attention allocation and complement the closed-loop K/V and fusion ablations. The attention analysis appears in Appendix H.1 (Figure S5).

The LoRA and video-branch controls (Table 4) and the fusion comparison (Table 5) are presented in Section 4.3. Supporting experiments and detailed analyses remain in the appendix. The failure-case visualization is provided in Appendix H.2 (Figure S6).

AuthorsOpenReview ↗

Follow-up on Reviewer gmw8's Assessment

Thank you for your reassessment and feedback. If our rebuttal and additional results have addressed your main concerns, we would be grateful if you could consider revising your score upward. If any concerns remain, we would be happy to provide further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.