- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes VID-Policy, which projects historical observation features from a frozen video generative model, MiniMax-H3, into keys and values for a pretrained vision-language-action policy. The added attention branch and the original branch are normalized separately and combined through a learnable gate. Training updates the projectors, gates, and optional LoRA parameters while retaining the base policy’s action-generation architecture and training objective.
Experiments use DIAL and WALA as base policies. In the preliminary DIAL development evaluation covering 24 tasks with five trials per task, the BF16 configuration achieves 97/120 successes, compared with 92/120 for Native DIAL, an improvement of approximately 4.17 percentage points. On the full RoboCasa-GR1-Tabletop benchmark, WALA-based VID-Policy achieves 868/1,200 successes (72.33%), compared with 70.50% for the local WALA baseline, an improvement of 1.83 percentage points. The paper also reports history perturbation experiments, quantization results, and computational costs.
Critical strengths
-
The interface provides a practical design for reusing existing policies. It allows a policy to incorporate external video-model representations while retaining its action-generation architecture. Separate softmax operations prevent the added video tokens from competing with the original tokens within a shared softmax, and zero-initialized gates make the added branch contribute zero at initialization. The DIAL and WALA implementations demonstrate compatibility with different action-generation architectures. (Section 3; Appendix H.)
-
The paper includes full closed-loop evaluation and reports limitations transparently. The main comparison covers all 24 tasks with 50 trials per task, using the same evaluation seeds and scene settings, and reports improvements on 12 tasks, regressions on seven, and ties on five. The authors distinguish Native policies from the OFF condition obtained by disabling the branch after joint training. They also report history-perturbation results that were not statistically significant and deployment differences that affect runtime comparisons. (Sections 4.4–4.5; Appendices A, B, and F.)
Critical weaknesses
-
The main comparison does not isolate the contribution of the video branch beyond policy adaptation. The configuration in Table 5 trains approximately 66.06M LoRA parameters and 15.34M projector/gate parameters, but the paper does not report an independently trained rank-32 LoRA-only baseline matched for initialization, adapter scope, training data, training steps, and model-selection protocol. The adapters in the OFF condition were learned jointly with the video branch, so this condition does not estimate what adaptation alone could achieve. The earlier LoRA-only experiments in Appendix C use a single task, few trials, and different implementation settings. A matched control is needed to establish the incremental benefit of the video branch beyond LoRA adaptation. The reported improvement corresponds to a net gain of 22 successful episodes across 1,200 evaluations. Reporting the numbers of paired episodes in which only one method succeeds, together with a confidence interval for the success-rate difference, would help quantify uncertainty. Please clarify whether the evaluation seeds were also used for checkpoint or configuration selection. (Table 5; Appendices A–D and F.)
-
The experiments do not establish the benefit of choosing video-generative features as the history representation. The paper emphasizes video generative priors, but the results leave open whether the gains reflect access to historical observations, an additional pretrained encoder, or advantages of the chosen video-generative features. The history-perturbation controls test responses to altered inputs; shuffled histories, repeated current frames, and equal-RMS noise do not substitute for meaningful alternative representation sources. A comparison with a non-generative image or video encoder, using the same observation history and a comparable policy-adaptation protocol, would help assess the value of the chosen feature source. This gap limits the evidence for the advantage of video-generative features, rather than the feasibility of the proposed interface. (Sections 4.3–4.5; Table 4.)
Additional comments. The paper reports source-precision timing comparisons and acknowledges deployment differences, but the incremental deployment cost relative to the base policy remains unclear. Comparable end-to-end latency and memory measurements would help interpret the practical significance of the gains.
Appendix G appropriately notes the receiver’s permutation invariance. Please clarify how this limitation applies to the implemented frame-wise extraction pipeline and the interpretation of the shuffle results in Section 4.4.
The paper already distinguishes its integration into an existing policy from VPP and acknowledges the influence of Flamingo. A brief discussion of its relationship to IP-Adapter’s decoupled attention would further clarify the interface’s technical contribution.
Questions for authors
The first two questions are central to my assessment; the third concerns uncertainty in the reported improvement.
-
Can you provide an independently trained rank-32 LoRA-only baseline matched to Table 5 in initialization, adapter scope, training data, training steps, and model-selection protocol?
-
Can you compare the chosen features with an alternative representation of the same observation history, such as a non-generative image or video encoder, under a comparable policy-adaptation protocol? This would help establish whether the benefit depends on the chosen video-generative feature source.
-
For the full-benchmark comparison, can you report the paired discordant outcome counts and a confidence interval for the success-rate difference? Please also clarify whether the evaluation seeds were used for checkpoint or configuration selection.
6 comments
Response to Reviewer Xfdd (Part II)
Additional comment 2. frame-wise extraction pipeline and the interpretation of the shuffle results in Section 4.4.
Response: Thank you for pointing out the distinction between input-frame order and feature-token order. The permutation-invariance result in Appendix G concerns jointly permuting an already extracted set of keys and values. In our shuffle control, the observed RGB frames are reordered before video-feature extraction, after which the selected hidden states are concatenated and projected into policy-side K/V. These are different operations: any sensitivity to input-frame order must arise in the feature-extraction pipeline, since concatenation followed by the receiver’s unmasked attention does not itself introduce order sensitivity.
In the six-task, 30-episode screen, correct history achieves 76.7% success and shuffled history achieves 70.0%. We report this comparison as an exploratory history-content control. The observed difference does not by itself establish temporal-order reasoning or identify the component responsible for the difference.
Additional comment 3. Relationship to IP-Adapter.
Response: Thank you for pointing out this connection. Compared with IP-Adapter [R1], the proposed VID-policy generally has the following differences.
(1) From the functional perspective, VID-policy is designed to adapt the video information into the VLA latents. However, IP-Adapter is to transfer knowledge from a reference image to a diffusion model.
(2) From the attention architecture level, VID-policy hierarchically extracts features from different levels. However, the IP-Adapter only utilizes the feature from final layers.
(3) From the adaptation level, IP-Adapter does the reference image feature adaptation. However, the proposed VID-policy adapts the target VLA layers.
Based on the aforementioned analysis, we can find that although the IP adapter shares some similarities with the proposed VID-policy. However, the VID-policy indeed has significant differences with the IP-Adapter.
[R1] Ye, Hu, et al. "IP-Adapter: Text compatible image prompt adapter for text-to-image diffusion models." arXiv preprint arXiv:2308.06721 (2023).
Thanks for your review.
Dear Reviewer Xfdd,
I wanted to kindly follow up on the review for our submission. I would greatly appreciate any update at your earliest convenience. Thank you for your time!
Response to Reviewer Xfdd (Part I)
W1 / Q1. A matched, independently trained rank-32 LoRA-only baseline is needed.
Response: Thank you for this suggestion. We have added a further experiment to train original WALA with only Rank-32 LoRA. The experimental result is shown below and in Table 4 of Section 4.3. We can see that the method (B) with only LoRA shows a very marginal improvement, indicating that LoRA is not the principle attribute to performance improvement.
W2 / Q2. Compare with a non-generative encoder using the same history.
Response: Thanks for your comments. We have added further ablation studies, which utilize InternVideo (training for video recognition but not generation) as the visual encoder to extract latent from the same history. The experimental results are shown in the table below. We can see that H3 indeed contains more useful knowledge for the embodied control than the non-generative encoder.
Q3. Paired uncertainty and use of evaluation seeds for model selection.
Response: Across the 1,200 matched task-seed episode pairs, Native WALA succeeds in 846 episodes, and VID-Policy succeeds in 880, giving a gain of 2.83 percentage points. The 95% paired-bootstrap confidence interval is [+0.5,+5.2] percentage points.
For the full-benchmark comparison, we evaluate WALA-based VID-Policy against Native WALA on the same 24 tasks and evaluation seeds, with 50 episodes per task. We fixed the model configuration and training budget in advance and evaluated the final training checkpoint. Evaluation outcomes were not used to select checkpoints or configurations. This configuration was trained in one adaptation run.
We also report VID-Policy results on three five-seed subsets, each covering 24 tasks with five episodes per task. The success rates are 73.33%, 74.17%, and 75.00%, compared with 73.33% on the full benchmark.
Additional comment 1. Comparable end-to-end latency and memory measurements would help interpret the practical significance of the gains.
Response: INT8 reduces the recorded H3 weight-file size from 61.73 to 31.70 GiB, a 48.6% storage reduction. In the archived BF16/INT8 timing comparison, four-frame H3 extraction takes 10.028/0.952 seconds (10.53 times), policy requests take 9.168/1.547 seconds (5.93 times), and optimizer updates take 402.675/75.269 seconds (5.35 times). The training measurements use different device placements and servers. We additionally use the Qwen router to select between Native WALA and H3-assisted predictions; Appendix F reports the implementation and deployment costs.
The updated inference costs are shown below. VID-policy-S uses Qwen at every action chunk (K1), while VID-policy-S-Parallel runs WALA and H3 on separate NPUs. Latency includes routing, H3 feature extraction, communication, and action prediction; loading and warmup are excluded. Throughput is measured over the full request loop. Memory sums per-process peak Torch allocations and is an upper bound on simultaneous allocation.
We also compare three routing strategies using the same evaluation seeds. K1 queries Qwen at every action chunk and selects the higher-scoring branch; K3 queries it once every three chunks and holds the decision between queries. Budget uses the same Qwen selector at every chunk, with a threshold updated from recent H3 preference scores to target approximately 35% H3 invocation. H3 runs when its score reaches the threshold; otherwise WALA runs without H3. Appendix F gives the threshold and initialization details.
K3 reduces Qwen calls but can retain H3 decisions for longer. In the timing measurements, H3 invocation rates are 75% for K1, 87.5% for K3, and 37.5% for Budget routing. The budgeted strategy achieves 75.83% SR with an average latency of 1.057 seconds per request.