- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper uses multi-camera videos during training to improve single-camera VLA policies. It learns one shared latent action from all views and trains the VLA to predict it from the primary camera, so only one camera is needed at deployment. Experiments show better success than prior methods, especially under camera and spatial shifts, though some distribution shifts remain challenging.
Critical strengths
A key contribution of the paper is a multi-view approach for enhancing latent action learning in VLA models. The method leverages observations from multiple cameras during training to learn a shared latent action, while requiring only a single camera at deployment. Experiments show that it achieves the highest average success rate among the compared latent-action methods across different benchmark settings, demonstrating the effectiveness of multi-view supervision for latent action learning.
Critical weaknesses
- W1. Mechanistic claim not directly validated. The paper claims that the shared decoder encourages the latent action to encode view-consistent physical changes rather than camera-specific content, but the experiments mainly show downstream performance. They do not directly demonstrate reduced camera-specific information or improved view invariance.
- W2. The contribution of the proposed multi-view design is not fully isolated. The comparison with the primary-only LAM establishes the benefit of incorporating additional views, but does not disentangle the contribution of multi-view information from that of the proposed cross-view fusion and shared-decoder objective. It therefore remains unclear whether the proposed architectural design is necessary.
Questions for authors
-
Q1. How can the authors directly verify that the shared decoder encourages the latent action to be more view-invariant and less camera-dependent, beyond showing improved policy performance?
-
Q2. How much do the proposed cross-view fusion and shared-decoder objective contribute to the gains, compared with simpler ways of incorporating additional views?
-
Q3. Could the authors include stronger baselines from other VLA paradigms, such as π0.5, especially on LIBERO-Pro and the real-robot experiments, to better assess the advantage over existing VLA methods?
Minor:
- The title of the AI-use statement is duplicated in the manuscript.
3 comments
Q2. Component ablations and overall supervision gains (W2)
Question: How much do the proposed cross-view fusion and shared-decoder objective contribute to the gains, compared with simpler ways of incorporating additional views?
Component controls at the representation level. All LAMs use the same training-video mixture and trajectory-sampling weights and are probed on the same LIBERO-Spatial transitions. PI retains multi-view reconstruction but restricts transition encoding to P. Mean replaces cross-view attention with averaging; +RR adds random codebook restarts. The loss for PI and Mean retains the original pooled per-camera reconstruction weighting. P/All and probe uncertainties follow Q1.
| LAM | Input | Perturbed | Clean MAE ↓ | Perturbed MAE ↓ | ΔMAE |
|---|---|---|---|---|---|
| Primary-only | P | P | 0.2695 ± 0.0004 | 0.2758 ± 0.0006 | +0.0064 ± 0.0004 |
| JMV-LA-PI | P | P | 0.2708 ± 0.0007 | 0.2739 ± 0.0004 | +0.0031 ± 0.0003 |
| JMV-LA | P | P | 0.2526 ± 0.0003 | 0.2581 ± 0.0002 | +0.0055 ± 0.0003 |
| JMV-LA | All | P | 0.2383 ± 0.0017 | 0.2371 ± 0.0009 | −0.0012 ± 0.0009 |
| JMV-LA | All | All | 0.2383 ± 0.0017 | 0.2405 ± 0.0014 | +0.0022 ± 0.0008 |
| Mean, no RR | P | P | 0.3543 ± 0.0004 | 0.3543 ± 0.0004 | 0.0000 ± 0.0000 |
| Mean, no RR | All | P | 0.3543 ± 0.0004 | 0.3543 ± 0.0004 | 0.0000 ± 0.0000 |
| Mean, no RR | All | All | 0.3543 ± 0.0004 | 0.3543 ± 0.0004 | 0.0000 ± 0.0000 |
| Mean+RR | P | P | 0.2439 ± 0.0009 | 0.2536 ± 0.0006 | +0.0097 ± 0.0009 |
| Mean+RR | All | P | 0.2474 ± 0.0014 | 0.2492 ± 0.0012 | +0.0018 ± 0.0009 |
| Mean+RR | All | All | 0.2474 ± 0.0014 | 0.2535 ± 0.0005 | +0.0061 ± 0.0012 |
Joint encoding. PI's clean MAE is 0.2708, versus 0.2526 for JMV-LA with P input and 0.2383 with All input. Joint multi-view training therefore improves overall readout even when extraction inputs are matched. Supplying both views to the same JMV-LA checkpoint further improves overall MAE, supporting complementary information for target construction.
Attention versus averaging. Averaging without restarts collapses to a constant token sequence in our run; its zero ΔMAE does not indicate useful invariance. Stabilized Mean+RR reaches all-view MAE 0.2474 and all-input perturbation ΔMAE 0.0061, versus 0.2383 and 0.0022 for JMV-LA without restarts. This supports attention's advantage in the all-view target-construction setting used for LIBERO supervision. Mean+RR has lower clean MAE under P extraction, so the attention advantage is specific to the all-view setting.
These controls separate joint transition encoding from multi-view reconstruction and compare attention with averaging. Decoder weight sharing itself has not been isolated through a shared-versus-per-view decoder comparison.
Matched policy control. Both policies use the same VLA initialization, pre-training data and budget, fine-tuning settings, action decoder, and latent-history mechanism. Only the target-generating LAM changes. JMV-LA targets for LIBERO use both primary and wrist frame pairs; both policies deploy with the primary camera only.
Clean LIBERO success (%).
| Target-generating LAM | Spatial | Object | Goal | Long | Average |
|---|---|---|---|---|---|
| Primary-only | 97.0 | 96.2 | 94.6 | 93.6 | 95.35 |
| JMV-LA | 97.2 | 98.0 | 96.0 | 94.2 | 96.35 |
LIBERO-10 camera-extrinsic perturbations (%).
| Target-generating LAM | Clean | ±5 cm | ±10 cm | ±5° | ±10° | Combo | Pert. Avg. | Retention |
|---|---|---|---|---|---|---|---|---|
| Primary-only | 93.6 | 70.4 | 29.0 | 67.2 | 22.0 | 48.0 | 47.32 | 50.56 |
| JMV-LA | 94.2 | 74.8 | 38.8 | 72.0 | 30.8 | 57.4 | 54.76 | 58.13 |
Success rates and retention are reported in percent. Pert. Avg. averages the five camera-extrinsic perturbations; Retention divides this mean by clean success. Combo combines ±5 cm translation and ±5° rotation.
LIBERO-10-Pro success (%).
| Target-generating LAM | Object | Position | Semantic | Task | Environment | Average |
|---|---|---|---|---|---|---|
| Primary-only | 49.0 | 0.4 | 90.8 | 13.4 | 47.2 | 40.16 |
| JMV-LA | 57.0 | 2.0 | 93.0 | 19.0 | 50.0 | 44.20 |
The full supervision design yields +1.00 percentage point on clean LIBERO, +7.44 points under camera perturbations, and +4.04 points on LIBERO-10-Pro. Camera-perturbation gains occur in all five conditions and exceed the 0.6-point clean LIBERO-10 gain. These policy comparisons use one training seed per configuration and measure the complete target-construction design.
Official Comment by Reviewer a2oS
Thank you for the detailed rebuttal and additional experiments. Your responses have addressed my concerns, and I am happy to raise my score.
We thank the reviewer for recognizing the practical value of using multi-view supervision for single-camera policies. We have added representation diagnostics and component ablations (Tables 8–9), together with a matched Primary-only policy control (Tables 3–4 and 13). Responses to Q1 and Q2 also address W1 and W2, respectively.
Q1. Representation-level evidence and the proposed mechanism (W1)
New representation diagnostics. We evaluate frozen LAMs on the same LIBERO-Spatial transitions. Primary-only encodes and reconstructs only the primary view. JMV-LA-PI encodes only the primary transition but reconstructs all available views with a shared decoder; JMV-LA additionally jointly encodes the available transitions. PI retains cross-attention and each reconstruction's own first-frame features. P and All below denote extraction-time inputs: primary only and primary plus wrist, respectively. They are distinct from the model names.
Each clean-trained action probe is tested under the same five image transformations, applied consistently to both frames of each perturbed view. MAE values are mean ± sample standard deviation over five MLP-probe seeds with fixed LAM checkpoints. D_view is clean–perturbed token disagreement; D_action compares distinct clean transitions from the same task. Their ratio describes perturbation-induced token change relative to between-transition token variation and is calculated from the displayed means.
What these measurements show. Under matched P input and P perturbation, PI and JMV-LA both have lower D_view and lower D_view/D_action than Primary-only. Thus, the two multi-view-reconstruction variants respond less to these image perturbations relative to their token variation across distinct transitions. Full JMV-LA also has lower clean and perturbed action-readout MAE. PI's clean MAE remains comparable to Primary-only, so its lower token sensitivity alone does not imply better action decodability.
With all-view extraction, JMV-LA's D_view/D_action is 0.1615 when only the primary view is perturbed and 0.3560 when both views are perturbed, compared with 0.4488 for the Primary-only LAM under primary-view perturbation. Both JMV-LA conditions therefore show lower perturbation-induced token change relative to between-transition token variation. Together with the action-readout results, these diagnostics support preserved action information and lower relative token sensitivity, rather than universally invariant token identities.
Connection to single-camera control. Latent-token invariance and policy-level camera robustness are distinct properties. We hypothesize that JMV-LA's overall multi-view target-construction pipeline provides more action-informative supervision for the single-camera VLA. Joint encoding and fusion incorporate complementary observations, while multi-view reconstruction constrains the shared bottleneck. Learning from targets with more stable action readout may help the VLA learn action-relevant features that remain useful under camera changes, consistent with the matched policy results below. These diagnostics support this overall explanation, but do not separately identify the effect of decoder weight sharing or establish camera-independent codes.