acceptodds
Under review as a conference paper at ICLR 2027

Joint Multi-View Latent Actions for Robust Vision-Language-Action Learning

Abstract

A latent action model (LAM) maps a pair of video frames to a discrete latent action that a vision-language-action (VLA) policy learns to predict, so that the policy can learn from videos without action labels. Existing LAMs infer the latent action, mostly, from one camera, whose frames record the physical change only as that camera projects it, together with camera-specific content; the latent action can therefore lose part of the change and carry that content instead. JMV-LA infers the latent action from all cameras a recording provides: a shared transition encoder summarizes every available camera's frame pair, learned queries read all summaries before quantization, and one shared decoder must predict every camera's next features from the single latent action and that camera's first frame, so that the latent action has to supply the change in a form that serves every projection. The VLA predicts it from the primary camera alone and needs no additional camera at deployment. Compared with LAMs learned from the primary camera only, JMV-LA yields lower action-decoding error and a readout that is more stable under image perturbations. The single-camera VLA supervised by JMV-LA achieves the highest average success among the compared policies on the four LIBERO suites, LIBERO-10 camera-pose perturbations, and LIBERO-Pro, as well as the highest observed success in six of seven ALOHA real-robot evaluation settings.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Latent Action Learning, Vision-Language-Action Models, Multi-View Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_a2oSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses multi-camera videos during training to improve single-camera VLA policies. It learns one shared latent action from all views and trains the VLA to predict it from the primary camera, so only one camera is needed at deployment. Experiments show better success than prior methods, especially under camera and spatial shifts, though some distribution shifts remain challenging.

Critical strengths

A key contribution of the paper is a multi-view approach for enhancing latent action learning in VLA models. The method leverages observations from multiple cameras during training to learn a shared latent action, while requiring only a single camera at deployment. Experiments show that it achieves the highest average success rate among the compared latent-action methods across different benchmark settings, demonstrating the effectiveness of multi-view supervision for latent action learning.

Critical weaknesses
  • W1. Mechanistic claim not directly validated. The paper claims that the shared decoder encourages the latent action to encode view-consistent physical changes rather than camera-specific content, but the experiments mainly show downstream performance. They do not directly demonstrate reduced camera-specific information or improved view invariance.
  • W2. The contribution of the proposed multi-view design is not fully isolated. The comparison with the primary-only LAM establishes the benefit of incorporating additional views, but does not disentangle the contribution of multi-view information from that of the proposed cross-view fusion and shared-decoder objective. It therefore remains unclear whether the proposed architectural design is necessary.
Questions for authors
  • Q1. How can the authors directly verify that the shared decoder encourages the latent action to be more view-invariant and less camera-dependent, beyond showing improved policy performance?

  • Q2. How much do the proposed cross-view fusion and shared-decoder objective contribute to the gains, compared with simpler ways of incorporating additional views?

  • Q3. Could the authors include stronger baselines from other VLA paradigms, such as π0.5, especially on LIBERO-Pro and the real-robot experiments, to better assess the advantage over existing VLA methods?

Minor:

  • The title of the AI-use statement is duplicated in the manuscript.
3 comments

We thank the reviewer for recognizing the practical value of using multi-view supervision for single-camera policies. We have added representation diagnostics and component ablations (Tables 8–9), together with a matched Primary-only policy control (Tables 3–4 and 13). Responses to Q1 and Q2 also address W1 and W2, respectively.

Q1. Representation-level evidence and the proposed mechanism (W1)

Question: How can the authors directly verify that the shared decoder encourages the latent action to be more view-invariant and less camera-dependent, beyond showing improved policy performance?

New representation diagnostics. We evaluate frozen LAMs on the same LIBERO-Spatial transitions. Primary-only encodes and reconstructs only the primary view. JMV-LA-PI encodes only the primary transition but reconstructs all available views with a shared decoder; JMV-LA additionally jointly encodes the available transitions. PI retains cross-attention and each reconstruction's own first-frame features. P and All below denote extraction-time inputs: primary only and primary plus wrist, respectively. They are distinct from the model names.

Each clean-trained action probe is tested under the same five image transformations, applied consistently to both frames of each perturbed view. MAE values are mean ± sample standard deviation over five MLP-probe seeds with fixed LAM checkpoints. D_view is clean–perturbed token disagreement; D_action compares distinct clean transitions from the same task. Their ratio describes perturbation-induced token change relative to between-transition token variation and is calculated from the displayed means.

LAMInput viewsPerturbed viewsClean MAE ↓Perturbed MAE ↓ΔMAED_viewD_actionD_view/D_action
Primary-onlyPP0.2695 ± 0.00040.2758 ± 0.0006+0.0064 ± 0.00040.30690.68380.4488
JMV-LA-PIPP0.2708 ± 0.00070.2739 ± 0.0004+0.0031 ± 0.00030.27190.73710.3689
JMV-LAPP0.2526 ± 0.00030.2581 ± 0.0002+0.0055 ± 0.00030.28250.82160.3438
JMV-LAAllP0.2383 ± 0.00170.2371 ± 0.0009−0.0012 ± 0.00090.14410.89230.1615
JMV-LAAllAll0.2383 ± 0.00170.2405 ± 0.0014+0.0022 ± 0.00080.31770.89230.3560

What these measurements show. Under matched P input and P perturbation, PI and JMV-LA both have lower D_view and lower D_view/D_action than Primary-only. Thus, the two multi-view-reconstruction variants respond less to these image perturbations relative to their token variation across distinct transitions. Full JMV-LA also has lower clean and perturbed action-readout MAE. PI's clean MAE remains comparable to Primary-only, so its lower token sensitivity alone does not imply better action decodability.

With all-view extraction, JMV-LA's D_view/D_action is 0.1615 when only the primary view is perturbed and 0.3560 when both views are perturbed, compared with 0.4488 for the Primary-only LAM under primary-view perturbation. Both JMV-LA conditions therefore show lower perturbation-induced token change relative to between-transition token variation. Together with the action-readout results, these diagnostics support preserved action information and lower relative token sensitivity, rather than universally invariant token identities.

Connection to single-camera control. Latent-token invariance and policy-level camera robustness are distinct properties. We hypothesize that JMV-LA's overall multi-view target-construction pipeline provides more action-informative supervision for the single-camera VLA. Joint encoding and fusion incorporate complementary observations, while multi-view reconstruction constrains the shared bottleneck. Learning from targets with more stable action readout may help the VLA learn action-relevant features that remain useful under camera changes, consistent with the matched policy results below. These diagnostics support this overall explanation, but do not separately identify the effect of decoder weight sharing or establish camera-independent codes.

Q2. Component ablations and overall supervision gains (W2)

Question: How much do the proposed cross-view fusion and shared-decoder objective contribute to the gains, compared with simpler ways of incorporating additional views?

Component controls at the representation level. All LAMs use the same training-video mixture and trajectory-sampling weights and are probed on the same LIBERO-Spatial transitions. PI retains multi-view reconstruction but restricts transition encoding to P. Mean replaces cross-view attention with averaging; +RR adds random codebook restarts. The loss for PI and Mean retains the original pooled per-camera reconstruction weighting. P/All and probe uncertainties follow Q1.

LAMInputPerturbedClean MAE ↓Perturbed MAE ↓ΔMAE
Primary-onlyPP0.2695 ± 0.00040.2758 ± 0.0006+0.0064 ± 0.0004
JMV-LA-PIPP0.2708 ± 0.00070.2739 ± 0.0004+0.0031 ± 0.0003
JMV-LAPP0.2526 ± 0.00030.2581 ± 0.0002+0.0055 ± 0.0003
JMV-LAAllP0.2383 ± 0.00170.2371 ± 0.0009−0.0012 ± 0.0009
JMV-LAAllAll0.2383 ± 0.00170.2405 ± 0.0014+0.0022 ± 0.0008
Mean, no RRPP0.3543 ± 0.00040.3543 ± 0.00040.0000 ± 0.0000
Mean, no RRAllP0.3543 ± 0.00040.3543 ± 0.00040.0000 ± 0.0000
Mean, no RRAllAll0.3543 ± 0.00040.3543 ± 0.00040.0000 ± 0.0000
Mean+RRPP0.2439 ± 0.00090.2536 ± 0.0006+0.0097 ± 0.0009
Mean+RRAllP0.2474 ± 0.00140.2492 ± 0.0012+0.0018 ± 0.0009
Mean+RRAllAll0.2474 ± 0.00140.2535 ± 0.0005+0.0061 ± 0.0012

Joint encoding. PI's clean MAE is 0.2708, versus 0.2526 for JMV-LA with P input and 0.2383 with All input. Joint multi-view training therefore improves overall readout even when extraction inputs are matched. Supplying both views to the same JMV-LA checkpoint further improves overall MAE, supporting complementary information for target construction.

Attention versus averaging. Averaging without restarts collapses to a constant token sequence in our run; its zero ΔMAE does not indicate useful invariance. Stabilized Mean+RR reaches all-view MAE 0.2474 and all-input perturbation ΔMAE 0.0061, versus 0.2383 and 0.0022 for JMV-LA without restarts. This supports attention's advantage in the all-view target-construction setting used for LIBERO supervision. Mean+RR has lower clean MAE under P extraction, so the attention advantage is specific to the all-view setting.

These controls separate joint transition encoding from multi-view reconstruction and compare attention with averaging. Decoder weight sharing itself has not been isolated through a shared-versus-per-view decoder comparison.

Matched policy control. Both policies use the same VLA initialization, pre-training data and budget, fine-tuning settings, action decoder, and latent-history mechanism. Only the target-generating LAM changes. JMV-LA targets for LIBERO use both primary and wrist frame pairs; both policies deploy with the primary camera only.

Clean LIBERO success (%).

Target-generating LAMSpatialObjectGoalLongAverage
Primary-only97.096.294.693.695.35
JMV-LA97.298.096.094.296.35

LIBERO-10 camera-extrinsic perturbations (%).

Target-generating LAMClean±5 cm±10 cm±5°±10°ComboPert. Avg.Retention
Primary-only93.670.429.067.222.048.047.3250.56
JMV-LA94.274.838.872.030.857.454.7658.13

Success rates and retention are reported in percent. Pert. Avg. averages the five camera-extrinsic perturbations; Retention divides this mean by clean success. Combo combines ±5 cm translation and ±5° rotation.

LIBERO-10-Pro success (%).

Target-generating LAMObjectPositionSemanticTaskEnvironmentAverage
Primary-only49.00.490.813.447.240.16
JMV-LA57.02.093.019.050.044.20

The full supervision design yields +1.00 percentage point on clean LIBERO, +7.44 points under camera perturbations, and +4.04 points on LIBERO-10-Pro. Camera-perturbation gains occur in all five conditions and exceed the 0.6-point clean LIBERO-10 gain. These policy comparisons use one training seed per configuration and measure the complete target-construction design.

Reviewer_a2oSOpenReview ↗

Official Comment by Reviewer a2oS

Thank you for the detailed rebuttal and additional experiments. Your responses have addressed my concerns, and I am happy to raise my score.

Reviewer_m96EOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that latent action models (LAMs) inferred from a single camera can miss parts of the physical change and absorb camera-specific appearance. JMV-LA instead encodes every available camera’s frame pair with a shared, camera-type-aware transition encoder. The VLA is trained to predict these multi-view-derived targets from the primary camera only, so deployment stays single-camera. Experiments include action-decoding probes against a matched primary-only LAM, LIBERO, LIBERO-10 camera-pose perturbations, LIBERO-Pro, and seven ALOHA real-robot settings.

Critical strengths
  1. The paper is well written and easy to follow. It clearly lays out the problem background, the core challenge, and the motivation, and it describes the method and implementation details thoroughly.

  2. The idea is simple and practical. Many existing robot datasets already provide synchronized multi-view recordings. Using these views only to construct better latent-action targets, without requiring any additional camera at deployment, is a clean design choice.

  3. The masked query-based fusion handles heterogeneous camera availability, with half the samples being primary-only, without zero-padding artifacts, and always outputs a fixed four-token target. The camera-type embedding for wrist versus external views is a sensible detail.

  4. Results are consistently positive across four evaluation settings. The camera-pose perturbation result in Table 4, with an 8.6-point gain over MVP-LAM, and the spatial OOD cube-to-cup result in Table 6 are the most notable.

Critical weaknesses
  1. Novelty over MVP-LAM and VILA is incremental. MVP-LAM already uses cross-view reconstruction to make latents action-centric. The difference here, fusing views before quantization rather than after, is reasonable but narrow. The paper never isolates this specific difference experimentally, for example by comparing it with MVP-LAM-style cross-view reconstruction on the same data mixture and backbone.

  2. The probe in Sec. 4.2 does not test what the policy actually uses. According to App. E.1, JMV-LA receives the wrist-camera pair at probe time, while the primary-only LAM does not. The lower MAE in Table 1 can therefore come from extra input at inference rather than a better learned latent space. The relevant question is whether JMV-LA tokens, computed from the primary view alone with other views masked, are more informative.

  3. The key policy-level ablation is missing. The method inherits almost the whole UniVLA pipeline: the two-stream controllable and uncontrollable latents, language conditioning in phase 1, the action decoder, latent history, and Prismatic initialization. Yet no VLA is trained with the primary-only LAM targets. Component ablations, such as fusion versus averaging, the camera-type embedding, and the shared versus per-view decoder, are also absent.

  4. The mechanism for viewpoint robustness is not explained. The policy sees only the perturbed primary image, and nothing in its inputs or training makes it invariant to camera extrinsics. Why multi-view targets should make a single-view policy more robust to camera pose is asserted but not analyzed. Retention is still only 58 percent, and the comparison includes only latent-action baselines, with none of the viewpoint-robust methods discussed in Sec. 2.2.

  5. Effect sizes on standard benchmarks are small and single-seed. On LIBERO, 96.35 versus 95.3 and 95.2 is within typical seed variance. On LIBERO-Pro, a large part of the gain comes from LIBERO-Spatial (Table 10). Position scores are at floor for every method, and OpenCLAP-SV wins two of the five categories.

  6. Target consistency across view counts is unexamined. Half the training samples are labeled from the primary view only and half from two or three views. Nothing shows that the same code carries the same meaning in both regimes. This matters because the VLA must predict a single consistent target from a single view.

Questions for authors
  1. What are the Table 1 and Table 2 results when JMV-LA receives only the primary view, with other views masked?

  2. Can the authors report LIBERO and LIBERO-10 perturbation results for a VLA trained on primary-only LAM targets, with everything else held fixed?

  3. Is codebook usage, or the action decodability of the same code, different between samples labeled from one view and samples labeled from several views?

  4. Could the authors provide a more comprehensive ablation study of the key design components, for example cross-attention fusion versus simple averaging, the camera-type embedding, and the shared decoder versus per-view decoders?

6 comments

We thank the reviewer for recognizing the practical motivation and for identifying the controls needed to assess our contribution. We have added primary-input-only representation probes, a matched policy-level ablation, and controlled comparisons of joint encoding, reconstruction weighting, and fusion (revised Tables 3–4, 8–9, and 13).

Q1. Primary-view-only probes and the role of multi-view targets (W2).

Question: What are the Table 1 and Table 2 results when JMV-LA receives only the primary view, with other views masked?

Following your suggestion, we evaluate both frozen LAMs using only the same primary-camera frame pairs, with all auxiliary views masked for JMV-LA. We retain the trajectory-held-out LIBERO-Spatial probe split and the same probe architecture and training settings. The following tables extend the evaluations in Tables 1–2; the complete ablations are in revised Tables 8–9. MAEs and R² are mean ± sample standard deviation over five MLP-probe seeds with fixed LAM checkpoints.

Action decodability with primary-view-only input.

LAMMAE ↓Trans. MAE ↓Rot. MAE ↓Global R² ↑Gripper MAE ↓
Primary-only0.2695 ± 0.00040.2683 ± 0.00050.2429 ± 0.00060.4441 ± 0.00240.3525 ± 0.0022
JMV-LA0.2526 ± 0.00030.2273 ± 0.00040.2390 ± 0.00070.4954 ± 0.00180.3695 ± 0.0019

With identical input views, JMV-LA has lower overall, translation, and rotation MAE. The primary-only model retains the lower gripper MAE. Thus, the improvement in overall action decodability persists without additional wrist-camera information at token extraction time.

Image-perturbation results with primary-view-only input.

Clean-trained probes are evaluated after the same five image transformations, applied consistently to both frames of the primary-camera pair. Token disagreements are fixed-checkpoint statistics; ΔMAE is computed within each probe seed before aggregation and rounding.

Perturbed viewLAMClean MAE ↓Perturbed MAE ↓ΔMAE ↓D_viewD_action
Primary onlyPrimary-only0.2695 ± 0.00040.2758 ± 0.0006+0.0064 ± 0.00040.30690.6838
Primary onlyJMV-LA0.2526 ± 0.00030.2581 ± 0.0002+0.0055 ± 0.00030.28250.8216

JMV-LA retains a lower perturbed MAE and has a modestly smaller MAE increase. Its lower D_view indicates fewer token changes under the tested image transformations, while its higher D_action indicates greater token differentiation between distinct transitions from the same task. Together with the controlled action-readout gains, these diagnostics are consistent with a more action-centric representation under primary-only input.

What the original two-view probe measures.

The original probe evaluates the action information contained in the teacher targets used for VLA supervision. For labeling LIBERO demonstrations, JMV-LA jointly uses the primary and wrist frame pairs to construct these targets, while the VLA learns to predict them from primary-camera observations and language. During the VLA pretraining on the OXE mixture, latent action target construction also uses each sample's all available views. Using complementary views to construct richer action supervision, while no wrist image or LAM execution is required at deployment, is the purpose of the design.

For the same JMV-LA checkpoint, two-view extraction further lowers overall MAE from 0.2526 to 0.2383 and raises global R² from 0.4954 to 0.5342 (Table 8), supporting the intended benefit of complementary observations for target construction. The original two-view probe and the new masked-view probe therefore address target quality and performance under matched extraction inputs, respectively. Both of them are frame-pair representation diagnostics, and the matched policy experiment below tests whether these targets improve single-camera closed-loop control.

Q2. Matched policies trained with primary-only versus JMV-LA targets (W3).

Question: Can the authors report LIBERO and LIBERO-10 perturbation results for a VLA trained on primary-only LAM targets, with everything else held fixed?

We have added the policy-level control in revised Tables 3–4. Both policies use the same downstream framework, VLA initialization, pre-training data and budget, fine-tuning settings, action decoder, and latent-history mechanism. They differ only in the LAM used to generate their supervision targets.

Standard LIBERO success rates (%).

Target-generating LAMSpatialObjectGoalLongAverage
Primary-only97.096.294.693.695.35
JMV-LA97.298.096.094.296.35

LIBERO-10 camera-extrinsic perturbation results (%).

Both policies use the primary camera only at deployment. Pert. Avg. is the mean success rate over the five perturbation conditions; Retention is this mean divided by clean success, expressed as a percentage. Combo combines ±5 cm translation and ±5° rotation.

Target-generating LAMClean±5 cm±10 cm±5°±10°ComboPert. Avg.Retention
Primary-only93.670.429.067.222.048.047.3250.56
JMV-LA94.274.838.872.030.857.454.7658.13

Average clean success across the four LIBERO suites increases from 95.35% to 96.35% (+1.00 percentage point). On LIBERO-10, clean success increases from 93.6% to 94.2% (+0.6 points), while mean success under camera-extrinsic perturbations increases from 47.32% to 54.76% (+7.44 points), with improvements under all five perturbation conditions. Retention rises from 50.56% to 58.13%.

With the policy-learning pipeline held fixed, these results support the benefit of replacing primary-only LAM targets with jointly constructed JMV-LA targets for single-camera control. The observed gain is larger under camera-pose shifts than in the clean LIBERO-10 setting.

Q3. Consistency across view counts (W6).

Question: Is codebook usage, or the action decodability of the same code, different between samples labeled from one view and samples labeled from several views?

Controlled comparison on the same multi-view samples. We use the same frozen JMV-LA checkpoint and codebook to extract tokens from the same LIBERO-Spatial transitions, supplying either the primary frame pair alone or both primary and wrist frame pairs. Only the input-view mask changes. Probes are fitted separately for the two input configurations using the same trajectory split, architecture, and training protocol. Values are mean ± sample standard deviation over five probe seeds (revised Tables 8–9).

Action decodability under one- and two-view extraction.

JMV-LA input viewsMAE ↓Trans. MAE ↓Rot. MAE ↓Global R² ↑Gripper MAE ↓
Primary only (1)0.2526 ± 0.00030.2273 ± 0.00040.2390 ± 0.00070.4954 ± 0.00180.3695 ± 0.0019
Primary + wrist (2)0.2383 ± 0.00170.2149 ± 0.00320.2531 ± 0.00070.5342 ± 0.00620.2640 ± 0.0017

Both input regimes produce labels that support action readout. Using the additional wrist view improves overall, translation, and gripper MAE and global R², while primary-only extraction has lower rotation MAE. This supports the benefit of incorporating complementary views when constructing latent-action targets from multi-view recordings.

Image perturbations under one- and two-view extraction.

Each probe is trained on clean tokens and evaluated under the same five image transformations. ΔMAE is computed per seed before rounding; token disagreements are fixed-checkpoint statistics.

JMV-LA input viewsPerturbed viewsClean MAE ↓Perturbed MAE ↓ΔMAE ↓D_viewD_action
Primary only (1)Primary only0.2526 ± 0.00030.2581 ± 0.0002+0.0055 ± 0.00030.28250.8216
Primary + wrist (2)Primary only0.2383 ± 0.00170.2371 ± 0.0009−0.0012 ± 0.00090.14410.8923
Primary + wrist (2)Both views0.2383 ± 0.00170.2405 ± 0.0014+0.0022 ± 0.00080.31770.8923

When only the primary view is perturbed in both configurations, two-view extraction gives lower perturbed MAE and D_view. Perturbing both views also leaves action-readout MAE close to its clean value. The higher D_action with two-view extraction indicates greater token differentiation across distinct same-task transitions, consistent with less repetitive code assignments.

Scope of the consistency claim. These results establish action decodability under both extraction regimes on a matched multi-view dataset, with better overall readout when both views are available. They do not establish that an individual code has identical physical-action semantics across datasets with different camera availability, or that a readout fitted under one view count transfers unchanged to another. That code-wise alignment question requires a separate test.

Why train on a mixture of primary-only and multi-view samples? The goal is to scale latent-action learning across heterogeneous video datasets: retain the large volume of primary-only data while exploiting every available observation in multi-view recordings to construct more action-informative targets. Primary-only samples account for 49.90% of our effective training mixture. This motivates the asymmetric cross-attention design: four learned queries attend to the available camera-transition tokens as keys and values, with missing views masked. The input can contain one, two, or three views, while the output remains four latent-action tokens. This provides a common prediction interface for the VLA across datasets with different camera configurations.

Reviewer_Pjs8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Joint Multi-View Latent Actions (JMV-LA), a framework that learns discrete latent-action representations from synchronized multi-camera robot videos to supervise single-view vision-language-action (VLA) policies. The method encodes each available camera's frame transition, fuses the resulting representations through learned cross-view attention before vector quantization, and uses a shared decoder to reconstruct future visual features across available views. The resulting four-token latent action is then predicted by a VLA that receives only a primary-camera image and a language instruction at deployment.

The authors evaluate the learned representations through action-decoding probes and image-space perturbations, and assess downstream policies on the four LIBERO suites, camera-pose perturbations, LIBERO-Pro, and real-robot ALOHA tasks. They report an 11.56% reduction in action-decoding MAE relative to a primary-view-only latent-action model, a 96.35% average success rate on LIBERO, and the highest observed success in six of seven real-robot evaluation settings. Overall, the paper investigates whether using complementary camera views during latent-action learning can improve downstream single-view robot control.

Critical strengths
  1. A practical and well-motivated use of multi-view supervision. JMV-LA uses synchronized camera observations to construct a shared latent-action target while allowing the deployed policy to operate with a single camera. This is a useful design choice for robot learning from heterogeneous video datasets, where multi-camera coverage may be available during training but not at deployment. The fixed-length quantized bottleneck also provides a consistent interface across samples with different camera availability.

  2. Broad empirical validation with encouraging robustness results. The paper evaluates both representation quality and downstream control, including LIBERO, camera-pose perturbations, LIBERO-Pro, and physical robot manipulation. The reported improvements under simulated camera-pose changes and several real-robot distribution shifts are encouraging. The results suggest that multi-view supervision can be useful beyond simply improving visual transition reconstruction.

Critical weaknesses
  1. The primary-only ablation does not isolate the contribution of joint multi-view fusion. The comparison changes several factors simultaneously: the available transition inputs, the cross-view fusion module, and the reconstruction targets. Furthermore, the multi-view reconstruction loss averages over available cameras, so a sample with three cameras contributes three times as many feature elements as a single-camera sample, whereas the primary-only baseline gives each sample one primary-view reconstruction target. This changes the effective weighting of training samples according to camera availability. Consequently, the observed improvement cannot be attributed exclusively to the proposed pre-quantization fusion mechanism. More controlled ablations, ideally with matched sample weighting and camera-availability subsets, would strengthen the central claim.

  2. The robustness evidence is nuanced, and uncertainty estimates for downstream performance are limited. The clean LIBERO gains over the strongest reported baselines are relatively small, and most baseline results are taken from existing publications or official releases rather than retrained under a fully matched protocol. Multi-seed uncertainty estimates for the main simulated success rates are not reported. In addition, the appendix's token-level diagnostics show similar token disagreement under all-view image perturbations for JMV-LA and the primary-only model (0.31770 versus 0.30685). The action-readout results and camera-pose-perturbation policy results are promising, but further analysis would clarify which aspects of robustness improve and how consistently these improvements hold across training runs.

Questions for authors
  1. Controlled ablations: Can the authors provide additional ablations that separate the effect of joint cross-view fusion from multi-view reconstruction and camera-dependent sample weighting? In particular, how much of the action-decoding improvement remains when the reconstruction loss is normalized per transition rather than per available camera, and when the models are compared on a matched subset of multi-view samples?

  2. Predictability of multi-view latent-action targets: Since the latent-action targets are inferred from future frames and all available camera views, but the VLA predicts them from the primary camera alone, how predictable are these targets from the deployed policy's inputs? Could the authors report latent-token prediction accuracy or cross-entropy stratified by camera availability and camera type? This would help establish whether the additional views provide useful action supervision or introduce target information that is difficult to infer from the primary view.

  3. Evaluation uncertainty and baseline comparability: Were the main policy results reproduced across multiple training seeds under a common downstream protocol, and can the authors report corresponding uncertainty estimates? For the real-robot evaluations, how sensitive are the comparisons to the sequential rather than interleaved evaluation order, given that the reported Wilson intervals do not account for temporal hardware variation?

4 comments

We thank the reviewer for recognizing the practical motivation and encouraging robustness results, and for identifying the controls needed to interpret them. We have added controlled LAM ablations, an offline target-prediction evaluation, and a matched policy-level comparison. We address the three questions below.

Q1. Controlled ablations

Question: Controlled ablations: Can the authors provide additional ablations that separate the effect of joint cross-view fusion from multi-view reconstruction and camera-dependent sample weighting? In particular, how much of the action-decoding improvement remains when the reconstruction loss is normalized per transition rather than per available camera, and when the models are compared on a matched subset of multi-view samples?

Data matching and additional controls. The Primary-only LAM and JMV-LA already use the same training-video mixture, trajectories, and trajectory-sampling weights. The Primary-only LAM retains trajectories that have multiple cameras and uses their primary-camera frame pairs. We agree that the original comparison changes the multi-view design as a whole and does not isolate fusion or reconstruction weighting. We have therefore added the following controls in Appendix Sections A.4–A.5 (Tables 8–9):

  • JMV-LA-PT: average reconstruction error over each transition's available views before averaging over transitions, giving transitions equal reconstruction weight regardless of camera count.
  • JMV-LA-PI: encode only the primary transition while retaining cross-attention, reconstruction of all available views, and the original pooled per-camera loss. Each reconstruction still receives its own view's first-frame features.
  • JMV-LA-Mean / Mean+RR: replace cross-view attention with view averaging, retaining the pooled reconstruction loss. +RR adds random codebook restarts to address collapse.

All variants are evaluated on the same LIBERO-Spatial probe split and paired transition samples, which have both primary and wrist recordings. P denotes primary-only token extraction; All denotes primary-plus-wrist extraction. MAE and global R² are reported as mean ± sample standard deviation over five MLP-probe seeds with fixed LAM checkpoints.

LAMExtraction viewsOverall MAE ↓Global R² ↑
Primary-onlyP0.2695 ± 0.00040.4441 ± 0.0024
JMV-LA-PIP0.2708 ± 0.00070.4426 ± 0.0020
JMV-LAP0.2526 ± 0.00030.4954 ± 0.0018
JMV-LA-PTAll0.2412 ± 0.00140.5240 ± 0.0056
JMV-LA-Mean, without restartsAll0.3543 ± 0.00040.1335 ± 0.0003
JMV-LA-Mean+RRAll0.2474 ± 0.00140.5096 ± 0.0050
JMV-LAAll0.2383 ± 0.00170.5342 ± 0.0062

Reconstruction weighting. Per-transition normalization gives an MAE of 0.2412, versus 0.2695 for Primary-only: approximately 10.5% lower MAE, retaining about 91% of the original absolute clean-MAE improvement based on the displayed means. Thus, most of the clean action-decoding gain remains after removing camera-count-dependent reconstruction weighting. Weighting does affect perturbation stability: under all-input image perturbations, PT has ΔMAE 0.0108, compared with 0.0022 for the original JMV-LA.

Joint transition encoding versus reconstruction alone. PI retains multi-view reconstruction but has MAE 0.2708. JMV-LA achieves 0.2526 even when evaluated from the primary view alone, and 0.2383 with both views. This supports the value of joint multi-view training beyond multi-view reconstruction alone, including when extraction-time inputs are matched.

Cross-attention versus averaging. For the LIBERO policy experiments, JMV-LA supervision targets are constructed from all available views (primary and wrist), making the all-view extraction setting directly relevant to how the targets are used. Averaging without restarts collapses to a constant sequence in our run. With random restarts, Mean+RR achieves an all-view MAE of 0.2474 and an all-view perturbation ΔMAE of 0.0061, compared with 0.2383 and 0.0022, respectively, for JMV-LA with cross-attention and no restarts. These results support an advantage of cross-attention in overall action decodability and readout stability for this two-view supervision setting. Table 8 also reports primary-only extraction, where Mean+RR has lower clean MAE than JMV-LA.

These comparisons match the training trajectory mixture and the multi-view evaluation samples. They do not include separate retraining restricted exclusively to the multi-view portion of the pretraining mixture.

Q2. Predictability of multi-view latent-action targets

Question: Predictability of multi-view latent-action targets: Since the latent-action targets are inferred from future frames and all available camera views, but the VLA predicts them from the primary camera alone, how predictable are these targets from the deployed policy's inputs? Could the authors report latent-token prediction accuracy or cross-entropy stratified by camera availability and camera type? This would help establish whether the additional views provide useful action supervision or introduce target information that is difficult to infer from the primary view.

Purpose and evaluation setup. We assess target predictability from primary-camera inputs using VLAs supervised by the Primary-only LAM and JMV-LA. The Primary-only teacher constructs targets from current and future primary-camera frames; JMV-LA uses synchronized primary and wrist frame pairs. Each VLA is pre-trained with its teacher's targets and fine-tuned separately on LIBERO-Spatial, Object, Goal, and Long. Each VLA receives the current primary image and instruction, without latent history, and generates four tokens (temperature 0.75, top-p 0.9, one generation seed). Predictions are scored against each VLA's teacher labels on the same 85 trajectories and 1,700 frame-pair samples across the four suites: 6,800 tokens per method (Appendix A.6, Table 10).

Prediction and label-distribution metrics. Token accuracy and teacher-forcing cross-entropy (CE) measure prediction quality; teacher-label entropy describes target diversity and concentration. Token accuracy is the fraction of teacher-label matches during generation using the model's preceding outputs. Teacher-forcing CE is the mean negative log-probability of the correct token, conditioned on true preceding tokens and normalized over the full VLA vocabulary; lower is better. Entropy uses each teacher's code frequencies within each suite and token position; higher values indicate more diverse labels. Metrics average four token positions and weight suites by sample counts. CE and entropy are in nats/token.

Label-frequency baselines. Concentrated labels can be predicted accurately without the current observation. We therefore compare each VLA with three classifiers using only its teacher's label frequencies. Majority always predicts the most frequent code; frequency-random samples from code frequencies; matched sampling applies the VLA's temperature and top-p settings to those frequencies. Frequencies are estimated for each teacher, suite, and token position, excluding the trajectory being scored. Sampling-baseline accuracies are analytic expectations. Gains over majority show the VLA's advantage over always guessing the most common code; gains over frequency-random and matched sampling show its advantage over frequency-based sampling without and with the VLA's generation rules. These percentage-point gains contextualize accuracy relative to each teacher's label distribution.

Teacher and corresponding VLATeacher-forcing CEToken accuracy (%)Majority baseline (%)Frequency-random baseline (%)Matched-sampling baseline (%)Suite-weighted teacher entropy
Primary-only1.627255.7438.1826.1832.551.9166
JMV-LA2.001939.1614.8210.9012.312.3522

Label concentration. The two teachers produce differently concentrated label distributions. JMV-LA has higher suite-weighted entropy (2.3522 versus 1.9166 nats/token) and lower majority-baseline accuracy (14.82% versus 38.18%), indicating that its labels are less dominated by a few frequent codes. This pattern is consistent with its higher between-transition token disagreement on LIBERO-Spatial (Table 9).

Prediction results in context. Label concentration affects how accurately a label can be guessed from frequencies alone, so we interpret prediction scores alongside these teacher-specific references. Primary-only has higher raw token accuracy (55.74% versus 39.16%) and lower CE (1.6272 versus 2.0019 nats/token). However, JMV-LA exceeds its majority and matched-sampling baselines by 24.35 and 26.86 percentage points, respectively, compared with 17.56 and 23.18 points for Primary-only. Gains over the frequency-random baseline are close: 28.26 points for JMV-LA and 29.56 for Primary-only. All gains use unrounded values. Thus, despite its lower raw accuracy on the more diverse targets, the JMV-LA-supervised VLA predicts more accurately than its label-only baselines.

This analysis covers LIBERO's fixed primary-plus-wrist teacher configuration. It has not yet been stratified across the naturally occurring one-, two-, and three-camera configurations or camera types in the heterogeneous pretraining mixture.

Q3. Evaluation uncertainty and baseline comparability

Question: Evaluation uncertainty and baseline comparability: Were the main policy results reproduced across multiple training seeds under a common downstream protocol, and can the authors report corresponding uncertainty estimates? For the real-robot evaluations, how sensitive are the comparisons to the sequential rather than interleaved evaluation order, given that the reported Wilson intervals do not account for temporal hardware variation?

Matched downstream protocol. We have added the policy trained with Primary-only LAM targets under the same VLA initialization, pre-training videos and budget, fine-tuning data and settings, action decoder, and latent-history mechanism as JMV-LA. Only the target-generating LAM changes. The resulting comparisons are reported in revised Tables 3–4 and 13:

Evaluation metric (%)Primary-only targetsJMV-LA targets
Mean clean success, four LIBERO suites95.3596.35
Clean LIBERO-10 success93.694.2
Mean perturbed LIBERO-10 success47.3254.76
LIBERO-10 retention50.5658.13
Mean LIBERO-10-Pro success40.1644.20

The matched comparison yields +1.00 percentage point on standard LIBERO, +7.44 points under camera-extrinsic perturbations, and +4.04 points on LIBERO-10-Pro. The perturbation gain occurs under all five conditions and is larger than the 0.6-point clean LIBERO-10 gain. The Pro control covers LIBERO-10 only. External baselines remain complete-system comparisons using published results, official checkpoints, or the stated reproductions; they were not all retrained under a common training protocol.

Training uncertainty. We acknowledge that our current policy results use one training seed per configuration and therefore do not quantify variability across independent policy training runs. The five seeds in Tables 8–9 correspond to MLP-probe initializations with fixed LAM checkpoints, measuring variation in action readout rather than LAM or policy training variability. While the clean LIBERO gains are modest, the matched LIBERO-10 comparison shows a larger observed improvement under camera-extrinsic perturbations (+7.44 percentage points) than under the clean camera (+0.6 points), suggesting that the benefit is more pronounced under the tested camera shifts.

Real-robot evaluation order. The compared policies use the same task demonstrations, predetermined checkpoint budgets, control protocol, and reset procedure. However, evaluation was sequential within each task and setting. The reported Wilson intervals quantify within-condition binomial uncertainty and do not account for temporal hardware variation. We therefore treat order sensitivity as a limitation of the current comparison.

Additional response to W2: Action-readout stability and downstream viewpoint robustness

Comment: In addition, the appendix's token-level diagnostics show similar token disagreement under all-view image perturbations for JMV-LA and the primary-only model (0.31770 versus 0.30685).

Our robustness evidence concerns two levels: preserving decodable action information under image perturbations, and maintaining single-camera policy success under camera-extrinsic shifts. Token identities can change while the representation continues to support accurate action readout.

The following four rows from revised Table 9 compare the same frozen LAMs under primary-only and all-view extraction. P denotes the primary view; All denotes primary plus wrist. MAE values are mean ± sample standard deviation over five clean-trained probes. D_view measures clean–perturbed token disagreement, while D_action measures token disagreement across distinct clean transitions from the same task. For additional context, we report D_view/D_action as a descriptive ratio of perturbation-induced token change to between-transition token variation. Ratios are calculated from the displayed disagreement means.

LAMInput viewsPerturbed viewsClean MAE ↓Perturbed MAE ↓ΔMAE ↓D_viewD_actionD_view/D_action (approx.)
Primary-onlyPP0.2695 ± 0.00040.2758 ± 0.0006+0.0064 ± 0.00040.30690.68380.4488
JMV-LAPP0.2526 ± 0.00030.2581 ± 0.0002+0.0055 ± 0.00030.28250.82160.3438
JMV-LAAllP0.2383 ± 0.00170.2371 ± 0.0009−0.0012 ± 0.00090.14410.89230.1615
JMV-LAAllAll0.2383 ± 0.00170.2405 ± 0.0014+0.0022 ± 0.00080.31770.89230.3560

Matched primary-only input. JMV-LA has lower clean and perturbed action-readout MAE than Primary-only, together with lower D_view (0.2825 versus 0.3069) and a lower D_view/D_action ratio (0.3438 versus 0.4488). Thus, its readout advantage persists with the same available input view, accompanied by less perturbation-induced token change relative to its between-transition variation.

All-view target construction. When only the primary image is perturbed, the unchanged wrist observation provides complementary information, and JMV-LA has lower perturbed MAE and D_view than the same checkpoint with primary-only input. When both input views are perturbed, the raw D_view remains similar to that of the Primary-only LAM, as noted by the reviewer, while the action-readout MAE increase is smaller (0.0022 versus 0.0064), and the D_view/D_action ratio is also lower than that of the Primary-only LAM (0.3560 versus 0.4488). These results support preservation of action information even when individual token identities change. We regard the disagreement ratio as supplementary context, not a measure of physical action distance or geometric invariance.

Learning from targets that preserve decodable action information under visual perturbations may encourage the VLA to learn action-relevant features that help maintain single-camera control under camera-extrinsic shifts. This is a possible explanation for the matched policy results above.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal and the major changes in the revised manuscript

We thank the reviewers for their constructive feedback. In addition to our point-by-point responses, we summarize the main additions to the revised manuscript below. Newly added experimental results are marked in red.

  1. Matched policy-level comparison (Tables 3–4 and 13). We added a VLA supervised by Primary-only LAM targets, using the same initialization, pre-training data and budget, and downstream training settings as the JMV-LA-supervised policy. JMV-LA improves average success by 1.00 percentage point on clean LIBERO, 7.44 points under LIBERO-10 camera-extrinsic perturbations, and 4.04 points on LIBERO-10-Pro. Both policies use only the primary camera at deployment.

  2. Matched-input probes and controlled LAM ablations (Tables 8–9). We compare Primary-only and JMV-LA with four ablations: primary-only transition encoding with multi-view reconstruction (PI), per-transition loss normalization (PT), and mean fusion without or with random codebook restarts (Mean/Mean+RR). At token extraction, we evaluate primary-only and primary-plus-wrist inputs; Primary-only and PI support only primary input. For image perturbations, two-view inputs are tested with either the primary view or both views perturbed, while single-view inputs use primary-view perturbations. JMV-LA achieves a clean overall action-readout MAE of 0.2383 with both views and 0.2526 with primary-only input, compared with 0.2695 for the Primary-only LAM. Its readout advantage therefore persists under matched single-view inputs, and incorporating the wrist view further improves overall action decodability. The remaining ablations examine how joint encoding, fusion, and reconstruction weighting affect action decodability and perturbation stability.

  3. Latent-target predictability analysis (Table 10). We added an offline prediction diagnostic for VLAs supervised by the Primary-only LAM or JMV-LA. Each VLA is trained to predict its respective teacher’s latent-action targets and fine-tuned separately on LIBERO-Spatial, Object, Goal, and Long. The corresponding suite-specific VLAs are evaluated on the same 85 trajectories and 1,700 frame-pair samples, with predictions scored against labels from their respective teachers. We report teacher-forcing cross-entropy, autoregressive token accuracy, three label-frequency baselines, and teacher-label entropy to interpret prediction performance alongside target diversity and concentration.

  4. Clearer supervision and evaluation protocols. We explicitly state that JMV-LA supervision targets use all available views, while policy deployment remains single-camera. We also clarify that the five-seed probe statistics measure MLP initialization variability with fixed LAM checkpoints.

Detailed results and protocols are included in the revised manuscript, with further interpretation and discussion of scope in our responses to each reviewer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.