- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents CARVE, a latent action model for learning from synchronized multi-view videos without action labels during representation pretraining. Its central premise is that different cameras provide complementary evidence of the same physical transition, so enforcing complete view invariance may discard useful information. CARVE decomposes each view’s transition representation into cross-view global and view-specific private components. To predict a target view, its dynamics model uses the global components from all views and the private component from that target view. The model is trained in three stages: single-view pretraining, motion-amplitude-guided adaptation across cameras, and synchronized multi-view decomposition. The authors evaluate CARVE on 15-step video prediction, linear action probing, LIBERO behavior cloning, and VP2 visual planning. They report improved prediction metrics and a 71.50% mean success rate on LIBERO, compared with 62.00% for the strongest baseline in their table. Ablations support the contributions of the sampling strategy and global–private design. Downstream control still uses physical-action supervision, and the method assumes synchronized, fixed camera views.
Critical strengths
- The paper argues that, compared with requiring the entire latent action representation to be fully consistent across views in a multi-view setting, preserving useful view-specific cues while sharing useful information across views is a more reasonable research direction. The motivation is clear and reasonable.
- The global–private decomposition and asymmetric routing are logically coherent: each target-view predictor receives all global components and only its matched private component.
- The most impressive aspect is the amplitude-guided sampling strategy, which selects temporal horizons based on observed feature changes and progressively moves from smaller to larger motion amplitudes.
- The three-stage training procedure progresses naturally from single-view pretraining to multi-view adaptation and finally synchronized decomposition and fusion.
- The paper provides systematic ablations of sampling, decomposition, training objectives, and routing, supporting their effectiveness in video reconstruction
Critical weaknesses
-
The decomposition objectives do not fully enforce the intended role of private components.In Eq. (5), the matched-private gain loss can be zero even when adding the private component does not improve reconstruction. The cross-view constraint similarly permits equal reconstruction errors. Although the ablations suggest that the private branch is useful, the objectives themselves do not rule out solutions in which it contributes little. Reporting the reconstruction errors with matched, mismatched, and absent private components would help clarify whether the intended functional separation is achieved.
-
The practical control benefits need stronger validation. The paper includes encouraging simulated control results on LIBERO and VP2, but no real-world manipulation experiments. Moreover, the key component ablations primarily evaluate visual reconstruction. Better reconstruction provides useful but indirect evidence of improved action generation and closed-loop control. Evaluating these components through task success rates, particularly under challenging physical interactions, would make the practical value of the learned representations more convincing.
-
The action-probing comparisons do not clearly isolate cross-view complementarity.According to Eq. (2), each global component is computed from jointly processed multi-view inputs. Using only therefore does not necessarily constitute a front-camera-only baseline. Concatenating additional components also increases feature dimensionality and probe capacity. The reported improvements consequently cannot cleanly distinguish additional camera evidence from additional output features. The paper should clarify the camera inputs used in each setting and provide input-level view ablations with matched representation dimensions.
-
Some baseline comparisons require better control of experimental conditions.The VP2 baseline results are taken from separate publications, without establishing that adaptation procedures and planning budgets are identical. These results provide useful context, but offer limited evidence of superiority under a shared protocol. The reconstruction experiments also omit closely related multi-view methods such as MVP-LAM and MuCoLA. Comparisons under matched training data and evaluation settings, or a clear explanation of any incompatibilities, would strengthen the empirical conclusions.
Questions for authors
- I appreciate the paper’s novelty and theoretical motivation. However, my main concern is the absence of real-world manipulation experiments, which limits confidence that the reported improvements translate into physical robot control. Could the authors provide a small-scale real-world evaluation, even using a platform such as SO-101, on a few basic manipulation tasks with a relevant baseline?
- I acknowledge that Table 5 provides ablation evidence for the private branch and the decomposition losses. However, could the authors clarify the margin in the gain loss in Eq. (5)? This loss can be zero even when the matched private component provides no improvement over the global-only reconstruction. Could the authors report the held-out joint, global-only, and mismatched-private reconstruction errors defined in Eq. (4) for the same trained model, or provide a well-founded theoretical explanation of how the objective encourages useful private components? I would consider raising my score if this concern is convincingly addressed.
- Could the authors clarify whether Table 3 changes the actual camera inputs or only the global components given to the probe, and explain how feature dimensionality is controlled? Alternatively, could they provide camera-input ablations with matched feature dimensions?
- Could the authors clarify the comparability of the VP2 baseline settings and explain the absence of MVP-LAM and MuCoLA from the reconstruction comparison, or supplement these comparisons under matched conditions?
13 comments
W2&Q1: Real-World Manipulation and Downstream Policy abalations
[W2&Q1: Real-World Manipulation and Downstream Policy abalations]
R2: We sincerely thank the reviewer for this constructive suggestion. To address this concern, we provide two additional evaluations: real-world robotic manipulation experiments and simulation-based policy success rate ablations.
2.1. Additional Real-world Robotic Experiments
We follow your suggestion by conducting additional real-world experiments on a ARX5 robotic arm equipped with two Intel RealSense D435 cameras. We evaluate three manipulation tasks involving different objects and physical interactions: (1) picking up a sponge and placing it on a plate, (2) removing a batten from a box and placing it on a plate, and (3) placing a rope on a plate.
Policy Setup. Following our LIBERO evaluation protocol, we adopt a BCTransformer-based policy for real-world manipulation. Specifically, we first use the frozen CARVE model to extract global and private latent actions from synchronized dual-view demonstration videos, which serve as supervision for policy learning. The policy takes dual-view RGB observations, robot proprioceptive states, and task embeddings as inputs to predict latent action chunks. A lightweight two-layer MLP is trained to decode latent actions into physical robot actions. During deployment, the policy operates in a closed-loop manner, executing the first predicted action and replanning based on updated observations.
For comparison, we additionally evaluate Direct BC, LAPA and DiLA under the same demonstration data and policy training protocol. Direct BC directly predicts physical actions without latent action supervision, while LAPA and DiLA follows the same policy learning pipeline using its own learned representations.
Experimental Results. We report the task success rates across three real-world manipulation tasks. Each method is evaluated over 40 trials per task, with success rates reported as percentages. The results are summarized below:
| Method | Sponge | Batten | Rope | Average |
|---|---|---|---|---|
| Direct BC | 55.0% | 47.5% | 37.5% | 46.67% |
| LAPA | 62.5% | 42.5% | 32.5% | 45.83% |
| DiLA | 57.5% | 55.0% | 42.5% | 51.67% |
| CARVE (ours) | 70.0% | 77.5% | 50.0% | 65.83% |
As shown in the table, CARVE achieves the highest success rates across all three real-world manipulation tasks, with an average success rate of 66.67%, outperforming Direct BC (46.67%), LAPA (45.00%), and DiLA (51.67%). These results extend our evaluation beyond simulation benchmarks and provide direct evidence of CARVE's practical advantages over existing latent action models in physical robotic manipulation. We additionally provide demonstration videos of all three tasks in the supplementary material.
2.2. Additional Simulation-based Policy Ablation Experiments
We appreciate the reviewer's insightful suggestion and agree that evaluating component contributions through downstream control performance would provide further evidence of their practical effectiveness beyond visual reconstruction.
To address this issue, we conduct additional policy success rate ablations on the four LIBERO suites. Specifically, we evaluate three controlled variants: (1) removing the global-private decomposition and directly fusing multi-view latent actions, (2) removing Stage I single-view pretraining, (3) replacing amplitude-guided sampling with uniform sampling. Each variant follows the same behavior cloning architecture, demonstration data, training schedule, and evaluation protocol as the full CARVE model.
| Method | Long | Spatial | Object | Goal | Mean |
|---|---|---|---|---|---|
| Action Fusion | 51.00 | 62.33 | 75.67 | 72.00 | 65.25 |
| w/o stage1 | 48.67 | 60.33 | 73.00 | 69.67 | 62.92 |
| Uniform Sampling | 54.00 | 64.33 | 77.33 | 74.00 | 67.42 |
| CARVE (Full) | 58.67 | 67.33 | 81.33 | 78.67 | 71.50 |
As shown in the table, CARVE achieves the highest success rates across all four LIBERO suites, with an average success rate of 71.50%, outperforming Action Fusion (65.25%), w/o Stage I (62.92%), and Uniform Sampling (67.42%). These results demonstrate that the benefits of global-private decomposition, single-view pretraining, and amplitude-guided sampling extend beyond visual reconstruction to downstream control performance. Together with the real-world experiments, they further validate the effectiveness of CARVE and its key design choices in robotic manipulation.
W3&Q3: Cross-View Complementarity under Controlled Camera Inputs and Feature Dimensions
[W3&Q3: Cross-View Complementarity under Controlled Camera Inputs and Feature Dimensions]
R3: We sincerely thank the reviewer for pointing out these important issues. We address the two concerns separately by clarifying the original evaluation protocol in Table 3 and providing additional input-level camera ablations with controlled feature dimensions.
Clarification of Table 3. First we would like to clarify that all three synchronized camera views are provided as inputs to CARVE across all settings in the original action-probing experiments. As defined in Eq. (2) of the main text, the global components are extracted through joint multi-view processing. Each global component is 16-dimensional, resulting in 16-, 32-, and 48-dimensional probe inputs when using one, two, or three components, respectively.
Additional input-level camera ablations. To further address this concern, we conduct additional input-level camera ablations on DROID Formal300, where the number of camera observations is controlled from the model input. Specifically, we consider three configurations:
-
Single-view (Front): Only front-camera observations are provided. The model directly extracts latent actions through the single-view spatio-temporal backbone, without passing through the global-private decomposition pathway.
-
Two-view (Front + Wrist): The model receives synchronized front and wrist observations and performs global-private decomposition and cross-view fusion using only these two views.
-
Three-view (Front + Side + Wrist): The complete CARVE architecture receives all three synchronized camera observations and performs global-private decomposition and fusion.
All configurations are trained and evaluated using the same data splits and evaluation protocol, with comparable training budgets. To control for representation dimensionality, we use the original 16-dimensional latent action for the single-view model and average the available view-indexed global components for the multi-view models, producing a fixed 16-dimensional representation in all settings. The same linear probing protocol is then applied to predict physical robot actions.
| Camera Inputs | Feature Dim. | RMSE | MAE |
|---|---|---|---|
| Front | 16 | 0.275 | 0.193 |
| Front + Wrist | 16 | 0.257 | 0.181 |
| Front + Side + Wrist | 16 | 0.246 | 0.173 |
As shown in the table, incorporating additional camera observations consistently improves action regression performance, reducing RMSE from 0.275 to 0.246 and MAE from 0.193 to 0.173. In particular, the three-view configuration outperforms the two-view configuration under the same global-private architecture and fixed 16-dimensional representation. These results demonstrate that additional synchronized camera observations provide complementary information beyond simply increasing feature dimensionality.
W1&Q2: Clarification and Empirical Validation of the Decomposition Objectives
[W1&Q2: Clarification and Empirical Validation of the Decomposition Objectives]
R1: We sincerely thank the reviewer for this insightful comment. We agree that the intended functional separation between global and private components requires more direct validation. To address this concern, we would like to first clarify the design of our decomposition objectives and then provide additional quantitative evidence from the same trained model.
1.1. Clarification and Correction of Decomposition Objectives
First, we would like to kindly clarify that this is a misunderstanding stems from typographical errors in Equation (5) of the original manuscript, and we sincerely apologize for the resulting confusion. Specifically, the margin terms in both Lgain and Lcross were mistakenly written as −ϵ instead of the intended +ϵ. We sincerely appreciate the reviewer for bringing this issue to our attention. We sincerely apologize for this oversight and have corrected both terms in the revised manuscript. The corrected gain and cross-view losses are:
Lgain=V1∑v=1V[Ejointv−Eglobalv+ϵ]+.
Lcross=V(V−1)1∑v=1V∑u=v[Eglobalv−Ecrossv←u+ϵ]+.
With these corrections, minimizing the gain loss explicitly encourages Ejointv≤Eglobalv−ϵ, requiring the matched private component to provide a positive reconstruction improvement over the global-only representation. Likewise, the corrected cross-view loss encourages Ecrossv←u≥Eglobalv+ϵ, so that mismatched private components do not improve reconstruction.
The three objectives serve complementary purposes. Lrec minimizes both joint and global-only reconstruction errors, discouraging trivial satisfaction of the gain constraint by degrading global-only reconstruction. Lgain encourages the matched private component to contribute additional information beyond the global representation. Meanwhile, Lcross encourages mismatched private components to yield reconstruction errors at least ϵ higher than global-only reconstruction, discouraging the use of private information from an unmatched view.
1.2. Quantitative Validation of Private-Component Complementarity
To directly examine whether the intended functional separation is achieved, we conduct an additional evaluation on the held-out DROID Formal300 dataset using the same fully trained CARVE model. Following Eq. (4), we compare four reconstruction settings:
Matched Private: Joint reconstruction using global components and the corresponding target-view private component.
Global-only: Reconstruction using only global components through the global-only decoder.
Mismatched Private: Reconstruction using global components and a private component from another synchronized view.
We report the reconstruction MSE for each setting. We additionally measure the proportion of transitions where the matched private component reduces reconstruction error compared with the global-only representation.
As shown in the table, the matched private components achieve the lowest reconstruction MSE across all three views. Moreover, matched private components improve reconstruction in 91.1% of transitions. These results demonstrate that matched private components provide complementary reconstruction information beyond the global representation, while mismatched private components do not offer the same benefit.