acceptodds
Under review as a conference paper at ICLR 2027

CARVE: Learning Latent Actions under Cross-View Visual Heterogeneity

Abstract

Latent action models learn action-centric dynamics from unlabeled robot videos, but existing methods predominantly operate on a single camera stream. Recent multi-view formulations instead emphasize view invariance, overlooking that synchronized cameras provide complementary but not interchangeable evidence about the same physical action. We introduce CARVE (Cross-view Action Representation with View-specific Evidence), a latent action framework that selectively shares transition information across synchronized views. For each target view, CARVE forms global components from synchronized multi-view context and retains target-view private components. A shared future-dynamics model routes global information across target views while restricting private information to its corresponding predictor. Because synchronized multi-view videos are substantially less abundant than single-view recordings, CARVE uses a progressive three-stage procedure that combines single-view pretraining, amplitude-guided multi-view adaptation, and global–private decomposition. CARVE consistently improves long-horizon video prediction and downstream robot control over strong latent action baselines. Representation analyses and ablations further show that selective cross-view sharing captures complementary action information while preserving target-specific transition cues.

Primary area:
applications to robotics, autonomy, planning
Keywords:
latent action, world models, multi-view learning, robot manipulation, representation learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cNhZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CARVE, a latent action model for learning from synchronized multi-view videos without action labels during representation pretraining. Its central premise is that different cameras provide complementary evidence of the same physical transition, so enforcing complete view invariance may discard useful information. CARVE decomposes each view’s transition representation into cross-view global and view-specific private components. To predict a target view, its dynamics model uses the global components from all views and the private component from that target view. The model is trained in three stages: single-view pretraining, motion-amplitude-guided adaptation across cameras, and synchronized multi-view decomposition. The authors evaluate CARVE on 15-step video prediction, linear action probing, LIBERO behavior cloning, and VP2 visual planning. They report improved prediction metrics and a 71.50% mean success rate on LIBERO, compared with 62.00% for the strongest baseline in their table. Ablations support the contributions of the sampling strategy and global–private design. Downstream control still uses physical-action supervision, and the method assumes synchronized, fixed camera views.

Critical strengths
  1. The paper argues that, compared with requiring the entire latent action representation to be fully consistent across views in a multi-view setting, preserving useful view-specific cues while sharing useful information across views is a more reasonable research direction. The motivation is clear and reasonable.
  2. The global–private decomposition and asymmetric routing are logically coherent: each target-view predictor receives all global components and only its matched private component.
  3. The most impressive aspect is the amplitude-guided sampling strategy, which selects temporal horizons based on observed feature changes and progressively moves from smaller to larger motion amplitudes.
  4. The three-stage training procedure progresses naturally from single-view pretraining to multi-view adaptation and finally synchronized decomposition and fusion.
  5. The paper provides systematic ablations of sampling, decomposition, training objectives, and routing, supporting their effectiveness in video reconstruction
Critical weaknesses
  1. The decomposition objectives do not fully enforce the intended role of private components.In Eq. (5), the matched-private gain loss can be zero even when adding the private component does not improve reconstruction. The cross-view constraint similarly permits equal reconstruction errors. Although the ablations suggest that the private branch is useful, the objectives themselves do not rule out solutions in which it contributes little. Reporting the reconstruction errors with matched, mismatched, and absent private components would help clarify whether the intended functional separation is achieved.

  2. The practical control benefits need stronger validation. The paper includes encouraging simulated control results on LIBERO and VP2, but no real-world manipulation experiments. Moreover, the key component ablations primarily evaluate visual reconstruction. Better reconstruction provides useful but indirect evidence of improved action generation and closed-loop control. Evaluating these components through task success rates, particularly under challenging physical interactions, would make the practical value of the learned representations more convincing.

  3. The action-probing comparisons do not clearly isolate cross-view complementarity.According to Eq. (2), each global component is computed from jointly processed multi-view inputs. Using only GfG_f therefore does not necessarily constitute a front-camera-only baseline. Concatenating additional components also increases feature dimensionality and probe capacity. The reported improvements consequently cannot cleanly distinguish additional camera evidence from additional output features. The paper should clarify the camera inputs used in each setting and provide input-level view ablations with matched representation dimensions.

  4. Some baseline comparisons require better control of experimental conditions.The VP2 baseline results are taken from separate publications, without establishing that adaptation procedures and planning budgets are identical. These results provide useful context, but offer limited evidence of superiority under a shared protocol. The reconstruction experiments also omit closely related multi-view methods such as MVP-LAM and MuCoLA. Comparisons under matched training data and evaluation settings, or a clear explanation of any incompatibilities, would strengthen the empirical conclusions.

Questions for authors
  1. I appreciate the paper’s novelty and theoretical motivation. However, my main concern is the absence of real-world manipulation experiments, which limits confidence that the reported improvements translate into physical robot control. Could the authors provide a small-scale real-world evaluation, even using a platform such as SO-101, on a few basic manipulation tasks with a relevant baseline?
  2. I acknowledge that Table 5 provides ablation evidence for the private branch and the decomposition losses. However, could the authors clarify the margin in the gain loss in Eq. (5)? This loss can be zero even when the matched private component provides no improvement over the global-only reconstruction. Could the authors report the held-out joint, global-only, and mismatched-private reconstruction errors defined in Eq. (4) for the same trained model, or provide a well-founded theoretical explanation of how the objective encourages useful private components? I would consider raising my score if this concern is convincingly addressed.
  3. Could the authors clarify whether Table 3 changes the actual camera inputs or only the global components given to the probe, and explain how feature dimensionality is controlled? Alternatively, could they provide camera-input ablations with matched feature dimensions?
  4. Could the authors clarify the comparability of the VP2 baseline settings and explain the absence of MVP-LAM and MuCoLA from the reconstruction comparison, or supplement these comparisons under matched conditions?
13 comments
AuthorsOpenReview ↗

W1&Q2: Clarification and Empirical Validation of the Decomposition Objectives

[W1&Q2: Clarification and Empirical Validation of the Decomposition Objectives]

R1: We sincerely thank the reviewer for this insightful comment. We agree that the intended functional separation between global and private components requires more direct validation. To address this concern, we would like to first clarify the design of our decomposition objectives and then provide additional quantitative evidence from the same trained model.

1.1. Clarification and Correction of Decomposition Objectives

First, we would like to kindly clarify that this is a misunderstanding stems from typographical errors in Equation (5) of the original manuscript, and we sincerely apologize for the resulting confusion. Specifically, the margin terms in both Lgain\mathcal{L}_{\mathrm{gain}} and Lcross\mathcal{L}_{\mathrm{cross}} were mistakenly written as −ϵ-\epsilon instead of the intended +ϵ+\epsilon. We sincerely appreciate the reviewer for bringing this issue to our attention. We sincerely apologize for this oversight and have corrected both terms in the revised manuscript. The corrected gain and cross-view losses are:

Lgain=1V∑v=1V[Ejointv−Eglobalv+ϵ]+.\mathcal{L}_{\mathrm{gain}} = \frac{1}{V}\sum_{v=1}^{V} \left[ E_{\mathrm{joint}}^{v} - E_{\mathrm{global}}^{v} + \epsilon \right]_{+}.

Lcross=1V(V−1)∑v=1V∑u≠v[Eglobalv−Ecrossv←u+ϵ]+.\mathcal{L}_{\mathrm{cross}} = \frac{1}{V(V-1)}\sum_{v=1}^{V}\sum_{u\ne v} \left[ E_{\mathrm{global}}^{v} - E_{\mathrm{cross}}^{v\leftarrow u} + \epsilon \right]_{+}.

With these corrections, minimizing the gain loss explicitly encourages Ejointv≤Eglobalv−ϵE_{\mathrm{joint}}^{v} \leq E_{\mathrm{global}}^{v} - \epsilon, requiring the matched private component to provide a positive reconstruction improvement over the global-only representation. Likewise, the corrected cross-view loss encourages Ecrossv←u≥Eglobalv+ϵE_{\mathrm{cross}}^{v\leftarrow u} \geq E_{\mathrm{global}}^{v} + \epsilon, so that mismatched private components do not improve reconstruction.

The three objectives serve complementary purposes. Lrec\mathcal{L}_{\mathrm{rec}} minimizes both joint and global-only reconstruction errors, discouraging trivial satisfaction of the gain constraint by degrading global-only reconstruction. Lgain\mathcal{L}_{\mathrm{gain}} encourages the matched private component to contribute additional information beyond the global representation. Meanwhile, Lcross\mathcal{L}_{\mathrm{cross}} encourages mismatched private components to yield reconstruction errors at least ϵ\epsilon higher than global-only reconstruction, discouraging the use of private information from an unmatched view.

1.2. Quantitative Validation of Private-Component Complementarity

To directly examine whether the intended functional separation is achieved, we conduct an additional evaluation on the held-out DROID Formal300 dataset using the same fully trained CARVE model. Following Eq. (4), we compare four reconstruction settings:

  • Matched Private: Joint reconstruction using global components and the corresponding target-view private component.

  • Global-only: Reconstruction using only global components through the global-only decoder.

  • Mismatched Private: Reconstruction using global components and a private component from another synchronized view.

We report the reconstruction MSE for each setting. We additionally measure the proportion of transitions where the matched private component reduces reconstruction error compared with the global-only representation.

ViewMatched ↓\downarrowGlobal ↓\downarrowMismatched ↓\downarrowImproved (%) ↑\uparrow
Front0.02180.03020.032091.4%
Side0.02260.03100.032889.8%
Wrist0.02090.02950.031292.1%
Overall0.02180.03020.032091.1%

As shown in the table, the matched private components achieve the lowest reconstruction MSE across all three views. Moreover, matched private components improve reconstruction in 91.1% of transitions. These results demonstrate that matched private components provide complementary reconstruction information beyond the global representation, while mismatched private components do not offer the same benefit.

AuthorsOpenReview ↗

W2&Q1: Real-World Manipulation and Downstream Policy abalations

[W2&Q1: Real-World Manipulation and Downstream Policy abalations]

R2: We sincerely thank the reviewer for this constructive suggestion. To address this concern, we provide two additional evaluations: real-world robotic manipulation experiments and simulation-based policy success rate ablations.

2.1. Additional Real-world Robotic Experiments

We follow your suggestion by conducting additional real-world experiments on a ARX5 robotic arm equipped with two Intel RealSense D435 cameras. We evaluate three manipulation tasks involving different objects and physical interactions: (1) picking up a sponge and placing it on a plate, (2) removing a batten from a box and placing it on a plate, and (3) placing a rope on a plate.

Policy Setup. Following our LIBERO evaluation protocol, we adopt a BCTransformer-based policy for real-world manipulation. Specifically, we first use the frozen CARVE model to extract global and private latent actions from synchronized dual-view demonstration videos, which serve as supervision for policy learning. The policy takes dual-view RGB observations, robot proprioceptive states, and task embeddings as inputs to predict latent action chunks. A lightweight two-layer MLP is trained to decode latent actions into physical robot actions. During deployment, the policy operates in a closed-loop manner, executing the first predicted action and replanning based on updated observations.

For comparison, we additionally evaluate Direct BC, LAPA and DiLA under the same demonstration data and policy training protocol. Direct BC directly predicts physical actions without latent action supervision, while LAPA and DiLA follows the same policy learning pipeline using its own learned representations.

Experimental Results. We report the task success rates across three real-world manipulation tasks. Each method is evaluated over 40 trials per task, with success rates reported as percentages. The results are summarized below:

MethodSponge ↑\uparrowBatten ↑\uparrowRope ↑\uparrowAverage ↑\uparrow
Direct BC55.0%47.5%37.5%46.67%
LAPA62.5%42.5%32.5%45.83%
DiLA57.5%55.0%42.5%51.67%
CARVE (ours)70.0%77.5%50.0%65.83%

As shown in the table, CARVE achieves the highest success rates across all three real-world manipulation tasks, with an average success rate of 66.67%, outperforming Direct BC (46.67%), LAPA (45.00%), and DiLA (51.67%). These results extend our evaluation beyond simulation benchmarks and provide direct evidence of CARVE's practical advantages over existing latent action models in physical robotic manipulation. We additionally provide demonstration videos of all three tasks in the supplementary material.

2.2. Additional Simulation-based Policy Ablation Experiments

We appreciate the reviewer's insightful suggestion and agree that evaluating component contributions through downstream control performance would provide further evidence of their practical effectiveness beyond visual reconstruction.

To address this issue, we conduct additional policy success rate ablations on the four LIBERO suites. Specifically, we evaluate three controlled variants: (1) removing the global-private decomposition and directly fusing multi-view latent actions, (2) removing Stage I single-view pretraining, (3) replacing amplitude-guided sampling with uniform sampling. Each variant follows the same behavior cloning architecture, demonstration data, training schedule, and evaluation protocol as the full CARVE model.

MethodLongSpatialObjectGoalMean
Action Fusion51.0062.3375.6772.0065.25
w/o stage148.6760.3373.0069.6762.92
Uniform Sampling54.0064.3377.3374.0067.42
CARVE (Full)58.6767.3381.3378.6771.50

As shown in the table, CARVE achieves the highest success rates across all four LIBERO suites, with an average success rate of 71.50%, outperforming Action Fusion (65.25%), w/o Stage I (62.92%), and Uniform Sampling (67.42%). These results demonstrate that the benefits of global-private decomposition, single-view pretraining, and amplitude-guided sampling extend beyond visual reconstruction to downstream control performance. Together with the real-world experiments, they further validate the effectiveness of CARVE and its key design choices in robotic manipulation.

AuthorsOpenReview ↗

W3&Q3: Cross-View Complementarity under Controlled Camera Inputs and Feature Dimensions

[W3&Q3: Cross-View Complementarity under Controlled Camera Inputs and Feature Dimensions]

R3: We sincerely thank the reviewer for pointing out these important issues. We address the two concerns separately by clarifying the original evaluation protocol in Table 3 and providing additional input-level camera ablations with controlled feature dimensions.

Clarification of Table 3. First we would like to clarify that all three synchronized camera views are provided as inputs to CARVE across all settings in the original action-probing experiments. As defined in Eq. (2) of the main text, the global components are extracted through joint multi-view processing. Each global component is 16-dimensional, resulting in 16-, 32-, and 48-dimensional probe inputs when using one, two, or three components, respectively.

Additional input-level camera ablations. To further address this concern, we conduct additional input-level camera ablations on DROID Formal300, where the number of camera observations is controlled from the model input. Specifically, we consider three configurations:

  • Single-view (Front): Only front-camera observations are provided. The model directly extracts latent actions through the single-view spatio-temporal backbone, without passing through the global-private decomposition pathway.

  • Two-view (Front + Wrist): The model receives synchronized front and wrist observations and performs global-private decomposition and cross-view fusion using only these two views.

  • Three-view (Front + Side + Wrist): The complete CARVE architecture receives all three synchronized camera observations and performs global-private decomposition and fusion.

All configurations are trained and evaluated using the same data splits and evaluation protocol, with comparable training budgets. To control for representation dimensionality, we use the original 16-dimensional latent action for the single-view model and average the available view-indexed global components for the multi-view models, producing a fixed 16-dimensional representation in all settings. The same linear probing protocol is then applied to predict physical robot actions.

Camera InputsFeature Dim.RMSE ↓\downarrowMAE ↓\downarrow
Front160.2750.193
Front + Wrist160.2570.181
Front + Side + Wrist160.2460.173

As shown in the table, incorporating additional camera observations consistently improves action regression performance, reducing RMSE from 0.275 to 0.246 and MAE from 0.193 to 0.173. In particular, the three-view configuration outperforms the two-view configuration under the same global-private architecture and fixed 16-dimensional representation. These results demonstrate that additional synchronized camera observations provide complementary information beyond simply increasing feature dimensionality.

Reviewer_pN48OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CARVE, a framework for latent action models that is designed for the synchronized view-to-action prediction task. The key insight is that instead of extracting shared/view-invariant evidence, the framework forms global components for context and retains individual view components for complementary evidence. A three-stage adaptation strategy is proposed to transfer a single-view latent action backbone to multi-view fusion. Experiments are conducted on tasks including long-horizon video reconstruction, action prediction, and visual planning.

Critical strengths
  • The proposed method is well designed, and all components are connected together logically. For example, the global and private decoupling architecture is completed with a proper loss objective to ensure the functional decomposition (Eq. 5); the progressive three-stage training is designed to leverage the cross-view information while overcoming the limited data from synchronized views. I found this overall design interesting and insightful.
  • The design choices of the proposed framework are thoroughly tested in the experimental section: e.g., global vs. private components decomposition, sampling strategy. The results successfully validate the key components.
Critical weaknesses
  • While the experimental section shows competitive results, it is a bit underwhelming (e.g., Table 3). Or maybe the proposed method should be better evaluated against other methods in the target scenario: abundant single-view and limited multi-view data. This is the main motivation and central claim of the paper, and it needs to be validated properly.
Questions for authors
  1. One of the main motivations is that we could have abundant single-view and limited multi-view data. But it seems to me that Sections 4.1 and 4.2 do not consider this setting explicitly. Can the authors clarify how limited (synchronized) multiview data play a role in the framework (Eqs. 2-10)?
  2. Lines 186- 187 say: "A fully shared latent action across views may suppress informative view-specific cues, whereas independent per-view latent actions fail to exploit complementary cross-view shared evidence." Can the authors share more evidence for this claim?
6 comments
AuthorsOpenReview ↗

Q1: Role of Single-View and Multi-View Data in Progressive Training

[Q1: Role of Single-View and Multi-View Data in Progressive Training]

R1: We sincerely thank the reviewer for this important question. We would like to clarify that CARVE is designed to alleviate the dependence on synchronized multi-view data through a progressive three-stage training strategy:

  • Stage I: Leverages abundant front-view videos to learn general action-centric representations and fundamental dynamics prediction capabilities.

  • Stage II: Since Stage I is trained exclusively on front-view data, the pretrained backbone has not yet adapted to heterogeneous camera viewpoints. Stage II therefore leverages limited multi-view data and our amplitude-guided sampling strategy to adapt the backbone to different camera viewpoints and motion distributions.

  • Stage III: While Stages I and II establish fundamental action representations across different camera viewpoints, they do not yet explicitly fuse synchronized cross-view information. Stage III therefore introduces synchronized multi-view observations and our global-private decomposition architecture to learn complementary cross-view fusion through asymmetric information routing.

In summary, Stage I learns from abundant front-view videos, Stage II adapts the backbone using limited multi-view data without requiring synchronization, and Stage III utilizes synchronized multi-view observations for global-private decomposition and fusion. This progressive training strategy reduces the dependence on synchronized multi-view data while effectively leveraging complementary information across heterogeneous camera views.

AuthorsOpenReview ↗

Q2: Effectiveness of Global-Private Decomposition over Fully Shared and Independent Representations

[Q2: Effectiveness of Global-Private Decomposition over Fully Shared and Independent Representations]

R2: We thank the reviewer for this valuable suggestion. We agree that the motivation for global-private decomposition would benefit from more direct evidence. To further substantiate our claim, we conduct two additional controlled experiments that examine the two extreme strategies of multi-view latent action learning.

  • Fully Shared Latent: All three synchronized views are jointly encoded into a single shared latent action representation, which is then used for predicting the future states of all three views.

  • Independent Per-view Latents: Each view independently extracts its own latent action representation and predicts its future state without any cross-view fusion or information exchange.

Both variants are trained under the same data availability, training schedule, and evaluation protocols as CARVE, with architectural settings kept consistent wherever applicable. We evaluate their effectiveness through action regression and 15-step video reconstruction on DROID Formal300.

MethodAction Reg. RMSE ↓\downarrowReconstruction PSNR ↑\uparrowReconstruction SSIM ↑\uparrowReconstruction LPIPS ↓\downarrow
Fully Shared Latent0.252816.300.5480.296
Independent Per-view0.263516.120.5370.306
CARVE (ours)0.231217.010.5750.269

As shown in the table, CARVE achieves the best performance in both action regression (RMSE of 0.2312) and reconstruction (PSNR of 17.01, SSIM of 0.575, and LPIPS of 0.269), consistently outperforming both variants. These results demonstrate the limitations of both extreme strategies. The fully shared representation underperforms CARVE, suggesting that enforcing a single common latent action across heterogeneous views may discard informative view-specific transition cues. Meanwhile, independently learned per-view representations also yield inferior performance, indicating that modeling each view in isolation fails to fully exploit complementary information from synchronized observations.

In contrast, CARVE achieves better performance by jointly modeling cross-view reusable information through global components while preserving target-specific transition cues through private components. These findings provide additional support for the proposed global-private decomposition and its asymmetric information-sharing mechanism.

We thank the reviewer again for this suggestion, which has helped us strengthen the justification of our architectural design.

AuthorsOpenReview ↗

W1: Performance Evaluation under Limited Multi-View Data

[W1: Performance Evaluation under Limited Multi-View Data]

R3: We sincerely appreciate the reviewer for this insightful comment. We agree that evaluating CARVE under abundant single-view but limited multi-view data is essential to substantiate our central motivation. To address this concern, we further clarify our progressive training strategy and provide additional experiments on multi-view data efficiency and real-world robotic manipulation.

1.1. Data Efficiency under Limited Multi-View Data

As detailed in Appendix A, our training data comprises 103,417 single-view videos and 19,001 synchronized multi-view trajectories, with the latter accounting for only 18.4% of the single-view data volume. This represents a setting with substantially limited multi-view data, under which CARVE achieves strong performance. To more effectively demonstrate the advantage of our method under limited multi-view training data, we further conduct additional experiments by reducing the amount of multi-view training data to 9.2%, 4.6%, and 1.8% of the single-view data volume, while keeping the single-view pretraining data unchanged. We compare CARVE with variants without Stage I or Stage II, as well as DiLA, MVP-LAM, and MuCoLA trained from scratch using the same limited multi-view training subsets. The competing methods follow their respective training protocols without additional single-view pretraining, and all methods are evaluated on the same test data under consistent evaluation settings.

MethodMetric1.8%4.6%9.2%18.4%
DiLARMSE ↓\downarrow0.3160.2890.2760.253
PSNR ↑\uparrow15.6916.0416.2716.58
MVP-LAMRMSE ↓\downarrow0.3050.2790.2650.247
PSNR ↑\uparrow15.8816.1716.5116.72
MuCoLARMSE ↓\downarrow0.2950.2730.2490.243
PSNR ↑\uparrow16.0516.3516.4816.82
CARVERMSE ↓\downarrow0.2630.2480.2390.231
PSNR ↑\uparrow16.3516.6416.8417.01

As shown in the table, CARVE consistently outperforms advanced methods across all data ratios. The additional comparisons with scratch-trained baselines further examine whether CARVE's progressive training strategy provides advantages over existing latent-action methods under limited multi-view supervision. Even with only 1.8% multi-view data, CARVE maintains strong performance, achieving an RMSE of 0.263 and a PSNR of 16.35. These results demonstrate that CARVE remains effective under increasingly imbalanced single-view and multi-view data settings.

1.2. Real-World Robotic Manipulation Evaluation

Beyond the existing benchmarks, we further evaluate CARVE on a real-world ARX5 robotic arm equipped with two Intel RealSense D435 cameras. We consider three manipulation tasks involving different objects and interaction patterns: (1) picking up a sponge and placing it on a plate, (2) removing a batten from a box and placing it on a plate, and (3) placing a rope on a plate.

Policy Setup. Following our LIBERO evaluation protocol, we adopt a BCTransformer-based policy for real-world manipulation. Specifically, we first use the frozen CARVE model to extract global and private latent actions from synchronized dual-view demonstration videos, which serve as supervision for policy learning. The policy takes dual-view RGB observations, robot proprioceptive states, and task embeddings as inputs to predict latent action chunks. A lightweight two-layer MLP is trained to decode latent actions into physical robot actions. During deployment, the policy operates in a closed-loop manner, executing the first predicted action and replanning based on updated observations. For comparison, we additionally train a Direct BC baseline using the same policy architecture and demonstrations, but directly predicting physical actions without latent action supervision.

Experimental Results. We evaluate the task success rates of the learned policies across the three real-world manipulation tasks. Each task is evaluated over 40 trials, and the success rates are reported as percentages. The results are summarized below:

MethodSponge ↑\uparrowBatten ↑\uparrowRope ↑\uparrowAverage ↑\uparrow
Direct BC55.0%47.5%37.5%46.67%
LAPA62.5%42.5%32.5%45.83%
DiLA57.5%55.0%42.5%51.67%
CARVE (ours)70.0%77.5%50.0%65.83%

The results demonstrate that CARVE achieves promising performance across all three real-world manipulation tasks, further supporting the practical effectiveness of the learned latent action representations for robotic control. We additionally provide demonstration videos of all three tasks in the supplementary material. We truly appreciate your thoughtful feedback and valuable suggestions! :)

Reviewer_YLJaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CARVE learns latent actions from synchronized multi-view videos. It separates global and private components and uses asymmetric routing for future prediction. It also introduces a three-stage training procedure. The results show improvements in video reconstruction and downstream control.

Critical strengths

The motivation is clear. Different views can provide complementary action information. The proposed architecture enables selective information sharing while preserving view-specific cues. The evaluation covers video prediction, action probing, behavior cloning, and visual planning.

Critical weaknesses

**C1: The role of private components needs clearer validation. **Table 5 includes Global-only and Private-only variants. However, it is unclear whether these variants are trained separately or evaluated by removing a branch from the trained full model. These settings answer different questions. Separate training tests the value of architecture. Removing a branch after training tests how the full model uses that branch.

**C2: The gain loss does not directly enforce a reconstruction gain. **Equation (5) defines the gain loss. This loss is zero when Ejoint≤Eglobal+ϵE_{\mathrm{joint}}\le E_{\mathrm{global}}+\epsilon. It allows the matched private component to provide no improvement or even slightly worsen reconstruction. Thus, this term alone does not require the matched private component to improve reconstruction. The intended constraint and its implementation need clarification.

Questions for authors

**Q1: **Are the Global-only and Private-only variants trained separately, or are branches removed from the trained full model without further training?

**Q2: **Is the negative margin in the gain loss intentional and consistent with the implementation? How often does the matched private component actually reduce reconstruction error compared with global-only input?

5 comments
AuthorsOpenReview ↗

W1&Q1: Clarification and Validation of Global-Private Component Contributions

[W1&Q1: Clarification and Validation of Global-Private Component Contributions]

R1: We sincerely appreciate the reviewer for highlighting this important distinction. In the original Table 5, the Global-only and Private-only variants are obtained by removing the corresponding components from the FDM inputs of the fully trained CARVE model, without any additional training or fine-tuning. These experiments therefore examine the individual contributions of the global and private components within the jointly trained architecture.

To further evaluate the architectural benefits of global-private decomposition, we additionally conduct two training-time ablations, where the private or global branch is removed from the beginning , respectively. Both variants share identical training data, optimization schedules, and evaluation protocols. The remaining components are optimized independently, with objectives involving the removed branch excluded.

The results are summarized below:

VariantPSNR ↑\uparrowSSIM ↑\uparrowLPIPS ↓\downarrow
Full CARVE17.0090.5750.269
Global-only (inference-time)15.9770.5340.286
Private-only (inference-time)16.3810.5300.278
Global-only (retrained)16.5200.5510.281
Private-only (retrained)16.3110.5380.285

The separately trained variants still underperform the full CARVE model, demonstrating that neither branch alone can fully capture the complementary information provided by their joint modeling. These findings further support the benefit of combining cross-view reusable information with view-specific transition cues.

We will clarify the distinction between inference-time removal and training-time ablation in Table 5 and Appendix D.1.

AuthorsOpenReview ↗

W2&Q2: Correction of Margin Formulation and Validation of Private-Component Effectiveness

[W2&Q2: Correction of Margin Formulation and Validation of Private-Component Effectiveness]

R2: We sincerely thank the reviewer for identifying this issue. We would like to kindly clarify that this is a misunderstanding arising from our typographical errors in Equation (5) of the original paper, and we sincerely apologize for any confusion caused by this. Specifically, the margin terms in both Lgain\mathcal{L}_{\mathrm{gain}} and Lcross\mathcal{L}_{\mathrm{cross}} should be +ϵ+\epsilon rather than the incorrectly written −ϵ-\epsilon. We thank the reviewer for bringing this issue to our attention. We have corrected both terms in the revised manuscript. We would also like to kindly clarify that our actual experiments were conducted using the intended positive margins, and the errors are only typographical in the manuscript and do not reflect our actual implementation. Thank you very much!

The corrected gain and cross-view losses are defined as:

Lgain=1V∑v=1V[Ejointv−Eglobalv+ϵ]+.\mathcal{L}_{\mathrm{gain}} = \frac{1}{V}\sum_{v=1}^{V} \left[ E_{\mathrm{joint}}^{v} - E_{\mathrm{global}}^{v} + \epsilon \right]_{+}.

Lcross=1V(V−1)∑v=1V∑u≠v[Eglobalv−Ecrossv←u+ϵ]+.\mathcal{L}_{\mathrm{cross}} = \frac{1}{V(V-1)}\sum_{v=1}^{V}\sum_{u\ne v} \left[ E_{\mathrm{global}}^{v} - E_{\mathrm{cross}}^{v\leftarrow u} + \epsilon \right]_{+}.

We would also like to respectfully point out that, with the corrected positive margins, our method can indeed achieve the desired behavior highlighted in your concern, where the matched private component is required to improve reconstruction. Specifically, the corrected loss becomes zero only when Ejointv≤Eglobalv−ϵE_{\mathrm{joint}}^{v} \leq E_{\mathrm{global}}^{v} - \epsilon, thereby encouraging the joint reconstruction to outperform the global-only reconstruction by a positive margin. Similarly, the corrected cross-view loss encourages Ecrossv←u≥Eglobalv+ϵE_{\mathrm{cross}}^{v\leftarrow u} \geq E_{\mathrm{global}}^{v} + \epsilon, so that mismatched private components do not provide reconstruction gains. Meanwhile, Lrec\mathcal{L}_{\mathrm{rec}} supervises both reconstruction paths, discouraging trivial satisfaction of this constraint by degrading global-only reconstruction. Together, these objectives encourage the global components to preserve reusable transition information while the matched private components capture complementary information not sufficiently represented by the global components.

To further verify whether the matched private components actually improve reconstruction, we conduct an additional quantitative analysis on the held-out DROID Formal300 dataset. We measure the action-token reconstruction MSE using global-only, mismatched and matched global-private inputs, and report the proportion of transitions where adding the matched private component reduces the reconstruction MSE.

ViewMatched ↓\downarrowGlobal ↓\downarrowMismatched ↓\downarrowImproved (%) ↑\uparrow
Front0.02180.03020.032091.4%
Side0.02260.03100.032889.8%
Wrist0.02090.02950.031292.1%
Overall0.02180.03020.032091.1%

The results show that incorporating the matched private components consistently reduces the average reconstruction MSE across all three views, with improvements observed in the majority of evaluated transitions. These findings provide additional evidence that the private components contribute complementary reconstruction information beyond the global representation, further supporting the intended design of Lgain\mathcal{L}_{\mathrm{gain}}.

We thank the reviewer for bringing this issue to our attention, which has helped us improve both the mathematical clarity and empirical validation of our method. Thank you once again for your valuable comments! :)

AuthorsOpenReview ↗

Gentle Follow-up on Our Rebuttal

Dear Reviewer YLJa,

Thank you very much for your valuable feedback on our work! As the author-reviewer discussion period is approaching its end, we would be happy to address any remaining questions or concerns you may have. We sincerely appreciate your time and consideration!

Best regards,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers

We sincerely thank the Area Chair and all reviewers for their thoughtful evaluations, constructive suggestions, and recognition of the strengths of our work. We are particularly encouraged by their positive assessments of CARVE's motivation, novelty, architectural design, and experimental contributions. Reviewer 1 highlighted that “The motivation is clear”, “The proposed architecture enables selective information sharing while preserving view-specific cues”, and “The evaluation covers video prediction, action probing, behavior cloning, and visual planning” (R1). Reviewer 2 acknowledged that “the experimental section shows competitive results” (R2). Reviewer 3 further commented that “The most impressive aspect is the amplitude-guided sampling strategy”, “I appreciate the paper's novelty and theoretical motivation”, and “I acknowledge that Table 5 provides ablation evidence for the private branch and the decomposition losses” (R3). We sincerely appreciate these encouraging assessments and the reviewers' careful examination of our methodological design and experimental validation.

After carefully reviewing all the feedback, we identify several common concerns and have provided corresponding clarifications and additional experiments:

  • Effectiveness of global-private decomposition (R1, R2, R3): We clarify and correct the margin terms in our decomposition objectives, provide quantitative reconstruction error analyses for matched and mismatched private components, and conduct additional controlled ablations to validate the complementary roles of global and private representations.

  • Learning with limited multi-view data (R2): We clarify how different data sources are utilized throughout the progressive three-stage training strategy and introduce additional data efficiency experiments under progressively reduced multi-view training data budgets.

  • Cross-view complementarity and representation fairness (R2, R3): We introduce comparisons between fully shared and independent per-view latent representations, together with input-level camera ablations that control representation dimensionality, providing more direct evidence for the benefits of cross-view information sharing.

  • Practical robotic control performance (R3): We extend our evaluation to real-world manipulation using an ARX5 robotic arm across three tasks, with comparisons against Direct BC and existing latent action methods. We further conduct simulation-based policy ablations on LIBERO to assess the contributions of our key components to downstream control performance.

  • Baseline comparability and experimental completeness (R3): We clarify the consistency of VP2 planning budgets, evaluate existing latent action methods using our action adaptation procedure under controlled settings, and supplement the reconstruction comparisons with MVP-LAM and MuCoLA.

In addition to these responses, we have incorporated targeted revisions into the manuscript. Specifically, we corrected the typographical errors in Eq. (5), added downstream policy ablations in Section 5.4 (Table 6), and provided further clarification in Appendices C.5 and D.1 regarding the sources of published VP2 baseline results and the inference-time settings of the Global-only and Private-only ablations, respectively. We sincerely thank the Area Chair and all reviewers once again for their valuable time, thoughtful feedback, and constructive suggestions!

AuthorsOpenReview ↗

Additional General Response to the Area Chair and Reviewers

We sincerely thank the Area Chair and all reviewers for their thoughtful feedback, valuable recognition and constructive suggestions. We particularly appreciate the reviewer's careful examination and follow-up questions, which have helped us further improve the clarity and reproducibility of our work.

In response to the reviewer's additional questions, we have provided two further clarifications:

  • Margin implementation and experimental consistency (R3): We have confirmed that all experimental results reported in the paper were obtained using the correct positive-margin implementation. We additionally provide experimental results with negative margins, demonstrating the importance of the positive-margin formulation in enforcing complementary information learning and improving performance.

  • Real-world experimental details (R3): We have expanded Appendix F to specify the experimental protocol, including 100 demonstrations per task, the BCTransformer-based policy architecture from the official LIBERO benchmark, and a separately trained two-layer MLP action decoder that maps latent actions to 7-dimensional robot actions.

We sincerely thank the Area Chair's and reviewers' continued attention to our work!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.