acceptodds
Under review as a conference paper at ICLR 2027

Beyond View Consistency: Rotation-aware Consistent Alignment for Viewpoint-Robust Robotic Manipulation

Abstract

Visual robotic manipulation policies remain highly sensitive to viewpoint variations, which can significantly degrade performance. Existing approaches commonly address this issue by learning view-consistent representations. However, directly enforcing cross-view consistency may overlook viewpoint-dependent task-relevant information for precise manipulation. To address this limitation, we propose Rotation-aware Consistent Alignment (RCA). RCA prioritizes capturing task-relevant features. RCA then accounts for camera rotation to adapt viewpoint-dependent features and align them in a unified latent space. In practice, our framework comprises two stages. The first stage learns task-relevant representations from each observation. The second stage adapts and aligns these representations across viewpoints using a FiLM mapping network conditioned on predicted viewpoint parameters. We evaluate RCA in Meta-World and PandaGym simulations and real-world robot experiments under diverse viewpoint perturbations. RCA improves average success rates over baselines by 19.8, 26.7, 4.2, and 6.9 percentage points in Meta-World azimuth, Meta-World camera shaking, PandaGym azimuth, and real-world azimuth evaluations, respectively.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Visual Robotic Manipulation, Viewpoint Robustness, Cross-View Alignment, Task-Relevant Representation Learning, Zero-Shot Viewpoint Generalization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZpieOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Rotation-aware Consistent Alignment (RCA) for visual robotic manipulation under camera viewpoint changes. RCA separates representation learning into two stages. First, it learns task-relevant visual features through inverse-dynamics supervision and entity-weighted reconstruction, augmented with frozen DINOv3 features. Second, it predicts the camera’s relative azimuth and uses this prediction to condition a FiLM-based feature mapping network. Cross-view contrastive learning, zero-offset preservation, and viewpoint-conditioned reconstruction jointly encourage alignment while retaining task information. The resulting representations support COMBO-based offline reinforcement learning in simulation and ACT-based imitation learning on a real robot. Evaluation covers four Meta-World tasks, PandaGym Reach, and a real-world can-placement task, including azimuth changes, camera shaking, and additional camera perturbations. The appendix provides component ablations, alternative mapping architectures, pretrained-feature controls, and task-state prediction experiments.

Critical strengths

RCA achieves 42.41% average success across the four Meta-World tasks, compared with 22.58% for ReViWo and 22.24% for Maniwhere (Table 7). Under camera shaking, it achieves 64.31%, compared with 37.64% for ReViWo (Table 1). These comparisons use the same representation-learning datasets, auxiliary data, and single-view control datasets. The real-world comparison with ReViWo and Maniwhere also uses identical ACT policy heads and shared representation-learning data, with RCA achieving 73.08% versus 42.31% and 30.77%, respectively. The consistency of these gains across tasks, perturbations, and two downstream learning paradigms provides meaningful evidence for the utility of the proposed representation pipeline.

The appendix addresses several important alternative explanations. Removing viewpoint conditioning reduces Window Close success from 84.00% to 66.45% (Table 21), and the comparison with a viewpoint-conditioned MLP supports the choice of FiLM modulation under matched training settings (Table 22). Providing ReViWo and Maniwhere with DINOv3 features does not close the performance gap (Table 25). Furthermore, the held-out-trajectory probes in Table 24 show that mapped representations retain recoverable robot–object position and manipulation-progress information. Together, these experiments provide substantive support for the contribution of conditional alignment and information preservation beyond the qualitative attention and t-SNE visualizations.

Critical weaknesses

Table 9 shows that RCA’s average success decreases from 66.91% within the sampled sectors to 17.90% outside them, with OOD success rates of 12.65%, 7.81%, and 2.13% on Window Close, Door Open, and Drawer Open. Although RCA still improves over the strongest aggregate OOD baseline, the practical extent of extrapolation remains limited. The predictor analysis does not adequately explain this limitation: Table 19 contains angular errors above 40° at two OOD viewpoints, whereas Table 20 evaluates imposed errors only up to 20° and only at four viewpoints within the sampled sectors. Moreover, an imposed error of 5° already reduces average success from 94.31% to 81.81%. A full-range comparison between predicted and ground-truth viewpoint conditioning is needed to distinguish pose-estimation failures from limitations of alignment or visual observability. This uncertainty directly concerns the mechanism responsible for the claimed robustness.

The hardware evaluation contains one task and ten trials per azimuth setting. The reported advantage over π0.5 corresponds to 95 versus 86 successes out of 130 trials, without uncertainty estimates or repeated-training results. Most evaluated azimuths also fall within the representation-training span; only the +60° setting lies outside it (Appendix A.2 and Table 2). In addition, RCA learns representations from six camera views, including action supervision through inverse dynamics, while ACT and π0.5 receive only the fixed-view demonstrations. This follows the stated downstream-training protocol, but leaves the practical comparison confounded by different access to task-specific visual data. Since the six-view recordings already contain action labels, an ACT baseline trained directly on those recordings would clarify the benefit of the proposed pipeline under the same available data budget. The matched comparisons with ReViWo and Maniwhere remain informative.

Questions for authors

Diagnostic evidence for OOD robustness: Can the authors report success with ground-truth versus predicted viewpoint conditioning over all 36 evaluation angles, separating ID and OOD results? In particular, does oracle conditioning substantially improve the low OOD performance on Window Close, Door Open, and Drawer Open? This would clarify whether the primary limitation is viewpoint estimation or the representation mapping itself.

Controlled real-world comparison: How does RCA compare with ACT trained directly on the existing six-view recordings and their action labels? Please also clarify whether evaluation initial conditions are matched across methods and provide uncertainty estimates or repeated-training results for the reported advantage over π0.5. These results would materially affect the interpretation of the practical comparison.

Implementation clarification: How is the viewpoint-conditioned reconstruction decoder (D_{\psi}^{\mathrm{rec}}) in Eq. (10) constructed, initialized, and optimized? In particular, is it shared with or initialized from the Stage-1 decoder, and are its parameters updated jointly with the mapper? Algorithm 1, step 19, explicitly mentions updating only (M_\theta), leaving an important part of the information-preservation objective unclear.

7 comments

We thank the reviewer for the suggestion.

W1/Q1: OOD Robustness

To investigate the OOD performance drop, we analyze viewpoint estimation errors, representation mapping, and occlusion. Our findings suggest that occlusion is the main cause of this drop. Below, we discuss these three factors.

(1) Viewpoint estimation errors. We evaluate RCA-GT, which replaces predicted viewpoint parameters with ground-truth values, across 36 evaluation viewpoints for each task. Per-task, per-viewpoint results are in Tables 3–6 of the revised paper. We summarize the results over ID and OOD viewpoints below:

TaskRCA ID (%)RCA-GT ID (%)RCA OOD (%)RCA-GT OOD (%)
Button Press78.4677.2549.0152.78
Window Close75.5974.6912.6521.95
Door Open69.3871.087.8111.05
Drawer Open44.2348.582.138.24
Average66.9167.9017.9023.50

Ground-truth conditioning improves OOD success by 9.30, 3.24, and 6.11 percentage points on Window Close, Door Open, and Drawer Open. The substantial ID–OOD gap remains, suggesting that viewpoint estimation errors are not the primary cause of the OOD performance drop.

(2) Representation mapping. To further quantify the impact of representation mapping, we bypass Stage-2 mapping on Drawer Open and train a separate policy directly at each OOD viewpoint using Stage-1 features (RCA-OOD-Train). Per-viewpoint results are reported in Table 28 of the revised paper. Average success rate over 18 OOD viewpoints:

RCA (ID)RCA-GT (OOD)RCA-OOD-Train (OOD)
44.23%8.24%16.08%

RCA-OOD-Train achieves 16.08% success, 7.84 percentage points above RCA-GT, but a substantial gap from ID performance remains. These results suggest that representation mapping is not the primary cause of OOD performance drop.

(3) Occlusion. After accounting for the first two factors, a substantial gap between OOD and ID performance remains. Despite directly training policy at each OOD viewpoint (RCA-OOD-Train), this gap persists. We provide visualizations and comparative evidence supporting that occlusion is the major cause of the OOD performance drop.

Evidence 1: Observation visualizations. We visualize observations at 36 evaluation viewpoints for each task (Figures 15–18 of the revised paper). In Door Open (Figure 17), the arm obscures the gripper and door handle at OOD viewpoints from +30° to +80°. From −150° to −80° OOD viewpoints, the camera mainly views the door from behind, where the door obscures the gripper and reduces handle visibility. Drawer Open and Window Close exhibit similar self-occlusion at positive OOD offsets. At some negative OOD offsets, the drawer handle is not visible (Figure 18), while the window frame obscures the gripper and reduces handle visibility (Figure 16). In contrast, many ID viewpoints provide more complete observations of task-relevant cues. These visibility issues coincide with low success rates at the corresponding viewpoints. For example, Window Close, Door Open, Drawer Open achieve 0% success at +50° and +70°, even with GT viewpoint conditioning.

Evidence 2: Within-task comparisons. In Door Open, the gripper is visible at -70°, where RCA-GT achieves 95.56% success. At -80°, the door occludes the gripper, and success drops to 9.45%. Window Close provides another within-task comparison. At −130°, the gripper is visible through the glass and RCA-GT achieves 33.89% success. At −110°, the window frame occludes the gripper, and success is 7.22%.

Evidence 3: Cross-task comparisons. (a) At OOD viewpoints between -150° and -80°, the gripper remains visible in Button Press, whereas in Door Open, the door obscures the gripper. Across all OOD viewpoints, RCA-GT achieves 52.78% average success on Button Press, compared with 11.05% on Door Open. (b) At -130°, Window Close’s transparent glass leaves the gripper and part of the handle visible, whereas the opaque structures in Door Open and Drawer Open obscure more interaction cues. Their RCA-GT success rates are 33.89%, 5.56%, and 1.11%.

Taken together, these findings suggest that occlusion is the major cause of the OOD performance drop.

(4) Sensitivity to viewpoint errors. Motivated by the two OOD viewpoints with prediction errors over 40° mentioned by the reviewer, we extend the angular-error injection experiment in the original Table 20 from a maximum of 20° to 50°:

Injected angular errorAverage success (%)
0°94.31
5°81.81
10°78.89
20°78.20
30°77.92
40°77.50
50°77.22
ReViWo73.75
Maniwhere69.86
MVMWM23.47

Increasing the injected error from 20° to 50° reduces average success from 78.20% to 77.22%, a decline of only 0.98 percentage points. The results indicate that RCA is less sensitive to further increases in viewpoint conditioning error within the tested 10°–50° range. Even with 50° error, RCA retains a performance advantage over baselines.

W2/Q2: Controlled Real-World Comparison

We thank the reviewer for highlighting these issues. We add ACT trained and π0.5 fine-tuned directly on the same six-view demonstrations and their action labels. We also provide an additional independent training run for RCA and π0.5, and clarify the evaluation initialization procedure.

(1) Comparisons using the same six-view data. To assess whether RCA’s gains come solely from access to multi-view data during representation learning, we add ACT trained and π0.5 fine-tuned directly on the same six-view demonstrations. RCA retains its original setting: its policy head uses only single-view demonstrations, whereas ACT (Six Views) and π0.5 (Six Views) use all six views for policy training or fine-tuning. The corresponding experimental results are summarized below:

Method0°−10°−20°−30°−40°−50°−60°+10°+20°+30°+40°+50°+60°Average (%)
RCA10/109/109/109/107/108/106/107/106/106/106/107/105/1073.08
π0.5 (Single View)9/109/109/108/106/106/106/108/106/105/105/105/104/1066.15
π0.5 (Six Views)8/107/108/107/107/108/107/107/107/106/106/107/105/1069.23
ACT (Single View)6/104/101/100/100/100/100/103/101/100/100/100/100/1011.54
ACT (Six Views)6/106/105/104/104/106/104/106/105/104/104/105/104/1048.46

Comparison with ACT. Using six-view demonstrations increases ACT’s average success from 11.54% to 48.46%, indicating that multi-view data improves its cross-view performance. RCA uses the same ACT policy-head architecture and trains the head only on single-view demonstrations, yet achieves 73.08% success, exceeding ACT (Six Views) by 24.62 percentage points. This supports the conclusion that RCA’s gains are not solely attributable to access to multi-view data.

Comparison with π0.5. Six-view fine-tuning increases π0.5’s average success from 66.15% to 69.23%. Success becomes more uniform across viewpoints, but decreases at the policy-training view and small offsets (0°, ±10°, −20°, −30°). RCA achieves 73.08%, an advantage of 3.85 percentage points over π0.5 (Six Views).

(2) Results from independent training runs. To further examine the comparison between RCA and single-view π0.5, we collect 50 new trajectories and conduct one additional independent training run for each method. We retrain all stages of RCA, including representation learning, mapping, and policy-head training. The table below combines the results of the original and additional runs:

Method0°−10°−20°−30°−40°−50°−60°+10°+20°+30°+40°+50°+60°Average (%)
RCA19/2018/2018/2017/2014/2016/2013/2016/2014/2012/2011/2014/2011/2074.23
π0.5 (Single View)18/2018/2017/2015/2013/2012/2012/2016/2013/2011/209/209/208/2065.77

Across the two runs, RCA achieves a pooled success rate of 74.23% (193/260), compared with 65.77% (171/260) for π0.5, corresponding to 22 more successes and an advantage of 8.46 percentage points. This additional run provides preliminary evidence of consistency across training runs.

(3) Evaluation initial conditions. All methods use the same scene reset procedure. Before each evaluation trial, we use a reset script to return the robot arm and gripper to their preset initial states. The initial positions of the can and basket are marked on the tabletop, allowing us to reposition them with reasonable accuracy. This procedure is used to standardize the initial conditions across methods.

(4) Scope of the real-world evaluation. Of the original 13 test viewpoints, 12 lie within the azimuth range covered by the six-view data. Only −50°, 0°, and +50° match the azimuths sampled for representation learning. The other nine lie within this range but are not sampled, allowing us to evaluate interpolation generalization. The +60° viewpoint lies outside this range and evaluates extrapolation generalization.

The table’s physical constraints limit the available viewing range to 180°. To further assess generalization beyond the training range, we additionally evaluate the +70° extrapolation viewpoint within the available space. RCA, π0.5 (Single View), ACT (Single View), and ACT (Six Views) achieve 4/10, 3/10, 0/10, and 2/10 successes, respectively.

Question 3: Reconstruction Decoder Implementation

We thank the reviewer for highlighting this implementation detail. The Stage-2 reconstruction decoder first concatenates the mapped features z~\tilde z with the predicted viewpoint parameters ξ^\hat\xi, then uses a randomly initialized input projection layer to match the required feature dimension. The decoder backbone uses the same architecture as the Stage-1 decoder and is initialized with its trained weights. It is a separate copy and does not share parameters with the Stage-1 decoder.

During Stage-2 training, both the input projection layer and the decoder backbone are optimized jointly with the mapping network. The reconstruction loss updates both the decoder and the mapping network, encouraging the aligned representations to retain task-relevant information needed to reconstruct task entities.

Step 19 of Algorithm 1 in the original manuscript omitted the reconstruction decoder’s parameter update. In the revised manuscript, we have corrected it to “jointly update MθM_\theta and DψrecD_\psi^{rec} using LmapL_{map}” and added details of the initialization procedure.

Reviewer_tBQ5OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Visual manipulation policies trained from a single fixed camera degrade when the viewpoint changes. The paper argues that existing view-consistency methods keep only features shared across views and discard viewpoint-dependent but task-relevant cues. It therefore proposes Rotation-aware Consistent Alignment (RCA), which adapts features according to camera ratation. RCA works in two stages. Stage1 learns task-relevant features from each view using inverse-dynamics supervision and entity-centric masked reconstruction. Stage2 predicts the camera’s azimuth offset and uses it to align features across views, trained with cross-view contrastive and reconstruction losses. Downstream policies are then trained on the aligned features from a single view.
Across Meta-World, PandaGym and a real-robot task, RCA achieves gains under azimuth changes and camera shaking. Overall, the paper’s contribution is to replace enforced view invariance with viewpoint-conditioned feature alignment.

Critical strengths

1.The paper is well written and the author does extensive experiments to show the effectiveness of RCA.

  1. The claim that ReViWo’s attention mask focuses on robot wrist is very helpful and insightful. And the author does some improvements to make the encoder pay more attention to the task-relevant areas.
Critical weaknesses
  1. There are some important problems that RCA fail to deal with: (1) resilience to occlusion; (2) the moving camera. For the first problem, many task-relevant objects can be occluded by robot arms or other things. So the task-relevant reconstruction must first locate the necessary pixels. For the second problem, many cameras are installed on the robot arm which can move along with the robot. So it would be better if RCA can handle the moving camera.

  2. In the real-robot study, RCA learns from multi-view data covering nearly the full test range, whereas π0.5 and ACT see only one view, and the reported gap over π0.5 is not statistically significant. I wish the author could provide a comparison against π0.5 trained with original data on the selected on view and few training data on all other view.

Questions for authors

See weaknesses.

4 comments

We thank the reviewer for valuable feedback.

Weakness 1: Occlusion and Moving Cameras

(1) Resilience to occlusion. We agree that occlusion is a common challenge in visual control. For reconstruction supervision, we use Grounded SAM with text prompts to identify manipulator and target-object regions in the training data (Section 4.1, Appendix C.3). RCA’s entity-centric reconstruction helps representations retain visible task cues; it does not provide direct pixel supervision for occluded parts of entities. RCA’s main contribution is showing that viewpoint awareness is important for cross-view alignment, while entity-centric reconstruction encourages representations to retain task-relevant information before alignment. Handling occlusion is complementary to this contribution. Severe occlusion of key task cues remains a limitation of RCA, as discussed in Limitations and Future Work (Appendix D), where we propose using observation history to retain cues from earlier frames when they become temporarily occluded. Such occlusion-handling techniques could be combined with our framework.

In addition, RCA achieves higher success rates than the compared baselines at several occluded viewpoints. As shown in Figure 15 of the revised manuscript, the button in Button Press is partially or heavily occluded in the initial observations at −150° to −90°. RCA achieves 40.4% average success over these seven viewpoints, compared with 20.4%, 16.4%, and 31.7% for ReViWo, Maniwhere, and MVMWM (Table 3). At +60°, where the arm occludes the gripper, RCA achieves 48.33% while ReViWo achieves 0%.

Moreover, RCA’s gains do not rely solely on the entity-centric reconstruction objective. Table 26 in paper shows that removing this objective still leaves RCA with higher average success than the compared baselines on the Button Press and Window Close tasks:

MethodButton Press (%)Window Close (%)
RCA93.5686.89
RCA w/o Entity-Centric Reconstruction83.4576.56
ReViWo66.3335.33
Maniwhere65.2248.78
MVMWM29.2220.00

(2) The moving camera. We agree that wrist-mounted cameras moving with the robot arm are an important practical application. RCA does not require a fixed camera azimuth: its viewpoint predictor estimates azimuth frame by frame at test time and adjusts feature mapping accordingly. In Table 1 of the paper, we evaluate dynamic viewpoint changes during policy execution through camera-shaking experiments. Across four Meta-World tasks, RCA achieves 64.31% average success, compared with 37.64% for ReViWo, a gain of 26.67 percentage points. These results support RCA’s adaptability to time-varying azimuths.

We can further explore extending RCA to wrist-mounted moving cameras. Specifically, we could extend viewpoint estimation and conditioning from azimuth to full 3D camera poses and train both the viewpoint predictor and the mapping network on corresponding data to accommodate viewpoint changes during manipulation. We have included this research direction in the Limitations and Future Work section of the revised manuscript (Appendix D).

Weakness 2: Real-World Comparisons Using Six-View Data and Independent Training Runs

We thank the reviewer for the suggestion. To clarify data usage, RCA uses six-view data for representation learning, while its downstream policy head is trained from scratch using demonstrations from a single fixed viewpoint. In the original experiments, ACT and π0.5 are trained and fine-tuned, respectively, on the same single-view demonstrations. We add (1) ACT trained and π0.5 fine-tuned directly on the same six-view demonstrations, and (2) one additional independent training run for both RCA and π0.5.

(1) Comparisons using the same six-view data. To assess whether RCA’s gains come solely from access to multi-view data during representation learning, we add ACT trained and π0.5 fine-tuned directly on the same six-view demonstrations. RCA retains its original setting: its policy head uses only single-view demonstrations, whereas ACT (Six Views) and π0.5 (Six Views) use all six views for policy training or fine-tuning. The corresponding experimental results are summarized below:

Method0°−10°−20°−30°−40°−50°−60°+10°+20°+30°+40°+50°+60°Average (%)
RCA10/109/109/109/107/108/106/107/106/106/106/107/105/1073.08
π0.5 (Single View)9/109/109/108/106/106/106/108/106/105/105/105/104/1066.15
π0.5 (Six Views)8/107/108/107/107/108/107/107/107/106/106/107/105/1069.23
ACT (Single View)6/104/101/100/100/100/100/103/101/100/100/100/100/1011.54
ACT (Six Views)6/106/105/104/104/106/104/106/105/104/104/105/104/1048.46

Comparison with ACT. Using six-view demonstrations increases ACT’s average success from 11.54% to 48.46%, indicating that multi-view data improves its cross-view performance. RCA uses the same ACT policy-head architecture and trains the head only on single-view demonstrations, yet achieves 73.08% success, exceeding ACT (Six Views) by 24.62 percentage points. This supports the conclusion that RCA’s gains are not solely attributable to access to multi-view data.

Comparison with π0.5. Six-view fine-tuning increases π0.5’s average success from 66.15% to 69.23%. Success becomes more uniform across viewpoints, but decreases at the policy-training view and small offsets (0°, ±10°, −20°, −30°). RCA achieves 73.08%, an advantage of 3.85 percentage points over π0.5 (Six Views).

(2) Results from independent training runs. To further examine the comparison between RCA and single-view π0.5, we collect 50 new trajectories and conduct one additional independent training run for each method. We retrain all stages of RCA, including representation learning, mapping, and policy-head training. The table below combines the results of the original and additional runs:

Method0°−10°−20°−30°−40°−50°−60°+10°+20°+30°+40°+50°+60°Average (%)
RCA19/2018/2018/2017/2014/2016/2013/2016/2014/2012/2011/2014/2011/2074.23
π0.5 (Single View)18/2018/2017/2015/2013/2012/2012/2016/2013/2011/209/209/208/2065.77

Across the two runs, RCA achieves a pooled success rate of 74.23% (193/260), compared with 65.77% (171/260) for π0.5, corresponding to 22 more successes and an advantage of 8.46 percentage points. This additional run provides preliminary evidence of consistency across training runs.

Thanks for the reply. I think right now the paper is ready. Good luck.

Thank you for your encouraging feedback and kind wishes. We’re glad our responses addressed your concerns. We sincerely appreciate the time you took to read the revisions and the thoughtful suggestions that helped us improve the paper.

Reviewer_Dx3UOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how camera viewpoint changes affect robotic policies. RCA learns task representations using inverse dynamics, entity-centric reconstruction, and DINOv3 features, then uses predicted azimuth angles to guide FiLM-based cross-view alignment. Its average success rate across the full azimuth range on Meta-World is 42.41%, compared with 22.58% for ReViWo. On the real-world task, RCA achieves 73.08%, compared with 66.15% for π0.5.

Critical strengths
  1. I find the idea of preserving task information before viewpoint-conditioned alignment reasonable, and the two-stage design fits this goal. The average success rate increases from 55.42% to 82.78%, supporting the use of conditioning at each layer.

  2. The simulation results are fairly convincing. Across four Meta-World tasks, RCA outperforms ReViWo by 19.83 percentage points over the full azimuth range and by 26.67 points under camera shaking. The per-angle results and the breakdown within and outside the training sectors also help show where the improvements occur.

  3. Adding the same DINOv3 features to the baselines and removing both feature augmentation and entity-centric reconstruction from RCA still support RCA’s advantage. The task-state prediction experiments also use trajectory-level data splits. These analyses help explain the gains beyond the final success rates.

Critical weaknesses
  1. The results support the overall design, but the goal of preserving task information before alignment is addressed mainly through auxiliary losses. The FiLM conditioning mechanism and the pretrained DINOv3 encoder are also adopted from existing work. Further evidence is needed on whether the aligned features retain sufficient task information and how viewpoint estimation errors affect control. The authors should explain these points more clearly.

  2. The evidence for a shared representation remains limited. Appendix A.16 trains a separate prediction head for each viewpoint, showing that task information can be recovered, but not whether one head can work across views. Appendix A.15 only examines training states. This makes it difficult to distinguish the contributions of information preservation and cross-view alignment.

  3. The generalization claims need to be qualified. In Table 9, average success rates within and outside the training sectors are 66.91% and 17.90%. Outside the sectors, Door Open and Drawer Open reach only 7.81% and 2.13%. The model explicitly predicts only azimuth, so results under other camera perturbations should not be interpreted as evidence of general 3D rotation modeling. Could the authors clarify the scope of these claims?

  4. The accuracy of the real-world observation pairs is unclear. More detail on scene resets, temporal alignment, and the handling of problematic trajectories is needed to assess the reliability of the positive pairs used for contrastive learning.

Questions for authors
  1. Once the encoder is frozen, how does the second stage avoid losing information needed for control during cross-view alignment? Please explain the role of each auxiliary loss using the existing ablations and discuss, how viewpoint prediction errors affect the FiLM mapping and final policy performance.

  2. Why does Appendix A.16 train a separate prediction head for each viewpoint? Are results available for a single head tested on unseen viewpoints, or for feature alignment on held-out trajectories? Could these results help distinguish the contributions of task information preservation and cross-view alignment?

  3. What causes the substantial performance drop outside the training sectors in Table 9, particularly for Door Open and Drawer Open? Could the authors use the existing results to discuss the effects of occlusion, viewpoint prediction, and feature mapping, and clarify which camera changes the method can handle when conditioned only on azimuth?

  4. What scene-reset and temporal-alignment procedures are used to collect the real-world multi-view data? Are differences in object and gripper positions checked across replays, and how failed grasps or timing mismatches handled? Please provide any existing data-quality checks that help establish the reliability of the positive pairs.

8 comments

We thank the reviewer for the careful and valuable suggestions.

W1/Q1: Information Preservation

RCA’s main contribution is showing that viewpoint awareness is important for cross-view alignment. The experiments show that task-relevant information is retained after alignment. We also explain how the auxiliary losses support information preservation and analyze how viewpoint prediction errors affect FiLM mapping and policy performance.

(1) Evidence of task information retention. We provide both qualitative and quantitative evidence.

Qualitatively, the attention visualizations of aligned features in Figure 4(a) of paper show that RCA attends more broadly than ReViWo to task-relevant regions, including the robot arm, gripper, and target object.

Quantitatively, the per-view prediction results in Table 24 of the paper show that Stage-2 aligned features and Stage-1 features yield comparable position and progress prediction errors. This supports task information retention after alignment. The results for ID and OOD viewpoints are summarized below:

Button PressPosition ID (cm)Position OOD (cm)Progress ID (mm)Progress OOD (mm)
ReViWo1.812.343.545.48
RCA Stage-1 features0.941.162.202.70
RCA Stage-2 aligned features0.851.102.032.86
Window ClosePosition ID (cm)Position OOD (cm)Progress ID (mm)Progress OOD (mm)
ReViWo2.773.1014.0613.69
RCA Stage-1 features1.111.315.094.78
RCA Stage-2 aligned features0.981.204.594.68

(2) Preserving task information during alignment. RCA constrains alignment in two complementary ways to preserve task information: suppressing representation collapse and retaining task-relevant information.

Suppressing representation collapse. The zero-offset preservation loss LzeroL_{zero} keeps the zero-offset mapping output close to Stage 1 features, providing informative representation anchors for alignment. The cross-view contrastive loss LcvcL_{cvc} aligns same-state features across views while preserving distinguishability between different states, further suppressing collapse.

Retaining task-relevant information. The mapping reconstruction loss LrecL_{rec} encourages aligned features to retain information about task entities.

Ablations. In Table 21, the full model achieves 84.00% average success on Window Close. Removing LzeroL_{zero} or LcvcL_{cvc} reduces success to 80.00% and 52.11%, respectively, supporting the importance of informative anchors and cross-view alignment for control. Removing LrecL_{rec} reduces success to 68.22%, consistent with reconstruction helping retain task information needed for control.

(3) How viewpoint prediction errors affect FiLM mapping and policy performance. Viewpoint estimation errors affect FiLM-based feature adaptation, thereby affecting the mapped features and the inputs to the policy. We examine two questions: (a) how much the current viewpoint predictor’s errors affect policy performance; and (b) how sensitive policy performance is to viewpoint conditioning errors.

(a) We compare RCA with RCA-GT, which uses ground-truth viewpoint conditioning, across all 36 evaluation viewpoints for each task. Per-task, per-viewpoint results are reported in Tables 3–6 of the revised paper. We summarize the results over ID and OOD viewpoints below:

TaskRCA ID (%)RCA-GT ID (%)RCA OOD (%)RCA-GT OOD (%)
Button Press78.4677.2549.0152.78
Window Close75.5974.6912.6521.95
Door Open69.3871.087.8111.05
Drawer Open44.2348.582.138.24
Average66.9167.9017.9023.50

RCA’s average ID success with predicted viewpoint is close to that with ground-truth conditioning (66.91% vs. 67.90%). Ground-truth conditioning improves OOD success from 17.90% to 23.50%. Average success across all viewpoints increases from 42.41% to 45.70%. These results suggest that the current predictor’s errors have a modest effect on policy performance and that the predictor is relatively reliable.

(b) To evaluate the policy’s sensitivity to viewpoint condition errors, we increase the maximum angular error manually injected into the ground-truth viewpoint parameters in Table 20 of the original paper from 20° to 50°. Per-viewpoint results are in Table 20 of the revised paper. Average success is summarized below:

Injected angular errorAverage success rate (%)
0°94.31
5°81.81
10°78.89
20°78.20
30°77.92
40°77.50
50°77.22
ReViWo73.75
Maniwhere69.86
MVMWM23.47

Increasing the injected error from 0° to 5° reduces average success from 94.31% to 81.81%, highlighting the importance of accurate viewpoint conditioning. Increasing it further to 50° reduces success to 77.22%, still above ReViWo (73.75%), Maniwhere (69.86%), and MVMWM (23.47%). Thus, RCA can tolerate a degree of viewpoint error and retains a performance advantage even under large errors.

W2/Q2: Cross-View Alignment at Unseen Viewpoints and on Held-Out Trajectories

Appendix A.16 uses a separate prediction head for each viewpoint to assess task-information retention after mapping, without requiring the head to generalize across viewpoints. To further test whether the aligned features form shared task representations, we add two experiments: (1) task-state prediction at unseen viewpoints with a single shared head; (2) feature alignment analysis on held-out trajectories.

(1) We train a single prediction head on frozen aligned features, then evaluate robot–object relative position and task progress prediction on independent test trajectories at 12 unseen viewpoints: 6 ID and 6 OOD viewpoints. The results on Button Press and Window Close are summarized below:

Button-PressPosition ID (cm)Position OOD (cm)Progress ID (mm)Progress OOD (mm)
ReViWo2.1211.243.5833.14
Direct encoder fine-tuning7.4710.9215.2529.26
Mapping w/o View2.577.375.6817.81
RCA Stage-1 features3.305.978.5115.52
RCA Stage-2 aligned features1.664.713.1810.39
Window-ClosePosition ID (cm)Position OOD (cm)Progress ID (mm)Progress OOD (mm)
ReViWo4.8214.0026.9484.24
Direct encoder fine-tuning13.7914.6376.9876.55
Mapping w/o View4.3610.7326.2859.28
RCA Stage-1 features6.7010.8740.3948.76
RCA Stage-2 aligned features3.077.8113.0638.03

RCA Stage-1 features and RCA Stage-2 aligned features refer to the representations before and after viewpoint-conditioned mapping, respectively. Stage-1 features already retain task information. Compared with Stage-1 features, Stage-2 aligned features yield lower task-state prediction errors at unseen viewpoints with a single shared head. This shows that preserving task information alone is insufficient to match the prediction accuracy achieved after alignment, highlighting the importance of cross-view alignment. Stage-2 aligned features also achieve lower prediction errors than the ReViWo baseline and the other variants, further supporting the effectiveness of viewpoint-conditioned cross-view alignment at unseen viewpoints.

(2) Mapping also improves feature alignment on held-out trajectories. We evaluate cross-view alignment on 400 unseen states from 20 held-out Button Press trajectories. We report mean cross-view cosine similarities for same-state and different-state pairs. The similarity margin measures how much higher the same-state similarity is than the different-state similarity. The results are summarized below:

MetricZero-offset vs. others (Before)Zero-offset vs. others (After)All view pairs (Before)All view pairs (After)
Same-state similarity0.24810.66730.26270.7588
Different-state similarity0.18900.30600.20030.3171
Similarity margin0.05910.36130.06240.4417

where “Zero-offset vs. others” compares features from the policy-training viewpoint with those from other viewpoints, and “All view pairs” includes all pairs of distinct viewpoints. “Before” and “After” refer to features before and after Stage-2 mapping, respectively. Mapping increases both same-state similarity and the similarity margin for Zero-offset vs. others and All view pairs. After mapping, same-state similarities on held-out trajectories (0.6673 and 0.7588) remain comparable to those on training states in Appendix A.15 (0.5790 and 0.7141). These results provide evidence that RCA’s cross-view alignment generalizes to unseen trajectories and states.

W3/Q3: OOD Performance Drop

To investigate the substantial performance drop outside the sampled sectors, we analyze viewpoint prediction errors, feature mapping, and occlusion. Our findings suggest that occlusion is the main cause of this drop. Below, we discuss these three factors and clarify the scope of RCA’s generalization.

(1) Viewpoint prediction errors. We evaluate RCA-GT, which replaces predicted viewpoint parameters with ground-truth values, across 36 evaluation viewpoints for each task. Per-task, per-viewpoint results are in Tables 3–6 of the revised paper. We summarize the results over ID and OOD viewpoints below:

TaskRCA ID (%)RCA-GT ID (%)RCA OOD (%)RCA-GT OOD (%)
Button Press78.4677.2549.0152.78
Window Close75.5974.6912.6521.95
Door Open69.3871.087.8111.05
Drawer Open44.2348.582.138.24
Average66.9167.9017.9023.50

Ground-truth conditioning improves OOD success by 3.24 and 6.11 percentage points on Door Open and Drawer Open, respectively. Across all four tasks, average OOD success increases from 17.90% to 23.50%, but the substantial ID–OOD gap remains. These results indicate that viewpoint prediction errors are not the primary cause of the substantial performance drop.

(2) Feature mapping. To further quantify the impact of feature mapping, we bypass Stage-2 mapping on Drawer Open and train a separate policy directly at each OOD viewpoint using Stage-1 features (RCA-OOD-Train). Per-viewpoint results are in Table 28 of the revised manuscript. We report the average success rate across the 18 OOD viewpoints below:

RCA (ID)RCA-GT (OOD)RCA-OOD-Train (OOD)
44.23%8.24%16.08%

RCA-OOD-Train achieves 16.08% success, 7.84 percentage points above RCA-GT, but a substantial gap from RCA’s ID performance remains. These results suggest that feature mapping is not the primary cause of the substantial performance drop.

(3) Occlusion. After accounting for the first two factors, a substantial gap between OOD and ID performance remains. Despite directly training policy at each OOD viewpoint (RCA-OOD-Train), this gap persists. We provide visualizations and comparative evidence supporting that occlusion is the major cause of the substantial OOD performance drop.

Evidence 1: Observation visualizations. We visualize observations at 36 evaluation viewpoints for each task (Figures 15–18 of the revised manuscript). As shown in Figure 17, Door Open exhibits robot self-occlusion at OOD viewpoints from +30° to +80°: other arm parts obscure the gripper, and the arm also occludes the door handle. From -150° to -80° OOD offsets, the camera views the door from behind, where the door obscures the gripper and reduces handle visibility at several viewpoints. Drawer Open exhibits similar robot self-occlusion at positive OOD offsets, while the drawer handle is not visible at several negative OOD offsets (Figure 18). In contrast, many ID viewpoints provide more complete observations of task-relevant cues. These visibility issues coincide with low success rates at the corresponding viewpoints. For example, both Door Open and Drawer Open achieve 0% success at +50° and +70°, even with ground-truth viewpoint conditioning.

Evidence 2: Adjacent viewpoints within the same task. In Door Open, the gripper is visible at -70°, where RCA-GT achieves 95.56% success. At -80°, the door occludes the gripper, and success drops to 9.45%.

Evidence 3: Cross-task comparisons. (a) At OOD viewpoints between -150° and -80°, the gripper remains visible in Button Press, whereas in Door Open, the door obscures the gripper at several viewpoints. Across all OOD viewpoints, RCA-GT achieves 52.78% average success on Button Press, compared with 11.05% on Door Open. (b) At -130°, Window Close’s transparent glass leaves the gripper and part of the handle visible, whereas the opaque structures in Door Open and Drawer Open obscure more interaction cues. Their RCA-GT success rates are 33.89%, 5.56%, and 1.11%, respectively.

Taken together, these findings suggest that occlusion is the major cause of the substantial OOD performance drop.

(4) Scope of generalization. RCA currently conditions feature mapping on azimuth, supporting adaptation to both fixed and time-varying azimuth offsets. We agree that the experiments on camera pitch, height, and distance changes should not be interpreted as evidence of general 3D rotation modeling. These experiments demonstrate some robustness to these unmodeled perturbations, suggesting potential for adaptation to a broader range of camera changes. Extending the conditioning from azimuth to full 3D camera poses is a potential direction for exploring general 3D rotation modeling. We have clarified the current scope and future research directions in the Limitations and Future Work section of the revised paper.

Comments on the paper

1 comment

Dear Reviewers and Area Chair,

We thank all reviewers for the time and effort devoted to reviewing our paper and for their valuable feedback. We have revised the manuscript and added experiments to address the main concerns:

1. Understanding the OOD performance drop. We analyze viewpoint estimation errors, feature mapping, and occlusion. We compare predicted and ground-truth viewpoint conditioning across all 36 evaluation viewpoints for each task and additionally train a separate policy directly at each OOD viewpoint on Drawer Open. A substantial performance gap remains. Together with observation visualizations and within-task and cross-task comparisons, these results provide evidence that occlusion of task-relevant cues is a major limitation on OOD performance.

2. Additional real-world comparisons and independent training runs. We add ACT trained and π0.5 fine-tuned on the same six-view demonstrations used for RCA’s representation learning. RCA achieves 73.08% success, compared with 48.46% for ACT and 69.23% for π0.5 under this setting. We also collect 50 new demonstration trajectories and conduct an additional independent training run for RCA and π0.5, retraining all stages of RCA. Across the two runs, their pooled success rates are 74.23% and 65.77%, respectively, providing preliminary evidence of consistency across training runs.

3. Generalization of shared task representations. We add task-state prediction at 12 unseen viewpoints using a single shared head, together with feature alignment analysis on 400 unseen states from 20 held-out trajectories. Aligned features achieve lower prediction errors than Stage-1 features, ReViWo, and the other variants, while alignment improvements persist on held-out trajectories. These results support the effectiveness and generalization of viewpoint-conditioned cross-view alignment.

We have also clarified the method’s current azimuth-based scope, the real-world data collection and quality-check procedures, and the reconstruction decoder’s initialization and joint optimization with the mapping network.

Thank you again for your thoughtful feedback. We would appreciate your assessment of these additions and would be happy to clarify any remaining concerns.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.