- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Rotation-aware Consistent Alignment (RCA) for visual robotic manipulation under camera viewpoint changes. RCA separates representation learning into two stages. First, it learns task-relevant visual features through inverse-dynamics supervision and entity-weighted reconstruction, augmented with frozen DINOv3 features. Second, it predicts the camera’s relative azimuth and uses this prediction to condition a FiLM-based feature mapping network. Cross-view contrastive learning, zero-offset preservation, and viewpoint-conditioned reconstruction jointly encourage alignment while retaining task information. The resulting representations support COMBO-based offline reinforcement learning in simulation and ACT-based imitation learning on a real robot. Evaluation covers four Meta-World tasks, PandaGym Reach, and a real-world can-placement task, including azimuth changes, camera shaking, and additional camera perturbations. The appendix provides component ablations, alternative mapping architectures, pretrained-feature controls, and task-state prediction experiments.
Critical strengths
RCA achieves 42.41% average success across the four Meta-World tasks, compared with 22.58% for ReViWo and 22.24% for Maniwhere (Table 7). Under camera shaking, it achieves 64.31%, compared with 37.64% for ReViWo (Table 1). These comparisons use the same representation-learning datasets, auxiliary data, and single-view control datasets. The real-world comparison with ReViWo and Maniwhere also uses identical ACT policy heads and shared representation-learning data, with RCA achieving 73.08% versus 42.31% and 30.77%, respectively. The consistency of these gains across tasks, perturbations, and two downstream learning paradigms provides meaningful evidence for the utility of the proposed representation pipeline.
The appendix addresses several important alternative explanations. Removing viewpoint conditioning reduces Window Close success from 84.00% to 66.45% (Table 21), and the comparison with a viewpoint-conditioned MLP supports the choice of FiLM modulation under matched training settings (Table 22). Providing ReViWo and Maniwhere with DINOv3 features does not close the performance gap (Table 25). Furthermore, the held-out-trajectory probes in Table 24 show that mapped representations retain recoverable robot–object position and manipulation-progress information. Together, these experiments provide substantive support for the contribution of conditional alignment and information preservation beyond the qualitative attention and t-SNE visualizations.
Critical weaknesses
Table 9 shows that RCA’s average success decreases from 66.91% within the sampled sectors to 17.90% outside them, with OOD success rates of 12.65%, 7.81%, and 2.13% on Window Close, Door Open, and Drawer Open. Although RCA still improves over the strongest aggregate OOD baseline, the practical extent of extrapolation remains limited. The predictor analysis does not adequately explain this limitation: Table 19 contains angular errors above 40° at two OOD viewpoints, whereas Table 20 evaluates imposed errors only up to 20° and only at four viewpoints within the sampled sectors. Moreover, an imposed error of 5° already reduces average success from 94.31% to 81.81%. A full-range comparison between predicted and ground-truth viewpoint conditioning is needed to distinguish pose-estimation failures from limitations of alignment or visual observability. This uncertainty directly concerns the mechanism responsible for the claimed robustness.
The hardware evaluation contains one task and ten trials per azimuth setting. The reported advantage over π0.5 corresponds to 95 versus 86 successes out of 130 trials, without uncertainty estimates or repeated-training results. Most evaluated azimuths also fall within the representation-training span; only the +60° setting lies outside it (Appendix A.2 and Table 2). In addition, RCA learns representations from six camera views, including action supervision through inverse dynamics, while ACT and π0.5 receive only the fixed-view demonstrations. This follows the stated downstream-training protocol, but leaves the practical comparison confounded by different access to task-specific visual data. Since the six-view recordings already contain action labels, an ACT baseline trained directly on those recordings would clarify the benefit of the proposed pipeline under the same available data budget. The matched comparisons with ReViWo and Maniwhere remain informative.
Questions for authors
Diagnostic evidence for OOD robustness: Can the authors report success with ground-truth versus predicted viewpoint conditioning over all 36 evaluation angles, separating ID and OOD results? In particular, does oracle conditioning substantially improve the low OOD performance on Window Close, Door Open, and Drawer Open? This would clarify whether the primary limitation is viewpoint estimation or the representation mapping itself.
Controlled real-world comparison: How does RCA compare with ACT trained directly on the existing six-view recordings and their action labels? Please also clarify whether evaluation initial conditions are matched across methods and provide uncertainty estimates or repeated-training results for the reported advantage over π0.5. These results would materially affect the interpretation of the practical comparison.
Implementation clarification: How is the viewpoint-conditioned reconstruction decoder (D_{\psi}^{\mathrm{rec}}) in Eq. (10) constructed, initialized, and optimized? In particular, is it shared with or initialized from the Stage-1 decoder, and are its parameters updated jointly with the mapper? Algorithm 1, step 19, explicitly mentions updating only (M_\theta), leaving an important part of the information-preservation objective unclear.
7 comments
W2/Q2: Controlled Real-World Comparison
We thank the reviewer for highlighting these issues. We add ACT trained and π0.5 fine-tuned directly on the same six-view demonstrations and their action labels. We also provide an additional independent training run for RCA and π0.5, and clarify the evaluation initialization procedure.
(1) Comparisons using the same six-view data. To assess whether RCA’s gains come solely from access to multi-view data during representation learning, we add ACT trained and π0.5 fine-tuned directly on the same six-view demonstrations. RCA retains its original setting: its policy head uses only single-view demonstrations, whereas ACT (Six Views) and π0.5 (Six Views) use all six views for policy training or fine-tuning. The corresponding experimental results are summarized below:
| Method | 0° | −10° | −20° | −30° | −40° | −50° | −60° | +10° | +20° | +30° | +40° | +50° | +60° | Average (%) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| RCA | 10/10 | 9/10 | 9/10 | 9/10 | 7/10 | 8/10 | 6/10 | 7/10 | 6/10 | 6/10 | 6/10 | 7/10 | 5/10 | 73.08 |
| π0.5 (Single View) | 9/10 | 9/10 | 9/10 | 8/10 | 6/10 | 6/10 | 6/10 | 8/10 | 6/10 | 5/10 | 5/10 | 5/10 | 4/10 | 66.15 |
| π0.5 (Six Views) | 8/10 | 7/10 | 8/10 | 7/10 | 7/10 | 8/10 | 7/10 | 7/10 | 7/10 | 6/10 | 6/10 | 7/10 | 5/10 | 69.23 |
| ACT (Single View) | 6/10 | 4/10 | 1/10 | 0/10 | 0/10 | 0/10 | 0/10 | 3/10 | 1/10 | 0/10 | 0/10 | 0/10 | 0/10 | 11.54 |
| ACT (Six Views) | 6/10 | 6/10 | 5/10 | 4/10 | 4/10 | 6/10 | 4/10 | 6/10 | 5/10 | 4/10 | 4/10 | 5/10 | 4/10 | 48.46 |
Comparison with ACT. Using six-view demonstrations increases ACT’s average success from 11.54% to 48.46%, indicating that multi-view data improves its cross-view performance. RCA uses the same ACT policy-head architecture and trains the head only on single-view demonstrations, yet achieves 73.08% success, exceeding ACT (Six Views) by 24.62 percentage points. This supports the conclusion that RCA’s gains are not solely attributable to access to multi-view data.
Comparison with π0.5. Six-view fine-tuning increases π0.5’s average success from 66.15% to 69.23%. Success becomes more uniform across viewpoints, but decreases at the policy-training view and small offsets (0°, ±10°, −20°, −30°). RCA achieves 73.08%, an advantage of 3.85 percentage points over π0.5 (Six Views).
(2) Results from independent training runs. To further examine the comparison between RCA and single-view π0.5, we collect 50 new trajectories and conduct one additional independent training run for each method. We retrain all stages of RCA, including representation learning, mapping, and policy-head training. The table below combines the results of the original and additional runs:
| Method | 0° | −10° | −20° | −30° | −40° | −50° | −60° | +10° | +20° | +30° | +40° | +50° | +60° | Average (%) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| RCA | 19/20 | 18/20 | 18/20 | 17/20 | 14/20 | 16/20 | 13/20 | 16/20 | 14/20 | 12/20 | 11/20 | 14/20 | 11/20 | 74.23 |
| π0.5 (Single View) | 18/20 | 18/20 | 17/20 | 15/20 | 13/20 | 12/20 | 12/20 | 16/20 | 13/20 | 11/20 | 9/20 | 9/20 | 8/20 | 65.77 |
Across the two runs, RCA achieves a pooled success rate of 74.23% (193/260), compared with 65.77% (171/260) for π0.5, corresponding to 22 more successes and an advantage of 8.46 percentage points. This additional run provides preliminary evidence of consistency across training runs.
(3) Evaluation initial conditions. All methods use the same scene reset procedure. Before each evaluation trial, we use a reset script to return the robot arm and gripper to their preset initial states. The initial positions of the can and basket are marked on the tabletop, allowing us to reposition them with reasonable accuracy. This procedure is used to standardize the initial conditions across methods.
(4) Scope of the real-world evaluation. Of the original 13 test viewpoints, 12 lie within the azimuth range covered by the six-view data. Only −50°, 0°, and +50° match the azimuths sampled for representation learning. The other nine lie within this range but are not sampled, allowing us to evaluate interpolation generalization. The +60° viewpoint lies outside this range and evaluates extrapolation generalization.
The table’s physical constraints limit the available viewing range to 180°. To further assess generalization beyond the training range, we additionally evaluate the +70° extrapolation viewpoint within the available space. RCA, π0.5 (Single View), ACT (Single View), and ACT (Six Views) achieve 4/10, 3/10, 0/10, and 2/10 successes, respectively.
Question 3: Reconstruction Decoder Implementation
We thank the reviewer for highlighting this implementation detail. The Stage-2 reconstruction decoder first concatenates the mapped features with the predicted viewpoint parameters , then uses a randomly initialized input projection layer to match the required feature dimension. The decoder backbone uses the same architecture as the Stage-1 decoder and is initialized with its trained weights. It is a separate copy and does not share parameters with the Stage-1 decoder.
During Stage-2 training, both the input projection layer and the decoder backbone are optimized jointly with the mapping network. The reconstruction loss updates both the decoder and the mapping network, encouraging the aligned representations to retain task-relevant information needed to reconstruct task entities.
Step 19 of Algorithm 1 in the original manuscript omitted the reconstruction decoder’s parameter update. In the revised manuscript, we have corrected it to “jointly update and using ” and added details of the initialization procedure.
We thank the reviewer for the suggestion.
W1/Q1: OOD Robustness
To investigate the OOD performance drop, we analyze viewpoint estimation errors, representation mapping, and occlusion. Our findings suggest that occlusion is the main cause of this drop. Below, we discuss these three factors.
(1) Viewpoint estimation errors. We evaluate RCA-GT, which replaces predicted viewpoint parameters with ground-truth values, across 36 evaluation viewpoints for each task. Per-task, per-viewpoint results are in Tables 3–6 of the revised paper. We summarize the results over ID and OOD viewpoints below:
Ground-truth conditioning improves OOD success by 9.30, 3.24, and 6.11 percentage points on Window Close, Door Open, and Drawer Open. The substantial ID–OOD gap remains, suggesting that viewpoint estimation errors are not the primary cause of the OOD performance drop.
(2) Representation mapping. To further quantify the impact of representation mapping, we bypass Stage-2 mapping on Drawer Open and train a separate policy directly at each OOD viewpoint using Stage-1 features (RCA-OOD-Train). Per-viewpoint results are reported in Table 28 of the revised paper. Average success rate over 18 OOD viewpoints:
RCA-OOD-Train achieves 16.08% success, 7.84 percentage points above RCA-GT, but a substantial gap from ID performance remains. These results suggest that representation mapping is not the primary cause of OOD performance drop.
(3) Occlusion. After accounting for the first two factors, a substantial gap between OOD and ID performance remains. Despite directly training policy at each OOD viewpoint (RCA-OOD-Train), this gap persists. We provide visualizations and comparative evidence supporting that occlusion is the major cause of the OOD performance drop.
Evidence 1: Observation visualizations. We visualize observations at 36 evaluation viewpoints for each task (Figures 15–18 of the revised paper). In Door Open (Figure 17), the arm obscures the gripper and door handle at OOD viewpoints from +30° to +80°. From −150° to −80° OOD viewpoints, the camera mainly views the door from behind, where the door obscures the gripper and reduces handle visibility. Drawer Open and Window Close exhibit similar self-occlusion at positive OOD offsets. At some negative OOD offsets, the drawer handle is not visible (Figure 18), while the window frame obscures the gripper and reduces handle visibility (Figure 16). In contrast, many ID viewpoints provide more complete observations of task-relevant cues. These visibility issues coincide with low success rates at the corresponding viewpoints. For example, Window Close, Door Open, Drawer Open achieve 0% success at +50° and +70°, even with GT viewpoint conditioning.
Evidence 2: Within-task comparisons. In Door Open, the gripper is visible at -70°, where RCA-GT achieves 95.56% success. At -80°, the door occludes the gripper, and success drops to 9.45%. Window Close provides another within-task comparison. At −130°, the gripper is visible through the glass and RCA-GT achieves 33.89% success. At −110°, the window frame occludes the gripper, and success is 7.22%.
Evidence 3: Cross-task comparisons. (a) At OOD viewpoints between -150° and -80°, the gripper remains visible in Button Press, whereas in Door Open, the door obscures the gripper. Across all OOD viewpoints, RCA-GT achieves 52.78% average success on Button Press, compared with 11.05% on Door Open. (b) At -130°, Window Close’s transparent glass leaves the gripper and part of the handle visible, whereas the opaque structures in Door Open and Drawer Open obscure more interaction cues. Their RCA-GT success rates are 33.89%, 5.56%, and 1.11%.
Taken together, these findings suggest that occlusion is the major cause of the OOD performance drop.
(4) Sensitivity to viewpoint errors. Motivated by the two OOD viewpoints with prediction errors over 40° mentioned by the reviewer, we extend the angular-error injection experiment in the original Table 20 from a maximum of 20° to 50°:
Increasing the injected error from 20° to 50° reduces average success from 78.20% to 77.22%, a decline of only 0.98 percentage points. The results indicate that RCA is less sensitive to further increases in viewpoint conditioning error within the tested 10°–50° range. Even with 50° error, RCA retains a performance advantage over baselines.