- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The basic idea is to learn an SSL-esque objective in which images in multiview scenes are mapped to canonical feature grids via a canonicalization loss. The loss encourages the output rotation-conditioned feature extractor to gradually approach the same features as the feature extractor applied to the canonical view image and conditioned via the identity rotation, or to another image feature rotated via another rotation towards the same canonical position. A robot policy can be trained on canonicalized outputs of this feature extractor, with the assumption that it will be more robust to viewpoint change.
Critical strengths
The idea is sound for narrow use-cases in which a canonical view is available in the scene. The SSL-esque formulation for aligning feature predictions to the canonical view is cool and reminds me of much prior work in novel view synthesis leveraging cycle consistency. The evaluation is thorough (MetaWorld, Libero, xARM).
The performance on evaluation relative to baseline methods is also very good, though the sample sizes are very small so it's difficult to draw too strong of a conclusion.
Critical weaknesses
It seems like the critical advantage of other viewpoint-augmentation work focused on robust robot policies (RoViAug, VISTA) is that they don't rely on viewpoint canonicalization while still attaining the benefits of viewpoint robustness. How would the proposed technique generalize to scenes where a canonical view is not available? It would be good to compare to either/both of these methods.
Questions for authors
Is the proposed training method stable or require extensive tuning? My experience with SSL-esque objectives is that they can be tricky to get right.
4 comments
Response to Reviewer aNHv (2/2)
Response (2/2)
We ran VISTA with its own training recipe: ZeroNVS re-renders every training frame from a random azimuth around the training camera, and a ResNet-18 encoder is trained from scratch together with the policy. The policy head and evaluation protocol are ours. Because Canonical's encoder is pretrained on our multi-view pool, this checks the VISTA recipe in our setting rather than comparing the two ideas on equal footing. On four MetaWorld tasks (3 seeds, 30 trials per camera per seed), ZeroNVS augmentation gives higher OOD success than the same recipe on real images on three of them, but where the gain appears differs by task: at the two OOD cameras inside its augmentation range on window-open, at the two outside it on door-open, and at both on drawer-close. On assembly, the recipe succeeds almost only at the training camera with either kind of training image, and augmentation does not change that.
| Task | Method | ID | OOD inside VISTA's range (125°, 170°) | OOD outside it (0°, 310°) |
|---|---|---|---|---|
| window-open | VISTA (ZeroNVS images) | 0.18 | 0.27 | 0.07 |
| VISTA recipe, real images | 0.17 | 0.00 | 0.01 | |
| Canonical (paper) | 0.52 | 0.50 | 0.63 | |
| door-open | VISTA (ZeroNVS images) | 0.27 | 0.11 | 0.21 |
| VISTA recipe, real images | 0.19 | 0.15 | 0.00 | |
| Canonical (paper) | 0.63 | 0.55 | 0.75 | |
| drawer-close | VISTA (ZeroNVS images) | 0.67 | 0.44 | 0.48 |
| VISTA recipe, real images | 0.30 | 0.00 | 0.14 | |
| Canonical (paper) | 0.90 | 0.87 | 0.92 | |
| assembly | VISTA (ZeroNVS images) | 0.05 | 0.01 | 0.00 |
| VISTA recipe, real images | 0.10 | 0.00 | 0.00 | |
| Canonical (paper) | 0.42 | 0.40 | 0.48 |
Canonical's rows are from Table 1 and the appendix.
On LIBERO-Goal, the same recipe does not learn the tasks within 50 epochs even on real images (success near zero with either kind of training image, 3 seeds), so the comparison is uninformative on that benchmark.
Question 1: is training stable, or does the SSL objective need extensive tuning
Thank you for the question. In our experience, training is stable. Each benchmark's encoder is pretrained once with a fixed schedule and a checkpoint fixed in advance (Appendix F), and with that encoder fixed, retraining the policy across seeds changes success by about 0.05 (Appendix I.3).
- We needed one general stabilization step: disabling the canonical-consistency and angle-prediction losses for the first several epochs while encoder features are still noisy. This is a warmup schedule, not a per-benchmark search.
- The two disclosed LIBERO-specific adjustments, which loss terms reach the encoder and which features VICReg regularizes, were chosen with a single-seed comparison at one camera rather than a full sweep.
- Other settings differ across benchmarks: the rotation group (SO(3) on MetaWorld and xArm, SO(2) on LIBERO-Goal), the canonical frame (a per-dataset choice), and the loss weights and canonical-pair sampling listed in Appendix E. LIBERO-Goal uses SO(2) because its cameras vary only in azimuth, since tabletop objects occlude the workspace at other viewpoints, and with SO(3) instead Canonical still outperforms every Table 2 baseline (Appendix E). The LIBERO-Goal canonical-loss weight and the removal of canonical-pair oversampling were set in advance and not tuned.
Sample size
On the small sample sizes the reviewer notes, we re-evaluated LIBERO-Goal with three policy seeds per method and ten times the rollouts of Table 2 (see our response to Reviewer UEU9, Weakness 1), and Canonical stays ahead of ReViWo and VILA at both ID and OOD cameras.
We thank the reviewer again for the suggestions. If our answers address your concerns, we would be grateful if you would consider raising your score.
Thanks for the responses. I appreciate the additional analysis and like the paper overall, however I feel the canonical view requirement narrows the scope versus existing methods so will maintain my rating of weak accept.
Removing the recorded canonical camera: two alternative designs
Thank you for the positive assessment. We agree that the evaluated instantiation requires one recorded camera to carry the canonical label. Other MetaWorld pool cameras can also carry it (Appendix I.6), and in the current training objective the recorded features of that camera anchor the canonical representation when it appears in a sampled pair. We have designed two alternatives intended to remove the need for a recorded canonical camera.
The first defines the canonical frame by convention within each dataset, for example as a robot-frontal view at which no camera is placed, with each recorded camera labeled by its rotation relative to that frame. Training would still use pairs of recorded views. The anchoring currently provided by the recorded canonical camera would be replaced by an objective that trains the warp to reconstruct each recorded view's features from the other view's prediction in that frame.
The second extends this convention across datasets. Each dataset would keep its own camera rig, all rotations would be expressed relative to a common robot-frontal frame, and the policy would operate on observations mapped into that frame. The canonical frame would then serve as a shared coordinate system for combining datasets recorded with different camera configurations.
Under both alternatives, Canonical still requires multi-view pretraining data with one rotation label per camera, in exchange for needing no rendering pipeline and no camera pose at deployment. We have not yet evaluated either alternative, and we will describe both in the revision. We expect the recorded canonical camera to be a requirement of the current training setup rather than an inherent limitation of the approach, and we will list it as a limitation of the method as tested.
Response to Reviewer aNHv (1/2)
Response (1/2)
We thank the reviewer for the constructive review and for pointing us to RoViAug and VISTA.
Weakness 1: no comparison or discussion against canonical-view-free augmentation methods such as RoViAug and VISTA
On generalization without a canonical view, our method assumes that the pretraining pool contains one by construction. The appendix shows robustness to which pool view is chosen, but not to having none at all, and we see the canonical-view-free setting as future work. To be concrete about what "canonical view" requires: it is not a privileged camera that must exist in the scene, it is a label on one camera in the multi-view pretraining pool (Appendix I.6). In all three benchmarks it is simply the camera the demonstrations were recorded from, and other MetaWorld pool cameras also work as targets. No canonical camera or pose is needed at deployment. What Canonical does require, and VISTA does not, is synchronized multi-view pretraining data with known relative rotations.