acceptodds
Under review as a conference paper at ICLR 2027

Canonical Observation Pretraining for Visuomotor Control

Abstract

Visuomotor policies learned from human demonstrations are brittle to camera viewpoint changes. Existing remedies each make a different assumption: novel-view synthesis requires a rendering pipeline, equivariant architectures require 3D inputs or symmetry hard-coded into the network, and view-invariant pretraining requires only multi-view RGB but suppresses the viewpoint factor, leaving no explicit mechanism for extrapolating viewpoint change. We present , a self-supervised pretraining method that instead keeps viewpoint as a recoverable latent factor, modeling camera viewpoint change as a rotation in SO(3) and warping features into a shared canonical frame before a behavior-cloned policy, which is left unchanged. Pretraining consumes multi-view RGB and the camera angle; at deployment an angle predictor recovers the rotation from the image itself, requiring no pose information. generalizes to camera viewpoints unseen during policy training on both MetaWorld and LIBERO-Goal, outperforming the strongest multi-view baseline by and , and it remains the strongest method at unseen viewpoints on a real-world xArm robot.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-Supervised Learning, Viewpoint Generalization, Imitation Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_aNHvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The basic idea is to learn an SSL-esque objective in which images in multiview scenes are mapped to canonical feature grids via a canonicalization loss. The loss encourages the output rotation-conditioned feature extractor to gradually approach the same features as the feature extractor applied to the canonical view image and conditioned via the identity rotation, or to another image feature rotated via another rotation towards the same canonical position. A robot policy can be trained on canonicalized outputs of this feature extractor, with the assumption that it will be more robust to viewpoint change.

Critical strengths

The idea is sound for narrow use-cases in which a canonical view is available in the scene. The SSL-esque formulation for aligning feature predictions to the canonical view is cool and reminds me of much prior work in novel view synthesis leveraging cycle consistency. The evaluation is thorough (MetaWorld, Libero, xARM).

The performance on evaluation relative to baseline methods is also very good, though the sample sizes are very small so it's difficult to draw too strong of a conclusion.

Critical weaknesses

It seems like the critical advantage of other viewpoint-augmentation work focused on robust robot policies (RoViAug, VISTA) is that they don't rely on viewpoint canonicalization while still attaining the benefits of viewpoint robustness. How would the proposed technique generalize to scenes where a canonical view is not available? It would be good to compare to either/both of these methods.

Questions for authors

Is the proposed training method stable or require extensive tuning? My experience with SSL-esque objectives is that they can be tricky to get right.

4 comments
AuthorsOpenReview ↗

Response to Reviewer aNHv (1/2)

Response (1/2)

We thank the reviewer for the constructive review and for pointing us to RoViAug and VISTA.

Weakness 1: no comparison or discussion against canonical-view-free augmentation methods such as RoViAug and VISTA

  • RoViAug, which we cite in the comparison table (Appendix I.7), addresses embodiment and viewpoint variation jointly. Our experiments hold the embodiment fixed and isolate camera variation, so we did not treat RoViAug as a matched baseline, and we will make this distinction clearer in the table.
  • Thank you for pointing us to VISTA, which we also cite in Appendix I.7 and will discuss in the main text. Like RoViAug, it relies on novel-view synthesis, using a pretrained ZeroNVS model to generate viewpoint augmentations. Since it targets viewpoint alone, we evaluate its recipe below.

On generalization without a canonical view, our method assumes that the pretraining pool contains one by construction. The appendix shows robustness to which pool view is chosen, but not to having none at all, and we see the canonical-view-free setting as future work. To be concrete about what "canonical view" requires: it is not a privileged camera that must exist in the scene, it is a label on one camera in the multi-view pretraining pool (Appendix I.6). In all three benchmarks it is simply the camera the demonstrations were recorded from, and other MetaWorld pool cameras also work as targets. No canonical camera or pose is needed at deployment. What Canonical does require, and VISTA does not, is synchronized multi-view pretraining data with known relative rotations.

AuthorsOpenReview ↗

Response to Reviewer aNHv (2/2)

Response (2/2)

We ran VISTA with its own training recipe: ZeroNVS re-renders every training frame from a random azimuth around the training camera, and a ResNet-18 encoder is trained from scratch together with the policy. The policy head and evaluation protocol are ours. Because Canonical's encoder is pretrained on our multi-view pool, this checks the VISTA recipe in our setting rather than comparing the two ideas on equal footing. On four MetaWorld tasks (3 seeds, 30 trials per camera per seed), ZeroNVS augmentation gives higher OOD success than the same recipe on real images on three of them, but where the gain appears differs by task: at the two OOD cameras inside its augmentation range on window-open, at the two outside it on door-open, and at both on drawer-close. On assembly, the recipe succeeds almost only at the training camera with either kind of training image, and augmentation does not change that.

TaskMethodIDOOD inside VISTA's range (125°, 170°)OOD outside it (0°, 310°)
window-openVISTA (ZeroNVS images)0.180.270.07
VISTA recipe, real images0.170.000.01
Canonical (paper)0.520.500.63
door-openVISTA (ZeroNVS images)0.270.110.21
VISTA recipe, real images0.190.150.00
Canonical (paper)0.630.550.75
drawer-closeVISTA (ZeroNVS images)0.670.440.48
VISTA recipe, real images0.300.000.14
Canonical (paper)0.900.870.92
assemblyVISTA (ZeroNVS images)0.050.010.00
VISTA recipe, real images0.100.000.00
Canonical (paper)0.420.400.48

Canonical's rows are from Table 1 and the appendix.

On LIBERO-Goal, the same recipe does not learn the tasks within 50 epochs even on real images (success near zero with either kind of training image, 3 seeds), so the comparison is uninformative on that benchmark.

Question 1: is training stable, or does the SSL objective need extensive tuning

Thank you for the question. In our experience, training is stable. Each benchmark's encoder is pretrained once with a fixed schedule and a checkpoint fixed in advance (Appendix F), and with that encoder fixed, retraining the policy across seeds changes success by about 0.05 (Appendix I.3).

  • We needed one general stabilization step: disabling the canonical-consistency and angle-prediction losses for the first several epochs while encoder features are still noisy. This is a warmup schedule, not a per-benchmark search.
  • The two disclosed LIBERO-specific adjustments, which loss terms reach the encoder and which features VICReg regularizes, were chosen with a single-seed comparison at one camera rather than a full sweep.
  • Other settings differ across benchmarks: the rotation group (SO(3) on MetaWorld and xArm, SO(2) on LIBERO-Goal), the canonical frame (a per-dataset choice), and the loss weights and canonical-pair sampling listed in Appendix E. LIBERO-Goal uses SO(2) because its cameras vary only in azimuth, since tabletop objects occlude the workspace at other viewpoints, and with SO(3) instead Canonical still outperforms every Table 2 baseline (Appendix E). The LIBERO-Goal canonical-loss weight and the removal of canonical-pair oversampling were set in advance and not tuned.

Sample size

On the small sample sizes the reviewer notes, we re-evaluated LIBERO-Goal with three policy seeds per method and ten times the rollouts of Table 2 (see our response to Reviewer UEU9, Weakness 1), and Canonical stays ahead of ReViWo and VILA at both ID and OOD cameras.

We thank the reviewer again for the suggestions. If our answers address your concerns, we would be grateful if you would consider raising your score.

Thanks for the responses. I appreciate the additional analysis and like the paper overall, however I feel the canonical view requirement narrows the scope versus existing methods so will maintain my rating of weak accept.

AuthorsOpenReview ↗

Removing the recorded canonical camera: two alternative designs

Thank you for the positive assessment. We agree that the evaluated instantiation requires one recorded camera to carry the canonical label. Other MetaWorld pool cameras can also carry it (Appendix I.6), and in the current training objective the recorded features of that camera anchor the canonical representation when it appears in a sampled pair. We have designed two alternatives intended to remove the need for a recorded canonical camera.

The first defines the canonical frame by convention within each dataset, for example as a robot-frontal view at which no camera is placed, with each recorded camera labeled by its rotation relative to that frame. Training would still use pairs of recorded views. The anchoring currently provided by the recorded canonical camera would be replaced by an objective that trains the warp to reconstruct each recorded view's features from the other view's prediction in that frame.

The second extends this convention across datasets. Each dataset would keep its own camera rig, all rotations would be expressed relative to a common robot-frontal frame, and the policy would operate on observations mapped into that frame. The canonical frame would then serve as a shared coordinate system for combining datasets recorded with different camera configurations.

Under both alternatives, Canonical still requires multi-view pretraining data with one rotation label per camera, in exchange for needing no rendering pipeline and no camera pose at deployment. We have not yet evaluated either alternative, and we will describe both in the revision. We expect the recorded canonical camera to be a requirement of the current training setup rather than an inherent limitation of the approach, and we will list it as a limitation of the method as tested.

Reviewer_UEU9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Canonical aims to address the sensitivity of single-view visuomotor policies to camera viewpoint changes. It uses synchronized multiview images and camera rotation labels to learn a rotation predictor and a rotation-conditioned feature transformation. Cross-view prediction and canonical consistency train this transformation to produce a shared canonical representation. The frozen modules process observations during both policy training and deployment, using predicted rotations instead of test-time camera parameters. Experiments on MetaWorld, vision-only LIBERO-Goal, and xArm combine control evaluations, ablations, and representation probes. The contribution is an observation-level approach to viewpoint robustness that does not require an equivariant policy architecture.

Critical strengths
  1. Practical viewpoint robustness. Canonical adapts observations rather than redesigning the policy, achieving 78% OOD success on vision-only LIBERO-Goal versus ReViWo’s 50% (Table 2). This demonstrates useful gains without proprioceptive inputs.
  2. Useful ablations. Removing the warp or replacing predicted rotations lowers average success (Tables 48-49), showing that the trained system benefits from its view-dependent transformation.
Critical weaknesses
  1. The evaluation and baseline protocols need clarification. Table 2 and Tables 45-46 do not specify whether "30 trials per camera" applies to each task or how task results are aggregated. Appendix F fails to explain whether the final policy uses the last checkpoint or one selected through validation. Table 25 should identify the specific module and training stage that supply VILA's features. In addition, Canonical adopts LIBERO-specific pretraining adjustments, while ReViWo and VILA use default hyperparameters, which warrants clearer baseline training and validation details. These clarifications are needed to assess the coverage and comparability of the reported gains.
  2. The similarity tests do not establish a match to the current state. The retrieval test in Appendix H compares different views of the same state, not different states at the canonical camera. It also reports indistinguishable cosine similarity for matched and randomly mismatched states, which the authors acknowledge as inconclusive. Tables 26 and 28 support cross-view robot-state prediction and retention of end-effector and relative object-position information. However, these tests do not show that transformed features are closer to the canonical-view features of the current state than to those of a different state. This limits the interpretation of these measurements.
Questions for authors
  1. Are the 30 trials per camera in Table 2 total or per task? Please list the evaluated tasks, report successes and trial counts by task, camera, and seed, and explain how scores are averaged.
  2. Did you evaluate the last policy checkpoint or select one using validation results? If validation was used, were the validation and final test episodes separate? Was the same policy selection procedure used for all methods?
  3. Which VILA module outputs the 512-dimensional features in Table 25? Did you use both stages of the original method, i.e., latent-action learning and latent behavior cloning? For ReViWo and VILA, please report the pretraining epochs or update counts, and explain how the training length was determined. Please refer to any available learning curves or validation results.
  4. Suggested analysis of existing data: Can you test state matching using existing held-out observations? Use a noncanonical view as the query. Use canonical-view observations of different states from the same task as candidates. Compare retrieval accuracy before and after the warp. Does the warp help retrieve the observation that shows the same physical state as the query?
4 comments
AuthorsOpenReview ↗

Response to Reviewer UEU9 (1/2)

Response (1/2)

We thank the reviewer for the close reading of our evaluation protocol and for proposing a concrete state-matching test.

Weakness 1 and Questions 1 to 2: evaluation protocol clarity on trial aggregation and checkpoint selection

Thank you for raising these points. We answer each below and will state them explicitly in the main text and captions of the revision.

  • Trials: on MetaWorld, Table 1 uses three policy-training seeds per method with 10 trials per camera per seed, giving 30 trials per camera for each task. On LIBERO-Goal, Table 2 evaluates one policy per method with 3 trials of each task per camera, giving 30 trials per camera pooled over the ten tasks. The paper describes this as three seeds, which is an error that we will correct in Section 4.3 and the captions. To give stronger evidence, the table below re-evaluates Canonical, ReViWo, and VILA with three policy seeds each and 10 trials of each task per camera per seed. The MetaWorld table already reports per-task columns, and the appendix will list successes and trial counts by task, camera, and seed for every method.
  • Tasks: MetaWorld uses assembly, door-open, drawer-close, and window-open. LIBERO-Goal uses all ten tasks of the suite: “open the middle drawer of the cabinet”, “open the top drawer and put the bowl inside”, “push the plate to the front of the stove”, “put the bowl on the plate”, “put the bowl on the stove”, “put the bowl on top of the cabinet”, “put the cream cheese in the bowl”, “put the wine bottle on the rack”, “put the wine bottle on top of the cabinet”, and “turn on the stove”.
  • Averaging: for each camera, the success rate is successes divided by trials, pooled over seeds on MetaWorld and over tasks on LIBERO-Goal. The ID and OOD numbers average these per-camera rates over the cameras in each split (6 ID and 4 OOD on MetaWorld, 5 and 5 on LIBERO-Goal). The ± in Tables 1 and 2 is the standard deviation across cameras, so it reflects viewpoint variation rather than seed variation.
  • Checkpoint selection: the appendix applies the same best-checkpoint selection rule to Canonical and every baseline, and the reported results come from a separate evaluation run after training, not from the selection rounds. One asymmetry is disclosed there: none of MetaWorld's selection-rollout cameras overlap an evaluation camera, but LIBERO-Goal's selection cameras do overlap the evaluation cameras, so selection includes out-of-distribution views there, and its initial states come from the same seed sequence as the final evaluation, so the two are not guaranteed to be disjoint. We will surface this asymmetry more clearly in the main text.
Task (LIBERO-Goal)Canonical IDCanonical OODReViWo IDReViWo OODVILA IDVILA OOD
open the middle drawer of the cabinet0.92 ± 0.090.74 ± 0.190.41 ± 0.080.31 ± 0.010.12 ± 0.070.11 ± 0.08
open the top drawer and put the bowl inside0.27 ± 0.340.22 ± 0.280.34 ± 0.110.20 ± 0.050.15 ± 0.060.13 ± 0.02
push the plate to the front of the stove0.89 ± 0.010.88 ± 0.070.73 ± 0.130.56 ± 0.090.30 ± 0.070.23 ± 0.05
put the bowl on the plate0.91 ± 0.060.87 ± 0.080.82 ± 0.070.56 ± 0.090.24 ± 0.140.09 ± 0.09
put the bowl on the stove0.96 ± 0.050.79 ± 0.080.95 ± 0.010.69 ± 0.020.48 ± 0.180.09 ± 0.06
put the bowl on top of the cabinet0.96 ± 0.030.89 ± 0.040.91 ± 0.010.71 ± 0.050.63 ± 0.110.25 ± 0.11
put the cream cheese in the bowl0.71 ± 0.200.58 ± 0.210.41 ± 0.220.25 ± 0.100.59 ± 0.130.20 ± 0.15
put the wine bottle on the rack0.51 ± 0.100.46 ± 0.230.29 ± 0.240.19 ± 0.190.33 ± 0.210.09 ± 0.08
put the wine bottle on top of the cabinet0.98 ± 0.020.94 ± 0.070.64 ± 0.110.47 ± 0.080.06 ± 0.060.03 ± 0.02
turn on the stove1.00 ± 0.000.97 ± 0.030.77 ± 0.080.55 ± 0.030.55 ± 0.360.39 ± 0.14
All tasks0.81 ± 0.050.73 ± 0.020.63 ± 0.040.45 ± 0.020.35 ± 0.090.16 ± 0.03

For each method, the three seeds are the Table 2 policy and two more trained policies, each evaluated at 10 trials of each task per camera (300 rollouts per camera, against 30 in Table 2). The goal image is rendered at the evaluation camera for every method, whereas Table 2's baseline rows used the goal image from the training camera. The ± is the standard deviation across the three seeds. Every average is lower than in Table 2, and the ordering of the three methods is unchanged. We will add this table to the revision and correct the goal-image statement in Section 4.1.

AuthorsOpenReview ↗

Response to Reviewer UEU9 (2/2)

Response (2/2)

Weakness 2 and Question 4: the retrieval and state-matching test does not establish correct-state identification

Thank you for proposing this test. We ran it on the full held-out split, querying with a noncanonical view against canonical-view candidates from different states of the same task, before and after the warp. The table reports the gap in mean-centered cosine similarity between the matched state and a state from another episode of the same task, where zero means no state discrimination.

BenchmarkEncoderGap ± SE (centered cosine)Interpretation
MetaWorldCanonical (Table 1 encoder)+0.006 ± 0.028≈ 0
MetaWorldCanonical (Appendix I.3 encoder)0.000 ± 0.027≈ 0
LIBERO-GoalCanonical (Table 2 encoder)+0.741 ± 0.008≈ 93 SE

The warp does not change how often the matched state is retrieved, but it widens the margin. On LIBERO-Goal, the matched state ranks above the other-episode state for 98% of queries both before and after the warp, and the gap grows from +0.671 to +0.741. On MetaWorld, the rate is about 50% (chance) both before and after. On MetaWorld, neither raw nor warped features show state-level discrimination, on the full 60-episode held-out split. This measures whole-feature cosine similarity, which on MetaWorld is dominated by the shared background scene. State information is nonetheless present in the same features: object position, which proprioception does not include, stays linearly decodable after the warp (R² = 0.81, Appendix H). The retrieval test is therefore uninformative on MetaWorld rather than evidence against state information. In our measurements, MetaWorld's canonical-view features change very little across states, which leaves little state information to match. This matches the state-mismatch control in Appendix H, which also found no difference between matched and mismatched states on MetaWorld. The paper's claims are at the camera level, and we leave state matching on MetaWorld to future work. The camera-identification result (91.9% and 92.2% nearest-camera hit rate) is unaffected.

Question 3: which VILA module and stage supply the 512-dimensional features, and were both stages used

Both VILA stages were used. Stage 1 (LAPO and IDM pretraining) trains the inverse and forward dynamics model that produces latent actions. Stage 2, the BC actor, loads the frozen Stage 1 output and trains the actor. The 512-dimensional feature the policy consumes is the BC-actor's encoder output after global average pooling, and the actor's action head is loaded but not used at inference.

Question 3 (continued): baseline training length and how it was set

BaselineStageTraining
ReViWotokenizer (VQ beta-VAE)200 epochs (about 55k steps, batch 64)
VILAStage 1 (LAPO / IDM)stopped at epoch 298 of 300, epoch-250 checkpoint used
VILAStage 2 (BC actor)300 epochs

Neither baseline used a per-benchmark hyperparameter search. ReViWo uses its paper's native defaults, with only batch size, epoch count, and resolution changed for practicality and fairness. VILA inherits its already-tuned LIBERO configuration unchanged. Encoder checkpoint selection follows each method's native convention: ReViWo keeps the checkpoint with the lowest training-batch loss, which was its final one, and VILA uses its last saved checkpoint (epoch 250), since Stage 1's loss was flat from about epoch 70 onward, so this checkpoint is not undertrained. The downstream BC policy checkpoint rule, peak online rollout success rate, is identical across every method including Canonical. We will merge this detail, with both baselines' training-loss curves, into the Baselines section of the revision.

We appreciate the reviewer's careful reading. If the re-evaluation and clarifications resolve your concerns, we would be grateful if you would consider raising your score.

Thank you for the clarifications and the new three-seed comparison. These strengthen the empirical evidence. However, the reported overlap between checkpoint selection and final LIBERO-Goal evaluation remains a concern. I will maintain my rating of 3.

AuthorsOpenReview ↗

Checkpoint selection overlap: retraining and held-out camera results

Thank you for highlighting this concern. We agree that, on LIBERO-Goal, the checkpoint-selection rollouts overlapped with the evaluation conditions, and this also applies to the three-seed results posted above. The selection rule and budget were identical across methods, but this does not remove the concern. We are therefore retraining Canonical and the two strongest Table 2 baselines, ReViWo and VILA, with three seeds each. Checkpoint selection now uses 15 pretraining-pool cameras that are not evaluation cameras, with starting states disjoint from the final evaluation. Final evaluation remains on the ten Table 2 cameras, and we will include these results in the revision.

In addition, we evaluated the nine existing Canonical, ReViWo, and VILA policies of the three-seed comparison on ten new cameras, five inside and five outside the pretraining pool as in Table 2. None of these cameras was used in checkpoint selection or in the Table 2 evaluation. The new ID azimuths are -17.95, -3.16, 6.40, 12.44, and 17.62 degrees, and the new OOD azimuths are -12.0, 7.8, 13.8, 19.0, and 22.5 degrees. The new OOD set has two cameras beyond the pool's span, against three in Table 2, and the new ID set excludes the 0-degree training view. We also evaluated the policies at -30 and +30 degrees, which appear in Appendix M.2. All runs use starting states disjoint from those of checkpoint selection and of the reported evaluations, and the goal image is rendered at the evaluation camera for every method, as in the three-seed table above. Each ID and OOD entry is the mean success over the five cameras of that split, and each -30 and +30 entry is a single camera. All entries are averaged over three seeds (± standard deviation across seeds), with 10 trials per task per camera per seed.

MethodID, 5 new pool camerasOOD, 5 new non-pool cameras-30°+30°
Canonical0.79 ± 0.060.77 ± 0.030.15 ± 0.070.13 ± 0.08
ReViWo0.58 ± 0.060.49 ± 0.030.00 ± 0.000.00 ± 0.00
VILA0.23 ± 0.060.19 ± 0.060.00 ± 0.000.00 ± 0.01

Canonical remains ahead of both baselines in every column, with margins over ReViWo (0.21 ID, 0.28 OOD) close to those in the three-seed table (0.18, 0.28). At -30 and +30 degrees, where tabletop objects substantially occlude the workspace, both baselines are at or near zero while Canonical retains some success. This evaluation does not remove the overlap in how these checkpoints were selected, but it tests them on cameras and starting states unused in selection. The retraining described above is designed to remove the overlap itself.

Reviewer_RewyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the sensitivity of visual imitation learning policies to camera viewpoint shifts in robotic manipulation. Prior approaches improve viewpoint robustness through view-invariant representation learning, explicit geometric representations such as 3D point clouds, or novel-view synthesis using image and video generation models. However, policies based on these approaches can still suffer substantial performance degradation when evaluated from camera viewpoints outside the training distribution. The authors propose "Canonical", a visual pretraining method that learns an SO(3)-conditioned transformation in latent feature space. Given an observation from a shifted camera viewpoint, the method transforms its visual features into a shared canonical feature space before passing them to a downstream policy. The learned features preserve information about the robot state and make camera viewpoint information linearly decodable, while improving policy robustness to unseen viewpoints.

Critical strengths
  • The paper provides evidence at both the representation and downstream control levels that explicitly transforming viewpoint-dependent features into a shared canonical feature space can be more effective than suppressing viewpoint information through view-invariant representation learning. Canonical improves cross-view feature alignment while preserving task-relevant robot and object-state information, and these benefits translate into improved policy success under unseen camera viewpoints.

  • The paper provides extensive analyses of the learned representation and canonical warp. Feature-alignment measurements, state-decoding probes, canonical-camera retrieval, and reconstruction-based analyses help explain what the representation learns. The no-warp, identity-conditioning, random-rotation, and ground-truth-rotation experiments further demonstrate that the performance gains depend meaningfully on the learned warp and its viewpoint-dependent conditioning.

  • Canonical operates on the policy receiving visual representation rather than imposing equivariance on the downstream policy architecture. The pretrained modules can be frozen and combined with different policy architectures, as demonstrated with DiTFlow and VQ-BeT. This design makes the method applicable to standard non-equivariant policies without requiring explicit 3D reconstruction or specialized geometric policy architectures.

Critical weaknesses
  • (Insufficient positioning relative to closely related work) The paper does not discuss "Augmenting Imitation Experience via Equivariant Representations" (Sharma et al., 2022) or "Canonical Policy" (Zhang et al., 2025), despite their close connections to learned cross-view transformations and canonical representations for robot policies. These works should be cited, and Canonical’s methodological differences should be clearly established. A direct comparison with Sharma et al. or an equivalent latent cross-view mapping baseline would further clarify the paper’s novelty.

  • (Potentially overstated SO(3)-equivariance terminology) Although the paper explicitly limits its claim to single-hop equivariance, the learned warp does not satisfy composition as an SO(3) group action. The terminology should therefore more consistently reflect that the method learns SO(3)-conditioned canonicalization rather than strict equivariance.

  • (Limited guidance for selecting the canonical viewpoint) The canonical viewpoint is manually defined for each benchmark, and the appendix shows that this choice can affect downstream performance. However, the paper provides no general criterion for selecting an appropriate canonical frame in a new environment. More systematic analysis or practical guidance would improve the method’s applicability.

  • (Minor presentation inconsistencies) The evaluation figure does not clearly show the policy-training, encoder-pretraining, and OOD camera viewpoints. Method naming is also inconsistent across figures, and several benchmark-specific design choices are distributed across the main text and appendix. These issues make the experimental setup and exact method configuration more difficult to follow.

Questions for authors
  • Appendix I.6 indicates that downstream performance can depend on the selected canonical viewpoint. What principled criterion should be used to choose this viewpoint in a new environment without relying on downstream evaluation results?

  • "Augmenting Imitation Experience via Equivariant Representations" (Sharma et al., 2022) and "Canonical Policy" (Zhang et al., 2025) share important ideas with the proposed method, including learned cross-view transformations and canonical policy representations. How is Canonical methodologically distinguished from these two approaches, and what constitutes its main conceptual advance?

  • The current experiments define a separate canonical frame within each benchmark. How could a shared canonical representation be defined when pretraining a single model across multiple datasets and robot embodiments that do not share a common camera or robot coordinate frame? This would help clarify the method’s scalability to larger and more heterogeneous pretraining settings.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Rewy (1/2)

Response (1/2)

We thank the reviewer for the careful review and for the pointers to related work.

Weakness 1: missing citations and positioning against Sharma et al. 2022 and Canonical Policy

Sharma et al. (2022) study navigation (AirSim, CARLA, and a real Husky robot), and their equivariance is to camera translation of 0.25 to 0.5 m rather than to SO(3) rotation. They assume known camera poses, augment embeddings and actions for a fixed policy rather than images, and do not self-calibrate or compose. Canonical Policy (Zhang et al., 2025) requires 3D point clouds as input and is compared only with point-cloud methods, whereas Canonical uses RGB images alone. Because the input, the transformation, and the mechanism differ, neither paper is a matched baseline for our setting. The closest matched comparison is the direct-invariance control in Appendix I.1, which keeps the encoder, data, policy, and compute fixed and replaces the rotation-conditioned warp with a view-agnostic mapping. Canonical outperforms it on both ID and OOD cameras. Nevertheless, we thank the reviewer for these pointers and will cite both papers with this distinction.

Weakness 2: SO(3)-equivariance terminology may overstate what is proven

We agree, and thank the reviewer for the suggestion. We will revise the terminology to consistently state SO(3)-conditioned canonicalization. Section 3's introduction already limits the claim to a single warp, with no composition across warps, and the appendix gives the composition experiment (0.96 for a single warp versus 0.59 for two composed warps). The Section 3.2 heading, our row of the comparison table in Appendix I.7, and several sentences in Appendices H and I will be revised to this wording. Composing warps is not needed at inference, since Canonical applies one warp per view. Making the warp compose is a limitation of the current model and a future direction, with a reconstruction-anchored objective as one route (Appendix I.4).

Weakness 3: no principled criterion for picking the canonical viewpoint

We agree that the paper gives no formal criterion. In practice, we pick a canonical view that is not near the elevation extreme of the pretraining pool. The ablation in Appendix I.6, which compares the default canonical view (136° azimuth) with three alternatives (324°, 219°, and a steep view at 35° azimuth and −51° elevation), shows a clear drop only at that steep, edge-of-pool case. The 219° view loses some OOD success, and each variant is a single seed. This is an empirical rule based on that table rather than a closed-form criterion, and we will add this guidance to the appendix.

Weakness 4: evaluation figure and naming presentation issues

The decoder-probe figures and the azimuth-decodability plot label the method “CANON” while the main text uses “Canonical”. Thank you for catching this, and we will fix it in the revision. We will add a schematic or caption to the dataset figure marking which cameras are used for policy training, encoder pretraining, and out-of-distribution evaluation per benchmark (that breakdown currently exists only in prose), and consolidate the benchmark-specific design choices currently spread across the main text and appendix into one summary table.

Question 1: what criterion to choose a canonical viewpoint in a new environment

Please see our response to Weakness 3. In short, we provide empirical guidance (avoid elevation extremes in the pretraining camera pool) rather than a closed-form rule.

Question 2: how is Canonical distinguished from Sharma et al. and Canonical Policy, and what is the main conceptual advance

The core advance is self-calibrated SO(3) canonicalization learned from RGB alone, using only one rotation label per camera during pretraining, no pose information at deployment, and no 3D point cloud input. Sharma et al. assume known poses and operate on 2D translations in a navigation setting, and Canonical Policy requires point clouds as its core input. Ours is RGB only, needs no pose at deployment, and targets rotation rather than translation or embodiment.

AuthorsOpenReview ↗

Response to Reviewer Rewy (2/2)

Response (2/2)

Question 3: how would a shared canonical frame work across multiple datasets and embodiments

Following DynaMo, which we build on, Canonical targets the low-data, in-domain setting, where the encoder is pretrained on the same dataset the policy is trained on for quick adaptation. Its canonical frame is therefore defined per dataset, and we agree that this limits the current method to a single dataset and embodiment. A shared frame across datasets and embodiments would need either a semantic canonical pose, for example frontal relative to the robot base, or per-dataset frames plus a learned mapping between them. We have not built or tested either option, and we will state this limitation explicitly in the revision as an important direction for scaling the method up.

We thank the reviewer again for the constructive feedback. If our responses address your concerns, we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Summary of responses and planned revisions

Thank you again for your review. As the discussion period ends, we summarize our responses and the revisions we committed to. We will revise the terminology to consistently state SO(3)-conditioned canonicalization. We will cite and discuss Sharma et al. and Canonical Policy, with the direct-invariance control in Appendix I.1 as the closest matched comparison, and add guidance on choosing the canonical view, namely avoiding the elevation extremes of the pretraining pool. We will also correct the method naming in the figures, mark the camera roles in the dataset figure, and collect the benchmark-specific design choices in one table. Regarding your question on a shared canonical frame across datasets and embodiments, our reply to Reviewer aNHv outlines two designs. The first defines the canonical frame by convention within each dataset, such as a robot-frontal view at which no camera is placed, and the second shares this convention across datasets so that datasets recorded with different camera rigs could be combined. Both designs concern the camera frame, not differences in appearance or kinematics between embodiments. As stated in our earlier reply, the current method is limited to a single dataset and embodiment. We have not yet evaluated these designs, and the revision will describe them and state this limitation. If these responses address your concerns, we would be grateful if you would consider updating your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful reviews and constructive suggestions. We are encouraged that the reviewers found that mapping viewpoint-dependent features into a shared canonical frame helps at both the representation and the control level, with extensive supporting analyses (Rewy), that Canonical gives practical viewpoint robustness without proprioceptive input, with useful ablations (UEU9), and that the idea is sound and the evaluation thorough across MetaWorld, LIBERO-Goal, and xArm (aNHv).

New results in this response:

  • LIBERO-Goal re-evaluation (UEU9, aNHv): Canonical, ReViWo, and VILA with three policy seeds each and ten times the rollouts of Table 2, with the goal image rendered at the evaluation camera for every method, including a per-task breakdown. Canonical stays ahead of both baselines on average at ID and OOD cameras. This also corrects the paper's description of the LIBERO-Goal protocol.
  • VISTA comparison (aNHv): VISTA's own training recipe with a matched real-image control, on three MetaWorld tasks and on LIBERO-Goal.
  • State-matching test (UEU9): the retrieval test the reviewer proposed, before and after the warp, on both benchmarks.

Planned revisions:

  • Terminology changed to SO(3)-conditioned canonicalization, and citations and discussion of Sharma et al., Canonical Policy, and VISTA (Rewy, aNHv).
  • Guidance for choosing the canonical view, consistent method naming in figures, a camera-role schematic, and a summary table of benchmark-specific design choices (Rewy).
  • A corrected LIBERO-Goal protocol statement and captions, the re-evaluation table, trial counts by task, camera, and seed, and baseline training details with loss curves (UEU9).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.