- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper focuses on learning latent 3D rotations from visual sequences without rotation labels. It jointly trains an action inference network and a transition model over frozen VAE features, using an SO(3) action representation and a reverse-prediction loss. It reports geometric alignment with physical rotations on synthetic datasets and improved object matching over static feature baselines on the Abstract dataset.
Critical strengths
-
The paper studies an interesting problem and presents the method clearly.
-
The evaluation examines several geometric properties, including rotation axes, magnitudes, inversion, and composition.
Critical weaknesses
-
The additional contribution over existing latent action methods is not sufficiently demonstrated. DiLA, already cited in the paper, learns continuous latent actions and regularizes the relationship between forward and backward actions. A direct comparison is needed to establish what the SO(3) representation and reverse-prediction loss add beyond this related design.
-
The main reverse-prediction constraint lacks a quantitative ablation. Figure 3 provides visualizations, but the paper does not report how removing this loss affects aligned axis error, composition error, or matching accuracy. The rotation-magnitude prior is also not ablated, leaving its contribution unclear.
-
The self-supervised method fails to learn meaningful rotations with DINOv2 features, as acknowledged in Section 5. The strong DINOv2 results in Table 3 use ground-truth actions and therefore do not demonstrate the proposed label-free method. This leaves the main result dependent on one VAE backbone, without a clear explanation or solution for the failure.
-
Generalization is tested only within a narrow synthetic setting. The main geometric tests use small rotations from the training range, and the held-out objects come from the same dataset-generation process. There are no tests on real sequences, unseen categories, or changed camera and lighting conditions, leaving the robustness of the recovered geometry unclear.
Questions for authors
Please see the weaknesses above.
We thank the reviewer for the feedback.
W1 (Comparison with DiLA). DiLA addresses a related but different problem. As we mention in the paper, its transformation analysis covers in-plane rotation, translation and scaling, with an additional analysis of relative yaw in navigation. It does not investigate rotations about arbitrary 3D axes, which is the subject of our work. DiLA's forward/backward term encourages the two inferred latent actions to point in opposite directions, whereas our reverse-prediction loss requires the inverted action to predict the backward transition through the transition model. For a label-free baseline in our setting, we now include MSP (Miyato et al., 2022), as suggested by reviewer ra5X (MSP: 31.7° axis error vs 13.2° for ours, added in Fig. 4).
W2 (ablations). We ablate both components and report mean ± std over five independently trained models.
Without reverse prediction, the inferred axes fail to align consistently with the underlying 3D rotations across objects. Inverse and composition errors increase substantially, and matching accuracy drops on both datasets. This demonstrates the importance of reverse prediction for recovering meaningful 3D rotational structure.
The magnitude prior serves a different role: it encodes the temporal-continuity assumption of our setting, where consecutive observations differ by small rotations. Without this constraint, the model can infer implausibly large rotation angles (≈120° for a 5° step), leading to degraded performance.
Figs. 4–5 and Table 1 report the geometric results for the reverse-prediction ablation. Table 2 now includes the model without reverse prediction, following the original protocol of Tables 2–4 (one trained model, std over three planning runs). In Appendix E, Table 6 reports the five-model matching results shown above for the full model and both ablations. Table 7 reports the geometric results for the magnitude-prior ablation.
W3 (DINOv2). With action supervision, DINOv2 features perform well (Table 3), indicating that viewpoint information is present. However, label-free action discovery requires features that vary systematically with small pose changes. DINOv2 is trained to emphasize semantic information and may be relatively invariant to such changes, making this signal harder to exploit without supervision. We therefore attribute the difficulty to the suitability of the visual representation for label-free discovery, rather than to an inherent limitation of the approach.
W4 (synthetic setting). Our goal is to test whether 3D rotation structure can be recovered from unlabelled observations at all. This setting allows for a systematic study of this question, and moving directly to real data would introduce other confounds. Such controlled object-centric settings are common in prior work that studies related questions (e.g. MSP, Miyato et al., 2022; SIE, Garrido et al., 2023; EquiCaps, Konstantinou et al., 2025). Extending to more general settings such as real sequences is an important next step that we state as future work.
References
Takeru Miyato, Masanori Koyama, and Kenji Fukumizu. Unsupervised learning of equivariant structure from sequences. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022.
Quentin Garrido, Laurent Najman, and Yann LeCun. Self-supervised learning of split invariant equivariant representations. In International Conference on Machine Learning (ICML), 2023.
Athinoulla Konstantinou, Georgios Leontidis, Mamatha Thota, and Aiden Durrant. EquiCaps: Predictor-free pose-aware pre-trained capsule networks. In IEEE/CVF International Conference on Computer Vision (ICCV), 2025.