acceptodds
Under review as a conference paper at ICLR 2027

Learning Latent 3D Rotations from Unlabeled Visual Sequences

Abstract

Learning how visual states transform under physical transformations is important for visual reasoning, but transformation-aware models typically assume access to transformation labels. We ask whether -parameterized actions, learned from unlabeled visual sequences, can recover meaningful structure of the underlying physical 3D rotations. Given object-centric sequences of rotating objects, we jointly train an action inference network and an action-conditioned transition model in the latent space of a frozen visual encoder. To discourage degenerate predictive codes, we introduce a reverse-prediction constraint that requires inferred actions to support consistent inverse transitions. Without using labeled data, the learned actions recover meaningful structure of the underlying 3D rotations. As a complementary evaluation, we show that the learned dynamics support object matching across 3D viewpoint changes through latent-action planning, outperforming static feature-similarity baselines on abstract shapes with limited semantic shortcuts.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, latent action learning, world models, 3D rotations, visual planning, 3D perception

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eyBuOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on learning latent 3D rotations from visual sequences without rotation labels. It jointly trains an action inference network and a transition model over frozen VAE features, using an SO(3) action representation and a reverse-prediction loss. It reports geometric alignment with physical rotations on synthetic datasets and improved object matching over static feature baselines on the Abstract dataset.

Critical strengths
  • The paper studies an interesting problem and presents the method clearly.

  • The evaluation examines several geometric properties, including rotation axes, magnitudes, inversion, and composition.

Critical weaknesses
  • The additional contribution over existing latent action methods is not sufficiently demonstrated. DiLA, already cited in the paper, learns continuous latent actions and regularizes the relationship between forward and backward actions. A direct comparison is needed to establish what the SO(3) representation and reverse-prediction loss add beyond this related design.

  • The main reverse-prediction constraint lacks a quantitative ablation. Figure 3 provides visualizations, but the paper does not report how removing this loss affects aligned axis error, composition error, or matching accuracy. The rotation-magnitude prior is also not ablated, leaving its contribution unclear.

  • The self-supervised method fails to learn meaningful rotations with DINOv2 features, as acknowledged in Section 5. The strong DINOv2 results in Table 3 use ground-truth actions and therefore do not demonstrate the proposed label-free method. This leaves the main result dependent on one VAE backbone, without a clear explanation or solution for the failure.

  • Generalization is tested only within a narrow synthetic setting. The main geometric tests use small rotations from the training range, and the held-out objects come from the same dataset-generation process. There are no tests on real sequences, unseen categories, or changed camera and lighting conditions, leaving the robustness of the recovered geometry unclear.

Questions for authors

Please see the weaknesses above.

1 comment

We thank the reviewer for the feedback.

W1 (Comparison with DiLA). DiLA addresses a related but different problem. As we mention in the paper, its transformation analysis covers in-plane rotation, translation and scaling, with an additional analysis of relative yaw in navigation. It does not investigate rotations about arbitrary 3D axes, which is the subject of our work. DiLA's forward/backward term encourages the two inferred latent actions to point in opposite directions, whereas our reverse-prediction loss requires the inverted action to predict the backward transition through the transition model. For a label-free baseline in our setting, we now include MSP (Miyato et al., 2022), as suggested by reviewer ra5X (MSP: 31.7° axis error vs 13.2° for ours, added in Fig. 4).

W2 (ablations). We ablate both components and report mean ± std over five independently trained models.

ModelAxis errorInverseCompositionMatching Abstract / ShapeNet
Ours13.2 ± 1.0°2.1 ± 0.2°5.1 ± 0.4°87.2 ± 1.1 / 74.9 ± 1.2
w/o reverse prediction59.4 ± 12.4°22.7 ± 2.1°15.1 ± 2.0°64.4 ± 5.0 / 66.1 ± 2.2
w/o magnitude prior74.5 ± 4.0°142.3 ± 37.8°113.2 ± 25.4°53.4 ± 2.8 / 57.8 ± 3.8

Without reverse prediction, the inferred axes fail to align consistently with the underlying 3D rotations across objects. Inverse and composition errors increase substantially, and matching accuracy drops on both datasets. This demonstrates the importance of reverse prediction for recovering meaningful 3D rotational structure.

The magnitude prior serves a different role: it encodes the temporal-continuity assumption of our setting, where consecutive observations differ by small rotations. Without this constraint, the model can infer implausibly large rotation angles (≈120° for a 5° step), leading to degraded performance.

Figs. 4–5 and Table 1 report the geometric results for the reverse-prediction ablation. Table 2 now includes the model without reverse prediction, following the original protocol of Tables 2–4 (one trained model, std over three planning runs). In Appendix E, Table 6 reports the five-model matching results shown above for the full model and both ablations. Table 7 reports the geometric results for the magnitude-prior ablation.

W3 (DINOv2). With action supervision, DINOv2 features perform well (Table 3), indicating that viewpoint information is present. However, label-free action discovery requires features that vary systematically with small pose changes. DINOv2 is trained to emphasize semantic information and may be relatively invariant to such changes, making this signal harder to exploit without supervision. We therefore attribute the difficulty to the suitability of the visual representation for label-free discovery, rather than to an inherent limitation of the approach.

W4 (synthetic setting). Our goal is to test whether 3D rotation structure can be recovered from unlabelled observations at all. This setting allows for a systematic study of this question, and moving directly to real data would introduce other confounds. Such controlled object-centric settings are common in prior work that studies related questions (e.g. MSP, Miyato et al., 2022; SIE, Garrido et al., 2023; EquiCaps, Konstantinou et al., 2025). Extending to more general settings such as real sequences is an important next step that we state as future work.

References

Takeru Miyato, Masanori Koyama, and Kenji Fukumizu. Unsupervised learning of equivariant structure from sequences. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022.

Quentin Garrido, Laurent Najman, and Yann LeCun. Self-supervised learning of split invariant equivariant representations. In International Conference on Machine Learning (ICML), 2023.

Athinoulla Konstantinou, Georgios Leontidis, Mamatha Thota, and Aiden Durrant. EquiCaps: Predictor-free pose-aware pre-trained capsule networks. In IEEE/CVF International Conference on Computer Vision (ICCV), 2025.

Reviewer_tao8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether latent actions inferred from unlabeled sequences of rotating objects can recover meaningful 3D rotation structure. It jointly trains an action inference network and an action-conditioned transition model in a frozen VAE feature space. Actions are mapped to SO(3) through a 6D representation, and training combines forward prediction, reverse prediction using the inverse action, and an angle regularizer. The evaluation examines axis alignment, rotation magnitude, and inverse and composition consistency on held-out objects, with a complementary object-matching task using latent-action planning.

Critical strengths
  1. The geometric evaluation goes beyond prediction error and tests whether the inferred actions share a coordinate system across objects. A single global alignment is fitted on training data and evaluated on held-out data, giving a mean axis error of 12.7 degrees, compared with 11.1 degrees for the action-supervised model (Section 4.2, Figure 4). The self-supervised model also has inverse and composition errors of 2.5 and 5.0 degrees, respectively (Table 1); the composition test includes successive rotations about different axes (Section 4.4). Taken together, these results provide meaningful evidence of shared rotational structure.

  2. The object-matching experiment gives a useful example of what the learned dynamics add to a weak static representation. On Abstract, self-supervised planning reaches 88.3% accuracy, compared with 53.7% for static matching using the same VAE encoder and 73.3% for the strongest static DINO baseline (Table 2). The protocol uses held-out objects and viewpoint changes of 60 to 100 degrees (Section 5 and Appendix C.1). This is a worthwhile proof of concept, even though the self-supervised model does not outperform the strongest static baseline on ShapeNet.

Critical weaknesses
  1. The role of reverse prediction, the central methodological contribution, is not quantified sufficiently. Sections 3.2 and 4.1 describe it as crucial for recovering meaningful rotations, but the comparison without reverse prediction is presented through the axis visualizations in Figure 3. The aligned axis error in Figure 4 and the inverse/composition errors in Table 1 do not include this ablation. A quantitative comparison under the same architecture and training schedule would show how much the proposed constraint actually improves geometric recovery. I would also want to see whether that improvement repeats across independent training runs. This matters given the sharp training transition in Figure 6 and the reported failure to learn meaningful rotations with DINOv2 features; neither observation establishes instability, but both make the reliability of the positive result important to assess.

  2. The matching results leave the contribution of the recovered geometry unresolved. Table 2 compares the self-supervised model with static encoders and an action-supervised model, but does not include an ablated or alternative self-supervised latent-action model under matched training and planning conditions. The results therefore establish an advantage over static feature comparison, while leaving open how much comes from the proposed geometric constraint rather than the additional sequence training and action search. A particularly direct control would run the model trained without reverse prediction through the same bidirectional planner. Table 4 already shows that bidirectional planning improves accuracy, so keeping the planner fixed is important when testing what the training constraint contributes. This gap weakens the connection between the geometric analysis and the downstream result.

Questions for authors
  1. Quantitative ablation: For the model trained without reverse prediction, can you report aligned axis error and inverse/composition errors alongside the full model, using the same architecture, training schedule, and held-out evaluation protocol?

  2. Matched planning control: What matching accuracy does the model trained without reverse prediction achieve with the same VAE features, bidirectional planner, and candidate-search budget? This would help separate the contribution of the training constraint from that of planning.

  3. Evaluation details: Section 3.3 says that the geometric analyses use held-out objects from both datasets. Are the results in Figures 4-6 and Table 1 pooled across ShapeNet and Abstract or reported for one dataset at a time, and how many samples and independently trained models underlie them? What do the plus/minus values in Tables 2-4 represent, and does that variation include retraining or only evaluation/planning randomness?

1 comment

We thank the reviewer for the constructive feedback. The suggested seed-variance analysis and matched planning control strengthened the evaluation, and we have added both to the paper.

W1 / Q1 (reverse-prediction ablation, across training runs). We quantify the effect of reverse prediction and, to address reproducibility, report mean ± std over five independently trained models for the self-supervised variants. These results are now in Fig. 4 and Table 1.

ModelAxis errorInverseComposition
Action-supervised (1 model)11.1°2.2°3.7°
Ours13.2 ± 1.0°2.1 ± 0.2°5.1 ± 0.4°
Ours w/o reverse prediction59.4 ± 12.4°22.7 ± 2.1°15.1 ± 2.0°

Without reverse prediction, no run aligns its axes with a common 3D frame, and inverse and composition errors increase substantially. Our model reaches a low error in every run. We additionally report that the loss drop in Fig. 6 occurs in every run, but the epoch varies (501 ± 123, range 334–647).

W2 / Q2 (matched planning control). As suggested, we evaluate the model without reverse prediction with exactly the same pipeline as ours. Mean ± std over the five trained models of each group:

ModelAbstractShapeNet
Ours87.2 ± 1.174.9 ± 1.2
Ours w/o reverse prediction64.4 ± 5.066.1 ± 2.2
Static VAE features53.755.0

Without reverse prediction, matching drops in every run. The planner samples physical rotations and evaluates them through the transition model. Without the reverse constraint, the action is not a consistent rotation across objects, so the same sampled rotation means different things for different objects. This links the geometric gap (W1) to the planning task. In the paper, Table 2 now includes one model without reverse prediction (± over 3 planning runs, original protocol), and the five-model matching results above are in the new Appendix E.

Q3 (evaluation details). Yes, all geometric results (Figs. 4–5, Table 1) are pooled over the held-out objects of both datasets and each is computed on 960 samples. Fig. 6 shows the training loss of one run. In the submission, each result came from one trained model. Figs. 4–5 and Table 1 now report mean ± std over five independently trained models. The ± in Tables 2–4 is the std over 3 planning runs of the same trained model, so it reflects planning randomness only.

Reviewer_ra5XOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a self-supervised method for learning latent actions that correspond to 3D rotations from unlabeled sequences of rotating objects. Most prior transformation-aware models need ground-truth transformation labels. This work removes that need by jointly training two networks in the latent space of a frozen encoder (a distilled Stable Diffusion VAE). An action inference network predicts the rotation between two consecutive latent states, parameterized in SO(3) with the 6D representation, and a transition network predicts the next latent state from the current state and that action. To keep the two networks from co-adapting to arbitrary codes that do not correspond to physical rotations, the authors add a reverse-prediction constraint: applying the inverse of the inferred rotation to the next state should give back the current state. After a single global alignment, the inferred rotation axes are close to the ground-truth axes (12.7° mean error, against 11.1° for an action-supervised model). The inferred magnitudes also grow roughly linearly with the true ones, and the actions approximately respect inversion and composition. Finally, the learned dynamics are used to plan in latent space and match objects across 3D viewpoint changes.

Critical strengths
  • I like the simplicity of the reverse-prediction constraint. It is well motivated and cheap to compute, since the inverse of a rotation is available in closed form. Figure 3 shows its effect: without it, the inferred axes are not consistent across objects that share the same rotation axis.
  • The paper is clearly written. The method in Section 3 is short and easy to follow, and each of Sections 4.1–4.4 tests a single property of the inferred rotations.
  • I like Figure 6, which shows the training dynamics. The sharp drop in next-state prediction loss coincides with the inferred axes forming a cluster, which shows when the rotational structure emerges.
  • I like the bidirectional planning idea, where rollouts from the source and the target are optimized to meet in the middle. It resembles bidirectional RRT, and Table 4 shows that it consistently improves over unidirectional planning (e.g., from 82.7% to 88.3% on Abstract for the self-supervised model).
  • The visual-planning experiment in Section 5 is well designed and tests the learned dynamics on a concrete task. On the Abstract dataset, the self-supervised model reaches 88.3% accuracy against 73.3% for the best static baseline (DINOv3-h), with 120× fewer parameters (7M vs. 840M). On ShapeNet it matches or beats most DINO baselines (e.g., 76.8% vs. 75.0% for DINOv3-s, which has 3× more parameters), although DINOv3-l (78.4%) and DINOv3-h (83.1%) are better.
Critical weaknesses
  • The dataset description is too brief to reproduce the data. Section 3.3 says that rendering details are given in Appendix C.1, but the appendix only gives the number of objects, the ShapeNet categories, the number of sequences per object and the test split. The renderer, image resolution, camera distance, lighting, textures and the distribution of the rotation axes are not stated, and neither is how the Abstract shapes are generated. Please add these details.
  • The quantitative evaluation in Section 4 is limited. The only comparison is with the action-supervised variant (Figure 4 and Table 1), and Section 4.3 has no comparison at all. The action-supervised variant is also described in only one sentence in Section 5 ("trained with ground-truth actions"). The claim that the reverse-prediction constraint is critical rests on the qualitative Figure 3 alone. I would like to see the axis error (Figure 4), the magnitude scaling (Figure 5) and the inverse and composition errors (Table 1) for the model trained without the constraint, and a comparison with at least one other label-free method, such as those listed in the next point.
  • The related work omits prior methods that infer latent transformations from unlabeled sequences or image pairs and obtain group-structured operators, e.g., Miyato et al. (2022), Koyama et al. (2024), and Mitchel et al. (2024). In particular, Mitchel et al. (2024) learn orthogonal latent maps between adjacent video frames without transformation labels, and show that relative SE(3) camera poses can be regressed from these maps. They also use an inverse-direction reconstruction term closely related to the proposed reverse-prediction constraint. The authors should discuss how their approach differs from these and, ideally, compare against at least one.

Takeru Miyato, Masanori Koyama, and Kenji Fukumizu. Unsupervised learning of equivariant structure from sequences. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, 2022.

Masanori Koyama, Kenji Fukumizu, Kohei Hayashi, and Takeru Miyato. Neural Fourier transform: A general approach to equivariant representation learning. In International Conference on Learning Representations (ICLR), 2024.

Thomas W. Mitchel, Michael Taylor, and Vincent Sitzmann. Neural isometries: Taming transformations for equivariant ML. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2024.

Questions for authors
  • Section 3.3 and Appendix C.1 state that the Abstract dataset is designed to reduce semantic and category-level cues. How are the shapes generated, and how does the procedure reduce these cues?
  • Appendix C.1 states that five sequences are generated per object, each around a sampled axis. How are the axes sampled, and did you study how the results depend on the number of sequences per object?
  • Could you describe the action-supervised baseline used in Section 4? In particular, are the ground-truth rotations given to the transition network in place of the inferred actions, or are they used as regression targets for the action inference network?
  • The 6D representation (Section 3.1) already guarantees valid rotations, but the reverse-prediction loss maps each action to a rotation matrix and back, and the magnitude prior needs the rotation angle (Section 3.2). Did you consider parameterizing the actions in the Lie algebra so(3) through the exponential map, where the inverse is a sign flip and the rotation angle is the vector norm? Since the rotations between frames are small (5°–20°), the discontinuity of this representation at 180° should not matter here.
  • What does the action network predict it we feed it two identical frames? Ideally it should be 0°, but Figure 5 hints that there will be a constant offset even for the same input.
2 comments

We appreciate the constructive feedback and the detailed suggestions, which helped us strengthen the paper.

W1/Q1 (Dataset details and Abstract shapes). Appendix C.1 now describes the data-generation pipeline. All objects are rendered in Blender (Cycles) at 224×224 with a perspective camera (65° field of view), placed so that the object's bounding sphere stays in view with a 5% margin throughout rotation. A single area light sits at the camera, and all materials are uniform grey without textures.

Abstract shapes are procedurally generated with Blender's Shape Generator add-on (5–10 random face extrusions with varying length, taper and twist, then subdivision smoothing). This produces irregular 3D shapes without predefined semantic categories, reducing category-level shortcuts. We will release the rendering and shape-generation code and the generated Abstract shapes.

W2a (reverse-prediction ablation). Our quantitative evaluations in Section 4 now include the model trained without reverse prediction (Figs. 4–5 and Table 1). Results are mean ± std over five training seeds for our self-supervised models, as requested by reviewer tao8:

ModelAxis err. (Fig. 4)Inverse (Tab. 1)Composition (Tab. 1)
Action-supervised11.1°2.2°3.7°
Ours13.2 ± 1.0°2.1 ± 0.2°5.1 ± 0.4°
Ours w/o reverse prediction59.4 ± 12.4°22.7 ± 2.1°15.1 ± 2.0°

Without reverse prediction, the axes no longer align with a common 3D frame and inverse and composition errors increase substantially. The inferred angle still grows with the true angle (Fig. 5), but the actions do not faithfully represent the underlying 3D rotations.

W2b (label-free baseline). Following your suggestion, we added a comparison with MSP, which learns latent transformations through sequence prediction, making it particularly close to our setup. The NFT paper uses MSP to implement its label-free variant, U-NFT, which we cover through our comparison. G-NFT assumes a known transformation group, but its implementation is limited to 1D time-warped signals, and extending it to SO(3) would require additional design choices. Neural Isometries is also closely related, including its inverse-direction reconstruction term. However, while it learns orthogonal latent maps and uses a supervised readout for 3D pose estimation, our central question is whether SO(3)-parameterized latent actions can recover the underlying physical rotations without transformation supervision. We prioritized MSP for its close fit to our setup, as adapting and validating Neural Isometries' separate implementation was not feasible within the rebuttal period. We have expanded the related work on equivariant learning to discuss all three methods.

We trained MSP with the official code on triplets from our training data. We tested two inputs: VAE latents (as for our model) and raw images (MSP's default). Since MSP's transformation is a learned latent matrix, not a 3D rotation, we use a linear probe fitted on training-object rotations to decode its transformations into SO(3), evaluating on held-out objects:

MethodAxis error ↓
MSP, VAE latents34.8°
MSP, raw images31.7°
Ours, same linear probe11.9 ± 0.5°
Ours, global alignment13.2 ± 1.0°

MSP captures part of the rotation, but its axes are far less accurate. Our model recovers them without any learned readout beyond one global alignment. An MLP probe gives the same picture (MSP 35.0° / 31.1° for latents / images, ours 12.0 ± 0.6°). We include MSP's axis error in Fig. 4. The rotation magnitude, inverse and composition results depend on the calibration of the angles, which MSP obtains through the probe, while our results in the paper use the intrinsic angles of the learned representation. A fair comparison therefore needs the same probe, so we compare both models this way in Appendix D. These errors should be read together with the axis error: when the axes are inaccurate, low inverse and composition errors can simply result from small predicted angles.

Q2 (axis sampling). Each generated sequence has its own axis, drawn uniformly on the sphere. We used 5 sequences per object but did not study the effect of this number.

Q3 (supervised variant). The true rotations are regression targets for the action network, not inputs to the transition network. In both variants the transition network receives the inferred action. The supervised variant adds an MSE loss between the inferred and the true 6D action. A new baselines paragraph in Section 4 clarifies this.

Q4 (so(3)). Parameterizing rotations in so(3) via the exponential map is a viable alternative, but we have not tested it.

Q5 (identical frames). Fig. 5 now includes identical frames (0°). Our model infers a mean angle of 1.5 ± 0.2° between identical frames (supervised: 1.3°), i.e. essentially no rotation, whereas the model without reverse prediction infers 11.4 ± 1.7°.

Thank you for the detailed rebuttal. You have answered all my questions, and the additional experiments clearly highlight your contributions compared to prior work. Therefore, I have raised my rating to weak accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.