acceptodds
Under review as a conference paper at ICLR 2027

Commutator-Guided Learning of Order-Aware Lie Actions in Mixed-Latent VAEs

Abstract

A key challenge for Lie-structured variational autoencoders is to learn how actions compose and reproduce their order-dependent physical effects in decoded observations. We propose a two-phase framework separating representation learning from physical-action grounding and algebraic refinement. Phase 1 (P1) combines a categorical branch for category-related content with a continuous branch for Lie transformations. We compare unsupervised training with paired-weak supervision, which uses observation pairs sharing category or transformation state. Phase 2 (P2) first grounds latent directions using action-labeled transitions while keeping the representation model frozen (P2a), then refines these directions and Lie generators toward prescribed commutator relations while retaining finite-action targets (P2b). To connect the algebraic refinement to observable accuracy, a rollout variant (P2b+) additionally matches decoded action trajectories through a frozen decoder. We evaluate latent actions, decoded observations and Lie structure on synthetic image benchmarks and controlled global rotations of real human-pose data, covering non-commuting actions and commuting controls. Paired-weak P1 improves mean categorical and geometric specialization, with trade-offs in reconstruction and FactorVAE (FVM) scores. In P2, algebraic refinement strengthens prescribed non-zero relations beyond grounding alone, while P2b+ achieves lower mean decoded endpoint error than Core P2b. Compared with the Commutative Lie Group VAE (CLG-VAE)-style strict control, which naively encourages commutativity by penalizing all basis-pair commutators, grounded refinement improves action-direction agreement across several image benchmarks, better recovers prescribed non-zero brackets, and reduces unwanted decoded order effects on the commuting benchmarks. These action-level gains coexist with competitive reconstruction and FVM scores relative to the strict control and external baselines. Results on the skeleton datasets likewise show improved recovery of non-commuting order effects and prescribed bracket directions and magnitudes. Together, these findings support grounding and refining the appropriate action relations rather than imposing blanket commutativity, while distinguishing algebraic agreement from accurate observable effects.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Lie-structured variational autoencoders, non-commutative transformations, few-shot action grounding, Lie algebra refinement, categorical–continuous representations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Eg4gOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a staged Lie-group VAE. P1 learns a categorical carrier and continuous Lie bases. P2a fits physical-action directions from labeled transitions with P1 frozen. P2b refines directions and bases toward prescribed commutator relations, and P2b+R adds decoded rollout. It compares against an all-pairs commutative control (CLG-VAE style) on dSprites, ModelNet10, 3DShapes, Cars3D, and globally rotated MI-Motion and 3DPW poses.

Critical strengths

The evaluation separates internal algebraic agreement from observable effects in latent and decoded space, uses five seeds with standard deviations, and reports mixed results openly. On MI-Motion, refinement clearly improves decoded order effects: the AB/BA gap cosine rises from 0.587 (Strict) to 0.785–0.802, and the gap-magnitude error drops from 0.120 to about 0.028.

Critical weaknesses
  1. Several key metrics measure what P2b is trained on. P2b fits bracket residuals to prescribed structure constants, and closure, bracket cosine and norm ratio then evaluate the same quantities, while the Strict control is commutative by design and, for images, has no physical-action targets. Where gains matter most, in observable effects, they are small or absent: on 3DPW, decoded endpoint SSE is 0.512/0.503 versus 0.500 for Strict.

  2. On two of four image benchmarks the learned actions barely align with the physical ones. On ModelNet10, the values reported as ×10⁻² give a decoded order-gap cosine of 0.030 and single-action cosine of 0.007 for P2b+R, both below P2a-only. On Cars3D, azimuth cosine is 0.066 (latent) and 0.077 (decoded). The method also assumes the group algebra is known in advance.

Questions for authors
  1. If the structure constants of SE(2)/SO(3) are given, why not use the known generators directly? What does learning the Lie bases add in this setting?

  2. For the image benchmarks, can you report Strict with the same physical-action targets, as done for skeletons in Table 5?

  3. Given cosines near zero on ModelNet10 and Cars3D azimuth, does any variant actually learn these actions?

Reviewer_UXXrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a two-phase framework for learning Lie-structured latent representations in VAEs. Phase 1 learns categorical carriers and continuous Lie transformations, while Phase 2 grounds latent actions using known physical transitions and refines their generators to satisfy prescribed commuting or non-commuting Lie algebra relations. The method additionally introduces decoded rollout supervision to improve observable action accuracy. Experiments on synthetic image datasets and human-pose datasets evaluate representation specialization, algebraic consistency, and physical-action alignment.

Critical strengths

None

Critical weaknesses
  1. The paper suffers from severe readability and presentation issues. The paper is exceptionally difficult to follow. The writing introduces numerous concepts, objectives, training stages, and evaluation metrics without sufficiently explaining their relationships or motivations.
  2. The problem formulation and central research objective are insufficiently clear and the proposed methodology lacks sufficient motivation and justification. The paper discusses several related but distinct objectives, however, it is unclear which of these constitutes the primary research problem and how the others are essential to addressing it. Also, the framework combines numerous objectives, including endpoint matching, algebraic closure etc. However, the paper does not clearly justify why this particular combination is necessary.
Questions for authors

What is the central research problem, and why is the proposed multi-stage framework necessary? The paper addresses several objectives. Could the authors clarify how these objectives are connected and why each stage is necessary?

2 comments
AuthorsOpenReview ↗

Response to the concerns about clarity, motivation, and the multi-stage framework

We thank the reviewer for highlighting the need to explain the relationship between the research objective, training stages, and losses more clearly. We have added targeted explanations to the Introduction and Sections 2–2.3.

What is the central research problem?

Our central question is: given observations linked by known physical transformations and prescribed composition laws, can learned latent actions reproduce both the transformations and the correct effects of composing them in different orders?

The objective is not simply to produce nonzero commutators. An order-sensitive latent model can still predict the wrong physical effect, while accurate reconstruction does not establish that its latent directions represent the intended actions. We therefore assess agreement with finite physical-action targets, prescribed local composition relations, and decoded consequences. Categorical–continuous specialization is a supporting representation question, not a separate definition of the primary objective.

Why separate representation learning, grounding, and refinement?

The stages address distinct gaps:

  1. P1 provides a reconstructive representation. It learns the carrier, continuous Lie parameterization, and decoder. Reconstruction alone, however, does not identify which combinations of the learned bases correspond to a specified rotation or translation.

  2. P2a identifies the physical actions. It learns action-specific combinations of the P1 bases using known transformation targets, while keeping P1 frozen. This assigns physical meaning to latent directions against fixed reference states and targets.

  3. P2b refines their composition relations. Starting from the grounded actions, it updates the directions and Lie bases toward the prescribed commuting or non-commuting relations while retaining finite-action targets. Encoder, carrier, and decoder parameters remain frozen. The optional rollout extension directly matches decoded consequences through the frozen decoder.

This decomposition allows us to study grounding and algebraic refinement separately from representation learning.

Why combine these losses?

The losses impose complementary constraints on the same action model:

  • Endpoint matching ties learned actions to specified finite transformations. Signed-gap matching explicitly emphasizes the difference between action orders. As now clarified, exact matching of both ordered endpoints already implies the correct gap; the additional term changes the optimization emphasis rather than introducing independent target information.
  • Bracket matching constrains local composition relations. Algebraic agreement alone does not ensure accurate finite decoded effects, motivating the retention of finite-action targets and optional decoded rollout.
  • Guard, scale, and anchor terms serve supporting regularization roles: where configured, they discourage collapsed responses, reinforce amplitude matching, and limit drift.
  • In P1, code and leakage penalties encourage categorical-branch use and discourage cross-branch correlation because reconstruction alone need not assign distinct roles to the branches. They do not guarantee semantic separation.

Our ablations assess the empirical contribution of these choices rather than establish that every term is indispensable. For example, the matched dSprites comparison of finite-only+R and Core P2b+R shows improved algebraic and decoded order-effect agreement from adding bracket losses, but slightly worse mean endpoint error. We therefore distinguish benefits for composition agreement from uniform improvement across metrics.

Manuscript revisions.

The Introduction now explicitly states the primary research question. Section 2 explains the rationale for staging; Section 2.1 motivates the code and leakage penalties; Section 2.2 clarifies signed-gap matching; and Section 2.3 connects the finite-action, bracket, decoded, scale, and anchor terms. We also clarify that latent-action, Lie-structure, and decoded-observation metrics assess complementary requirements of the central objective, they are diagnostics, not additional training objectives.

Thank you for the response, especially for the clarification of the central research question. Now I understand the research objective. However, I still find that the paper is not organized clearly enough to support the main idea since the method involves many stages and their individual contributions are not clearly separated and demonstrated through experiments. Therefore, I will maintain my score.

Reviewer_PPSuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a two-phase framework for learning order-aware Lie actions in a mixed-latent VAE. In Phase 1 (P1), the categorical branch constructs a carrier, while the continuous branch learns a Lie transformation G(t)=exp⁡(∑ktkAk)G(t)=\exp(\sum_k t_kA_k) that acts on the carrier. In Phase 2 (P2), the learned representation is frozen. P2a grounds physical actions as combinations of the learned Lie bases using action-labeled transitions, and P2b jointly refines the action directions and Lie bases toward prescribed commutator relations while retaining finite-action targets. P2b+R additionally matches decoded action trajectories through the frozen decoder. The method is evaluated on four image benchmarks and two skeleton datasets. The P2 variants are compared with Strict and LF+64, while reconstruction and FVM are additionally compared with β\beta-VAE and CFASL.

Critical strengths

S1.

The paper makes a useful distinction between algebraic agreement and accurate observable effects. The evaluation separately considers latent action accuracy, agreement with the prescribed Lie structure, and decoded physical effects. The use of 3DShapes and Cars3D as commuting controls is also valuable because it evaluates not only whether desired order effects are recovered, but also whether unwanted order effects are suppressed.


S2.

• On the skeleton datasets, the refined variants recover the prescribed SO(3) relations with high accuracy. The bracket cosine of Core P2b/P2b+R is approximately 0.980.98 on MI-Motion and 1.001.00 on 3DPW, while the norm ratio reaches approximately 0.950.95–0.960.96. The Strict comparison on these datasets retains physical-action supervision, anchors, and the refinement budget, providing an appropriate control for comparing prescribed nonzero brackets with blanket commutativity.

Critical weaknesses

W1.

On ModelNet10, the prescribed SO(3) actions are not accurately recovered. Core P2b and P2b+R improve the bracket metrics over Strict and P1-only, but the bracket cosine and norm ratio remain at about one-third of both the target value and what the same method achieves on the skeleton datasets, and the latent single-action cosine stays close to zero (Tables 5–6). The results therefore show relative improvement over the controls, not accurate recovery of the prescribed SO(3) actions.


W2.

The image comparison with Strict confounds commutator guidance with action supervision. For the image benchmarks, Strict is a commutativity control that does not receive physical-action targets, as stated in the note of Table 6. Nevertheless, the abstract uses the improved action-direction agreement over Strict as evidence in favor of grounded refinement. This comparison does not isolate the contribution of prescribed commutator guidance because the amount and type of supervision differ. The skeleton experiments provide a supervision-matched Strict control in Table 5. An analogous image control on the non-commuting image benchmarks should retain the same physical-action targets, anchors, trainable parameters, and refinement budget as Core P2b, while replacing the prescribed nonzero-bracket losses with the all-pairs basis commutativity penalty, as in Table 5. Without such a comparison, the specific benefit of commutator-guided refinement on the image benchmarks remains unclear.


W3.

The mixed-latent representation is not shown to matter for order-aware action recovery. The image P2 experiments start from unsupervised P1 checkpoints. However, under unsupervised P1, category information is more predictable from the continuous branch than from the categorical branch on three of the four image benchmarks. Paired-weak P1 improves branch specialization, but these checkpoints are not used for downstream P2 evaluation. The skeleton experiments also contain no categorical branch. Therefore, the current experiments do not establish that the categorical carrier or improved categorical–continuous specialization is necessary for the reported order-aware action recovery.


W4.

The coexistence of action-level gains with competitive FVM is not evidence about P2. P2 keeps the encoder frozen. Consequently, the FVM scores are structurally identical across P2a-only, Core P2b, P2b+R, and LF+64. The observation that action-level gains coexist with competitive FVM therefore describes the inherited P1 representation relative to the external baselines. Since FVM is unchanged by construction once the encoder is frozen, it is not an independent empirical result of P2 refinement and does not show that P2 improves representation quality. The abstract and conclusion should distinguish these claims more clearly.


Minor Comnet

The reference for CFASL lists the first author is wrong. The current entry also points to a Semantic Scholar URL. Please replace it with the official TMLR/OpenReview BibTeX entry and check the remaining references against their official records.

Questions for authors

Please revise the weaknesses.

2 comments
AuthorsOpenReview ↗

Answer to the raised comments

Response to W1

We agree that the ModelNet10 results demonstrate relative improvement, not accurate recovery of the prescribed SO(3) actions. Core P2b achieves substantial relative improvements in the reported structural means over the original Strict control: mean bracket cosine increases from 0.006 to 0.324, and mean norm ratio from 0.079 to 0.320. Nevertheless, both remain below their ideal value of one, and latent single-action cosine remains near zero.

The supervision-matched comparison reported in W2 further supports improved bracket direction and magnitude agreement with physical-action supervision held fixed, although accurate SO(3) action recovery remains unresolved.

Unlike the skeleton setting, where rotations act directly on observed 3D coordinates, the image setting requires the representation and decoder to capture their observable consequences. This difference may contribute to the performance gap, but our experiments do not isolate its cause. Our conclusion is therefore improved structural agreement relative to the controls, rather than accurate SO(3) action recovery on ModelNet10.

Response to W2

We agree that the original image Strict comparison does not isolate commutator guidance because it also changes action supervision. We therefore conducted the requested supervision-matched comparison on dSprites and ModelNet10, rerunning Core P2b and Grounded Strict across five seeds.

Within each dataset, both arms start from identical seed-specific P2a checkpoints and retain the same physical-action targets, finite-action losses, trainable parameters, optimizer schedules, refinement budgets, evaluation records, and anchor weights and refresh policies. Grounded Strict replaces the prescribed nonzero-bracket losses with an all-45 basis-pair commutativity penalty; the dSprites commuting-relation loss is retained in both arms. Neither arm uses decoded rollout. The commutativity coefficient is fixed per dataset using training-only initial-gradient calibration, without evaluation-based tuning. Both ModelNet10 arms use paired-weak P1 parents.

With physical-action supervision matched, Core P2b improves mean agreement with the prescribed nonzero relations on both benchmarks:

Dataset / metricGrounded StrictCore P2b
dSprites: closure ↓0.623 ± 0.0260.529 ± 0.063
dSprites: R,TxR,T_x bracket cosine ↑0.404 ± 0.2550.619 ± 0.324
dSprites: R,TyR,T_y bracket cosine ↑0.433 ± 0.1740.740 ± 0.132
dSprites: R,TxR,T_x norm ratio → 10.098 ± 0.0280.216 ± 0.072
dSprites: R,TyR,T_y norm ratio → 10.106 ± 0.0280.205 ± 0.076
ModelNet10: cyclic closure ↓1.046 ± 0.0120.885 ± 0.081
ModelNet10: mean bracket cosine ↑0.008 ± 0.0350.321 ± 0.124
ModelNet10: mean norm ratio → 10.213 ± 0.0300.314 ± 0.060

Mean ± sample SD over five seeds. ModelNet10 cyclic metrics are averaged within each seed before aggregation. Bold indicates the better displayed mean, not statistical significance.

These results show improvements in both bracket direction and magnitude: the norm ratios move closer to their target of one, although magnitude recovery remains incomplete. On dSprites, Core P2b additionally improves mean decoded order-gap direction, order assignment, and gap-magnitude error for both rotation–translation pairs. Thus, the benefit extends beyond structural metrics to decoded non-commuting order effects on this benchmark.

The gains are specific rather than uniform. Grounded Strict achieves slightly lower mean endpoint error on both datasets, a smaller commuting translation false gap on dSprites, and higher latent and decoded direction cosines on ModelNet10. Accurate ModelNet10 action recovery remains unresolved, as acknowledged in W1.

The central finding is therefore improved recovery of prescribed nonzero relations with action supervision matched, with additional decoded order-effect benefits on dSprites. We distinguish this evidence from the original Strict comparison, which evaluates grounding and refinement jointly.

Manuscript changes. We have added a summary of this comparison to “P2: physical actions, algebra, and decoded effects,” under “Grounded refinement versus Strict.” The full results and matched protocol are reported in Appendix F.3, under “Supervision-matched commutativity control,” and referenced from the main text. The revised discussion distinguishes this evidence from the original supervision-unmatched comparison, emphasizing improved structural agreement while acknowledging incomplete magnitude recovery and dataset-dependent decoded benefits.

AuthorsOpenReview ↗

Anwer to the raised questions

Response to W3

We agree that the original experiments do not establish that the categorical carrier or improved branch specialization is necessary for order-aware action recovery.

Our architectural motivation is to distinguish category-related content from continuous geometry. In our model, Lie-algebra-generated actions parameterize continuous transformations, such as rotations and translations, while category identity is treated as a discrete factor preserved by those transformations. The categorical carrier supplies a category-dependent starting state, which the continuous Lie action transforms. This provides an inductive bias toward separating content from transformation; it does not guarantee semantic separation or imply that continuous-only models cannot represent categories. Our P1 audits explicitly expose this limitation under unsupervised training.

The skeleton experiments do not model a categorical factor and therefore use a shared carrier. They evaluate physical-action grounding and algebraic refinement, rather than the contribution of a categorical branch.

To address the missing downstream evaluation, we have now completed P2a, Core P2b and P2b+RR training from five paired-weak P1 checkpoints on dSprites, comparing them with the archived unsupervised-P1 results using the corresponding downstream recipes and identical development evaluation records. For the final P2b+RR models:

MetricUnsupervised P1Paired-weak P1
Decoded endpoint L2L_2 ↓8.509 ± 0.8677.451 ± 0.391
Decoded R,TxR,T_x gap cosine ↑0.146 ± 0.0860.282 ± 0.068
Decoded R,TyR,T_y gap cosine ↑0.153 ± 0.0460.287 ± 0.052
Commuting translation false gap ↓0.366 ± 0.1970.165 ± 0.078
Reconstruction BCE ↓20.217 ± 3.35562.685 ± 12.165

Mean ± sample SD over five seeds. Bold indicates the better displayed mean, not statistical significance.

Paired-weak initialization also improves mean order assignment for both rotation–translation pairs. These results show better downstream direction, order-assignment and endpoint accuracy, with approximately 12.4% lower mean endpoint error. However, decoded gap-magnitude errors and reconstruction BCE are worse, so the benefits are not uniform.

This experiment supplies the previously missing downstream evaluation of paired-weak P1 and shows that the alternative P1 training configuration can support improved action-level results. It does not establish categorical-branch necessity or isolate specialization as the cause: the existing P1 families differ in objectives and training duration—49 paired-weak versus 53 unsupervised epochs. These are development-set comparisons, not evidence of held-out generalization. We therefore distinguish the motivation for the mixed-latent architecture, measured branch specialization, and downstream performance, rather than treating any one as proof of the others.

Manuscript changes. We have added these findings and a comparison table to the P1 results subsection, under “Downstream effect of P1 supervision,” reporting both the action-level improvements and reconstruction and gap-magnitude trade-offs.

Response to W4

We agree. For P2 variants sharing the same P1 checkpoint, FVM evaluates the frozen encoder and is therefore unchanged by construction. Its comparison with external baselines characterizes the inherited P1 representation, not a benefit of algebraic refinement.

We have clarified this in the abstract and the “Representation quality” results paragraph, explicitly separating inherited FVM scores from P2’s action-level gains. Reconstruction is discussed separately: unlike encoder-based FVM, it can change when P2 updates the Lie bases, even with the encoder and observation decoder frozen.

Response to the minor comment: CFASL reference

Thank you for identifying this error. We have corrected the CFASL entry using the official TMLR/OpenReview bibliographic record. We have also checked the remaining references against their official records.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.