- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a staged Lie-group VAE. P1 learns a categorical carrier and continuous Lie bases. P2a fits physical-action directions from labeled transitions with P1 frozen. P2b refines directions and bases toward prescribed commutator relations, and P2b+R adds decoded rollout. It compares against an all-pairs commutative control (CLG-VAE style) on dSprites, ModelNet10, 3DShapes, Cars3D, and globally rotated MI-Motion and 3DPW poses.
Critical strengths
The evaluation separates internal algebraic agreement from observable effects in latent and decoded space, uses five seeds with standard deviations, and reports mixed results openly. On MI-Motion, refinement clearly improves decoded order effects: the AB/BA gap cosine rises from 0.587 (Strict) to 0.785–0.802, and the gap-magnitude error drops from 0.120 to about 0.028.
Critical weaknesses
-
Several key metrics measure what P2b is trained on. P2b fits bracket residuals to prescribed structure constants, and closure, bracket cosine and norm ratio then evaluate the same quantities, while the Strict control is commutative by design and, for images, has no physical-action targets. Where gains matter most, in observable effects, they are small or absent: on 3DPW, decoded endpoint SSE is 0.512/0.503 versus 0.500 for Strict.
-
On two of four image benchmarks the learned actions barely align with the physical ones. On ModelNet10, the values reported as ×10⁻² give a decoded order-gap cosine of 0.030 and single-action cosine of 0.007 for P2b+R, both below P2a-only. On Cars3D, azimuth cosine is 0.066 (latent) and 0.077 (decoded). The method also assumes the group algebra is known in advance.
Questions for authors
-
If the structure constants of SE(2)/SO(3) are given, why not use the known generators directly? What does learning the Lie bases add in this setting?
-
For the image benchmarks, can you report Strict with the same physical-action targets, as done for skeletons in Table 5?
-
Given cosines near zero on ModelNet10 and Cars3D azimuth, does any variant actually learn these actions?
Response to the concerns about clarity, motivation, and the multi-stage framework
We thank the reviewer for highlighting the need to explain the relationship between the research objective, training stages, and losses more clearly. We have added targeted explanations to the Introduction and Sections 2–2.3.
What is the central research problem?
Our central question is: given observations linked by known physical transformations and prescribed composition laws, can learned latent actions reproduce both the transformations and the correct effects of composing them in different orders?
The objective is not simply to produce nonzero commutators. An order-sensitive latent model can still predict the wrong physical effect, while accurate reconstruction does not establish that its latent directions represent the intended actions. We therefore assess agreement with finite physical-action targets, prescribed local composition relations, and decoded consequences. Categorical–continuous specialization is a supporting representation question, not a separate definition of the primary objective.
Why separate representation learning, grounding, and refinement?
The stages address distinct gaps:
P1 provides a reconstructive representation. It learns the carrier, continuous Lie parameterization, and decoder. Reconstruction alone, however, does not identify which combinations of the learned bases correspond to a specified rotation or translation.
P2a identifies the physical actions. It learns action-specific combinations of the P1 bases using known transformation targets, while keeping P1 frozen. This assigns physical meaning to latent directions against fixed reference states and targets.
P2b refines their composition relations. Starting from the grounded actions, it updates the directions and Lie bases toward the prescribed commuting or non-commuting relations while retaining finite-action targets. Encoder, carrier, and decoder parameters remain frozen. The optional rollout extension directly matches decoded consequences through the frozen decoder.
This decomposition allows us to study grounding and algebraic refinement separately from representation learning.
Why combine these losses?
The losses impose complementary constraints on the same action model:
Our ablations assess the empirical contribution of these choices rather than establish that every term is indispensable. For example, the matched dSprites comparison of finite-only+R and Core P2b+R shows improved algebraic and decoded order-effect agreement from adding bracket losses, but slightly worse mean endpoint error. We therefore distinguish benefits for composition agreement from uniform improvement across metrics.
Manuscript revisions.
The Introduction now explicitly states the primary research question. Section 2 explains the rationale for staging; Section 2.1 motivates the code and leakage penalties; Section 2.2 clarifies signed-gap matching; and Section 2.3 connects the finite-action, bracket, decoded, scale, and anchor terms. We also clarify that latent-action, Lie-structure, and decoded-observation metrics assess complementary requirements of the central objective, they are diagnostics, not additional training objectives.