- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes MaDeL, a two-stage feature loss for generative modeling: one learned representation is trained to recover clean samples from corrupted inputs, while a complementary representation is trained to recover the corruption. The resulting feature losses are intended to distinguish manifold-parallel variation from off-manifold deviation, and are added to otherwise standard generative regression objectives. The authors provide local theory showing that, under suitable bottlenecks, the two encoder Jacobians align with tangent and normal directions, and evaluate the method on synthetic, Earth/climate, torsion-angle, and protein-backbone benchmarks.
Critical strengths
- The paper addresses a clear and practically important limitation of isotropic generative regression objectives: equal-magnitude errors can have very different meanings depending on whether they are tangent or normal to the data support.
- The proposed method is conceptually simple and modular: it modifies the regression loss rather than requiring an explicit manifold parametrization or a pretrained geometry model.
- The paired x-prediction and \delta-prediction objectives give a clean, interpretable motivation for complementary tangent- and normal-sensitive features, and the controlled OTM comparison isolates the effect of the loss.
- The theoretical results are useful as a local explanation of the intended representation geometry, even if they are not mathematically deep.
Critical weaknesses
- Global/topological scope is not adequately addressed. The theory characterizes locally optimal encoder Jacobians at individual points and is explicitly local and leading-order in the corruption level. It does not establish that a single globally trained encoder realizes the claimed tangent/normal decomposition everywhere on a compact or topologically nontrivial manifold. This distinction matters because the paper’s motivation concerns manifolds such as circles, spheres, and tori. A minimal-dimensional tangent encoder with full rank everywhere on a compact boundaryless r-manifold would give a local diffeomorphism onto an open subset of \mathbb R^r, which is incompatible with compactness. Similarly, a full-rank normal encoder with n_\perp=d-r would globally trivialize the normal bundle, which need not be possible. These are not objections to the local propositions, but they show that local optimality cannot be directly promoted to a global guarantee. The paper should state more clearly which claims are local, which are empirical, and what topological or global limitations follow for compact manifolds such as S^1, S^2, and \mathbb T^d.
- The relation between learned features and an explicit parametrization should be clarified. The method does not require the user to supply a chart, but Figure 2 uses an oracle decomposition in polar coordinates, where the angular coordinate is many-to-one. This makes the toy example highly structured and potentially easier than the general setting. It would strengthen the paper to clarify whether the learned encoders are expected to recover a comparable decomposition without such an oracle coordinate, and whether many-to-one latent representations cause practical ambiguity or failure modes. In particular, are there distributions for which the learned feature geometry is substantially worse than an intrinsic Riemannian formulation?
- The empirical corruption models do not directly match the theoretical assumptions. The theory assumes additive isotropic Gaussian corruption in the ambient space. The sphere experiments instead use tangent-space corruption, the torus experiments use wrapped angular corruption, and the protein experiments use structured corruptions including segment displacements, chain breaks, and bond/angle/torsion perturbations. These choices may be sensible, but they weaken the direct connection between Propositions 3–4 and the experiments. For example, tangent corruption on S^2 does not expose the encoder to arbitrary ambient normal perturbations, while wrapped corruption on the torus remains on the known state manifold. Please explain why these deviations are appropriate, whether isotropic Gaussian corruption is insufficient, and which theoretical assumptions are actually exercised in each experiment.
- The Earth/climate encoder dimensions do not match the normal-bottleneck theory. For data on S^2\subset\mathbb R^3, the encoder uses n_\parallel=n_\perp=2. Thus n_\parallel+n_\perp=4>3, so the setup is overparameterized; more importantly, n_\perp=2 exceeds the codimension d-r=1, so the condition needed for the orthogonal branch to lie in the normal space is not met. Please explain the intended data manifold and intrinsic dimension here, and clarify whether the Earth experiments are meant as direct validation of the bottleneck theory or as a practical application in which the theory only provides motivation.
- The comparative claims need clearer scoping. The main tables compare all methods under single-step sampling, while the appendix states that the flow-based methods are also evaluated at 2, 5, and 10 steps. Single-step evaluation is legitimate if cost-effective one-step generation is the intended setting, but it should not support a broader claim of superiority over diffusion/flow methods. Please state the intended contribution more explicitly. If the goal is a better one-step generator, report the multi-step results and emphasize the NFE/cost setting. If the goal is a generally improved geometric loss, include performance of the baselines under their intended multi-step sampling regimes.
Questions for authors
- Why is tangent-space Gaussian corruption used for the spherical data when the theory assumes additive isotropic Gaussian corruption in the ambient space? Does isotropic ambient corruption perform worse, or does it create a different learning problem?
- For protein backbones, why use highly structured corruptions rather than the isotropic corruption assumed in the theory? Is Gaussian corruption alone insufficient, or was the structured corruption chosen for domain-specific validity?
- For Earth and climate data on S^2, why choose n_\parallel=n_\perp=2? Since n_\perp=2>d-r=1, this does not satisfy the normal-bottleneck condition in Proposition 4. What manifold dimension and normal space are the authors intending to model?
- Do the learned encoders recover a usable decomposition without an oracle coordinate system such as polar coordinates? How sensitive are the results to the many-to-one nature of such coordinates, and are there distributions where this ambiguity materially harms generation?
- Can the authors report the multi-step results for RFM, MeanFlow, and RMF in the main comparison, or otherwise clarify that the claimed improvements are specific to the one-step regime?
5 comments
Reply to Reviewer zm74 (Part 2 of 2)
[C3] Reviewer zm74 asked why the Earth encoder uses two normal features despite the sphere’s codimension of one, and whether this is a practical capacity choice or a direct validation of the theory.
We assure the reviewer that MaDeL still improves over the baseline with feature dimensions satisfying the theoretical dimension bounds. Practical choices with greater feature capacity can further improve performance: on Earth with intrinsic corruption, we obtain additional gains when the combined feature dimension exceeds the ambient dimension. Appendix A.3 (Effect of Feature Dimensions) compares these configurations:
Earth (Table 5, three seeds). Reducing the normal branch from two features to one preserves the improvement over OTM on every dataset under both corruption types. With intrinsic corruption, average error is 5.39° with one normal feature, compared with 5.79° for OTM and 5.08° with two. With ambient corruption, the two choices perform similarly (5.35° versus 5.37°). Thus, the gains persist within the bounds, although extra capacity can help.
Helix (Fig. 10). Reducing the parallel branch from three features to one also preserves the qualitative benefit: both configurations recover the helical structure and substantially improve upon OT + MSE. The original three features accommodate the containing three-dimensional signal subspace (Appendix D.2), but are not necessary to obtain this improvement.
[C4] Reviewer zm74 asked us to report multi-step comparisons or explicitly scope the Earth and torsion-angle claims to single-step generation.
Thank you for pointing out the need to scope these comparisons more clearly. Our Earth and torsion-angle results demonstrate improvements in single-step generation (NFE = 1). We now state this setting explicitly in Sec. 5, Training and evaluation (lines 403–404), and Appendix D.1, with results in Sec. 5.1 and Table 2.
A Kind Reminder for Comments
We sincerely thank Reviewer zm74 for the constructive feedback on the local scope of the theory, corruption choices, feature dimensions, and evaluation setting. Our responses and revised paper address these concerns through the following evidence and clarifications:
- Local geometry and global limitations: Distinguished the oracle circle illustration from learned features (Sections 3–4) and added a learned-circle experiment (Appendix A.1, Figures 6–7). Learned features show useful local sensitivities but can develop severe distortion; branch-wise gradient normalization reduces the off-manifold rate from 69.8% to 2.9%. We distinguish these empirical findings from a global decomposition guarantee.
- Ambient versus intrinsic corruption: Clarified the distinction between the known state space and the unknown data manifold (Section 4.3) and compared both corruption choices (Section 5.1, Table 2, Figure 5). Ambient Gaussian corruption also improves generation; intrinsic corruption further improves average Earth error from 5.37° to 5.08°. On SCOPe, both variants improve steric geometry at 1M steps, with intrinsic corruption generally performing better at five and ten network evaluations.
- Feature capacity and theoretical dimension bounds: Added Earth and helix dimension ablations (Appendix A.3, Table 5, Figure 10). The gains persist when feature dimensions satisfy the theoretical dimension bounds, while additional capacity can improve the Earth results under intrinsic corruption.
- Single-step evaluation scope: Explicitly stated that the Earth and torsion-angle comparisons evaluate single-step generation with NFE = 1 (Section 5, Appendix D.1). The comparative claims for these benchmarks concern this sampling budget.
We appreciate the reviewer's time and would welcome any remaining questions about these revisions. Changes in the revised paper are highlighted in blue.
Reply to Reviewer zm74 (Part 1 of 2)
[C1] Reviewer
zm74asked us to distinguish local theoretical guarantees from globally learned geometry, clarify the role of oracle coordinates, and examine whether topological constraints or feature distortion can harm generation.Thank you for highlighting the topological constraints. The new learned-circle experiment shows useful angular and radial sensitivities over much of the manifold, alongside a localized distortion that harms generation; branch-wise gradient normalization mitigates this failure (Appendix A.1, Figs. 6–7). These findings support useful local geometry, not a globally valid coordinate system: Propositions 3–4 remain local (Sec. 4.2), and MaDeL uses feature differences without requiring a unique global parameterization (Sec. 4.3).
We have clarified that the angle/radius construction in Sec. 3 is an oracle illustration. The opening of Sec. 4 (lines 190–196) explicitly distinguishes it from learning features when these coordinates are unavailable. We also added a learned-circle experiment (Appendix A.1, Figs. 6–7): learned scalar features show angular and radial sensitivity over much of the circle, but the angular feature develops a narrow region of severe distortion. This demonstrates that useful local alignment can coexist with poor global conditioning, and that ill-conditioned learned features can harm generation. The off-manifold rate rises from 42.2% for MSE alone to 69.8% with the unnormalized feature loss; branch-wise gradient normalization reduces it to 2.9%, with mean radial error falling from 0.175 to 0.021.
Thus, poor global conditioning can harm generation even when local alignment is useful, but its adverse effects can be mitigated through appropriate optimization techniques, as demonstrated here by branch-wise gradient normalization. We did not observe this pathology in our main experiments. We distinguish this empirical observation from a global guarantee: the theory remains local, and the oracle construction is illustrative. The curved-image experiment in Appendix A.2 (Figs. 8–9) additionally shows complementary learned sensitivities across the tested curvatures and finite noise levels.
[C2] Reviewer
zm74asked us to justify intrinsic and structured corruptions given the theory’s isotropic Gaussian assumption, and compare corruption choices on Earth, torsion angles, and protein backbones.Thank you for asking us to clarify these choices. We have added ambient-versus-intrinsic corruption comparisons on Earth and torsion angles (Sec. 5.1, Table 2) and on SCOPe protein backbones (Sec. 5.1, Fig. 5). These comparisons address whether MaDeL's gains depend on corruption choices outside the theory's isotropic Gaussian assumption. We also revised Sec. 4.3, Learning the encoders (lines 346–349), to distinguish the known state space from the unknown data manifold and clarify which experiments fall within the theoretical assumptions.
Clarification of the theory's scope. The sphere and torus are known state spaces; the data distribution supported within them is unknown. Intrinsic corruption uses tangent-space Gaussian perturbations on Earth and wrapped angular perturbations on the tori (Appendices D.3–D.4), whereas ambient Gaussian corruption can move samples outside these state spaces. The revised text makes clear that intrinsic-corruption experiments are practical extensions, rather than direct tests of the isotropic-noise propositions. For protein backbones, the structured corruptions target domain-specific defects, including segment displacements, chain breaks, and bond, angle, and torsion perturbations (Appendix D.5).
Added Earth and torsion-angle comparisons (Sec. 5.1, Table 2; lines 466–475). With isotropic ambient Gaussian corruption, adding the feature loss reduces average Earth error from 5.79° for OTM without the feature loss to 5.37°, and torsion-angle error from 10.47° to 9.52°. Intrinsic corruption gives 5.08° and 9.51°, respectively, with the preferred corruption varying by dataset. These results address the concern that the improvements might require intrinsic corruption: gains persist with the corruption assumed by the theory, while intrinsic corruption provides a further benefit on Earth.
Added SCOPe comparison (Sec. 5.1, Fig. 5). We compare ambient Gaussian and structured intrinsic corruption at the same budget of 1M generator steps. Both reduce steric overlap relative to RMF. Their results are similar at one NFE, while intrinsic corruption generally gives lower overlap at five and ten NFEs, especially for longer backbones. This comparison tests whether the protein results depend on the hand-designed structural corruptions: ambient corruption also yields improvements, although structured corruption offers additional benefits in these settings. Together, the added experiments establish intrinsic corruption as a useful practical option rather than a requirement for MaDeL's gains.