acceptodds
Under review as a conference paper at ICLR 2027

MaDeL: Manifold-Decomposed Feature Losses for Generative Modeling

Abstract

Generative models are often trained with isotropic objectives such as mean-squared error. For data concentrated near a low-dimensional manifold, however, such losses conflate displacement along the manifold, which may represent valid variation, with displacement away from it, which produces invalid samples. This mismatch is especially problematic in sparse, highly constrained domains, where ambient-space regression can encourage off-manifold interpolation. We ask whether a generative objective can distinguish manifold-parallel variation from manifold-orthogonal deviation directly from data, without explicitly estimating the manifold. We introduce a manifold-decomposed feature loss (MaDeL) that learns complementary representations from corrupted observations: one is trained to recover the clean sample, while the other is trained to recover the corruption. We show that, under a feature bottleneck, their Jacobians align with the tangent and normal spaces, exactly for linear manifolds and locally for smooth manifolds. Together, these representations define an anisotropic objective that separately measures intrinsic variation and off-manifold deviation. Across synthetic, Earth and climate science, and torsion-angle benchmarks, MaDeL improves support recovery and average angular under single-step sampling; on protein backbones, it reduces steric clashes across one- and few-step sampling budgets.

Primary area:
generative models
Keywords:
Manifold, Generative Modeling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zm74OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MaDeL, a two-stage feature loss for generative modeling: one learned representation is trained to recover clean samples from corrupted inputs, while a complementary representation is trained to recover the corruption. The resulting feature losses are intended to distinguish manifold-parallel variation from off-manifold deviation, and are added to otherwise standard generative regression objectives. The authors provide local theory showing that, under suitable bottlenecks, the two encoder Jacobians align with tangent and normal directions, and evaluate the method on synthetic, Earth/climate, torsion-angle, and protein-backbone benchmarks.

Critical strengths
  1. The paper addresses a clear and practically important limitation of isotropic generative regression objectives: equal-magnitude errors can have very different meanings depending on whether they are tangent or normal to the data support.
  2. The proposed method is conceptually simple and modular: it modifies the regression loss rather than requiring an explicit manifold parametrization or a pretrained geometry model.
  3. The paired x-prediction and \delta-prediction objectives give a clean, interpretable motivation for complementary tangent- and normal-sensitive features, and the controlled OTM comparison isolates the effect of the loss.
  4. The theoretical results are useful as a local explanation of the intended representation geometry, even if they are not mathematically deep.
Critical weaknesses
  1. Global/topological scope is not adequately addressed. The theory characterizes locally optimal encoder Jacobians at individual points and is explicitly local and leading-order in the corruption level. It does not establish that a single globally trained encoder realizes the claimed tangent/normal decomposition everywhere on a compact or topologically nontrivial manifold. This distinction matters because the paper’s motivation concerns manifolds such as circles, spheres, and tori. A minimal-dimensional tangent encoder with full rank everywhere on a compact boundaryless r-manifold would give a local diffeomorphism onto an open subset of \mathbb R^r, which is incompatible with compactness. Similarly, a full-rank normal encoder with n_\perp=d-r would globally trivialize the normal bundle, which need not be possible. These are not objections to the local propositions, but they show that local optimality cannot be directly promoted to a global guarantee. The paper should state more clearly which claims are local, which are empirical, and what topological or global limitations follow for compact manifolds such as S^1, S^2, and \mathbb T^d.
  2. The relation between learned features and an explicit parametrization should be clarified. The method does not require the user to supply a chart, but Figure 2 uses an oracle decomposition in polar coordinates, where the angular coordinate is many-to-one. This makes the toy example highly structured and potentially easier than the general setting. It would strengthen the paper to clarify whether the learned encoders are expected to recover a comparable decomposition without such an oracle coordinate, and whether many-to-one latent representations cause practical ambiguity or failure modes. In particular, are there distributions for which the learned feature geometry is substantially worse than an intrinsic Riemannian formulation?
  3. The empirical corruption models do not directly match the theoretical assumptions. The theory assumes additive isotropic Gaussian corruption in the ambient space. The sphere experiments instead use tangent-space corruption, the torus experiments use wrapped angular corruption, and the protein experiments use structured corruptions including segment displacements, chain breaks, and bond/angle/torsion perturbations. These choices may be sensible, but they weaken the direct connection between Propositions 3–4 and the experiments. For example, tangent corruption on S^2 does not expose the encoder to arbitrary ambient normal perturbations, while wrapped corruption on the torus remains on the known state manifold. Please explain why these deviations are appropriate, whether isotropic Gaussian corruption is insufficient, and which theoretical assumptions are actually exercised in each experiment.
  4. The Earth/climate encoder dimensions do not match the normal-bottleneck theory. For data on S^2\subset\mathbb R^3, the encoder uses n_\parallel=n_\perp=2. Thus n_\parallel+n_\perp=4>3, so the setup is overparameterized; more importantly, n_\perp=2 exceeds the codimension d-r=1, so the condition needed for the orthogonal branch to lie in the normal space is not met. Please explain the intended data manifold and intrinsic dimension here, and clarify whether the Earth experiments are meant as direct validation of the bottleneck theory or as a practical application in which the theory only provides motivation.
  5. The comparative claims need clearer scoping. The main tables compare all methods under single-step sampling, while the appendix states that the flow-based methods are also evaluated at 2, 5, and 10 steps. Single-step evaluation is legitimate if cost-effective one-step generation is the intended setting, but it should not support a broader claim of superiority over diffusion/flow methods. Please state the intended contribution more explicitly. If the goal is a better one-step generator, report the multi-step results and emphasize the NFE/cost setting. If the goal is a generally improved geometric loss, include performance of the baselines under their intended multi-step sampling regimes.
Questions for authors
  1. Why is tangent-space Gaussian corruption used for the spherical data when the theory assumes additive isotropic Gaussian corruption in the ambient space? Does isotropic ambient corruption perform worse, or does it create a different learning problem?
  2. For protein backbones, why use highly structured corruptions rather than the isotropic corruption assumed in the theory? Is Gaussian corruption alone insufficient, or was the structured corruption chosen for domain-specific validity?
  3. For Earth and climate data on S^2, why choose n_\parallel=n_\perp=2? Since n_\perp=2>d-r=1, this does not satisfy the normal-bottleneck condition in Proposition 4. What manifold dimension and normal space are the authors intending to model?
  4. Do the learned encoders recover a usable decomposition without an oracle coordinate system such as polar coordinates? How sensitive are the results to the many-to-one nature of such coordinates, and are there distributions where this ambiguity materially harms generation?
  5. Can the authors report the multi-step results for RFM, MeanFlow, and RMF in the main comparison, or otherwise clarify that the claimed improvements are specific to the one-step regime?
5 comments
AuthorsOpenReview ↗

Reply to Reviewer zm74 (Part 1 of 2)


[C1] Reviewer zm74 asked us to distinguish local theoretical guarantees from globally learned geometry, clarify the role of oracle coordinates, and examine whether topological constraints or feature distortion can harm generation.

Thank you for highlighting the topological constraints. The new learned-circle experiment shows useful angular and radial sensitivities over much of the manifold, alongside a localized distortion that harms generation; branch-wise gradient normalization mitigates this failure (Appendix A.1, Figs. 6–7). These findings support useful local geometry, not a globally valid coordinate system: Propositions 3–4 remain local (Sec. 4.2), and MaDeL uses feature differences without requiring a unique global parameterization (Sec. 4.3).

We have clarified that the angle/radius construction in Sec. 3 is an oracle illustration. The opening of Sec. 4 (lines 190–196) explicitly distinguishes it from learning features when these coordinates are unavailable. We also added a learned-circle experiment (Appendix A.1, Figs. 6–7): learned scalar features show angular and radial sensitivity over much of the circle, but the angular feature develops a narrow region of severe distortion. This demonstrates that useful local alignment can coexist with poor global conditioning, and that ill-conditioned learned features can harm generation. The off-manifold rate rises from 42.2% for MSE alone to 69.8% with the unnormalized feature loss; branch-wise gradient normalization reduces it to 2.9%, with mean radial error falling from 0.175 to 0.021.

Thus, poor global conditioning can harm generation even when local alignment is useful, but its adverse effects can be mitigated through appropriate optimization techniques, as demonstrated here by branch-wise gradient normalization. We did not observe this pathology in our main experiments. We distinguish this empirical observation from a global guarantee: the theory remains local, and the oracle construction is illustrative. The curved-image experiment in Appendix A.2 (Figs. 8–9) additionally shows complementary learned sensitivities across the tested curvatures and finite noise levels.


[C2] Reviewer zm74 asked us to justify intrinsic and structured corruptions given the theory’s isotropic Gaussian assumption, and compare corruption choices on Earth, torsion angles, and protein backbones.

Thank you for asking us to clarify these choices. We have added ambient-versus-intrinsic corruption comparisons on Earth and torsion angles (Sec. 5.1, Table 2) and on SCOPe protein backbones (Sec. 5.1, Fig. 5). These comparisons address whether MaDeL's gains depend on corruption choices outside the theory's isotropic Gaussian assumption. We also revised Sec. 4.3, Learning the encoders (lines 346–349), to distinguish the known state space from the unknown data manifold and clarify which experiments fall within the theoretical assumptions.

Clarification of the theory's scope. The sphere and torus are known state spaces; the data distribution supported within them is unknown. Intrinsic corruption uses tangent-space Gaussian perturbations on Earth and wrapped angular perturbations on the tori (Appendices D.3–D.4), whereas ambient Gaussian corruption can move samples outside these state spaces. The revised text makes clear that intrinsic-corruption experiments are practical extensions, rather than direct tests of the isotropic-noise propositions. For protein backbones, the structured corruptions target domain-specific defects, including segment displacements, chain breaks, and bond, angle, and torsion perturbations (Appendix D.5).

Added Earth and torsion-angle comparisons (Sec. 5.1, Table 2; lines 466–475). With isotropic ambient Gaussian corruption, adding the feature loss reduces average Earth error from 5.79° for OTM without the feature loss to 5.37°, and torsion-angle error from 10.47° to 9.52°. Intrinsic corruption gives 5.08° and 9.51°, respectively, with the preferred corruption varying by dataset. These results address the concern that the improvements might require intrinsic corruption: gains persist with the corruption assumed by the theory, while intrinsic corruption provides a further benefit on Earth.

Added SCOPe comparison (Sec. 5.1, Fig. 5). We compare ambient Gaussian and structured intrinsic corruption at the same budget of 1M generator steps. Both reduce steric overlap relative to RMF. Their results are similar at one NFE, while intrinsic corruption generally gives lower overlap at five and ten NFEs, especially for longer backbones. This comparison tests whether the protein results depend on the hand-designed structural corruptions: ambient corruption also yields improvements, although structured corruption offers additional benefits in these settings. Together, the added experiments establish intrinsic corruption as a useful practical option rather than a requirement for MaDeL's gains.


AuthorsOpenReview ↗

Reply to Reviewer zm74 (Part 2 of 2)


[C3] Reviewer zm74 asked why the Earth encoder uses two normal features despite the sphere’s codimension of one, and whether this is a practical capacity choice or a direct validation of the theory.

We assure the reviewer that MaDeL still improves over the baseline with feature dimensions satisfying the theoretical dimension bounds. Practical choices with greater feature capacity can further improve performance: on Earth with intrinsic corruption, we obtain additional gains when the combined feature dimension exceeds the ambient dimension. Appendix A.3 (Effect of Feature Dimensions) compares these configurations:

Earth (Table 5, three seeds). Reducing the normal branch from two features to one preserves the improvement over OTM on every dataset under both corruption types. With intrinsic corruption, average error is 5.39° with one normal feature, compared with 5.79° for OTM and 5.08° with two. With ambient corruption, the two choices perform similarly (5.35° versus 5.37°). Thus, the gains persist within the bounds, although extra capacity can help.

Helix (Fig. 10). Reducing the parallel branch from three features to one also preserves the qualitative benefit: both configurations recover the helical structure and substantially improve upon OT + MSE. The original three features accommodate the containing three-dimensional signal subspace (Appendix D.2), but are not necessary to obtain this improvement.


[C4] Reviewer zm74 asked us to report multi-step comparisons or explicitly scope the Earth and torsion-angle claims to single-step generation.

Thank you for pointing out the need to scope these comparisons more clearly. Our Earth and torsion-angle results demonstrate improvements in single-step generation (NFE = 1). We now state this setting explicitly in Sec. 5, Training and evaluation (lines 403–404), and Appendix D.1, with results in Sec. 5.1 and Table 2.


AuthorsOpenReview ↗

A Kind Reminder for Comments

We sincerely thank Reviewer zm74 for the constructive feedback on the local scope of the theory, corruption choices, feature dimensions, and evaluation setting. Our responses and revised paper address these concerns through the following evidence and clarifications:

  • Local geometry and global limitations: Distinguished the oracle circle illustration from learned features (Sections 3–4) and added a learned-circle experiment (Appendix A.1, Figures 6–7). Learned features show useful local sensitivities but can develop severe distortion; branch-wise gradient normalization reduces the off-manifold rate from 69.8% to 2.9%. We distinguish these empirical findings from a global decomposition guarantee.
  • Ambient versus intrinsic corruption: Clarified the distinction between the known state space and the unknown data manifold (Section 4.3) and compared both corruption choices (Section 5.1, Table 2, Figure 5). Ambient Gaussian corruption also improves generation; intrinsic corruption further improves average Earth error from 5.37° to 5.08°. On SCOPe, both variants improve steric geometry at 1M steps, with intrinsic corruption generally performing better at five and ten network evaluations.
  • Feature capacity and theoretical dimension bounds: Added Earth and helix dimension ablations (Appendix A.3, Table 5, Figure 10). The gains persist when feature dimensions satisfy the theoretical dimension bounds, while additional capacity can improve the Earth results under intrinsic corruption.
  • Single-step evaluation scope: Explicitly stated that the Earth and torsion-angle comparisons evaluate single-step generation with NFE = 1 (Section 5, Appendix D.1). The comparative claims for these benchmarks concern this sampling budget.

We appreciate the reviewer's time and would welcome any remaining questions about these revisions. Changes in the revised paper are highlighted in blue.

Reviewer_EzMsOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes an objective for generative modeling that directly distinguishes manifold-parallel variation from manifold-orthogonal deviation from data, without explicitly estimating the manifold. Training involves two stages: in the first, a denoising autoencoder with a shared encoder and dual decoder learns to separate manifold-parallel and manifold-orthogonal features. Then the second stage involves learning a generator with the frozen encoder that can provide complementary update directions along and away from the data manifold. With experiments on several datasets that live in a lower-dimensional manifold, the authors show that their approach performs better for most cases.

Critical strengths
  1. The paper is well written. I appreciate the intuition behind the theory presented.
  2. The problem is well motivated. Many areas of application will benefit from manifold-aware generative modeling and feature extraction.
  3. The empirical performance shows clear improvement with this approach over the baseline.
Critical weaknesses

An important baseline: "Metric Flow Matching for Smooth Interpolations on the Data Manifold" (Kapusniak et al. 2024) is missing from the empirical study

Questions for authors

None

5 comments
AuthorsOpenReview ↗

Reply to Reviewer EzMs


[C1] Reviewer EzMs asked us to include “Metric Flow Matching for Smooth Interpolations on the Data Manifold” (Kapusniak et al., 2024) as a baseline.

Thank you for pointing out this relevant baseline. MaDeL achieves lower average single-step errors than Metric Flow Matching (MFM; Kapusniak et al., 2024) on both Earth and torsion angles at both tested baseline training budgets. We added MFM with its native training objective and report three-seed results (Sec. 5, Baselines, lines 414–420; Sec. 5.1, Table 2).

At 10k training steps, MFM obtains average angular errors of 70.32 ± 3.45° on Earth and 68.49 ± 3.51° on torsion angles; extending training to the sample-matched 90k budget gives 71.03 ± 3.57° and 69.25 ± 3.60°, respectively. Intrinsic-corruption MaDeL obtains 5.08 ± 0.04° and 9.51 ± 0.27°. MaDeL achieves lower errors in our single-step evaluation.


I thank the authors for running this additional baseline. Can the authors clarify if the MaDeL numbers are also at 10k or 90k steps, as was reported for MFM? Having a table and details about the experiments would help me understand more.

Thank you for the prompt reply. Both MaDeL variants use 10k generator-training steps, preceded by 5k encoder-training steps. MFM is evaluated at both 10k and 90k generator-training steps. We reproduce the comparison below for convenience.

MethodEncoder stepsGenerator stepsEarth ↓Protein/RNA torsion angles ↓
MFM—10k70.32 ± 3.45°68.49 ± 3.51°
MFM—90k71.03 ± 3.57°69.25 ± 3.60°
OTM + MaDeL (ambient corruption)5k10k5.37 ± 0.29°9.52 ± 0.33°
OTM + MaDeL (intrinsic corruption)5k10k5.08 ± 0.04°9.51 ± 0.27°

Values are average angular Wasserstein-1 distances across the four Earth datasets and five protein/RNA torsion-angle datasets, respectively; lower is better. We report means and standard deviations over three training seeds, computing each dataset average within a seed before summarizing across seeds. “+ MaDeL” denotes adding our frozen feature loss to the coordinate-space OT-matching baseline.

All generators are trained with batch size 128 and are evaluated with one network evaluation (NFE = 1). MaDeL’s encoder is trained with batch size 2,048 and is frozen before generator training. Consequently, MaDeL processes 5k × 2,048 + 10k × 128 = 11.52 million training samples, matching MFM at 90k × 128. The 10k MFM comparison matches generator-training steps, while the 90k comparison matches the combined encoder-and-generator sample count.

The full per-dataset results appear in Table 2, with the training setup in Section 5 and dataset-specific details in Appendices D.3–D.4.

Reviewer_kaTXOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MaDeL, a feature-space regression loss for generative models intended to separate errors along the data manifold from errors away from it. In a first stage, a shared encoder is trained on corrupted samples x + δ, with one block of features decoded to predict the clean sample x and a disjoint block decoded to predict the corruption δ. The encoder is then frozen, and the generator is trained with the usual ambient regression loss plus squared feature-space discrepancies in each block (Eq. 23). For linear Gaussian data and linear rank-constrained autoencoders, the authors show that x-prediction selects the top principal directions and δ-prediction the bottom ones (Props. 1-2); they extend this to smooth manifolds by linearizing around a point (Props. 3-4) and conclude that the two feature-loss gradients lie in the tangent and normal spaces, respectively (Prop. 5). Experiments use a one-step generator trained by minibatch OT matching on a helix embedded in high dimension, Earth science datasets on S², and protein/RNA torsion angles, reporting modest improvements in angular W1; on protein backbones, adding the loss to Riemannian MeanFlow reduces steric clashes while leaving designability unchanged.

Critical strengths

The underlying observation is reasonable: an isotropic regression loss treats on-manifold and off-manifold errors the same, and for sparse, highly constrained data (e.g., backbone geometry) it is natural to want a loss that penalizes the latter more heavily. The method is simple to bolt onto an existing objective, and the improvements, while small, are fairly consistent across the Earth and torsion benchmarks (Table 2), with a clear reduction in steric clashes on protein backbones (Fig. 5).

Critical weaknesses
  1. The motivation and theory do not convincingly support the method. The paper starts by framing diffusion and consistency models under a single isotropic regression objective (Eq. 1), which is an odd way to begin: these are quite different model classes, and in consistency training the regression target is the model's own prediction at a neighboring time rather than a data sample, so the analogy to x-prediction is loose. Sec. 3 then motivates learned feature losses, which is reasonable but a bit trivial (of course learning a feature encoder changes the geometry of the loss), and only in Sec. 4 is this connected to x-prediction and δ-prediction. I found this structure strange and confusing. More substantively, I don't really buy the general manifold theory. Props. 3-4 linearize the manifold and recover the linear result to first order. For a linear manifold the corruption splits cleanly into parallel and orthogonal components, but for a curved manifold at finite noise this is much less clear, and the local statement says little about what a single global encoder trained over all of the data will learn. The experiments also do not appear to operate in the regime the theory describes: Props. 2, 4, and 5 require n_⊥ ≤ d − r, but on the Earth datasets (S² in R³, so d − r = 1) the encoder uses n_⊥ = 2 (App. C.3), and on the helix it uses n_∥ = 3 (App. C.2), larger than the intrinsic dimension. The only direct check that the learned Jacobians align as claimed is Table 4, on the helix.
  2. The experiments are skimpy and not well controlled. The method requires training a separate encoder before generative training, and this is not cheap relative to the generator: for Earth and torsion angles the encoder sees roughly four times as many samples as the generator (10k iterations at batch 2048 versus 20k at batch 256, App. C.3-C.4), and the protein encoder is trained for 250k iterations at batch 2048. No baseline is given comparable extra compute, so it is unclear whether the gains come from the decomposed geometry or simply from more training. The results also appear to be from a single seed (App. C.1) with no error bars, the improvements are small (e.g., 5.89° to 5.59° average on Earth), the helix result is qualitative only, the ablation (Table 3) is run only on Earth, and on proteins designability is unchanged. Taken together, the empirical case is not strong enough to compensate for the weaknesses of the theory.
Questions for authors
  1. (Experiments) Can the authors report a compute-matched baseline, in which OTM (and RMF for proteins) is given additional training equal to the cost of training the encoder? If the gains persist, this would substantially strengthen the paper.
  2. (Experiments) How many seeds are the results in Tables 2 and 3 based on? Please report means and standard deviations over several seeds, since the reported gains are small.
  3. (Clarification) On the Earth datasets, n_⊥ = 2 exceeds the codimension d − r = 1, and on the helix n_∥ = 3 exceeds the intrinsic dimension. How should the theory of Sec. 4 be interpreted in these settings, and how do results change when the bottleneck dimensions satisfy the stated conditions?
  4. (Clarification) Beyond the helix (Table 4), is there evidence that the learned encoders actually align with tangent and normal directions on curved manifolds at the corruption levels used in practice?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer kaTX (Part 1 of 2)


[C1] Reviewer kaTX asked us to clarify the shared regression-loss framing for diffusion and consistency models and explain why complementary features are needed and how they connect to clean-sample and corruption prediction.

Thank you for highlighting the ambiguity. Our clarification separates two stages: clean-sample and corruption prediction learn complementary features, which are then frozen to train the generator with its method-specific targets. Sec. 2 (lines 100–106, 124–125) now clarifies that diffusion and consistency models share a regression-loss form while retaining their different targets and procedures; the common limitation is that isotropic losses do not distinguish movement along the data manifold from movement away from it.

Sec. 3 now explains why both kinds of features are needed: features that capture variation along the manifold can overlook departures from it, so we pair them with features sensitive to those departures. The circle example illustrates this idea using angle and radius (lines 174–179). Sec. 4 (lines 190–196) then explains how to learn complementary features when such coordinates are unavailable. We also make the two training stages explicit in Sec. 4.3 (lines 312–323, 350–370) and Algorithm 1: first learn the features through clean-sample and corruption prediction, then freeze them to train the generator with its own targets. The theory explains why these two auxiliary tasks encourage complementary geometric sensitivities.


[C2] Reviewer kaTX asked whether globally trained encoders retain tangent/normal alignment on curved manifolds at finite noise, and requested evidence beyond the helix.

Our added experiments show that globally learned features retain complementary tangent and normal sensitivities at finite noise, including on a curved image manifold in a 1,024-dimensional space. The circle experiment also exposes a localized distortion that can harm generation and shows that gradient normalization mitigates it. These are empirical findings beyond the local theoretical characterization in Sec. 4.2; the evidence is summarized below:

Curved image manifold (Appendix A.2, Figs. 8–9). We train encoders on translated disc images, whose position varies along a curved surface in a 1,024-dimensional image space. Across the tested disc sizes and noise levels, clean-sample prediction remains more sensitive to movement along this surface, while corruption prediction remains more sensitive to movement away from it. This directly supports the predicted behavior for globally trained encoders at finite noise.

Circle and failure mode (Appendix A.1, Figs. 6–7). Learned features capture angular and radial variation over much of the circle, but develop a sharply distorted region that can destabilize generation. Adding the feature loss without normalization worsens the off-manifold rate from 42.2% to 69.8%; normalizing each branch's gradients reduces it to 2.9%. This shows both useful learned geometry and a concrete failure mode with a practical mitigation.


[C3] Reviewer kaTX asked how to interpret the theory when Earth and helix feature dimensions exceed its bottleneck bounds, and whether the gains persist when those bounds are satisfied.

We assure the reviewer that MaDeL still improves over the baseline with feature dimensions satisfying the theoretical dimension bounds. Practical choices with greater feature capacity can further improve performance: on Earth with intrinsic corruption, we obtain additional gains when the combined feature dimension exceeds the ambient dimension. Appendix A.3 (Effect of Feature Dimensions) compares these configurations:

Earth (Table 5, three seeds). Reducing the normal branch from two features to one preserves the improvement over OTM on every dataset under both corruption types. With intrinsic corruption, average error is 5.39° with one normal feature, compared with 5.79° for OTM and 5.08° with two. With ambient corruption, the two choices perform similarly (5.35° versus 5.37°). Thus, the gains persist within the bounds, although extra capacity can help.

Helix (Fig. 10). Reducing the parallel branch from three features to one also preserves the qualitative benefit: both configurations recover the helical structure and substantially improve upon OT + MSE. The original three features accommodate the containing three-dimensional signal subspace (Appendix D.2), but are not necessary to obtain this improvement.


AuthorsOpenReview ↗

Reply to Reviewer kaTX (Part 2 of 2)


[C4] Reviewer kaTX asked for baselines given additional training to account for encoder pretraining and distinguish the effect of the feature loss from that of a larger training budget.

Thank you for raising the training-budget concern; we added these comparisons to account for the additional encoder-training stage. MaDeL's improvements persist against sample-matched baselines on Earth and torsion angles and longer-trained RMF baselines on SCOPe (Sec. 5.1, Table 2 and Fig. 5).

Earth and torsion angles (Table 2, three seeds; lines 403–411, 466–475). For sample-matched comparison, we consider MaDeL with 5k encoder steps at batch size 2,048 and 10k generator steps at batch size 128. We give each baseline 90k generator steps at batch size 128, matching MaDeL's combined 11.52 million training samples. Intrinsic-corruption MaDeL achieves average errors of 5.08 ± 0.04° on Earth and 9.51 ± 0.27° on torsion angles, compared with 5.69 ± 0.20° and 10.18 ± 0.33° for the longer-trained OTM baseline. Ambient-corruption MaDeL also improves both averages (5.37 ± 0.29° and 9.52 ± 0.33°).

SCOPe (Fig. 5). Extending RMF from 1M to 1.2M steps does not reproduce the reduction in steric overlap obtained by intrinsic-corruption MaDeL at 1M steps, across the tested backbone lengths and sampling budgets. Ambient-corruption MaDeL also reduces overlap at 1M steps. The improvements persist against the longer-trained baselines.


[C5] Reviewer kaTX asked for means and standard deviations across multiple seeds to assess the reliability of the Earth and torsion-angle results and ablations.

The three-seed results show lower average errors for MaDeL on both Earth and torsion angles, and the Earth ablation favors combining both feature branches (Tables 2–3). We now report means and standard deviations over three training seeds, with average-column statistics computed by averaging datasets within each seed before summarizing across seeds (Sec. 5, Training and evaluation, lines 403–411).

Main results (Sec. 5.1, Table 2; lines 466–475). With intrinsic corruption, MaDeL reduces average angular error from 5.79 ± 0.34° to 5.08 ± 0.04° on Earth and from 10.47 ± 0.56° to 9.51 ± 0.27° on torsion angles. Every dataset mean improves over OTM at the same generator budget, and both averages remain lower than those of sample-matched OTM trained for 90k steps. Ambient-corruption MaDeL also improves both averages.

Component ablation (Sec. 5.2, Table 3). Combining both branches gives 5.08 ± 0.04°, compared with 5.41 ± 0.65° for an overparameterized autoencoder, 5.61 ± 0.22° for the parallel branch alone, and 5.33 ± 0.19° for the normal branch alone. This supports the benefit of complementary features beyond adding an auxiliary feature loss.


AuthorsOpenReview ↗

A Kind Reminder for Comments

We sincerely thank Reviewer kaTX for the constructive feedback on the motivation, theoretical interpretation, and experimental comparisons. Our responses and revised paper address these concerns through the following evidence and clarifications:

  • Motivation and two-stage training: Reorganized the motivation and clarified that clean-sample and corruption prediction train auxiliary features, which are then frozen for generator training with method-specific targets (Sections 2–4.3, Algorithm 1).
  • Learned geometry at finite noise: Added alignment measurements on a curved image manifold in 1,024 dimensions across different curvatures and noise levels (Appendix A.2, Figures 8–9). The learned branches retain complementary tangent and normal sensitivities. The circle experiment additionally exposes a distortion-related failure mode and demonstrates gradient normalization as a mitigation (Appendix A.1, Figures 6–7).
  • Feature-dimension ablations: Added configurations satisfying the theoretical dimension bounds (Appendix A.3, Table 5, Figure 10). On Earth, one normal feature still improves every dataset mean over OTM under both corruption types; with intrinsic corruption, two normal features further reduce average error from 5.39° to 5.08°.
  • Training-budget comparisons: Added sample-matched baselines on Earth and torsion angles and longer-trained RMF baselines on SCOPe (Section 5.1, Table 2, Figure 5). Intrinsic-corruption MaDeL achieves average errors of 5.08° and 9.51°, compared with 5.69° and 10.18° for sample-matched OTM; the SCOPe steric-overlap improvements also persist against longer-trained RMF.
  • Three-seed results: Reported means and standard deviations for the Earth and torsion comparisons and the Earth component ablation (Tables 2–3). Combining the feature branches gives average Earth error of 5.08 ± 0.04°, compared with 5.79 ± 0.34° for OTM.

We appreciate the reviewer's time and would welcome any remaining questions about these revisions. Changes in the revised paper are highlighted in blue.

Dear Reviewer kaTX, thank you again for your thoughtful and constructive feedback.

As the discussion period approaches its end, we would appreciate any feedback on whether our responses and the revised manuscript have adequately addressed your concerns.

Please let us know if any questions or concerns remain. We would be happy to provide further clarification.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all three reviewers for their careful reading and constructive feedback. Their comments helped us sharpen the motivation, clarify the scope of the theory, and strengthen the experimental evidence.

Reviewer kaTX recognized the method's simplicity, fairly consistent gains on Earth and torsion benchmarks, and clear reduction in protein steric clashes; Reviewer EzMs appreciated the writing, motivation, theoretical intuition, and empirical improvements. Reviewer zm74 highlighted the practical importance of distinguishing tangent and normal errors, the method's modularity, the interpretable paired prediction tasks, and the theory's usefulness as a local explanation of the learned geometry.

We respond to each reviewer individually below and indicate where the revised manuscript addresses each concern. Changes are highlighted in blue. Unless otherwise indicated, section, figure, table, equation, and line references in our responses refer to the revised manuscript. The main revisions are as follows.

  1. Clearer motivation and training procedure (kaTX C1). Sections 2–4 now explain why we combine features sensitive to variation along the data manifold with features sensitive to departures from it. Section 4.3 and Algorithm 1 distinguish auxiliary encoder training from generator training with method-specific targets. The two prediction tasks learn the feature geometry; they do not replace the underlying generative objective.

  2. Learned geometry and its limitations (kaTX C2; zm74 C1). Appendix A.1 (Figs. 6–7) examines learned circle features without oracle coordinates, showing both useful directional alignment and a distortion-related failure mode that branch-wise gradient normalization mitigates. Appendix A.2 (Figs. 8–9) adds alignment measurements on a curved image manifold across curvatures and finite noise levels. These experiments distinguish useful learned local geometry from a global decomposition guarantee.

  3. Stronger baseline and training-budget comparisons (kaTX C4–C5; EzMs C1). Section 5.1 and Table 2 add Metric Flow Matching and sample-matched, longer-trained baselines on Earth and torsion angles, with means and standard deviations over three seeds. Figure 5 adds longer-trained RMF comparisons on SCOPe. The improvements persist against the longer-trained baselines. The Earth and torsion comparisons are explicitly scoped to single-step generation (Sec. 5 and Appendix D.1; zm74 C4).

  4. Ambient versus intrinsic corruption (zm74 C2). Section 4.3 distinguishes the known state space from the unknown data manifold, while Section 5.1 compares corruption choices on Earth and torsion angles over three seeds (Table 2) and on SCOPe at 1M generator steps (Fig. 5). Ambient Gaussian corruption also yields gains; intrinsic corruption can further improve them, particularly on Earth and for multi-step SCOPe generation.

  5. Feature components and dimensions (kaTX C3; zm74 C3). Section 5.2 and Appendix A.3 expand the feature-loss component and dimension ablations (Tables 3 and 5; Fig. 10). Combining the branches improves average Earth error over either branch alone, and the benefits persist with feature dimensions satisfying the theoretical dimension bounds. The gains are not contingent on exceeding those bounds.

We hope these responses and revisions address the reviewers' concerns, and we welcome further discussion.

AuthorsOpenReview ↗

A Kind Reminder for Comments

Dear Area Chair and Reviewers,

We sincerely thank the reviewers for their constructive feedback and careful evaluation of our work. We have provided detailed point-by-point responses and updated the manuscript with additional experiments and clarifications addressing the reviewers’ concerns. The key changes are summarized in our individual reviewer responses, and all revisions in the manuscript are highlighted in blue.

We would greatly appreciate the Area Chair’s help in inviting reviewers to engage in further discussion and consider our responses. We welcome any remaining questions and would be happy to provide further clarification.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.