acceptodds
Withdrawn from ICLR 2027

Beyond Gaussian Bottlenecks: Topologically Aligned Encoding of Vision-Transformer Feature Spaces

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pre-trained Vision Transformer (ViT) representations are rich and broadly useful, but their high dimensionality makes them expensive to store, transfer, and reuse. We study how to compress these representations while preserving downstream utility, and identify a fundamental geometric obstacle: layer-normalized ViT features concentrate on hyperspherical manifolds, whereas standard Gaussian variational autoencoders model latents in Euclidean space. This mismatch wastes capacity on a radial dimension that carries no semantic information and induces posterior collapse under strong compression. We propose SVAE, a geometry-aware variational autoencoder that replaces the Gaussian bottleneck with a product of Power Spherical distributions, aligning the latent topology with the data manifold while remaining numerically stable in high dimensions. Across five transformer backbones spanning distinct training objectives and modalities, including DINOv2, CLIP, DUSt3R, VGGT, and PAGE-4D, S²VAE consistently outperforms Gaussian baselines under aggressive compression (up to 512×), with the gap widening as the bottleneck tightens. Beyond reconstruction fidelity, we show that the learned latent space is directly usable for downstream tasks. Lightweight decoders with fewer than 0.5M parameters can recover task-relevant signals from compressed latents, and generative models trained in the latent space produce outputs that remain structurally consistent across modalities. These results indicate that geometry-aligned latent design enables compact representations without sacrificing downstream utility.

Primary area:
generative models
Keywords:
VAEs, generative models, transformers, feature compression

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7iTnOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies compression of frozen ViT features using VAEs whose bottleneck is a product of low-dimensional Power Spherical distributions. Reconstructed features are passed to frozen downstream heads. The main quantitative results show stronger performance than standard Gaussian. Additional experiments study compression dimensions, alternative bottlenecks, lightweight depth probes, and qualitative generation using a DiT trained on the learned latents.

Critical strengths
  1. The paper is easy to follow
  2. The author shows that spherical representation outperforms standard Gaussians on downstream 3D tasks. There are a ablation studies on dimensionality of VAE.
Critical weaknesses

Unclear goal. The author has an unclear goal for this paper. Specifically, if the goal is to have a better VAE for 3D generation, then there should be quantitative and qualitative results on 3D generation with various baselines, for example, Gen3R: 3D Scene Generation Meets Feed-Forward Reconstruction. If the author with to improve downstream 3D tasks such as depth prediction, camera pose estimation, and 3D reconstruction, then there should also be proper baselines. I do not know which problem the author is trying to solve.

Improper experimental evaluations. Comparing with a standard Gaussian under 3D tasks does not show a contribution to the community. The generation results are simply two images. The quantitative evaluation results are very limited. Only two tables, each with two rows. There are also limited visual examples. The author only shows one thing: if we add a VAE between the encoder and decoder of VGGT, we want the VAE to have a spherical distribution to minimize error. But "so what" is not discussed.

Questions for authors

Please clarify the research problem and goals with complete experiments and resubmit to another venue. I have no reason to change my rating to 3 or 4 based on any rebuttal, rather than rewriting the paper. If my concerns are resolved, the paper will be largely rewritten, and this should not be a case under ICLR policy.

Reviewer_e8toOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript argues that ViT features after LayerNorm roughly lie on a hypersphere, while the Euclidean latent space of a Gaussian VAE doesn't match that and causes posterior collapse. So it proposes S²VAE, which uses a product of multiple low dimensional Power Spherical distributions as the bottleneck, and it backs this up with better compression and reconstruction quality.

Critical strengths
  1. The product of spheres does solve the problem that earlier spherical VAEs only worked up to ≤64 dimensions, and the single sphere ablation (AbsRel 1.21, total failure) strongly supports this design.
  2. It covers a variety of modalities (geometry, semantics, text, pixels), and it reports latency and memory overhead (almost no extra cost).
Critical weaknesses
  1. The biggest problem with the manuscript is that it misses a lot of related work, and I think some of these omissions seriously hurt its novelty. Specifically, "Increasing Expressivity of a Hyperspherical VAE" (NeurIPS 2019) points out that the traditional Hyperspherical VAE lacks expressive power in high dimensions, and it proposes extending the spherical latent with a Product Space, validated on image datasets. This is very close to your Section 5.2. The only difference is the choice of distribution: you use a Product of Power Spherical distributions, while the 2019 work uses the vMF family. So the core design of splitting one high dimensional spherical latent into several low dimensional spheres to improve expressivity and trainability is not new. Also, "VFMF: World Modeling by Forecasting Vision Foundation Model Features" (ICML 2026) uses a VAE to compress pretrained DINO features, then generates the latents with Flow Matching, and the decoded latents support depth, surface normals, segmentation, RGB and other tasks. That is highly similar to your pipeline of feature compression, generation and multi task decoding. I think what you really need to show is whether the spherical design of S²VAE brings an extra, quantifiable advantage in VFMF style settings. You can't just show generation results and attribute them to spherical geometry.

  2. You cite "Laminating Representation Autoencoders for Efficient Diffusion", but FlatDINO itself is a method that uses a continuous VAE to compress DINOv2 patch features, squeezing 256 patch tokens into 32 continuous tokens and then using them for DiT generation. I feel your Section 2.1 doesn't cover the existing work on ViT feature encoding completely enough. You should discuss FlatDINO explicitly in the main text and explain how S²VAE differs from it.

  3. Based on the first two weaknesses, I actually suggest that the independent novelty of S²VAE is more accurately located in the application of Power Spherical, the design for compressing ViT features, and the validation on geometric tasks, rather than in the Product of Spheres itself. You should add a Product of vMF distributions baseline, keeping the same spherical grouping, network architecture and training budget. That way you can isolate the actual contribution of Power Spherical over the earlier product spherical VAEs.

  4. I also have some suggestions about details. For example, you wrote these sentences: "S²VAE consistently outperforms Gaussian baselines under aggressive compression (up to 512×), with the gap widening as the bottleneck tightens." "with the performance gap widening as compression increases, reaching up to 512×." "with the advantage growing as compression tightens" (Contribution 3). But the support in the main text and appendix is weak. The main experiments are basically fixed at D=128. In Appendix K, the only place with a sweep (D=8 to 128), the AbsRel gap is roughly flat across dimensions and does not clearly grow as compression gets stronger. Also, I could not find how the "512×" is computed, and there is no corresponding experiment.

  5. You also say "Across five transformer backbones ... S²VAE consistently outperforms Gaussian baselines" and "Across all models, the spherical bottleneck consistently outperforms a matched Gaussian baseline". From what I read, quantitative comparisons with the Gaussian baseline only appear on VGGT (Table 2 etc.) and DUSt3R (Table 3). DINOv2 (Table 1 has no baseline), CLIP, PAGE-4D and RAE only have S²VAE's own results or qualitative figures. "Across all models" and "consistently" have no evidence behind them.

  6. The premise that "ViT features lie on a hypersphere" is poorly supported by evidence and doesn't match the features actually used. Take these statements: "The intermediate hidden states at layer ℓ form a tokenized feature map x^(ℓ) ∈ R^{T×D}", "Across all three backbones and all layers the CV stays below 0.15", "The shells also sit at radii that depend strongly on depth (VGGT norms grow from ∼6 at shallow layers to ∼100 at the deepest layers)", and "indicating that the spherical concentration is induced by Layer Normalization itself rather than by training". What gets encoded is the hidden state of the Transformer (the residual stream), while the LayerNorm output is only the input to each sublayer. The norm of the residual stream accumulates with depth (the manuscript itself measured 6→100), so it is not a "spherical shell" produced directly by LayerNorm. Also, the derivation in Appendix B.2 only considers ε and ignores the learnable affine parameters γ and β of LayerNorm. Together with the mean subtraction, the actual shape should be an ellipsoid rather than a sphere. A CV < 0.15 only shows that norms are fairly concentrated across tokens. It doesn't mean a "thin shell", and it can't show that semantic information is carried by direction. A randomly initialized network behaving the same way actually suggests this is a structural byproduct, not the semantic structure of the data. This check was only done on VGGT and DINOv2, yet it is extended to the CLIP text encoder, CroCo, PAGE-4D and so on.

Questions for authors

Could the authors better distinguish the novelty of S²VAE from prior work, particularly product of spheres VAEs, VFMF, and FlatDINO, and provide stronger experimental evidence for the claimed compression advantages, consistent improvements across backbones, and hyperspherical structure of ViT features?

Reviewer_tenCOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the compression of pretrained Transformer features while preserving their downstream outputs. S²VAE replaces a Gaussian bottleneck with a product of low-dimensional Power Spherical distributions. Training combines feature reconstruction, auxiliary regularization, and task-specific losses where applicable. The paper reports reconstruction results across several pretrained models, matched Gaussian comparisons on VGGT and DUSt3R, and applications to lightweight depth prediction and latent-space generation. Most reconstruction evaluations measure fidelity to frozen model outputs rather than sensor-measured ground truth.

Critical strengths

Table 2 provides a useful matched comparison across three training seeds, with the same architecture and training objective. S²VAE reduces VGGT depth AbsRel from 0.040 to 0.027 and point-cloud CD from 0.029 to 0.014. This supports the bottleneck choice without attributing the difference to auxiliary losses that both models share.

The small depth probes demonstrate that task-relevant information remains directly accessible in the compressed representation, without reconstructing the original features or using the full VGGT decoder. Appendix N also includes normalized-Gaussian and FSQ comparisons, extending the evaluation beyond a standard Gaussian baseline.

Critical weaknesses

The posterior-collapse claim is not established by the reported activity measure. Section 7 does not specify whether variance is computed from sampled latents, predicted directions, or posterior expectations. If samples were used, a posterior collapsed to the uniform prior would still have prior-level variance and pass the activity threshold. The reported 128/128 active coordinates therefore do not, as described, demonstrate that the spherical bottleneck avoids collapse. This leaves the paper’s central explanation of the performance gains unresolved.

The complete loss weights, including the KL settings for the main comparisons, are not clearly specified. This matters because the proposed explanation depends on KL pressure. There is also an unexplained difference between the full-model CD of 0.014 in Table 2 and 0.14 in Table 5, although both are described as using unfiltered point sets. Different configurations may explain this, but they need to be identified before the auxiliary-loss ablation can be related to the main result.

The claims of consistent cross-backbone superiority and a widening advantage under stronger compression are broader than the evidence. Quantitative Gaussian comparisons are concentrated on VGGT and DUSt3R; the other backbone experiments mainly demonstrate reconstruction feasibility. In Table 7, the AbsRel advantage stays around 0.005–0.007 across dimensions, without a clear widening trend. These fixed-budget results support an empirical advantage, but not the stronger conclusion that geometry rather than capacity becomes the limiting factor.

Questions for authors

Is latent activity measured from samples, predicted directions, or posterior expectations? What concentration and KL statistics support the interpretation that it reflects retained information rather than sampling variance?

Please specify the full loss weights and configuration differences between Tables 2 and 5. Does the advantage over Gaussian and normalized-Gaussian bottlenecks persist across a small KL-weight sweep?

Is there a matched Gaussian comparison on a non-geometry backbone, and which results establish a widening advantage under stronger compression? Otherwise, these claims should be narrowed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.