acceptodds
Under review as a conference paper at ICLR 2027

Hyperspherical Autoencoder for High-Fidelity Image Reconstruction and Generation

Abstract

Recent studies have explored using pretrained Vision Foundation Models (VFMs) such as DINO for generative autoencoders, showing strong generative performance. Unfortunately, existing approaches often suffer from limited reconstruction fidelity due to the loss of high-frequency details. In this work, we present the \em Hyperspherical Autoencoder (HAE), a framework that bridges semantic representation and pixel-level reconstruction. Our key insight is that while semantic information in contrastive representations is primarily directional, enforcing strict magnitude matching hinders the preservation of fine-grained details. To address this, we introduce a \em Directional Feature Alignment objective that enforces semantic consistency while allowing flexible feature magnitudes for detail retention, alongside a \em Hierarchical Convolutional Patch Embedding module to enhance local structure preservation. Furthermore, observing that SSL-based representations intrinsically lie on a hypersphere, we employ \em Riemannian Flow Matching to train a Diffusion Transformer (DiT) directly on this spherical latent manifold. Notably, our manifold-aware DiT exhibits highly efficient convergence, achieving an exceptional gFID of 1.96 alongside a reconstruction rFID of 0.78 and a PSNR of 25.2 dB, validating the advantages of our manifold-aware approach.

Primary area:
generative models
Keywords:
Latent Diffusion model, Autoencoder, Visual Foundation Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Qrf6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to make pretrained DINO features more suitable for image reconstruction and generation. It replaces the original DINO patch embedding with a hierarchical convolutional patch embedding (HCPE) to better preserve local details, while using cosine-based teacher alignment to retain the semantic representation of the frozen DINO encoder. The resulting latent features are then normalized and modeled with Riemannian Flow Matching for image generation. According to the experiment, HAE achieves strong reconstruction results. However, the experimental design still need more declaration, and the method requires clearer positioning and discussion with respect to related work.

Critical strengths

The paper presents a clear and well-motivated pipeline for adapting a frozen vision foundation model to image reconstruction and latent-space generation.

Critical weaknesses

Here are all my questions, the question 1-2 are my main concerns:

  1. the Riemannian Flow Matching component is closely related to recent geometry-aware latent flow-matching approaches that project latent tokenas onto a hypersphere and replace Euclidean interpolation with spherical geodesics or SLERP. In particular, Aligning Latent Geometry for Spherical Flow Matching in Image Generation appears highly relevant but are not discussed. Please cite these works and clarify the technical distinction from the proposed formulation. It also would be helpful to explain whether the main novelty lies in the hyperspherical flow formulation itself, or in constructing a reconstruction-capable DINO latent via HCPE and directional alignment before applying this formulation.

  2. The current experiments do not disentangle the contributions of HCPE, the decoder architecture, and the training recipe. The system-level RAE-to-HAE gap is large, whereas the only explicitly parameter-matched stem-only control is short-horizon and much smaller.

  3. The direction-dominant latent geometry is motivated mainly by qualitative perturbation examples. A quantitative measurement of decoder sensitivity to radial versus tangential perturbations, ideally along the generation trajectory, would make the geometric motivation more convincing.

  4. More details of the linear-probing protocol are recommended to provide, including which tokens or features are used, the probe architecture and training schedule, and whether the reported results use the same data preprocessing as the frozen DINO teacher.

Questions for authors

Please refer to the Critical Weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer Qrf6 (1/3)

[W1] Relation to spherical flow matching and the source of novelty.

We thank the reviewer for pointing out this closely related work. We clarify that the primary novelty of HAE lies not in spherical flow matching itself, but in learning reconstruction-capable, semantically aligned VFM latents that can be effectively used for generation. We will add Meral et al. to Section 2 and clarify this distinction in the revised manuscript.

Specifically, Meral et al. focus on adapting the geometry of existing pretrained VAE latents for spherical flow matching, projecting latent tokens onto a fixed-radius sphere and fine-tuning the decoder while keeping the encoder frozen. In contrast, HAE focuses on constructing the latent representation itself: HCPE adapts the input embedding before the frozen VFM Transformer, while directional alignment preserves the teacher's semantic information. The resulting representation supports high-fidelity reconstruction while providing a direction-sensitive latent space suitable for spherical generation.

Our experimental results further support the effectiveness of this representation-learning approach. RJF (Kumar & Patel, 2026) applies geometry-aware flow matching to RAE latents and achieves 3.62 unguided gFID, compared to 2.65 for HAE, using the same LightningDiT-XL/1 generator architecture and an 80-epoch training budget (Table 3). Although the underlying representations differ (DINOv2-B vs. DINOv3-L), this comparison suggests that the quality of the learned latent representation is an important factor beyond the use of spherical flow matching.

We also compare HAE with the systems reported by Meral et al. (Table R5):

Table R5. Comparison with reported spherical flow-matching systems (ImageNet 256×256, unguided).

SystemGeneratorEpochsSampling stepsUnguided gFID ↓
Meral et al. (FLUX.2)SiT-XL/28025010.71
Meral et al. (VA-VAE)SiT-XL/18025010.36
RJF (DINOv2-B)LightningDiT-XL/180503.62
HAE (DINOv3-L)LightningDiT-XL/180502.65

HAE achieves the lowest reported unguided gFID among these systems while using only 50 sampling steps, compared to 250 for Meral et al. We acknowledge that these are not strictly controlled comparisons, as the systems differ in their tokenizers, generator architectures, and training recipes. Nevertheless, the results provide encouraging evidence of HAE's generation quality and sampling efficiency.

Taken together, these results highlight HAE's distinct contribution: rather than applying spherical flow matching to existing pretrained latents, HAE learns a reconstruction-capable VFM representation that preserves semantic information and can be effectively exploited for generation.

[W2] Separating HCPE from the decoder architecture and training recipe.

We clarify that the full performance gap between RAE and HAE in Table 1 should not be attributed to HCPE alone, as the two systems also differ in their decoder architectures and training recipes. To separate HCPE from these two factors, we conducted an additional experiment that inserts HCPE into the original RAE pipeline while keeping the RAE decoder architecture and training recipe.

Specifically, we replace the patch-embedding stem of RAE's DINOv2-B encoder with HCPE and align its output with the frozen DINOv2-B teacher using the same directional alignment as in HAE. DINOv2-B is sensitive to changes in its patch-embedding stem: when a randomly initialized HCPE is trained directly with the decoder for reconstruction, its alignment with the teacher easily breaks down. We therefore first warm up HCPE alone by matching its output to that of the original DINOv2-B patch embedding for 10,000 steps with a batch size of 256 (approximately two ImageNet epochs), and then train it jointly with the decoder. The RAE baseline keeps the original frozen DINOv2-B patch embedding and follows the same training recipe; the two runs differ only in HCPE, its warm-up, and the directional alignment loss. As a reference, the publicly released RAE decoder reproduces its reported rFID under our evaluation (0.569 vs. 0.57).

Table R6. Inserting HCPE into the RAE pipeline (DINOv2-B, ImageNet 256×256). Cosine similarity is computed between each encoder's output and the frozen DINOv2-B features.

ModelrFID ↓PSNR (dB) ↑SSIM ↑LPIPS ↓Cosine similarity
RAE, released decoder0.56918.860.48360.25561.000
RAE, baseline trained in our setup0.54718.810.47450.25941.000
RAE + HCPE0.47221.100.56370.21760.961
AuthorsOpenReview ↗

Response to Reviewer Qrf6 (2/3)

[W2] Separating HCPE from the decoder architecture and training recipe (continued).

With the decoder architecture and training recipe fixed, HCPE improves all four reconstruction metrics, raising PSNR from 18.81 to 21.10 dB (+2.29 dB) and lowering rFID from 0.547 to 0.472. The baseline trained in our setup closely matches the released RAE decoder (18.81 vs. 18.86 dB), indicating that the gain does not come from retraining the decoder.

This gain is obtained while largely preserving the teacher's semantics. The HCPE encoder retains a cosine similarity of 0.961 to the frozen DINOv2-B features, and its linear-probe Top-1 accuracy stays within 0.5 points of the frozen encoder for both patch-token and CLS features (Table R7).

Table R7. ImageNet linear probing of the encoders in Table R6 (Top-1 / Top-5, %).

EncoderMean of patch tokensCLS token
Frozen DINOv2-B (original RAE encoder)82.33 / 96.4883.88 / 96.67
DINOv2-B with HCPE81.91 / 96.2583.47 / 96.47
Difference−0.42 / −0.23−0.41 / −0.20

Table 10 complements this result by isolating the stem design itself, varying only the stem while keeping the decoder architecture, training recipe, random seed, data order, initialization, training objectives, and stem capacity fixed. Under these matched conditions, HCPE consistently outperforms both the capacity-matched channel-mixing baseline and the early-collapse baseline across all four metrics (+0.26 dB and +0.62 dB PSNR, respectively). Together with Table R6, these results demonstrate that HCPE provides a measurable improvement independent of changes to the decoder or training recipe.

We acknowledge that the 2.29 dB gain in the RAE pipeline does not account for the full system-level gap in Table 1, and that Table 10 is conducted under a shorter training horizon. Our claim is that HCPE makes an independently verified contribution to HAE's performance, not that it accounts for the entire RAE-to-HAE gap. This distinction is also stated in Appendix A.1.

[W3] Quantitative evidence for direction-dominant latent geometry.

Following the reviewer's suggestion, we conducted a controlled quantitative experiment to compare the decoder's sensitivity to radial and angular perturbations. The results in Table R8 show that angular perturbations consistently lead to worse reconstruction quality than radial perturbations across all four metrics, even when the perturbation magnitudes are exactly matched. This provides direct quantitative support for our geometric motivation: the HAE decoder is substantially more sensitive to changes in latent direction than to changes in magnitude.

To ensure a fair comparison, we apply radial and angular perturbations with exactly the same relative displacement d = ‖z′ − z‖₂ / ‖z‖₂ for each encoded latent token z. Radial perturbations scale the token by 1 − d or 1 + d, preserving its direction, while angular perturbations preserve its norm and rotate it toward a random orthogonal direction by θ = 2 arcsin(d/2). All reconstructions are evaluated using the same HAE-R decoder on the 50,000 ImageNet validation images.

Table R8. Reconstruction under matched radial and angular perturbations.

Relative displacement dPerturbationrFID ↓PSNR (dB) ↑SSIM ↑LPIPS-VGG ↓
0None (baseline)0.3726.200.75390.1353
0.10Radial: 0.9× or 1.1×0.40025.6070.77470.1359
0.10Angular: 5.73°0.62225.0910.75920.1496
0.25Radial: 0.75× or 1.25×1.52122.7850.70000.1962
0.25Angular: 14.36°9.30821.0190.61370.2918
0.50Radial: 0.5× or 1.5×9.79018.9750.58200.3296
0.50Angular: 28.96°102.32415.9570.37840.5371

Bold indicates the better result within each matched-displacement pair.

The performance gap becomes particularly pronounced at larger perturbations. For example, at d = 0.25, angular perturbations yield an rFID of 9.308 compared to 1.521 for radial perturbations, along with substantially worse LPIPS (0.2918 vs. 0.1962). Similar trends hold at all tested displacement levels, confirming that latent direction has a greater impact on reconstruction quality than latent magnitude under matched perturbations.

While this experiment measures sensitivity at encoded latents rather than along the full generation trajectory, these latents represent the target distribution that the generator aims to recover at the end of sampling. Thus, the results directly support the direction-dominant geometry underlying HAE, while a trajectory-level analysis remains an interesting direction for further investigation.

AuthorsOpenReview ↗

Response to Reviewer Qrf6 (3/3)

[W4] Clarification of the linear-probing protocol.

We thank the reviewer for pointing out the need for additional details. We clarify that HAE and the frozen DINOv3 teacher are evaluated under exactly the same linear-probing protocol, namely the official DINOv3 linear evaluation (Siméoni et al., 2025) without modification, including feature extraction, classifier architecture, training schedule, and input preprocessing. The complete protocol is given below and will be added to the appendix of the revised manuscript to ensure reproducibility.

Specifically, the evaluation uses:

  • Features: The output of the last Transformer block after the final LayerNorm. Following DINOv3, the class token is concatenated with the average-pooled patch tokens.
  • Classifier: A single linear layer trained on these features, with both feature extractors frozen.
  • Training: SGD with momentum 0.9 and no weight decay, using a cosine learning-rate schedule annealed to zero, for 12,500 iterations (10 epochs of 1,250 iterations) with a batch size of 128 per GPU. The learning rate is selected by a grid search over 13 values from 10⁻⁵ to 10⁻¹, each scaled by (total batch size)/256; all classifiers are trained in parallel on the same features, and the best one is reported. For HAE, the best grid value was 5×10⁻².
  • Preprocessing: RandomResizedCrop with bicubic interpolation and horizontal flip for training, and resize followed by center crop for evaluation, applied identically to HAE and the frozen DINOv3 teacher.

Under this identical evaluation protocol, the fully trained HAE achieves 87% ImageNet Top-1 accuracy (97% Top-5), compared to 89% Top-1 (98% Top-5) for the frozen DINOv3 teacher. This shows that HAE preserves most of the teacher's discriminative capability while learning its generative latent representation.

We also clarify that the 84.0% and 6.6% results in Table 6 correspond to a separate, controlled comparison of cosine and raw-MSE alignment after only 5,000 training updates, rather than the fully trained HAE model. Both alignment variants are evaluated using the same linear-probing protocol described above. Thus, Table 6 isolates the effect of the alignment objective, whereas the 87% result reflects the representation quality of the fully trained HAE.

Reviewer_N1KVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper turns a frozen DINOv3-L into an autoencoder usable for both reconstruction and latent diffusion, combining a trainable convolutional patch stem, a cosine feature-alignment loss, and Riemannian flow matching on per-patch hyperspheres.

Critical strengths

The generation results are impressive. A plain, unmodified DiT reaches a gFID 1.79 unguided, and at 80 epochs it clearly beats VA-VAE and RAE (2.65 vs roughly 4.3, Tab. 3). That is a practical result others can build on.

I also like that reconstruction improves a lot over RAE while the semantic encoder stays frozen. That was a real weakness of RAE-style latents, and the paper addresses it directly.

Critical weaknesses

My main worry is that the paper never tells us which ingredient is doing the work. Tab. 2 shows gFID 32.64 without the RAE-style latent smoothing and 2.65 with it, so a large share of the generation quality may come from borrowed smoothing rather than the new stem or alignment loss, and there is no standard stem plus smoothing versus HCPE plus smoothing comparison. The only evidence for the spherical flow is Tab. 4, where the Euclidean baseline (gFID 15.95 vs 2.65) is barely described, so it may simply be under-tuned. The reconstruction story has the same issue: against a parameter-matched control, HCPE adds only +0.26 dB (Tab. 10). Everything is single seed.

Questions for authors

What exactly is the Euclidean flow-matching baseline in Tab. 4 (normalization, time shift, tuning)?

What is the gFID of a standard stem with the same latent smoothing, and with a properly tuned MSE alignment?

2 comments
AuthorsOpenReview ↗

Response to Reviewer N1KV (1/2)

[W1] Disentangling the contributions of HAE's components.

We agree that latent smoothing plays a critical role in HAE's generation performance, as demonstrated by the substantial gFID improvement from 32.64 to 2.65 in Table 2. We explicitly acknowledge that smoothing is adopted from RAE (Section 3.3) and do not claim it as our contribution. However, since RAE also employs latent smoothing, its use alone does not explain the performance differences between HAE and RAE.

More importantly, our controlled experiments provide separate evidence for the effectiveness of HAE's proposed components:

  • HCPE (Tables 10 and R3): With the decoder, training recipe, and stem capacity fixed, HCPE outperforms the alternative stems across all four reconstruction metrics, demonstrating its independent contribution to reconstruction quality. In an additional experiment, inserting HCPE into the original RAE pipeline, with the RAE decoder architecture and training recipe kept, improves PSNR from 18.81 to 21.10 dB and rFID from 0.547 to 0.472 (Table R3).
  • Directional alignment (Table 6): Under otherwise identical conditions, cosine alignment preserves substantially more semantic information than raw-MSE alignment (84.0% vs. 6.6% linear-probe Top-1 accuracy after 5,000 updates).
  • RFM (Table 4): Under matched optimization and sampling settings, RFM substantially outperforms EFM (2.65 vs. 15.95 gFID), demonstrating the importance of the flow formulation for generation quality.

Table R3. Inserting HCPE into the original RAE pipeline (DINOv2-B, ImageNet 256×256).

ModelrFID ↓PSNR (dB) ↑SSIM ↑LPIPS ↓
RAE baseline0.54718.810.47450.2594
RAE + HCPE0.47221.100.56370.2176

We acknowledge that these ablations do not directly isolate the contribution of HCPE to gFID under latent smoothing. Rather, they establish the distinct benefits of the proposed components for reconstruction fidelity, semantic preservation, and generation dynamics, respectively.

At the system level, HAE improves reconstruction PSNR from 18.9 dB (RAE) to 26.2 dB using the same DINOv3-L backbone (Table 1), while at 80 epochs achieving lower unguided gFID than the reported RAE and VA-VAE baselines that use similarly sized standard generators (2.65 vs. 4.28 and 4.29; Table 3). Our claim is therefore not that any single component explains the overall performance gain, but that HAE combines independently validated design improvements into an effective reconstruction and generation framework.

[W2] Contribution of HCPE to reconstruction quality.

We acknowledge that HCPE's PSNR improvement over the parameter-matched standard-patch control is modest (+0.26 dB). However, its benefit is not limited to PSNR: HCPE consistently achieves the best performance across all four reconstruction metrics under controlled conditions (Table 10).

All three stems have approximately 2.74M parameters and share the same 10-epoch Stage-1 training protocol, random seed, data order, decoder initialization, and training objectives. Thus, the observed differences can be attributed to the stem design rather than differences in model capacity or training configuration.

Table 10 (from the paper). Parameter-matched stem controls after 10 epochs of Stage-1 training.

StemrFID ↓PSNR ↑SSIM ↑LPIPS ↓
Early-collapse control6.048121.10530.54090.3022
Standard-patch capacity control3.470121.46190.56490.2545
HCPE3.205621.72590.58400.2509

Importantly, the comparison with the early-collapse control further highlights the benefit of progressive spatial processing. Despite having matched convolutional depth and parameter capacity, HCPE improves PSNR by 0.62 dB and reduces rFID from 6.05 to 3.21. Together with the standard-patch control, these results suggest that HCPE's advantage arises from its hierarchical spatial processing rather than simply increased capacity or convolutional depth.

We also acknowledge that these results are based on a single seed and a relatively short training horizon, and therefore do not establish the statistical robustness or long-term magnitude of the improvement. Nevertheless, the consistent gains across reconstruction metrics and both controls provide controlled evidence for the effectiveness of HCPE's architectural design.

We will also cite Xiao et al. (2021) in Section 3.2 to acknowledge prior work on convolutional stems. Unlike their setting of training ViTs from scratch, HCPE is designed to adapt a frozen pretrained VFM for reconstruction, and our ablations provide evidence for the benefit of progressive spatial processing in this setting.

AuthorsOpenReview ↗

Response to Reviewer N1KV (2/2)

[Q1] Configuration and fairness of the EFM baseline.

We thank the reviewer for raising this concern. To clarify the comparison in Table 4, EFM and RFM are trained using the same HAE encoder and LightningDiT-XL/1 generator, with identical optimization settings, time sampling schedules, training budgets, and sampling budgets. The detailed configurations are listed in Table R4 and will be added to the appendix of the revised manuscript.

Table R4. Configurations of EFM and RFM for the comparison in Table 4.

SettingEFMRFM
Encoder / GeneratorHAE / LightningDiT-XL/1HAE / LightningDiT-XL/1
Learning rate2×10⁻⁴2×10⁻⁴
Time samplingLogit-normal, shift α = 8Logit-normal, shift α = 8
Training80 epochs80 epochs
EvaluationUnguided, 50 NFEUnguided, 50 NFE
Latent normalizationMean/std standardization using encoded ImageNet statisticsPer-patch normalization to radius R = 32
DecodingInverse standardizationRescaling from R = 32 to s = 14 (Appendix E)

The key differences lie in how the two methods represent and model the latent space. EFM operates on standardized Euclidean latents, whereas RFM normalizes each patch token onto a sphere and employs a spherical flow path. The latter also eliminates the need to estimate dataset-level normalization statistics.

Under the same training and sampling budgets, RFM achieves substantially better generation quality than EFM (2.65 vs. 15.95 gFID; Table 4). This result provides evidence that spherical latent modeling is more effective than the evaluated Euclidean formulation for HAE representations, rather than the improvement being attributable to differences in the generator, learning rate, or computational budget.

We acknowledge that using identical hyperparameters does not guarantee that EFM is optimally tuned. Nevertheless, the comparison establishes a substantial performance advantage for RFM under a shared training and evaluation protocol, supporting the effectiveness of our spherical formulation in the HAE setting.

[Q2] Standard-stem and tuned-MSE comparisons.

We thank the reviewer for suggesting these additional comparisons. The requested baselines correspond to two controlled modifications of HAE: (1) replacing HCPE with a standard patch-embedding stem while retaining the same latent smoothing, cosine alignment, and generator configuration; and (2) replacing cosine alignment with a properly tuned MSE-based objective while keeping the remaining components unchanged.

We evaluate both alternatives at the representation level, where these two design choices act, and train the generator only on the latent that meets HAE's two goals: high-fidelity reconstruction and preservation of the teacher's semantics.

  • Standard stem (Tables 10 and R3): The stem is designed and trained in Stage 1, whose objective is reconstruction fidelity together with teacher alignment; latent smoothing enters only in Stage 2. We therefore select the stem by reconstruction quality. With the decoder, training recipe, and stem capacity fixed, the parameter-matched standard patch-embedding stem yields lower reconstruction quality than HCPE across all four metrics (Table 10), and replacing the standard patch embedding with HCPE in the original RAE pipeline likewise improves all four metrics (Table R3). The standard stem is therefore not the design selected by our Stage-1 criterion. We do not rule out that a standard stem could attain a comparable, or even lower, gFID under the same latent smoothing. However, a stem with lower reconstruction fidelity does not meet the reconstruction objective that motivates HAE, so we do not train a generator on it, and we do not claim that HCPE by itself lowers gFID.
  • MSE alignment (Table 6): Under otherwise identical conditions, raw-MSE alignment retains only 6.6% linear-probe Top-1 accuracy, compared to 84.0% for cosine alignment. As discussed in Section 3.3, direct MSE penalizes differences in feature magnitude as well as direction. A latent that has lost most of the teacher's semantic structure removes the motivation for building the autoencoder on a pretrained VFM, so training a generator on such a latent falls outside the scope of this work.

While the paper does not report gFID for these two alternatives, our claims are made at the level where each component is validated: HCPE for reconstruction fidelity (Table 10) and directional alignment for semantic preservation (Table 6). The generation quality of HAE is then evaluated on the resulting latent, at the system level (Table 3) and through the controlled comparison of EFM and RFM (Table 4). We believe that these provide clear evidence of the representation-level benefits of HCPE and cosine alignment, particularly in reconstruction fidelity and semantic preservation, respectively.

Reviewer_jn4vOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes the Hyperspherical Autoencoder (HAE). It retains a frozen DINOv3-L Transformer, replaces the standard patch embedding with a four-stage hierarchical convolutional patch embedding (HCPE), and aligns student and teacher features by cosine direction. Each patch latent is then normalized to a fixed-radius hypersphere and used to train a DiT with Riemannian Flow Matching (RFM) along spherical geodesics.

Critical strengths
  1. Strong empirical gains with meaningful controlled experiments. Under the same frozen backbone, HCPE substantially improves reconstruction and is supported by parameter-matched stem controls under the same training setup. Directional alignment also clearly outperforms direct MSE alignment.
  2. A coherent system connecting VFM reconstruction adaptation and spherical generative geometry. With the same latents, model, and 80-epoch budget, RFM achieves gFID 2.65 compared with 15.95 for EFM.
Critical weaknesses
  1. The scope, related-work positioning, and attribution of the RFM–EFM gain are incomplete. Directional alignment, radial insensitivity, HCPE, and spherical RFM are all evaluated only with DINOv3-L.
  2. Prior works including DINO-Tok, SVG, One Layer Is Enough (FAE), and LV-RAE investigate closely relevant directions such as encoder adaptation, multi-scale feature modeling, and low-level correction for VFM-based Autoencoder. However, the distinct independent contribution of HAE is not clearly articulated.
Questions for authors

Could the authors provide experimental results of HAE on other VFM models to demonstrate its generalization and universal applicability?

Comments on the paper

4 comments
AuthorsOpenReview ↗

Response to Reviewer jn4v [2/2]

[Q] Generalizability to other VFMs.

We agree that evaluating HAE on other VFMs is important to establish its generalizability beyond DINOv3-L. To address this concern, we applied HAE to SigLIP2, replacing its patch-embedding stem with HCPE and using the frozen SigLIP2 encoder as the teacher for directional alignment, while keeping the remaining experimental settings unchanged. This experiment is designed to examine whether HAE's key design principles can be effectively transferred to a different pretrained VFM, rather than relying on properties specific to DINOv3-L.

As with DINOv2-B (see below), SigLIP2 is sensitive to changes in its patch-embedding stem: when a randomly initialized HCPE is trained directly with the decoder for reconstruction, its alignment with the teacher easily breaks down. We therefore first warm up HCPE alone by matching its output to that of the original patch embedding for 10,000 steps with a batch size of 256 (approximately two ImageNet epochs), and then train it jointly with the decoder. Training is still in progress: Table R1 compares the two models at the current checkpoint, after 5 epochs, and we will update it as soon as the remaining training is complete.

Table R1. SigLIP2 with and without HCPE at the current checkpoint (5 epochs, 6.4M images seen; training in progress), evaluated on the 50k ImageNet 256×256 validation images.

ModelrFID ↓PSNR (dB) ↑SSIM ↑LPIPS ↓
SigLIP2 (original patch embedding)5.1318.810.4980.307
SigLIP2 + HCPE4.7520.890.5440.284

At the same point in training, HCPE already improves all four reconstruction metrics on SigLIP2, raising PSNR from 18.81 to 20.89 dB (+2.08 dB) and lowering rFID from 5.13 to 4.75.

As an additional reference, HCPE and directional alignment also transfer to DINOv2-B. We inserted HCPE into the original RAE pipeline, keeping the RAE decoder architecture and training recipe and using the same HCPE warm-up. HCPE improves all four reconstruction metrics, while linear-probe Top-1 accuracy stays within 0.5 points of the frozen DINOv2-B encoder (Table R2).

Table R2. Inserting HCPE into the original RAE pipeline (DINOv2-B, ImageNet 256×256).

ModelrFID ↓PSNR (dB) ↑SSIM ↑LPIPS ↓Linear-probe Top-1 (CLS, %)
RAE baseline0.54718.810.47450.259483.88
RAE + HCPE0.47221.100.56370.217683.47

We acknowledge that these experiments do not evaluate generation, and that the SigLIP2 results are intermediate and report reconstruction metrics only. Nevertheless, HCPE improves reconstruction on both SigLIP2 and DINOv2-B, and the DINOv2-B encoder stays within 0.5 points of the teacher's linear-probe accuracy, providing evidence that HCPE and directional alignment are not specific to DINOv3-L.

AuthorsOpenReview ↗

Follow-up to Reviewer jn4v: Final SigLIP2 Results

Update on [W1] and [Q]: final SigLIP2 results.

We thank the reviewer again for suggesting an evaluation on other VFMs. In our initial response, we reported intermediate SigLIP2 results at 5 epochs and noted that we would update them once training was complete. Training is now complete for both variants, and Table R1 below replaces the intermediate table.

Both variants follow the same schedule: Stage 1 runs for 20 epochs, followed by adversarial training without latent smoothing (the reconstruction-oriented HAE-R setting), and for each we report the checkpoint with the lowest rFID (epoch 9 of this stage for both). The variant with the original patch embedding keeps the SigLIP2 encoder frozen and trains only the decoder. The HCPE warm-up and all other settings are as described in our initial response.

Table R1 (updated). HAE with a SigLIP2 encoder, with and without HCPE, evaluated on the 50k ImageNet 256×256 validation images (EMA weights).

ModelrFID ↓PSNR (dB) ↑SSIM ↑LPIPS ↓Linear-probe Top-1 (patch mean, %)
SigLIP2 (original patch embedding)0.59520.630.5670.21785.70
SigLIP2 + HCPE0.4823.900.6920.17585.66

HCPE improves all four reconstruction metrics on SigLIP2: PSNR increases by 3.27 dB (from 20.63 to 23.90 dB) and rFID decreases from 0.595 to 0.48. Linear-probe Top-1 accuracy, measured with the official DINOv3 protocol on the average-pooled patch tokens (the SigLIP2 vision tower has no class token), is 85.66% with HCPE, compared with 85.70% for the original SigLIP2 encoder, indicating that SigLIP2's semantics are preserved. Together, these results show that HCPE and directional alignment can be trained on a VFM outside the DINO family, improving its reconstruction while preserving its semantics, providing further evidence that HAE is not specific to DINOv3-L. We will include these results in the revised manuscript.

Thanks for the detailed rebuttal. It satisfactorily resolves my questions. I lean towards acceptance and retain my initial score.

We sincerely thank Reviewer jn4v for the positive feedback and for the constructive suggestions throughout the review process. We are glad that our response resolved your questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.