- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper aims to make pretrained DINO features more suitable for image reconstruction and generation. It replaces the original DINO patch embedding with a hierarchical convolutional patch embedding (HCPE) to better preserve local details, while using cosine-based teacher alignment to retain the semantic representation of the frozen DINO encoder. The resulting latent features are then normalized and modeled with Riemannian Flow Matching for image generation. According to the experiment, HAE achieves strong reconstruction results. However, the experimental design still need more declaration, and the method requires clearer positioning and discussion with respect to related work.
Critical strengths
The paper presents a clear and well-motivated pipeline for adapting a frozen vision foundation model to image reconstruction and latent-space generation.
Critical weaknesses
Here are all my questions, the question 1-2 are my main concerns:
-
the Riemannian Flow Matching component is closely related to recent geometry-aware latent flow-matching approaches that project latent tokenas onto a hypersphere and replace Euclidean interpolation with spherical geodesics or SLERP. In particular, Aligning Latent Geometry for Spherical Flow Matching in Image Generation appears highly relevant but are not discussed. Please cite these works and clarify the technical distinction from the proposed formulation. It also would be helpful to explain whether the main novelty lies in the hyperspherical flow formulation itself, or in constructing a reconstruction-capable DINO latent via HCPE and directional alignment before applying this formulation.
-
The current experiments do not disentangle the contributions of HCPE, the decoder architecture, and the training recipe. The system-level RAE-to-HAE gap is large, whereas the only explicitly parameter-matched stem-only control is short-horizon and much smaller.
-
The direction-dominant latent geometry is motivated mainly by qualitative perturbation examples. A quantitative measurement of decoder sensitivity to radial versus tangential perturbations, ideally along the generation trajectory, would make the geometric motivation more convincing.
-
More details of the linear-probing protocol are recommended to provide, including which tokens or features are used, the probe architecture and training schedule, and whether the reported results use the same data preprocessing as the frozen DINO teacher.
Questions for authors
Please refer to the Critical Weaknesses
5 comments
Response to Reviewer Qrf6 (2/3)
[W2] Separating HCPE from the decoder architecture and training recipe (continued).
With the decoder architecture and training recipe fixed, HCPE improves all four reconstruction metrics, raising PSNR from 18.81 to 21.10 dB (+2.29 dB) and lowering rFID from 0.547 to 0.472. The baseline trained in our setup closely matches the released RAE decoder (18.81 vs. 18.86 dB), indicating that the gain does not come from retraining the decoder.
This gain is obtained while largely preserving the teacher's semantics. The HCPE encoder retains a cosine similarity of 0.961 to the frozen DINOv2-B features, and its linear-probe Top-1 accuracy stays within 0.5 points of the frozen encoder for both patch-token and CLS features (Table R7).
Table R7. ImageNet linear probing of the encoders in Table R6 (Top-1 / Top-5, %).
| Encoder | Mean of patch tokens | CLS token |
|---|---|---|
| Frozen DINOv2-B (original RAE encoder) | 82.33 / 96.48 | 83.88 / 96.67 |
| DINOv2-B with HCPE | 81.91 / 96.25 | 83.47 / 96.47 |
| Difference | −0.42 / −0.23 | −0.41 / −0.20 |
Table 10 complements this result by isolating the stem design itself, varying only the stem while keeping the decoder architecture, training recipe, random seed, data order, initialization, training objectives, and stem capacity fixed. Under these matched conditions, HCPE consistently outperforms both the capacity-matched channel-mixing baseline and the early-collapse baseline across all four metrics (+0.26 dB and +0.62 dB PSNR, respectively). Together with Table R6, these results demonstrate that HCPE provides a measurable improvement independent of changes to the decoder or training recipe.
We acknowledge that the 2.29 dB gain in the RAE pipeline does not account for the full system-level gap in Table 1, and that Table 10 is conducted under a shorter training horizon. Our claim is that HCPE makes an independently verified contribution to HAE's performance, not that it accounts for the entire RAE-to-HAE gap. This distinction is also stated in Appendix A.1.
[W3] Quantitative evidence for direction-dominant latent geometry.
Following the reviewer's suggestion, we conducted a controlled quantitative experiment to compare the decoder's sensitivity to radial and angular perturbations. The results in Table R8 show that angular perturbations consistently lead to worse reconstruction quality than radial perturbations across all four metrics, even when the perturbation magnitudes are exactly matched. This provides direct quantitative support for our geometric motivation: the HAE decoder is substantially more sensitive to changes in latent direction than to changes in magnitude.
To ensure a fair comparison, we apply radial and angular perturbations with exactly the same relative displacement d = ‖z′ − z‖₂ / ‖z‖₂ for each encoded latent token z. Radial perturbations scale the token by 1 − d or 1 + d, preserving its direction, while angular perturbations preserve its norm and rotate it toward a random orthogonal direction by θ = 2 arcsin(d/2). All reconstructions are evaluated using the same HAE-R decoder on the 50,000 ImageNet validation images.
Table R8. Reconstruction under matched radial and angular perturbations.
| Relative displacement d | Perturbation | rFID ↓ | PSNR (dB) ↑ | SSIM ↑ | LPIPS-VGG ↓ |
|---|---|---|---|---|---|
| 0 | None (baseline) | 0.37 | 26.20 | 0.7539 | 0.1353 |
| 0.10 | Radial: 0.9× or 1.1× | 0.400 | 25.607 | 0.7747 | 0.1359 |
| 0.10 | Angular: 5.73° | 0.622 | 25.091 | 0.7592 | 0.1496 |
| 0.25 | Radial: 0.75× or 1.25× | 1.521 | 22.785 | 0.7000 | 0.1962 |
| 0.25 | Angular: 14.36° | 9.308 | 21.019 | 0.6137 | 0.2918 |
| 0.50 | Radial: 0.5× or 1.5× | 9.790 | 18.975 | 0.5820 | 0.3296 |
| 0.50 | Angular: 28.96° | 102.324 | 15.957 | 0.3784 | 0.5371 |
Bold indicates the better result within each matched-displacement pair.
The performance gap becomes particularly pronounced at larger perturbations. For example, at d = 0.25, angular perturbations yield an rFID of 9.308 compared to 1.521 for radial perturbations, along with substantially worse LPIPS (0.2918 vs. 0.1962). Similar trends hold at all tested displacement levels, confirming that latent direction has a greater impact on reconstruction quality than latent magnitude under matched perturbations.
While this experiment measures sensitivity at encoded latents rather than along the full generation trajectory, these latents represent the target distribution that the generator aims to recover at the end of sampling. Thus, the results directly support the direction-dominant geometry underlying HAE, while a trajectory-level analysis remains an interesting direction for further investigation.
Response to Reviewer Qrf6 (3/3)
[W4] Clarification of the linear-probing protocol.
We thank the reviewer for pointing out the need for additional details. We clarify that HAE and the frozen DINOv3 teacher are evaluated under exactly the same linear-probing protocol, namely the official DINOv3 linear evaluation (Siméoni et al., 2025) without modification, including feature extraction, classifier architecture, training schedule, and input preprocessing. The complete protocol is given below and will be added to the appendix of the revised manuscript to ensure reproducibility.
Specifically, the evaluation uses:
- Features: The output of the last Transformer block after the final LayerNorm. Following DINOv3, the class token is concatenated with the average-pooled patch tokens.
- Classifier: A single linear layer trained on these features, with both feature extractors frozen.
- Training: SGD with momentum 0.9 and no weight decay, using a cosine learning-rate schedule annealed to zero, for 12,500 iterations (10 epochs of 1,250 iterations) with a batch size of 128 per GPU. The learning rate is selected by a grid search over 13 values from 10⁻⁵ to 10⁻¹, each scaled by (total batch size)/256; all classifiers are trained in parallel on the same features, and the best one is reported. For HAE, the best grid value was 5×10⁻².
- Preprocessing: RandomResizedCrop with bicubic interpolation and horizontal flip for training, and resize followed by center crop for evaluation, applied identically to HAE and the frozen DINOv3 teacher.
Under this identical evaluation protocol, the fully trained HAE achieves 87% ImageNet Top-1 accuracy (97% Top-5), compared to 89% Top-1 (98% Top-5) for the frozen DINOv3 teacher. This shows that HAE preserves most of the teacher's discriminative capability while learning its generative latent representation.
We also clarify that the 84.0% and 6.6% results in Table 6 correspond to a separate, controlled comparison of cosine and raw-MSE alignment after only 5,000 training updates, rather than the fully trained HAE model. Both alignment variants are evaluated using the same linear-probing protocol described above. Thus, Table 6 isolates the effect of the alignment objective, whereas the 87% result reflects the representation quality of the fully trained HAE.
Response to Reviewer Qrf6 (1/3)
[W1] Relation to spherical flow matching and the source of novelty.
We thank the reviewer for pointing out this closely related work. We clarify that the primary novelty of HAE lies not in spherical flow matching itself, but in learning reconstruction-capable, semantically aligned VFM latents that can be effectively used for generation. We will add Meral et al. to Section 2 and clarify this distinction in the revised manuscript.
Specifically, Meral et al. focus on adapting the geometry of existing pretrained VAE latents for spherical flow matching, projecting latent tokens onto a fixed-radius sphere and fine-tuning the decoder while keeping the encoder frozen. In contrast, HAE focuses on constructing the latent representation itself: HCPE adapts the input embedding before the frozen VFM Transformer, while directional alignment preserves the teacher's semantic information. The resulting representation supports high-fidelity reconstruction while providing a direction-sensitive latent space suitable for spherical generation.
Our experimental results further support the effectiveness of this representation-learning approach. RJF (Kumar & Patel, 2026) applies geometry-aware flow matching to RAE latents and achieves 3.62 unguided gFID, compared to 2.65 for HAE, using the same LightningDiT-XL/1 generator architecture and an 80-epoch training budget (Table 3). Although the underlying representations differ (DINOv2-B vs. DINOv3-L), this comparison suggests that the quality of the learned latent representation is an important factor beyond the use of spherical flow matching.
We also compare HAE with the systems reported by Meral et al. (Table R5):
Table R5. Comparison with reported spherical flow-matching systems (ImageNet 256×256, unguided).
HAE achieves the lowest reported unguided gFID among these systems while using only 50 sampling steps, compared to 250 for Meral et al. We acknowledge that these are not strictly controlled comparisons, as the systems differ in their tokenizers, generator architectures, and training recipes. Nevertheless, the results provide encouraging evidence of HAE's generation quality and sampling efficiency.
Taken together, these results highlight HAE's distinct contribution: rather than applying spherical flow matching to existing pretrained latents, HAE learns a reconstruction-capable VFM representation that preserves semantic information and can be effectively exploited for generation.
[W2] Separating HCPE from the decoder architecture and training recipe.
We clarify that the full performance gap between RAE and HAE in Table 1 should not be attributed to HCPE alone, as the two systems also differ in their decoder architectures and training recipes. To separate HCPE from these two factors, we conducted an additional experiment that inserts HCPE into the original RAE pipeline while keeping the RAE decoder architecture and training recipe.
Specifically, we replace the patch-embedding stem of RAE's DINOv2-B encoder with HCPE and align its output with the frozen DINOv2-B teacher using the same directional alignment as in HAE. DINOv2-B is sensitive to changes in its patch-embedding stem: when a randomly initialized HCPE is trained directly with the decoder for reconstruction, its alignment with the teacher easily breaks down. We therefore first warm up HCPE alone by matching its output to that of the original DINOv2-B patch embedding for 10,000 steps with a batch size of 256 (approximately two ImageNet epochs), and then train it jointly with the decoder. The RAE baseline keeps the original frozen DINOv2-B patch embedding and follows the same training recipe; the two runs differ only in HCPE, its warm-up, and the directional alignment loss. As a reference, the publicly released RAE decoder reproduces its reported rFID under our evaluation (0.569 vs. 0.57).
Table R6. Inserting HCPE into the RAE pipeline (DINOv2-B, ImageNet 256×256). Cosine similarity is computed between each encoder's output and the frozen DINOv2-B features.