- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a tokenizer that aggregates features from all layers of DINOv3, achieving a better reconstruction and generation trade-off. Experimental results on class-conditional generation and text-to-image generation demonstrate the effectiveness of the proposed method.
Critical strengths
The proposed method is simple and easy to follow. Experimental results show the method achieves a good trade-off between reconstruction and generation.
Critical weaknesses
-
The method seems incremental. Previous methods such as RAEv2 and DRoRAE have demonstrated the effectiveness of feature aggregation, while the paper introduces a heuristic aggregation method with some manually specified hyperparameters, such as the number of groups, dropout probabilities, and groupwise norm caps.
-
Some experimental comparisons are not controlled, making it difficult to isolate the contribution of the proposed fusion method.
- Most methods in Table 1 use different encoder backbones from the one used by the proposed method, and the tokenizer parameter counts are also different.
- Tables 1, 2, and 3 demonstrate the trade-off between reconstruction and generation, while some methods are excluded in some tables. For example, RAEv2 K=23 is omitted from Table 1.
- Some configurations are not consistent across methods, such as the guidance settings and the number of sampling steps. For example, RAEv2 uses 50 sampling steps, while HiRAE uses 100.
Questions for authors
- What constitutes a better overall trade-off, given that RAEv2 achieves better gFID in Table 2 and RAEv2 + DRoRAE achieves better rFID in Table 6?
- Does RAEv2 + DRoRAE in Table 6 use the three-stage training strategy proposed in the original DRoRAE paper?
9 comments
Response to W2.
Some experimental comparisons are not controlled, making it difficult to isolate the contribution of the proposed fusion method.
Response:
Thank you for emphasizing the importance of controlled comparisons. Following your feedback, we conducted two additional ablations within the same 24-expert architecture to isolate the contribution of residual control at the same model capacity. We hold the other training and evaluation settings fixed so that the comparisons focus on changes in residual constraints.
The first ablation (A–D) toggles norm caps and dropout independently to examine their individual and combined effects. The second (D/E/F) keeps dropout settings fixed and compares depth-dependent budgets with a single global cap and equal group budgets. D and F have the same total norm budget, allowing us to test its allocation across depth groups. E provides a simpler alternative that jointly caps the summed residual.
Both use the full HiRAE configuration D as the shared reference and follow the experimental setup and evaluation procedures described in the paper.
Norm caps and dropout (A–D). We toggle the two components independently. When enabled, the shallow, middle, and deep caps are 0.025 / 0.075 / 0.150, and dropout probabilities are 0.50 / 0.25 / 0.10.
| ID | Norm caps | Dropout | rFID ↓ | PSNR ↑ | LPIPS ↓ | Guided gFID ↓ | Unguided gFID ↓ | Guided FDr6 ↓ | Unguided FDr6 ↓ |
|---|---|---|---|---|---|---|---|---|---|
| A | ✗ | ✗ | 0.0149 | 40.5858 | 0.00115 | 9.5624 | 22.9793 | 16.7579 | 33.3217 |
| B | ✓ | ✗ | 0.1775 | 26.9631 | 0.03803 | 2.2340 | 2.8564 | 2.8470 | 6.2798 |
| C | ✗ | ✓ | 0.2165 | 33.5174 | 0.01203 | 4.6550 | 10.4927 | 9.0812 | 19.7496 |
| D (HiRAE) | ✓ | ✓ | 0.2070 | 26.0565 | 0.04352 | 2.1916 | 2.5265 | 2.5810 | 5.4996 |
A–D separate the two components while holding the architecture and capacity fixed:
- Caps improve generation with and without dropout. A→B and C→D add caps without and with dropout, respectively, and both improve generation. The benefit of bounding residual magnitude therefore appears under both dropout conditions.
- Dropout complements the norm constraint. B→D improves all four generation measures while retaining the same caps, showing an additional benefit within the norm-constrained model.
- Reconstruction fidelity alone does not identify the best representation for generation. A reconstructs best but generates worst, whereas D leads on all four generation metrics. This difference explains why residual control matters even when unconstrained fusion can reconstruct accurately.
Budget allocation (D/E/F). D uses shallow, middle, and deep group caps of 0.025 / 0.075 / 0.150 times the anchor norm. E jointly caps the sum of the three group residuals at 25% of the anchor norm. F assigns each group an equal cap of 0.25/3 times the anchor norm. All three use the same dropout probabilities.
| ID | rFID ↓ | PSNR ↑ | LPIPS ↓ | Guided gFID ↓ | Unguided gFID ↓ | Guided FDr6 ↓ | Unguided FDr6 ↓ |
|---|---|---|---|---|---|---|---|
| D (HiRAE) | 0.2070 | 26.0565 | 0.04352 | 2.1916 | 2.5265 | 2.5810 | 5.4996 |
| E | 0.1763 | 29.9312 | 0.02217 | 2.2490 | 3.0377 | 3.1652 | 7.6314 |
| F | 0.2001 | 27.5131 | 0.03473 | 2.2001 | 2.6261 | 2.8561 | 6.2569 |
D/F apply group caps before the progressive-opening coefficients. E applies its global cap to the summed residual after dropout and progressive opening.
- Grouped depth-dependent control improves on a global cap. D improves all four generation metrics over E, supporting the grouped design relative to a single constraint on the summed residual.
- D/F isolate the allocation of a fixed budget across depth. They share dropout probabilities and the total cap budget, with similar mean residual-to-anchor norm ratios (0.2014 and 0.2030). D improves all four generation metrics, although the guided gFID difference is small. This supports depth-dependent allocation at a comparable mean residual magnitude.
- The allocation changes the reconstruction-generation balance. E and F reconstruct better, while D generates better. The role of the budgets is therefore to control the trade-off between these objectives, with the generation benefit established at fixed model capacity.
We thank the reviewer for suggesting these controls, which let us attribute the observed generation gains to residual-control choices within the same architecture.
Response to W3.
Most methods in Table 1 use different encoder backbones from the one used by the proposed method, and the tokenizer parameter counts are also different.
Response:
Thank you for drawing attention to the backbone and parameter-count differences. Table 1 compares representative baselines and advanced tokenizer methods using results from their main published configurations. We retain their original encoders, architectures, and corresponding results to provide context for performance within each method's own framework.
Specifically, RAE, DRoRAE, and DecQ use DINOv2-B; HAE uses DINOv3-L; and SD-VAE, VA-VAE, and REPA-E use their respective VAE configurations. Keeping these backbones preserves the representations and architectures used in the original papers' training and evaluation.
| Representative configurations in Table 1 | Encoder or tokenizer design | Purpose of the comparison |
|---|---|---|
| SD-VAE, VA-VAE, REPA-E | Their respective VAE and training/tuning configurations | Reconstruction references for reported complete systems |
| RAE, DRoRAE, DecQ | DINOv2-B with their respective aggregation and decoding designs | Reconstruction references for reported representation tokenizers |
| HAE | DINOv3-L with spherical latents | Another complete configuration within the same backbone family |
| RAEv2, HiRAE | DINOv3-L with the same latent interface | Direct comparison within a common framework |
Within this broader comparison, HiRAE and RAEv2 share the DINOv3-L backbone and latent interface. HiRAE-7 also uses the same seven selected layers as RAEv2 K=7, providing a compact fusion variant within this common framework.
To make the performance-capacity trade-off explicit, we report reconstruction and guided generation alongside tokenizer size:
| Tokenizer | Total parameters | Increase over RAEv2 K=7 | rFID ↓ | Guided gFID ↓ |
|---|---|---|---|---|
| RAEv2 K=7 | 718.800M | — | 0.299 | 1.060 |
| HiRAE-7 | 777.663M | +8.19% | 0.217 | 1.038 |
| HiRAE-24 | 920.470M | +28.06% | 0.209 | 1.038 |
The metrics follow the paper's reported configurations: RAEv2 uses 50 sampling steps, and both HiRAE variants use 100. W5 reports the matched-step generation comparisons.
- HiRAE-7 retains most of the reconstruction gain with an 8.19% parameter increase. It reduces rFID from RAEv2's 0.299 to 0.217, close to HiRAE-24's 0.209. The compact variant therefore recovers most of the larger model's reconstruction improvement.
- Guided-generation quality is also comparable to HiRAE-24. Both variants report guided gFID of 1.038. At 50 steps, their guided gFID values remain nearly identical: 1.0470 for HiRAE-7 and 1.0471 for HiRAE-24.
These results support HiRAE-7 as a compact option when a layer subset is available. HiRAE-24 uses the full hierarchy to avoid manual layer selection and further improves reconstruction. This comparison characterizes the performance-capacity trade-off; W2 separately tests the residual-control design at fixed capacity. We appreciate the suggestion and will clarify these choices and their associated costs in the revision.
We sincerely thank the reviewer for the careful assessment and for recognizing the simplicity of the method and its reconstruction-generation trade-off. We appreciate your detailed comments on the contribution, experimental controls, and evaluation settings, which help us explain the method's value and scope more clearly.
Response to W1.
Response:
Thank you for encouraging us to clarify the contribution relative to existing feature aggregation methods. RAEv2 uses fixed aggregation of selected encoder layers, while DRoRAE learns full-depth fusion through staged training. Building on these approaches, HiRAE introduces depth-dependent control of how fusion modifies the deepest representation during joint fusion and decoder training.
This design addresses the tension between adding reconstruction detail and preserving a representation suitable for generation. Reconstruction-driven fusion can favor shallow-layer detail; HiRAE assigns tighter residual budgets to shallower groups and retains the deepest representation as an anchor. This lets us learn contributions from all encoder layers while explicitly controlling their influence on the latent space. The group count and numerical budgets are empirical choices within this design.
With the 24-expert architecture fixed, our comparisons show that norm caps and dropout each improve generation, and that depth-dependent budgets provide further gains over global and equal-budget constraints. We report the detailed results in W2.
To examine grouping granularity, we retain all 24 layer experts and map the same depth prior to two, three, or four contiguous groups:
All configurations share a budget of 16,000 tokenizer updates and 5,004 generator updates, with 5K reconstruction images and 10K generated samples per setting.
Together, these results support depth-dependent residual control as a practical way to use the full encoder hierarchy for reconstruction and generation. We appreciate the suggestion and will make this contribution and its empirical basis clearer in the revision.