acceptodds
Under review as a conference paper at ICLR 2027

HiRAE: Hierarchical Representation Autoencoding with Residual Budgets

14 views

Abstract

Pretrained visual representations support image generation, but may not fully pre- serve the fine-grained details needed for faithful reconstruction. Meanwhile, inter- mediate encoder layers contain complementary visual details, but learning to fuse them for reconstruction can produce a latent distribution that is difficult to model. Existing fusion methods require empirical tuning of layer selection or staged opti- mization of fusion and decoding, increasing configuration effort or training com- plexity. We introduce HiRAE (Hierarchical Representation Autoencoder), which learns a hierarchical fusion framework over the full encoder hierarchy to improve reconstruction fidelity while maintaining compatibility with generative modeling. HiRAE groups encoder layers by depth and learns residual corrections to the deep- est representation. Group-wise norm caps bound these corrections relative to the deep anchor, with tighter budgets for shallower groups. Our HiRAE-24 preserves the latent token count and channel dimension. On ImageNet-256, HiRAE-24 re- duces reconstruction FID from 0.299 to 0.209 relative to RAEv2 while maintain- ing competitive guided generation quality. For text-to-image generation, HiRAE- 24 improves alignment over RAEv2 on GenEval, DPG-Bench, and GenAI-Bench both before and after supervised fine-tuning. Under the same generator-training and evaluation protocol, post-fine-tuning GenEval increases from 84.86 to 87.70.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Representation AutoEncoder, RepreMulti-layer Feature Fusion, Visual Tokenization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iAuHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a tokenizer that aggregates features from all layers of DINOv3, achieving a better reconstruction and generation trade-off. Experimental results on class-conditional generation and text-to-image generation demonstrate the effectiveness of the proposed method.

Critical strengths

The proposed method is simple and easy to follow. Experimental results show the method achieves a good trade-off between reconstruction and generation.

Critical weaknesses
  • The method seems incremental. Previous methods such as RAEv2 and DRoRAE have demonstrated the effectiveness of feature aggregation, while the paper introduces a heuristic aggregation method with some manually specified hyperparameters, such as the number of groups, dropout probabilities, and groupwise norm caps.

  • Some experimental comparisons are not controlled, making it difficult to isolate the contribution of the proposed fusion method.

    • Most methods in Table 1 use different encoder backbones from the one used by the proposed method, and the tokenizer parameter counts are also different.
    • Tables 1, 2, and 3 demonstrate the trade-off between reconstruction and generation, while some methods are excluded in some tables. For example, RAEv2 K=23 is omitted from Table 1.
    • Some configurations are not consistent across methods, such as the guidance settings and the number of sampling steps. For example, RAEv2 uses 50 sampling steps, while HiRAE uses 100.
Questions for authors
  • What constitutes a better overall trade-off, given that RAEv2 achieves better gFID in Table 2 and RAEv2 + DRoRAE achieves better rFID in Table 6?
  • Does RAEv2 + DRoRAE in Table 6 use the three-stage training strategy proposed in the original DRoRAE paper?
9 comments

We sincerely thank the reviewer for the careful assessment and for recognizing the simplicity of the method and its reconstruction-generation trade-off. We appreciate your detailed comments on the contribution, experimental controls, and evaluation settings, which help us explain the method's value and scope more clearly.

Response to W1.

The method seems incremental. Previous methods such as RAEv2 and DRoRAE have demonstrated the effectiveness of feature aggregation, while the paper introduces a heuristic aggregation method with some manually specified hyperparameters, such as the number of groups, dropout probabilities, and groupwise norm caps.

Response:

Thank you for encouraging us to clarify the contribution relative to existing feature aggregation methods. RAEv2 uses fixed aggregation of selected encoder layers, while DRoRAE learns full-depth fusion through staged training. Building on these approaches, HiRAE introduces depth-dependent control of how fusion modifies the deepest representation during joint fusion and decoder training.

This design addresses the tension between adding reconstruction detail and preserving a representation suitable for generation. Reconstruction-driven fusion can favor shallow-layer detail; HiRAE assigns tighter residual budgets to shallower groups and retains the deepest representation as an anchor. This lets us learn contributions from all encoder layers while explicitly controlling their influence on the latent space. The group count and numerical budgets are empirical choices within this design.

With the 24-expert architecture fixed, our comparisons show that norm caps and dropout each improve generation, and that depth-dependent budgets provide further gains over global and equal-budget constraints. We report the detailed results in W2.

To examine grouping granularity, we retain all 24 layer experts and map the same depth prior to two, three, or four contiguous groups:

GroupsReconstruction rFID ↓Unguided gFID ↓Guided gFID ↓
23.29344.79529.385
33.36344.05528.354
43.45745.17231.321

All configurations share a budget of 16,000 tokenizer updates and 5,004 generator updates, with 5K reconstruction images and 10K generated samples per setting.

  • Three groups provide the best generation results in this comparison. They yield the lowest guided and unguided gFID, while two groups give slightly better reconstruction. This supports the three-group choice for balancing the two objectives under the tested budget.
  • The comparison tests control granularity with the same layer coverage. Every configuration retains all 24 experts and the same depth prior. Increasing the number of groups to four does not improve generation, supporting three groups as an effective granularity for this design.

Together, these results support depth-dependent residual control as a practical way to use the full encoder hierarchy for reconstruction and generation. We appreciate the suggestion and will make this contribution and its empirical basis clearer in the revision.

Response to W2.

Some experimental comparisons are not controlled, making it difficult to isolate the contribution of the proposed fusion method.

Response:

Thank you for emphasizing the importance of controlled comparisons. Following your feedback, we conducted two additional ablations within the same 24-expert architecture to isolate the contribution of residual control at the same model capacity. We hold the other training and evaluation settings fixed so that the comparisons focus on changes in residual constraints.

The first ablation (A–D) toggles norm caps and dropout independently to examine their individual and combined effects. The second (D/E/F) keeps dropout settings fixed and compares depth-dependent budgets with a single global cap and equal group budgets. D and F have the same total norm budget, allowing us to test its allocation across depth groups. E provides a simpler alternative that jointly caps the summed residual.

Both use the full HiRAE configuration D as the shared reference and follow the experimental setup and evaluation procedures described in the paper.

Norm caps and dropout (A–D). We toggle the two components independently. When enabled, the shallow, middle, and deep caps are 0.025 / 0.075 / 0.150, and dropout probabilities are 0.50 / 0.25 / 0.10.

IDNorm capsDropoutrFID ↓PSNR ↑LPIPS ↓Guided gFID ↓Unguided gFID ↓Guided FDr6 ↓Unguided FDr6 ↓
A✗✗0.014940.58580.001159.562422.979316.757933.3217
B✓✗0.177526.96310.038032.23402.85642.84706.2798
C✗✓0.216533.51740.012034.655010.49279.081219.7496
D (HiRAE)✓✓0.207026.05650.043522.19162.52652.58105.4996

A–D separate the two components while holding the architecture and capacity fixed:

  • Caps improve generation with and without dropout. A→B and C→D add caps without and with dropout, respectively, and both improve generation. The benefit of bounding residual magnitude therefore appears under both dropout conditions.
  • Dropout complements the norm constraint. B→D improves all four generation measures while retaining the same caps, showing an additional benefit within the norm-constrained model.
  • Reconstruction fidelity alone does not identify the best representation for generation. A reconstructs best but generates worst, whereas D leads on all four generation metrics. This difference explains why residual control matters even when unconstrained fusion can reconstruct accurately.

Budget allocation (D/E/F). D uses shallow, middle, and deep group caps of 0.025 / 0.075 / 0.150 times the anchor norm. E jointly caps the sum of the three group residuals at 25% of the anchor norm. F assigns each group an equal cap of 0.25/3 times the anchor norm. All three use the same dropout probabilities.

IDrFID ↓PSNR ↑LPIPS ↓Guided gFID ↓Unguided gFID ↓Guided FDr6 ↓Unguided FDr6 ↓
D (HiRAE)0.207026.05650.043522.19162.52652.58105.4996
E0.176329.93120.022172.24903.03773.16527.6314
F0.200127.51310.034732.20012.62612.85616.2569

D/F apply group caps before the progressive-opening coefficients. E applies its global cap to the summed residual after dropout and progressive opening.

  • Grouped depth-dependent control improves on a global cap. D improves all four generation metrics over E, supporting the grouped design relative to a single constraint on the summed residual.
  • D/F isolate the allocation of a fixed budget across depth. They share dropout probabilities and the total cap budget, with similar mean residual-to-anchor norm ratios (0.2014 and 0.2030). D improves all four generation metrics, although the guided gFID difference is small. This supports depth-dependent allocation at a comparable mean residual magnitude.
  • The allocation changes the reconstruction-generation balance. E and F reconstruct better, while D generates better. The role of the budgets is therefore to control the trade-off between these objectives, with the generation benefit established at fixed model capacity.

We thank the reviewer for suggesting these controls, which let us attribute the observed generation gains to residual-control choices within the same architecture.

Response to W3.

Most methods in Table 1 use different encoder backbones from the one used by the proposed method, and the tokenizer parameter counts are also different.

Response:

Thank you for drawing attention to the backbone and parameter-count differences. Table 1 compares representative baselines and advanced tokenizer methods using results from their main published configurations. We retain their original encoders, architectures, and corresponding results to provide context for performance within each method's own framework.

Specifically, RAE, DRoRAE, and DecQ use DINOv2-B; HAE uses DINOv3-L; and SD-VAE, VA-VAE, and REPA-E use their respective VAE configurations. Keeping these backbones preserves the representations and architectures used in the original papers' training and evaluation.

Representative configurations in Table 1Encoder or tokenizer designPurpose of the comparison
SD-VAE, VA-VAE, REPA-ETheir respective VAE and training/tuning configurationsReconstruction references for reported complete systems
RAE, DRoRAE, DecQDINOv2-B with their respective aggregation and decoding designsReconstruction references for reported representation tokenizers
HAEDINOv3-L with spherical latentsAnother complete configuration within the same backbone family
RAEv2, HiRAEDINOv3-L with the same latent interfaceDirect comparison within a common framework

Within this broader comparison, HiRAE and RAEv2 share the DINOv3-L backbone and latent interface. HiRAE-7 also uses the same seven selected layers as RAEv2 K=7, providing a compact fusion variant within this common framework.

To make the performance-capacity trade-off explicit, we report reconstruction and guided generation alongside tokenizer size:

TokenizerTotal parametersIncrease over RAEv2 K=7rFID ↓Guided gFID ↓
RAEv2 K=7718.800M—0.2991.060
HiRAE-7777.663M+8.19%0.2171.038
HiRAE-24920.470M+28.06%0.2091.038

The metrics follow the paper's reported configurations: RAEv2 uses 50 sampling steps, and both HiRAE variants use 100. W5 reports the matched-step generation comparisons.

  • HiRAE-7 retains most of the reconstruction gain with an 8.19% parameter increase. It reduces rFID from RAEv2's 0.299 to 0.217, close to HiRAE-24's 0.209. The compact variant therefore recovers most of the larger model's reconstruction improvement.
  • Guided-generation quality is also comparable to HiRAE-24. Both variants report guided gFID of 1.038. At 50 steps, their guided gFID values remain nearly identical: 1.0470 for HiRAE-7 and 1.0471 for HiRAE-24.

These results support HiRAE-7 as a compact option when a layer subset is available. HiRAE-24 uses the full hierarchy to avoid manual layer selection and further improves reconstruction. This comparison characterizes the performance-capacity trade-off; W2 separately tests the residual-control design at fixed capacity. We appreciate the suggestion and will clarify these choices and their associated costs in the revision.

Reviewer_5wT8OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HiRAE combines features from all layers of a frozen vision encoder to improve reconstruction without changing the latent shape used by the generator. Layer-specific experts and a spatial router produce residual corrections to the deepest representation. These corrections are grouped by depth and constrained through norm caps and dropout. The paper evaluates reconstruction, class-conditional generation, and text-to-image generation, alongside analyses of latent structure and decoder sensitivity.

Critical strengths
  1. The reconstruction improvement is supported by several metrics, rather than rFID alone. Relative to RAEv2, rFID falls, PSNR improves, and LPIPS falls on the matched subset. The latent token count and channel dimension remain unchanged.

  2. The text-to-image experiments provide useful evidence beyond reconstruction. With the same generator-training recipe, HiRAE-24 improves all three reported alignment benchmarks before and after finetuning. The matched full-depth fusion comparison also shows why reconstruction quality alone is insufficient: the DRoRAE-style variant reconstructs better but generates worse.

Critical weaknesses
  1. The ablations do not cleanly establish the contribution of depth-dependent budgets. Table 5 removes norm caps and dropout together, and its unregularized variant uses depth-mode experts rather than the main 24-expert configuration. The DRoRAE-style comparison also changes routing and other regularization choices. It remains unclear how much of the gain requires different budgets across depth, rather than a simpler global constraint.

  2. The generation benefit is less consistent than the reconstruction benefit. Guided ImageNet FID changes very little, while unguided FID worsens. The paper acknowledges this, but the results leave open whether the new latent space is easier to model or mainly works better with guidance. The text-to-image gains are more encouraging.

Questions for authors
  1. With the 24-expert architecture fixed, how do caps alone, dropout alone, a single global cap, and equal group budgets compare with the proposed depth-dependent settings?

  2. Were guidance strengths selected independently for each tokenizer? Would the generation conclusions hold across a guidance sweep and repeated sampling seeds, particularly for the small ImageNet FID difference and the GenEval improvement?

9 comments

We sincerely thank the reviewer for the thoughtful feedback and for recognizing the paired reconstruction and text-to-image results. We especially appreciate your suggestions for clarifying component contributions and the role of guidance.

Response to W1.

The ablations do not cleanly establish the contribution of depth-dependent budgets. Table 5 removes norm caps and dropout together, and its unregularized variant uses depth-mode experts rather than the main 24-expert configuration. The DRoRAE-style comparison also changes routing and other regularization choices. It remains unclear how much of the gain requires different budgets across depth, rather than a simpler global constraint.

Response:

Thank you for pointing out the need to separate these effects more carefully in the original ablations. To examine the contributions of norm caps, dropout, and depth-dependent budgets more directly, we conducted two additional comparisons within the same 24-expert architecture. With expert capacity and routing architecture fixed, these comparisons focus on the effects of residual constraints.

  • The first comparison (A–D) toggles norm caps and dropout independently to examine their individual and combined effects, addressing the removal of both components together in the original ablation.
  • The second (D/E/F) uses the same dropout settings and compares depth-dependent budgets with a single global cap and equal group budgets. It tests whether simpler constraints suffice and whether allocating budgets by depth provides additional generation gains.

Both use the full HiRAE configuration D as the shared reference. The DRoRAE-style comparison evaluates complete fusion methods, while these ablations test component contributions within the same architecture. We follow the experimental setup and evaluation procedures described in the paper.

Norm caps and dropout (A–D). We toggle the two components independently. When enabled, the shallow, middle, and deep caps are 0.025 / 0.075 / 0.150, and dropout probabilities are 0.50 / 0.25 / 0.10.

IDNorm capsDropoutrFID ↓PSNR ↑LPIPS ↓Guided gFID ↓Unguided gFID ↓Guided FDr6 ↓Unguided FDr6 ↓
A✗✗0.014940.58580.001159.562422.979316.757933.3217
B✓✗0.177526.96310.038032.23402.85642.84706.2798
C✗✓0.216533.51740.012034.655010.49279.081219.7496
D (HiRAE)✓✓0.207026.05650.043522.19162.52652.58105.4996
  • Caps help with and without dropout. Without dropout, A→B reduces guided gFID from 9.5624 to 2.2340 and unguided gFID from 22.9793 to 2.8564. With dropout, C→D also improves generation. These paired comparisons identify a benefit from bounding residual magnitude under both dropout conditions.
  • Dropout has an additional effect within the capped model. B→D reduces unguided gFID by 11.5%, guided and unguided FDr6 by 9.3% and 12.4%, and guided gFID by 1.9%. Since the caps stay fixed, this comparison separates the contribution of dropout from that of the norm constraint.
  • Reconstruction fidelity alone would select a different configuration. A reaches PSNR 40.5858 and LPIPS 0.00115 yet has the weakest generation results; D leads on all four generation metrics. This divergence explains the need to evaluate residual control through both reconstruction and generation.

Budget allocation (D/E/F). D uses shallow, middle, and deep group caps of 0.025 / 0.075 / 0.150 times the anchor norm. E jointly caps the sum of the three group residuals at 25% of the anchor norm. F assigns each group an equal cap of 0.25/3 times the anchor norm. All three use the same dropout probabilities.

IDrFID ↓PSNR ↑LPIPS ↓Guided gFID ↓Unguided gFID ↓Guided FDr6 ↓Unguided FDr6 ↓
D (HiRAE)0.207026.05650.043522.19162.52652.58105.4996
E0.176329.93120.022172.24903.03773.16527.6314
F0.200127.51310.034732.20012.62612.85616.2569

D/F apply group caps before the progressive-opening coefficients. E applies its global cap to the summed residual after dropout and progressive opening.

  • Grouped depth-dependent control improves on a single global constraint. D reduces guided/unguided gFID by 2.6%/16.8% and FDr6 by 18.5%/27.9% relative to E. This establishes a generation benefit over the simpler global-cap alternative under the tested setup.
  • D/F isolate how a fixed group budget is allocated. D improves all four generation metrics over F, with a small guided gFID difference. Their dropout probabilities and total cap budget match, and their mean residual-to-anchor norm ratios are similar (0.2014 and 0.2030). The gain therefore supports depth-dependent allocation at a comparable mean residual magnitude.
  • Budget allocation changes which objective is favored. E and F reconstruct better, while D generates better. The results favor D for generation among these constraints, with the reconstruction cost explicit.

Together, A–D separate the effects of caps and dropout, while D/F identify the additional value of depth-dependent allocation at fixed capacity. We thank the reviewer for suggesting the controls that make this attribution possible.

Response to W2.

The generation benefit is less consistent than the reconstruction benefit. Guided ImageNet FID changes very little, while unguided FID worsens. The paper acknowledges this, but the results leave open whether the new latent space is easier to model or mainly works better with guidance. The text-to-image gains are more encouraging.

Response:

Thank you for highlighting the distinction between latent modeling and the benefits of guidance, and for recognizing the improvements in text-to-image generation.

As discussed in the paper, intermediate-layer features provide reconstruction details missing from the deepest representation, but reconstruction-driven fusion can favor shallow-layer detail and make the latent space harder to model. HiRAE keeps the deepest representation as an anchor and uses depth-dependent residual budgets to bound corrections from different depths, improving reconstruction while maintaining compatibility with generative modeling.

RAEv2's results also reflect the modeling difficulty that arises when incorporating more encoder layers: increasing K from 7 to 23 improves reconstruction rFID but worsens both guided and unguided gFID. The generators in the table all use 80 training epochs and 50 Euler sampling steps. RAEv2 values come from Tables 14 and 16 of its paper:

ModelReconstruction rFID ↓Guided gFID ↓Unguided gFID ↓
RAEv2 K=70.2991.0601.650
RAEv2 K=230.1851.2503.010
HiRAE-240.2091.04712.3847
  • The K=23 comparison supports gains with and without guidance. HiRAE-24 uses all 24 layers and reduces guided gFID from 1.250 to 1.0471 and unguided gFID from 3.010 to 2.3847, reductions of 16.2% and 20.8%. The unguided improvement shows that the advantage over this reference also appears without added guidance, supporting more effective generative use of the full encoder hierarchy.
  • The advantage over K=7 is guidance-dependent. HiRAE improves reconstruction and guided generation, while K=7 retains lower unguided gFID. The K=7 comparison therefore supports a favorable trade-off for guided generation, while the K=23 comparison shows gains in both sampling modes.
  • Text-to-image results show a downstream alignment benefit. Under the same generator-training and sampling settings, HiRAE-24 outperforms RAEv2 on GenEval, DPG-Bench, and GenAI-Bench both before and after supervised fine-tuning (SFT). After SFT, scores rise from 84.86 to 87.70, 84.90 to 86.35, and 71.69 to 72.66, respectively. These gains connect the tokenizer choice to better text-image alignment under a common generation recipe.
Reviewer_cfCpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes using all features from all frozen DINOv3 layers combined to regularize RAE, termed hiRAE (hierarchical). In order not to collapse to reconstruction, the paper also proposes using deepest representation. Both reconstruction and generation show good improvements.

Critical strengths

(+) good reconstruction results: Compared with RAEv2, HiRAE's rFID improves from 0.299 to 0.209, PSNR 22.667 to 26.377, LPIPS improves from 0.074 to 0.043

(+) consistent text-to-image improvements: Compared with RAEv2, on t2i, HiRAE improves GenEval from 56.42 to 60.94, DPG-Bench from 81.22 to 82.73, and GenAI-Bench from 67.02 to 68.08 before fine-tuning. After fine-tuning, the scores improve from 84.86 to 87.70, 84.90 to 86.35, and 71.69 to 72.66. On imagenet-256, it improves 1.060->1.038

(+) the design of the architecture makes sense and combining features from all layers of DINOv3 is reasonable

(+) no extra added cost in text-to-image generation inference

Critical weaknesses

(-) it is hard to tell HiRAE's performance reconstruction gain from the visual examples, except for faces and small text despite huge gain from 0.299 to 0.209. RAEv2's small faces and text are not the strongest (e.g. a 16-channel regular VAE would do better than F.1 RECONSTRUCTION COMPARISONS.

(-) text-to-image generation quality improvement is moderate. The picked examples in F.4 TEXT-TO-IMAGE COMPARISONS is not impressive at all.

(-) paper builds largely on existing hierarchical fusion and routing ideas. Its main distinction is probably depth-dependent residual regularization? It should be more clearly stated.

Questions for authors

none

4 comments

We sincerely thank the reviewer for the careful reading and for recognizing the reconstruction results, text-to-image improvements, and rationale for hierarchical feature fusion. We appreciate your suggestions on the visual examples and contribution, which help us present the work more clearly.

Response to W1.

it is hard to tell HiRAE's performance reconstruction gain from the visual examples, except for faces and small text despite huge gain from 0.299 to 0.209. RAEv2's small faces and text are not the strongest (e.g. a 16-channel regular VAE would do better than F.1 RECONSTRUCTION COMPARISONS.

Response:

Thank you for this specific and helpful comment. We agree that the examples in F.1 could better convey HiRAE's reconstruction improvements beyond faces and small text. We will select more representative reconstruction examples in the revision and provide matched close-ups of textures, edges, and local color details to make the differences easier to assess.

The quantitative results support HiRAE's improvement over RAEv2 in reconstruction distribution quality, paired-image fidelity, and local detail:

MetricRAEv2HiRAE-24
rFID ↓0.2990.209
PSNR ↑22.66726.377
LPIPS ↓0.0740.043
Sobel MSE ×10³ ↓1.8300.816
Laplacian MSE ×10³ ↓43.37832.645

PSNR improves by 3.710 dB and LPIPS decreases by 0.0310 on average. These paired-image gains show that the lower rFID accompanies improved fidelity to the inputs, with the edge metrics providing complementary evidence of local-detail preservation.

To examine whether the gains extend across image content, we group the same 5,000 inputs into four texture quartiles by their original Sobel-gradient strength, with 1,250 images per group. The grouping depends only on the inputs, independently of either model's reconstructions.

Original-image textureRAEv2 LPIPS ↓HiRAE-24 LPIPS ↓Absolute reduction ↑Relative reduction ↑
Q1: Lowest0.047280.024860.0224247.42%
Q20.062780.034190.0285945.53%
Q30.079110.045440.0336742.56%
Q4: Highest0.105810.066330.0394837.31%
  • The improvement extends across texture levels. HiRAE lowers LPIPS in all four groups. Because the grouping uses only the original images, this pattern is independent of the reconstruction examples selected for display and supports a broader fidelity improvement across the evaluated inputs.
  • High-texture images benefit where the baseline error is largest. The most textured group has both the highest RAEv2 LPIPS and the largest absolute reduction. Less textured images show larger relative reductions. Together, these results support gains across texture complexity, including inputs with substantial texture detail.

We appreciate the reviewer's suggestion to make the qualitative presentation more informative. We will use these analyses to improve the example selection and make the measured reconstruction gains easier to see.

Response to W2.

text-to-image generation quality improvement is moderate. The picked examples in F.4 TEXT-TO-IMAGE COMPARISONS is not impressive at all.

Response:

Thank you for the helpful feedback on the qualitative examples. We agree that our selection in F.4 does not convey HiRAE's improvements in text-image alignment clearly enough. In the revision, we will select more representative examples from the generated outputs, keeping prompts, seeds, and sampling settings matched across models to make the comparison clearer.

The quantitative results show gains on GenEval, DPG-Bench, and GenAI-Bench both before and after supervised fine-tuning, under shared training and sampling settings. The gains before fine-tuning show that the benefit is already present in the pretrained generators, while their persistence afterward supports its relevance after downstream adaptation. This agreement supports our claim of improved text-image alignment across the evaluated benchmarks and stages; the revised examples should make that measured benefit easier to assess visually.

Response to W3.

paper builds largely on existing hierarchical fusion and routing ideas. Its main distinction is probably depth-dependent residual regularization? It should be more clearly stated.

Response:

Thank you for helping us clarify the central contribution. We agree that depth-dependent residual regularization deserves greater emphasis. The experts and router provide the fusion mechanism; depth-dependent residual control determines how intermediate-layer information modifies the deepest representation during joint fusion and decoder training. This design addresses the tension between recovering reconstruction detail and maintaining a latent space suitable for generation.

We conducted two additional ablations within the fixed 24-expert architecture. A–D separate the effects of norm caps and dropout; D/E/F compare depth-dependent budgets with a global cap and equal group budgets while keeping dropout fixed. Both use full HiRAE (D) as the shared reference and follow the paper's training and evaluation procedures. Holding the architecture and capacity fixed lets these comparisons test the contribution of residual control directly.

Norm caps and dropout (A–D). We toggle the two components independently. When enabled, the shallow, middle, and deep caps are 0.025 / 0.075 / 0.150, and dropout probabilities are 0.50 / 0.25 / 0.10.

IDNorm capsDropoutrFID ↓PSNR ↑LPIPS ↓gFID-G ↓gFID-U ↓FDr6-G ↓FDr6-U ↓
A✗✗0.014940.58580.001159.562422.979316.757933.3217
B✓✗0.177526.96310.038032.23402.85642.84706.2798
C✗✓0.216533.51740.012034.655010.49279.081219.7496
D (HiRAE)✓✓0.207026.05650.043522.19162.52652.58105.4996
  • Accurate reconstruction alone does not ensure useful generative latents. Unconstrained A reconstructs best but has the weakest generation results. This mismatch motivates controlling reconstruction-driven fusion even when it can fit the inputs accurately.
  • Explicit norm bounds provide the larger generation benefit. Caps-only B outperforms dropout-only C on all four generation metrics, although both improve over A. Residual dropout alone therefore does not reproduce the benefit of bounding residual magnitude in this setting.
  • Dropout complements the norm bounds. Adding dropout to B yields D and improves all four metrics further. The two controls thus make distinct contributions within the same fusion architecture.

Budget allocation (D/E/F). D uses shallow, middle, and deep group caps of 0.025 / 0.075 / 0.150 times the anchor norm. E jointly caps the sum of the three group residuals at 25% of the anchor norm. F assigns each group an equal cap of 0.25/3 times the anchor norm. All three use the same dropout probabilities.

IDrFID ↓PSNR ↑LPIPS ↓gFID-G ↓gFID-U ↓FDr6-G ↓FDr6-U ↓
D (HiRAE)0.207026.05650.043522.19162.52652.58105.4996
E0.176329.93120.022172.24903.03773.16527.6314
F0.200127.51310.034732.20012.62612.85616.2569
  • The distribution of the group budget matters for generation. D and F share the total cap budget and dropout settings. D improves all four generation metrics, although the guided gFID difference is small, supporting allocation by depth over a uniform allocation within the grouped model.
  • The benefit concerns the reconstruction-generation balance. D also improves all four generation metrics over global constraint E, while E and F reconstruct better. These comparisons favor D when choosing the residual constraints for generation, even though it does not minimize reconstruction error among these variants.

These comparisons support a specific contribution: depth-dependent residual control improves generation within a fixed fusion architecture. Following your suggestion, we will emphasize this contribution in the revision and explain how experts and routing implement it.

To complement these design ablations, we also compare RAEv2 K=7 and HiRAE-24 under the same sampling settings. We use the official RAEv2 generator and train the HiRAE generator for 80 epochs. Both use 256×256 resolution, 50 Euler steps, CFG=1, IG=1.78 on [0.1,1.0], and sampling seeds 42, 43, and 44, with 50K images per seed. The table gives the average results across the three sampling runs.

MetricRAEv2HiRAE-24
gFID ↓1.10191.0423
FDr6 ↓2.08561.9081

The guidance sweep uses the same sampling settings with seed=42:

IGRAEv2 gFID ↓HiRAE-24 gFID ↓RAEv2 FDr6 ↓HiRAE-24 FDr6 ↓
1.01.63762.38473.88444.9716
1.31.22211.32612.71622.9876
1.51.11771.09502.31432.3408
1.781.10411.04712.07461.8979
2.01.15201.11002.07701.7638
  • The guided gain repeats across sampling runs. HiRAE lowers gFID and FDr6 across all three seeds, with mean reductions of 5.4% and 8.5%. This connects the component-level evidence to improved generation from the resulting tokenizer under shared sampling settings.
  • The sweep defines the scope of that gain. HiRAE also improves both metrics at IG=2.0, while RAEv2 performs better at IG=1.0 and 1.3. Together with the reconstruction gains in W1, these results support a useful trade-off for guided generation, with the advantage depending on guidance strength.

We thank the reviewer for helping us present the design contribution more clearly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.