acceptodds
Withdrawn from ICLR 2027

Beyond the Last Layer: Multi-Layer Representation Fusion for Visual Tokenization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Representation autoencoders that reuse frozen pretrained vision encoders as visual tokenizers have achieved strong reconstruction and generation quality. However, using only the last encoder layer leaves intermediate features unavailable to the decoder, limiting the reconstruction of fine visual details. We improve detail reconstruction by supplying the decoder with fused intermediate-layer features. We propose DRoRAE(Depth-Routed Representation AutoEncoder), which uses energy-constrained routing for adaptive multi-layer aggregation and incremental correction to retain a contribution from the original representation. Starting from a pretrained base model, we first train fusion with the decoder frozen, then fine-tune the decoder with fusion fixed. On ImageNet-256, DRoRAE reduces rFID from 0.570 to 0.290 and improves class-conditional generation FID from 1.740 to 1.650 (with AutoGuidance). For text-to-image generation, DRoRAE achieves a overall GenEval score of 0.600, compared with 0.560 for the base model. With encoder features and all other settings fixed, increasing fusion expert capacity improves reconstruction. Over the measured range, rFID decreases approximately log-linearly with fusion expert capacity ().

Primary area:
generative models
Keywords:
Representation AutoEncoder, Unified Tokenizer, Image Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ttccOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces multi-layer feature fusion into RAE, leveraging intermediate-layer features of DINOv2 to supplement the final-layer representations. The method adopts positive and negative routing weights while preserving the original final-layer representations, and then trains the fusion module and decoder in stages. In experiments, the reconstruction rFID decreased from 0.570 to 0.290; after incorporating AutoGuidance, the class-conditional generation FID dropped from 1.740 to 1.650, and the text-to-image GenEval increased from 0.560 to 0.600.

Critical strengths
  1. The reconstruction performance is significantly improved. Improvements are observed in rFID, PSNR, LPIPS and SSIM. Fine details such as digits and human faces in the demonstrations are clearer, while the number and dimension of tokens to be predicted by the generative model are not increased.
  2. The trade-off between reconstruction and generation echoes the findings of VAVAE. Experiments including joint training and encoder fine-tuning show that better reconstruction does not necessarily facilitate generation. Experiments on preserving final-layer representations and staged training provide concrete evidence for this observation.
Critical weaknesses
  1. Lack of direct comparison with existing multi-layer RAE methods. RAEv2[1] has improved reconstruction using simple multi-layer summation.
  2. Limited generation gains. With AutoGuidance enabled, the FID decreased from 1.740 to 1.650; without guidance, it instead rose from 2.160 to 2.678.

[1] Singh, Jaskirat, et al. “Improved Baselines with Representation Autoencoders.” Advances in Neural Information Processing Systems. 2026.

Questions for authors
  1. Under the same settings, what performance can simple multi-layer summation achieve? How much additional gain does the method proposed in this paper bring compared with RAEv2?
  2. Why does the generation performance degrade without guidance after multi-layer fusion? When comparing different guidance strengths under the same guidance training budget, does the proposed method still retain advantages?
Reviewer_CZxiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors identify a limitation of existing Representation Autoencoder (RAE) methods: they feed only the final-layer features of a vision foundation model (VFM) into the decoder, overlooking the rich fine-grained information available in intermediate layers. To overcome this limitation, they propose DRoRAE, which supplies the decoder with a learned fusion of features from all encoder layers while keeping the latent interface unchanged. The approach combines per-layer expert MLPs, a router that assigns per-token signed weights with energy-based normalization, and an incremental correction step, and is trained using a three-stage procedure. On ImageNet-256, DRoRAE achieves a reconstruction FID of 0.29 and a class-conditional generation FID of 1.65 under the AutoGuidance setting. It also attains a GenEval score of 0.60, improving over the RAE baseline. Additionally, the paper reports an approximately log-linear decrease in reconstruction FID as fusion capacity increases.

Critical strengths

Simple and plug-and-play design. DRoRAE consists of three lightweight components: layer-wise experts, energy-constrained routing, and incremental correction. It incorporates intermediate-layer features into decoding while preserving the original RAE latent interface and leaving the frozen encoder and downstream generator unchanged. This makes it a plug-and-play extension to the RAE architecture, with strong practical utility and potential for broader applicability.

Improved performance over the baselines. On ImageNet-256, DRoRAE improves reconstruction quality over RAE and other baselines, achieving lower rFID with the same backbone and decoder architecture. When combined with AutoGuidance, it also achieves better gFID using the same generator and training budget. These downstream benefits further extend to text-to-image generation.

Comprehensive experimental analysis. Beyond reporting the main reconstruction and generation results, the paper investigates the sources of improvement, including a decomposition of FID into mean and covariance terms. It also presents systematic ablation studies examining the contributions of individual components and the effects of key design choices. Together, these analyses provide clear and convincing support for the conclusions.

Critical weaknesses

Limited novelty. Multi-layer feature fusion has been extensively studied in vision foundation models and MLLMs (e.g., [1, 2]). The proposed approach primarily adapts this established idea to the RAE architecture, and its conceptual novelty appears limited.

The generation gains depend on AutoGuidance, weakening support for the central claim. The results indicate that AutoGuidance plays a substantial role in the final generation performance. Without guidance, DRoRAE actually performs worse than the RAE baseline in gFID (2.680 vs. 2.160). It only outperforms RAE when AutoGuidance is applied, but the improvement is modest (1.650 vs. 1.740). This pattern is more consistent with a representation that benefits more from guidance than with a clear improvement in the intrinsic suitability of the latent space for generation. Thus, the central claim that multi-layer fusion improves downstream generation is not sufficiently substantiated.

The experimental conclusions are validated only on DINOv2-Base. The paper does not discuss whether the method remains effective with stronger backbones, such as DINOv2-Large or DINOv2-Giant. Moreover, the evidence for the claimed log-linear scaling relationship is relatively weak: the layer-count fit yields an R2 of only 0.49, increasing to 0.59 in the combined analysis. Given the small magnitude of the rFID changes, the overall scaling effect is not particularly compelling. [1] Multimodal Language Models See Better When They Look Shallower [2] Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practices

Questions for authors

Comparison with simpler fusion strategies. Under the same training budget, does DRoRAE outperform simpler multi-layer feature fusion methods, such as directly summing features from multiple layers?

Complementary reconstruction metrics. The ablation studies and the scaling experiments in Fig. 6 report only rFID. Pprior work has highlighted the limitations of rFID as a reconstruction quality metric, were complementary metrics such as PSNR, SSIM, and LPIPS also evaluated? If so, are their trends consistent with those observed for rFID?

Choice of the number of fused layers. Fig. 6 suggests that fusing nine layers yields the best performance. What motivated the final design choice to fuse features from all layers instead?

Computational overhead. How much additional training time and GPU memory does the proposed fusion module require compared with the RAE baseline?

Reviewer_K3QGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DRoRAE is a multi-layer representation fusion method for visual tokenization in image reconstruction and generation. It uses layer-wise MLPs and signed, normalized routing weights to aggregate intermediate features from a frozen DINOv2 encoder. The fused features are combined with the original last-layer representation while preserving the latent shape, with fusion and decoder adaptation trained sequentially. Experiments reduce reconstruction rFID from 0.570 to 0.290 and improve guided generation FID from 1.740 to 1.650, although unguided generation FID worsens from 2.160 to 2.678.

Critical strengths
  • Substantial reconstruction gains. rFID improves from 0.570 to 0.290, with consistent improvements in PSNR, LPIPS, and SSIM. With the decoder frozen, fusion alone achieves 0.470.
  • Downstream evaluation with a fixed latent interface. DRoRAE preserves RAE’s latent shape and improves guided generation FID from 1.740 to 1.650 and GenEval from 0.560 to 0.600.
Critical weaknesses

Reconstruction gains do not consistently improve generation. Unguided generation FID worsens from 2.160 to 2.678 at the same 80-epoch generator budget (Table 13). With AutoGuidance, the improvement is small, while IS decreases and precision/recall remain unchanged (Table 1). This limits the practical generation benefit.

Improvements over RAE do not establish an overall performance advantage over other tokenizers. DRoRAE trails VA-VAE on rFID, PSNR, LPIPS, and SSIM (Table 2), and its GenEval score of 0.600 is below VA-VAE and RPiAE-VB at 0.610 (Table 3). Relative to RAE, two-object performance is unchanged and position performance slightly decreases. These results limit the evidence for an overall performance advantage beyond the RAE baseline.

Questions for authors

Is the proposed routing better than simple fusion? Please compare against multi-layer concatenation followed by a linear projection or MLP, retaining the same residual interpolation and training procedure. Report both reconstruction and generation results.

Where do the gains come from? The layer-count sweep increases both input layers and expert capacity. Please add representative single-layer baselines, a capacity-matched last-layer adapter, and the same experts with input-independent learned routing weights to separate the benefits of intermediate features, extra capacity, and adaptive routing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.