acceptodds
Under review as a conference paper at ICLR 2027

Pooling Representation Autoencoders for Efficient Diffusion

12 views

Abstract

Representation autoencoders (RAEs) generate images from pretrained visual features, but their dense token grids make generative modeling expensive. Motivated by local feature correlations, we introduce PoolDINO, a learned affine pooling operator that merges neighboring tokens. Training the pooling operator jointly with the RGB decoder preserves the standard two-stage RAE procedure without a separate feature autoencoder. On ImageNet-256, token compression retains comparable generation quality under internal guidance, while compression trades some quality for greater efficiency. At a fixed budget of 100 sampling steps, latent-sampling throughput increases by and , respectively, relative to the unpooled baseline. Classification and dense prediction evaluations show that comparable guided generation quality can coexist with weaker performance on other tasks.

Primary area:
generative models
Keywords:
representation autoencoders, efficiency, flow matching, diffusion, image generation, generative modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vqtPOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose PoolDINO, an extension work based on existing RAEv2 pipeline. The authors identify the strong local spatial correlation between latent patches from DINO vision encoders. Therefore, they applied different types of pooling layers to spatially compress latents to fewer tokens. It boosts the image generation training and inference speed with certain trade-off with generation quality. It speeds up image generation by 3.7× (4× compression) and 9.0× (16× compression), with comparable guided FID at 4× compression (1.09 vs. 1.08), but degraded quality at 16× (1.44).

Critical strengths
  • The method is simple, with minimal changes to the existing RAEv2 pipeline. A single learned affine pooling layer is trained jointly with the transformer decoder
  • The efficiency gain is significant. At 4× compression, guided FID is comparable to the RAEv2 baseline (1.09 vs. 1.08) with 3.7× higher sampling throughput.
Critical weaknesses
  • The paper lacks one important baseline: changing the DDT patch size from 1 to 2. This also compresses the number of tokens while keeping the RAEv2 reconstruction quality.
  • The paper writing is poor. In the main paper, two full pages are used for curated samples with large white space, while useful contents remain in the appendix.
  • The paper lacks analysis of why the unguided performance drops a lot while the guided performance barely changes. For example, average pooling has much better unguided FID than learned pooling (1.70 vs. 3.00), but worse IG FID (1.29 vs. 1.09).
  • Section 5.3 of the paper seems disconnected from the main story. The classification and dense prediction results do not seem to correlate with the generation performance.
Questions for authors
  • Please refer to the weakness section above.
  • The biggest concern is the lack of comparison to larger patch-size baselines (e.g., DDT with patch size 2 / 4). I would consider raising my score if PoolDINO outperforms this baseline under the same compute with additional analysis into this direction.
3 comments

Thank you for the review and for recognizing the method’s simplicity and efficiency gains. We have completed the main baseline you requested.

Patch-size-2 comparison. We train the unpooled RAEv2 baseline with patch size 2 in both the DDT encoder and prediction head. It processes the same 64 visual tokens as PoolDINO 2×2, using the same transformer widths and depths, optimizer, auxiliary objectives, and 80-epoch training recipe. Both are evaluated with 100 Euler steps and separately tuned IG.

ModelUnguided FID ↓Best IG FID ↓IS at best IG FID ↑
Unpooled RAEv2, patch size 23.82781.9709238.90
PoolDINO, learned 2×2 pooling3.00191.0890250.36

The best IG scales are 1.50 and 1.75, respectively. PoolDINO achieves 44.7% lower best-tested FID and lower FID at every shared tested scale. Analytical generator forward costs, including the IG head, are approximately 132.1 GFLOPs for the patch-size-2 baseline and 129.4 GFLOPs for PoolDINO per image, so the baseline has slightly more compute.

The patch-size-4 experiment is underway. We will try to report its results by tomorrow and will include them in the final version of the manuscript.

Pooling necessarily discards information, so the key question is which information to preserve. PoolDINO learns pooling jointly with the RGB decoder, encouraging it to retain features useful for image reconstruction. In contrast, pooling learned during generator training is optimized to predict DINO patches. The information most useful for this task need not be the information most useful for generating realistic images. Patchification shortens the transformer sequence but retains the full unpooled denoising target, whereas PoolDINO performs diffusion directly on the compressed features, preserving RAE’s two-stage pipeline. This comparison supports learning what to preserve through image reconstruction rather than DINO patch prediction.

Why guidance changes the ranking. Ease of diffusion modeling and image reconstruction quality are distinct. We hypothesize that average pooling simplifies the latent distribution by discarding information, while learned pooling retains more reconstruction-relevant detail that may be harder to model. At 4× compression, learned pooling has better reconstruction FID (0.36 versus 0.65 in Table 1) and guided FID, despite worse unguided FID. IG may mitigate this modeling difficulty without changing the pooling operator or decoder.

Main-paper organization. We thank the reviewer for this suggestion and agree that the space can be used more effectively. We will incorporate the most relevant information from the appendix into the main text.

Purpose of Section 5.3. As stated at the beginning of the section, our evaluation is diagnostic: classification, segmentation, and depth estimation assess what semantic and geometric information remains accessible in the compressed latents learned through RGB reconstruction. These tasks characterize the representation rather than serve as proxies for generation quality. The weaker transfer despite comparable guided generation is therefore an informative property of the learned latents. We will smake its connection to the information preserved by pooling clearer.

Thank you for running the patch-size-2 experiment. It addresses my main concern about the comparison with the simplest baseline. However, it is still unclear why learned pooling benefits more from guidance than average pooling. Therefore, I have raised my score.

We are running an experiment where we take the 2x2 linear projection of the DiT with patch-size-2 and we train a model that tries to recover the original DINO patches from them. Then we will compute the rFID from these reconstructions, and it will be an approximated lower bound on the quality that we can get from these pooled features.

This would separate the numerical error and diffusion model error and focus on the information lost by the DiT pooling.

We expect this rFID to be worse than that of PoolDINO, because there was no signal in the training of the DiT poling that would pressure the optimizer to choose the features that are most useful for image reconstruction.

Reviewer_LymjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

PoolDINO compresses frozen representation-autoencoder features using a shared local affine projection trained jointly with an RGB decoder. A diffusion transformer is subsequently trained in the compressed latent space. The method preserves the two-stage RAE workflow and studies the resulting tradeoffs in reconstruction, generation, sampling efficiency, and representation transfer.

Critical strengths
  1. The central efficiency–quality result is supported by a useful matched comparison. At 80 training epochs and 100 Euler steps, fourfold compression achieves IG FID 1.09 versus 1.08 for the unpooled reference, with a reported 3.7× latent-sampling speedup. Learned pooling also substantially improves reconstruction over average pooling: rFID 0.36 versus 0.65 at 4× compression and 0.41 versus 1.81 at 16×. The decoder architecture and input length are controlled.

  2. The paper reports its tradeoffs clearly. It includes unguided degradation, guidance sweeps, longer training, hardware and batch-size measurements, and classification and dense-prediction results that do not consistently favor learned pooling. These analyses distinguish strong guided generation from strong representation transfer and make the empirical contribution more informative.

Critical weaknesses
  1. The contribution of the auxiliary training objectives remains unresolved. Equation (5) includes both an intermediate clean-latent objective and a dense-feature reconstruction term with weight 0.5. Appendix E varies sampling-time guidance while retaining both training heads. The experiments therefore establish the complete system’s performance, but do not isolate whether dense-feature supervision is particularly important for compressed generation. This is relevant because average pooling performs better unguided at 4× compression, with FID 1.70 versus 3.00, before learned pooling becomes stronger under IG.

  2. Some reproduction details remain implicit. Equation (2) presents velocity MSE, while Equation (5) refers to an x-prediction objective without specifying its exact time weighting. Table 7 does not state the training-time distribution. The evaluation description also needs explicit FID reference statistics and the class-sampling protocol. These details matter for reproducing and interpreting the close guided FIDs.

Questions for authors
  1. How much does the dense-feature reconstruction term contribute at 4× and 16× compression when the internal-guidance head and other settings remain fixed?

  2. Please specify the time-sampling distribution, losses and time weighting for both x-prediction heads, FID reference statistics, and class-sampling protocol. Were these choices identical across learned-pooling, average-pooling, and unpooled controls?

  3. Do existing 16× average-pooling generation results establish whether the advantage of learned pooling persists at stronger compression? The 4× control is particularly informative because its unguided and guided rankings differ.

I look forward to the rebuttal and would be open to raising my score if it resolves the concerns about the auxiliary-loss contribution and the reproducibility details.

Related-work suggestion for the revision: The semantic-retention results in Section 5.3 would benefit from a brief connection to unified visual tokenizers. Please discuss HYDRA and UniTok, and cite them where relevant. HYDRA uses a channel bottleneck within a progressive tokenizer, while UniTok expands discrete capacity through multiple codebooks. Relating these designs to spatial pooling would help explain the different tradeoffs between compression, reconstruction, generation, and semantic information.

1 comment

Thank you for the careful review and for recognizing the controlled comparisons and the transparent reporting of trade-offs.

Contribution of the dense-feature reconstruction loss. We have completed the requested training ablation at 4× compression. We remove only the dense-feature reconstruction term, retaining the IG head, its training objective, and the remaining settings.

Dense-feature loss weightUnguided FID ↓Best IG FID ↓Selected IG scaleIS at selected scale ↑
0.5, original model3.00191.08901.75250.36
0, new ablation3.07361.08642.00251.47

Removing the loss shifts the preferred guidance scale but leaves the best guided performance essentially unchanged. After retuning IG, the ablated model obtains FID 1.0864. Thus, the strong guided result at 4× does not depend on dense-feature supervision. We will extend the same ablation to the remaining experiments in the paper. If these experiments show no benefit from dense-feature supervision, we will remove this component from the method in the revised manuscript.

Time sampling and loss weighting. Thank you for identifying these missing training details. We provide them below and will include them explicitly in the revised manuscript to improve reproducibility. We follow the time-sampling procedure described in RAEv2. In the paper’s noise-to-data convention,

zt=(1−t)ϵ+tzˉ,z_t=(1-t)\epsilon+t\bar z,

the training-time distribution is equivalently

\qquad \eta\sim\mathcal N(0,1), \qquad \kappa=\max\left(1,\sqrt{\frac{ND}{4096}}\right),$$ where $N$ is the number of latent tokens and $D=1024$. This gives κ=8, 4, √8, and 2 for unpooled, 4×, 8×, and 16× compression, respectively. Both the final and intermediate heads predict the clean normalized latent. Each receives the loss $$\mathbb E\!\left[ \frac{\operatorname{MSE}(\hat z,\bar z)} {\max(1-t,0.05)^2} \right],$$ with coefficient 1. This is the velocity-MSE objective implemented through clean-latent predictions, with a clamped denominator. The dense-feature head receives ordinary MSE with coefficient 0.5 and no additional time weighting or cosine term. We will revise Equations 2 and 5 to make their relationship explicit. **Evaluation and consistency across controls.** We use 50,000 generated images, exactly 50 per ImageNet class. Evaluation uses EMA parameters, 100 Euler steps, and BF16 inference. FID uses the RAEv2 `fd_evaluator` backend with Inception-v3-compat features and the precomputed `imagenet_256_fid_stats.npz` reference distributed in `nanovisionx/nanogen-evals-stats`. [Reference statistics](https://huggingface.co/datasets/nanovisionx/nanogen-evals-stats) These choices and the loss definitions are shared across learned pooling, average pooling, and unpooled controls. The time shift follows the same dimensionality-based rule, and normalization statistics are estimated separately for each representation. More precisely, normalization uses training-set means and variances for each token position and channel. Both training stages use deterministic ADM-style center cropping to 256×256 and ImageNet normalization, without random crops or horizontal flips. We will consolidate the essential training and evaluation details in the main text. **Average pooling at 16×.** We did not initially evaluate gFID for 4×4 average pooling because its reconstruction quality was substantially worse: Table 1 reports rFID 1.81, compared with 0.41 for learned pooling. Given our limited compute budget, we prioritized more informative experiments. The weak reconstruction performance of this decoder leads us to expect poor generation quality, although reconstruction results alone do not establish a generation disadvantage. For completeness, we will train a generator with this representation and report its gFID in the final version of the manuscript. **HYDRA and UniTok.** Thank you for these suggestions. As stated in Section 5.3, our evaluation is diagnostic: classification, segmentation, and depth estimation assess what semantic and geometric information remains accessible in the compressed latents learned through RGB reconstruction. [HYDRA](https://arxiv.org/abs/2603.15228) and [UniTok](https://arxiv.org/abs/2502.20321) study representations designed to support both generation and understanding. We will cite them as related work on how representation design and training objectives influence information retention.
Reviewer_pUMHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PoolDINO, which reduces the token count of representation-autoencoder latents for cheaper diffusion. Starting from the RAEv2 latent (multi-layer DINOv3-L/16 features, 16×16×1024), a learned affine operator P\mathcal{P} maps each non-overlapping py×pxp_y\times p_x window to one 1024-d token; this is equivalent to a convolution with kernel = stride = window size. P\mathcal{P} is trained jointly with a ViT-XL RGB decoder (L1 + LPIPS + GAN, with noise augmentation), and pooled tokens are repeated back to 16×16 before decoding so decoder FLOPs stay fixed. A DiT$^{\text{DH}}$-XL is then trained on the pooled grid with x-prediction, an internal-guidance (IG) head, and a dense-feature reconstruction head. On ImageNet-256, 4× compression (64 tokens) reaches IG FID 1.09 vs. 1.08 for the unpooled baseline, and 16× (16 tokens) reaches 1.44, with 3.7× and 9.0× latent-sampling throughput. Longer training at about 77–81% of the baseline’s compute improves these to 1.05 and 1.29. The paper also reports that learned pooling transfers worse than average pooling, PCA, and random projection on ImageNet classification, and does not consistently beat average pooling on ADE20K and NYUv2.

Critical strengths

The paper is well written and esay to follow.

The method is simple with a substantial practical speedup. A single affine layer, with no extra feature autoencoder stage, gives 4× fewer tokens at an essentially unchanged guided FID (1.09 vs. 1.08), and 3.7× latent-sampling throughput at 100 steps.

Critical weaknesses
  1. Limited novelty. Spatially compressing RAE latents for efficient diffusion has already been explored by FlatDINO [R3], and FAE [R4] compresses the channel dimension. The proposed operator is a single convolution with kernel = stride = pp, which is exactly the patch-embedding layer that DiT [R1] uses to shorten token sequences, the standard way to reduce token count on the generator side [R2]. The only change is where this layer lives: it is trained with the RGB decoder in stage 1 and then frozen, instead of being learned by the generator. Avoiding a separate feature-autoencoder stage, compared with [R3, R4], is a useful engineering simplification but not a new idea.

  2. The baseline needed to justify the design is missing. The natural comparison is the unpooled RAEv2 latent [R5] modeled by DiT$^{\text{DH}}$-XL with patch size 2 (and 4), which gives the same 64 (16) tokens at nearly the same generator FLOPs, with no change to stage 1.

[R1] W. Peebles and S. Xie. Scalable Diffusion Models with Transformers. In ICCV, 2023.

[R2] J. Chen, H. Cai, J. Chen, E. Xie, S. Yang, H. Tang, M. Li, Y. Lu, and S. Han. Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models. In ICLR, 2025. arXiv:2410.10733.

[R3] R. Calvo-González and F. Fleuret. Laminating Representation Autoencoders for Efficient Diffusion. arXiv:2602.04873, 2026.

[R4] Y. Gao, C. Chen, T. Chen, and J. Gu. One Layer Is Enough: Adapting Pretrained Visual Encoders for Image Generation. arXiv:2512.07829, 2025.

[R5] J. Singh, B. Zheng, Z. Wu, R. Zhang, E. Shechtman, and S. Xie. Improved Baselines with Representation Autoencoders. arXiv:2605.18324, 2026.

Questions for authors

Can you train DiT$^{\text{DH}}$-XL with x-prediction [R7] on the unpooled RAEv2 latent [R5] using patch size 2 (and 4), with the same recipe, epochs, and guidance sweep, and compare it to learned 2×2 (and 4×4) pooling at matched tokens and FLOPs?

Average pooling gives much better unguided FID than learned pooling (1.70 vs. 3.00) but responds less to IG (1.29 vs. 1.09). Do you have an explanation for this reversal, and what does it imply about why learned pooling helps generation?

Were the guidance scales in Tab. 16 selected on the same 50K samples whose FIDs are reported?

1 comment

Thank you for the thoughtful review and for recognizing the simplicity and practical efficiency of the method. We address your questions below.

Comparison with larger patch sizes. We have completed the requested patch-size-2 baseline on unpooled RAEv2 features. Both models process 64 visual tokens and use the same transformer widths and depths, 80-epoch training recipe, optimizer, auxiliary objectives, and 100-step Euler evaluation. We tune IG separately for each model.

ModelBest tested IG scaleFID ↓IS ↑
Unpooled RAEv2, patch size 21.501.9709238.90
PoolDINO, learned 2×2 pooling1.751.0890250.36

PoolDINO achieves 44.7% lower best-tested FID and lower FID at every shared tested IG scale.

Contribution and relation to prior work. Our central contribution is empirically demonstrating that the feature information preserved during decoder training matters for image generation under a reduced token budget. Diffusability and generation quality are distinct: a representation can be easy to model with diffusion yet fail to decode faithfully into pixel space. Joint training with the RGB decoder directs aggregation toward reconstruction-relevant information, whereas average pooling applies a fixed rule. Like increasing the DiT patch size, our method requires no additional training stage beyond RAE, whereas FlatDINO and FAE introduce an extra stage. Although the affine operator is standard (Section 4.1), our results show that adapting compression to image synthesis preserves strong guided generation while substantially reducing diffusion cost. We will clarify this contribution.

Following your references to DiT and DC-AE, our new baseline directly tests generator-side patchification. The key distinction is the denoising target: larger patches shorten the transformer sequence but retain the full unpooled target, while PoolDINO performs diffusion directly on compressed features. The comparison supports learning compression before diffusion. The operator could also apply to other high-dimensional spatial features, though we have only evaluated it on DINO.

As discussed in Section 2 (Related Work), FlatDINO also reduces token count, but uses a Transformer feature autoencoder rather than learning aggregation through RGB reconstruction. Its learned tokens often become spatially localized, motivating our explicit local structure.

Section 2 also distinguishes FAE, which compresses channels without reducing token count.

Both FlatDINO and FAE require three training stages rather than two; PoolDINO retains RAE's two-stage training pipeline and introduces no extra stage. We will acknowledge FAE's convergence benefits more clearly and soften our description of its computational benefits.

Average pooling and the guidance reversal. We hypothesize a trade-off between diffusability and image reconstruction: other aggregation schemes may be easier to model but discard details needed for faithful decoding. Our representations are explicitly trained for image reconstruction (reconstruction FID 0.36 versus 0.65 for average pooling), but may be less diffusable. IG may mitigate this modeling difficulty, but cannot recover information lost during aggregation.

Guidance-scale selection. Yes. Each candidate scale was evaluated using 50,000 generated images, and the reported FID is the lowest value obtained in that sweep. We did not generate an independent sample set after selecting the scale. We used this procedure consistently across our controls and will state it explicitly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.