- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose PoolDINO, an extension work based on existing RAEv2 pipeline. The authors identify the strong local spatial correlation between latent patches from DINO vision encoders. Therefore, they applied different types of pooling layers to spatially compress latents to fewer tokens. It boosts the image generation training and inference speed with certain trade-off with generation quality. It speeds up image generation by 3.7× (4× compression) and 9.0× (16× compression), with comparable guided FID at 4× compression (1.09 vs. 1.08), but degraded quality at 16× (1.44).
Critical strengths
- The method is simple, with minimal changes to the existing RAEv2 pipeline. A single learned affine pooling layer is trained jointly with the transformer decoder
- The efficiency gain is significant. At 4× compression, guided FID is comparable to the RAEv2 baseline (1.09 vs. 1.08) with 3.7× higher sampling throughput.
Critical weaknesses
- The paper lacks one important baseline: changing the DDT patch size from 1 to 2. This also compresses the number of tokens while keeping the RAEv2 reconstruction quality.
- The paper writing is poor. In the main paper, two full pages are used for curated samples with large white space, while useful contents remain in the appendix.
- The paper lacks analysis of why the unguided performance drops a lot while the guided performance barely changes. For example, average pooling has much better unguided FID than learned pooling (1.70 vs. 3.00), but worse IG FID (1.29 vs. 1.09).
- Section 5.3 of the paper seems disconnected from the main story. The classification and dense prediction results do not seem to correlate with the generation performance.
Questions for authors
- Please refer to the weakness section above.
- The biggest concern is the lack of comparison to larger patch-size baselines (e.g., DDT with patch size 2 / 4). I would consider raising my score if PoolDINO outperforms this baseline under the same compute with additional analysis into this direction.
3 comments
Thank you for running the patch-size-2 experiment. It addresses my main concern about the comparison with the simplest baseline. However, it is still unclear why learned pooling benefits more from guidance than average pooling. Therefore, I have raised my score.
We are running an experiment where we take the 2x2 linear projection of the DiT with patch-size-2 and we train a model that tries to recover the original DINO patches from them. Then we will compute the rFID from these reconstructions, and it will be an approximated lower bound on the quality that we can get from these pooled features.
This would separate the numerical error and diffusion model error and focus on the information lost by the DiT pooling.
We expect this rFID to be worse than that of PoolDINO, because there was no signal in the training of the DiT poling that would pressure the optimizer to choose the features that are most useful for image reconstruction.
Thank you for the review and for recognizing the method’s simplicity and efficiency gains. We have completed the main baseline you requested.
Patch-size-2 comparison. We train the unpooled RAEv2 baseline with patch size 2 in both the DDT encoder and prediction head. It processes the same 64 visual tokens as PoolDINO 2×2, using the same transformer widths and depths, optimizer, auxiliary objectives, and 80-epoch training recipe. Both are evaluated with 100 Euler steps and separately tuned IG.
The best IG scales are 1.50 and 1.75, respectively. PoolDINO achieves 44.7% lower best-tested FID and lower FID at every shared tested scale. Analytical generator forward costs, including the IG head, are approximately 132.1 GFLOPs for the patch-size-2 baseline and 129.4 GFLOPs for PoolDINO per image, so the baseline has slightly more compute.
The patch-size-4 experiment is underway. We will try to report its results by tomorrow and will include them in the final version of the manuscript.
Pooling necessarily discards information, so the key question is which information to preserve. PoolDINO learns pooling jointly with the RGB decoder, encouraging it to retain features useful for image reconstruction. In contrast, pooling learned during generator training is optimized to predict DINO patches. The information most useful for this task need not be the information most useful for generating realistic images. Patchification shortens the transformer sequence but retains the full unpooled denoising target, whereas PoolDINO performs diffusion directly on the compressed features, preserving RAE’s two-stage pipeline. This comparison supports learning what to preserve through image reconstruction rather than DINO patch prediction.
Why guidance changes the ranking. Ease of diffusion modeling and image reconstruction quality are distinct. We hypothesize that average pooling simplifies the latent distribution by discarding information, while learned pooling retains more reconstruction-relevant detail that may be harder to model. At 4× compression, learned pooling has better reconstruction FID (0.36 versus 0.65 in Table 1) and guided FID, despite worse unguided FID. IG may mitigate this modeling difficulty without changing the pooling operator or decoder.
Main-paper organization. We thank the reviewer for this suggestion and agree that the space can be used more effectively. We will incorporate the most relevant information from the appendix into the main text.
Purpose of Section 5.3. As stated at the beginning of the section, our evaluation is diagnostic: classification, segmentation, and depth estimation assess what semantic and geometric information remains accessible in the compressed latents learned through RGB reconstruction. These tasks characterize the representation rather than serve as proxies for generation quality. The weaker transfer despite comparable guided generation is therefore an informative property of the learned latents. We will smake its connection to the information preserved by pooling clearer.