- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work aims to improve the capabilities of deep compression autoencoders (DC-AE) and their associated diffusion models. Specifically, the authors propose APEX, a channel-wise asynchronous denoising method designed for extreme compression. APEX deliberately organizes latent channels in a structure-to-detail hierarchy and denoises them asynchronously in sequence. The results demonstrate superior generative performance over existing methods on standard benchmarks under 64× and 128× spatial compression.
Critical strengths
APEX presents a framework for enhancing DC-AE-based diffusion models through resolution-aware autoencoder training and channel-wise asynchronous diffusion. The proposed approach is novel and elegant, and the experimental results provide sufficient evidence for its effectiveness. The paper is relatively easy to follow, and the experimental details are sufficiently thorough to enable experts to reproduce the reported findings.
Critical weaknesses
-
My main concern is that the paper focuses primarily on describing the method's implementation and evaluating its effectiveness through quantitative metrics, while providing limited analysis of the underlying motivation and its effectiveness beyond these metrics. Please see the question 1 below for more details.
-
The experimental evaluation could also be more comprehensive. For example, comparisons of the reconstruction performance of APEX DC-AE itself, and more detailed explanations of the ablation variants would strengthen the paper. Further details are provided in the questions 2-6.
Questions for authors
I have no major concerns regarding the method's implementation or the reported experimental performance. My questions primarily concern the motivation and the analysis of its effectiveness, and the completeness of the experimental evaluation. I will be happy to raise my initial assessment if these issues can be addressed through more in-depth discussion. The questions are listed in descending order of importance.
-
What is the underlying motivation for APEX, and is there an analysis on why it works? More specifically, why organizing latent information and the generation order in the manner adopted by APEX might and do improve the diffusion models operating on the deeply compressed tokens? How APEX differs from the original DC-AE 1.5 (which also has a structured latent) in channel-wise latent information distribution and intermediate diffusion state? Could the authors strengthen the discussion on these parts?
-
Is APEX specifically designed for extreme compression? The paper primarily demonstrates its advantages at compression ratios of 64× and 128×. Does APEX also provide improvements at 32× or 16× compression?
-
Could the authors compare the reconstruction performance of APEX DC-AE with that of other DC-AEs or VAEs? Such a comparison provides a more direct assessment of the impact of the proposed resolution-aware autoencoder training.
-
I spent some time on the specific implementation of each ablation step in Table 2. Could the authors clarify the implementation?
-
Although the paper mentions that uniform staggering gaps are adopted for implementation convenience, I'm wondering whether non-uniform gaps might be more appropriate, and is there any analysis?
-
It would be better if the axes in Figure 1(c) show actual reconstruction and generation metrics.
4 comments
Response to the rebuttal
Thank you for your response, and I believe some of my concerns have been solved (e.g., the motivation, explanation of ablations, and analysis on non-uniform gaps). Nevertheless, I do have several remaining questions (that expect a further discussion) and suggestion.
Remaining questions
- Quantitative analysis on how APEX diffusion functions. All experimental results of APEX at present contain only qualitative visualization and the final generation performance (e.g., gFID). Is it possible to provide additional quantitative analysis or statistical results on the intermediate diffusion state, that help understand how APEX acts differently and more efficiently during the iterative denoising compared to the baseline DC-AEs? I believe this is a key analysis reflecting the core and actual contribution of APEX, and would strengthen the overall claims beyond simple visualization.
- Compare reconstruction performance of APEX DC-AE with that of other DC-AEs or VAEs. Thank you for the further clarification on Table 1. However, it seems that Table 1 does not perform direct reconstruction comparison between DC-AEs and APEX under the same model configuration. Could you provide direct reconstruction metrics (e.g., PSNR, SSIM, LPIPS, rFID) of APEX and DC-AE at f64c128 and f128c512? These results would explicitly tell the reconstruction-generation tradeoff of APEX and strengthen the experiments.
Suggestion for the manuscript
- If possible, please include a comparison of APEX and DC-AE at f32c32, or even f16, in the future. This would display the limit of APEX beyond extreme compression.
- Swap axes in Figure 1(c). My initial expectation is to provide direct quantitative values of metrics (e.g., rFID, gFID). In current version, Figure 1(c) still can not show the explicit improvement of APEX compared to a flat latent.
Response to Reviewer xdX7
Thank you for the follow-up! We ran additional experiments to address your remaining questions and updated Figure 1(c) as requested.
1. Quantitative analysis of intermediate denoising
We conducted a new experiment comparing how APEX and DC-AE generate different image frequencies throughout inference, and the results were quite interesting! Due to the current page limit, we included it in Appendix A, but we plan to move it to the main body in the final version.
For both methods, we generated 1,024 images and measured the frequency-wise error of their intermediate clean-image predictions relative to the final generated images. We report both absolute error and error normalized by the final image's energy in each frequency band. The results are shown in Appendix A.5 and Fig. 15.
We observed two interesting behaviors:
-
APEX is much more stable when generating high frequencies. DC-AE's high-frequency error initially increases before decreasing, suggesting that its predictions are being revised throughout inference. In contrast, APEX's error decreases steadily and remains substantially lower throughout most of sampling. This difference holds even after normalizing by each model's final high-frequency energy.
-
Different frequencies converge at different stages. In APEX, low-frequency content converges earlier, while higher frequencies continue to be refined later. This is consistent with the order in which APEX denoises its levels, where fine details are generated using cleaner structural information from earlier levels.
Importantly, these are two models trained with matched compute and evaluated at the same inference budget, and APEX also achieves better final gFID (2.85 vs 3.22). The new analysis provides a quantitative view of how their generation processes differ and helps explain why APEX is effective.
2. Reconstruction performance
We added a direct comparison of the reconstruction metrics in Appendix Table 6, including PSNR, SSIM, LPIPS, and rFID.
| Compression | Method | rFID ↓ | PSNR ↑ | SSIM ↑ | LPIPS ↓ | gFID ↓ (CFG 1.5) |
|---|---|---|---|---|---|---|
| f64c128 | APEX | 0.22 | 26.14 | 0.69 | 0.082 | 2.85 |
| f64c128 | DC-AE | 0.22 | 26.15 | 0.71 | 0.080 | 3.22 |
| f128c512 | APEX | 0.23 | 25.68 | 0.70 | 0.085 | 8.98 |
| f128c512 | DC-AE | 0.23 | 25.73 | 0.70 | 0.084 | 14.44 |
The reconstruction quality is very similar, while APEX achieves much better generation quality. This further supports our main claim that APEX improves generation without sacrificing reconstruction.
Suggestion 1: Evaluation at f32 and f16
We agree that evaluating APEX at lower spatial compression would be useful for understanding where its advantages are most significant. Our focus in this work is the extreme-compression regime, and we leave a broader evaluation at f32 and f16 for future work.
Suggestion 2: Figure 1(c)
We updated Figure 1(c) to show the actual measured rFID and gFID values of APEX and DC-AE across the f64 configurations. The figure now directly shows the improvement in the reconstruction-generation tradeoff.
To summarize, we have addressed the points from your original review and follow-up in the revised manuscript:
- Motivation and comparison to DC-AE 1.5
- Ablation details App. F.11
- Analysis of non-uniform gaps
- Quantitative analysis of intermediate denoising: Appendix A.5 and Fig. 15
- Direct reconstruction metrics: Appendix Table 6
- Figure 1(c): measured rFID and gFID values
Your initial review noted that you would be happy to raise your assessment if these issues were addressed. Given the new results and revisions, we'd really appreciate it if you could update your score.
As the discussion period closes in under 48 hours, if any concern remains, please let us know as soon as possible, and we will do our best to clarify it in the remaining time.
Thank you for your follow-up, and the additional experiments during the remaining discussion time. My major concern regarding the analysis on why APEX works and how it functions has been resolved, and I appreciate the revision presented in the latest manuscript. I would recommend a future exploration on the limit of APEX under shallow compression (e.g., f32 and f16), as discussed in the response, and a deeper investigation on the non-uniform gap, as mentioned in the manuscript. I will update my score. Good luck.
Response to Reviewer xdX7
Thank you, we're glad you found the work interesting! We expand on the motivation below.
What is the underlying motivation of APEX?
To make generation in high-compression settings easier by separating two sources of uncertainty: image structure and fine details.
Where does the uncertainty come from?
To decrease token count, we increase spatial compression. Each latent token encodes a larger image region. To maintain reconstruction quality, we increase the number of channels. This allows the VAE to encode patch-specific details, like exact grass texture. Each token then contains an "entangled" representation of structure and detail (most added capacity goes to the latter).
Why does this make diffusion harder?
Given a noisy latent xt, the optimal clean prediction under MSE loss is E[x0∣xt]. During training, we don't know this expectation and instead supervise the model using individual clean samples. When the latent encodes both structure and instance-specific details, uncertainty over possible clean outputs is higher, making prediction harder.
How does APEX help?
We separate these two sources of uncertainty and therefore reduce the difficulty of prediction throughout the denoising process. On the AE side, we organize channels from coarse structure to fine detail. During generation, denoising starts only on the first (coarsest) level: the model is only asked to predict this information (in effect, it operates in a simplified space). The hard task of predicting fine detail is only given to the model once earlier levels are cleaner, again reducing uncertainty.
How does APEX differ from DC-AE 1.5?
There are two main differences: how we organize information across channel levels, and how we denoise them.
On the AE side, DC-AE 1.5 trains its levels to reconstruct the full-resolution image, using small increments of 4 channels per level. APEX instead matches each level to a progressively higher target resolution, similar to an image pyramid. We also use much larger levels (e.g., 8 -> 32 -> 128 channels), allowing us to separate structure and detail into a small number of meaningful stages. This is intentional: too many small levels constrain the AE's representation and reduce the overlap between denoising windows, making sampling less efficient. Please see Figure 7 for a qualitative demonstration of the benefit of resolution-aware training vs. DC-AE 1.5.
More importantly, DC-AE 1.5 does not change the diffusion sampling process: all channels are denoised synchronously, at the same noise level. APEX instead denoises different channel groups at staggered local times, allowing later levels to predict fine details using cleaner structural information from earlier levels.
Is APEX specifically designed for extreme compression?
Yes. APEX is designed for the regime where generation quality is the bottleneck, not reconstruction quality. As one increases spatial compression, one needs to add more channels to maintain reconstruction quality. While reconstruction quality does not necessarily degrade, generation quality degrades catastrophically, and that's the problem that APEX addresses.
At lower spatial compression, like 32x and 16x, the same reconstruction quality can be achieved with fewer channels. Our f64c128 model has a compression ratio of 96:1 and requires 128 channels. At 32x only 32 channels are needed, and at 16x only 8. At these widths, the channel-scaling problem is less severe because each token encodes less sample-specific high-frequency information, so the motivation for APEX is weaker.
Compare APEX to DC-AE.
Table 1 reports reconstruction quality. Following DC-AE 1.5, which showed that a channel hierarchy can preserve reconstruction quality, we match all methods to the same rFID at each latent configuration. This ensures that generation gains do not sacrifice reconstruction quality. For example, the matched rFID is 0.20 at f128c1024 and 0.22 at f64c128. We now highlight this comparison in the revised manuscript. Appendix A.3 shows example reconstructions.
Explanation of Ablations:
Appendix F.11 provides implementation details, and we have substantially expanded it for Table 2. Please let us know if anything remains unclear.
Could non-uniform gaps improve performance?
Yes, likely. Larger gaps provide cleaner structural context for later levels, but reduce the overlap between their denoising windows, making training less efficient. There is no reason to believe that a single gap is optimal for all levels.
However, we found that a uniform gap of g=0.25, selected based on Figure 8(a), already yields strong results without architecture-tuning. We leave the exploration of non-uniform gaps for future work.
Swap axes in Figure 1(c).
Certainly. We updated the figure.
We hope we've answered your questions! Please let us know if there's anything else you'd like us to explain. We'd be very happy if you could consider raising your score.