acceptodds
Under review as a conference paper at ICLR 2027

APEX: Asynchronous Prefix Denoising for Extreme Compression

Abstract

Fewer latent tokens promise cheaper diffusion. Yet aggressive spatial compression creates a reconstruction-generation dilemma: adding channels improves reconstruction but makes generation harder. Our goal is to make extreme spatial compression competitive by shifting this frontier. We introduce APEX, short for Asynchronous Prefix denoising for EXtreme compression, which pairs a hierarchical latent space with asynchronous denoising over channel groups. Our autoencoder orders channel prefixes from structure to detail, while one shared denoiser generates them at staggered local times without adding tokens. We show that naively combining a channel hierarchy with asynchronous denoising can make generation worse. APEX succeeds by matching what each level represents, when it denoises, and how training is allocated across the trajectory. On ImageNet-512, APEX outperforms existing methods at every evaluated channel count under 64x and 128x spatial compression, substantially improving this frontier. At matched training and guided sampling compute, it also outperforms a 256-token model at every evaluated budget, with larger gains at lower compute. APEX makes extreme spatial compression practical for compute-efficient latent diffusion.

Primary area:
generative models
Keywords:
generative models, diffusion models, latent diffusion, flow matching, autoencoders, image generation, transformers, computational efficiency, deep compression

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xdX7OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work aims to improve the capabilities of deep compression autoencoders (DC-AE) and their associated diffusion models. Specifically, the authors propose APEX, a channel-wise asynchronous denoising method designed for extreme compression. APEX deliberately organizes latent channels in a structure-to-detail hierarchy and denoises them asynchronously in sequence. The results demonstrate superior generative performance over existing methods on standard benchmarks under 64× and 128× spatial compression.

Critical strengths

APEX presents a framework for enhancing DC-AE-based diffusion models through resolution-aware autoencoder training and channel-wise asynchronous diffusion. The proposed approach is novel and elegant, and the experimental results provide sufficient evidence for its effectiveness. The paper is relatively easy to follow, and the experimental details are sufficiently thorough to enable experts to reproduce the reported findings.

Critical weaknesses
  1. My main concern is that the paper focuses primarily on describing the method's implementation and evaluating its effectiveness through quantitative metrics, while providing limited analysis of the underlying motivation and its effectiveness beyond these metrics. Please see the question 1 below for more details.

  2. The experimental evaluation could also be more comprehensive. For example, comparisons of the reconstruction performance of APEX DC-AE itself, and more detailed explanations of the ablation variants would strengthen the paper. Further details are provided in the questions 2-6.

Questions for authors

I have no major concerns regarding the method's implementation or the reported experimental performance. My questions primarily concern the motivation and the analysis of its effectiveness, and the completeness of the experimental evaluation. I will be happy to raise my initial assessment if these issues can be addressed through more in-depth discussion. The questions are listed in descending order of importance.

  1. What is the underlying motivation for APEX, and is there an analysis on why it works? More specifically, why organizing latent information and the generation order in the manner adopted by APEX might and do improve the diffusion models operating on the deeply compressed tokens? How APEX differs from the original DC-AE 1.5 (which also has a structured latent) in channel-wise latent information distribution and intermediate diffusion state? Could the authors strengthen the discussion on these parts?

  2. Is APEX specifically designed for extreme compression? The paper primarily demonstrates its advantages at compression ratios of 64× and 128×. Does APEX also provide improvements at 32× or 16× compression?

  3. Could the authors compare the reconstruction performance of APEX DC-AE with that of other DC-AEs or VAEs? Such a comparison provides a more direct assessment of the impact of the proposed resolution-aware autoencoder training.

  4. I spent some time on the specific implementation of each ablation step in Table 2. Could the authors clarify the implementation?

  5. Although the paper mentions that uniform staggering gaps are adopted for implementation convenience, I'm wondering whether non-uniform gaps might be more appropriate, and is there any analysis?

  6. It would be better if the axes in Figure 1(c) show actual reconstruction and generation metrics.

4 comments
AuthorsOpenReview ↗

Response to Reviewer xdX7

Thank you, we're glad you found the work interesting! We expand on the motivation below.

What is the underlying motivation of APEX?

To make generation in high-compression settings easier by separating two sources of uncertainty: image structure and fine details.

Where does the uncertainty come from?

To decrease token count, we increase spatial compression. Each latent token encodes a larger image region. To maintain reconstruction quality, we increase the number of channels. This allows the VAE to encode patch-specific details, like exact grass texture. Each token then contains an "entangled" representation of structure and detail (most added capacity goes to the latter).

Why does this make diffusion harder?

Given a noisy latent xtx_t, the optimal clean prediction under MSE loss is E[x0∣xt]E[x_0 \mid x_t]. During training, we don't know this expectation and instead supervise the model using individual clean samples. When the latent encodes both structure and instance-specific details, uncertainty over possible clean outputs is higher, making prediction harder.

How does APEX help?

We separate these two sources of uncertainty and therefore reduce the difficulty of prediction throughout the denoising process. On the AE side, we organize channels from coarse structure to fine detail. During generation, denoising starts only on the first (coarsest) level: the model is only asked to predict this information (in effect, it operates in a simplified space). The hard task of predicting fine detail is only given to the model once earlier levels are cleaner, again reducing uncertainty.

How does APEX differ from DC-AE 1.5?

There are two main differences: how we organize information across channel levels, and how we denoise them.

On the AE side, DC-AE 1.5 trains its levels to reconstruct the full-resolution image, using small increments of 4 channels per level. APEX instead matches each level to a progressively higher target resolution, similar to an image pyramid. We also use much larger levels (e.g., 8 -> 32 -> 128 channels), allowing us to separate structure and detail into a small number of meaningful stages. This is intentional: too many small levels constrain the AE's representation and reduce the overlap between denoising windows, making sampling less efficient. Please see Figure 7 for a qualitative demonstration of the benefit of resolution-aware training vs. DC-AE 1.5.

More importantly, DC-AE 1.5 does not change the diffusion sampling process: all channels are denoised synchronously, at the same noise level. APEX instead denoises different channel groups at staggered local times, allowing later levels to predict fine details using cleaner structural information from earlier levels.

Is APEX specifically designed for extreme compression?

Yes. APEX is designed for the regime where generation quality is the bottleneck, not reconstruction quality. As one increases spatial compression, one needs to add more channels to maintain reconstruction quality. While reconstruction quality does not necessarily degrade, generation quality degrades catastrophically, and that's the problem that APEX addresses.

At lower spatial compression, like 32x and 16x, the same reconstruction quality can be achieved with fewer channels. Our f64c128 model has a compression ratio of 96:1 and requires 128 channels. At 32x only 32 channels are needed, and at 16x only 8. At these widths, the channel-scaling problem is less severe because each token encodes less sample-specific high-frequency information, so the motivation for APEX is weaker.

Compare APEX to DC-AE.

Table 1 reports reconstruction quality. Following DC-AE 1.5, which showed that a channel hierarchy can preserve reconstruction quality, we match all methods to the same rFID at each latent configuration. This ensures that generation gains do not sacrifice reconstruction quality. For example, the matched rFID is 0.20 at f128c1024 and 0.22 at f64c128. We now highlight this comparison in the revised manuscript. Appendix A.3 shows example reconstructions.

Explanation of Ablations:

Appendix F.11 provides implementation details, and we have substantially expanded it for Table 2. Please let us know if anything remains unclear.

Could non-uniform gaps improve performance?

Yes, likely. Larger gaps provide cleaner structural context for later levels, but reduce the overlap between their denoising windows, making training less efficient. There is no reason to believe that a single gap is optimal for all levels.

However, we found that a uniform gap of g=0.25g=0.25, selected based on Figure 8(a), already yields strong results without architecture-tuning. We leave the exploration of non-uniform gaps for future work.

Swap axes in Figure 1(c).

Certainly. We updated the figure.

We hope we've answered your questions! Please let us know if there's anything else you'd like us to explain. We'd be very happy if you could consider raising your score.

Reviewer_xdX7OpenReview ↗

Response to the rebuttal

Thank you for your response, and I believe some of my concerns have been solved (e.g., the motivation, explanation of ablations, and analysis on non-uniform gaps). Nevertheless, I do have several remaining questions (that expect a further discussion) and suggestion.

Remaining questions

  1. Quantitative analysis on how APEX diffusion functions. All experimental results of APEX at present contain only qualitative visualization and the final generation performance (e.g., gFID). Is it possible to provide additional quantitative analysis or statistical results on the intermediate diffusion state, that help understand how APEX acts differently and more efficiently during the iterative denoising compared to the baseline DC-AEs? I believe this is a key analysis reflecting the core and actual contribution of APEX, and would strengthen the overall claims beyond simple visualization.
  2. Compare reconstruction performance of APEX DC-AE with that of other DC-AEs or VAEs. Thank you for the further clarification on Table 1. However, it seems that Table 1 does not perform direct reconstruction comparison between DC-AEs and APEX under the same model configuration. Could you provide direct reconstruction metrics (e.g., PSNR, SSIM, LPIPS, rFID) of APEX and DC-AE at f64c128 and f128c512? These results would explicitly tell the reconstruction-generation tradeoff of APEX and strengthen the experiments.

Suggestion for the manuscript

  1. If possible, please include a comparison of APEX and DC-AE at f32c32, or even f16, in the future. This would display the limit of APEX beyond extreme compression.
  2. Swap axes in Figure 1(c). My initial expectation is to provide direct quantitative values of metrics (e.g., rFID, gFID). In current version, Figure 1(c) still can not show the explicit improvement of APEX compared to a flat latent.
AuthorsOpenReview ↗

Response to Reviewer xdX7

Thank you for the follow-up! We ran additional experiments to address your remaining questions and updated Figure 1(c) as requested.

1. Quantitative analysis of intermediate denoising

We conducted a new experiment comparing how APEX and DC-AE generate different image frequencies throughout inference, and the results were quite interesting! Due to the current page limit, we included it in Appendix A, but we plan to move it to the main body in the final version.

For both methods, we generated 1,024 images and measured the frequency-wise error of their intermediate clean-image predictions relative to the final generated images. We report both absolute error and error normalized by the final image's energy in each frequency band. The results are shown in Appendix A.5 and Fig. 15.

We observed two interesting behaviors:

  1. APEX is much more stable when generating high frequencies. DC-AE's high-frequency error initially increases before decreasing, suggesting that its predictions are being revised throughout inference. In contrast, APEX's error decreases steadily and remains substantially lower throughout most of sampling. This difference holds even after normalizing by each model's final high-frequency energy.

  2. Different frequencies converge at different stages. In APEX, low-frequency content converges earlier, while higher frequencies continue to be refined later. This is consistent with the order in which APEX denoises its levels, where fine details are generated using cleaner structural information from earlier levels.

Importantly, these are two models trained with matched compute and evaluated at the same inference budget, and APEX also achieves better final gFID (2.85 vs 3.22). The new analysis provides a quantitative view of how their generation processes differ and helps explain why APEX is effective.

2. Reconstruction performance

We added a direct comparison of the reconstruction metrics in Appendix Table 6, including PSNR, SSIM, LPIPS, and rFID.

CompressionMethodrFID ↓PSNR ↑SSIM ↑LPIPS ↓gFID ↓ (CFG 1.5)
f64c128APEX0.2226.140.690.0822.85
f64c128DC-AE0.2226.150.710.0803.22
f128c512APEX0.2325.680.700.0858.98
f128c512DC-AE0.2325.730.700.08414.44

The reconstruction quality is very similar, while APEX achieves much better generation quality. This further supports our main claim that APEX improves generation without sacrificing reconstruction.

Suggestion 1: Evaluation at f32 and f16

We agree that evaluating APEX at lower spatial compression would be useful for understanding where its advantages are most significant. Our focus in this work is the extreme-compression regime, and we leave a broader evaluation at f32 and f16 for future work.

Suggestion 2: Figure 1(c)

We updated Figure 1(c) to show the actual measured rFID and gFID values of APEX and DC-AE across the f64 configurations. The figure now directly shows the improvement in the reconstruction-generation tradeoff.

To summarize, we have addressed the points from your original review and follow-up in the revised manuscript:

  • Motivation and comparison to DC-AE 1.5
  • Ablation details App. F.11
  • Analysis of non-uniform gaps
  • Quantitative analysis of intermediate denoising: Appendix A.5 and Fig. 15
  • Direct reconstruction metrics: Appendix Table 6
  • Figure 1(c): measured rFID and gFID values

Your initial review noted that you would be happy to raise your assessment if these issues were addressed. Given the new results and revisions, we'd really appreciate it if you could update your score.

As the discussion period closes in under 48 hours, if any concern remains, please let us know as soon as possible, and we will do our best to clarify it in the remaining time.

Thank you for your follow-up, and the additional experiments during the remaining discussion time. My major concern regarding the analysis on why APEX works and how it functions has been resolved, and I appreciate the revision presented in the latest manuscript. I would recommend a future exploration on the limit of APEX under shallow compression (e.g., f32 and f16), as discussed in the response, and a deeper investigation on the non-uniform gap, as mentioned in the manuscript. I will update my score. Good luck.

Reviewer_nRc2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets the reconstruction–generation dilemma in latent diffusion under extreme spatial compression (64 x/128 x), where adding latent channels improves reconstruction but makes generation harder. It proposes APEX, which (i) trains an autoencoder whose cumulative channel prefixes are supervised at progressively higher target resolutions, so they are ordered from structure to detail, and (ii) uses one shared Transformer that denoises the channel groups at staggered local times on a global clock, with level-balanced losses and a structure-heavy clock-sampling density, without adding tokens.

Critical strengths

Across five latent geometries, APEX degrades far more slowly with width than all baselines at matched training FLOPs and NFE, and the same ordering holds unguided.

The method is simple and adds no tokens, no external encoders and no per-step cost, since a single network evaluation updates all channel levels with one shared Euler step. This makes it easy to adopt on top of existing DC-AE and SiT pipelines.

APEX is compared with a compute-matched 256-token model, with guidance tuned separately at every inference budget. The flat 64-token model falls behind the 256-token model once sampling compute grows, and APEX removes this crossover.

Critical weaknesses

Guidance scale is reselected at each budget on the same 50K-sample evaluation. Meanwhile, the margin over the 256-token model at two and four times the reference budget is only about 0.1 and 0.04 gFID, and several ablation gaps are between 0.4 and 0.65. The claim of superiority at every evaluated budget and the ranking of design choices may therefore fall within run-to-run and selection noise.

The main channel-scaling comparison also fixes the guidance scale at 1.5 for every method, even though the paper’s own sweep shows that a change of 0.1 can shift gFID by about 0.4. In addition, the conceptually closest methods, such as Denoising-VAE and the asynchronous schedules of SFD and Learning When to Denoise, are discussed but not compared. Where the very large baseline gaps at wide latents come from is therefore partly uncertain

The 64-token model is shown to beat a compute-matched 256-token model at only a single, small training budget of about 1e20 FLOPs. At that budget the 256-token model receives only 250K steps and is likely not converged, and the margin already shrinks to about 0.04 gFID at four times the reference inference budget. The advantage may therefore vanish or reverse at larger budgets, and a curve over several training budgets would be needed to generalize it. Moreover, the larger gains at low inference compute arise because the 256-token model must sample with only 16 to 32 steps. This reflects the few-step sampling regime rather than a general benefit of compression, and it could change with distillation or better samplers.

The channel-scaling comparison covers only widths at or above the default DC-AE setting, where APEX is expected to help most. It uses three widths at 64-fold compression and two at 128-fold, so superiority at every channel count and a shifted reconstruction and generation frontier are never tested on narrower latents or against each method’s best width.

The qualitative evidence consists of only a few APEX samples from 64-token models. There are no side-by-side comparisons with baselines, no samples from the headline 16-token model or the 256-token comparison. The large gFID gaps therefore cannot be checked visually, and neither can the claimed reduction of high-frequency leakage

Questions for authors

Does APEX at 64 tokens still beat the 256-token model when both are trained with a larger budget, for example two to four times the current one?

What are the rFID and gFID of all methods at narrower widths, such as 32 or 64 channels at 64-fold compression and 128 or 256 channels at 128-fold compression?

4 comments
AuthorsOpenReview ↗

Response to Reviewer nRc2

Thank you for your detailed review. We ran additional experiments to investigate your concerns, and the results are clear: APEX's advantage persists even after optimizing CFG separately for each method and evaluating FID across 10 random seeds.

Statistical Robustness and CFG Selection

We agree that some of the reported FID differences are small, and that using a fixed CFG of 1.5 may not be optimal for every method. Therefore, we conducted additional CFG sweeps, evaluated across 10 random seeds, and computed 95% confidence intervals.

Figure 5 - Channel Scaling:

We focused on f64c128, where the FID differences between methods are smallest. We swept CFG separately for APEX, DC-AE, and DC-AE 1.5 (the strongest baseline).

MethodBest CFG valuegFID (best CFG, mean ± std.)95% CIgFID (w/o CFG)95% CI
APEX1.42.74 ± 0.033[2.72, 2.77]10.90 ± 0.075[10.84, 10.95]
DC-AE1.43.21 ± 0.042[3.18, 3.24]13.35 ± 0.073[13.29, 13.40]
DC-AE 1.51.43.16 ± 0.059[3.12, 3.20]13.25 ± 0.061[13.21, 13.29]

Even after optimizing CFG separately for each method, APEX maintains its advantage over both baselines. The confidence intervals verify that this advantage is not simply due to sampling variance.

Figure 6 - Generation Quality at Matched Compute:

We repeated the evaluations at 2x and 4x inference compute across 10 random seeds and computed 95% confidence intervals.

Inference computeMethodgFID (mean ± std.)95% CI
2xAPEX f64c128 (64 tokens)2.72 ± 0.027[2.70, 2.73]
2xDC-AE f32c32 (256 tokens)2.83 ± 0.036[2.81, 2.86]
4xAPEX f64c128 (64 tokens)2.71 ± 0.038[2.69, 2.74]
4xDC-AE f32c32 (256 tokens)2.79 ± 0.039[2.76, 2.82]

Again, APEX's advantage is stable and statistically significant.

Both 10-seed analyses are now reported in Appendix E.3 of the revised manuscript.

Qualitative comparisons: Appendix Figure 10 compares APEX with DC-AE and DC-AE 1.5 at 16 tokens. Figure 14 compares 64-token APEX with 256-token DC-AE. Figure 7 illustrates the reduction in high-freq. leakage.

What happens with a larger training budget?

Increasing training budget could change the performance of the models. However, our contribution is creating a window in which extreme compression is not only cheaper, but actually superior in terms of generation quality. Normally, fewer tokens make diffusion substantially cheaper, but generation quality deteriorates so much that the computational savings are not worth it. APEX changes this. As shown in Fig 6, our 64-token model outperforms the 256-token baseline with 4x fewer tokens, including at large inference budgets, e.g. when the 256-token model uses 250 steps.

We do not claim that APEX with fewer tokens will always outperform a model with more tokens at large compute budgets. What we demonstrate is that APEX creates a regime where you actually want to use fewer tokens, not just to save compute, but also to get better generation.

Comparison with Related Work

SFD and Learning When to Denoise solve a different problem: how to leverage DINOv2 to accelerate diffusion. Both combine a DINO embedding with a separate texture latent, and denoise them asynchronously. While we share the idea of async denoising, our focus is not on how to integrate pretrained embeddings, but to show that by structuring AE latents and async denoising them, we can make otherwise unusable latent spaces practical.

Denoising-VAE belongs to the same family of spectral regularization approaches as Spectrum Matching, which we already include as a baseline in Fig 5. Both methods improve latent diffusibility by modifying the frequencies in the latent space. APEX takes a different approach: rather than regularizing the latent spectrum, we create a structure that we know how to exploit in inference. This allows us to preserve fine details in the latent rather than suppressing them to make diffusion easier.

Why not evaluate fewer channels?

Our results show that APEX improves the reconstruction-generation tradeoff even across different channel counts. APEX f64c512 has better reconstruction (rFID 0.13 vs. 0.18) and generation (gFID 4.90 vs. 7.11) than DC-AE 1.5 f64c256, despite using twice as many channels.

At 64x compression, 32 or 64 channels yield compression ratios of 384:1 and 192:1 (likewise for 128/256 channels at 128x). In our experiments, the DCAE autoencoder could not reconstruct the image with reasonable quality at these ratios. Therefore, generation is not the problem, but rather reconstruction. APEX targets the regime where generation quality is the bottleneck, not reconstruction quality.

We hope we've addressed your main concerns with the additional experiments. If anything remains unresolved, please let us know. In light of these results, we would appreciate it if you could reconsider your score.

AuthorsOpenReview ↗

Summary of Rebuttal

Before the discussion closes, we’d like to highlight the additional experiments included in the revised manuscript in response to your concerns:

  • Statistical robustness (Appendix E.3): We evaluated the main comparisons across 10 random seeds and computed 95% confidence intervals. APEX maintains its advantage, including at 2× and 4× inference compute, where the differences are smallest.
  • CFG optimization (Appendix E.3): We independently optimized CFG for APEX, DC-AE, and DC-AE 1.5. APEX continues to outperform both baselines, confirming that its advantage is not due to using a fixed guidance scale.
  • Qualitative comparisons (Figs. 10 and 14): We added side-by-side generations against the baselines at 16 tokens and the compute-matched 64-token vs. 256-token comparison, as well as reconstruction visualizations for all autoencoders.
  • Intermediate denoising analysis (Appendix A.5, Fig. 15): We added a quantitative frequency-wise analysis of 1,024 generation trajectories, showing that APEX produces more stable high-frequency predictions than DCAE throughout most of sampling.

We hope these new experiments help resolve your concerns and provide stronger support for our conclusions. We’d be grateful if you could take them into account when reconsidering your assessment!

Thanks for the extra experiments, they resolve my concerns on variance and qualitative results.

But the evaluation still feels selective. Results are only shown for one small training budget, default or wider channels, and without the closest baselines. The claim that narrower widths can’t reconstruct images also has no numbers or examples behind it.

The rebuttal calls Denoising-VAE a spectral regularization method, but the paper’s related work describes it as low-pass latent levels with frequency-aligned diffusion, basically the closest design to APEX. These don’t match.

I also agree with Reviewer sDCq that the novelty is mostly in combining existing ideas, and that the training allocation for deeper hierarchies was reused from the three-level setup without validation.

I’ll keep my score.

AuthorsOpenReview ↗

Response to Reviewer nRc2

Thank you for considering the additional experiments and acknowledging that they resolve your concerns regarding statistical robustness and qualitative comparisons. We respect your assessment and would like to clarify a few remaining points.

Training budget: Our compute-matched comparison deliberately targets a limited training budget, where extreme compression should offer the greatest efficiency advantage. Yet conventional methods suffer such severe generation degradation that this advantage is lost. APEX changes this, making extreme compression preferable at matched compute in the evaluated regime. We do not claim superiority at arbitrarily large budgets.

Narrower widths not evaluated: Existing literature already evaluates these overall compression ratios and demonstrates severe reconstruction degradation. FreqWarm (Fig. 1) reports rFID ~13.7 at 384:1 and ~5.2 at 192:1. DC-AE 1.5 (Fig. 1b) similarly reports rFID worsening from 0.26 at c128 to 1.60 at c16. We therefore focus on the regime where generation, rather than reconstruction, is the bottleneck.

Denoising-VAE: Its multi-level spectral regularization closely resembles Spectrum Matching's frequency-aligned reconstruction, which we include as a representative baseline for this family of approaches. Its additional EMA-teacher diffusion alignment is a distinct training mechanism, rather than asynchronous channel-wise denoising. Our evaluation is designed to cover representative methodological families, rather than every individual method.

Novelty: While APEX combines and adapts existing ideas, as sDCq's full assessment emphasizes, the improvements under extreme compression are substantial and consistent, and constitute a useful contribution.

Thank you again for your thoughtful review and discussion.

Reviewer_sDCqOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents APEX, a framework for improving latent diffusion under extreme spatial compression. It combines a hierarchy of cumulative channel prefixes, resolution-aware autoencoder supervision, and asynchronous denoising with a shared Transformer. A global clock controls the relative progress of different channel groups, while the training distribution emphasizes structure generation. Experiments on ImageNet-512 show substantial improvements over relevant baselines, particularly for wide latents with very few spatial tokens.

Critical strengths
  1. Substantial improvements in the most challenging configurations. The f128c1024 result is particularly compelling: with only 16 spatial tokens, APEX achieves a guided gFID of 18.77, compared with 51.47 for DC-AE 1.5 and 89.47 for DC-AE. The advantage also holds without guidance, with gFID improving from 84.92 to 48.56 relative to DC-AE 1.5 (Table 7). These are meaningful improvements in a setting where increasing latent capacity severely challenges existing generators.

  2. Consistent improvements while preserving reconstruction capacity. The gains extend across the evaluated f64 and f128 configurations, with substantially less degradation as channel count increases. The authors report matched full-latent reconstruction quality within each configuration, making the generation improvements relevant to the reconstruction–generation trade-off. The shared denoiser and fixed spatial token count also make the approach practically attractive.

Overall assessment: Although the methodological novelty is primarily in combining and adapting existing ideas, the improvements under extreme compression are substantial and consistent enough to constitute a useful contribution. I particularly value the f128c1024 results. My remaining concerns focus on strengthening the evidence that the additional latent capacity is meaningfully used during generation and providing a firmer justification for the scheduling choices in deeper hierarchies.

Critical weaknesses
  1. The contribution of the widest refinement groups to generation remains insufficiently characterized. Good full-latent reconstruction does not necessarily imply that the diffusion model successfully generates the information carried by the tail channels. Figures 3, 9, and 10 provide useful qualitative evidence, and Figure 8(c) demonstrates a quantitative benefit for f64c128. However, an analogous analysis is missing for f128c1024. Moreover, Figure 8(c) reallocates the sampling budget across different trajectory lengths, so it does not isolate the contribution of the tail while holding the generated prefix fixed. It remains unclear how much useful detail the final 512-channel group contributes in the configuration with the strongest relative improvement.

  2. The scheduling choices need stronger support for deeper hierarchies. The gap and training-allocation ablations are informative but mainly concern the three-level f64c128 model. Under the reported five-level f128c1024 configuration, the final group is active only in the tail interval, which receives 10% of the training-clock sampling probability. Its full local trajectory is covered, but further diagnostics would help establish whether this allocation adequately learns detail refinement. Connecting schedule choices to the actual contribution of generated tail channels would strengthen the paper’s central explanation.

Questions for authors
  1. For f128c1024, could you evaluate the same completed generated latents with all 1024 channels and with the final 512 channels zeroed out before decoding? Paired detail crops and gFID/sFID, together with the corresponding truncation effect on encoded real-image latents, would help distinguish reconstruction capacity from effective detail generation.

  2. How were the gap and clock-sampling allocation validated for the five-level model? Could you provide per-group denoising diagnostics across local timesteps, or a limited sensitivity analysis showing that the final refinement group is adequately learned? A sampling-grid comparison that changes the allocation of integration steps while keeping the trained clock mapping and total NFE fixed could also help assess whether tail refinement is limited by the sampling budget.

4 comments
AuthorsOpenReview ↗

Response to Reviewer sDCq

Thank you very much for your positive feedback and thorough review! We agree that further analysis of the contribution of the tail levels for our deepest architectures will strengthen the paper (especially for the deeper configurations), and we've conducted thorough experiments accordingly.

Contribution of the final refinement level in f128c32-128-256-512-1024: Figure 8(c) shows per-level gFID when we reallocate all inference steps to the first k levels on the f64c128 architecture. As requested, we ran the experiment on f128c32-128-256-512-1024 without resampling or reallocating the inference steps, which directly isolates the contribution of the final group. Each completed 1024-channel generated latent is decoded twice: once using all channels, and once with the final 512 channels set to zero. Both decodes therefore use exactly the same generated first 512 channels. We now report this control in Appendix Table 12.

PrefixrFIDgFID (CFG 1.5)sFID (CFG 1.5)gFID (w/o CFG)sFID (w/o CFG)
First 4/5 levels (channels 0 to 512)0.2521.2216.9149.3623.23
All levels0.2018.7714.0548.5618.98

Result: The final group improves guided gFID from 21.22 to 18.77, guided sFID from 16.91 to 14.05, and encoded-real rFID from 0.25 to 0.20. The unguided metrics improve as well.

We also measured this qualitatively in the updated manuscript:

  • Figure 10 in Appendix A shows generations from f128c512 and f128c1024, with comparisons to baselines. APEX's improvement is substantial, especially at f128c1024.
  • Figure 12 shows per-level decodes for f128c1024 and the RGB differences between the generated levels, with crops and zoom-ins.
  • Figure 13 shows per-level reconstructions for all AE that we trained.

Analysis of inference schedule allocation: We designed an experiment to test what happens when a certain level is "prioritized" during inference. Normally, we use a standard uniformly spaced Euler schedule. Here, for a given level, we double the density of inference steps within its active range (the interval of the global clock during which that level is being denoised), relative to the rest of the trajectory. The total number of steps is held constant. We now report the complete grid in Appendix Table 13.

Emphasized levelgFID (CFG 1.5)sFID (CFG 1.5)gFID (w/o CFG)sFID (w/o CFG)
018.5914.1548.4418.83
118.9314.4049.0119.22
219.1414.5449.2119.41
319.0814.5549.2219.35
418.8114.3348.7419.04
Uniform Euler18.7714.0548.5618.98

Result: At fixed NFE, prioritizing the final group gives gFID 18.81, compared with 18.77 under uniform Euler. We did not see an improvement from this tail-focused allocation. This experiment only changes the inference allocation.

How were the gap and clock-sampling allocation selected for the five-level model? We transferred g=0.25g = 0.25 and the structure-heavy training allocation from the three-level ablations in Figures 8(a)-(b), and kept them fixed across architectures. Our motivation for emphasizing L0 is that it carries the main generative burden, as opposed to later levels that do not have to generate “from scratch”. We chose pL0=0.9p_{L_0}=0.9 rather than 1.0 because otherwise, if the tail levels don’t overlap with L0, they will not be sampled at all. The fact that the final tail level still contributes demonstrates that the approach remains effective in deep hierarchies.

We hope these new experiments address your concerns! Please let us know if anything remains unclear. We would appreciate it if you could consider raising your score based on these results.

AuthorsOpenReview ↗

Follow-up on Additional Experiments

Thank you again for your review! We wanted to follow up while there's still time for discussion.

We ran the additional experiments you suggested, including the paired analysis of the final refinement level at f128c1024 and the inference schedule allocation study. Both are now included in the revised manuscript, and we believe the results directly address your concerns.

We'd really appreciate your thoughts on these new results, and hope you'll consider raising your score if you find that your concerns have been addressed. Please let us know if there's anything else you'd like us to clarify!

AuthorsOpenReview ↗

Summary of Rebuttal

Before the discussion closes, we’d like to summarize the additional experiments included in the revised manuscript in response to your concerns:

  • Contribution of the final refinement level (Table 12): The requested paired truncation experiment at f128c1024 shows that the final 512 channels improve gFID from 21.22 → 18.77.
  • Inference schedule allocation (Table 13): We tested different step allocations at fixed NFE and found that the final level is not starved of inference steps: prioritizing the final level does not improve over uniform Euler (18.81 vs. 18.77 gFID).
  • Per-level visualizations (Figs. 10–13): We added generation comparisons, per-level decodes, and detail crops to illustrate how the refinement levels contribute to image quality.

We believe these experiments directly address your main concerns and strengthen the evidence for APEX’s effectiveness in deep hierarchies. We’d appreciate your consideration of these results in your final assessment!

Thanks for the additional experiments. The new truncation experiment clearly shows that the final refinement level contributes to generation quality, and the schedule allocation results further support the effectiveness of the five-level hierarchy.

The additional quantitative analysis of intermediate denoising also provides a better understanding of why APEX works, while the reconstruction comparison addresses my remaining question about the reconstruction-generation tradeoff.

I believe my concerns have been adequately addressed by the new experiments and revisions. I am satisfied with the response and will maintain my positive assessment.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response

Dear Reviewers and Area Chair,

We thank all reviewers for their thoughtful feedback. We are glad that reviewers recognized the substantial and consistent improvements achieved by APEX under extreme compression (sDCq, nRc2), found the method to be novel and elegant (xdX7), and highlighted its simplicity and computational efficiency (nRc2). We also appreciate that reviewers found the paper informative (xdX7) and the results particularly compelling in the challenging 16-token setting (sDCq).

We have updated the manuscript with additional experiments, analyses, and visualizations addressing the reviewers' main concerns. The main additions are:

  • Statistical robustness and CFG optimization (nRc2): We evaluated the main comparisons across 10 random seeds, computed 95% confidence intervals, and optimized CFG separately for each method. APEX maintains its advantage over the baselines, including at higher inference-compute budgets where the differences are smallest.
  • Contribution of the refinement levels (sDCq): We performed additional experiments demonstrating that the final refinement level contributes meaningfully to both reconstruction and generation quality, confirming that the additional latent capacity is effectively used.
  • Additional visualizations and explanations (sDCq, nRc2, xdX7): We added extensive qualitative visualizations of our autoencoders and generated samples across all evaluated models, including comparisons with the baselines. We also expanded the explanation of APEX's motivation and provided a detailed breakdown of the ablations.

Our individual responses provide the full results and address the remaining questions. We believe these additions further strengthen the evidence for APEX's contributions and hope they help inform the discussion and final assessment.

AuthorsOpenReview ↗

Follow-up on Rebuttal and New Experiments

Dear Reviewers and Area Chair,

We posted our rebuttal and revised manuscript on November 12, including substantial new experiments that directly address the concerns raised in the reviews.

We believe these results significantly strengthen the paper, and we'd really appreciate it if you could take a look. With the discussion period ending on November 18, we want to make sure there's enough time to discuss any remaining follow-up questions.

Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.