acceptodds
Under review as a conference paper at ICLR 2027

Successive-Refinement Latent Variable Generative Models

Abstract

Many generative models construct samples progressively, through sequences of latent variables, denoising steps, or iterative refinements. We study when this process forms a meaningful hierarchy, in which each stage reveals new information, improves the current reconstruction, and enables coarse-to-fine control. Our focus is on hierarchical latent variable models (HLVMs), where multiple latent variables concur to build a coarse-to-fine generation pipeline. We analyze them through *successive refinement*, the classical theory of layered lossy coding, which asks whether every partial representation achieves the smallest reconstruction error permitted by the information it carries about the source. This view yields two core principles: I) control the cumulative information rate available at each stage; II) supervise every intermediate reconstruction. We design a novel objective for hierarchical models following these principles, and show it recovers the best model-realizable hierarchy at the achieved information rates. We then analyze two canonical HLVMs under the same criteria. We find the ELBO of hierarchical variational autoencoders (HVAEs) does not incentivize hierarchical representations, as it constrains only the total rate and final reconstruction, whereas denoising diffusion probabilistic models (DDPMs) realize a special case in which the encoder is fixed, and rate–distortion optimal only for Gaussian sources under MSE. We instantiate the proposed objective in an HVAE baseline, the Successive Refinement HVAE (SR-HVAE). On analytically tractable sources it matches every prediction of the SR theory. On images it makes every latent active, enables reaching rate regions the standard ELBO never targets, and improves overall generative ability.

Primary area:
generative models
Keywords:
hierarchical latent variable models, hierarchical generation, diffusion models, variational autoencoders, rate-distortion theory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KPd4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper connects hierarchical latent models to successive refinement, derives conditional rate–distortion guarantees, and proposes rate targeting with prefix reconstruction losses. Experiments examine whether the resulting hierarchies use additional latent information effectively and whether this improves generation. Validation covers Gaussian and Gaussian-mixture sources, MNIST, and CelebA. With the same HVAE architecture and approximately matched rates, the proposed objective improves intermediate reconstructions; separate prior-refitting experiments evaluate unconditional generation.

Critical strengths

The analysis clarifies when a hierarchy can be optimal at every prefix, and controlled experiments demonstrate a practical benefit for intermediate reconstruction. With approximately matched rates and the same HVAE architecture, prefixes performing no better than the data mean decrease from 4/32 to 0/32 on MNIST and from 12/32 to 0/32 on CelebA (Table 12).

Critical weaknesses

The incremental contribution over Shu and Ermon (2022) remains unclear. Their formulation already combines reconstruction losses and cumulative rate constraints for all prefixes. This paper adds SR analysis, replaces hard budgets with rate-targeting penalties, and introduces slope-based weights, but provides no experimental comparison with that method. Thus, the benefit attributable to the new training objective is not established.

The generation results do not consistently support the stated benefits. At approximately matched rates, SR-HVAE has worse FID than HVAE on MNIST (22.80 vs. 21.59) and CelebA (53.14 vs. 48.26; Table 2). Improvements with extended training or prior refitting are encouraging, but do not isolate the benefit of the objective. The clear reconstruction gains therefore do not establish better unconditional generation.

Questions for authors

What additional theoretical or empirical insight distinguishes this approach from Shu and Ermon’s all-prefix formulation?

How should the claim of improved generation be interpreted given the matched-rate results and the additional training or prior changes in the stronger results?

5 comments
AuthorsOpenReview ↗

Answer to Reviewer KPd4, part 1/2

We thank the Reviewer for the careful reading, and for finding the paper clear and sound. In the following, line numbers refer to the revised version of our manuscript (changes in red).

W1 / Q1: contribution over Shu and Ermon (2022)

The incremental contribution over Shu and Ermon (2022) remains unclear. [...] [The paper] provides no experimental comparison with that method. What additional theoretical or empirical insight distinguishes this approach from Shu and Ermon’s all-prefix formulation?

We thank the reviewer for raising this. We extended Appendix A (Related Works, L675–686) to better highlight the distinction between the two works. Accordingly, we revised the abstract (L25–27), the introduction (L92–93), our contribution II (L104–106), and Section 3 (L303–304) to no longer present the special-case reading of DDPMs as our result. Beyond that correction, several distinction between the two works remain, and we discuss them in detail as follows.

Shu & Ermon formalise HLVMs, call a hierarchy a good progressive coder when every prefix is RD Pareto-optimal, and propose the sum of prefix distortions under cumulative rate budgets (their Eq. 3), of which the DDPM objective is a special case. They test the progressive coding hypothesis with two VDVAEs. While it is true that we address a similar question, several distinctions emerge.

1. When every prefix can be optimal, and what an all-prefix objective achieves. Shu & Ermon note that there is no guarantee that (Rk,Dk)(R_k, D_k) is Pareto-optimal for k<Tk < T, and they leave the rate targets as a hyperparameter. Our paper uses SR to answer both points:

  • A hierarchy can be RD-optimal at every prefix only if the source–distortion pair is successively refinable (Theorem 1, L141–143; App. B.5). Otherwise every nested code has a gap at some prefix (Cor. 2.1, L285–289).
  • For our objective, under the assumptions stated with it, Prop. 2 (L275–279) guarantees an O(λ−1)O(\lambda^{-1}) rate error and Pareto optimality at the achieved allocation. In the SR case, Cor. 2.1 adds frontier attainment at every prefix.
  • The analytic sources verify both sides against the exact frontier (Fig. 1; L369–377).

2. Design choices with consequences. Eq. 10 differs from their Eq. 3 in two precise ways, backed up by theory.

  • Two-sided targets instead of budgets. A budget Rk≤γkR_k \le \gamma_k lets a level add nothing (Rk=Rk−1R_k = R_{k-1}), which an optimiser may prefer on a non-refinable source. A target forces every level to add rate, and, under variational tightness, true information (L242–243; Cor. 2.1, Eqs. 82–83, L1190–1197).
  • Slope-normalised weights. We express every level in rate-equivalent units, so the coarse levels, where the RD slope is steepest, do not dominate the sum (L250–260). λ\lambda then acts as an exchange rate, and λ=1\lambda = 1 matches the reference trade-off (L266–269).

We additionally measure the effect of a different rate schedule: at a fixed total rate, its allocation alone moves the final FID from 26.6 to 11.1 on MNIST (Fig. 3; L460–462).

3. What the DDPM special case achieves in RD terms. Shu & Ermon show that DDPMs are trained as progressive coders. We show that SR quantifies how good a coder this special case is:

  • For isotropic Gaussian sources under MSE, every DDPM prefix is RD-optimal and the reconstructions form the SR Markov chain (Prop. 1, L194–199).
  • Outside this case, per-prefix RD optimality is no longer guaranteed, independently of the denoiser's capacity, and the reconstructions no longer inherit the Markov chain (L210–213). On the SBGM non-refinable source, the DDPM leaves the frontier near RcritR_{\text{crit}} and never rejoins it within the tested range (Fig. 1b; L372–377), with a per-level excess Dk/D(Rk)D_k/D(R_k) up to 1.33, identical across seeds (Table 6).

This gives a precise trade-off between a fixed encoder and a learned one (L298–315). The fixed encoder has exact rates and a channel penalty that cannot be recovered. The learned encoder adapts to the source but pays the prior–posterior mismatch of Eq. 8, which can be reduced without touching the hierarchy. Such a distinction between learnable and fixed encoders is not analyzed in Shu and Ermon.

AuthorsOpenReview ↗

Answer to Reviewer KPd4, part 2/2

4. A refinement of the progressive coding hypothesis. Our results indicate that good progressive coding is not sufficient for good sampling: sampling quality also depends on the prior–posterior gap and on the rate regime.

  • At the matched rate, SR-HVAE is a far better progressive coder than the HVAE (no prefix stuck at the data mean, against 4/32 and 12/32; L455–457; Table 12), yet it is not a better sampler.
  • Decoding SR-HVAE from its own posterior instead of its prior changes FID from 11.09 to 3.36 on MNIST and from 50.79 to 24.21 on CelebA (Table 2). Much of the remaining gap therefore sits in the prior, not in the coding.
  • On MNIST at the matched rate, the DDPM, a good progressive coder by construction, is the weakest sampler of the three (46.75 FID). At RT=0.2R_T = 0.2 it reaches 10.47 (Table 2): the rate itself was starved.

5. Training the all-prefix objective. Shu & Ermon test their hypothesis with a two-stage procedure: two separately trained VDVAEs, a single split, a distortion target on the first stage, and the ELBO on the second. As a result:

  • only one intermediate prefix is supervised, and the rate of the second stage is set by the ELBO;
  • in their design, TT stages would require TT models trained one after another;
  • freezing each stage at its own optimum costs nothing only when the source is successively refinable. On a non-refinable source it forces excess rate or distortion on later stages, whereas a joint objective selects a compromise across all prefixes (Prop. 2). On SBGM, SR-HVAE leaves the frontier near RcritR_{\text{crit}} and returns to it immediately afterwards (Fig. 1b; L372–376).

We train the all-prefix objective end-to-end, in a single run, at every level (32 on images, 16 on the analytic sources). We evaluate every prefix against an external reference: the exact frontier on the analytic sources and an achievable curve on images (L417–419).

To sum up, Shu and Ermon formalized the HLVM model family, identified a meaningful objective and showed that DDPMs pursue it, while HVAEs do not. We go beyond this theoretically (with SR formulation, DDPM analysis and objective guarantees), insightfully (by showing the practical trade-offs between prescribed and learned encoders) and experimentally (enabling a formulation that trains the objective without freezing intermediate models and on T steps concurrently).

Experimental comparison. We did not compare directly for two reasons. 1: their experiment tests the hypothesis in a different setting (VDVAE model using 2 stage split). 2: our evaluation already isolates the objective: against the parameter-identical ELBO baseline (Table 11), and against the exact frontier on the analytic sources. If after reading our answer the Reviewer still wishes, we can run their two-stage baseline on MNIST with our architecture: their Eq. 3 on the first T/2T/2 levels, then frozen, followed by the remaining T/2T/2.

W2 / Q2: interpreting the generation results

At approximately matched rates, SR-HVAE has worse FID than HVAE [...]. Improvements with extended training or prior refitting are encouraging, but do not isolate the benefit of the objective. How should the claim of improved generation be interpreted?

The matched-rate rows are an ablation at the ELBO's own operating point (L414–416), a regime where every model family performs poorly, DDPMs included (L430–431). Choosing the rate is principle (i) of our objective (L217–218). The comparison of the two objectives is therefore at the extended budget, with parameter-identical models (Table 11). There, SR-HVAE reaches 11.09 vs. 21.59 FID on MNIST; on CelebA it has better precision and recall, while its FID is slightly worse (50.79 vs. 48.26).

The prior refit does not confound this. It is applied to both models, with encoder and decoder frozen, and isolates the prior–posterior mismatch (last block of Table 2, L452–453; bold: best):

MNIST FID ↓prec. ↑rec. ↑CelebA FID ↓prec. ↑rec. ↑
HVAE, prior refit23.720.3530.19948.220.3560.004
SR-HVAE, prior refit6.460.5920.64543.320.4350.024

SR-HVAE is better on every metric. The same refit does not help the HVAE (L473–476): a better prior pays off only when the hierarchy carries information worth sampling.

We thank the Reviewer again for its constructive comments. We hope our clarifications address the Reviewer's concerns and, if so, that the Reviewer will consider raising the score. We remain at the Reviewer's disposal for any further question, including the MNIST comparison described above.

Reviewer_KPd4OpenReview ↗

Follow-up after author response

Thank you for the detailed response and clarifications. I appreciate the improved results when both models' priors are refitted. However, the generation gains before prior refitting remain mixed: SR-HVAE has worse FID at matched rates, and increasing the rate does not consistently improve FID over HVAE across datasets. Together with the missing direct comparison to Shu and Ermon, this leaves me unconvinced that the proposed objective offers a sufficiently strong empirical advantage.

The theoretical analysis provides a useful interpretation, but its additional insight into model behavior and guidance for model design still seem limited. For example, the guarantees hold for arbitrary fixed positive weights and do not establish an advantage of the proposed slope-based weighting. I appreciate the authors' efforts, but these concerns remain, so I am maintaining my weak-reject recommendation.

Reviewer_HsLmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies hierarchical latent-variable generative models through the rate–distortion framework of successive refinement (SR). It proposes SR-HVAE, which assigns a cumulative variational-rate target to every latent prefix and supervises the corresponding prefix reconstruction. The paper also analyzes standard HVAEs and DDPMs, observing that the HVAE ELBO directly constrains only the total variational rate and final reconstruction, while DDPMs realize rate–distortion-optimal SR under idealized conditions including an isotropic Gaussian source and MSE distortion.

Critical strengths
  1. A explanatory unifying framework. The SR perspective places ineffective HVAE prefixes, DDPM noise schedules, and stage-wise information allocation in a common rate–distortion framework while clearly identifying the conditions required for strict optimality.
Critical weaknesses
  1. The formal guarantees are weaker than the central claim that every latent adds true information and improves reconstruction. The controlled variational rate satisfies equation (8), so a positive rate increment may reflect additional prior slack rather than additional conditional information about the source. Proposition 2 provides rate-error control and weighted Pareto optimality at the achieved allocation, but does not generally imply (D_{k+1}<D_k) or the reconstruction Markov structure required by SR.
  2. The image experiments do not support a strong generation claim. At matched rate, SR-HVAE has worse final ancestral FID than the vanilla HVAE on both MNIST and CelebA. At the extended rate on CelebA, it remains substantially worse than an equal-depth DDPM.
Questions for authors

Can the authors estimate the per-prefix prior slack or conditional mutual information and identify which result guarantees that every stage adds true information and satisfies (D_k<D_{k-1})?

5 comments
AuthorsOpenReview ↗

Answer to Reviewer HsLm, part 1/2

We thank the Reviewer for the careful reading, and for seeing the paper as a clear, unifying framework between HVAEs, DDPMs and optimality conditions. In the following, line numbers refer to the revised version of our manuscript (changes in red).

W1 / Q1: formal guarantees

A positive rate increment may reflect additional prior slack rather than additional conditional information about the source. Proposition 2 [...] does not generally imply Dk+1<DkD_{k+1} < D_k or the reconstruction Markov structure required by SR. Can the authors estimate the per-prefix prior slack [...] and identify which result guarantees that every stage adds true information and satisfies Dk<Dk−1D_k < D_{k-1}?

The Reviewer is right that reading Prop. 2 alone does not give these properties. However, Prop. 2 is better read under its assumptions (L275–279, including variational tightness), Cor. 2.1 and its proof. In the ensemble, they give the following.

  1. True information at every stage. For λ\lambda large enough the achieved allocation is strictly ordered (Eqs. 82–83, L1190–1197), so I(X;Zk∣Z1:k−1)=Rk−Rk−1>0I(X; Z_k \mid Z_{1:k-1}) = R_k - R_{k-1} > 0 (Eq. 61; L242–243).
  2. SR sources. Dk=D(Rk)D_k = D(R_k) at every level, hence Dk<Dk−1D_k < D_{k-1}. Moreover, Rk≥I(X;X^(1:k))≥I(X;X^(k))≥R(Dk)=RkR_k \ge I(X; \hat X^{(1:k)}) \ge I(X; \hat X^{(k)}) \ge R(D_k) = R_k forces I(X;X^(1:k−1)∣X^(k))=0I(X; \hat X^{(1:k-1)} \mid \hat X^{(k)}) = 0, the compatibility of Eq. 41 (same argument as App. B.4, Eqs. 34–36).
  3. Non-SR sources. No nested code is frontier-optimal and compatible at every prefix (Eq. 42), so no objective can guarantee it. Prop. 2 gives the Pareto-optimal compromise; distortion is non-increasing when the level-kk decoder can ignore ZkZ_k, but not strictly decreasing in general.

The original claim (L296 of the submission) was therefore imprecise. We thank the Reviewer for noticing this, and revised it (L298–303) to: "Under the assumptions of Proposition 2, and for λ\lambda large enough that the achieved allocation is ordered, every latent adds true source information, and when the source is SR each prefix is a strictly better, compatible reconstruction (Corollary 2.1). When the source is not SR, no nested code reaches the frontier at every prefix; our objective then selects a Pareto-optimal compromise at the achieved rates."

Estimating the slack. As the Reviewer points out, in practice δk\delta_k in Equation 8 need not vanish, and a positive variational increment may partly reflect prior slack. The paper states this (L240–243), discusses it (L309–315) and acknowledges it both as a limitation (L485–486) and as a main direction for future work (L486–488). We also attempt to estimate the slack during the experimental evaluation, depending on the case.

Analytic sources. We bracket δk\delta_k at every level (Table 9), which bounds every true increment from below. A second bound needs no auxiliary estimator: by the RD converse (Eq. 58), Rk≥R(Dk)R_k \ge R(D_k), while Rk−1≤Rk−1varR_{k-1} \le R^{\text{var}}_{k-1}. Over 10 seeds:

GaussianSBGM
levels with a positive lower bound on I(X;Zk∣Z1:k−1)I(X; Z_k \mid Z_{1:k-1}), every seed16/1616/16
smallest bound for k≥2k \ge 2, bracket (bits/dim)0.1350.162
share of each variational increment certified as true information, k≥2k \ge 2, bracket≥ 79%≥ 86%
same, estimator-free bound≥ 60%≥ 76%
Dk<Dk−1D_k < D_{k-1} at every level10/10 seeds10/10 seeds

Image sources. A per-level bound is out of reach: D(R)D(R) is unknown, contrastive MI bounds saturate at log⁡2N≈\log_2 N \approx 13–16 bits against about 200 (MNIST) and 1,200 (CelebA) bits per image, and upper bounds need density models over 2,048 or 16,384 latent dimensions. We report instead:

  • RkvarR^{\text{var}}_k strictly increasing and DkD_k strictly decreasing at all 31 transitions of every SR-HVAE run;
  • the generative cost of the slack, as the gap between prior and posterior decoding (Table 2; L421–423);
  • the share the prior alone explains: refitting it recovers 60% of that gap on MNIST and 28% on CelebA (L99–100).
AuthorsOpenReview ↗

Answer to Reviewer HsLm, part 2/2

W2: generation claim

The image experiments do not support a strong generation claim. At matched rate, SR-HVAE has worse final ancestral FID than the vanilla HVAE [...]. At the extended rate on CelebA, it remains substantially worse than an equal-depth DDPM.

We agree that a strong generation claim is not supported in general. The claim is relative to the ELBO-trained HVAE, at the rates our objective selects (abstract, L29–31; contribution III, L107–110; L464–467). Our limitations state that the DDPM wins on ancestral sampling at 64×64 (L483–485). We revised the paper to avoid any ambiguities on how the "better generative claim" should be read, which we summarize as follows.

Against the HVAE. The matched-rate row is an ablation that pins SR-HVAE to the ELBO's own operating point (L414–416), a regime where every model family performs poorly (L430–431). The comparison of the two objectives is at the rate each selects, and like-for-like with both priors refitted:

MNIST FID ↓prec. ↑rec. ↑CelebA FID ↓prec. ↑rec. ↑
HVAE, own rate (0.032 / 0.013)21.590.2370.28848.260.2860.005
SR-HVAE, matched rate (ablation)22.800.2940.23653.140.3090.004
SR-HVAE, targeted rate (0.2 / 0.1)11.090.3790.63250.790.3670.024
HVAE, prior refit23.720.3530.19948.220.3560.004
SR-HVAE, prior refit6.460.5920.64543.320.4350.024

Rows are from the revised Table 2 (L432–453; bold: best). The HVAE prior refit was previously reported only in Table 14, we moved it to main with the revision. Note that the refitting HVAE does not help: the refit is beneficial only to the model trained with our objective, i.e. the one whose hierarchy carries meaningful information (L473–476).

Against the DDPM. Both are instances of the same objective, with different encoders (L298–315).

  • Optimal conditions. The prescribed encoder is per-stage RD-optimal only for isotropic Gaussian sources (Prop. 1), and per-stage optimality is no longer guaranteed otherwise, whatever the denoiser's capacity (L210–212). On SBGM it leaves the frontier near RcritR_{\text{crit}} and never rejoins it (Fig. 1b; excess up to 1.33, Table 6), and SR-HVAE outperforms even a 1000-step DDPM on every generative metric (Table 1).
  • Practice. On images, the learned encoder has more to recover: the prior slack, and the joint optimisation of encoder and decoder at finite capacity (L482–485). The DDPM's prefixes are closer to the reference (median excess 1.16 vs. 1.98 on MNIST, 1.21 vs. 1.58 on CelebA; Table 12), and it samples better.

A prescribed encoder is therefore currently the convenient choice on images; closing the prior slack is the natural next step (L486–488).

Overall, our paper should be seen as a comparison between objectives (ELBO vs SR/Hierarchical), with SR-HVAE and DDPM as two instances of the same hierarchical loss, where only the encoder changes. As stated in the introduction (L49–50), the main research question is: how can we train a generative model so that every stage actively adds new information, improving reconstructions?
We believe Successive Refinement is the correct tool to investigate these questions, and both our objective and its guarantees derive from that. Experimentally, our goal was not to show or claim absolute superiority of SR-HVAE over DDPMs, but instead to understand how a properly learned hierarchy affects generation quality, including its strengths and limitations.

We hope our answer clarified the concerns and questions of the Reviewer. If this is the case, we hope it will lead to a score increase. Otherwise, we remain at the Reviewer's full disposal for further discussion or any clarification.

Reviewer_HsLmOpenReview ↗

Response

Thank you for the detailed response and revisions. The clarification resolves some of my concerns about the scope of the formal results. The revised statement (L298–303) and generation claim is substantially clearer.

My remaining concern is the practical scalability of the framework beyond small-scale unconditional generation. The current image experiments are limited to low resolution and small-scale dataset, where SR-HVAE already exhibits substantial prior mismatch, requires prior refitting, and remains behind DDPM in ancestral generation. These difficulties are likely to become more severe in text-to-image or text-to-video generation.

Reviewer_daW7OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to train hierarchical latent variable models so that successive latent stages produce progressively better reconstructions. Drawing on successive refinement theory, it proposes an objective that controls the cumulative information rate and supervises reconstruction at every stage. The authors apply this objective to hierarchical VAEs, yielding SR-HVAE, and connect their framework to diffusion models.

The image experiments on MNIST and CelebA64 compare reconstruction distortion and FID across stages. The main benefit is improved intermediate reconstructions over a standard HVAE. However, my reading of the results is that these improvements do not translate into consistently better final generation quality. My main concern is whether the practical value of progressive reconstruction is sufficiently established to justify this trade-off.

Critical strengths
  • Improved intermediate reconstruction is the strongest positive result. SR-HVAE produces better reconstructions from partial latent representations than the standard HVAE baseline. This could be useful when progressive reconstruction is an explicit requirement, although the paper needs to establish the practical value of that capability more clearly.

  • Successive refinement provides a coherent motivation for the training objective. Controlling cumulative information rates and supervising intermediate reconstructions gives a concrete way to train for this capability. The connection to diffusion models is also conceptually interesting, although its novelty needs clarification.

Critical weaknesses
  • The practical benefit of progressive reconstruction is not sufficiently established. This is my main reason for leaning toward rejection. The method improves intermediate reconstructions, but my reading of Table 2 is that the standard HVAE achieves slightly better final FID on MNIST and more clearly better final FID on CelebA64. The paper needs to demonstrate a use case where access to intermediate reconstructions justifies this trade-off.

  • The theoretical motivation does not establish an advantage for the final output. Successive refinement asks whether intermediate reconstructions can be optimal without sacrificing final rate–distortion optimality. It does not imply that intermediate supervision improves the final optimum. Likewise, poor intermediate reconstruction does not by itself mean that earlier latents are unused; they may contribute through later stages. The paper should distinguish these claims more carefully. In my opinion, this approach is defensible only when progressive compression, and hence performance along the rate–distortion curve, is an explicit requirement. It is not clear to me which applications require this.

  • The novelty of the connection to diffusion models needs clarification. The authors should explain what their successive refinement formulation adds beyond existing progressive coding and rate–distortion interpretations of diffusion (Ho et al., 2020). A precise new theoretical result could strengthen the contribution, but the connection alone does not resolve the limited practical advantage demonstrated by SR-HVAE.

  • The baseline is weak for intermediate reconstruction, making the gains unsurprising. Figure 2 compares SR-HVAE, which explicitly supervises intermediate reconstructions, with a standard HVAE trained only for the final output. Better intermediate results are therefore expected and provide limited evidence for the proposed method’s advantage. The more telling result is that the end-to-end baseline achieves better final FID in Table 2. This suggests that intermediate supervision may come at a cost to final generation quality, which weakens the case for adopting it unless progressive reconstruction is explicitly needed.

Questions for authors
  1. Practical value. What concrete application benefits from SR-HVAE’s improved intermediate reconstructions enough to justify the apparent loss in final FID in Table 2? What advantage would SR-HVAE offer over diffusion models in that setting?

  2. Novelty of the diffusion connection. What new result does the successive refinement formulation establish beyond existing progressive coding and rate–distortion interpretations of diffusion (Ho et al., 2020) ?

5 comments
AuthorsOpenReview ↗

Answer to Reviewer daW7 Part 1/2

We thank the Reviewer for the constructive review, and for recognizing that successive refinement gives a coherent motivation and that improved intermediate reconstruction is a positive result. In the following, line numbers refer to the revised version of our manuscript (changes in red).

W1 / Q1: reading of Table 2

My reading of Table 2 is that the standard HVAE achieves slightly better final FID on MNIST and more clearly better final FID on CelebA64. [...] What concrete application benefits [...] enough to justify the apparent loss in final FID?

We recognize that the layout of the original Table 2 invited this reading, and we restructured it (L432–453). The rows where the HVAE wins are a matched-rate ablation: SR-HVAE is pinned to the total rate the ELBO reaches on its own, 0.032 and 0.013 bits/dim (L414–416). This regime is starved for every model family (L430–431): the DDPM itself degrades from 10.47 FID at the extended rate to 46.75 on MNIST, and from 30.84 to 38.37 on CelebA. However, the ELBO objective cannot choose its rate (L82–84), while our objective can, so the comparison remains fair also at extended rate. Moreover, since both HVAE and SR-HVAE have a learnable encoder, the final comparison is with both priors refitted, encoder and decoder frozen:

MNIST FID ↓prec. ↑rec. ↑CelebA FID ↓prec. ↑rec. ↑
HVAE, own rate (0.032 / 0.013)21.590.2370.28848.260.2860.005
SR-HVAE, matched rate (ablation)22.800.2940.23653.140.3090.004
SR-HVAE, targeted rate (0.2 / 0.1)11.090.3790.63250.790.3670.024
HVAE, prior refit23.720.3530.19948.220.3560.004
SR-HVAE, prior refit6.460.5920.64543.320.4350.024

At its own rate, SR-HVAE is better on all metrics on MNIST and on precision and recall on CelebA; with both priors refitted it is better on every metric and both datasets.

Applications (Q1). Concrete applications are therefore not a tradeoff on sample quality, but an addition to it. Examples include:

  • progressive coding and transmission: since every prefix is a usable reconstruction, whereas the first 12 HVAE levels on CelebA carry 26% of the rate yet reconstruct no better than the data mean (L86–88);
  • coarse-to-fine generation: samples are built progressively (Fig. 3, center), allowing to partially sample and discard or generate multiple details from one coarse image;
  • coarse-to-fine editing: keeping the first jj levels of a real image and resampling the rest yields variations that retain more of its structure as jj grows (new Fig. 4, App. E.9, L1743–1757).

Also note that all these applications are more beneficial when the targeted rates can be selected (the targeted rate setup, which the ELBO does not permit).

W2 / Q1: theoretical motivation

Successive refinement [...] does not imply that intermediate supervision improves the final optimum. Likewise, poor intermediate reconstruction does not by itself mean that earlier latents are unused. [...] What advantage would SR-HVAE offer over diffusion models?

We agree, and the paper does not claim a better final optimum based on theory. The theory characterizes when asking every stage to be optimal costs nothing at the final one: for SR sources our objective reaches the frontier at every level (Cor. 2.1, L285–289; Fig. 1a); otherwise it selects a Pareto-optimal compromise with an unavoidable gap (Prop. 2; Fig. 1b). The gains on the final output are shown empirically (see W1), and come from two sources:

  • the objective chooses the rate region and its allocation, which the ELBO cannot: on MNIST increasing the rate budget moves FID from 22.8 to 13.6 (L89–90), and deciding rate allocations spans 26.6 to 11.1 (105.5 to 50.8 on CelebA; L460–462);
  • a better prior pays off only where the hierarchy carries information. Decoded from the posterior (no prior slack), SR-HVAE reaches 3.36 vs. 26.03 FID on MNIST and 24.21 vs. 40.83 on CelebA. Refitting the prior improves SR-HVAE (11.09 → 6.46, 50.79 → 43.32) but not the HVAE (refit block of Table 2; L473–476).

We also agree that the term "unused" was imprecise, and we now write "uninformative" (L86–87): the HVAE's early latents carry rate but contribute only through later stages. On CelebA, it reaches half of its final distortion reduction only after 59% of its rate, against 4% for SR-HVAE.

AuthorsOpenReview ↗

Answer to Reviewer daW7 part2/2

W2 / Q1 (continued): advantage over diffusion. Our work should be intended as a comparison between objectives, not models. In this respect, both DDPMs and SR-HVAE are instances of the same objective (L303–304) and differ only in the encoder (L298–315). A prescribed encoder (DDPM) has exact rates, but is per-stage RD-optimal only for isotropic Gaussian sources (Prop. 1). A learned encoder (SR-HVAE) adapts to the source, but pays the prior slack. On SBGM, where the prior slack is low, SR-HVAE outperforms even a 1000-step DDPM on every generative metric (Table 1). On images, the slack is higher and the prescribed encoder is currently the better sampler (L483–485). In short: the learnead encoder is an advantage when the prior slack can be kept small, since in that case the prescribed encoder's penalty is higher.

W3 / Q2: novelty of the diffusion connection

What new result does the successive refinement formulation establish beyond existing progressive coding and rate–distortion interpretations of diffusion (Ho et al., 2020)?

Ho et al. (2020) show that a DDPM defines a progressive code and report its overall RD curve. Successive refinement asks whether each stage is RD-optimal and whether the stages are compatible. This yields three results:

  1. Prop. 1 (L194–199). For isotropic Gaussian sources under MSE, every global minimiser of the DDPM loss attains Dk=D(Rk)D_k = D(R_k) at every level, and its reconstructions form the SR Markov chain.
  2. Its limit (L210–213). Outside this case, both properties are no longer guaranteed, whatever the denoiser's capacity. On SBGM the DDPM leaves the frontier near RcritR_{\text{crit}} and never rejoins it (Fig. 1b), with a per-level excess up to 1.33, identical across seeds, against at most 1.08 for SR-HVAE (Table 6).
  3. Learnable encoders (Prop. 2, Cor. 2.1, L275–289). Our objective has rate error O(λ−1)O(\lambda^{-1}), is Pareto-optimal at the achieved rates, reaches the frontier at every level for SR sources, and has an unavoidable gap otherwise.

These results separate the two encoders. A fixed encoder has exact rates but pays a penalty set by the channel (L305–309). A learned encoder adapts to the source, staying within 8% of the frontier at every level on SBGM (Table 6), but pays the prior slack. Of the two penalties, only the prior slack can be reduced without touching the hierarchy (L311–315).

W4: weak baseline

Better intermediate results are therefore expected and provide limited evidence. [...] The end-to-end baseline achieves better final FID.

We agree that the HVAE's poor intermediate reconstructions are expected, since the ELBO constrains only the total rate and the final reconstruction (L70–71). This is precisely the paper's claim, and Fig. 2 is its measurement: HVAE and SR-HVAE share architecture, parameters (Table 11) and prefix readout, so the difference is due to the objective alone. A multistage architecture does not, by itself, guarantee a good hierarchy (L45–46). However, such baseline is not the final result. The models are also tested on generative metrics, and also there our objective shows better metrics than ELBO (see answer to W1).

We thank the Reviewer for the constructive suggestions. We believe the revised version of Table 2 is now much clearer, and hope the Reviewer's main concern is now solved, leading to a re-evaluation of our paper.

Dear Reviewer daW7,

Thank you again for your thoughtful review. As the discussion period closes on November 18, we wanted to check whether our response addressed your concerns, and in particular the reading of Table 2, which has been reported as the main reason for rejection in the original review. We would be happy to clarify anything further or discuss any remaining concerns before the deadline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.