- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper connects hierarchical latent models to successive refinement, derives conditional rate–distortion guarantees, and proposes rate targeting with prefix reconstruction losses. Experiments examine whether the resulting hierarchies use additional latent information effectively and whether this improves generation. Validation covers Gaussian and Gaussian-mixture sources, MNIST, and CelebA. With the same HVAE architecture and approximately matched rates, the proposed objective improves intermediate reconstructions; separate prior-refitting experiments evaluate unconditional generation.
Critical strengths
The analysis clarifies when a hierarchy can be optimal at every prefix, and controlled experiments demonstrate a practical benefit for intermediate reconstruction. With approximately matched rates and the same HVAE architecture, prefixes performing no better than the data mean decrease from 4/32 to 0/32 on MNIST and from 12/32 to 0/32 on CelebA (Table 12).
Critical weaknesses
The incremental contribution over Shu and Ermon (2022) remains unclear. Their formulation already combines reconstruction losses and cumulative rate constraints for all prefixes. This paper adds SR analysis, replaces hard budgets with rate-targeting penalties, and introduces slope-based weights, but provides no experimental comparison with that method. Thus, the benefit attributable to the new training objective is not established.
The generation results do not consistently support the stated benefits. At approximately matched rates, SR-HVAE has worse FID than HVAE on MNIST (22.80 vs. 21.59) and CelebA (53.14 vs. 48.26; Table 2). Improvements with extended training or prior refitting are encouraging, but do not isolate the benefit of the objective. The clear reconstruction gains therefore do not establish better unconditional generation.
Questions for authors
What additional theoretical or empirical insight distinguishes this approach from Shu and Ermon’s all-prefix formulation?
How should the claim of improved generation be interpreted given the matched-rate results and the additional training or prior changes in the stronger results?
5 comments
Answer to Reviewer KPd4, part 2/2
4. A refinement of the progressive coding hypothesis. Our results indicate that good progressive coding is not sufficient for good sampling: sampling quality also depends on the prior–posterior gap and on the rate regime.
- At the matched rate, SR-HVAE is a far better progressive coder than the HVAE (no prefix stuck at the data mean, against 4/32 and 12/32; L455–457; Table 12), yet it is not a better sampler.
- Decoding SR-HVAE from its own posterior instead of its prior changes FID from 11.09 to 3.36 on MNIST and from 50.79 to 24.21 on CelebA (Table 2). Much of the remaining gap therefore sits in the prior, not in the coding.
- On MNIST at the matched rate, the DDPM, a good progressive coder by construction, is the weakest sampler of the three (46.75 FID). At it reaches 10.47 (Table 2): the rate itself was starved.
5. Training the all-prefix objective. Shu & Ermon test their hypothesis with a two-stage procedure: two separately trained VDVAEs, a single split, a distortion target on the first stage, and the ELBO on the second. As a result:
- only one intermediate prefix is supervised, and the rate of the second stage is set by the ELBO;
- in their design, stages would require models trained one after another;
- freezing each stage at its own optimum costs nothing only when the source is successively refinable. On a non-refinable source it forces excess rate or distortion on later stages, whereas a joint objective selects a compromise across all prefixes (Prop. 2). On SBGM, SR-HVAE leaves the frontier near and returns to it immediately afterwards (Fig. 1b; L372–376).
We train the all-prefix objective end-to-end, in a single run, at every level (32 on images, 16 on the analytic sources). We evaluate every prefix against an external reference: the exact frontier on the analytic sources and an achievable curve on images (L417–419).
To sum up, Shu and Ermon formalized the HLVM model family, identified a meaningful objective and showed that DDPMs pursue it, while HVAEs do not. We go beyond this theoretically (with SR formulation, DDPM analysis and objective guarantees), insightfully (by showing the practical trade-offs between prescribed and learned encoders) and experimentally (enabling a formulation that trains the objective without freezing intermediate models and on T steps concurrently).
Experimental comparison. We did not compare directly for two reasons. 1: their experiment tests the hypothesis in a different setting (VDVAE model using 2 stage split). 2: our evaluation already isolates the objective: against the parameter-identical ELBO baseline (Table 11), and against the exact frontier on the analytic sources. If after reading our answer the Reviewer still wishes, we can run their two-stage baseline on MNIST with our architecture: their Eq. 3 on the first levels, then frozen, followed by the remaining .
W2 / Q2: interpreting the generation results
At approximately matched rates, SR-HVAE has worse FID than HVAE [...]. Improvements with extended training or prior refitting are encouraging, but do not isolate the benefit of the objective. How should the claim of improved generation be interpreted?
The matched-rate rows are an ablation at the ELBO's own operating point (L414–416), a regime where every model family performs poorly, DDPMs included (L430–431). Choosing the rate is principle (i) of our objective (L217–218). The comparison of the two objectives is therefore at the extended budget, with parameter-identical models (Table 11). There, SR-HVAE reaches 11.09 vs. 21.59 FID on MNIST; on CelebA it has better precision and recall, while its FID is slightly worse (50.79 vs. 48.26).
The prior refit does not confound this. It is applied to both models, with encoder and decoder frozen, and isolates the prior–posterior mismatch (last block of Table 2, L452–453; bold: best):
| MNIST FID ↓ | prec. ↑ | rec. ↑ | CelebA FID ↓ | prec. ↑ | rec. ↑ | |
|---|---|---|---|---|---|---|
| HVAE, prior refit | 23.72 | 0.353 | 0.199 | 48.22 | 0.356 | 0.004 |
| SR-HVAE, prior refit | 6.46 | 0.592 | 0.645 | 43.32 | 0.435 | 0.024 |
SR-HVAE is better on every metric. The same refit does not help the HVAE (L473–476): a better prior pays off only when the hierarchy carries information worth sampling.
We thank the Reviewer again for its constructive comments. We hope our clarifications address the Reviewer's concerns and, if so, that the Reviewer will consider raising the score. We remain at the Reviewer's disposal for any further question, including the MNIST comparison described above.
Follow-up after author response
Thank you for the detailed response and clarifications. I appreciate the improved results when both models' priors are refitted. However, the generation gains before prior refitting remain mixed: SR-HVAE has worse FID at matched rates, and increasing the rate does not consistently improve FID over HVAE across datasets. Together with the missing direct comparison to Shu and Ermon, this leaves me unconvinced that the proposed objective offers a sufficiently strong empirical advantage.
The theoretical analysis provides a useful interpretation, but its additional insight into model behavior and guidance for model design still seem limited. For example, the guarantees hold for arbitrary fixed positive weights and do not establish an advantage of the proposed slope-based weighting. I appreciate the authors' efforts, but these concerns remain, so I am maintaining my weak-reject recommendation.
Answer to Reviewer KPd4, part 1/2
We thank the Reviewer for the careful reading, and for finding the paper clear and sound. In the following, line numbers refer to the revised version of our manuscript (changes in red).
W1 / Q1: contribution over Shu and Ermon (2022)
We thank the reviewer for raising this. We extended Appendix A (Related Works, L675–686) to better highlight the distinction between the two works. Accordingly, we revised the abstract (L25–27), the introduction (L92–93), our contribution II (L104–106), and Section 3 (L303–304) to no longer present the special-case reading of DDPMs as our result. Beyond that correction, several distinction between the two works remain, and we discuss them in detail as follows.
Shu & Ermon formalise HLVMs, call a hierarchy a good progressive coder when every prefix is RD Pareto-optimal, and propose the sum of prefix distortions under cumulative rate budgets (their Eq. 3), of which the DDPM objective is a special case. They test the progressive coding hypothesis with two VDVAEs. While it is true that we address a similar question, several distinctions emerge.
1. When every prefix can be optimal, and what an all-prefix objective achieves. Shu & Ermon note that there is no guarantee that (Rk,Dk) is Pareto-optimal for k<T, and they leave the rate targets as a hyperparameter. Our paper uses SR to answer both points:
2. Design choices with consequences. Eq. 10 differs from their Eq. 3 in two precise ways, backed up by theory.
We additionally measure the effect of a different rate schedule: at a fixed total rate, its allocation alone moves the final FID from 26.6 to 11.1 on MNIST (Fig. 3; L460–462).
3. What the DDPM special case achieves in RD terms. Shu & Ermon show that DDPMs are trained as progressive coders. We show that SR quantifies how good a coder this special case is:
This gives a precise trade-off between a fixed encoder and a learned one (L298–315). The fixed encoder has exact rates and a channel penalty that cannot be recovered. The learned encoder adapts to the source but pays the prior–posterior mismatch of Eq. 8, which can be reduced without touching the hierarchy. Such a distinction between learnable and fixed encoders is not analyzed in Shu and Ermon.