- Rating
- 3
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose to utilize an energy-based model to represent a prior distribution in hierarchical latent-variable models (HiSC-EBM, thought that should read rather as HiSC-EBP). Their key motivation is to improve the poor expressiveness of hierarchical priors that rely on the common (first-order Markov) conditional independence assumption adopted in the individual factors of the standard auto-regressive factorization. They achieve this by augmenting the latent space by a set of categorical latent variables that are pair-wise dependent on the original set of continuous latent variables only in a specific sub-set of the hierarchy and modeling this pair-wise dependency via an EBM modulation (correction) term. The authors then propose a variational learning scheme that is hierarchically similar to the prior, following the standard recipe to amortized posterior inference in combination with Langevin Dynamics to handle sampling from the intractable prior part in the variational objective. The performance of the authors' models is finally demonstrated on three small image datasets in the unsupervised, supervised, and semi-supervised learning regimes.
Critical strengths
The paper is a nice contribution into the rich literature landscape on various flavors of mechanistic instantiation of the standard variational calculus.
The related work is thorough (although more papers from 2025 and 2026 would further strengthen Section 3).
Critical weaknesses
I have only a few rather minor issues (hereafter just to help improve the paper):
The paper uses rather non-standard terminology, e.g., "amortize the posterior sampling", "generation model", "inference model".
Proposition 2 is a well-known standard result that the authors present without acknowledging any reference. This proposition does not have to be in the paper since (10) and (11), and the text following these equations, are clear enough.
The best results in Tables 2 and 3 are not highlighted as in Table 1 (why if you have that elsewhere?).
It is unclear whether the results in Section 4 (apart from Tables 4 and 5) are from a single experimental run or an average over several runs with different random seeds. In certain cases, the values among the compared models are tight and the resulting ranking may flip when more seeds are considered. For example, it is hard to judge the FID score in Table 7 without any uncertainty interval.
The images in Figure 4 are way too small. It is hard to see anything in such small images.
There is no reference to Table 1 in Section 4.1. Is the second line of Table 1 related to the only line in Table 9? How was the prior in Table 1 trained?
Section 5 is very short for its three parts, each would deserve more thorough comments.
Questions for authors
Can you explain why changing the coupling layer in Table 7 has so little impact on the FID score? Can you also explain why there is such a small effect on the accuracy in Table 8 when changing the number of coupling layers?
Thanks for your detailed response and additional experiments. Most of my concerns are addressed, and I've increased my rating. One remaining point I'd appreciate clarification on is how the categorical assignments at different layers correspond to meaningful coarse-to-fine semantic relationships, beyond the hierarchical clustering metrics.