acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Symbol Coupling for Generative Modeling and Symbolic Inference

Abstract

Hierarchical latent variable models provide a promising framework for representing data at multiple levels of abstraction. Their priors, however, typically use Gaussian conditionals, which model dependence across layers while leaving latent units within each layer conditionally independent. Moreover, lower layers can absorb most of the data variation, leaving upper layers weakly informative for semantic abstraction. We propose HiSC-EBM, a hierarchical symbol-coupling energy-based prior. Layer-wise energy terms provide informative corrections to the Gaussian hierarchy, while discrete symbols, such as class labels and attributes, are coupled to selected upper layers and the remaining layers remain available to model residual variation. The resulting joint prior therefore provides a unified formulation in which marginalizing the symbols supports generation, inferring them enables prediction, and conditioning on them allows controllable generation. Because unobserved symbols can be marginalized in closed form, the same formulation accommodates fully labeled, partially labeled, and unlabeled data. The generator, prior, and inference model are learned jointly from scratch within a variational framework. Various experiments and analyses demonstrate the effectiveness of the proposed method.

Primary area:
generative models
Keywords:
Hierarchical latent variable generative model, hierarchical energy-based prior model, hybrid generative model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ASDROpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose to utilize an energy-based model to represent a prior distribution in hierarchical latent-variable models (HiSC-EBM, thought that should read rather as HiSC-EBP). Their key motivation is to improve the poor expressiveness of hierarchical priors that rely on the common (first-order Markov) conditional independence assumption adopted in the individual factors of the standard auto-regressive factorization. They achieve this by augmenting the latent space by a set of categorical latent variables that are pair-wise dependent on the original set of continuous latent variables only in a specific sub-set of the hierarchy and modeling this pair-wise dependency via an EBM modulation (correction) term. The authors then propose a variational learning scheme that is hierarchically similar to the prior, following the standard recipe to amortized posterior inference in combination with Langevin Dynamics to handle sampling from the intractable prior part in the variational objective. The performance of the authors' models is finally demonstrated on three small image datasets in the unsupervised, supervised, and semi-supervised learning regimes.

Critical strengths

The paper is a nice contribution into the rich literature landscape on various flavors of mechanistic instantiation of the standard variational calculus.

The related work is thorough (although more papers from 2025 and 2026 would further strengthen Section 3).

Critical weaknesses

I have only a few rather minor issues (hereafter just to help improve the paper):

The paper uses rather non-standard terminology, e.g., "amortize the posterior sampling", "generation model", "inference model".

Proposition 2 is a well-known standard result that the authors present without acknowledging any reference. This proposition does not have to be in the paper since (10) and (11), and the text following these equations, are clear enough.

The best results in Tables 2 and 3 are not highlighted as in Table 1 (why if you have that elsewhere?).

It is unclear whether the results in Section 4 (apart from Tables 4 and 5) are from a single experimental run or an average over several runs with different random seeds. In certain cases, the values among the compared models are tight and the resulting ranking may flip when more seeds are considered. For example, it is hard to judge the FID score in Table 7 without any uncertainty interval.

The images in Figure 4 are way too small. It is hard to see anything in such small images.

There is no reference to Table 1 in Section 4.1. Is the second line of Table 1 related to the only line in Table 9? How was the prior in Table 1 trained?

Section 5 is very short for its three parts, each would deserve more thorough comments.

Questions for authors

Can you explain why changing the coupling layer in Table 7 has so little impact on the FID score? Can you also explain why there is such a small effect on the accuracy in Table 8 when changing the number of coupling layers?

Reviewer_dwxTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HiSC-EBM, a hierarchical energy-based prior that associates discrete semantic symbols with selected layers of a hierarchical generative model. The proposed prior supports image synthesis, symbolic inference, and conditional generation through a unified formulation. It also allows supervised, semi-supervised, and unsupervised learning by marginalizing unobserved symbols. The model is trained with variational learning, showing evidence of improved generation quality, competitive classification accuracy, hierarchical clustering, conditional generation, and anomaly detection across several image datasets.

Critical strengths

The proposed prior combines discrete symbols with continuous latents for hierarchical generative modeling, resulting in a unified and flexible approach for multiple tasks, including generation, symbolic inference, and conditional generation in supervised/unsupervised settings. This integration is meaningful and allows the multi-task capabilities to emerge from the joint probabilistic formulation.

Critical weaknesses
  1. Although the paper demonstrates unsupervised hierarchical clustering in Table 4 and Figure 6, it remains unclear what semantic information is captured at each layer and whether the relationships between layers form meaningful coarse-to-fine abstractions. The formulation itself doesn't explicitly enforce nested semantic relationships, and the reported results in Table 4 remain substantially lower than TreeVAE. A more systematic analysis of the semantic organization at each level, potentially using the coarse and fine labels of CIFAR-100, would strengthen the claim of hierarchical semantic discovery.

  2. Section 4.4 shows controllable generation through qualitative examples. Since it is one of the main capabilities of the proposed framework, quantitative evaluation, eg., attribute consistency, preservation of upper-layer attributes, and conditional generation quality, would provide stronger evidence of its effectiveness.

Questions for authors
  1. Table 1 reports FID for HiSC-EBM trained with symbolic supervision. Could the authors clarify whether the compared baselines use equivalent supervision? If not, could the authors additionally report the unsupervised HiSC-EBM results across all three datasets to enable a more controlled comparison?

  2. Table 9 shows that symbol coupling improves FID even without supervision. Could the authors clarify the reason for this improvement? In particular, would the improvement remain when comparing against an uncoupled EBM with a similar parameter count?

2 comments

Thanks for your detailed response and additional experiments. Most of my concerns are addressed, and I've increased my rating. One remaining point I'd appreciate clarification on is how the categorical assignments at different layers correspond to meaningful coarse-to-fine semantic relationships, beyond the hierarchical clustering metrics.

Thank you for your positive reassessment. The hierarchy visualizations and class distributions within branches suggest that broader upper-level groups are refined into more class-specific subgroups at lower levels. However, these observations do not establish consistent semantic alignment across layers. We will incorporate the reviewers’ feedback and clarify both the interpretation of the categorical assignments and the limitations of this evidence in the final manuscript. We appreciate your informative and constructive review.

Reviewer_kVzUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HiSC-EBM, a hierarchical symbol-coupled energy-based prior that unifies image generation and symbolic inference within a single probabilistic framework. To address the limited expressiveness of Gaussian hierarchical priors and the lack of semantic information in higher latent layers, the method introduces layer-wise energy corrections and selectively couples discrete symbols, such as class labels and attributes, with upper-level latent representations. The model jointly trains the generator, energy-based prior, and inference network using a variational learning objective, while analytically marginalizing unobserved symbols to support supervised, semi-supervised, and unsupervised learning.

Critical strengths
  • The paper presents a coherent probabilistic framework that integrates hierarchical energy-based priors with discrete symbol coupling. A key strength is that the same model supports image generation, classification, and controllable generation, while analytically marginalizing unobserved symbols to accommodate different supervision settings.

  • The proposed method achieves good generative performance across multiple benchmarks, outperforming the evaluated EBM-based baselines in FID.

Critical weaknesses
  • The proposed framework largely combines existing techniques, including hierarchical EBM priors [1] and symbol-vector coupling [2]. While extending symbol coupling to multiple latent layers is meaningful, the paper does not sufficiently demonstrate its advantages over simpler alternatives. In particular, a controlled comparison against a hierarchical EBM with an auxiliary classification head, under matched architecture and supervision, is missing. This makes it difficult to determine whether the improvements arise specifically from the proposed coupling mechanism.
  • A central motivation of the paper is to encourage meaningful semantic abstraction across latent layers. However, the experimental evidence supporting this claim remains limited. The unsupervised hierarchy discovery results substantially underperform TreeVAE, while the multi-level controllable generation experiments rely primarily on qualitative visualizations. More rigorous quantitative evaluations of semantic disentanglement, attribute preservation, and cross-layer independence are needed to establish whether the proposed approach effectively learns meaningful hierarchical representations.

[1] Cui J, Wu YN, Han T. Learning joint latent space ebm prior model for multi-layer generator. CVPR 2023.

[2] Pang B, Wu YN. Latent space energy-based model of symbol-vector coupling for text generation and classification. ICML 2021.

Questions for authors
  • Could the authors clarify the key technical advances beyond combining these existing frameworks? More importantly, could the authors compare against a hierarchical EBM with an auxiliary classification head under matched architecture, supervision, and parameter count to demonstrate the necessity of explicit symbol coupling?
  • A central motivation is to encourage different latent layers to capture distinct levels of semantic abstraction. However, the unsupervised hierarchy discovery results substantially underperform TreeVAE, and the multi-level controllable generation experiments are primarily qualitative. Could the authors provide quantitative evaluations of semantic disentanglement, attribute preservation, and cross-layer interference to demonstrate that meaningful hierarchical representations are actually learned?
  • In Table 1, the reported generation results are obtained from supervised HiSC-EBM models, while some baselines do not use equivalent label supervision. Could the authors provide comparisons under matched supervision, model capacity, and computational budgets? How much of the reported FID improvement can be attributed specifically to symbol coupling rather than additional supervision?
  • The method requires manually specifying which latent layers are coupled with particular symbols. How sensitive is performance to these assignments, especially when multiple semantic attributes are involved?
  • Proposition 3 establishes optimal prediction under the maximization of the conditional log-likelihood (log⁡p(y∣x)\log p(y\mid x)). However, the actual training objective in Eq. 12 optimizes a variational lower bound involving (Eqϕ(z∣x)[log⁡p(y∣z)]\mathbb{E}{q_\phi(z\mid x)}[\log p(y\mid z)]), which differs from the conditional log-likelihood due to posterior approximation and the Jensen gap. Furthermore, Eq. 33 performs prediction using the posterior mean rather than marginalizing over the latent posterior as assumed in Proposition 3. Could the authors clarify whether the theoretical result extends to the actual training and inference procedures? How significant are these approximation gaps in practice, and how does posterior-mean prediction compare with Monte Carlo posterior marginalization in terms of classification accuracy and calibration?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.