acceptodds
Under review as a conference paper at ICLR 2027

Disentanglement with Holographic Reduced Representations

Abstract

The process of disentanglement, that focuses on separating the factors of variation in a dataset using neural networks, is a long standing challenge in machine learning. Prior solutions to this problem included the design of variational autoencoders and generative adversarial networks that use concepts from variational inference and information-theoretic constraints in their loss functions, respectively. In contrast to these prior works that rely on continuous representations in their models, we propose a new design that naturally views disentangled representations as symbolic structures due to the compositional nature of the relationship between the concepts that make up a sample from a distribution. However, learning discrete symbolic structures using traditional neural network architectures while maintaining differentiability is challenging, often requiring complex architectures to accomplish. To this end, in this paper, we propose an unsupervised learning algorithm that makes use of holographic reduced representations (HRR) for disentanglement using neural networks. We find that the unbinding operation, defined over HRR vectors, yields a suitable inductive bias for the separation of factors, and yields competitive results compared to other baselines, as measured using latent traversals and disentanglement metrics. We complement these empirical findings with an information-theoretic analysis of the HRR unbinding channel. We prove that unbinding induces approximately independent symbol-value pairs (also called slots) and derive a per-slot capacity bound that quantifies how many distinct symbolic concepts the representation can reliably encode, providing a quantitative account of the inductive bias toward disentanglement. The disentangled representations produced in this process differ from other autoencoder based models in that the individual latent units are vectors themselves which are summed together, differing from the paradigm of latent units behaving as scalar dimensions of low dimensional vectors. We show that this type of HRR representation is more robust to noise than other disentangled representations and can maintain good reconstruction performance across a range of SNRs, while simultaneously gracefully degrading in the recoverable semantic content. Our results thus show that the use of vector symbolic architectures (VSA), such as HRRs, may hold a promising potential for representation learning, paving the way toward exploring new ways in which the symbolic benefits of VSAs can be used to represent data with neural networks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Disentanglement, Deep Learning, Holographic Reduced Representations, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7UKJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper edits the vq-vae architecture to use holographic reduced representations in the latent space of the autoencoder. They claim this vq-vae variant has good disentanglement property. They compare the proposed method with other classical VAE variants and QLAE in disentanglement tasks using disentanglement metrics.

Critical strengths
  1. Combining VSA like HRR into modern popular deep learning structure has big potential. Symbolic system can provide strong representation and reasoning ability, and improve the explainability of modern deep learning structure.

  2. As a method for disentanglement, this paper enforces disentanglement by editing network structure and uses new latent representation, instead of directly adding disentanglement-specific loss term. Actually this paper's loss, except for the original vq-vae loss, is just regularization of the HRR. disentanglement is enforced indirectly and naturally. It is elegant and I like this kind of work.

Critical weaknesses

All the tasks of this paper are traditional disentanglement tasks and can be done by many disentanglement methods like vq-vae, the authors' method just has better performance. As a method combining VSA and vq-vae, the author seems to have not fully utilized HRR's compositional reasoning and compositional representation ability.

If I ignore the potential advantage of VSA and only assess this paper as a disentanglement method, it seems this paper's comparison experiments are not enough and it is hard to evaluate the contribution of this paper. In the main experiment results in Table 2 in the main text, the author compares HRR with beta-VAE (2017), beta-TCVAE (2018), VQ-VAE (2017) and QLAE (2023). The first three methods are very classical methods which were published 8-9 years ago. QLAE is the only one recent work in Table 2 and Table 2 is the main experiment results of this paper. More recent work comparison is needed. For example EncDiff, Bayes-QLAE and so on.

Yang, Tao, et al. "Diffusion model with cross attention as an inductive bias for disentanglement." Advances in Neural Information Processing Systems 37 (2024): 82465-82492.

Patil, Vaishnavi, et al. "A Bayesian Nonparametric Framework For Learning Disentangled Representations." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors

In recent years, for common disentanglement metrics, some work has pointed out that a high disentanglement metric doesn't mean truly disentanglement or controllable generation. Do you think these issues of disentanglement metrics affect your paper?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 7UKJ

We thank the reviewer for their time and feedback. We provide our responses to the reviewer's question and concerns below.

Question 1

In recent years, for common disentanglement metrics, some work has pointed out that a high disentanglement metric doesn't mean truly disentanglement or controllable generation. Do you think these issues of disentanglement metrics affect your paper?

We agree with the reviewer that performance on disentanglement metrics alone is not always indicative of performance on downstream tasks, and it is a gap that applies across works in the field. We measure performance on disentanglement metrics in our paper as they are still important quantitative metrics that the field uses to assess separation of factors in a representation [1-5]. This serves a primary motivation of the paper, which is investigating how VSA encoders, which are typically hand-designed per application, can be learned from data. Such encoders could map high dimensional data to compositional representations that can be manipulated within VSA frameworks. However, a learning algorithm that wishes to do this needs to be able to recognize meaningful structure in the data in order to properly represent it in a compositional representation. We chose disentanglement metrics as they measure what we are interested in, which is the degree to which factors of generation are meaningfully separated across the mm pairs in our learned representations. Thus, our results on the disentanglement metrics are a step towards the broader goal of learning representations that take better advantage of the compositional structures that VSAs can create. We expect that learning to map data to these richer compositional structures from data may require imposing similar VSA-related inductive biases to the ones we have shown in our paper. In particular, modifying the process through which the latent representation gets reconstructed could lead to learning representations with nested structure, such as a filler that is itself a bound set of role-filler pairs, by unbinding in a specific sequence. In such a scenario, disentanglement metrics still provide a way to measure if the representation separates factors in a meaningful way.

[1] Hsu, K., Dorrell, W., Whittington, J. C. R., Wu, J., & Finn, C. (2023). Disentanglement via latent quantization. Thirty-Seventh Conference on Neural Information Processing Systems.

[2] Hsu, K., Hamid, J. I., Burns, K., Finn, C., & Wu, J. (2024). Tripod: Three complementary inductive biases for disentangled representation learning. Proceedings of the 41st International Conference on Machine Learning, ICML’24.

[3] Yang, T., Lan, C., Lu, Y., & Zheng, N. (2024). Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement. Advances in Neural Information Processing Systems 37, 82465–82492.

[4] Patil, V. S., Patil, S., Evanusa, M., Kundu, A. K., Fermuller, C., & JaJa, J. (2026). A bayesian nonparametric framework for learning disentangled representations. The Fourteenth International Conference on Learning Representations.

[5] Sun, B., Pagnucco, M., & Song, Y. (2024). Fully Distributed, Flexible Compositional Visual Representations via Soft Tensor Products. Advances in Neural Information Processing Systems, 37, 139470–139560.

Comparison to Other Recent Works

We note the reviewer's comment that a comparison with more recent work may need to be made in order to evaluate the contributions of our paper. We collected data for three more baselines and post them as a separate comment in the discussion for other reviewers to see as well.

Thanks for the additional experimental results. It address my concern. I will raise my score from 2 to 3.

Reviewer_D1Z6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an unsupervised autoencoder based on Holographic Reduced Representations (HRRs) for learning decoupled visual representations. The model uses fixed symbol vectors to extract slot values from the encoded latent variables, processes these values through a denoising network and vector quantization, and then rebinds them to reconstruct the input. The training process combines reconstruction, quantization, and structural regularization objectives. This paper also analyzes retrieval dependencies and latent capacity from an information-theoretic perspective. Experiments conducted on four visual benchmark datasets evaluate the decoupling performance, robustness to latent noise, compositional generalization ability, and latent component exchange capability; additional ablation experiments and baseline comparisons are provided in the appendix.

Critical strengths
  1. This paper proposes an interesting method for unsupervised decoupling using HRR coupling and decoupling. The model combines these operations with a shared denoising network and vector quantization to retrieve and manipulate individual slots in a latent representation of fixed dimension. Its training objective does not require factor labels or paired examples with known factor variations, making this method a useful approach for learning structured representations.
  2. The main experiments provide encouraging evidence across four visual benchmark tests. As shown in Table 2, among the reported methods, the proposed method achieved the highest average scores on all three InfoMEC metrics and all three DCI metrics, although it did not consistently outperform all baseline methods on every dataset. Results for five initial values are reported (confidence intervals are provided in the appendix), which also helps assess the reliability of these improvements.
  3. The appendices contain additional evaluation results. Appendix H examines the contributions of the codebook generator, the denoising network, quantization, and other design choices, while Table 22 compares HRR with another binding operation under matching capacity budgets. Noise-robustness tests and experiments with various retention factors also go beyond standard decoupling scores, helping to elucidate the actual behavior of the learned representations.
  4. The authors also provide code, which helps reproduce the results presented in the paper and verify the implementation details of the model and the training process.
Critical weaknesses

1.The relationship between Proposition 2 and the implemented model remains unclear. Appendix D.3 introduces independent Gaussian posterior noise to address the singularity of the retrieval covariance, while Section 3.1 describes a deterministic encoding and quantization process. This lower bound on variance is crucial for both the validity and the order of magnitude of the bound, but there is no operation in the described model that ensures its validity. In the absence of such a connection, the proposition pertains only to an auxiliary stochastic model and does not prove that the implemented method possesses the claimed independence property. 2. The current proof does not provide sufficient justification for Proposition 1. Appendix D.6 omits the term O(m2/(t4d))O(m^2/(t^4d)) when combining the upper bounds for each time slot. More fundamentally, interference generated by other input-dependent time slots cannot automatically be treated as independent AWGN. Adjusting its covariance does not establish the required channel model, just as the soft-norm regularization term does not enforce strict power constraints. The discrete bound mlog⁡km\log k remains valid, but the additional continuous capacity bound requires revising the derivation and making explicit assumptions. 3. Even if both of these propositions hold, they do not prove semantic factor separation. Section 3.4 demonstrates that, to reach the capacity limit, different factors must be assigned to different slots; however, independent and non-redundant encodings may still mix factors. For independent, uniform binary factors AA and BB, the representation (A⊕B,B)(A\oplus B,B) (where ⊕\oplus denotes the XOR operation) has independent components and preserves all information about AA and BB, yet its first component mixes the two factors. Furthermore, Proposition 2 only specifies the dependencies among retrievals under given input conditions; it does not clarify how semantic factors are distributed across slots in the data distribution. Therefore, the assertion in Section 4.4 that “single-factor editing can be derived from Proposition 2” requires further justification. 4. The averaged compositional generalization scores do not fully reflect the weak probe performance on some factors directly involved in the withheld combinations. In Tables 30–31, HRR’s object-hue probe score drops from 0.8540.854 on in-distribution examples to 0.1780.178 on held-out combinations. Table 32 reports a clipped R2R^2 of zero for horizontal axis, despite an overall average score of 0.7200.720. These results support improved average linear decodability, but provide limited evidence of reliable recovery of all factors involved in the unseen combinations. It remains unclear whether these low scores reflect limitations of the learned representations or the probing protocol.

Questions for authors

1.How is the positive posterior variance floor required by Proposition 2 realized in the implemented model? 2.Can the authors justify the AWGN and power assumptions in Proposition 1 and provide a bound that retains the conditional dependence correction? 3.What additional argument connects the capacity bound and conditional retrieval independence to semantic factor alignment and single-factor edits? 4/What evidence distinguishes representation limitations from probing limitations for the low held-out scores on object hue and horizontal axis?

3 comments
AuthorsOpenReview ↗

Response to Reviewer D1Z6, Part 1

We thank the reviewer for their detailed feedback and comments on the theoretical contributions of our paper. We provide our responses to their questions below, in two parts.

Question 1

How is the positive posterior variance floor required by Proposition 2 realized in the implemented model?

Our implemented model is deterministic, as Section 3.1 describes, so the variance floor is not present in our implementation. The t2t^2 floor is a regularization used in the analysis to complete the proof. Proposition 2 describes a stochastic model of the unbinding step in which the symbol vectors are drawn from the HRR distribution, as done in our implementation, and where the slot values are held fixed. In the stochastic model, the joint covariance of the mm retrievals is singular so the expression for their total correlation diverges. Adding independent Gaussian noise of variance t2t^2 to every retrieval coordinate makes the covariance invertible and allows us to derive the stated bound. The proposition is meant to characterize the statistical dependence among the retrievals of a bundle of role-filler pairs under unbinding. We will rewrite Proposition 2 and its corresponding proof to state the floor as an assumption that is made for the analysis.

Question 2

Can the authors justify the AWGN and power assumptions in Proposition 1 and provide a bound that retains the conditional dependence correction?

We agree with the reviewer on their described points in W2, and have restated Proposition 1 so that each bound states the assumptions.

As mentioned by the reviewer, the discrete bound remains valid. For any encoder and any fixed symbol vectors,

I(x;z^)≤H(v^1,…,v^m)≤∑i=1mH(v^i)≤mlog⁡k.I(\mathbf{x};\hat{\mathbf{z}}) \le H(\hat{\mathbf{v}} _ 1,\dots,\hat{\mathbf{v}} _ m) \le \sum _ {i=1}^{m} H(\hat{\mathbf{v}} _ i) \le m\log k .

The continuous bound is stated under two assumptions on the retrieval v~i=vi(x)+ϵi\tilde{\mathbf{v}} _ i = \mathbf{v} _ i(\mathbf{x}) + \boldsymbol{\epsilon} _ i, where ϵi\boldsymbol{\epsilon}_i is the retrieval error of slot ii.

(a)Ex∥vi(x)∥2=Pfor every i=1,…,m,\text{(a)}\qquad \mathbb{E} _ {\mathbf{x}} \Vert \mathbf{v} _ i(\mathbf{x}) \Vert^2 = P \qquad \text{for every } i=1,\dots,m ,

(b)ϵi∼N(0,mPdId) and ϵi is independent of xfor every i=1,…,m.\text{(b)}\qquad \boldsymbol{\epsilon} _ i \sim \mathcal{N}\Bigl(\mathbf{0}, \tfrac{mP}{d} \mathbf{I} _ d\Bigr) \text{ and } \boldsymbol{\epsilon} _ i \text{ is independent of } \mathbf{x} \qquad \text{for every } i=1,\dots,m .

Assumption (a) gives the mm slot values a common expected squared norm. Assumption (b) replaces the retrieval error with Gaussian noise that is independent of the input, where mP/dmP/d is the leading-order per-coordinate variance of the retrieval error in the retrieval analysis. The retrieval error in the trained model is cross-talk from the other input-dependent slots, as the reviewer notes, so (b) is a modeling assumption. Under (a) and (b) each slot satisfies I(x;v^i)≤d2log⁡(1+1m)I(\mathbf{x};\hat{\mathbf{v}}_i)\le \tfrac{d}{2}\log\bigl(1+\tfrac{1}{m}\bigr). Furthermore,

I(x;z^)≤min⁡(md2log⁡(1+1m)+Δ,mlog⁡k),Δ=∑i=2mI(v^i;V^<i∣x).I(\mathbf{x};\hat{\mathbf{z}}) \le \min\Bigl( m \tfrac{d}{2}\log\bigl(1+\tfrac{1}{m}\bigr)+\Delta, \quad m\log k \Bigr), \qquad \Delta=\sum _ {i=2}^{m} I\bigl(\hat{\mathbf{v}} _ i;\hat{\mathbf{V}} _ {\lt i}\mid\mathbf{x}\bigr) .

The term Δ\Delta is the conditional dependence correction from the proof for Proposition 1.

AuthorsOpenReview ↗

Response to Reviewer D1Z6, Part 2

Question 3

What additional argument connects the capacity bound and conditional retrieval independence to semantic factor alignment and single-factor edits?

Semantic factor alignment and single-factor edits were empirical observations that we made throughout the experiments that we ran, and these observations are supported through the results presented in Table 2 and Figure 2. Table 2 in particular shows that our method performs best on aggregate on the InfoMEC and DCI metrics which together support semantic factor alignment. However, the reviewer is correct that independent slots can still mix factors. We thus acknowledge that the statements in Section 3.4 and Section 4.4, on reaching the capacity limit and on where single-factor editing arises from, are not established by Proposition 1 and Proposition 2. We will remove the claim that reaching the capacity limit requires distinct factors in distinct slots, and we will present factor alignment and single-factor edits as empirical observations.

Question 4

What evidence distinguishes representation limitations from probing limitations for the low held-out scores on object hue and horizontal axis?

Our paper's compositional generalization study focuses on information that is linearly decodable from a latent representation. This purposefully limits the expressivity of the probe to instead focus on the extent to which information about the encoded object can be recovered by a linear map. Thus, our relatively unexpressive probes are a designed component of our experiments as we are interested in linear decodability.

On the horizontal_axis attribute on MPI3D-C we see that our method, as well as all the baselines, perform poorly on the held out images as evidenced by the mean scores of zero across that row. However, Table 32 also shows that even on the held-out images, our model is able to best linearly encode information about shape even at horizontal_axis combinations that it didn't see during training, across all evaluated models, attaining a higher mean accuracy score than the next best performing baseline by 5 percentage points (0.879 against 0.829). We see a similar pattern for Shapes3D, where Table 30 also shows that our model is able to attain the highest accuracy on predicting the shape, even at unseen combinations with object_hue.

We take the low held-out scores as evidence that the evaluated methods do not reliably make information about object_hue and horizontal_axis linearly decodable on the withheld combinations. Although the averaged scores do not isolate the performance on these two factors, they still reflect that our model is able to best linearly encode factors amongst the baselines and, as discussed, is best able to correctly linearly encode other attributes (such as shape and object_shape) even when paired with factors for combinations it has not seen previously.

Thank you for your response. I appreciate your efforts to address my concerns, and some of the clarifications are helpful. However, I believe these revisions also weaken the paper's original theoretical contributions. In particular, my concern regarding Q1 remains unresolved. The authors acknowledge that the implemented model is deterministic and does not satisfy the positive variance assumption required by Proposition 2. This assumption is not merely a technical convenience, as it is essential for establishing the conditional independence bound. Consequently, the proposition characterizes an auxiliary stochastic model rather than the actual implementation, and its relevance to the proposed method remains unclear. Furthermore, the acknowledgment that Propositions 1 and 2 do not establish semantic factor separation further limits the claimed theoretical contribution. While I recognize the empirical merits of this work, these concerns remain significant, and I will maintain my original rate.

Reviewer_dqwaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary This work focuses on disentanglement in the discrete, symbolic setting. The nondifferentiable nature of symbolic structures makes disentangled representation learning challenging. The authors approach this using holographic reduced representations (HRR) with neural networks. In particular, the unbinding operation of HRR vectors is a suitable inductive bias for the separation of factors. Their experiments and analyses indicate that unbinding induces disentanglement (independence). The paper provides the following contributions: a method that learns HRR-structured latent representations without supervision, and an explanation of why HRR structure should produce disentanglement via a theoretical upper bound on latent information.

The introduction covers the related work well. There is sufficient coverage of traditional disentanglement methods (i.e., in AE/VAE) and VSA based methods. The preliminaries are well explained and clear.

The authors propose a CNN-based autoencoder which maps x -> z. It includes an HRR unbinding and vector quantization step. The symbol vectors are randomly initialized and remain frozen during training. The codebook is initialized randomly but is not frozen. The latent z is unbound by each of the frozen symbol vectors, yielding estimates of the values. VQ is used to produce the final value vectors, where each v_i is in the codebook. The “denoised retrievals” are then rebound with their original symbols, and the decoder attempts to reconstruct the original input x.

The authors define a structural regularizer to the latent, the denoised values, and the codebook. The regularizer is intended to cause V and C to reproduce per-component variance of HRR vectors and to cause z to match the expected squared norm and per-component variance of the sum of HRR pairs. They use the straight-through estimator to propagate gradients through the VQ step. The method adopts some VQ-VAE losses and has 6 loss terms in total, with 5 hyperparameters. The latent channel is modeled as a Gaussian-corrupted channel, and this permits an upper bound on the information of x and the latent. The authors provide good intuition on the sources of information bottleneck as they relate to the codebook size and the slot-dimension tradeoff. Proposition 2 connects the HRR approach to factorization of the posterior.

After hyperparameter tuning, the method works well in disentangling 5 datasets, as measured by DCI-disentanglement and InfoMEC. The latents are robust to noise, akin to VQ-VAE. Activation swapping experiments indicate that HRR is effective in disentangling latent features. The probing experiments indicate that the HRR approach can generalize encodings to unseen combinations better than previous approaches.

Critical strengths

(1) Very clear paper that is elegantly presented

(2) The mathematical foundations are solid and well-written

(3) Adequate related work and introduction to the problem

(4) The experiments, while sparse in some regards, indicate that the method is promising and useful

Critical weaknesses

(1) The unsupervised disentanglement evaluation is relatively sparse in terms of baselines. beta-vae is very old and is known to have excessive capacity issues. One could also try methods like FactorVAE, NashAE, etc

(2) The hyperparameter space for the method is high dimensional, and the hyperparameter search for HRR requires far more resources (and supervision, which somewhat defeats the point of unsupervised disentanglement).

(3) The choice of m appears to depend on the ground truth number of factors. In real settings, this is typically unknown.

(4) Missing comparison of training cost of each method (incl. hyperparam optimization). It is possible I could have missed this. Wall-clock time experiments would do.

Questions for authors

Corresponding to "weaknesses" section:

(2) Do the hyperparameters optimized for one set work well on a holdout set? For example, do they work well on a dataset that was not included in the hyperparameter search?

(3) How well do hyperparameters transfer across datasets when m is kept fixed?

(4) What is a comparison of training time/expense for HRR and the baselines?

2 comments
AuthorsOpenReview ↗

Response to Reviewer dqwa

We thank the reviewer for their time and feedback on our experiments. We provide our responses to the reviewer's questions and concerns below.

Question 1

Do the hyperparameters optimized for one set work well on a holdout set? For example, do they work well on a dataset that was not included in the hyperparameter search?

We followed the protocol of prior work [1] and ran a hyperparameter sweep separately on each dataset, for our model and for every baseline. For our model the four searches led us to a single configuration which we use on all four datasets, apart from mm. Our model was trained with the same model architecture and optimizer parameters, for the same number of gradient steps, with the same weighting for the loss term regularizers. The only hyperparameter that was varied between datasets was mm, analogous to the number of scalar dimensions for the latent representation in most disentanglement works. We refer the reviewer to Table 5 and Table 6 of the appendix for details on the hyperparameters that were used across all datasets.

In our paper, Table 21 demonstrates the effect of varying mm across individual datasets. However, we had not investigated the effect of using a constant mm across different datasets. To explore this, we are conducting an experiment in which we set mm at 3 values, m∈9,10,14m \in \\{9, 10, 14\\}, for each dataset. This set ranges from the lowest to the highest value of mm that was used for the original experiments. We are currently running those experiments and will follow up with the results soon.

Question 2

How well do hyperparameters transfer across datasets when m is kept fixed?

With our interpretation of the reviewer's second question, we believe that it is answered in Question 1. Though we encourage the reviewer to tell us if it is not answered.

Question 3

What is a comparison of training time/expense for HRR and the baselines?

The table below shows a comparison of training times between our model and the baselines in the paper, when run on an NVIDIA L40S, measured in hours. We also include training time for a new baseline, EncDiff [2], that we included after receiving the reviews. Due to constraints on computational resources we are currently only able to provide data on Shapes3D and MPI3D-C for EncDiff, but we can provide a more complete set of results for a final version of our paper.

ModelShapes3DMPI3D-CFalcor3DIsaac3D
beta-VAE1.080.960.941.11
beta-TCVAE0.980.980.981.13
VQ-VAE0.981.011.011.10
QLAE0.980.980.981.07
HRR (ours)1.161.171.171.27
EncDiff6.576.57----

References

[1] Hsu, K., Dorrell, W., Whittington, J. C. R., Wu, J., & Finn, C. (2023). Disentanglement via latent quantization. Thirty-Seventh Conference on Neural Information Processing Systems.

[2] Yang, T., Lan, C., Lu, Y., & Zheng, N. (2024). Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement. Advances in Neural Information Processing Systems 37, 82465–82492.

Results on Additional Baselines

We note the reviewer's concern that our paper did not include sufficient baselines. We collected results for three additional baselines which we include in a separate comment in this discussion.

AuthorsOpenReview ↗

Response to Reviewer dqwa, Follow up to Question 1

As discussed in the first part of our response, we provide the results for our experiment that varies mm across individual datasets in the table below, in which we report mean InfoM across five seeds. We see that varying mm only results in small differences in InfoM in each row, indicating that the metric is not overly sensitive to mm.

Datasetm = 9m = 10m = 14
Shapes3D0.827 ± 0.0580.857 ± 0.0810.842 ± 0.051
Falcor3D0.649 ± 0.0760.674 ± 0.0480.660 ± 0.041
Isaac3D0.720 ± 0.0680.731 ± 0.0520.716 ± 0.057
MPI3D-C0.510 ± 0.1240.527 ± 0.0410.549 ± 0.044

InfoM, mean ± 95% half-width over five seeds. Bold marks the m used in the paper for the respective dataset.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Results on Additional Baselines

Additional Baselines

In addition to the results in our main paper, we also collected results on three more baselines: FactorVAE [1], NashAE [2], and EncDiff [3]. We noted Reviewer 7UKJ's suggestion of Bayes-QLAE [4] but did not find their code to be publicly available. We are currently in touch with the main author of the paper who has notified us that they are on travel for the next few days. We will try to include those results when they are available.

For FactorVAE and NashAE we conducted a hyperparameter search and collected results with the same protocol used for our paper. For EncDiff, we were not able to run a new hyperparameter sweep given the computational expense of training a single model, thus we use the released hyperparameter configuration for their model. We are currently only able to provide complete results for Shapes3D and MPI3D-C, due to constraints on computational resources, but are awaiting results for Isaac3D and Falcor3D.

The values in the tables are the mean and 95% confidence intervals over five seeds. We see that our method performs best on InfoM for Shapes3D, and is within the confidence intervals of the baselines on InfoM for MPI3D-C. Our method also attains the best DCI-D values for both datasets.

Shapes3D

ModelInfoMInfoCInfoEDCI-DDCI-IDCI-C
FactorVAE0.644 ± 0.0780.483 ± 0.0700.819 ± 0.1290.687 ± 0.1040.999 ± 0.0030.576 ± 0.091
NashAE0.673 ± 0.0720.481 ± 0.0770.951 ± 0.1130.591 ± 0.1320.965 ± 0.0430.499 ± 0.106
EncDiff0.585 ± 0.0460.342 ± 0.0270.998 ± 0.0040.675 ± 0.0281.000 ± 0.0010.472 ± 0.020
HRR (ours)0.850 ± 0.0220.609 ± 0.0290.999 ± 0.0010.867 ± 0.0280.998 ± 0.0020.400 ± 0.011

MPI3D-C

ModelInfoMInfoCInfoEDCI-DDCI-IDCI-C
FactorVAE0.524 ± 0.0500.475 ± 0.0280.560 ± 0.0590.446 ± 0.0500.799 ± 0.0250.423 ± 0.044
NashAE0.543 ± 0.0870.476 ± 0.1170.513 ± 0.1060.281 ± 0.0730.732 ± 0.0390.277 ± 0.064
EncDiff0.512 ± 0.0320.353 ± 0.0820.768 ± 0.0390.509 ± 0.0300.857 ± 0.0140.401 ± 0.020
HRR (ours)0.531 ± 0.0340.535 ± 0.0420.853 ± 0.0100.545 ± 0.0140.783 ± 0.0100.346 ± 0.021

[1] Kim, H., & Mnih, A. (2018). Disentangling by factorising. In J. Dy & A. Krause (Eds.), Proceedings of the 35th international conference on machine learning (Vol. 80, pp. 2649–2658). PMLR.

[2] Yeats, E., Liu, F., Womble, D., & Li, H. (2022). NashAE: Disentangling Representations Through Adversarial Covariance Minimization. In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, & T. Hassner (Eds.), Computer Vision – ECCV 2022 (pp. 36–51). Springer Nature Switzerland.

[3] Yang, T., Lan, C., Lu, Y., & Zheng, N. (2024). Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement. Advances in Neural Information Processing Systems 37, 82465–82492.

[4] Patil, V. S., Patil, S., Evanusa, M., Kundu, A. K., Fermuller, C., & JaJa, J. (2026). A Bayesian nonparametric framework for learning disentangled representations. The Fourteenth International Conference on Learning Representations.

AuthorsOpenReview ↗

Summary of the Discussion Period

We would like to thank all of the reviewers for their feedback and comments which have improved our paper. To close out the discussion period we would like to provide a recap of some of the points that were addressed.

Experimental Additions

Additional Baselines

Reviewers dqwa and 7UKJ both mentioned that our paper could benefit from including additional baselines for a broader comparison with more prior works. Following this feedback, we implemented three additional baselines, namely FactorVAE [1], NashAE [2], and EncDiff [3]. We noted Reviewer 7UKJ's particular suggestion for including Bayes-QLAE [4] but we were unable to reproduce their published results and have reached out to the authors for help with this. We shared the results in a separate comment in this discussion where we showed that our method remained competitive in both the InfoMEC and DCI disentanglement metrics.

Hyperparameter Transfer and Training Time

Reviewer dqwa had questions regarding how well the hyperparameters for our model transferred across datasets, and how long our method took to train relative to the baselines. We pointed out that our approach keeps all hyperparameters fixed across datasets with the exception of mm, which serves a purpose analogous to the number of scalar entries in representations from other related works. Following the protocol from prior disentanglement work, this parameter is varied per the number of ground truth factors in each dataset. Additionally, we provided a table on wall-clock training time comparisons, showing that our method only incurs a modest increase to training time and is still far below more expensive diffusion based approaches such as EncDiff. We also provided a table of InfoM for our model trained with m=9m = 9, 1010 and 1414 on each dataset.

Clarification of Theoretical Results and Experimental Framework

Reviewer D1Z6's questions regarding our theoretical contributions led us to clarify the specific assumptions behind some of our results, such as the AWGN and power assumptions underlying Proposition 1, which we will include as revisions in a final version of our paper. We also addressed the reviewer's concern regarding our compositional generalization study by explaining that our experiments focus on information that is linearly decodable, which is why we use linear probes.

Paper Positioning

Finally, Reviewer 7UKJ brought up a concern regarding how recent results on disentanglement metrics may affect our paper. Acknowledging that good disentanglement metrics alone are not enough to guarantee good performance on some downstream tasks, we explained that disentanglement metrics are still a good quantitative measure of the separation of factors within a representation. This serves a broader motivation for our paper, learning a mapping to compositional structures directly from data, which the vector symbolic architecture (VSA) field has historically relied on hand-designed encoders for. Within such a compositional representation, disentanglement metrics still provide a way to measure if factors are meaningfully separated throughout it.

References

[1] Kim, H., & Mnih, A. (2018). Disentangling by factorising. In J. Dy & A. Krause (Eds.), Proceedings of the 35th international conference on machine learning (Vol. 80, pp. 2649–2658). PMLR.

[2] Yeats, E., Liu, F., Womble, D., & Li, H. (2022). NashAE: Disentangling Representations Through Adversarial Covariance Minimization. In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, & T. Hassner (Eds.), Computer Vision – ECCV 2022 (pp. 36–51). Springer Nature Switzerland.

[3] Yang, T., Lan, C., Lu, Y., & Zheng, N. (2024). Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement. Advances in Neural Information Processing Systems 37, 82465–82492.

[4] Patil, V. S., Patil, S., Evanusa, M., Kundu, A. K., Fermuller, C., & JaJa, J. (2026). A bayesian nonparametric framework for learning disentangled representations. The Fourteenth International Conference on Learning Representations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.