Disentanglement with Holographic Reduced Representations

The process of disentanglement, that focuses on separating the factors of variation in a dataset using neural networks, is a long standing challenge in machine learning. Prior solutions to this problem included the design of variational autoencoders and generative adversarial networks that use concepts from variational inference and information-theoretic constraints in their loss functions, respectively. In contrast to these prior works that rely on continuous representations in their models, we propose a new design that naturally views disentangled representations as symbolic structures due to the compositional nature of the relationship between the concepts that make up a sample from a distribution. However, learning discrete symbolic structures using traditional neural network architectures while maintaining differentiability is challenging, often requiring complex architectures to accomplish. To this end, in this paper, we propose an unsupervised learning algorithm that makes use of holographic reduced representations (HRR) for disentanglement using neural networks. We find that the unbinding operation, defined over HRR vectors, yields a suitable inductive bias for the separation of factors, and yields competitive results compared to other baselines, as measured using latent traversals and disentanglement metrics. We complement these empirical findings with an information-theoretic analysis of the HRR unbinding channel. We prove that unbinding induces approximately independent symbol-value pairs (also called slots) and derive a per-slot capacity bound that quantifies how many distinct symbolic concepts the representation can reliably encode, providing a quantitative account of the inductive bias toward disentanglement. The disentangled representations produced in this process differ from other autoencoder based models in that the individual latent units are vectors themselves which are summed together, differing from the paradigm of latent units behaving as scalar dimensions of low dimensional vectors. We show that this type of HRR representation is more robust to noise than other disentangled representations and can maintain good reconstruction performance across a range of SNRs, while simultaneously gracefully degrading in the recoverable semantic content. Our results thus show that the use of vector symbolic architectures (VSA), such as HRRs, may hold a promising potential for representation learning, paving the way toward exploring new ways in which the symbolic benefits of VSAs can be used to represent data with neural networks.