acceptodds
Under review as a conference paper at ICLR 2027

Winner-Take-All bottlenecks enforce disentangled symbolic representations in multi-task learning

Abstract

Winner-take-all (WTA) networks constitute a central circuit motif in cortical networks of the brain. In addition, WTA-like activations are abundant in modern deep learning models in the form of the softmax activation for example in attention layers of transformers. While their role in the extraction of latent factors has been studied for relatively simple generative models, their role in the context of highly non-linearly entangled latent factors has remained elusive. In this article, we show that a WTA bottleneck within a deep neural network can enforce under certain well-defined conditions the extraction of categorical latent factors of the data in a multi-task learning setup. In particular, we prove that under specific conditions, including that the structure of the WTA bottleneck matches the structure of the latent factors and the model solves all tasks perfectly, the emerging representation is a structured permutation of those latent factors. When the structures do not match, we empirically demonstrate on two datasets that a symbolic representation often emerges when the multi-task learning objective is optimized close to its global optimum. This representation is a weaker form of structured permutation in which multiple WTA output neurons may jointly encode a single latent factor category. We additionally evaluate the generalization behavior of the acquired symbolic representation. Our proposed model provides insights into the generalization capabilities of deep neural networks with WTA-like components and may serve as an interface between symbolic and subsymbolic AI systems.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Learning, Multi-Task Learning, Disentangled Representations, Winner-Take-All Networks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_EQFBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a network trained on many tasks at once can be made to recover the discrete factors that generated its inputs. Each input is produced from a handful of categorical factors (for example shape, colour, position) by an unknown nonlinear mixing, and each task is a binary classification whose label probability depends on those factors through a logistic-linear rule. The proposed network has a bottleneck made of several winner-take-all (WTA) groups: in each group exactly one unit is active, so the bottleneck is a set of discrete choices, and a linear layer reads the task predictions off it (Sec. 3, Fig. 1a).

The main result is Theorem 1 (proof in App. A.2). If the network solves all tasks exactly, there are at least as many linearly independent tasks as categories, and the WTA groups have the same sizes as the true factors, then the bottleneck must equal the true factors up to relabelling: each group encodes one factor and each unit one category.

The experiments test this, and then what happens when the assumptions are relaxed, with 20 training runs per setting. On synthetic data with five factors of five categories each and a matching bottleneck, most runs solve the tasks essentially perfectly (15 of 20 with 30 tasks), and every such run recovers the factors as the theorem predicts (Sec. 5.1). When the bottleneck does not match the factors, which is the realistic case since the number of categories is not known in advance, the paper looks for a weaker "symbolic" code in which several units may jointly stand for one category. This often appears but not reliably: in 17 of 20 runs with five groups and 6 of 20 with seven groups, rising to 14 of 20 when the readout weights are ℓ1\ell_1-regularised (Sec. 5.2). A coloured variant of dSprites with 1920 images gives a similar picture (9 of 20 runs with four groups, 15 of 20 with six; Sec. 5.4). Finally, a new task is learned from fewer examples, and generalises better to held-out factor combinations, when a classifier is trained on the bottleneck code than when it is trained on the raw inputs (Sec. 5.3, Fig. 2b–c).

Critical strengths

1. An exact, checkable identifiability statement for categorical latents. Under its assumptions, Theorem 1 gives recovery of the latents themselves up to relabelling, which is stronger than the linear decodability that the same multi-task argument yields without the bottleneck (Eqs. 8–11). I followed Lemma 1 and the proof in App. A.2 and found no error: the steps C=DQ+1b⊤C=DQ+\mathbf 1\mathbf b^\top, b=0\mathbf b=\mathbf 0, one block per column, bijection between blocks, and the final counting argument on block sizes all hold. This matters for my assessment because it is the paper's main claim and it is correct as stated.

2. Transparent reporting that lets the reader see where the claim stops. Every setting is run with 20 seeds and the per-run MAE and number of recovered categories are tabulated (Tables A3–A7, A14–A16, A20). The counts and percentages quoted in the text agree with those tables. The paper also states its failure modes openly (L246–249, L372–373, L442–443), and it reports the non-WTA control for the generalisation experiment in the appendix (Tables A8–A10).

3. Very clear presentation. A reader can reconstruct exactly what was done - the setup is fully defined, the theorem is stated precisely, and the text states its own failure modes.

Critical weaknesses

1. The guarantee holds in a narrow regime in which the supervision already determines z\mathbf z, and the evidence outside that regime is small-scale and mixed.

  • Strength of supervision. The targets are exact, noise-free posteriors σ(Wz)\sigma(W\mathbf z) (L269–270, L942–946), i.e. n≥ln\ge l generic real-valued linear functions of the one-hot code, from which z\mathbf z follows by a fixed linear map (Eq. 10). The linear-decodability step is taken from Vafidis et al. (L199), so the new part is the combinatorial refinement in Theorem 1, where Φ\Phi enters only through injectivity. The theorem further needs an exactly optimal predictor on all p=∏ilip=\prod_i l_i latent combinations and known numbers of latents and categories.
  • No guarantee where it would be used. The paper notes that category counts are unknown in practice (L287–288), and there the result is empirical only. Perfect task performance does not imply a symbolic code in that case (Table A5 runs 9 and 19; Table A6, where 12 of 18 runs with MAE ∼10−8\sim10^{-8} are not fully symbolic). Success rates range from 6/20 to 17/20 with the number of heads, and the ℓ1\ell_1 runs that help have MAE ≈10−3\approx10^{-3}, outside the paper's own "perfectly solved" criterion (Tables A7, A16).
  • Scale. One synthetic MLP mixing and a 1920-image dSprites subset, with learning rate and temperature set per setup (Table A1) and no sensitivity analysis or competing method.

2. Nothing in the paper shows a benefit that is specific to the WTA bottleneck or to the symbolic format. The abstract closes by claiming "insights into the generalization capabilities of deep neural networks with WTA-like components" and an interface to symbolic AI. Sec. 5.3 and Fig. 2b–c compare a readout on z^\hat{\mathbf z} only with a readout on the raw input x\mathbf x. The readout on the multi-task representation a\mathbf a of the same network without WTA heads is in the appendix only, and it performs the same: on the random split with 100 samples, test AUC 0.992 [0.988–0.994] (Table A8) versus 0.992 [0.985–0.995] with WTA (Table A11); Tables A9–A10 versus A12–A13 give 0.990 vs 0.989 and 0.965 vs 0.959 (the captions identify A9–A10 as MLPa\mathrm{MLP}_{\mathbf a}; the z^\hat{\mathbf z} headers in the test block of A9 and in A10 look like a typo). What the experiments establish is therefore parity: the generalisation gain comes from multi-task pre-training (as in Johnston & Fusi, 2023), and discretising the bottleneck does not lose it. No experiment uses a task for which an axis-aligned discrete code is needed, so the neurosymbolic motivation is untested. Likewise, the softmax/attention motivation of the abstract and introduction is at odds with L246–249, where softmax gates are reported to give "blurry" outputs that were insufficient.

Questions for authors

Clarifications

  1. Is there any experiment in which the WTA/symbolic representation outperforms the non-WTA multi-task representation a\mathbf a (Tables A8–A10 vs A11–A13)?
  2. In Sec. 5.3 and 5.4, was the multi-task encoder trained on latent combinations that later appear in Ztest\mathcal Z_{\text{test}} of the downstream task (App. A.3 suggests uniform sampling over all of Z\mathcal Z)? If so, combinations described as "never been observed during training" (L402–403) and "unseen" (Fig. 2c) were seen by the encoder; does the symbolic code still emerge, and generalise, when they are withheld from encoder training as well?
  3. Do the results hold when the targets are binary labels sampled from P(Ai∣z)P(A_i\mid\mathbf z), as a learner would observe them, instead of the exact posterior probabilities?
  4. The theory requires rank⁡(W)=l\operatorname{rank}(W)=l, hence n≥l=25n\ge l=25 (L456–458). Because each of the m=5m=5 one-hot blocks of z\mathbf z sums to one, the codes span only l−m+1=21l-m+1=21 dimensions, so injectivity would need only 21 tasks, yet structured permutations emerge with 20 and 15 tasks (Table A4), below both bounds. Is recovery in practice driven by the discreteness of the bottleneck rather than by the linear-decoding step (Eqs. 8–11), and can the theorem be stated under the weaker condition?
  5. How entangled is Φ\Phi? Please report the accuracy of a linear probe for z\mathbf z from x\mathbf x, and the softmax-gate results mentioned at L246–249, as baselines.

Suggestions for future work

  • A characterisation of when a symbolic representation is guaranteed in the unmatched case (for example via the readout sparsity explored in Sec. 5.2), and for partially observed latent combinations.
  • A positioning against work that uses a discrete bottleneck as the inductive bias for disentanglement, which the related-work section does not cover (e.g. Hsu et al., "Disentanglement via Latent Quantization", NeurIPS 2023, arXiv:2305.18378, where each latent dimension is quantised to a small learned codebook). Earlier empirical work on disentanglement emerging from multi-task training (e.g. Maziarka et al., ECML PKDD 2022, arXiv:2110.03498) could also be discussed alongside Johnston & Fusi (2023) and Vafidis et al. (2025) at L084–093.
13 comments

We split the remarks into sub-points to facilitate readability of the response and pair them with related points in the "Questions For Authors" section.

R3W1a: Strength of supervision. The targets are exact, noise-free posteriors σ(Wz)\sigma(W\mathbf{z}) (L269–270, L942–946), i.e. n≥ln \geq l generic real-valued linear functions of the one-hot code, from which z\mathbf{z} follows by a fixed linear map (Eq. 10). The linear-decodability step is taken from Vafidis et al. (L199), so the new part is the combinatorial refinement in Theorem 1, where Φ\Phi enters only through injectivity. The theorem further needs an exactly optimal predictor on all p=∏ilip=\prod_i l_i latent combinations and known numbers of latents and categories.

R3Q3: Do the results hold when the targets are binary labels sampled from P(Ai∣z)P(A_i \vert \mathbf{z}), as a learner would observe them, instead of the exact posterior probabilities?

Answer to R3W1a and R3Q3: We agree that the influence of noise on the supervision targets is an interesting question. Binary labels as proposed in Q3 are also quite natural and can be viewed as a very specific noise on the targets (i.e., quantization noise). We therefore conducted new experiments with the visual entanglement (dsprites) where the supervision was performed with binary classification targets instead of posterior probabilities (binarization threshold 0.50.5). We found that in 99 out of 2020 runs, a fully symbolic representation emerged. On average 24.824.8 out of 2929 categories were encoded in the bottleneck. 28.5728.57 categories were encoded in the bottleneck on average over runs with an MAE of at most 0.040.04.

The mean MAE on the test set was high (0.0540.054) as expected, since the 00 and 11 target values could not be represented with the sigmoid activation functions. These results were achieved using 200 tasks instead of the 50 tasks in the main experiment, indicating that the information loss due to target binarization can be compensated for by an increased number of tasks.

We report these results in Appendix A.12 and incorporated a short summary at the end of Section 5.4.



R3W1b: No guarantee where it would be used. The paper notes that category counts are unknown in practice (L287–288), and there the result is empirical only. Perfect task performance does not imply a symbolic code in that case (Table A5 runs 9 and 19; Table A6, where 12 of 18 runs with MAE ∼10−8\sim 10^{-8} are not fully symbolic). Success rates range from 6/20 to 17/20 with the number of heads, and the ℓ1\ell_1 runs that help have MAE ≈10−3\approx 10^{-3}, outside the paper's own "perfectly solved" criterion (Tables A7, A16).

Answer to R3W1b: It is correct that perfect task performance does not always imply the emergence of a complete symbolic representation in the unmatched setting where our theory does not directly apply. However, we found that it does lead to at least a nearly complete symbolic representation in which almost all latent categories are correctly encoded. We noted this at lines 320-322 (recovery rate of categories 94.3394.33% over all runs and 99.399.3% for perfect runs; perfect run if MAE$<10^{-6}$ after multi-task learning) and lines 357 (recovery rate 87.1787.17% over all runs and 97.0497.04% for perfect runs; the latter number was added in the revised version). Excluding runs with an MAE greater than 0.010.01, the average category recovery rate across Tables A5 (generic MLP entanglement, 5 WTA heads), A6 (generic MLP entanglement, 7 WTA heads), A14 (visual entanglement, 4 WTA heads), and A15 (visual entanglement, 6 WTA heads) is 97.597.5%, with the lowest rate being 95.495.4%. Notably, here, recovery is evaluated using a strict exact-match criterion: a latent category is considered recovered only if a set of WTA bottleneck output neurons or one neuron encodes it correctly across all samples. Even a single mismatch causes the entire category to be counted as not recovered and prevents a complete symbolic representation.

An universal threshold guaranteeing a nearly complete symbolic representation is difficult to establish in practice due to noise in inputs and targets, as well as regularization. Thus, a rule of thumb is more appropriate (see our answer to R2W1). First, the maximum achievable performance is identified. Then, the smallest possible bottleneck is selected without substantially compromising performance.

R3W1c: Scale. One synthetic MLP mixing and a 1920-image dSprites subset, with learning rate and temperature set per setup (Table A1) and no sensitivity analysis or competing method.

Answer to R3W1c: We note that our experiments were performed to test our main contribution, the theoretical result, and to investigate empirically how this result extends to scenarios that do not match its preconditions. Of course, it is an important question how the approach performs on more complex data sets, but we leave this largely to future work. Nevertheless, we agree that a sensitivity analysis is called for, and we performed additional simulations with the visual entanglement (dsprites). Results are shown in the Table below. The results show that symbolic representations emerge over a large range of learning rates η\eta and Gumbel softmax temperatures τ\tau. Low values of η\eta and τ\tau performed clearly worse, indicating the need for stochasticity in the learning dynamics in order to find a good representation in the WTA bottleneck.

We report these results in Appendix A.13 in the revised version and refer to it at the end of Section 5.4.

Table 1: Results for different values of τ\tau and learning rate η\eta for visual entanglement (dSprites). MCRMCR: mean number of recovered categories; MCRperfMCR_{perf}: mean number of recovered categories over perfect runs (MAE≤10−6MAE \leq 10^{-6}). Each row summarizes 2020 runs with different seeds, and the total number of latent categories was 2929.

τ\tauη\etamean MAEMCRMCR# symb. rep.# perf. runsMCRperfMCR_{perf}
100.0010.0260922.6520NA
200.0010.0385716.4530NA
300.0010.0629911.5520NA
10.0050.124964.500NA
50.0050.0042526.3510627.83
100.0050.0033926.913828.5
200.0050.0079625.99728.57
300.0050.0159023.859628.5
400.0050.0154224.258329
10.010.236981.6500NA
50.010.0046725.8101028.4
100.010.0008627.4121128.73
200.010.0012727.05111427.86
300.010.0038226.1591627.25
400.010.0085825.75121128.55

R3W2: Nothing in the paper shows a benefit that is specific to the WTA bottleneck or to the symbolic format. The abstract closes by claiming "insights into the generalization capabilities of deep neural networks with WTA-like components" and an interface to symbolic AI. Sec. 5.3 and Fig. 2b–c compare a readout on z^\hat{\mathbf{z}} only with a readout on the raw input x\mathbf{x}. The readout on the multi-task representation a\mathbf{a} of the same network without WTA heads is in the appendix only, and it performs the same: on the random split with 100 samples, test AUC 0.992 [0.988–0.994] (Table A8) versus 0.992 [0.985–0.995] with WTA (Table A11); Tables A9–A10 versus A12–A13 give 0.990 vs 0.989 and 0.965 vs 0.959 (the captions identify A9–A10 as MLPa\mathrm{MLP}_{a}; z^\hat{\mathbf{z}} the headers in the test block of A9 and in A10 look like a typo. What the experiments establish is therefore parity: the generalisation gain comes from multi-task pre-training (as in Johnston & Fusi, 2023), and discretising the bottleneck does not lose it. No experiment uses a task for which an axis-aligned discrete code is needed, so the neurosymbolic motivation is untested. Likewise, the softmax/attention motivation of the abstract and introduction is at odds with L246–249, where softmax gates are reported to give "blurry" outputs that were insufficient.

R3Q1: Is there any experiment in which the WTA/symbolic representation outperforms the non-WTA multi-task representation a\mathbf{a} (Tables A8–A10 vs A11–A13)?

Answer to R3W2 and R3Q1: We thank the reviewer for raising this important point. The space of symbolic methods that can be utilized is huge, and thorough investigation is left for future work. As a first plausibility check, we have performed new experiments and used decision trees, a simple symbolic and interpretable method, as the downstream model. It allows to clearly determine how the model arrives at a given result based on specific factors.
We found that, when using a decision tree as downstream model, the generalization performance is clearly better when trained on z^\hat{\mathbf{z}} than when trained on the bottleneck representation a\mathbf{a} from a network without a WTA bottlenck. The results for both data sets are listed in the tables below.

We briefly summarize the results at the end of Section 5.3 and 5.4 and added the new Table 1 in the revised version. Details and further tables are given in Appendix A.9 and A.10.

Table 2: Test performance (AUC; random train-test split) on visual entanglement (Sec. 5.4) of a decision tree (DT) downstream model which uses the WTA bottleneck representation z^\hat{\mathbf{z}} or the abstract representation a\mathbf{a} of a multi-task learning model without a WTA bottleneck. The WTA bottleneck consisted of 4 WTA heads, each with 10 neurons. # train: number of training examples for the downstream model; med.: median of 2020 runs.

# trainmed. a\mathbf{a}Q1 a\mathbf{a}Q3 a\mathbf{a}med. z^\hat{\mathbf{z}}Q1 z^\hat{\mathbf{z}}Q3 z^\hat{\mathbf{z}}
1000.7210.6740.7670.8430.8190.862
3000.7670.7500.7970.8890.8760.915
5000.8000.7610.8310.9090.9000.924
10000.8070.7890.8510.9390.9290.957
15000.8540.8290.8730.9590.9450.969

Table 3: Test performance (AUC; random train-test split) on generic MLP entanglement (Sec. 5.2) of a downstream decision tree (DT) model which uses the WTA bottleneck representation z^\hat{\mathbf{z}} or the abstract representation a\mathbf{a} of a multi-task learning model without a WTA bottleneck. The WTA bottleneck consisted of 5 WTA heads, each with 10 neurons. # train: number of training examples for the downstream model; med.: median of 2020 runs.

# trainmed. a\mathbf{a}Q1 a\mathbf{a}Q3 a\mathbf{a}med. z^\hat{\mathbf{z}}Q1 z^\hat{\mathbf{z}}Q3 z^\hat{\mathbf{z}}
1000.6860.6520.7230.8200.7650.865
5000.7700.7510.7880.8900.8670.922
10000.8100.7870.8170.9190.8950.942
20000.8200.8060.8480.9310.9190.952
44000.8530.8300.8680.9520.9460.963
Reviewer_WY6kOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the properties of using winner-take-all (WTA) bottlenecks for learning disentangled and symbolic representations. Theoretically, the proposed methods demonstrate that the representations learned from multi-WTA heads can be structured permutations of the real generative factors when the model is perfectly optimized by enough linear binary classification tasks (so the weight W is full rank). Designed experiments proved that the theoretical conclusions are true. Additional experiments were conducted to generalize the proposed WTA framework to more general/real-life cases, such as when the number of classification tasks is smaller, and number of categories of latent factors is unknown.

Critical strengths

(1) Although the assumptions of the proposed methods are quite specific and strict (e.g., a perfectly trained classifier measured by whether small errors are achieved, and assigning the exact real settings of categorical latent variables in WTA heads), the main findings of this paper are very interesting and promising. Traditional disentangled representation learning methods usually encourage a single neuron/node in a continuous space to learn a single representation, which can be very challenging. However, this paper's methods and insights provide another approach to the learning problem: perhaps, in some scenarios, learning a group of neurons/nodes together as a one-hot code of a categorical variable can be more appropriate and achieve better disentanglement results. Therefore, I believe this paper will be inspiring to the ICLR community.

(2) The visual plots of the symbolic representation results (like Fig. 2a) look sound and align well with the proposed theory. So, the group of neurons from one WTA head indeed form a cluster for one generative factor, and show great disentanglement.

(3) The mathematical definitions, derivations, and explanations for the problem setup and proposed methods are very detailed and sound, which make the overall paper easy to follow.

Critical weaknesses

(W1) The main weakness is in the generalization/application aspect, i.e., how one can apply the method to real-life datasets, or how to design their own model using the knowledge from this paper. Two critical parameters of the proposed methods need to be addressed as they are usually unknown in the general case: first, the number of the latent factors, and second, the number of categories of each latent factor. While the latter one is briefly evaluated in Section 5.2, the former is not mentioned in the paper, if I haven't missed it. In general, I believe setting a larger number could be helpful, like the authors did in the experiment. However, a more comprehensive evaluation on the effects of choice of the two numbers in the model and a suggested rule-of-thumb when one or both are unknown will strengthen this paper.

(W2) Following up on (W1), an experiment on a more complex dataset can be very helpful. I noticed that the dSprites dataset used in this paper didn't include the orientation factor. I remember the categorical number of the orientation factor in the original dSprites is like 30 or 40, which is much larger than other factors. What's more, the orientation is also one of the most difficult factors to disentangle in the dSprites dataset. So, what's the reason for not including the original orientation factor in the experiment? Scalability issues? Disentanglement issues?

Minor weakness:

(M1) Please avoid using vague and unclear descriptions in the introduction (or anywhere). For example, what do the authors intend to say in "(but see (Locatello et al., 2019))" (line 66)? Disentangled representations do not benefit generalization? Or learning disentangled representations is not guaranteed?

Questions for authors

(1) I appreciate the authors' detailed report on how many runs are successful with different seeds, which is a key part of the proposed theory. I have two follow-up questions here, for the purpose of applying the methods to a more complex dataset. First, what if I set up a less restrictive successful criterion, say from 10^-6 to 10^-1 for MAE? Did the authors try, and will there be any difference? Second, will the successful rates drop significantly when training on a more complex dataset? Did the authors try or have any anticipation on this?

3 comments

R2W1: The main weakness is in the generalization/application aspect, i.e., how one can apply the method to real-life datasets, or how to design their own model using the knowledge from this paper. Two critical parameters of the proposed methods need to be addressed as they are usually unknown in the general case: first, the number of the latent factors, and second, the number of categories of each latent factor. While the latter one is briefly evaluated in Section 5.2, the former is not mentioned in the paper, if I haven't missed it. In general, I believe setting a larger number could be helpful, like the authors did in the experiment. However, a more comprehensive evaluation on the effects of choice of the two numbers in the model and a suggested rule-of-thumb when one or both are unknown will strengthen this paper.

Answer to R2W1: Notice that we did investigate in our initial submission a setting where the number of WTA heads does not match the number of latent factors (and the number of WTA head neurons does not match the number of latent factor categories) in Section 5.2 (line 323ff) as well as in Section 5.4 (line 434ff). We also described there the possibility to use ℓ1\ell_1 regularization to force the model to use as few WTA output neurons as possible, hence obtaining a minimal code.

Rule of thumb: in general, one possibility in practice is to vary the size and number of WTA heads and use a setup with the smallest bottleneck which still achieves low classification performance. This ensures that the latents can be represented (otherwise, small classification performance is not possible) with a minimal bottleneck which favors disentanglement (as suggested by our theorem). Starting with a very large bottleneck is beneficial for identifying the maximum classification performance and then finding a bottleneck that is as small as possible with only minimal performance loss.

To test this approach, we performed additional simulations with the visual entanglement (dsprites) where we have 4 latent factors with 29 categories in total (categories per factors were 3, 8, 8, 10). We performed multi-task training using a varying number of WTA heads and neurons per head. The results are shown in the table below.

Table 1: Results for different numbers of WTA heads and neurons per head. MCRMCR: mean number of recovered categories; MCRperfMCR_{perf}: mean number of recovered categories over perfect runs (MAE≤10−6MAE \leq 10^{-6}). Each row summarizes 2020 runs with different seeds and the total number of latent categories was 2929.

# WTA# nrnsmean MAEMCRMCR# symb. rep.# perf. runsMCRperfMCR_{perf}
8166.4×10−76.4 \times 10^{-7}27.881727.59
6124.9×10−74.9 \times 10^{-7}28.45161528.8
6108.6×10−78.6 \times 10^{-7}28.55151328.92
4120.0009228.05161328.15
4100.0079625.99728.57
480.0377119.700NA
3100.126452.5500NA

We found that if the bottleneck cannot represent the latent structure (last two rows), then the error of the predictors is large, indicating the mismatch. In the other cases, the error was reduced and symbolic representations emerged. Interestingly, slightly larger bottleneck structures had an advantage in terms of the number of runs where symbolic representations emerge, whereas a too large bottleneck (8 WTAs with 16 neurons each) lead to the reduction of the probability of symbolic representation emergence despite a small MAE. This provides evidence for our hypothesis that a small bottleneck with a low error is a good choice.

We incorporated a paragraph in Section 5.4 about these results, added a section in the Appendix (Appendix A.14), and discussed it briefly in Discussion.

R2W2: Following up on (W1), an experiment on a more complex dataset can be very helpful. I noticed that the dSprites dataset used in this paper didn't include the orientation factor. I remember the categorical number of the orientation factor in the original dSprites is like 30 or 40, which is much larger than other factors. What's more, the orientation is also one of the most difficult factors to disentangle in the dSprites dataset. So, what's the reason for not including the original orientation factor in the experiment? Scalability issues? Disentanglement issues?

Answer to R2W2: We did not include orientation because it is a closer to a continuous factor. Since there are basically only two truly categorical factors (shape, color), we decided to add x and y positions in order to have a sufficient number of factors. It is definitely a valid point to examine the emergence of symbolic representations when the factors are very difficult to isolate. We will examine this more closely in future research.



R2MW1 (Minor Weakness): Please avoid using vague and unclear descriptions in the introduction (or anywhere). For example, what do the authors intend to say in "(but see (Locatello et al., 2019))" (line 66)? Disentangled representations do not benefit generalization? Or learning disentangled representations is not guaranteed?

Answer to R2MW1: We changed the sentence to "It has been argued that disentangled representations benefit generalization [1,2], although Locatello et al. [3] have shown in an empirical study that this is not necessarily the case".

[1] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.

[2] W Jeffrey Johnston and Stefano Fusi. Abstract representations emerge naturally in neural networks trained to perform multiple tasks. Nature Communications, 14(1):1040, 2023.

[3] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 4114–4124. PMLR, 2019.



R2Q1: I appreciate the authors' detailed report on how many runs are successful with different seeds, which is a key part of the proposed theory. I have two follow-up questions here, for the purpose of applying the methods to a more complex dataset. First, what if I set up a less restrictive successful criterion, say from 10−610^-6 to 10−110^-1 for MAE? Did the authors try, and will there be any difference?

Answer to R2Q1: For more complex datasets, less restrictive thresholds are likely needed, since such low MAE values may not be achievable in those cases. The experiments with ℓ1\ell_1 regularization (see Table A7 and A16) are particularly interesting in this context, as the regularization prevents low MAE values from being achieved but still results in symbolic representations. An absolutely universal threshold is therefore likely unrealistic. We visualize the results with the visual entanglement (dsprites) (see Table A14 in the initial submission) in the new Figure A4 (revised paper version). One sees that a large number of categories could be recovered on a wide range of MAEs, with a clear drop at MAEs above 10−210^{-2} in this setup.



R2Q2: Second, will the successful rates drop significantly when training on a more complex dataset? Did the authors try or have any anticipation on this?

Answer to R2Q2: We see the current manuscript as a first step that introduces the idea based on a clean theoretical result. Of course, it is an important question how the approach performs on more complex data sets. As a first step in that direction, we tested it with the visual entanglement (dsprites) in the manuscript, which has been used by many papers in the area of disentangled representation learning as a basic benchmark. We did find that the method still performs reasonable here, but at this point we cannot anticipate its behavior on real-world data.

I appreciate the authors' detailed response/explanations, and my concerns/questions have been addressed. Therefore, I will keep my score in a positive acceptance recommendation.

Reviewer_4CHvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether winner-take-all (WTA) networks combined with multi-task learning can help neural networks learn disentangled and symbolic representations. The authors theoretically show that (under specific conditions) the network can recover categorical latent factors up to a structured permutation (of the latent factors). The paper evaluate the proposed approach using synthetic data, including dSprites, to examine the emergence of symbolic representations and their downstream generalization performance.

Critical strengths
  • The motivation of this paper is interesting, particularly the connection between WTA mechanisms in biological neural circuits and disentangled representation learning.

  • The paper provides a theoretical explanation (Sec 4) of how WTA bottlenecks can recover categorical latent factors through multi-task learning. This can can recover categorical latent factors from nonlinearly entangled observations. This connection is quite interesting.

  • While the theoritical analysis was for matched WTA structures, the experiments also showed the both matched and unmatched structures, and it showed that symbolic representations can emerge even when the number of WTA neurons does not match the number of latent categories. This can serve as useful approach to explore for future studies as well.

Critical weaknesses
  • I think the biggest weakness of this work is the strong assumptions in the theoretical results. I believe it requires knowing the latent factor structure, having sufficiently informative classification tasks, and achieving perfect predictions. The first one is the most challenging, and disentanglement as a field has made a lot of efforts to avoid such strong assumptions, from the design of metrics to learning strategies. So this work's strong assumptions feel like going backward.

  • Also, it seems like perfect task performance does not always imply symbolic representations. As presented in Sec 5.2, 19 out of 20 models achieve perfect task performance, but only 17 develop fully symbolic representations, especially given these are simulated settings. Now, with 7 WTA heads, symbolic representations emerge in only 6 out of those 20 runs. These results suggest that successful multi-task learning does not necessarily guarantee symbolic disentanglement. This raises questions about the reliability of the approach.

  • While the paper has provided the generalization experiments, I think these are not strong to imply broader generalization. These experiments are conducted using the same underlying latent factors and Φ. Although the downstream classification tasks are new, the experiments mainly evaluate unseen combinations of "existing categories" rather than changes in the data distribution or entirely new factors.

Questions for authors
  1. Please provide your response to the listed questions.

  2. I wonder how this work relates to recent research on monosemanticity and feature superposition. In particular, could WTA bottlenecks help reduce feature superposition and encourage more monosemantic representations? I would encourage the authors to discuss these connections in the related work section and clarify how their approach relates to recent developments in mechanistic interpretability.

4 comments

R1W1: I think the biggest weakness of this work is the strong assumptions in the theoretical results. I believe it requires knowing the latent factor structure, having sufficiently informative classification tasks, and achieving perfect predictions. The first one is the most challenging, and disentanglement as a field has made a lot of efforts to avoid such strong assumptions, from the design of metrics to learning strategies. So this work's strong assumptions feel like going backward.

Answer to R1W1: Our theoretical treatment is not intended to replace practical disentanglement methods but rather to identify conditions under which disentanglement is guaranteed. Theoretical results often need assumptions which are not fully satisfied in practice. Nevertheless, they can give valuable insights and guidance that can inspire further developments. Regarding the first assumption (knowledge of the latent structure), we have thoroughly investigated violations of this assumption in Section 5.2. Our results show that complete or nearly complete symbolic representations emerge even when the bottleneck is designed to be substantially larger (more WTA heads and more neurons per head) than the underlying latent factor structure.

Table 1: Results for different numbers of WTA heads and neurons per head. MCRMCR: mean number of recovered categories; MCRperfMCR_{perf}: mean number of recovered categories over perfect runs (MAE≤10−6MAE \leq 10^{-6}). Each row summarizes 2020 runs with different seeds and the total number of latent categories was 2929.

# WTA# nrnsmean MAEMCRMCR# symb. rep.# perf. runsMCRperfMCR_{perf}
8166.4×10−76.4 \times 10^{-7}27.881727.59
6124.9×10−74.9 \times 10^{-7}28.45161528.8
6108.6×10−78.6 \times 10^{-7}28.55151328.92
4120.0009228.05161328.15
4100.0079625.99728.57
480.0377119.700NA
3100.126452.5500NA


R1W2: Also, it seems like perfect task performance does not always imply symbolic representations. As presented in Sec 5.2, 19 out of 20 models achieve perfect task performance, but only 17 develop fully symbolic representations, especially given these are simulated settings. Now, with 7 WTA heads, symbolic representations emerge in only 6 out of those 20 runs. These results suggest that successful multi-task learning does not necessarily guarantee symbolic disentanglement. This raises questions about the reliability of the approach.

Answer to R1W2: It is correct that perfect task performance does not always imply the emergence of a complete symbolic representation in the unmatched setting where our theory does not directly apply. However, we found that it does lead to at least a nearly complete symbolic representation in which almost all latent categories are correctly encoded. We noted this at lines 320-322 (recovery rate of categories 94.3394.33% over all runs and 99.399.3% for perfect runs; perfect run if MAE$<$$10^{-6}$ after multi-task learning) and lines 357 (recovery rate 87.1787.17% over all runs and 97.0497.04% for perfect runs; the latter number was added in the revised version). Excluding runs with an MAE greater than 0.010.01, the average category recovery rate across Tables A5 (generic MLP entanglement, 5 WTA heads), A6 (generic MLP entanglement, 7 WTA heads), A14 (visual entanglement, 4 WTA heads), and A15 (visual entanglement, 6 WTA heads) is 97.597.5%, with the lowest rate being 95.495.4%. Notably, here, recovery is evaluated using a strict exact-match criterion: a latent category is considered recovered only if a set of WTA bottleneck output neurons or one neuron encodes it correctly across all samples. Even a single mismatch causes the entire category to be counted as not recovered and prevents a complete symbolic representation.

R1W3: While the paper has provided the generalization experiments, I think these are not strong to imply broader generalization. These experiments are conducted using the same underlying latent factors and Φ\Phi. Although the downstream classification tasks are new, the experiments mainly evaluate unseen combinations of "existing categories" rather than changes in the data distribution or entirely new factors.

Answer to R1W3: We considered three different train-test splits. The first two, the "random train-test split" and the "pair-of-categories train-test split", evaluate how well the downstream model generalizes to unseen combinations of latent categories.

The third, the "constant-category train-test split", evaluates generalization to previously unseen categories. Specifically, one latent factor was restricted to a single category during training, while its category was allowed to vary in the test set. Thus, the downstream model was evaluated on categories that were not observed during training. We argue that the appearance of novel categories in the test set is a significant distribution shift in the input.

To further test model generalization, we conducted new experiments with the visual entanglement (dsprites) where we introduced three new factors in each image: stripes that globally span the image with different stripe types, stripe color and stripe orientation. These factors were irrelevant for the tasks. Some combinations of these factors were only present in the test set. We asked the question whether the learned symbolic representation on the test set was affected by these unseen factors, and whether downstream predictors generalize in this setting. This setup implements a profound distribution shift between train and test set as it induces a global change of the input images. We removed 12 of the 120 possible confounding factor combinations from the train set and used 6 of them for the test set and 6 for a validation set used for downstream models.

We used 50 tasks for multi-task learning. We asked the question whether the WTA representation is a symbolic representation of the task-relevant latent variables on the test set, even though this context is unknown. In 6 out of 20 runs, the WTA bottleneck exhibited a symbolic representation for test set inputs and 8181% of all categories could be recovered. In 6 out of 20 runs the multi-task model solved the tasks perfectly on the test set and in these 6 runs 98.398.3% of all categories could be recovered. The generalization performances of the downstream models MLPz^MLP_{\hat{\mathbf{z}}} were excellent, see table below. In this experiment, the test and validation sets were relatively small, so a question for future work is: How large does the training set need to be to ensure that a symbolic representation emerges and that downstream generalization works well?

Table 2: Test performance (AUCAUC) of MLPxMLP_{\mathbf{x}} and MLPz^MLP_{\hat{\mathbf{z}}} with confounding factors for the visual entanglement (dSprites). The median and the first and third quartile of the AUCAUC across 1818 runs with different seeds are reported.

# trainmed. x\mathbf{x}Q1 x\mathbf{x}Q3 x\mathbf{x}med. z^\hat{\mathbf{z}}Q1 z^\hat{\mathbf{z}}Q3 z^\hat{\mathbf{z}}
1000.6910.6190.7410.9960.9950.998
3000.8150.7940.8351.0000.9991.000
5000.8610.8440.8951.0000.9991.000
10000.9250.9120.9391.0001.0001.000
15000.9540.9490.9631.0001.0001.000

R1Q1 I wonder how this work relates to recent research on monosemanticity and feature superposition. In particular, could WTA bottlenecks help reduce feature superposition and encourage more monosemantic representations? I would encourage the authors to discuss these connections in the related work section and clarify how their approach relates to recent developments in mechanistic interpretability.

Answer to R1Q1: We thank the reviewer for pointing out this related body of work. We added the following paragraph to the related work section:

Our work relates to recent research on monosemantic representations and feature superposition in deep neural networks [1,2,3,4]. A structured permutation of the original semantic features is a monosemantic representation as each neuron encodes only one specific feature value. In a symbolic representation, individual neurons are not necessarily monosemantic. In this representation, a group of neurons represents a specific feature value. Hence, the question of monosemanticity is elevated to neuron groups, a concept that has not been considered in the literature so far. Note that for structured permutations, features are linearly decodable via the permutation matrix. For symbolic representations within a WTA head, this is also true as the decoding is performed by a logical or over a set of binary variables where at most one assumes the value 11. Hence, these codes imply linear recoverability [4].

[1] Adam S Jermyn, Nicholas Schiefer, and Evan Hubinger. Engineering monosemanticity in toy models. arXiv preprint arXiv:2211.09169, 2022.

[2] Hanqi Yan, Yanzheng Xiang, Guangyi Chen, Yifei Wang, Lin Gui, and Yulan He. Encourage or inhibit monosemanticity? revisit monosemanticity from a feature decorrelation perspective. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10423–10435, 2024.

[3] Robert Huben, Hoagy Cunningham, Logan Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In International Conference on Learning Representations, volume 2024, pages 7827–7845, 2024.

[4] Dai Shi, Xiaoyu Li, Andi Han, and José Miguel Hernández-Lobato. Feature superposition in neural networks: From theory to practice, 2026. URL https: //arxiv.org/abs/2609.06862.

Reviewer_4CHvOpenReview ↗

Comment to authors

I thank the authors for their thoughtful responses and for conducting additional experiments to address my concerns. I particularly appreciate the new experiments showing that symbolic representations can emerge even when the WTA structure does not match the underlying latent factors. The additional analysis of category recovery is also helpful. More importantly, the new experiments with unseen visual factors in dSprites provide stronger evidence of generalization under distribution shifts.

While some limitations remain, particularly regarding the theoretical assumptions, I appreciate the authors' efforts to address my concerns. I believe the additional experiments have strengthened the paper, and I have increased my score accordingly.

Comments on the paper

1 comment

We would like to thank all reviewers for pointing out ambiguities, suggesting new experiments, and highlighting weaknesses. The reviewing process has certainly strengthened the paper substantially. Below, we summarize the most important points and new experiments once again.

Reliable emergence of symbolic representations: By reporting the number of symbolically represented latent categories, we clarified that sufficiently high multi-task learning performance consistently yielded either fully or nearly symbolic representations even when the latent factor structure and bottleneck structure were mismatched.

New experiments regarding the emergence of symbolic representations:

  • symbolic representations emerged also if the training set did not consists of all latent factor category combinations ("Answer to R3Q2" first experiment)
  • symbolic representations emerged in unknown context (see "Answer to R1W3" and "Answer to R3Q2" second experiment)
  • symbolic representations emerged when binary targets were used for multi-task learning (see "Answer to R3W1a and R3Q3"; binary task labels)
  • symbolic representations emerged when binary targets were sampled from the posterior P(Ai∣z)P(A_i\mid\mathbf z) (see "Author' Follow-up Response (2/2) Part 2" to Reviewer EQFB )

Usefulness in practice: We conducted additional experiments where we varied the number of WTA heads and neurons per head (see "Answer to R1W1" and "Answer to R2W1"). The results showed that nearly complete symbolic representations emerged even when the bottleneck was substantially larger than the latent factor structure. These findings suggest that our approach is applicable in practice, as a suitable WTA bottleneck structure can likely be identified using the rule of thumb derived from our experiments (see "Answer to R2W1"). This rule of thumb should be further validated in future research. We also performed a sensitivity analysis regarding the learning rate η\eta and the Gumbel softmax temperature τ\tau (see "Answer to R3W1c"). The results show that symbolic representations emerge over a large range of learning rates and Gumbel softmax temperatures. Low values of η\eta and τ\tau performed clearly worse, indicating the need for stochasticity in the learning dynamics in order to find a good representation in the WTA bottleneck.

Neurosymbolic AI: A key advantage of our bottleneck approach is its potential to combine the resulting symbolic representations with symbolic AI methods. To investigate whether the symbolic format provides a benefit, we conducted additional experiments using a decision tree as a interpretable symbolic downstream model for new tasks (see "Answer to R3W2 and R3Q1"). We compared its performance using representations from a WTA bottleneck and representations from a multi-task model without a WTA bottleneck. The symbolic representations yielded better generalization, providing initial evidence that symbolic AI methods benefit from symbolic representations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.