acceptodds
Under review as a conference paper at ICLR 2027

Learning Abstract Disentangled Factors from Entangled Conjunctive Codes through Compositional Tasks

Abstract

Abstract, or disentangled, representations in neural networks encode latent factors in separate subspaces and can support compositional generalization. Yet latent factors are often observed only through entangled combinations, posing a challenge for networks to disentangle them. Here we study compositional tasks whose inputs and outputs consist of conjunctions of discrete latent factors (e.g., shape color). Each combination of factor values in a conjunction (e.g., *square* *blue*) receives a distinct one-hot code. We find that even when a latent factor is always entangled with other factors in the input and the output, it can become abstract in task-trained networks through **rebinding**. Rebinding occurs when is *bound* to (i.e. encoded in a conjunction with) one factor in the input and a different factor in the output. We illustrate this mechanism in a rule transfer task inspired by Raven's Progressive Matrices, for which we analytically derive the optimal hidden representation of single-hidden-layer ReLU networks. The solution uses subpopulations of mixed-selective neurons that each supports abstraction for a subset of factor values. We then derive the optimal linear network representation for arbitrary tasks in the family. The theory formalizes the rebinding principle and predicts that abstraction increases when a factor is reused across many distinct conjunctions. The predicted degree of abstraction qualitatively matches that observed in nonlinear networks and closely tracks their generalization performance to held-out contexts in a transfer task. Together, these results show how compositional task structure can drive networks to learn abstract representations from entangled observations.

Primary area:
applications to neuroscience & cognitive science
Keywords:
representational geometry, abstract representation, deep learning theory, representation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Afw5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how neural networks learn abstract representations when latent factors appear only through entangled conjunctive codes. It identifies rebinding, where a factor changes partners between inputs and outputs, as a mechanism promoting abstraction. For a simplified rule transfer task, the authors analytically characterize optimal representations in shallow ReLU networks, revealing rectangular mixed-selective neuronal populations. A complementary linear theory predicts abstraction across broader compositional tasks through interaction strengths. Experiments support these predictions qualitatively, demonstrate greater early-layer abstraction with depth, and associate predicted abstraction with downstream transfer across contexts. The study connects task structure, representational geometry, and flexible knowledge reuse.

Critical strengths
  • Rebinding offers an intuitive account of how task supervision can expose shared factors despite orthogonal conjunctive encodings. The distinction between removing old bindings and constructing output-required bindings is particularly informative.
  • The nonlinear analysis establishes example-object invariance, a quantitative abstraction bound, and rectangular neuronal tuning. The general linear analysis provides explicit predictions linking input and output interaction strengths to hidden geometry.
  • The experiments progress from rule transfer to fully conjunctive recombination, random task banks, deeper networks, and downstream transfer. This provides complementary evidence at the population, neuron, and behavioral levels.
Critical weaknesses
  • Representation training includes every factor combination. The downstream experiment tests a new readout on previously encountered representations across held-out probe contexts, rather than testing the original network on unseen combinations or factor values. This limits conclusions about compositional generalization.
  • Exact nonlinear results concern one specialized task under sufficiently wide networks and vanishing regularization. The broader theory is linear and cannot construct missing output conjunctions, producing systematic discrepancies precisely where nonlinear computation matters. Conditions governing its predictive accuracy remain insufficiently characterized.
  • The association between predicted parallelism and transfer does not isolate abstraction from task complexity, factor decodability, or representation dimensionality. Controlled comparisons and interventions that vary abstraction while matching these properties would strengthen the claimed functional connection.
Questions for authors

NA

4 comments
AuthorsOpenReview ↗

Compositional generalization (weakness 1)

Thank you for pointing out the limitation in our transfer experiment, we agree it provides only a partial test of compositional generalization, since the original network is trained on all factor combinations and only the new readout is trained on a restricted subset of contexts. To address this limitation, we conducted additional experiments where we withhold factor combinations from training directly to test compositional generalization and its relationship to representational abstraction.

1. Compositional generalization to unseen combinations.

We trained ReLU networks on our bank of 200 tasks, randomly withholding 60%, 70%, or 80% of factor combinations during training. All networks achieved 100% training accuracy. We then evaluated these networks on the held-out combinations, without retraining any readout. We computed accuracy for each output component (each prediction was counted as correct if the highest-scoring output unit matched the target one-hot unit), and also computed a more stringent full task accuracy that requires all output components to be correct. We found substantial compositional generalization: even with 60% factor combinations withheld, the mean output component accuracy is 98%, and the full task accuracy is 95%. Thus, networks can generalize to unseen combinations of factor values.

Fraction held outOutput component accuracyFull task accuracy
60%98.2%94.7%
70%93.8%83.5%
80%81.1%55.6%

2. Relationship between abstraction and compositional generalization.

We next investigated whether this generalization is related to the abstract representations studied in our paper, focusing on the more challenging setting with 80% of combinations withheld. To connect abstraction to generalization, we measured PS specifically using target factors within each output component, across context factors outside that component (averaging over all choices of target and context). We found that both theory-predicted and observed PS were positively correlated with generalization accuracy on unseen combinations across output components from all tasks (ρ = 0.27 and 0.51, respectively), with the observed PS producing a higher correlation. Our original linear theory assumes training on all factor combinations, but here networks are trained on only a subset of factor combinations. We therefore derived the approximate linear network representation under random subsampling, which predicted the PS of any factor pair from only the task's input-output structure and the fraction of combinations included in training (Appendix C.3 in updated manuscript). PS predicted from this derivation shows a stronger correlation with the empirical generalization accuracy (ρ = 0.65). Together, these results show the degree of abstraction is associated with the network's ability to generalize to unseen factor combinations.

AuthorsOpenReview ↗

Characterizing deviation of ReLU networks from linear theory (weakness 2)

Thank you for bringing attention to the deviation of ReLU networks from linear theory. We agree that the linear theory cannot capture the synthesis of new output conjunctions, and that its discrepancies with ReLU networks are particularly important in this regime. We performed an additional analysis to quantify the discrepancies between PS from the linear theory and ReLU network simulations across our bank of 200 tasks. We find three distinct regimes depending on whether a factor pair AB is bound in the input and/or output:

  1. AB absent from the output: The linear theory is almost exact, with a mean absolute PS deviation of 0.0007 across 2,430 factor pairs, regardless of whether AB is present in the input.
  2. AB present in both input and output: The linear theory is reasonably accurate, with a mean absolute error of 0.08 across 1,460 pairs.
  3. AB present only in the output: Discrepancies are largest, with a mean absolute error of 0.31 across 1,080 pairs. In this case, the linear theory predicts PS = 1, whereas ReLU networks exhibit a wide range of PS values. The deviation arises because linear networks cannot synthesize the AB interaction from the input, but nonlinear networks can.

In regime 3, variation in ReLU network PS can be partly explained by two properties of task structure:

i) Reuse of the target factor across novel output combinations. For a target A and context B, we count the number of other factors C that appear jointly with A in an output component but not in any input component (i.e. needs to be synthesized nonlinearly from the input). Mean ReLU network PS(A|B) increases from 0.58 to 0.81 as this count increases from zero to two. Although not directly predicted by the linear theory, this is consistent with our broader conclusion that increased reuse of a factor across diverse partners promotes abstraction.

ii) Relative binding demand, measured by the heuristic kABout/(kAinkAout)\sqrt{k_{AB}^{\mathrm{out}}/(k_A^{\mathrm{in}}k_A^{\mathrm{out}})}, which compares the required AB interaction strength in the output against the input and output strengths of A. This measure is negatively correlated with ReLU PS (Pearson r = −0.49).

Together, these two task properties predict ReLU PS with a held-out Pearson correlation of r = 0.76 (five-fold cross-validation, using linear regression with entire tasks held out). We note that, in ReLU networks with two hidden layers, the first hidden layer exhibits nearly full abstraction in this regime (mean PS = 0.98), closely matching the linear prediction of PS = 1, as the first hidden layer does not have to synthesize the novel output conjunctions.

For regime 2, PS deviation also depends on whether the output requires new interactions involving the target factor that are absent from the input. The mean absolute error is only 0.01 when no such interactions are required, compared with 0.11 otherwise. Thus, while the linear theory cannot fully capture representations learned through nonlinear computation, its discrepancies are systematically related to interpretable properties of task structure.

AuthorsOpenReview ↗

Controlling for confounding factors in the abstraction-transfer relationship (weakness 3)

Thank you for raising this concern. We performed additional analyses to test whether the relationship between predicted PS and transfer performance can be explained by these alternative factors.

1. Factor decodability. We trained linear decoders for the individual factors involved in the transfer target, using 80% of conditions for training and 20% for testing. All relevant factors were decoded with 100% accuracy, indicating that differences in transfer performance cannot be attributed simply to differences in individual factor decodability.

2. Task complexity and representation dimensionality. We considered several measures of task complexity: log input/output rank, number of output-relevant factors, mean output component size (i.e. the number of factors), and the fraction of output dimensions absent from the input span. We additionally measured dimensionality of the trained networks using participation ratio (PR). We compared the ability of these measures and theoretical PS to predict transfer accuracy, using five-fold task-level cross-validation, with four-fold inner cross-validation for regularization selection. We tested both linear regression and additive spline regression (a nonlinear predictor).

PredictorsLinear regression R²Additive spline R²
Task complexity + PR0.180.16
Theoretical PS0.390.68
Theoretical PS + task complexity + PR0.440.72

Theoretical PS substantially outperforms the combined task complexity and dimensionality predictors under both models. Adding PS to these controls yields a large improvement in predictive performance. These results indicate that the relationship between theoretical PS and transfer performance is not merely explained by differences in individual factor decodability, task complexity, or representation dimensionality.

Thank you again for your review, we will incorporate these additional analyses and clarifications into the revised manuscript.

Dear Reviewer,

I hope this message finds you well. As the discussion period draws to a close, we would like to sincerely thank you for the time and care you have put into reviewing our paper. Your thoughtful comments have been very helpful in improving our work. We hope that our responses and additional experiments have addressed your concerns. If there are any remaining questions or suggestions, we would be very happy to discuss them further.

Thank you again for your time, constructive feedback, and consideration!

Reviewer_sqHaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors study of the phenomenon of abstraction in simple neural networks using a composition task inspired by Raven's Progressive Matrices. They find that models are able to bind rules to novel objects (i.e they can perform rebinding) even though inputs only present conjunctions of said rules and objects using one-hot encodings. They define abstraction mechanistically as disentanglement and measure it's level in the hidden layers of models using a Parallel Score; applying (i.e. binding) a rule to two different objects should induce equivalent linear transformations (i.e. parallel) in latent space. They use this analysis to show that abstraction emerges faster the more composition tasks the model must learn.

Critical strengths

The paper makes a straightforward point: that networks can learn to perform simple remapping from input conjunctive codes to output or target ones. They analyze the internal neuron activations a show that neurons involved in the remapping learn to ignore the unbounded example in the Raven matrix input, extracting the rule.

Critical weaknesses

The main weakness of the paper is that similar setups have been studied before and that field doesn't consider this an instance of rebinding. This is because unbinding the components of a conjunctive code into its components is trivial if we have explicit supervision. Notice that in this case the authors don't directly supervise the individual components of a conjunctive code, but they do so implicitly via the input. And indeed the trivial solution is to take the embedding of the example, subtract it from the embedding of the rule-transformed example and add the embedding of the cue. The only task of the neural network is to create the appropriate mappings from the conjunctive codes. Using conjunctive codes in this way thus trivializes the binding problem. Indeed, what is interesting about abstraction is that it operates on perceptual inputs which need to be parsed in some non-trivial way to bind properly, which is what actual RPMs test.

This leads us to the second problem which is that the authors don't test for generalization at all. And indeed if they were to test for generalization it would just be a test of whether the model can project a new object (or its rule-transformed version) onto good enough embedding.

References:

  1. https://arxiv.org/html/2012.05208v1
  2. https://pubmed.ncbi.nlm.nih.gov/7488855/
  3. https://pmc.ncbi.nlm.nih.gov/articles/PMC8019313/
  4. https://reasoninglab.psych.ucla.edu/wp-content/uploads/sites/273/2021/04/Hummel-Holyoak.1997.pdf
Questions for authors

I have no questions.

3 comments
AuthorsOpenReview ↗

Clarifying the scope and novelty of our contribution (Part 1/2)

Thank you for your review and for pointing us to the rich literature on the binding problem. We apologize for the ambiguity in our terminology and would like to clarify our research question and contributions.

1. Scope and terminology. Our simplified task does not address the traditional binding problem, which classically arises when multiple objects are represented by a disentangled code of their features, and the challenge is to represent which features belong to the same object or go together. Conjunctive coding is one possible solution to this problem, alongside other mechanisms explored in the literature you have highlighted.

Our research question is not to solve the binding problem. It is almost the reverse: starting from conjunctive codes (which do not suffer from the binding problem), how can networks learn disentangled representations of the underlying factors? We use binding in a narrow, operational sense: two factors are bound when they are encoded conjunctively, and rebinding refers to a factor being conjunctively encoded with different partners in the input and output. We use one-hot conjunctions in the input and output to remove any pre-existing similarity between observations sharing a factor. While this does not capture the rich structure of real-world perceptual inputs, it allows us to isolate how the input–output task structure shapes learned representations. We agree that producing new rule-object conjunctions is not, by itself, a difficult task in this setting. Our central question is what internal representation networks learn when trained to perform such tasks, and we identify rebinding (as defined here) as a task structure property that promotes disentanglement. In general, learning disentangled representations without explicit supervision of the underlying factors is a fundamental challenge in representation learning. Our contribution is to identify and characterize a specific mechanism through which compositional task structure can promote such disentanglement.

2. Why disentangled representations are nontrivial in our tasks. Conjunctive codes can be mapped to their underlying factors (e.g. through a linear projection), but this does not mean that a network necessarily learns a disentangled representation. In our rule transfer task, the input does not provide explicit rule identities or similarities between conjunctions sharing a rule. In fact, the rule cannot be determined from the input distribution alone: rule identities can be permuted independently for each example object without changing the input distribution. It is the input–output mapping that provides the information needed to identify the shared factors. The suggested subtraction/addition solution (subtracting the embedding of the example from the embedding of the rule-transformed example and adding the embedding of the cue) assumes an appropriate additive structure in the embeddings, which is not present in our input, and whether it is possible to learn such representations from conjunctive codes is precisely our question.

Moreover, perfect task performance does not require disentanglement. For example, a hidden layer could simply use a one-hot code for each example-rule-cue conjunction, achieving perfect performance while having zero rule parallelism across cues or examples. Our key contribution is to show how disentangled representations can emerge even when neither explicitly specified by the inputs nor required for task performance. We quantify the exact learned geometry and single-neuron tuning (Sec. 3), and further show that abstraction emerges when no factor is presented individually (Sec. 4). We derive how different input-output conjunction structures determine the degree of abstraction in the entire task family (Sec. 5-6). These are novel results not established by the cited studies.

AuthorsOpenReview ↗

Generalization and its relationship to abstraction (Part 2/2)

3. Generalization. Previous work has shown that abstract or disentangled representations can facilitate compositional generalization, so our primary focus is on understanding how such representations could emerge through learning. We agree that generalization to novel factor values is not addressed by our study. However, we do test a simple form of generalization in Sec. 7 (Fig. 5d): after training on a task, we freeze the learned representation and train a new readout on a novel factor pair target using examples from only one context, and evaluate the readout on 80 held-out contexts. Transfer accuracy varies substantially across tasks and is strongly predicted by our theory of abstraction (Spearman ρ = 0.79).

Furthermore, we have conducted additional experiments where we withhold some factor combinations from training, and directly test whether networks can compositionally generalize to these heldout combinations. In particular, we trained ReLU networks on our bank of 200 tasks, randomly withholding 60%, 70%, or 80% of factor combinations during training. All networks achieved 100% training accuracy. We then evaluated these networks on the held-out combinations, without retraining any readout. We computed accuracy for each output component (each prediction was counted as correct if the highest-scoring output unit matched the target one-hot unit), and also computed a more stringent full task accuracy that requires all output components to be correct. We found substantial compositional generalization: even with 60% factor combinations withheld, the mean output component accuracy is 98%, and the full task accuracy is 95%. Thus, networks can generalize to unseen combinations of factor values.

Fraction held outOutput component accuracyFull task accuracy
60%98.2%94.7%
70%93.8%83.5%
80%81.1%55.6%

We next investigated whether this generalization is related to the abstract representations studied in our paper, focusing on the challenging setting with 80% of combinations withheld. We measured PS specifically using target factors within each output component, across context factors outside that component (averaging over all choices of target and context). We found that both theory-predicted and observed PS were positively correlated with generalization accuracy on unseen combinations across output components from all tasks (ρ = 0.27 and 0.51, respectively), with the observed PS producing a higher correlation. We further extended our linear theory to derive an approximate PS when trained on a random subset of all trials (Appendix C.3 in updated manuscript), and this predicted PS shows a stronger correlation with the empirical generalization accuracy (ρ = 0.65). Together, these results show the degree of abstraction is associated with the network's ability to generalize to unseen factor combinations.

Thank you again for your review, we will incorporate these additional analyses and clarifications into the revised manuscript.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, we would like to sincerely thank you for the time and care you have put into reviewing our paper. Your thoughtful comments have been very helpful in clarifying our research question and strengthening the presentation of our contributions.

We hope that our responses and additional clarifications have addressed your concerns, and we would be very grateful if you would consider revisiting your assessment and raising your score in light of these discussions.

If there are any remaining questions or suggestions, we would be more than happy to address them. Thank you again for your time, constructive feedback, and consideration!

Reviewer_SEHhOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how abstract representations can emerge through rebinding, where latent factors are combined with different factors in the input and output. The authors use one-hot encodings for factor conjunctions, so combinations sharing the same latent factor do not have explicit similarity.

The paper first analyses a rule-transfer task inspired by Raven's Progressive Matrices and characterises the hidden representations of sufficiently wide ReLU networks at global optima. It then extends the study to a compositional recombination task. A linear-network theory is developed to predict abstraction from the input-output structure, and experiments with ReLU networks examine these predictions. Finally, the paper investigates the relationship between predicted abstraction and transfer performance in held-out contexts.

Critical strengths
  • The paper addresses an interesting question of how abstract representations can emerge from task structure without directly providing individual latent factors.

  • The proposed rebinding perspective is interesting, particularly the idea that reusing latent factors with different partners can promote abstraction.

  • The paper combines theoretical analysis with experiments across different compositional task structures.

  • The paper also examines whether the predicted abstraction is associated with better transfer performance.

Critical weaknesses
  1. Presentation of the theoretical analysis I found the theoretical sections difficult to follow. While the overall idea of rebinding is understandable, the intuition behind some theoretical results is not sufficiently clear. In particular, the paper would benefit from more accessible explanations of why the optimal representations become abstract and how the theoretical predictions relate to the observed representations.

  2. Validation of the linear theory in nonlinear networks The linear theory predicts full abstraction when interactions are not shared between the input and output. However, ReLU networks exhibit only partial abstraction for some output-bound factor pairs. Although the experiments support the general trends, the differences between the theoretical predictions and the observed ReLU representations are not fully characterised.

  3. Relationship between abstraction and transfer performance Section 7 shows a positive correlation between the minimum PS predicted by the linear theory and transfer accuracy. However, the paper does not directly report the relationship between the PS measured from the learned ReLU representations and transfer performance. This makes it difficult to assess how closely the learned abstraction itself is associated with improved transfer.

Questions for authors
  1. Theoretical analysis The paper analyses global optima of sufficiently wide ReLU networks in the vanishing-regularisation limit. Could the authors clarify how their analysis extends existing theoretical frameworks for regularised ReLU networks, particularly in characterising the representations induced by rebinding?

  2. Linear theory versus ReLU networks The linear theory predicts full abstraction for unshared interactions, whereas the ReLU experiments show partial abstraction for output-bound factor pairs. Could the authors provide a quantitative comparison between the predicted and measured PS values across the tested task structures, and identify the conditions where the differences are largest?

  3. Marginal PS The paper uses marginal PS to quantify abstraction. Since marginalisation averages over the remaining factors, could this averaging conceal context-dependent coding in the original representations? It would be helpful to compare marginal PS with PS calculated without marginalisation for representative tasks.

  4. Abstraction and transfer Section 7 reports a correlation between predicted minimum PS and transfer accuracy. Could the authors also report the correlation between the measured PS of the ReLU representations and transfer accuracy? Additionally, does the minimum predicted PS provide better predictive performance than the average predicted PS?

4 comments
AuthorsOpenReview ↗

Clarifying theoretical analyses (weakness 1)

We apologize for the lack of clarity in our explanation of the intuition underlying the theoretical results in the current manuscript. The intuition behind the optimal representation can be understood more easily by considering linear networks. When the input and output kernel matrices share an eigenbasis, the nonzero eigenmodes of the optimal hidden representation kernel are exactly the nonzero eigenvectors shared by the two kernels. In the rule transfer task (Sec. 3) and the recombination task (Sec. 4), the shared eigenvectors between the input and output matrices are exactly those eigenvectors that represent individual latent factors in the task. Consequently, the optimal hidden representation of a linear network is perfectly abstract with respect to these factors. However, such a representation cannot reproduce the desired outputs through a linear readout, because the outputs depend on conjunctions of latent factors. Introducing ReLU nonlinearity allows the network to produce these conjunctions while retaining part of the abstract geometry. Thus, partially abstract representations arise from the structure shared by the input and output kernel matrices (which correspond to an abstract representation of the latent factors), while the nonlinearity introduces the conjunctive structure necessary to solve the task.

For the rule transfer task, this intuition can be developed further. The task contains three latent factors: the example object, the cue object, and the rule. Since the desired output depends only on the cue object and the rule, the example object is irrelevant to the task. If we disregard this task-irrelevant factor, the input representation is perfectly abstract with respect to the cue object and rule. Because the hidden preactivations are linear transformations of this input representation, they inherit its abstract representation geometry. Applying the ReLU nonlinearity introduces conjunctive structures while partially preserving the geometry of this abstract representation. This provides an intuitive explanation for why the hidden representation tends to retain substantial abstraction, as reflected in its high parallelism score (PS).

AuthorsOpenReview ↗

Characterizing linear–ReLU discrepancies and the relationship between PS and transfer

Characterizing deviation of ReLU networks from linear theory (weakness 2 & question 2)

Thank you for bringing attention to the deviation of ReLU networks from linear theory. We performed an additional analysis to characterize the discrepancies between PS from the linear theory and ReLU network simulations across our bank of 200 tasks. We find three distinct regimes depending on whether a factor pair AB is bound in the input and/or output:

  1. AB absent from the output: The linear theory is almost exact, with a mean absolute PS deviation of 0.0007 across 2,430 factor pairs, regardless of whether AB is present in the input.
  2. AB present in both input and output: The linear theory is reasonably accurate, with a mean absolute error of 0.08 across 1,460 pairs.
  3. AB present only in the output: Discrepancies are largest, with a mean absolute error of 0.31 across 1,080 pairs. In this case, the linear theory predicts PS = 1, whereas ReLU networks exhibit a wide range of PS values. The deviation arises because linear networks cannot synthesize the AB interaction from the input, but nonlinear networks can.

In regime 3, variation in ReLU network PS can be partly explained by two properties of task structure:

i) Reuse of the target factor across novel output combinations. For a target A and context B, we count the number of other factors C that appear jointly with A in an output component but not in any input component (i.e. needs to be synthesized nonlinearly from the input). Mean ReLU network PS(A|B) increases from 0.58 to 0.81 as this count increases from zero to two. Although not directly predicted by the linear theory, this is consistent with our broader conclusion that increased reuse of a factor across diverse partners promotes abstraction.

ii) Relative binding demand, measured by the heuristic (\sqrt{k_{AB}^{\mathrm{out}}/(k_A^{\mathrm{in}}k_A^{\mathrm{out}})}), which compares the required AB interaction strength in the output against the input and output strengths of A. This measure is negatively correlated with ReLU PS (Pearson r = −0.49).

Together, these two task properties predict ReLU PS with a held-out Pearson correlation of r = 0.76 (five-fold cross-validation, using linear regression with entire tasks held out). We note that, in ReLU networks with two hidden layers, the first hidden layer exhibits nearly full abstraction in this regime (mean PS = 0.98), closely matching the linear prediction of PS = 1, as the first hidden layer does not have to synthesize the novel output conjunctions.

For regime 2, PS deviation also depends on whether the output requires new interactions involving the target factor that are absent from the input. The mean absolute error is only 0.01 when no such interactions are required, compared with 0.11 otherwise. Thus, while the linear theory cannot fully capture representations learned through nonlinear computation, its discrepancies are systematically related to interpretable properties of task structure.

Relationship between PS and transfer accuracy (weakness 3 & question 4)

Thank you for pointing out the nuances in comparing abstraction and transfer performance in Sec. 7. We performed an additional analysis comparing transfer accuracy with PS directly observed from the learned ReLU representations, and using either the minimum or the mean when aggregating PS across relevant factor pairs. The Spearman correlation of PS with transfer accuracy is

PS statisticObserved ReLU PSTheory-predicted PS
Minimum0.790.79
Average0.660.73

The correlations are similar for observed PS and theory-predicted PS, and are slightly higher for minimum PS than average PS.

AuthorsOpenReview ↗

Theoretical contributions and marginal PS

Clarifying theoretical contributions beyond existing frameworks (question 1)

Thank you for pointing out this clarification point that was not sufficiently clear in the manuscript. The technical tools we use build directly on two recent works that characterize optimal solutions of regularized rectified linear unit (ReLU) networks with multidimensional outputs [1,2].

[1] shows that for sufficiently wide networks, the original optimization problem can be reformulated as a convex optimization problem over the sets of copositive matrices through strong duality. Here, the optimization is over a set of auxiliary variables whose relationship to the optimal hidden representation is less direct. The related work in [2] reformulates the original optimization problem as a convex optimization problem over preactivation and postactivation kernel matrices.

Our main theoretical contribution is to apply this framework to a three-factor rule transfer task (Sec. 3), a representative task that involves rebinding, and to characterize the optimal hidden representation kernels and single-neuron selectivity for different numbers of objects and rules. We reparameterize the kernel matrices in terms of distributions over neuronal preactivations and derive the Karush–Kuhn–Tucker (KKT) conditions characterizing the optimal solutions. Because this task involves nonlinear conjunctions of latent factors, obtaining an explicit characterization of its optimal representations is highly nontrivial.

[1] Sahiner, Arda, et al. "Vector-output relu neural network problems are copositive programs: Convex analysis of two layer networks and polynomial-time algorithms." arXiv preprint arXiv:2012.13329 (2020).
[2] Jacot, Arthur, et al. "Feature Learning in L2L_2-regularized DNNs: Attraction/Repulsion and Sparsity." Advances in Neural Information Processing Systems 35 (2022): 6763-6774.

Marginal vs context-specific PS (question 3)

Thank you for raising the point on marginal PS, it is possible that context-dependent coding could be concealed by the marginalization. We performed an additional analysis of PS without marginalizing over the remaining factors. Specifically, for a target factor A and a context factor B, we computed the cosine similarity between the coding directions of A across two values of B, while fixing all other factors to a particular combination of values. We repeated this for all combinations of the remaining factors and all pairs of A and B values. For ReLU networks trained on either the rule transfer task or the recombination task, every context-specific PS value differs from its corresponding marginal PS by less than 10−310^{-3}. Thus, for these representative tasks, marginal PS accurately reflects the underlying context-specific PS rather than concealing substantial variation. This observation is also consistent with a stronger result we previously established for linear networks (Appendix C.2): when no interaction involving a target factor is shared between the input and output, its coding is invariant across all combinations of the remaining factors, even without marginalization – this is a particularly strong form of how rebinding leads to fully context-invariant abstract coding. We note, however, that this equivalence does not hold universally. Across our bank of 200 tasks, the absolute difference between marginal PS and the mean context-specific PS is 0.056 on average, while the maximum absolute deviation of individual context-specific PS values from marginal PS is 0.066 on average. Thus, marginal PS is generally a useful summary, though it does not always capture the full distribution of context-specific PS.

Thank you again for your review, we will incorporate these additional analyses and clarifications into the revised manuscript.

Dear Reviewer,

I hope this message finds you well. As the discussion period draws to a close, we would like to sincerely thank you for the time and care you have put into reviewing our paper. Your thoughtful comments have been very helpful in improving our work. We hope that our responses and additional experiments have addressed your concerns. If there are any remaining questions or suggestions, we would be very happy to discuss them further.

Thank you again for your time, constructive feedback, and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.