- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how neural networks learn abstract representations when latent factors appear only through entangled conjunctive codes. It identifies rebinding, where a factor changes partners between inputs and outputs, as a mechanism promoting abstraction. For a simplified rule transfer task, the authors analytically characterize optimal representations in shallow ReLU networks, revealing rectangular mixed-selective neuronal populations. A complementary linear theory predicts abstraction across broader compositional tasks through interaction strengths. Experiments support these predictions qualitatively, demonstrate greater early-layer abstraction with depth, and associate predicted abstraction with downstream transfer across contexts. The study connects task structure, representational geometry, and flexible knowledge reuse.
Critical strengths
- Rebinding offers an intuitive account of how task supervision can expose shared factors despite orthogonal conjunctive encodings. The distinction between removing old bindings and constructing output-required bindings is particularly informative.
- The nonlinear analysis establishes example-object invariance, a quantitative abstraction bound, and rectangular neuronal tuning. The general linear analysis provides explicit predictions linking input and output interaction strengths to hidden geometry.
- The experiments progress from rule transfer to fully conjunctive recombination, random task banks, deeper networks, and downstream transfer. This provides complementary evidence at the population, neuron, and behavioral levels.
Critical weaknesses
- Representation training includes every factor combination. The downstream experiment tests a new readout on previously encountered representations across held-out probe contexts, rather than testing the original network on unseen combinations or factor values. This limits conclusions about compositional generalization.
- Exact nonlinear results concern one specialized task under sufficiently wide networks and vanishing regularization. The broader theory is linear and cannot construct missing output conjunctions, producing systematic discrepancies precisely where nonlinear computation matters. Conditions governing its predictive accuracy remain insufficiently characterized.
- The association between predicted parallelism and transfer does not isolate abstraction from task complexity, factor decodability, or representation dimensionality. Controlled comparisons and interventions that vary abstraction while matching these properties would strengthen the claimed functional connection.
Questions for authors
NA
4 comments
Characterizing deviation of ReLU networks from linear theory (weakness 2)
Thank you for bringing attention to the deviation of ReLU networks from linear theory. We agree that the linear theory cannot capture the synthesis of new output conjunctions, and that its discrepancies with ReLU networks are particularly important in this regime. We performed an additional analysis to quantify the discrepancies between PS from the linear theory and ReLU network simulations across our bank of 200 tasks. We find three distinct regimes depending on whether a factor pair AB is bound in the input and/or output:
- AB absent from the output: The linear theory is almost exact, with a mean absolute PS deviation of 0.0007 across 2,430 factor pairs, regardless of whether AB is present in the input.
- AB present in both input and output: The linear theory is reasonably accurate, with a mean absolute error of 0.08 across 1,460 pairs.
- AB present only in the output: Discrepancies are largest, with a mean absolute error of 0.31 across 1,080 pairs. In this case, the linear theory predicts PS = 1, whereas ReLU networks exhibit a wide range of PS values. The deviation arises because linear networks cannot synthesize the AB interaction from the input, but nonlinear networks can.
In regime 3, variation in ReLU network PS can be partly explained by two properties of task structure:
i) Reuse of the target factor across novel output combinations. For a target A and context B, we count the number of other factors C that appear jointly with A in an output component but not in any input component (i.e. needs to be synthesized nonlinearly from the input). Mean ReLU network PS(A|B) increases from 0.58 to 0.81 as this count increases from zero to two. Although not directly predicted by the linear theory, this is consistent with our broader conclusion that increased reuse of a factor across diverse partners promotes abstraction.
ii) Relative binding demand, measured by the heuristic , which compares the required AB interaction strength in the output against the input and output strengths of A. This measure is negatively correlated with ReLU PS (Pearson r = −0.49).
Together, these two task properties predict ReLU PS with a held-out Pearson correlation of r = 0.76 (five-fold cross-validation, using linear regression with entire tasks held out). We note that, in ReLU networks with two hidden layers, the first hidden layer exhibits nearly full abstraction in this regime (mean PS = 0.98), closely matching the linear prediction of PS = 1, as the first hidden layer does not have to synthesize the novel output conjunctions.
For regime 2, PS deviation also depends on whether the output requires new interactions involving the target factor that are absent from the input. The mean absolute error is only 0.01 when no such interactions are required, compared with 0.11 otherwise. Thus, while the linear theory cannot fully capture representations learned through nonlinear computation, its discrepancies are systematically related to interpretable properties of task structure.
Controlling for confounding factors in the abstraction-transfer relationship (weakness 3)
Thank you for raising this concern. We performed additional analyses to test whether the relationship between predicted PS and transfer performance can be explained by these alternative factors.
1. Factor decodability. We trained linear decoders for the individual factors involved in the transfer target, using 80% of conditions for training and 20% for testing. All relevant factors were decoded with 100% accuracy, indicating that differences in transfer performance cannot be attributed simply to differences in individual factor decodability.
2. Task complexity and representation dimensionality. We considered several measures of task complexity: log input/output rank, number of output-relevant factors, mean output component size (i.e. the number of factors), and the fraction of output dimensions absent from the input span. We additionally measured dimensionality of the trained networks using participation ratio (PR). We compared the ability of these measures and theoretical PS to predict transfer accuracy, using five-fold task-level cross-validation, with four-fold inner cross-validation for regularization selection. We tested both linear regression and additive spline regression (a nonlinear predictor).
| Predictors | Linear regression R² | Additive spline R² |
|---|---|---|
| Task complexity + PR | 0.18 | 0.16 |
| Theoretical PS | 0.39 | 0.68 |
| Theoretical PS + task complexity + PR | 0.44 | 0.72 |
Theoretical PS substantially outperforms the combined task complexity and dimensionality predictors under both models. Adding PS to these controls yields a large improvement in predictive performance. These results indicate that the relationship between theoretical PS and transfer performance is not merely explained by differences in individual factor decodability, task complexity, or representation dimensionality.
Thank you again for your review, we will incorporate these additional analyses and clarifications into the revised manuscript.
Dear Reviewer,
I hope this message finds you well. As the discussion period draws to a close, we would like to sincerely thank you for the time and care you have put into reviewing our paper. Your thoughtful comments have been very helpful in improving our work. We hope that our responses and additional experiments have addressed your concerns. If there are any remaining questions or suggestions, we would be very happy to discuss them further.
Thank you again for your time, constructive feedback, and consideration!
Compositional generalization (weakness 1)
Thank you for pointing out the limitation in our transfer experiment, we agree it provides only a partial test of compositional generalization, since the original network is trained on all factor combinations and only the new readout is trained on a restricted subset of contexts. To address this limitation, we conducted additional experiments where we withhold factor combinations from training directly to test compositional generalization and its relationship to representational abstraction.
1. Compositional generalization to unseen combinations.
We trained ReLU networks on our bank of 200 tasks, randomly withholding 60%, 70%, or 80% of factor combinations during training. All networks achieved 100% training accuracy. We then evaluated these networks on the held-out combinations, without retraining any readout. We computed accuracy for each output component (each prediction was counted as correct if the highest-scoring output unit matched the target one-hot unit), and also computed a more stringent full task accuracy that requires all output components to be correct. We found substantial compositional generalization: even with 60% factor combinations withheld, the mean output component accuracy is 98%, and the full task accuracy is 95%. Thus, networks can generalize to unseen combinations of factor values.
2. Relationship between abstraction and compositional generalization.
We next investigated whether this generalization is related to the abstract representations studied in our paper, focusing on the more challenging setting with 80% of combinations withheld. To connect abstraction to generalization, we measured PS specifically using target factors within each output component, across context factors outside that component (averaging over all choices of target and context). We found that both theory-predicted and observed PS were positively correlated with generalization accuracy on unseen combinations across output components from all tasks (ρ = 0.27 and 0.51, respectively), with the observed PS producing a higher correlation. Our original linear theory assumes training on all factor combinations, but here networks are trained on only a subset of factor combinations. We therefore derived the approximate linear network representation under random subsampling, which predicted the PS of any factor pair from only the task's input-output structure and the fraction of combinations included in training (Appendix C.3 in updated manuscript). PS predicted from this derivation shows a stronger correlation with the empirical generalization accuracy (ρ = 0.65). Together, these results show the degree of abstraction is associated with the network's ability to generalize to unseen factor combinations.