acceptodds
Under review as a conference paper at ICLR 2027

A Spectral Theory of Compositional Learning

Abstract

How does compositional reasoning emerge during learning? We address this question by mathematically analyzing the learning dynamics of deep linear networks. We train these networks in structured synthetic environments and derive a theory linking the structure of experience to compositional learning. Our theory predicts when compositional inferences emerge, whether they are identifiable from the available evidence, and how new linking evidence can rapidly unlock previously unavailable inferences. These results provide a qualitative explanation for several phenomena observed in human cognition. They account for why a composition can fail despite knowing its premises, why similar compositions can emerge at different times, and how a single linking fact can suddenly enable many new inferences. Taken together, these findings establish a mathematical link between the statistical structure of experience and the development of compositional reasoning.

Primary area:
learning theory
Keywords:
compositional learning, learning dynamics, spectral analysis, deep linear networks, relational representations

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_SYFJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper investigates how compositional reasoning emerges during learning through three main aspects: law-specific emergence, structural identifiability, and relation integration. The authors analyze the learning dynamics of shallow and deep linear neural networks and develop a theory linking the structure of training evidence to the emergence of compositional inference. Their findings explain why a model may fail to perform compositional inference despite learning the premises, and how additional linking evidence can enable previously unavailable inferences. The authors evaluate their theoretical predictions through experiments on structured synthetic relational worlds using networks of three different depths (concepts for compositional).

Critical strengths
  • The paper investigates an important question in compositional learning: how neural models learn to combine known relations into previously unseen inferences.
  • The paper combines theoretical analysis with controlled experiments to investigate how compositional reasoning emerges during learning.
  • The findings distinguish between compositional failure caused by slow learning and failure caused by insufficient evidence, even when the model has learned the premises. They also explain how additional linking evidence can enable previously unavailable inferences.
  • The theory makes predictions about the learning trajectories and emergence times of individual compositional laws, going beyond aggregate performance.
  • The authors evaluate these predictions across independently generated relational worlds and three network depths, providing evidence for the theory within the studied setting.
Critical weaknesses
  • Although the findings are interesting, the analysis and experiments are limited to shallow and deep linear networks. The connection to nonlinear architectures, including Transformers used for autoregressive generation, remains untested.
  • The experiments evaluate network depths of only one, two, and three layers. Although the theoretical formulation accommodates general depth, further evaluation would help establish whether its predictive accuracy persists in deeper networks.
  • The evaluation focuses on learning continuous relational embeddings and retrieving a target entity from a predefined candidate set. Generalization to more complex prediction settings requires further investigation.
  • All synthetic worlds use predefined additive relations. This formulation excludes noncommutative composition, where applying relation A followed by B differs from applying B followed by A, limiting the range of compositional structures covered by the study.
  • For deeper networks, the authors integrate the derived dynamics from the same initial weights as the trained networks. The close agreement supports their approximation of training dynamics, but does not show that a simple spectral summary predicts emergence across architectures or settings.
Questions for authors

Please address the weaknesses raised above.

2 comments

We thank the reviewer for the constructive assessment and address each point below.

1. Nonlinear architectures. "The connection to nonlinear architectures, including Transformers... remains untested."

While our experiments use linear networks, our theory separates what depends on linearity from what holds across architectures (Section 6, Paragraph 4). Because Proposition 4.1 depends only on the solution set of AE=CAE = C, it determines identifiability and the effect of linking facts for any nonlinear model that can represent those embeddings (Section 4.2). By contrast, our timing predictions rely on linear dynamics, which have previously anticipated key learning phenomena in nonlinear networks and Transformers (Section 1, Paragraph 3). To test whether this spectral ordering carries over to nonlinear models directly, we outline an empirical test checking whether multi-hop compositions on knowledge-graph benchmarks (FB15k-237 and WN18RR) emerge in the order that the spectrum of their training facts predicts (Section 6, Paragraph 4).

2. Greater network depths (N=4N=4 and N=5N=5). "The experiments evaluate network depths of only one, two, and three layers."

We extended Experiment 1 to N=4N = 4 and N=5N = 5 on 30 worlds each, using the main experiment’s settings and no fitted parameters (Section 5.1, Paragraph 5; Appendix C.12, Table C.13). All 480 law-worlds per depth emerged in both the networks and the predictions, with median relative timing errors of 0.0% (N=4N = 4) and 0.5% (N=5N = 5) and trajectory agreement of R2=0.98R^2 = 0.98 to 1.001.00. The theory also predicted the continued narrowing of the spread across laws, to 1.15-fold at N=4N = 4 (1.16 predicted) and 1.05-fold at N=5N = 5 (1.06 predicted). Agreement on individual emergence times is looser at these depths (R2R^2 = 0.87 and 0.90), in part because the 16 laws now emerge within 5–15% of each other, which leaves little variance to explain.

3. Fixed candidate sets. "The evaluation focuses on learning continuous relational embeddings and retrieving a target entity from a predefined candidate set."

We agree that richer prediction settings, such as generative prediction, remain untested, and Section 6 (Paragraph 4) lists evaluation within a fixed candidate set as a limitation. Within our framework, however, the theory predicts the composed point itself, and the candidate set only determines how that point is scored (Section 3.3). Alongside retrieval accuracy, we report geometric error, the distance between the composed point and its true target in embedding space (Figure C.3). The theory predicts these continuous trajectories with R2≥0.997R^2 \ge 0.997 in all experiments (Table C.1), and across 40 randomly generated designs, candidate spacing is uncorrelated with timing error (∣r∣≤0.08|r| \le 0.08; Appendix C.9).

4. Non-commutative composition. "All synthetic worlds use predefined additive relations. This formulation excludes noncommutative composition..."

Our framework extends beyond additive laws in two ways (Section 6, Paragraph 5; Appendix A.7). Commuting multiplicative relations reduce directly to our linear setting after taking logarithms. Non-commutative matrix relations (Rz=RyRxR_z = R_y R_x) make the constraints bilinear, so the consistent solutions form a curved manifold. Linearizing around a consistent solution θ∗\theta^{\ast} shows that the null space of the constraint Jacobian J(θ∗)J(\theta^{\ast}) governs local identifiability, while the spectrum of J(θ∗)⊤J(θ∗)J(\theta^{\ast})^\top J(\theta^{\ast}) sets local learning rates (Rothenberg, 1971). Developing a global dynamical theory for non-commutative relations remains an open problem.

5. Spectral summary vs. numerical integration. "The close agreement supports their approximation of training dynamics, but does not show that a simple spectral summary predicts emergence across architectures or settings."

To isolate what a spectral summary predicts without integrating coupled layers from initial weights, we decomposed the deep dynamics in the eigenbasis of HH (Section 4.4; Appendix A.5). At N=1N = 1, emergence follows in closed form from the eigenvalues of HH, the law’s loadings on them and the initial errors (Proposition 4.2). At N≥2N \ge 2, each isolated mode grows at a rate set by ck=λksk=∣∣qk⊤B∣∣2c_k = \lambda_k s_k = ||q_k^\top B||_2, which depends only on the training constraints AA and CC (Table C.6). This summary alone reproduces the compression of emergence times from 5.4-fold at N=1N = 1 to 1.8-fold at N=2N = 2, close to the networks’ 2.1-fold (Section 5.1, Paragraph 4; Table C.7). What it leaves out is the interaction between modes through the shared d×dd \times d layers. Without it, the summary overestimates emergence times 2.8-fold at N=2N = 2 and 7.1-fold at N=3N = 3, and misses the further compression to 1.2-fold at N=3N = 3 (Tables C.7, C.8). Individual emergence times at depth therefore require the full dynamics.

Thank you to the authors for their responses. These responses resolved my initial confusion and concerns about the paper. The additional explanation better supports the theory and provides more comprehensive coverage. Based on my knowledge, they are convincing, so I am increasing my initial score.

Reviewer_Sp6nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies compositional learning in additive relational embeddings. Each entity has a position x_a ∈ ℝ^16, each relation is a displacement r, and training minimizes ‖AE − C‖²/(2|D|), combining fact errors x_a + r − x_b with anchor penalties. A shallow model learns E directly; deeper models factorize E = W_N⋯W_1, changing the optimization dynamics without restricting the representable embeddings. A law such as r_z = r_x + r_y is inferred from observed relational constraints rather than directly imposed as a training penalty.

  • Theory. Let H = AᵀA/|D|. Assuming a solution consistent with the training facts also satisfies the law, Prop. 4.1 states that the law holds in every consistent solution exactly when its contrast lies in the positive eigenspace of H. The authors identify this as the classical estimability criterion. The measure ρ = ‖P₀ℓ‖/‖ℓ‖ quantifies the contrast's unresolved projection. Prop. 4.2 expresses the shallow law residual as a sum of exponentially decaying modes. Retrieval also depends on source, target and distractor geometry. The shallow predictor uses a discrete mean-update solution; deeper predictions numerically integrate coupled factor dynamics, under which the end-to-end null-space component can change.
  • Experiments. Three experiments each use 200 synthetic worlds per depth. Experiment 1 reports different emergence times for 16 laws, with the range of median times shrinking from 5.4× at N = 1 to 1.2× at N = 3. Experiment 2 contrasts an underdetermined composition, with generally poor retrieval, against the same structure after one closing fact resolves its ambiguity. Experiment 3 adds a link that determines 80 withheld cross-lattice queries and observes their acquisition alongside transient premise forgetting. Predicted and observed trajectories agree closely; event-time agreement is measured on the sampled evaluation grids.
  • Framing. The results are offered as a qualitative account of differences in compositional learning and knowledge integration, motivated by human associative inference, asynchronous generalization and knowledge assembly.
Critical strengths
  1. The framework separates insufficient evidence from slow learning in a clear and useful way. Prop. 4.1 identifies when the observed facts uniquely determine a compositional law within the additive model class; the trajectory analysis then asks when that law becomes retrievable. The linking experiments make this distinction tangible by showing how one additional fact changes what can be inferred. I also find the transient forgetting of premises interesting, since resolving the relative position of two lattices does not itself require disrupting knowledge within them. The use of 200 worlds per depth, explicit event definitions and replay checks gives these observations a solid empirical basis within the chosen setting.
Critical weaknesses
  1. The additional mechanistic insight beyond established linear-learning theory needs to be clearer.

    The mathematical framework is coherent: Prop. 4.1 applies classical estimability, as the authors acknowledge, and Prop. 4.2 follows from the spectral decomposition of a quadratic objective. Applying these tools to relational laws is useful. My question is what this application teaches us beyond existing accounts of learning and generalization in linear networks, such as Saxe et al. (2019).

    The close agreement between predicted and observed trajectories is meaningful. The shallow predictor solves the discrete mean update, while the deeper predictors integrate the derived factor dynamics from the same initial weights without fitting the observed trajectories. This supports the approximation to per-row SGD. However, it does not isolate why a particular law emerges earlier: spectral structure, initialization and retrieval geometry all enter the prediction, as Appendix C.1 acknowledges. For the deeper models, a similarly clear explanation of why the spread in emergence times shrinks from 5.4× to 1.2× is still missing.

    A controlled change in constraint placement while holding presentation frequencies fixed would help establish what topology contributes. Frequency is already encoded in H, so a strong frequency baseline would not refute the spectral account; it would clarify its added explanatory value. Such comparisons should also distinguish differences between laws from variation across worlds within a law. The 3,200 law–world pairs are repeated realizations of a smaller collection of structures, not independent topology designs.

    The linking experiments currently resolve a single token-space null direction. Their time courses and transient forgetting are interesting, but the static outcome is straightforward: in Experiment 3, fixing the relative translation determines the cross-lattice queries. A case in which a link resolves only some of several ambiguities would demonstrate more of the criterion's diagnostic value. Throughout this discussion, non-identifiability should mean that the evidence permits conflicting answers, rather than that an optimizer cannot happen to retrieve the correct one.

  2. The cognitive interpretation remains plausible but too loosely connected to the motivating evidence.

    There are reasonable parallels to human learning: familiar premises can coexist with poor composition, different relations can be learned at different times, and a new fact can connect previously separate knowledge. The paper appropriately presents these as qualitative connections. What is less clear is whether its particular mechanism explains the relevant behavioral findings, or makes a prediction that distinguishes it from other models.

    Knowledge assembly provides a concrete example. In Nelli et al. (2023), most participants integrated the lists after 20 boundary-training trials, and the computational account used certainty-dependent mutual updates. Here, median times to 72/80 correct cross-lattice queries are 24,082, 3,771 and 2,250 epochs. These counts cannot be directly compared with human trials because the tasks and learning rates differ. More importantly, every post-link epoch rehearses the original facts and anchors, whereas the human boundary phase presents only the boundary pair. Understanding how much the present result depends on rehearsal would make the proposed connection more informative.

    The comparison would also benefit from Lippl et al. (2024), on additive representations and learning dynamics in transitive inference, and Miconi and Kay (2025), on rapid list linking in plastic neural networks. Their tasks differ from the multi-relation grids here, so these works do not make the present study redundant. They do provide useful alternatives against which to explain its contribution.

    The identifiability criterion already has a scientific use within the additive setting. My concern is therefore not the absence of a practical application by itself, but whether the broader cognitive interpretation adds a testable explanation. A distinctive behavioral prediction, evaluated against an alternative account, would make that contribution much easier to assess.

Questions for authors
  1. Explaining the effect of depth. Could you explain which feature of the coupled dynamics compresses the spread of law-specific emergence times from 5.4× to 1.2×? The trajectory predictions are accurate, but the underlying explanation remains less clear. A simple approximation or controlled analysis separating depth from initialization and effective learning speed would help identify the additional mechanistic insight.

  2. Contribution of constraint topology. Could a controlled comparison separate the effects of constraint placement, presentation frequency and retrieval geometry on emergence times? For example, changing constraint placement while keeping presentation counts fixed would help establish what topology contributes. Any predictor comparison would also benefit from distinguishing between-law differences from variation across worlds within the same law.

  3. Connection to human knowledge assembly. How much does Experiment 3 depend on rehearsing the original facts and anchors after the link is introduced? This differs from the boundary-only phase in Nelli et al., so clarifying the role of rehearsal would help identify which aspect of human assembly the model explains. What prediction distinguishes this account from the related models of Lippl et al. or Miconi and Kay?

  4. Interpreting ambiguity in Experiment 2. Could you confirm whether the two blocks use separate relation tokens, giving ρ = 1/√15 under the reconstruction described above? It would also help to clarify whether the statement that further training cannot settle the law refers to unique determination by the evidence. An optimizer may still select a correct answer even when other consistent solutions disagree.

  5. Diagnostic reach of the framework. In a setting with several unresolved directions, what would the framework predict when a new link resolves only some of them? An analytical example or focused experiment could show whether it distinguishes queries that become identifiable from those that remain ambiguous, and clarify what additional predictions can be made about their subsequent retrieval dynamics.

2 comments
AuthorsOpenReview ↗

Response (Part 1 of 2)

We thank the reviewer for a thorough and constructive review. Each question led us to add a new theoretical derivation or controlled experiment in the revised manuscript.

On mechanistic insight beyond established linear-learning theory. "The additional mechanistic insight beyond established linear-learning theory needs to be clearer."

Standard spectral analyses, such as Saxe et al. (2019), follow outputs that a network is explicitly trained to produce, tying each output to a single mode of the input-output map (Section 1, Paragraphs 3-4; Section 2, Paragraphs 2-3). By contrast, training never targets a compositional law directly. Because a law's error projects across many spectral modes of HH at once and can include unconstrained null-space directions, no single mode determines whether a network can learn the law or when it will emerge. Analyzing this multi-mode projection separates slow learning from non-identifiable evidence before training begins (Section 4.2), shows how global constraint topology governs timing beyond local presentation frequency (Section 5.1), and explains why network depth compresses timing differences across laws (Section 4.4).

On Q1 (Depth compression). "Could you explain which feature of the coupled dynamics compresses the spread of law-specific emergence times from 5.4x to 1.2x?"

Decomposing the factorized dynamics in the eigenbasis of HH reveals two mechanisms that drive this compression (Section 4.4; Section 5.1, Paragraph 4; Appendices A.5 and C.8). First, within each isolated mode, target distance cancels out the eigenvalue. At N=1N=1, mode kk decays at rate λk\lambda_k, which varies 514-fold within a law (Table C.6). At N≥2N \ge 2, an isolated mode grows at rate ck=λkskc_k = \lambda_k s_k, where sk=∥qk⊤B∥2/λks_k = \Vert q_k^\top B \Vert_2 / \lambda_k is its target distance from the origin. Because weakly constrained modes have targets that lie correspondingly further away, λk\lambda_k cancels identically in ck=∥qk⊤B∥2c_k = \Vert q_k^\top B \Vert_2, reducing the 514-fold spread in λk\lambda_k to a 6.0-fold spread in ckc_k. An isolated-mode model with this cancellation alone narrows the spread across laws from 5.4-fold at N=1N=1 to 1.8-fold at N=2N=2, matching the networks' 2.1-fold spread closely (Table C.7). Second, all modes update and multiply the same d×dd \times d hidden factors, so fast-growing modes enlarge these shared factors early and accelerate slower modes. Giving each mode private d×dd \times d factors widens the within-world spread at N=3N=3 from 1.53-fold to 2.82-fold in 29 of 30 worlds (p=5.8×10−8p = 5.8 \times 10^{-8}, Table C.8). This compression holds across a 4-fold range of initialization scales and a 2-fold range of learning rates (Appendix C.7, Table C.5). Additionally, a uniform speed-up of all modes does not account for it, as the shallow solution run on a clock set by the network’s layer norms misses the emergence times of individual laws (R2≤0.14R^2 \le 0.14 at N=2N = 2; Appendix C.8).

On Q2 (Topology vs. frequency). "Could a controlled comparison separate the effects of constraint placement, presentation frequency and retrieval geometry on emergence times?"

We ran two new analyses to separate these effects (Section 5.1, Paragraphs 2-3; Appendices C.6 and C.9). First, we evaluated 40 random lattice designs under both uniform presentation counts (isolating constraint topology) and unequal counts (Appendix B.1; Appendix C.9, Table C.9). With no fitted parameters, our spectral theory predicted emergence times accurately in both regimes (R2=0.93R^2 = 0.93 to 1.001.00), whereas a regression on the number of times each law’s xx, yy and zz facts were presented per epoch explained at most 46% of held-out variance under uniform counts, where that number tracks lattice size, and failed (R2<0R^2 < 0) under unequal counts. As the reviewer notes, frequency is already encoded in HH, so this baseline shows what the spectrum adds rather than competing with it. Candidate spacing did not correlate with prediction error (∣r∣≤0.08|r| \le 0.08). Second, after subtracting each law’s mean emergence time in Experiment 1, our theory still explained 97.2% to 100.0% of the within-law timing variance across worlds (Table C.4).

AuthorsOpenReview ↗

Response (part 2 of 2)

On Q3 (Rehearsal and related models). "How much does Experiment 3 depend on rehearsing the original facts and anchors after the link is introduced? ... What prediction distinguishes this account from the related models of Lippl et al. or Miconi and Kay?"

Integration in our gradient-trained networks depends critically on rehearsing the original facts alongside the link (Section 5.3, Paragraph 3; Appendix C.11, Table C.11). All 50 worlds per depth integrated the two lattices under full rehearsal, whereas none integrated them when we trained the link with 25% of the original facts, anchors only, or the link alone, matching our theoretical predictions. Without rehearsal, gradient descent satisfies the link by distorting the shared relation vector rxr_x instead of translating the second lattice, which erases xx-premises (2 of 12 retained) while sparing yy-premises (10 of 12, Table C.12). This clarifies which aspect of human assembly the model explains. Participants in Nelli et al. (2023) integrated the lists after about 20 trials on the boundary pair alone, without rehearsing the original lists, which our networks cannot do. The model therefore accounts for which inferences a link makes available and in what order, but not for how quickly people integrate from the boundary pair alone.

Existing assembly models study a single ordering relation (list rank), explaining how a fixed additive representation supports within-list generalization (Lippl et al., 2024) or how certainty-dependent updates (Nelli et al., 2023) and meta-learned plasticity (Miconi & Kay, 2025) link two lists from boundary trials (Section 2, Paragraph 4; Section 6, Paragraph 3). Our multi-relation analysis shows that rapid integration from linking facts alone requires a stabilization mechanism that protects shared relation vectors while entity positions shift, yielding two predictions that these single-relation models do not address (Section 6, Paragraph 3). First, incomplete protection of the linking relation during assembly should selectively disrupt premises using that relation relative to other premises (Table C.12). Second, when a learner acquires three structures separately, a single link between two of them should enable inferences between that pair only (Appendix C.10).

On Q4 (Ambiguity in Experiment 2). "Could you confirm whether the two blocks use separate relation tokens, giving ρ=1/15\rho = 1/\sqrt{15} under the reconstruction described above?"

Yes, the two blocks share no relation tokens, leaving a one-dimensional null space in the open block with ρ=1/15≈0.2582\rho = 1/\sqrt{15} \approx 0.2582, which we define throughout as an unsquared ratio of norms (Appendices A.2 and B.1). We now use identifiable and non-identifiable consistently (Sections 4.2 and 5.2) to clarify that non-identifiability is a property of the training evidence rather than the optimizer. Specifically, we construct two embeddings that both fit every training fact with zero loss (<3×10−28< 3 \times 10^{-28}) while giving the composed law different values (Appendix A.2). As the reviewer notes, an optimizer could still select a correct solution, so Appendix C.5 reports which one learning selects. At N=1N = 1 the free component stays at its initial value, and at N=2,3N = 2, 3 it drifts by a median of 5–8% of the law’s size, leaving held-out accuracy at 1–6% without the link versus 100% with the link (Table C.3).

On Q5 (Partial resolution). "In a setting with several unresolved directions, what would the framework predict when a new link resolves only some of them?"

When ker⁡A\ker A has several free directions, a link removes only the direction along which it projects, making a query identifiable if and only if its unresolved component is parallel to that direction (Section 4.2; Appendix A.6). We tested this in a three-lattice experiment where networks learned lattices AA, BB and CC separately with only AA anchored (Section 5.3, Paragraph 4; Appendix C.10, Figure C.4, Table C.10). A single link between AA and BB made every comparison between them identifiable (ρ=0\rho = 0), while every comparison involving CC remained non-identifiable (ρ\rho between 0.05 and 0.19). Across all 50 worlds per depth, networks reached 100% accuracy on comparisons between AA and BB at the predicted times, while comparisons with CC stayed at chance levels.

Reviewer_rZMaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates the emergence of compositional reasoning in neural networks by analyzing the learning dynamics of deep linear networks within synthetic relational environments. By framing relational learning as a linear constraint problem, the authors derive a spectral theory that predicts when specific compositional laws become mathematically identifiable and when they are successfully retrieved during training. The work formally separates conditions in which compositional generalization fails due to slow learning from those in which it fails due to fundamentally underdetermined evidence. Furthermore, it demonstrates how introducing a single linking fact can resolve this ambiguity.

Critical strengths

The paper provides a clear theoretical background on compositional learning and a clear mathematical distinction between laws that are identifiable but slow to learn and those that are undetermined, providing a good ground to evaluate learning behaviors on them.

The theory provides an interesting potential explanation for behaviors observed in more complex ML methods, such as the sudden emergence of capabilities (e.g., due to linking facts) and forgetting in continual learning (e.g., due to restructuring).

Critical weaknesses

The theoretical results and experiments are limited to linear neural networks and simple additive relational environments. This leaves a gap in understanding whether the same mathematical principles influence learning in real-world, non-linear ML models.

Questions for authors
  • Q1: Given the reliance on additive laws, do you think that your spectral theory could be adapted to environments requiring non-additive relational rules (e.g., explicit symbolic logic or multiplicative interactions)?
2 comments

We thank the reviewer for the positive assessment of our work and for the constructive questions regarding the scope of our linear and additive assumptions.

On the restriction to linear neural networks. "The theoretical results and experiments are limited to linear neural networks and simple additive relational environments."

Our theoretical results separate into an algebraic identifiability criterion, which holds across architectures, and dynamical timing predictions, which rely on linear dynamics (Section 6, Paragraph 4). Because Proposition 4.1 depends only on the set of embeddings that satisfy the training constraints AE=CAE = C, whether a compositional law is identifiable and which unseen inferences a linking fact unlocks apply unchanged to any nonlinear model capable of representing those embeddings (Section 4.2). By contrast, our trajectory predictions rely on linear learning dynamics. Exact analyses of deep linear networks have consistently anticipated learning phenomena in nonlinear architectures and Transformers (Section 1, Paragraph 3). To connect our timing predictions to nonlinear models directly, we outline a concrete empirical test asking whether multi-hop compositions on knowledge-graph benchmarks, such as FB15k-237 and WN18RR, emerge in the order predicted by the spectrum of their training facts (Section 6, Paragraph 4).

On adapting the theory to non-additive rules. "...do you think that your spectral theory could be adapted to environments requiring non-additive relational rules (e.g., explicit symbolic logic or multiplicative interactions)?"

Yes, our framework extends beyond additive rules in two ways (now explained in Section 6, Paragraph 5 and Appendix A.7). First, relations that compose by commuting elementwise multiplication (xb=r⊙xax_b = r \odot x_a) reduce directly to our linear setting after taking logarithms (log⁡xb=log⁡r+log⁡xa\log x_b = \log r + \log x_a), so Propositions 4.1 and 4.2 apply unchanged in log-coordinates. Second, for non-commutative relations acting as matrices (Rz=RyRxR_z = R_y R_x), the training constraints become bilinear and the set of consistent solutions forms a curved manifold rather than a flat subspace. Linearizing the constraints around a consistent solution θ∗\theta^{\ast} yields a local identifiability criterion and local spectral decay rates governed by the constraint Jacobian J(θ∗)J(\theta^{\ast}) in place of AA (Rothenberg, 1971). Under this local criterion, a composition is locally identifiable when its derivative has no component in the null space of J(θ∗)J(\theta^{\ast}). Developing a global dynamical theory for non-commutative composition remains an open problem for future work.

Explicit symbolic logic falls outside our framework as such, since the criterion compares continuous solutions of learned parameters. It applies to symbolic rules only once they are realized as constraints on learned representations, as in the two cases above.

Thank you for the detailed response and revisions. I find the distinction between the identifiability criterion and the learning dynamics helpful. While the applicability to nonlinear architectures remains to be tested, I think the authors have sufficiently clarified the scope and limitations of their approach. I will therefore maintain my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revisions

We thank all three reviewers for their constructive feedback and for highlighting the value of separating compositional failures caused by slow learning from those caused by insufficient evidence. In the revised manuscript (with changes highlighted in blue), we have added new theoretical derivations and experiments that directly address the questions raised across the reviews, alongside edits throughout the text to improve clarity and standardize terminology.

Summary of New Theoretical Results and Experiments

  • Mechanism of depth compression (Reviewers Sp6n and SYFJ). We derived an eigenbasis decomposition of the deep factorized dynamics (Section 4.4, Appendix A.5) and ran controlled ablations (Appendix C.8, Tables C.6-C.8) showing that depth compresses emergence times through two mechanisms, namely exact cancellation of the constraint eigenvalue against target distance within isolated modes (N=1N=1 to N=2N=2) and cross-mode acceleration through shared hidden factors (N≥3N \ge 3). This compression is invariant across initialization scales and learning rates (Appendix C.7, Table C.5) and continues at depths N=4N=4 and N=5N=5 (Appendix C.12, Table C.13).
  • Constraint topology vs. presentation frequency (Reviewers Sp6n and SYFJ). Across 40 newly generated random lattice topologies evaluated under both uniform and unequal presentation counts (Section 5.1, Appendix C.9, Table C.9), our parameter-free spectral predictions achieve R2=0.93R^2 = 0.93 to 1.001.00, whereas presentation frequency alone explains at most 46% of the variance under uniform counts and fails (R2<0R^2 < 0) under unequal counts. Within-law timing predictions across random worlds also achieve R2=0.972R^2 = 0.972 to 1.0001.000 (Appendix C.6, Table C.4).
  • Selective unlocking across multiple structures (Reviewer Sp6n). We derived the exact null-space projection for multi-dimensional unconstrained subspaces (Section 4.2, Appendix A.6) and tested it in a new three-lattice experiment (Section 5.3, Appendix C.10, Figure C.4, Table C.10), confirming that a single link between lattices AA and BB unlocks all A↔BA \leftrightarrow B inferences at the predicted times while leaving comparisons involving an unlinked lattice CC at chance.
  • Role of rehearsal in knowledge integration (Reviewer Sp6n). New rehearsal ablations in Experiment 3 (Section 5.3, Appendix C.11, Tables C.11-C.12) show that without rehearsing original premises alongside the linking fact, gradient descent absorbs the link by distorting the shared relation vector rather than shifting the unanchored lattice. This suggests that rapid integration in humans relies on a mechanism that protects established relation vectors (Section 6, Paragraph 3).
  • Scope across nonlinear architectures and non-additive relations (Reviewers rZMa and SYFJ). We expanded Section 6 (Paragraphs 4-5) and added Appendix A.7. These additions clarify that Proposition 4.1 does not depend on the network being linear and therefore also applies to nonlinear architectures. They also outline a direct empirical test of our timing predictions on knowledge-graph benchmarks (FB15k-237 and WN18RR), and extend our framework to commuting multiplicative relations and, locally, to non-commutative matrix relations via the constraint Jacobian.

We address each reviewer's remaining points in the individual responses below. The main text remains within the 10-page limit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.