acceptodds
Withdrawn from ICLR 2027

Do LLMs Learn Universal Symbolic Representations? An Early Exploration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Exploring internal cognitive patterns of large language models (LLMs) has attracted growing research interest. In this paper, we use interaction patterns as an interpretable and verifiable metric to rigorously decompose the inference logic of an LLM into symbolic AND-OR interactions, with theoretical guarantees of explanatory completeness. We find that mainstream open-weight LLMs exhibit highly consistent symbolic interaction patterns despite substantial differences in parameter scale and training data. Compared with model-specific, non-shared interactions, these cross-model shared interaction patterns exhibit lower structural complexity and weaker positive-negative cancellation, making them more compelling as inference patterns. We further show that a large fraction of LLM prediction scores is attributable to these shared interaction patterns. Overall, our work validates the hypothesis that LLMs encode inherent symbolic patterns and provides a precise and verifiable decomposition of cognitive patterns that are universally shared across LLMs.

Primary area:
interpretability and explainable AI
Keywords:
Representation Learning, Representation Similarity, Cross-Model Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_XBNbOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors investigate 16 classic open-weight LLMs which learn shared symbolic inference patterns. The authors report that despite differences in architecture and parameter scale, diverse LLMs rely on similar interaction patterns to predict the target token given the same input prompt. The authors also report several phenomenons.

Critical strengths
  1. The author propose an interesting way to analyze the LLMs.
  2. Plenty of open-source LLMs are tested.
Critical weaknesses
  1. There is limited contribution in the work. It seems that the technique is merely a combination of the existing work.
  2. Although the authors report plenty of experimental results, it seems that the statistical result may not lead to the explanable interpretation.
Questions for authors
  1. How is the interaction function IandI^{and} determined?
  2. Other than the AND-OR logic, the LLMs seem to be able to handle comlex tasks. Are those tasks be explainable?
Reviewer_WXcWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper applies the AND-OR interaction framework from Chen et al. (2024) and Ren et al. (2024a) to cross-model comparison. The procedure is: select 10 content words from each prompt as variables, run forward inference on all 2102^{10} masked states to obtain v(xS)v(x_S), then decompose the output into sparse AND/OR interaction effects via Möbius transform and L1 optimization. The authors extract interactions pairwise across 16 open-weight LLMs (360M–32B) on two datasets, and report that 62–78% of the prediction score is attributable to interactions shared across model pairs. Shared interactions tend to be lower-order, exhibit less positive-negative cancellation, and have stronger individual effects. The paper concludes that LLMs encode universal symbolic inference patterns.

Critical strengths

The paper asks a good question—whether different LLMs converge to similar inference logic. Compared to methods like CKA that only produce a scalar similarity, the AND-OR interaction framework can point to which specific token subsets carry shared combinatorial effects. That granularity is meaningful. The interactions shown in Figure 2 (e.g., {strong, spatial} → "variation") do have readable semantic content. The experiments cover 16 models from multiple independent families, and the ablations (Tables 3–5) are solid.

Critical weaknesses

1. The paper claims to reveal shared symbolic reasoning structures inside models, but the method can only decompose input-output behavior. There is a gap between the two that has not been bridged.

The Möbius transform holds for any set function—the proof in Appendix B never uses the fact that vv is a neural network. So universal matching (Eq. 2) is a property of the mathematical tool, not a finding about the model being analyzed. Yet the abstract calls the decomposition coefficients "cognitive patterns," and Section 2.2 calls them "primitive inference patterns," giving the impression that the model internally organizes its computation along AND-OR lines. Supporting that would require some form of causal verification—for instance, locating the attention heads responsible for a shared interaction and ablating them. The paper does not attempt this.

What concerns me more is the following. The Möbius coefficients are determined by the values of v(xS)v(x_S) across 2n2^n masked states. The AND-OR split does introduce a nonlinear optimization over {γT}\{\gamma_T\}, but if two models produce similar v(xS)v(x_S) across those states, running the same optimization on similar functions will naturally tend to yield similar decompositions. So "shared interactions" may simply be another way of saying "behavioral similarity." If the paper compared corr(vA(xS),vB(xS))\text{corr}(v_A(x_S), v_B(x_S)) against κ\kappa, it could settle whether the interaction decomposition carries information beyond behavioral correlation. This experiment is cheap with the existing data, but the paper does not do it.

To be fair, behavioral consistency across all 2102^{10} masked states is stronger than just agreeing on the unmasked prediction—I acknowledge that. But given that all these models were pretrained on massive English corpora with the same next-token objective, similar responses to common collocations and syntactic structures are not surprising.

2. Even setting the methodological issues aside, the finding itself is not very surprising, and the more interesting questions are left untouched.

Most modern open-weight LLMs were pretrained on heavily overlapping English data. Sharing common inference structures is a natural consequence. The paper spends considerable effort showing that "the shared part exists," but that is not the most curious question here. What would be more interesting is: which interactions differ across models? Can those differences be traced to specific training data or training decisions? Section 5 briefly argues that training data overlap may not be the main cause, but the evidence is indirect (within-family similarity is not necessarily stronger than cross-family), and the paper never really digs into what the non-shared interactions encode. If the work examined differences rather than just commonalities, it would be considerably more valuable to the community.

3. Several experimental design issues stack up and make me uneasy about the conclusions.

Sparsity: Algorithm 1 explicitly minimizes ∥Iand∥1+∥Ior∥1\|I^{\text{and}}\|_1 + \|I^{\text{or}}\|_1, and then the paper reports that the extracted interactions are sparse. I do not think this is entirely circular—doing the same optimization on a random function would probably not yield such a sparse decomposition, so the LLM output function does have some structure that favors sparsity. But the paper does not run that control, nor does it independently verify whether the theoretical conditions in Appendix A are actually satisfied. As it reads now, it looks like the method pursues sparsity and then claims to have discovered it.

Target token selection (lines 363–370) picks the token with the lowest summed rank across both models—i.e., one both models are already confident about. I understand the methodological rationale: analyzing tokens with very low confidence gives poor signal-to-noise. But the paper's conclusion ("universal symbolic representations") goes far beyond this conditioned setting.

κ\kappa has no random baseline. With n=10n=10 there are only 1023 candidate subsets, each model retains K=80K=80, and low-order subsets tend to have the strongest effects and be selected by both models. The chance overlap under these conditions is not negligible. Without a null model, it is hard to tell whether 62–78% is actually high.

Questions for authors
  1. For each model pair and prompt, compute corr(vA(xS),vB(xS))\text{corr}(v_A(x_S), v_B(x_S)) across all 2102^{10} masked states and compare it against κ\kappa. The data already exist; this should be quick. If the two are strongly correlated, the added value of the interaction decomposition needs to be re-argued.

  2. Randomly permute interaction identities or signs and recompute κ\kappa. What are the expected value and variance?

  3. Switch the target token to the one where the two models disagree most. How does κ\kappa change?

  4. What do the non-shared interactions encode? Can they be linked to known training differences (data sources, training stages, etc.) between models?

Reviewer_rtm3OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether different LLMs rely on similar input interactions when predicting the same token. The authors use the existing AND-OR interaction framework, select ten words per prompt, and compare interactions present in both models with effects of the same sign. Experiments on 16 models use financial and general text data. The authors report that shared interactions have larger net effects, involve fewer words, and show less positive-negative cancellation. In a separate BERT experiment, the degree of sharing first increases and then decreases during training.

Critical strengths
  1. The research question is interesting and has both theoretical and practical relevance. Comparing the input interactions used for individual predictions offers a useful perspective alongside existing work on similarity between representation spaces.
  2. The difference in cancellation between shared and non-shared interactions is worth examining. The same overall pattern appears on both datasets, which makes the observation more convincing.
  3. The study covers 16 models ranging from 360M to 32B and includes financial and general text. The appendix also checks sensitivity to prompt count, retained interaction count, and word selection, with reasonably stable results, which helps support the consistency of the observations.
Critical weaknesses
  1. The paper equates AND-OR interactions among input tokens with symbolic representations, but such interactions may merely reflect statistical correlations rather than genuine symbolic reasoning mechanisms.
  2. The universal-matching property guarantees the reconstruction of model outputs but does not establish that different LLMs learn the same internal symbolic mechanisms.
  3. The experiments primarily focus on next-token prediction, without evaluating explicit logical or symbolic reasoning tasks. This weakens the connection between the experimental findings and the paper's central claim.
  4. The evaluation uses only 100 text samples and 10 selected input variables per sample. The sensitivity of the findings to variable selection, masking strategies, and interaction extraction settings remains unclear.
  5. The core AND-OR interaction decomposition and its theoretical guarantees are adopted from prior work. The main contribution lies in applying existing techniques to cross-model comparisons rather than developing a fundamentally new method.
Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.