acceptodds
Under review as a conference paper at ICLR 2027

Causal and Interpretable Structures in LLM Compositional Tasks

Abstract

Large language models are able to solve tasks whose answers depend on not only individual input tokens, but also on relations among them. How is such relational information represented and processed across transformer layers? We study activations from ensembles of prompts that require inferring relationships between three tokens corresponding to a cyclic concept (months, hours, weekdays, and musical notes) to correctly predict the next token. Across model families (Llama, Qwen, Gemma, and Mistral) and cyclic concepts, we find a consistent layerwise progression in how the joint dependence among the tokens is geometrically organized and causally used: intermediate layers use a joint representation based on the inferred relationship between two tokens, while later layers use a joint representation associated with all three tokens to correctly complete the task. We also find other relationships between tokens that are geometrically structured but remain causally inert in the next-token prediction. Crucially, when taken together, these geometric and causal investigations reveal the representation-level mechanism that progressively organizes and composes the relational information to form the answer. More surprisingly, restricting the models to such causally relevant joint representations improves next-token prediction accuracy.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, concept composition, explainable AI, representation geometry, representation learning, causal interventions, manifolds, LLMs

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_X4g4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The main research question is: How do LLMs represent and compose relationships between multiple input tokens to solve tasks? The paper introduce a task: γ=B−A,\gamma = B-A , $ D = C+ \gamma$. A, B and C ranges from 1 to 12. The paper uses functional ANOVA to decompose the representation (last-token residual-stream activations) into individual-variable effects, pairwise interactions and a three-way interaction. They use Fourier analysis and energy fractions to characterize the representations. They also use causal analysis, apply ablation, replacement, and steering to determine whether these representations actually influence predictions.

They find that the formation progression is 2-stage along the layers. Pairwise relation hABh_{AB} form first, around when γ\gamma is established, and 3-way relation hABCh_{ABC} form afterwards, around the formation of the answer.

Some relations are causally relevant while others are not. hABh_{AB} is important in middle layers, hABCh_{ABC} is important in final layers. Ablating hABh_{AB} also affects the emergence of hABCh_{ABC}. On the other hand, other geometrically structured pairwise interactions like hBCh_{BC} do exists but they have little causal influence.

Critical strengths

The paper gives us a mathematically well-defined representation decomposition with causal experiments. The experiments clearly traces where does the relationship between 2 or 3 variables emerges, how do they interact with each other, and whether they are important or not.

Critical weaknesses
  1. The paper only examined one task: the month prediction. This gives a close-form and symmetric representation for all factors. This is not the case in general during LLM inference. So, the title "Causal and Interpretable Structures in LLM Compositional Tasks" may have a broader claim than what the experiment actually suggests.
Questions for authors
  1. How will your analysis generalize, if each factor takes more than 1 token to represent? In specific, what if factors have an indefinite (and not same) amount of tokens
1 comment

Dear X4g4,

Thanks for your positive assessment of our work and for finding the 2-stage behavior interesting. We appreciate your questions and constructive feedback on the scope and generalizability of our analysis. Below you will find our response to your concerns.

Q1.

How will your analysis generalize, if each factor takes more than 1 token to represent? In specific, what if factors have an indefinite (and not same) amount of tokens

If each factor has an indefinite but same amount of tokens, the analysis generalizes since we perform an ANOVA decomposition at the last-token site.

However, if each factor has an indefinite but not the same amount of tokens, the position of the last-token site changes according to the number of tokens required to represent a factor. One could devise experiments with padding, such as, in the case of numeric tokens and left pad each numeric factor with zeros. This ensures that the last-token site remains at a fixed position.

For every other experiment, this would require architecture-aware changes to the experimental design. One example is RoPE, where one would need to fix the position IDs so that differing lengths do not induce an extraneous rotation at the last token site. We will include this discussion about the experimental design in the paper to guide future work. Thanks for the question!

W1.

The paper only examined one task: the month prediction. This gives a close-form and symmetric representation for all factors. This is not the case in general during LLM inference. So, the title "Causal and Interpretable Structures in LLM Compositional Tasks" may have a broader claim than what the experiment actually suggests.

In the main section, we discuss only the months prediction. In appendices, we also observe consistent findings across hours, weekdays, and musical notes prediction. We will additionally import the discussion about other cyclic domains from appendices to the main discussion (section 7) to address the generalizability concern.

Reviewer_ihNGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors focus on establishing connections among input tokens for tasks solved with large language models. More specifically, the authors study activations from prompt ensembles that require inferring relationships among three tokens corresponding to a cyclic concept to predict the next token correctly. The authors find a consistent layerwise progression, showing how the joint dependence among the tokens is geometrically organized and causally used. Moreover, they find other relationships between tokens that are geometrically structured but remain causally inert in the next-token prediction. Finally, they observe that these geometric and causal investigations reveal the representation-level mechanism that progressively organizes and composes the relational information to form the answer.

Critical strengths
  1. The authors find that H_BC and H_CA are strongly connected by C-B and C-A yet are causally independent; this is a very important result. This is a solid warning against inferring function from geometry.

  2. The ANOVA decomposition is exact and orthogonal. It isolates joint dependence from main effects, which standard probing and PCA on raw activations cannot do.

  3. The paper is well written and clearly explains the LEMMA.

  4. The results cover seven models from 3B to 24B parameters, four domains, top-1 and top-3 metrics, and symmetrized steering variants. This layerwise technique is qualitatively consistent in the appendix figures.

  5. The authors honestly report the negative results also. This paper states that H_ABC transfer is inconsistent across domains and that Gemma transition is more diffusive.

Critical weaknesses
  1. There is no test shown that the "inference" hypothesis is correct.
  2. Layer boundaries such as "17-18" are read visually from plots. The claim of a "sharp" transition lacks a quantitative changepoint estimate.
  3. Replicates differ only in name, noun, and date numbers. This is narrow template variation, so generalization to different phrasing is untested.
  4. Steering is evaluated with a top-3 logit margin, where top-3 is a lenient criterion and a median can hide a wide spread. Top-1 effects on the fraction of prompts where the target becomes the argmax are shown in the main text.
  5. H_AB transfer works partially and is asymmetric: months-to-hours reduces performance; hours-to-months improves it. This is not clearly explained.
Questions for authors
  1. Is H_ABC organization predictive of the model's actual predictions?
  2. Why is hours-to-months asymmetric?
  3. Could you please explain how you formed Equation 2?
  4. Why did you choose these four concepts (months, weekdays, hours, musical notes)?
  5. "We vary A, B, C independently over calendar months while keeping everything else fixed." (line number 39)- is mod 12 fixed for all (days or weekdays)?
  6. Does the distance between A and B have any relation or dependency?
  7. What happens when you retain H_A, H_B, and H_C?
3 comments

Dear ihNG,

Thank you for your positive assessment of our work and we appreciate that you find our results important. We also appreciate your thoughtful questions, which have helped us clarify and strengthen the paper. Below you will find our response to your questions, along with relevant changes that we made to the paper based on your comments.

Q1.

Is H_ABC organization predictive of the model's actual predictions?

HABCH_{ABC} by construction is not conditioned on model's actual prediction. However, the ablation and replacement experiments show that HABCH_{ABC} is indeed necessary and sufficient for the model's actual predictions.

Q2., W5.

Why is hours-to-months asymmetric?

H_AB transfer works partially and is asymmetric: months-to-hours reduces performance; hours-to-months improves it. This is not clearly explained.

Thanks for this interesting question! We hypothesize that different baseline accuracies can explain the asymmetry in cross-domain transfer. To test this hypothesis, we first matched months and hours prompts by their cyclic indices (A,B,C)(A,B,C) across all 10 replicate pairs. For hours-to-months transplantation of HABH_{AB} at layer 15, we then asked whether a correct hours answer helped correct a previously incorrect months answer and vice-versa. For Llama-3.1-8B, these net corrections account for 80.8% of the total net increase in top-1 accuracy. Applying the same analysis in both directions for top-1 accuracy gives:

ModelHours →\to months: fraction of net increaseMonths →\to hours: fraction of net decrease
Llama-3.1-8B80.8±3.4%80.8 \pm 3.4\%80.5±13.2%80.5 \pm 13.2\%
Llama-3.2-3B89.0±13.1%89.0 \pm 13.1\%84.8±4.3%84.8 \pm 4.3\%

Every replicate pair shows the same pattern: months accuracy increases, hours accuracy decreases, and prompts where only one domain was initially correct account for most of each change. This supports the hypothesis that the baseline differences help explain the asymmetry. We have added this analysis in Appendix L.1. However, this does not explain the mechanism behind this asymmetry. We think that it would be an interesting future direction.

Q3.

Could you please explain how you formed Equation 2?

The intuition behind Equation 2 is as follows: we wanted to understand how multiple factors, say AA and BB, jointly influence the activations after averaging out irrelevant factors. For example, if we change the value of AA from "January" to "February", we expect to see a change in the activation. However, if we change values of both (A,B)(A,B) at same time from ("January", "March") to ("February", "April"), the part of the activation that depends on the difference between AA and BB should not change. The functional ANOVA decomposition provides a principled way of averaging out irrelevant factors. For example, take AA and BB. We want to understand how they affect the function FF. The first thing to do is center the ensemble FF to get F−μF-\mu. To figure out the effect of AA, we marginalize the centered ensemble over BB to get a term hAh_A. We similarly get hBh_B after marginalizing over AA. The leftover term is hAB=F−μ−hA−hBh_{AB} = F-\mu-h_A-h_B. Equation (2) is analogous to this derivation but for three factors. Appendix B details all the relevant lemmas, propositions, and theorems. We will update the manuscript to explain the intuition behind Equation (2). Thanks for pointing it out!

Q4.

Why did you choose these four concepts (months, weekdays, hours, musical notes)?

Months, weekdays, and hours are well studied examples of cyclic concepts that lie on a circular low-dimensional manifold, say M\mathcal{M} (see Engels et al., 2025; Modell et al., 2025; and the references in the related works section). We wanted to understand how multiple factors of the same manifold M×M×M\mathcal{M}\times\mathcal{M}\times\mathcal{M} interact when given a composition task, such as the ones studied in our work. Therefore, we used the same concepts already studied in the literature and they also give a definitive unique answer for the task. We additionally included musical notes to get a comparison point for weekdays since both concepts are defined modular-7.

Q5.

"We vary A, B, C independently over calendar months while keeping everything else fixed." (line number 39)- is mod 12 fixed for all (days or weekdays)?

We consider four cyclic domains: months, hours, weekdays, and musical notes. The former two domains are evaluated with respect to mod 12 and the latter two are evaluated against mod 7.

Line 39 is intended to convey that we only change variables A,B,CA,B,C without changing the phrasing. We have updated that sentence to "We vary A, B, C independently over calendar months without changing the phrasing." Thank you for pointing this out.

AuthorsOpenReview ↗

Rebuttal (cont.)

Q6.

Does the distance between A and B have any relation or dependency?

Yes, the distance between AA and BB is described by γ\gamma. Figure 1 shows that HABH_{AB} is clustered according to γ\gamma, with an apparent circular arrangement. Our Fourier analysis in Section 3 quantifies the extent of this clustering behavior by defining mode fraction. Much of the variation in HABH_{AB} can be described by γ\gamma since the mode fraction is ∼0.9\sim 0.9. However, we have not explicitly modeled this geometric arrangement in the residual space directly by computing the distance between each cluster. One can invoke a low-rank approximation to create such a model (see Appendix K of Engels et al. (2025)), which we leave to future work.

Q7.

What happens when you retain H_A, H_B, and H_C?

On retention of additive terms HA,HB,HCH_{A}, H_B, H_C along with the mean, the layerwise behavior mimics that of μ\mu-only baseline as shown in Figure 3. The top-3 accuracy drops to near chance around layer 15. In the table below, we have identified the layer where the largest absolute difference between the proposed experiment and the μ\mu-only baseline experiment is observed.

MetricLayerμ\mu-only accuracyRetained accuracyMaximum absolute difference
Top-11 accuracy31318.3%±0.0%8.3\% \pm 0.0\%1.8%±0.8%1.8\% \pm 0.8\%−6.6±0.8-6.6 \pm 0.8
Top-33 accuracy222218.6%±3.5%18.6\% \pm 3.5\%23.4%±1.0%23.4\% \pm 1.0\%+4.8±3.7+4.8 \pm 3.7

The top-1 and top-3 baseline accuracies are ∼62%\sim 62\% and ∼85%\sim 85\%. We have added the sweep over every layer in Appendix I.1. Thank you for this question!

W1.

There is no test shown that the "inference" hypothesis is correct.

The "interference" hypothesis introduced in the Discussion is speculative and we have improved the Discussion section to explicitly call it conjectural.

W2.

Layer boundaries such as "17-18" are read visually from plots. The claim of a "sharp" transition lacks a quantitative changepoint estimate.

Layer boundaries are read off from several complementary experiments including: Fourier analysis, energy analysis, ablations, replacements, and steering.

W3.

Replicates differ only in name, noun, and date numbers. This is narrow template variation, so generalization to different phrasing is untested.

We use one prompt phrasing per cyclic domain, with different phrasings across domains. We have not tested generalization to alternative phrasings within each domain.

W4.

Steering is evaluated with a top-3 logit margin, where top-3 is a lenient criterion and a median can hide a wide spread. Top-1 effects on the fraction of prompts where the target becomes the argmax are shown in the main text.

We agree with the reviewer that top-3 logit margin is lenient. We chose this metric to match the rest of the paper that uses a top-3 accuracy in the main section. Reviewer VazF had a similar concern with Q1. We have added an additional analysis that reports the top-1 accuracy on shifting a single factor in Appendix G. We observe the same behavior: HABH_{AB} retains causal relevance in the middle layers 15--17 of Llama-3.1-8B, but the relevance shifts to HABCH_{ABC} at layer 18. We will include this point in the steering section. Thank you for raising this point.

Reviewer_ihNGOpenReview ↗

Response

I acknowledge and appreciate the effort of the authors.

Reviewer_VazFOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This work analyzes the layer-wise "geometry" of next token depending on a previous token vs several previous tokens (and the relationship among them), where the next token belongs to a cyclic concept (hours, days of week, months, musical notes). It uses a functional ANOVA decomposition of last-token activations over prompts in which three cyclic tokens (A, B, C) vary independently, separating out pairwise (H_AB, H_BC, H_CA) and three-way (H_ABC) interaction terms. It discovers progressive representation: lower layers infer relation between two tokens, and later layers tie this to a third token for correctly computing the next token. Then it shows some of this representation is used for predicting the next token, by conducting ablation, replacement, steering, and cross-domain transplant experiments. They also discover that restricting the model to such joint representations improved next-token prediction. (although not clear to me why this would be "surprising").

Edit: The authors answered most of my questions (and some weakness points) satisfactorily, even though some issues remain. If I had the option, I would have raised my rating to 3.5 score. But since all reviewers gave 3 (at the time of writing), I will be nice and raise my rating to 4, because I think this paper deserves a rating higher than 3 at the bottom line.

Critical strengths
  • clean, principled decomposition. The exact functional ANOVA analysis (without approximation) is cool. The Fourier-based mode fraction is well matched to cyclic variables.

  • Geometry paired with causality. The paper does not stop at showing structure. Ablation, replacement, and steering show a handoff from H_AB to H_ABC at layers ~17–18. The finding that H_BC and H_CA are strongly organized yet causally inert is a useful cautionary result for interpretability work that equates decodable structure with use. (However, I wish the authors tried to explain the function of H_BC and H_CA and how the model is restoring this info upon ablation!)

  • Breadth and robustness. Results are repeated across seven models, four cyclic domains, and 10 prompt replicates, with top-1 and top-3 metrics, two steering constructions (averaged and symmetrized).

  • section 6: nice cross-domain transfer experiments

  • the paper is generally well organized (but see org weakness too)

Critical weaknesses
  • Limited mechanistic interpretability. The claim of revealing “the representation-level mechanism” is overstated. The paper shows that terms are organized, and some are necessary or sufficient at the last token, but not how H_ABC is computed from H_AB and C, which heads or MLPs are responsible.

  • the “retaining improves accuracy” claim. The abstract’s claim that restriction “improves next-token prediction” rests on gains over a baseline of ~85% top-3 accuracy. There are no significance tests, no per-γ breakdown (the baseline fails mostly at large γ), and the “destructive interference” explanation in the Discussion section is untested.

  • why is this finding "surprising"?

  • the Discussion section is too thin to my taste, and there's no Conclusions section, as is expected in this scientific paper genre.

  • missing citations: geometry of concepts papers only from 2025. Consider adding Mikolov et al. (2013), Pennington et al. (2014), Park, Choe & Veitch (2024), etc.

  • The purpose of 4.2 (ENSEMBLE REPLACEMENT) is not clear. the retaining action (of a single interaction ensemble together with the mean) is also not clear. And therefore, it is not clear how "retaining" a factor improves performance.

  • I expected to see an interpretation of the ablation and replacement results in the discussion section, beyond repeating the findings and suggesting a vague "different function" conjecture. It would make the paper much stronger.

Questions for authors
  • Section 4 (ablation and replacement tests): why not change value of a day/month (A, B or C), perhaps changing H_AB or H_BC etc., and see if the next token prediction changes accordingly? This similar to the steering section 5, but focuses on A/B/C as opposed to delta.

  • if ablating H_BC or H_CA doesn't change much, how does the model recover this info, as opposed to H_AB ?

  • Are the steering vectors evaluated on held-out data?

  • How do you know the layer transitions mark where the computation happens, rather than where it is readable at the last token?

  • can you explain or clarify the "retaining action" (Section 4.2)

  • why is the improved perforamce finding "surprising"?

5 comments

Dear VazF,

Thank you for your positive assessment of our work and for recognizing the strengths of the exact ANOVA decomposition. We appreciate the constructive feedback to strengthen our paper. Below you will find answers to the questions raised, along with relevant changes we made to the manuscript, including those in the Discussion section.

Q1.

Section 4 (ablation and replacement tests): why not change value of a day/month (A, B or C), perhaps changing H_AB or H_BC etc., and see if the next token prediction changes accordingly? This similar to the steering section 5, but focuses on A/B/C as opposed to delta.

The suggested test was indeed our motivation to conduct steering experiments of Section 5. We would also like to draw your attention to Appendix K.3, where we change the individual value of each factor A,A, BB, or CC by δ\delta and replace each ensemble (HAB,HABC)(H_{AB}, H_{ABC}) by a symmetrized ensemble shifted by δ\delta. The objective of Appendix K.3 is still to steer but it is much closer to the proposal.

To test your proposal directly, we shifted factors individually:

  • A→A−4A \to A-4 in HABH_{AB},
  • B→B+4B \to B+4 in HBCH_{BC},
  • C→C+4C \to C+4 in HCAH_{CA},
  • C→C+4C \to C+4 in HABCH_{ABC}.

In each case, the expected answer shifts D→D+4D\to D+4 and we computed the top-1 accuracy to check if the next token prediction changes accordingly. The unsteered baseline over all replicates is 2.0±0.2%2.0 \pm 0.2\%. Below we have tabulated the maximum absolute change in top-1 accuracy in steered and unsteered model and marked the layer where each maximum is observed. This shows that the model is sensitive to HABH_{AB} in the middle layers and HABCH_{ABC} in the later layers. In contrast, the model's prediction is insensitive to a change in HBCH_{BC} and HCAH_{CA} across the transformer layers. We have added this experiment as Figure 11, Appendix G.

EnsembleLayerSteered top-1 (mean ±\pm SD)Maximum change = steered top-1 accuracy −- unsteered top-1 accuracy
HABH_{AB}151544.5%±3.8%44.5\% \pm 3.8\%+42.5±3.7+42.5 \pm 3.7
HBCH_{BC}15150.9%±0.5%0.9\% \pm 0.5\%−1.2±0.3-1.2 \pm 0.3
HCAH_{CA}15153.8%±0.7%3.8\% \pm 0.7\%+1.8±0.5+1.8 \pm 0.5
HABCH_{ABC}292970.1%±4.9%70.1\% \pm 4.9\%+68.0±4.8+68.0 \pm 4.8

Q2.

if ablating H_BC or H_CA doesn't change much, how does the model recover this info, as opposed to H_AB ?

There are two possible interpretations of ablation experiments in Section 4.1: either the model does not use the ensembles HBCH_{BC} and HCAH_{CA} for the next-token prediction or the model uses the recovered information. To test the distinction, we conducted a stronger ablation experiment where we repeatedly ablate the causally irrelevant ensembles HBCH_{BC} and HCAH_{CA} after ablation at layer 15. At every subsequent layer 16--31, we recompute the ANOVA decomposition of each head and ablate, say HBCH_{BC}, in each head. We further recompute the ANOVA decomposition of the residual stream after MLP at each layer and ablate HBCH_{BC}. Using the same prompts for Llama-3.1-8B as in the main section, we observe that this strong ablation still does not change top-3 accuracy appreciably.

InterventionTop-3 accuracy
Baseline85.4±3.3%85.4 \pm 3.3\%
Single HBCH_{BC} ablation84.5±3.0%84.5 \pm 3.0\%
Repeated HBCH_{BC} ablation85.7±3.2%85.7 \pm 3.2\%
Single HCAH_{CA} ablation84.4±3.2%84.4 \pm 3.2\%
Repeated HCAH_{CA} ablation84.7±3.1%84.7 \pm 3.1\%

In contrast, a single ablation of HABH_{AB} at layer 15 changed the top-3 accuracy from ∼85%\sim 85\% to ∼50%\sim 50\%. This result supports the interpretation that the model does not use the ensembles HBCH_{BC} and HCAH_{CA} irrespective of whether it can recover the information. We have added this analysis to Appendix H.1. We thank the reviewer for pointing it out.

Q3.

Are the steering vectors evaluated on held-out data?

Our motivation for conducting steering experiments was closer to the premise of Q1 (that is, does a change in HABH_{AB} at a middle layer affect the downstream prediction?), rather than a claim of a novel steering method. We note this at the beginning of Section 5 and additionally mention that our method is equivalent to the difference-of-means steering (Rimsky et al. (2024); Subramani et al. (2022)). As a result, we reported the steering results within the same ensemble and same replicate. We do, however, understand the reviewer's concern and updated the manuscript with an additional Appendix K.2 that evaluates steering vectors on the held-out replicates (50/50 split between train/test). The results still identify HABH_{AB} being causally active between layers 15--17 and HABCH_{ABC} being causally active after layer 18.

Steering experiment in Section 6 can be interpreted as another test of generalizability where we construct target vectors using one domain and use them to steer the second domain. In the revised manuscript, we will explicitly note the role of this additional appendix in the main section. Thank you for your feedback!

AuthorsOpenReview ↗

Rebuttal (cont.)

Q4.

How do you know the layer transitions mark where the computation happens, rather than where it is readable at the last token?

Thank you for bringing up this nuanced question. The reported layer transitions describe the change in the causal relevance of representations at the last token. In Section 4, we describe this phenomenon as a "transition in causal relevance". We will explicitly mention that this transition does not necessarily mark where computation happens.

Our evidence nevertheless goes beyond readability. In Figure 3, we see that the ablation of HABH_{AB} in layers 15--17 decreases the top-3 accuracy from ∼85%\sim 85\% to ∼50%\sim 50\%. This change in accuracy cannot be attributed to readability alone since HABH_{AB} is organized and clustered according to the intermediate variable γ\gamma and not according to the readout sensitive answer D=C+γD = C + \gamma. Appendix F further shows that ablation of HABH_{AB} at layer 15 suppresses the emergence of HABCH_{ABC} at layer 18 by computing the mode and energy fractions.

Another evidence is provided by Section 6 where we replace HABH_{AB} in the months domain using that of hours domain at the same layer. We observe the successful prediction of months answers. Because HABH_{AB} depends only on AA and BB, it cannot specify the final answer D=C+B−AD= C+B-A. This is not what we would expect if the experimental observations were driven by readability alone.

Independent evidence of computation is provided by Feucht et al. (2026). They use prompts, for example, "Q: What month is six months after August? A:" and identify 28 MLP neurons in layer 18 of Llama-3.1-8B responsible for the computation. This lines up with our observation that the causal relevance shifts from HABH_{AB} to HABCH_{ABC} at layer 18.

Q5., W6.

can you explain or clarify the "retaining action" (Section 4.2)

The purpose of 4.2 (ENSEMBLE REPLACEMENT) is not clear. the retaining action (of a single interaction ensemble together with the mean) is also not clear. And therefore, it is not clear how "retaining" a factor improves performance.

Section 4.1 (ablation experiments) identifies which components of the ensemble are necessary at a particular layer for the next-token prediction.

Section 4.2 (replacement experiments) asks a different question: which components of the ensemble are sufficient to output the final answer? That is, we probe for the minimum number of decomposed terms required to perform as well as the unmodified baseline. We observed in Section 4.1 that HABCH_{ABC} is necessary in the later layers to correctly predict the answer. Therefore, it is a natural candidate to test for sufficiency. The same applies for HABH_{AB} in the middle layers.

To test for sufficiency, one could possibly replace the entire activation ensemble with HABCH_{ABC}, that is, ϕ~(A,B,C)=hABC.\widetilde\phi (A,B,C) = h_{ABC}. However, this induces an extremely large change in the activation vector. The average squared-norm of activation vectors before any intervention at layer 22 is EABC∥ϕ(A,B,C)∥22≈475E_{ABC}\|\phi(A,B,C)\|^2_2 \approx 475, but EABC∥hABC∥22≈44E_{ABC}\|h_{ABC}\|^2_2 \approx 44. By retaining only HABCH_{ABC}, we are discarding about ∼91\sim 91% of the energy. In comparison, ∥μ∥22≈371\|\mu\|_2^2 \approx 371, or ∼78\sim 78% of the total energy.

Therefore, we also report replacement with μ\mu alone as a baseline: ϕ~(A,B,C)=μ\widetilde\phi (A,B,C) = \mu. For example, at layer 22 for Llama-3.1-8B, the μ\mu-only performance (measured in terms of top-3 accuracy) in Figure 3 of the paper is ∼19\sim19%, but on replacement by μ+HABC\mu+ H_{ABC}, the performance jumps to ∼97\sim97%. The baseline without any intervention is ∼85\sim 85%. In the revised manuscript, we will update Section 4.2 to clearly explain the intuition and reasoning behind our decision to retain the interaction ensemble and the mean.

Q6., W3.

why is the improved perforamce finding "surprising"?

why is this finding "surprising"?

ANOVA decomposition is, crucially, not conditioned on the ground-truth correct answer or the next token predicted by a model.

One can, in theory, stop the inference at layer 22 of Llama-3.1-8B, collect all the activations of the ensemble and decompose each vector into the mean, first-order, second-order, and third-order terms. Findings of section 4.2 (replacement experiments) can be equivalently stated as: ablating first- and second-order terms at layer 22 and letting the model run uninterrupted after the intervention performs better, as compared to a model that was never intervened upon. This is the surprising part for us.

AuthorsOpenReview ↗

Rebuttal (cont.)

W1.

Limited mechanistic interpretability. The claim of revealing “the representation-level mechanism” is overstated. The paper shows that terms are organized, and some are necessary or sufficient at the last token, but not how H_ABC is computed from H_AB and C, which heads or MLPs are responsible.

We indeed conduct our causal experiments at the level of layers. To this end, that sentence in the abstract is changed from "Crucially, when taken together, these geometric and causal investigations reveal the representation-level mechanism that progressively organizes and composes the relational information to form the answer." to "Crucially, when taken together, these geometric and causal investigations trace how relational information is progressively organized and composed across layers to form the answer."

W2.

the “retaining improves accuracy” claim. The abstract’s claim that restriction “improves next-token prediction” rests on gains over a baseline of ~85% top-3 accuracy. There are no significance tests, no per-γ breakdown (the baseline fails mostly at large γ), and the “destructive interference” explanation in the Discussion section is untested.

We evaluate all combinations of (A,B,C)(A,B,C) in each replicate and score them according to the top-3 highest logits, so there is no sampling uncertainty in accuracy. To figure out the significance of our ablation and replacement experiments, we performed the same experiments using a random vector scaled to match the norm of the corresponding interaction term. We draw 100100 such vectors for each experiment corresponding to a single replicate. We have listed the Student-t 95%95\% CI for the difference between control and our observations for some specific experiments below. We have added the control experiments for a single replicate to Appendix J. We will add the corresponding results for all replicates once the experiment finishes.

InterventionExperimental ObservationRandom mean ±\pm SDExperiment − random [95% CI]
Ablate H_AB, layer 1545.7%81.2 ± 0.5%−35.5 [−35.6, −35.4]
Retain H_AB, layer 1581.5%28.7 ± 0.8%+52.8 [52.6, 53.0]
Ablate H_ABC, layer 2232.2%81.5 ± 0.4%−49.3 [−49.4, −49.3]
Retain H_ABC, layer 2295.8%20.4 ± 0.7%+75.4 [75.3, 75.5]

We have added per-γ\gamma breakdown for the increase in performance after the replacement experiment at layer 22 in Appendix I.2. We omit the full table for brevity, but highlight the largest improvements in the top-3 accuracy.

γ\gammaBaseline (%)μ+HABC\mu + H_{ABC} (%)Improvement
7756.1±6.456.1 \pm 6.489.3±4.089.3 \pm 4.0+33.2±4.8+33.2 \pm 4.8
9941.5±5.441.5 \pm 5.483.9±3.583.9 \pm 3.5+42.4±4.7+42.4 \pm 4.7

The "destructive interference" hypothesis is introduced in the Discussion section to speculate about the role of first- and second-order terms in the later layers. We observed that ablating them in Section 4.2 improved the performance. We hypothesize that this is possible due to superposition of different ensembles that play a different functional role in the residual stream (Elhage et al., 2021; 2022).

W4., W7.

the Discussion section is too thin to my taste, and there's no Conclusions section, as is expected in this scientific paper genre.

I expected to see an interpretation of the ablation and replacement results in the discussion section, beyond repeating the findings and suggesting a vague "different function" conjecture. It would make the paper much stronger.

Thanks a lot for this feedback. We agree that the functional interpretation of interaction terms, ablation and replacement experiments was understated. We have expanded the Discussion section to emphasize the aforementioned points. We have further improved the speculative "destructive interference" argument, as pointed out in W2.

W5.

missing citations: geometry of concepts papers only from 2025. Consider adding Mikolov et al. (2013), Pennington et al. (2014), Park, Choe & Veitch (2024), etc.

We thank the reviewer for pointing them out and we have added them to the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.