- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The main research question is: How do LLMs represent and compose relationships between multiple input tokens to solve tasks? The paper introduce a task: $ D = C+ \gamma$. A, B and C ranges from 1 to 12. The paper uses functional ANOVA to decompose the representation (last-token residual-stream activations) into individual-variable effects, pairwise interactions and a three-way interaction. They use Fourier analysis and energy fractions to characterize the representations. They also use causal analysis, apply ablation, replacement, and steering to determine whether these representations actually influence predictions.
They find that the formation progression is 2-stage along the layers. Pairwise relation form first, around when is established, and 3-way relation form afterwards, around the formation of the answer.
Some relations are causally relevant while others are not. is important in middle layers, is important in final layers. Ablating also affects the emergence of . On the other hand, other geometrically structured pairwise interactions like do exists but they have little causal influence.
Critical strengths
The paper gives us a mathematically well-defined representation decomposition with causal experiments. The experiments clearly traces where does the relationship between 2 or 3 variables emerges, how do they interact with each other, and whether they are important or not.
Critical weaknesses
- The paper only examined one task: the month prediction. This gives a close-form and symmetric representation for all factors. This is not the case in general during LLM inference. So, the title "Causal and Interpretable Structures in LLM Compositional Tasks" may have a broader claim than what the experiment actually suggests.
Questions for authors
- How will your analysis generalize, if each factor takes more than 1 token to represent? In specific, what if factors have an indefinite (and not same) amount of tokens
Rebuttal
Dear X4g4,
Thanks for your positive assessment of our work and for finding the 2-stage behavior interesting. We appreciate your questions and constructive feedback on the scope and generalizability of our analysis. Below you will find our response to your concerns.
Q1.
If each factor has an indefinite but same amount of tokens, the analysis generalizes since we perform an ANOVA decomposition at the last-token site.
However, if each factor has an indefinite but not the same amount of tokens, the position of the last-token site changes according to the number of tokens required to represent a factor. One could devise experiments with padding, such as, in the case of numeric tokens and left pad each numeric factor with zeros. This ensures that the last-token site remains at a fixed position.
For every other experiment, this would require architecture-aware changes to the experimental design. One example is RoPE, where one would need to fix the position IDs so that differing lengths do not induce an extraneous rotation at the last token site. We will include this discussion about the experimental design in the paper to guide future work. Thanks for the question!
W1.
In the main section, we discuss only the months prediction. In appendices, we also observe consistent findings across hours, weekdays, and musical notes prediction. We will additionally import the discussion about other cyclic domains from appendices to the main discussion (section 7) to address the generalizability concern.