- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Interpretability works have found structured representations of semantic information (circular arrangement of moths, weekdays, Engels et al, 2025) and arithmetic operands (Kantamneni and Tegmark, 2025; Feucht et al, 2026). This paper instead studies the reminder, the representation orthogonal to the identified structure. With ablation experiments this paper finds that often the reminder is sufficient for the LM to perform the task. The authors draw the conclusion that LM representations are much richer than their visible geometry.
Critical strengths
The paper does follow a nice framework of separating structure vs deviation (everything but the structure). The paper pits different kind of interventions against each other, such as introducing conflict with rotation on structure vs amplifying the deviation, to understand what actually matters for LM computation.
Critical weaknesses
-
The paper is very poorly written. For example: terms such as "embedding campaign", "zero patch", ... are mentioned in passing without any explicit definition, and it is really up to the reader to figure out what they actually means. The paper also have a heavy LLM smell.
-
I think the core technical and conceptual contribution of this paper is weak and not novel, nor do they provide any actionable insights. Please see my questions for authors below for more details.
Questions for authors
First of all, this paper is very difficult to read. I am familiar with most of the works this paper is based on, but I still had a hard time understanding the contents. How the paper presented the ideas is very tedious and that needs significant improvement before it is worthy of publication. My current assessment is potentially due to my incomplete understanding of some of the technical/conceptual details and I am happy to revise my scores if the authors are able to give satisfactory response and update the paper to improve readability.
-
Most of your headline scores are on the
"embedding campaign". You never explicitly define what that means. Such terms are unconventional and I personally have never seen them in any previous interp papers, so it is up to the reader to work out what it means. How I understood it as (and correct me if I am wrong): you are editing the representation after the input embedding and checking the downstream effects. A few questions on this.- (1a) In Figure 2 you report that ablating the structure (deviation-only) leaves performance largely intact for months and letters. The works you cite (Engels, Feucht) find this structures in middle to later layers. And, the LM might have multiple opportunities to "re-build" the structure, as long as it knows that the input is
"January". After your edit, have you tried applying the on your edited month embedding to check if it still decodes to"January"? This approach is just the inverse of Logit Lens with input embeddings. If it does decode do the month (i.e"January") then the deviation-only scores are trivial and the insights you are drawing from them don't really mean anything. - (1b) Same question for structure-only. And, is the structure even formed that early in the computation so that these interventions are expected to give target effects? In how many cases you are just corrupting the semantic information (i.e: "January", checked with the reverse Logit Lens I suggested) and the LM just can't recover the structure as it doesn't even know what the input is?
- (1a) In Figure 2 you report that ablating the structure (deviation-only) leaves performance largely intact for months and letters. The works you cite (Engels, Feucht) find this structures in middle to later layers. And, the LM might have multiple opportunities to "re-build" the structure, as long as it knows that the input is
-
In your "activation campaign", you target one layer at a time. Again, the LMs are distributed systems and potentially get multiple opportunities to "re-build" the structure. Have you considered targeting a window of layers? Specifically targetting the window of layers Engels, Feucht papers found for the representation/computation?
-
Previous literature (see Gur Arich et al, 2026; Cui et al, 2026, Sen Sharma et al, 2025, among others) have also found mix of different mechanisms inside LM computation. When you ablated on structure, maybe the LM can still do the same computation based a redundent computation which got abosrbed in the deviation. Maybe you are also finding that in your rotation experiment on Figure 3a: as you are amplifying the deviation, you are just suppressing one structure with it alternate computation absorbed in the deviation. I am also asking this because of 2 lines in your conclusion.
- "The results carry a caution and a corrective for intervention-based interpretability": Sure! We knew already that the LM does redundant computations. Interpretability is more of an effort to catalogue all such mechanisms one at a time and understand how they work together. I don't see how your results are a caution or corrective for this. And, if it is, what do you suggest interp researchers should do differently?
- "The representations ... language models compute are richer than their visible geometry, and this paper supplies instruments for studying the remainder": Really? Maybe I missed something, but where is the "instrument" that you are referring to here? With your ablations/interventions, all you are really saying is that "the reminders matter, and is often enough", which is not a new insight.