- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
As i understand it, the fundamental question being looked at in this paper is: when different neural networks lean to solve the same problem, how much of their internal weights is determined by the problem rather than architecture/initialisation etc. I think it has been reasonably well shown that attention contains common structure across LLMs, and this papers focus is on a simple aggregate statistic. What the authors introduce is a notion of functional identifiability via a jacobian to see if any two independent LLMs approximate the same function, then the jacobians of their next token logits could be similar. I do not know this field well, but this seems novel and, to be, interesting and clever.
Critical strengths
This paper is really excellently presented and clear. The per marginal attention is sensible, and i like how simple it is, and the definitions are clear. I found figure 2 and 3 really useful as a visualisation. The experimental setting is on an extensive number of open models of varying size. As i said above I like the jacobian idea, and the fact their simpler aggregate is an approximation of this is a very useful result.
Critical weaknesses
Honestly, It would be nit picking to for me to write anything here. I enjoyed reading this paper, and I could not find anything obviously wrong as a "critical weakness".
Questions for authors
I suppose the one thing is the correlation of 0.6 between the two is decent but to me still a little weak. However, this is all relative. So i guess this correlation established agreement in token ranks, but it does not show fully that marginal attention approximates the jacobian influence numerically, Given how much the authors have done I do not expect them to fix this, just wondering what are their thoughts on this. I dont know if its possible to determine whether the component of attention that agrees across models is specifically the component predicted by Jacobian influence.
2 comments
Thanks! happy to maintain my positive assessment! great work
We want to thank the reviewer for appreciating our work and strongly supporting its acceptance.
On whether the shared component of attention is the one predicted by the Jacobian
We also find this question interesting. Even though our main experiments do not isolate this component directly, we believe one result supports the “shared” interpretation strongly: the Jacobian-based influence of Mamba and RWKV models (models which have no attention mechanism) agrees with the attention consensus of the census about as well as that of a transformer (noted in Section 4.1). Our reading is that the shared structure we observe does not strictly depend on attention being the underlying mechanism, which is what our Jacobian account predicts.
We hope our replies clarify the points raised, and we are happy to discuss them further.