- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This model uses analogical reasoning (concretely, hypernymy and friends) to try and establish whether generative models 'compute' (really, activate) with the same 'variable' (trajectory/activation).
Critical strengths
Thorough experiments, and grammatically correct
Critical weaknesses
I think this paper has some critical weaknesses, divided in two major categories:
- The writing is unscientific and subjective
- Technically, the paper is good but must be considerably better to meet the bar.
I do appreciate the effort the authors put into this specific problem, which--at least to me--is compelling and has a few interesting branches. However, as it stands, the authors need to put in more effort in this work so it meets the bar. Here's how I would improve this paper:
In #1, regarding writing: broadly, you need to clearly separate your methods, results, and discussion sections. There is a reason for that: (1) it is easier to understand the setup; what you found (objectively); and how you interpreted these (subjectively). This paper does not do that, and hence every single aspect of the work is confirmatory to the thesis, not supportive. For example:
- The section titles are leading.
- Do not start an abstract with a question.It is not just style (which is unacademic), it simply does not fulfill the definition of an abstract: a condensed version of your paper. Clip out numbers and keep it readable.
- You should put the interpretation of the results in the discussion section, not in the results headers.
- Proofread your paper. In the abstract: 'Invariance, not effect size, identifies the variables models share'. What does this mean? It does appear that this does not contribute to the abstract at all.
- Captions are descriptive, which is good, but do not tell me what is the takeaway from it. For example, the medians over models plot seems like a clump without any interpretability--what am I looking at? Why am I looking at it?
In #2, regarding technical work:
- Maths need to be way more formal. L157-L160: 'recipient and donor activations at a declared site'. What is this? More importantly, what are really? Vectors? Tensors? What is a unit direction? Is this a basis or a metric tensor or something else? Also, this does not seem to contribute to the bulk of the paper.
- It is unclear to me what are your experiments. You have the models (without citations) and the 'stimuli and consumers'. But the latter seem to be entirely unrelated to the maps? I think this section is good, but needs to be clearer: what are we using the maps for? etc.
- Mechanistic interpretability's common pitfall is that the results are always confirmatory (hence my insistence on the earlier section). This lacks alternate interpretations of the results, in addition to better tests. Spearman is not sufficient to establish the conclusion since it is a correlation measure. Here, better tests would mean contrastive experiments.
- Terminology is unsound (compute, variable, etc). Fixing this is of crucial importance in any MI work.
Fundamentally, I think that this paper is not objective enough to be a scientific contribution. The problem is interesting but it really seems that it needed considerably more time to be a paper at the level of ICLR.
Questions for authors
What were, in a nutshell, the conclusions drawn from your work? I see a discussion section, but the conclusion is not in the work. I'd like to see that clearly stated so it can be tied to the thesis and properly evaluated.