- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
I think (but I'm not sure) that the paper aims to measure whether the capabilities of an LLM (across different domains or languages) are tied to specific populations of neurons of their MLP blocks, in the sense of these neurons being necessary for only 1 domain at a time (meaning that they are necessary for that domain and not others).
The contribution/novelty (which is not stated clearly) seems to be to conduct such an analysis more thoroughly than prior work.
Critical strengths
(S1) The experiments cover multiple dimensions: 3 models from different families, 2 granularities of domains (subjects/languages).
(S2) The findings complement those of prior smaller-scale studies of the topic.
Critical weaknesses
(W1) The paper is generally poorly written and organized. The motivation for the research questions is not even clearly established. The first paragraph of the introduction launches into a full-on survey of the prior work without explaining why we even care about what-neuron-does-what. The term "parametric shells" appears out of nowhere and I'm still not sure what it means exactly (what's a "shell" here?).
(W1b) The language used in important technical descriptions is also (seemingly) unnecessarily complicated, to the point that I'm not sure I understand what the authors mean. For example: the "concentrated, causally necessary neuron populations whose removal selectively degrades a target domain while sparing others". Does this just mean "neurons necessary to only one domain*"?
(W2) The methodology seems to be the same as in prior work (see citations L80). Two pages (Section 3) describe the methodology, which is fine to make the paper self-contained, but it's not clear if any of it is novel. If notm it should be more clearly labeled as background/review material.
(W3) The paper claims to "assess when parametric shells exist and when they do not". I guess that the culmination of this quest is the final section titled "The Granularity Principle"? The story about "non-overlapping regions of the token distribution" and "pushing the gradients" is very hand-wavy. I'm not saying that the findings are not valid, but the observations made on pretrained/instruction-tuned models do not seem sufficient to really establish the conditions under which the observed organization of neurons can appear.
Minor comments:
-
The term "modalitiy" to refer to language vs. GSM-8k: I don't think this is a good qualified. A different "modality" usually refers to a different type of input like images in vision-and-language models. The contents of GSM-8k are full English sentences.
-
Wouldn't Section 4.7 be easier to follow with plots rather than listing numbers in the text?
-
The word "null" is used multiple times in a way I don't understand. E.g. "the subject level null", "without a null condition", etc. Do you mean a "null hypothesis"? Either way this is one example among many things that make the paper hard to follow.
-
The writing mostly grammatically fine on the surface, but the underlying structure and organization of ideas is really messy. I don't know if this is the reason, but in my experience this is what usually happens in LLM-generated text (rather than LLM-edited text).
Questions for authors
I don't have precise questions and I don't anticipate to raise my evaluation of the paper.
The paper, in both subject and form is at the level of a workshop paper at best.
This could work can be part of a future good paper, but much more work would be necessary. As a constructive comment, I would suggest the authors to first reflect on why do we even care about measuring the "parametric modularity" in LLMs? The answers to this question could guide the future work that could be done around the current experiments to expand them into a future heftier paper.