- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies how shared patterns relate to grammar and brain responses. It finds that Concept (similarity of relations among aligned meanings across languages ) gives the most consistent cross-model language relations, but the selected units do not show stronger grammatical effects than random units. Number interventions ( using steering vectors to alter singular or plural information and affect subjectverb agreement) also transfer from English to Hebrew. Shared representations (the part predicted from other models’ representations) do not show clearly stronger correspondence with brain responses than model-specific representations.
Critical strengths
- Connecting intracranial brain recordings with LLM representations is interesting. Although the conclusions are limited, the released data and code could support further research in this direction.
- The paper goes beyond measuring representation similarity and tests whether the selected units have a causal role in grammar. Comparing their effects with matched random units is useful, since similar representations alone do not establish functional importance.
Critical weaknesses
-
Many results are negative or inconclusive, and the findings seem scattered. The paper does not clearly bring them together into a coherent account of what multilingual models share.
-
Interpreting Concept as meaning seems overstated. Both Concept and Subspace measure aspects of representational geometry, but use different features and similarity measures. It is therefore unclear whether their difference supports the proposed distinction between meaning relations and activation organization.
-
The grammatical interventions only test subject–verb number agreement. Different forms of number marking provide useful variation, but they still test the same grammatical phenomenon, so broader claims about grammar seem overstated.
Questions for authors
-
Line 162: both edits act at layer 14 . Why layer 14?
-
Why is Concept interpreted as meaning, while Subspace is interpreted as activation organization? Both compare representation geometry. Could the difference come from the features and metrics used?
-
Why should the selected units carry number agreement information? Could their weak effects simply mean that the selection method does not pick out units relevant to this task?
-
What is new about the English-to-Hebrew steering experiment compared with earlier work on cross-language grammatical interventions?
-
Do the claims about grammar only refer to subject–verb number agreement? If not, what supports the broader claims?
-
The brain correspondence is small and the trial-reassignment test gives p = .072. How strong a conclusion can be drawn here? How can we interprete the results?
-
What is the main finding that connects the three parts of the paper? I find the individual results hard to put together into one clear conclusion.