- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Existing research shows that different LM may provide similar representation structure (not the raw tensor which is uncomparable, but about the proximity structure). However, this paper claims that metrics like RSA only give a global measure, which may "obscure how agreement varies across geometric scales and which linguistic properties support it".
The paper introduces 2 modified metrics: NW-RSA and CF-RSA. NW-RSA assigns more weights to nearby representations, essentially calculating a localized measure. CF-RSA measure the turbulence effect when changing one side of the representation (when g(.) is applied).
Experiment 1 (5.1) shows that broad geometric agreement coexists with weaker agreement among nearby inputs. Global RSA is in general larger than NW-RSA. When the scale of comparison gets larger (less localized and more globalized), so does the NW-RSA.
Experiment 2 (5.2-5.3) shows that representation structure is aligned within most linguistic equivalence class (probably excluding Syntactic group). Such alignment is selectively sensitive to perturbations: only counterfactual changes to logical scope produce particularly large reductions in alignment.
Experiment 3 (5.4-5.5) shows that alignment is relatively robust to paraphrasing but weaker when sentence structure or lexical meaning is disrupted.
Overall, the paper shows that while different models may preserve the global representation structure, this kind of alignment depends on the geometric scale of comparison and the linguistic properties of the inputs.
Critical strengths
-
The research topic is interesting and thoughtful. The similarity of representation structure across models has similarity on a global scale, but this alone does not tell us what exactly do the models agree on. The paper inspects it from different layer of granularity.
-
The experimental framework for investigating what exactly the models have representational alignment is very systematic.
Critical weaknesses
- The main analysis use the penultimate (second-to-last) layer and mean-pool hidden states over all non-special, non-padding tokens. While appendix E have some additional results on the effect of depth, the exact reason to choose second-to-last layer as the main research target needs a further clarification.
Questions for authors
-
Regarding results 5.1, the finding is that broad geometric agreement coexists with weaker agreement among nearby inputs. Is it possible that the lower local RSA arise partly from the fact that within a small neighborhood, all sentences have very similar distances, and in this case, small variations in these distances could then produce substantial changes in their rankings? If this is the case, can we still say that the local representation structure across models are different, or it's merely a statistical confounder?
-
Figure 4 shows that some linguistic equivalence classes have RSA comparable to global RSA, but "Syntactic" class has a significantly lower RSA. Do you think this tells you something about the linguistic property itself? Do you think this has implications that language models are good at identifying certain linguistic property, while others (like syntactic) are not important factor of the model representation.