- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Probes of large language models have demonstrated that the models encode geospatial properties with respect to entities such as countries and cities. However, it is unclear how much of the spatial information is directly encoded in the initial entity tokens compared to later model activations. To explore how the initial entity embeddings contribute to the overall spatial encodings, the authors train a linear probe that extracts an entity's coordinates on a globe, and then compare the extracted coordinates against the true coordinates using spherical distance. The authors also design the Spelling Share metric, which uses the error distance between the true coordinates and predicted coordinates to measure the contribution of the initial token embeddings to the final spatial representation.
Critical strengths
Significance - Studying the spatial awareness at varying levels of spatial granularity (i.e. village --> town --> city --> country) is useful to ensure that the conclusions hold for a wide range of spatial categorizations.
Critical weaknesses
- Clarity - The writing of this paper is quite confusing due to wording and phrasing issues and missing details. (1) Some of the writing is very difficult to interpret. For example, lines 42 - 44 are discussing how linear probes have been used before. The authors write that the probes have been fitted to "the colour a word names and the state of board." I do not understand what that means. Does the first part refer to using a probe to see what color a model maps a target word to? Also, how does this relate to the second part, which I assume is referring to the state of a boardgame? Also, line 105 states "They probed sentence embeddings beside the same encoder with random weights..." It is unclear what the authors are trying to say. Are the authors saying that two of the same model are probed and compared, with one model being trained and the other untrained? What does beside mean? I should note, these questions are rhetorical and meant to highlight examples where the writing is extremely unclear. There are more instances of this lack of clarity throughout the paper. (2) The methodology is unclear. Details behind design choices are missing and there is seemingly conflicting information. See Questions for Authors for specifics.
- Soundness / Significance - It is difficult for me to understand how the Spelling Share metric actually measures an entity name's contribution to the model's spatial understanding of that entity. For me, this calls into question whether the results are meaningful. See Questions for Authors for specifics.
Questions for authors
- Where are the probes being placed? Line 207 states that probes are placed at "the residual stream at the last token of the name at layer l, with layer 0 the embedding output." The phrasing is a confusing, but I am interpreting this mean that the probes are placed directly after the token embeddings are obtained in the model during a forward pass. This would be directly after passing tokens to a model and before the embeddings are passed to the first transformer layer. But then, that doesn't fully make sense because the line mentions the residual stream. Line 211 states that "the layer l* of each probe is one whose median ... is lowest," seeming to indicate that probes can be placed at any transformer layer. To me, it seems line 207 and line 211 are giving conflicting information.
- How is the Spelling Share measuring an entity name's contribution to the model's spatial encoding of the entity? It seems to be more of an error ratio since it is comparing the error distance of one encoding to a baseline error distance. Also, why does the ratio need to be converted to bits?