- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Linguistic Cartography: Metric Spatial and Hierarchical Platial Representations in Language Models investigates how platial and spatial information are represented in LLMs. The authors begin their analysis with simpler questions of whether these structures are represented by the model at all, and, upon confirmation of the hypothesis, evaluate where in the model's depth the information lies, how they are modeled, how they conflict and coexist with each other, and how model size impacts these representations.
Critical strengths
S1 - The authors construct two datasets for this analysis. GPH-30K in particular is carefully built and documented: sampling uses per-country quotas, square-root allocation and a country-share cap to avoid domination by a few large countries, duplicated place names are removed to reduce prompt ambiguity, and the planned release includes an audit script. HGCD is presented as a curated set of conflict locations, but the curation procedure is not specified (see Q1).
S2 - The study provides a comprehensible analysis of the representation of platial/spatial information in LLMs, considering the depth at which information appears, SAP and CHP differences, generalization, and the interaction of platial and spatial representations, both for coexistence and under conflicting scenarios. The large experimental suite adds great depth to the understanding of how geospatial information is represented in LLMs. A few additional results could further enrich the analysis and better justify the paper's conclusions (see Q2, Q3 and Q4). I don't necessarily expect all of these questions to be resolved, but I think they provide some valid additional analysis.
Critical weaknesses
W1 - Depth comparisons between spatial and platial information (Figure 5A) use R² and top-1 accuracy, respectively. These two metrics are substantially different in behavior, and the observed later emergence of platial representations might be a consequence of these metric differences: R² can rise more smoothly with small increases in information, whereas accuracy may require a decision threshold to be crossed for correct predictions to occur. Considering a different metric for platial evaluation would improve comparability here, e.g. normalized label entropy explained by a probabilistic classifier, (H_prior − cross-entropy)/H_prior, where H_prior is the entropy of the class distribution.
W2 - The dataset sampling makes a significant effort to control geographic bias (per-country quotas, square-root allocation, country-share cap), but the paper contains no study of geographic bias in the observed results. This matters for two reasons. First, the country and admin1 probes only consider classes with at least 20 and 10 locations, respectively (App. C.5). These locations remain in the dataset and are used in other experiments, but they are excluded from these two probes. This keeps only 168 of 226 country classes and 728 of 3,624 admin1 classes, which biases these evaluations toward better represented and more densely sampled areas, and partly reintroduces the imbalance the sampling procedure was designed to correct. Second, LLMs themselves are geographically biased, and this bias likely affects the results independently of the filtering. Investigating both would require a breakdown of probe accuracy by continent (or subregion), reporting (i) the number of places and classes retained per continent after the minimum-count filters, and (ii) per-region accuracy, ideally also for experiments that use all locations (e.g., coordinate probes and parent retrieval). This would show whether the reported accuracies hold uniformly or are driven by the regions that survive filtering or by the model's own regional biases.
Questions for authors
Q1 - How is HGCD curated? Simply selecting all city pairs that satisfy the 10 km and 100 km criteria would yield far more than 318 cities, so either the initial candidate set was small or a substantial curation step was applied. Please describe the candidate pool, the curation procedure, and the number of pairs of each type. Furthermore, what is the reasoning behind two cities more than 100 km apart in the same country/state constituting a conflict? By this criterion, the majority of same-country city pairs in any large country would constitute a conflict.
Q2 - What role does language play in the model's ability to correctly identify locations? For instance, 'Rio de Janeiro' is obviously a Portuguese name, which should make it easier for an LLM to identify the country it's in. The opposite behavior may also hold: San Diego is a Spanish name in an English-speaking country; does this potentially misleading linguistic cue negatively affect the model's understanding? Furthermore, is performance hindered for names that have been transliterated in the dataset (e.g., Chinese cities being represented with romanized names)?
Q3 - How are similarities between city names and country/admin1 names understood by the models? I expect any LLM to easily understand that Guatemala City is in Guatemala, for instance, while opposite cases, such as Kansas City being in Missouri rather than Kansas, could cause some confusion.
Q4 - Admin1 accuracy is low (Table 1), which the authors treat as partial recovery. Admin1 is harder both because of finer granularity and because admin1 units differ widely in size and meaning across countries. What do the errors look like? Do wrong admin1 predictions (from the Table 1 probe, or place→admin1 Global@1) fall within the correct country or subregion, and how far is the predicted admin1 from the true one? If errors are geographically coherent, low admin1 accuracy would reflect coarse-but-coherent knowledge rather than absence, which changes how the "partially recoverable" result should be read.
4 comments
Response to Reviewer a4SD
Part 2 of 2
Reply to Q1 (HGCD curation). HGCD is organized by pair type rather than mined by distance thresholds, which is why it is small; its conflicts are relational, so a far same-country pair alone is not one. Our Sec. 2.1 wording ("<10 km", ">100 km") described typical cases (25 of the 95 twin-city pairs are farther than 10 km); the revision states the typology instead. There was no systematic candidate pool: the pairs are well-documented instances of each type (e.g., San Diego/Tijuana, Baarle-Hertog/Baarle-Nassau, Kansas City MO/KS) across 69 countries, so HGCD is a diagnostic rather than a sample. Coordinates are city centres (254 of the 260 locations matched to GeoNames lie within 14 km of the entry; median 0.4 km).
| type | pairs | distance (median, range) |
|---|---|---|
| international twin cities | 95 (93 cross-border) | 4.2 km (0 to 51) |
| cross-border enclaves / exclaves | 18 | 9.9 km (0.3 to 209) |
| enclave to own capital | 6 | 528 km (27 to 1,092) |
| divided domestic cities | 23 | 3.0 km (0 to 73) |
| controls: close/same, far/same, far/different, close/different | 6 / 10 / 10 / 10 | 15 / 2,990 / 12,019 / 429 km |
A conflict arises at an anchor: for San Diego, the near cross-border city (Tijuana, 25 km) and a far compatriot (Los Angeles, 179 km) receive opposite predictions from the two views. Testing such contrasts at the anchor cities of 69 predefined cross-border pairs, all 15 models show a significant proximity effect (+0.010 to +0.053) and a significant membership effect (+0.016 to +0.063), membership the larger in all 15 (significantly in 11). A systematically mined conflict set is a natural extension. Details: new Appendix "HGCD composition and conflict-pair test".
Reply to Q2 (language in the name). Language helps, and the models go well beyond it.
- A spelling-only character n-gram classifier recovers 43.7% of countries, against 0.589 for the LLM probe on the same places and folds.
- Spelling points to the right country (13,104 places): 0.90; no confident cue (16,043): 0.35; misleading cue (853, your San Diego case): 0.26. The misleading cue hurts, as you expected, but does not decide.
- Stripping diacritics from the same place (6,740 places): 0.61 → 0.52.
- Romanized names (56 non-Latin-script countries) are not harder: 0.63 vs 0.56 for Latin-script countries (Asia: 0.69 vs 0.73).
This fits the "linguistic cartography" framing: the internal map is built partly from linguistic form (new Appendix "Name cues").
Reply to Q3 (names containing other place names). Misleading overlaps hurt and helpful ones do not help, so the probes are not string-matching. Such names are rare because duplicated names were removed.
- Another admin1 of the same country (62; Kansas City→Kansas): admin1 accuracy 0.08 vs 0.23; country not lowered (0.69 vs 0.60).
- Another country's name (37; Benin City): country 0.46 vs 0.60.
- Own admin1 (1,120; Shanghai): admin1 at baseline (0.22). Own country (17; Guatemala City): 0.25 on the 12 in the probe.
- On HGCD, removing name-sharing pairs or partialling out name similarity changes the aggregate Δr by ≤ 0.011 in all 17 models.
Reply to Q4 (admin1 errors). As you anticipated, the errors are coarse but geographically coherent.
- 55% [39%, 62%] of wrong predictions fall in the correct country (0.7% by chance), 69% in the correct subregion.
- Same-country errors lie a median 532 km from the true unit (750 km for a random unit of that country); all errors 1,017 km (7,681 km at random).
- Top-5 accuracy is 0.51 (top-1 0.23); admin1 accuracy is 0.29 when the country is right, 0.08 when it is wrong.
- place→admin1 Global@1 shows the same: 46% of errors in the correct country (1,133 vs 7,481 km).
Sec. 3.1 now reads "coarse but geographically coherent" instead of "partially recoverable" (details: Appendix "Geographic breakdown").
I thank the authors for their detailed responses and additional analyses. My main concerns have been adequately addressed, and the new analyses and experiments makes the paper stronger and more complete.
We sincerely thank you for your thoughtful review and constructive suggestions, and we are glad that the new analyses, now included in the revised paper, have addressed your concerns.
Response to Reviewer a4SD
Part 1 of 2
Thanks very much for your expert review and all the concrete suggestions. We ran every requested analysis on all 15 model checkpoints (re-extracted; Table 1 reproduced within 0.001). New results are in the revised PDF (blue; new appendices).
Your points, as we understand them. W1: the Fig. 5A depth comparison mixes R² and top-1 accuracy. W2: the min-count filters and the LLMs' own geographic bias may skew accuracy. Q1: how HGCD was curated, and why far same-country pairs count as conflicts. Q2/Q3: the role of language in place names. Q4: what the admin1 errors look like.
Reply to W1 (metric mismatch in Fig. 5A). The late rise is not a metric artifact: rescored with your NIG = (H_prior − CE)/H_prior, it remains. The platial curve of Fig. 5A is the model's own country-label prediction (final norm and unembedding at each layer; legend: "country label prediction"), which Sec. 3.4 contrasts with spatial information from a coordinate probe.
So spatial information is early and platial label prediction is late, as Fig. 5A shows. In addition, platial information itself is available as early as the matched spatial cells (country NIG is higher at every depth after the input layer, by 0.05–0.12), which strengthens coexistence. Fig. 5A is unchanged; the NIG analyses are in a new appendix figure and tables (Appendix "Information versus label prediction").
Reply to W2 (filters and geographic bias). The filters do not drive the results, and chance-corrected accuracy is positive in every region. Medians over the 15 models (kept = places (classes) after the country / admin1 filters):