acceptodds
Under review as a conference paper at ICLR 2027

Linguistic Cartography: Metric Spatial and Hierarchical Platial Representations in Language Models

Abstract

Language models can encode where a place lies on the Earth's surface, but geographic representation is not only about the coordinate-based metric space. A city is also nested in a hierarchical platial structure: state/province, country, subregion and continent. We study how these two geographic forms coexist in the neural representations of language models. We build GPH-30K, a primary source of geographic hierarchy with 30,000 globally distributed populated places with both latitude/longitude information and a complete administrative path. This benchmark dataset supports metric spatial experiments and hierarchical platial experiments. On an open-weight Llama/Qwen model panel, coordinate probes recover latitude/longitude while hierarchy probes recover platial path from the same residual stream. Layer-wise parent-recovery maps show that the place hierarchy is not a single flat attribute: country-to-subregion relations emerge earlier, admin1-to-country relations peak late, and place-to-admin1 remains the hardest edge. Final-layer embedding geometry also varies by model family and generation: Qwen-2.5 models are consistently more spatial metric-leaning, whereas Llama and Qwen3 are closer to balanced or platial hierarchy-leaning. Finally, on a conflict diagnostic set HGCD where coordinate distance and administrative-tree distance make opposite predictions, smaller Qwen/Llama checkpoints are mostly spatial, while large Llama and Qwen-2.5 checkpoints can become mixed or platial. These results support a coexistence view of linguistic cartography: language models contain both continuous metric spatial representations and discrete hierarchical platial representations, which emerge at different depths and dominate geometry under different contexts.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, spatial representations, interpretability, representational similarity analysis, geographic knowledge, cognitive maps

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_a4SDOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Linguistic Cartography: Metric Spatial and Hierarchical Platial Representations in Language Models investigates how platial and spatial information are represented in LLMs. The authors begin their analysis with simpler questions of whether these structures are represented by the model at all, and, upon confirmation of the hypothesis, evaluate where in the model's depth the information lies, how they are modeled, how they conflict and coexist with each other, and how model size impacts these representations.

Critical strengths

S1 - The authors construct two datasets for this analysis. GPH-30K in particular is carefully built and documented: sampling uses per-country quotas, square-root allocation and a country-share cap to avoid domination by a few large countries, duplicated place names are removed to reduce prompt ambiguity, and the planned release includes an audit script. HGCD is presented as a curated set of conflict locations, but the curation procedure is not specified (see Q1).

S2 - The study provides a comprehensible analysis of the representation of platial/spatial information in LLMs, considering the depth at which information appears, SAP and CHP differences, generalization, and the interaction of platial and spatial representations, both for coexistence and under conflicting scenarios. The large experimental suite adds great depth to the understanding of how geospatial information is represented in LLMs. A few additional results could further enrich the analysis and better justify the paper's conclusions (see Q2, Q3 and Q4). I don't necessarily expect all of these questions to be resolved, but I think they provide some valid additional analysis.

Critical weaknesses

W1 - Depth comparisons between spatial and platial information (Figure 5A) use R² and top-1 accuracy, respectively. These two metrics are substantially different in behavior, and the observed later emergence of platial representations might be a consequence of these metric differences: R² can rise more smoothly with small increases in information, whereas accuracy may require a decision threshold to be crossed for correct predictions to occur. Considering a different metric for platial evaluation would improve comparability here, e.g. normalized label entropy explained by a probabilistic classifier, (H_prior − cross-entropy)/H_prior, where H_prior is the entropy of the class distribution.

W2 - The dataset sampling makes a significant effort to control geographic bias (per-country quotas, square-root allocation, country-share cap), but the paper contains no study of geographic bias in the observed results. This matters for two reasons. First, the country and admin1 probes only consider classes with at least 20 and 10 locations, respectively (App. C.5). These locations remain in the dataset and are used in other experiments, but they are excluded from these two probes. This keeps only 168 of 226 country classes and 728 of 3,624 admin1 classes, which biases these evaluations toward better represented and more densely sampled areas, and partly reintroduces the imbalance the sampling procedure was designed to correct. Second, LLMs themselves are geographically biased, and this bias likely affects the results independently of the filtering. Investigating both would require a breakdown of probe accuracy by continent (or subregion), reporting (i) the number of places and classes retained per continent after the minimum-count filters, and (ii) per-region accuracy, ideally also for experiments that use all locations (e.g., coordinate probes and parent retrieval). This would show whether the reported accuracies hold uniformly or are driven by the regions that survive filtering or by the model's own regional biases.

Questions for authors

Q1 - How is HGCD curated? Simply selecting all city pairs that satisfy the 10 km and 100 km criteria would yield far more than 318 cities, so either the initial candidate set was small or a substantial curation step was applied. Please describe the candidate pool, the curation procedure, and the number of pairs of each type. Furthermore, what is the reasoning behind two cities more than 100 km apart in the same country/state constituting a conflict? By this criterion, the majority of same-country city pairs in any large country would constitute a conflict.

Q2 - What role does language play in the model's ability to correctly identify locations? For instance, 'Rio de Janeiro' is obviously a Portuguese name, which should make it easier for an LLM to identify the country it's in. The opposite behavior may also hold: San Diego is a Spanish name in an English-speaking country; does this potentially misleading linguistic cue negatively affect the model's understanding? Furthermore, is performance hindered for names that have been transliterated in the dataset (e.g., Chinese cities being represented with romanized names)?

Q3 - How are similarities between city names and country/admin1 names understood by the models? I expect any LLM to easily understand that Guatemala City is in Guatemala, for instance, while opposite cases, such as Kansas City being in Missouri rather than Kansas, could cause some confusion.

Q4 - Admin1 accuracy is low (Table 1), which the authors treat as partial recovery. Admin1 is harder both because of finer granularity and because admin1 units differ widely in size and meaning across countries. What do the errors look like? Do wrong admin1 predictions (from the Table 1 probe, or place→admin1 Global@1) fall within the correct country or subregion, and how far is the predicted admin1 from the true one? If errors are geographically coherent, low admin1 accuracy would reflect coarse-but-coherent knowledge rather than absence, which changes how the "partially recoverable" result should be read.

4 comments
AuthorsOpenReview ↗

Response to Reviewer a4SD

Part 1 of 2

Thanks very much for your expert review and all the concrete suggestions. We ran every requested analysis on all 15 model checkpoints (re-extracted; Table 1 reproduced within 0.001). New results are in the revised PDF (blue; new appendices).

Your points, as we understand them. W1: the Fig. 5A depth comparison mixes R² and top-1 accuracy. W2: the min-count filters and the LLMs' own geographic bias may skew accuracy. Q1: how HGCD was curated, and why far same-country pairs count as conflicts. Q2/Q3: the role of language in place names. Q4: what the admin1 errors look like.

Reply to W1 (metric mismatch in Fig. 5A). The late rise is not a metric artifact: rescored with your NIG = (H_prior − CE)/H_prior, it remains. The platial curve of Fig. 5A is the model's own country-label prediction (final norm and unembedding at each layer; legend: "country label prediction"), which Sec. 3.4 contrasts with spatial information from a coordinate probe.

  • Same panel, your metric (HGCD, the same four models and prompt): the label prediction is still below the class prior at half depth in Qwen-2.5-7B, Llama-3.1-8B and Llama-2-7B (−0.22 to −0.20) and peaks at 0.91–1.0 relative depth; Mistral-7B rises earlier (peak 0.75). NIG never places the rise more than 0.03 earlier than accuracy does.
  • At scale (29,504 GPH-30K places, four layer-wise models): NIG at half depth −0.46 to −0.02, peak at 0.86–1.0.
  • Same readout for both views (one prompt; logistic probe on PCA-128 scored with NIG; spatial targets are K = 168 sphere cells matching the 168 countries). First sampled depth (of nine) reaching 90% of the maximum:
modelcoord R² (ridge)cells K=168country (probe)country (own prediction)
Qwen-2.5-7B0.360.360.361.00
Qwen3-8B0.390.750.391.00
Llama-3.2-3B0.360.500.360.86
Llama-3.1-8B0.380.500.381.00

So spatial information is early and platial label prediction is late, as Fig. 5A shows. In addition, platial information itself is available as early as the matched spatial cells (country NIG is higher at every depth after the input layer, by 0.05–0.12), which strengthens coexistence. Fig. 5A is unchanged; the NIG analyses are in a new appendix figure and tables (Appendix "Information versus label prediction").

Reply to W2 (filters and geographic bias). The filters do not drive the results, and chance-corrected accuracy is positive in every region. Medians over the 15 models (kept = places (classes) after the country / admin1 filters):

continentplaceskept: country / admin1country acc / κadmin1 acccoord error (km)admin1→country Cond.@1
Africa4,7874,711 (49) / 2,247 (128)0.24 / 0.220.061,9050.57
Asia10,15310,102 (46) / 7,937 (244)0.70 / 0.670.221,3160.78
Europe8,0137,943 (40) / 5,943 (197)0.68 / 0.660.339680.78
N. America3,5523,412 (17) / 2,584 (78)0.65 / 0.500.242,0010.59
Oceania780646 (5) / 510 (9)0.45 / 0.100.236,7220.53
S. America2,7152,690 (11) / 2,133 (72)0.59 / 0.420.151,7750.42
  • Without the filters (226 countries, 3,624 admin1 units), country accuracy moves only from 0.599 to 0.589 (admin1 0.228 → 0.153, since most added classes have 1–9 examples).
  • In every region and model, κ > 0 and admin1→country retrieval beats chance. Accuracy tracks regional data density, the known geographic bias of LLMs (Manvi et al., 2024): highest in Asia and Europe, lowest in Africa; Oceania, where one country holds 51% of places, is hardest.
  • We acknowledge the sparse tail: the 496 places in countries with fewer than 20 places reach 0.139 in class-free retrieval (chance 0.102). Denser coverage of small states is a natural extension of GPH-30K. All tables: new Appendix "Geographic breakdown".
AuthorsOpenReview ↗

Response to Reviewer a4SD

Part 2 of 2

Reply to Q1 (HGCD curation). HGCD is organized by pair type rather than mined by distance thresholds, which is why it is small; its conflicts are relational, so a far same-country pair alone is not one. Our Sec. 2.1 wording ("<10 km", ">100 km") described typical cases (25 of the 95 twin-city pairs are farther than 10 km); the revision states the typology instead. There was no systematic candidate pool: the pairs are well-documented instances of each type (e.g., San Diego/Tijuana, Baarle-Hertog/Baarle-Nassau, Kansas City MO/KS) across 69 countries, so HGCD is a diagnostic rather than a sample. Coordinates are city centres (254 of the 260 locations matched to GeoNames lie within 14 km of the entry; median 0.4 km).

typepairsdistance (median, range)
international twin cities95 (93 cross-border)4.2 km (0 to 51)
cross-border enclaves / exclaves189.9 km (0.3 to 209)
enclave to own capital6528 km (27 to 1,092)
divided domestic cities233.0 km (0 to 73)
controls: close/same, far/same, far/different, close/different6 / 10 / 10 / 1015 / 2,990 / 12,019 / 429 km

A conflict arises at an anchor: for San Diego, the near cross-border city (Tijuana, 25 km) and a far compatriot (Los Angeles, 179 km) receive opposite predictions from the two views. Testing such contrasts at the anchor cities of 69 predefined cross-border pairs, all 15 models show a significant proximity effect (+0.010 to +0.053) and a significant membership effect (+0.016 to +0.063), membership the larger in all 15 (significantly in 11). A systematically mined conflict set is a natural extension. Details: new Appendix "HGCD composition and conflict-pair test".

Reply to Q2 (language in the name). Language helps, and the models go well beyond it.

  • A spelling-only character n-gram classifier recovers 43.7% of countries, against 0.589 for the LLM probe on the same places and folds.
  • Spelling points to the right country (13,104 places): 0.90; no confident cue (16,043): 0.35; misleading cue (853, your San Diego case): 0.26. The misleading cue hurts, as you expected, but does not decide.
  • Stripping diacritics from the same place (6,740 places): 0.61 → 0.52.
  • Romanized names (56 non-Latin-script countries) are not harder: 0.63 vs 0.56 for Latin-script countries (Asia: 0.69 vs 0.73).

This fits the "linguistic cartography" framing: the internal map is built partly from linguistic form (new Appendix "Name cues").

Reply to Q3 (names containing other place names). Misleading overlaps hurt and helpful ones do not help, so the probes are not string-matching. Such names are rare because duplicated names were removed.

  • Another admin1 of the same country (62; Kansas City→Kansas): admin1 accuracy 0.08 vs 0.23; country not lowered (0.69 vs 0.60).
  • Another country's name (37; Benin City): country 0.46 vs 0.60.
  • Own admin1 (1,120; Shanghai): admin1 at baseline (0.22). Own country (17; Guatemala City): 0.25 on the 12 in the probe.
  • On HGCD, removing name-sharing pairs or partialling out name similarity changes the aggregate Δr by ≤ 0.011 in all 17 models.

Reply to Q4 (admin1 errors). As you anticipated, the errors are coarse but geographically coherent.

  • 55% [39%, 62%] of wrong predictions fall in the correct country (0.7% by chance), 69% in the correct subregion.
  • Same-country errors lie a median 532 km from the true unit (750 km for a random unit of that country); all errors 1,017 km (7,681 km at random).
  • Top-5 accuracy is 0.51 (top-1 0.23); admin1 accuracy is 0.29 when the country is right, 0.08 when it is wrong.
  • place→admin1 Global@1 shows the same: 46% of errors in the correct country (1,133 vs 7,481 km).

Sec. 3.1 now reads "coarse but geographically coherent" instead of "partially recoverable" (details: Appendix "Geographic breakdown").

I thank the authors for their detailed responses and additional analyses. My main concerns have been adequately addressed, and the new analyses and experiments makes the paper stronger and more complete.

We sincerely thank you for your thoughtful review and constructive suggestions, and we are glad that the new analyses, now included in the revised paper, have addressed your concerns.

Reviewer_p5PAOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how metric geography and administrative hierarchies coexist in language model representations. Using GPH-30K, HGCD, and multiple model families, it combines probing, parent retrieval, and subspace erasure to examine recoverability, depth profiles, and geometric alignment.

Critical strengths

GPH-30K and the broad model evaluation provide useful evidence that coordinates and administrative membership are jointly recoverable.

Critical weaknesses

The HGCD analysis does not clearly isolate the predefined conflict pairs. Aggregate distance correlations may not directly measure the intended spatial-administrative disagreements.

Questions for authors

Which pairs enter the HGCD correlations, and do the conclusions hold when evaluated directly on the predefined conflict cases?

1 comment
AuthorsOpenReview ↗

Response to Reviewer p5PA

Thanks very much for your precise questions and comments.

Your point, as we understand it. W1/Q1: the HGCD correlations may not isolate the predefined conflict pairs; do the conclusions hold on those cases?

Reply to W1 & Q1. The HGCD correlations use all 50,403 location pairs, i.e., the geometry of the whole conflict set; on the predefined conflict cases themselves, the coexistence conclusion still holds in all 15 main-panel models. Sec. 3.4 now states which pairs enter Δr, and a new Appendix "HGCD composition and conflict-pair test" adds the pair-level tests.

  • Which pairs. All C(318, 2) = 50,403; the 178 predefined pairs are 0.35%. Removing name-sharing pairs (Nogales/Nogales) or partialling out name similarity changes no sign of the aggregate in any of the 17 models (|change| ≤ 0.011).
  • Controlled test at the anchor cities of 69 predefined cross-border pairs (117 anchors; GPH-30K comparison places near < 60 km or far 300–1,500 km, in the same or another country; places sharing a name token with the anchor dropped):
range over 15 models
proximity effect (near minus far, membership fixed)+0.010 to +0.053, CI excludes 0 in 15/15
membership effect (same minus other country, distance fixed)+0.016 to +0.063, CI excludes 0 in 15/15
near foreign minus far compatriot−0.030 to −0.006: compatriot closer in 15/15, CI excludes 0 in 11/15

With two other distance bands and a name-similarity control (45 model × band cells), membership stays significant in all 45 and proximity in 36 (all misses at near < 30 km, 71 anchors).

  • Within HGCD. Anchored triplets (is X closer to its partner N than to a compatriot F more than 100 km away?) are balanced: the spatial view wins 43%–56% after excluding the 17 of 117 name-sharing groups (16/17 intervals include 50%; 49%–60% with them). Δr on the 178 pairs alone is negative in all 17 models, partly because of 26 name-sharing pairs (17 identical); without them it is positive in 9 of 17.

So both views shape the representation at the border: hierarchical platial membership weighs more against fixed spatial distance bands, and neither view dominates within HGCD. The aggregate Δr mostly covers non-conflict pairs and answers a different question; with 318 places its per-model intervals are about ±0.10, and its robust reading is at the tier level (the small and mid-size median is negative in 92% of location resamples, and all three 70B-class checkpoints exceed it in 95%).

Reviewer_ukTCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates two distinct forms of geographic knowledge encoded inside large‑language‑model residual‑stream representations: metric‑spatial representations based on latitude/longitude physical distance, and hierarchical platial representations corresponding to administrative membership hierarchies (place → admin1 → country → subregion → continent). To enable controlled comparison, the authors built two new benchmarks: GPH‑30K (30 000 globally distributed populated locations with both coordinate values and full administrative paths) and HGCD (a conflict diagnostic dataset where physical proximity and administrative hierarchy yield opposing predictions). Using decoder‑agnostic interpretability tools including linear probing, child‑to‑parent vector retrieval, representational‑similarity analysis (RSA), and INLP selective subspace erasure across a broad panel of Llama‑ and Qwen‑family models of varying sizes, the authors demonstrate that both forms of geographic information are linearly recoverable from the same hidden‑state stream. Hierarchical edges emerge at different transformer depths; small models prioritize metric‑spatial geometry under conflict, while large‑model behaviour diverges by model family. The core conclusion is a coexistence view: continuous metric‑spatial and discrete hierarchical platial representations co‑exist within LLMs, emerge at different network depths, and each can dominate embedding geometry depending on model properties and context.

Critical strengths
  1. This work develops two custom geographic benchmarks enabling systematic large-scale evaluation of both metric-spatial and hierarchical-platial representations. Existing LLM geography research mostly focuses only on coordinate probing or uses small ad-hoc test cases. GPH‑30K unifies coordinates and full administrative hierarchy labels for 30 000 real-world locations, while HGCD isolates conflict scenarios where the two geographic systems contradict each other, resolving key evaluation bottlenecks and providing reusable testbeds for follow-up geospatial interpretability research.

  2. All core empirical evidence relies on decoder-agnostic hidden-state analysis instead of token generation outputs. Many prior geospatial LLM studies conflate internal neural representations with generation choices, tokenization and unembedding artefacts. This experimental design ensures results directly characterize information encoded inside model residual states rather than the verbal answers generated by the model.

  3. INLP selective subspace erasure provides partial causal evidence that platial hierarchy is not merely a downstream byproduct of metric coordinate representations. After erasing nearly all recoverable latitude and longitude signals from hidden states, the critical admin1→country relational retrieval performance remains largely intact, supporting the claim that these two types of geographic knowledge occupy partially independent subspaces.

Critical weaknesses
  1. Generalizable hierarchical platial relational knowledge can only be reliably demonstrated for the single admin1→country tree edge. Most other administrative hierarchy edges achieve strong in-sample cross-validation performance but fall to chance under strict parent-disjoint evaluation. This means most hierarchy recovery comes from memorized place-parent pairs rather than generalizable relational reasoning, limiting the scope of the paper’s claim for complete tree-like platial representations inside LLMs.

  2. The small magnitude of Δr raises concerns that the observed model-family differences may be statistical noise rather than genuine differences in learned representations. The plotted Δr values for tested models generally range only from −0.09 to +0.07. While bootstrap confidence intervals are provided, partial correlations of such small magnitude are highly susceptible to sampling noise, which casts doubt on the conclusion that Qwen‑2.5 models favour spatial geometry while Llama models favour platial geometry.

  3. The paper cannot quantify how much embedding variance comes from corpus statistics versus Transformer-specific learning. The PPMI‑SVD baseline only proves training text co-occurrence contains geographic signals. Without formal variance partitioning, it cannot rule out the possibility that much of the observed coexistence signal originates purely from corpus statistics rather than an internally learned geographic world model

Questions for authors
  1. Given that only the admin1‑>country edge shows robust parent‑disjoint generalization, how precisely do you bound your main claim about “hierarchical platial representations”? Do you claim LLMs learn a full tree‑structured relational hierarchy, or only partial relational knowledge limited to this single edge plus memorized in‑support patterns for other hierarchy links?

  2. The Δr partial‑correlation effect sizes are small for most models. Can you quantify the expected noise floor for Δr under your experimental setup? Would the core family‑level trend (Qwen‑2.5 spatial‑leaning vs Llama platial‑leaning) persist after noise adjustment?

  3. The INLP metric‑erasure gives only one‑sided dissociation: erasing metric signal preserves hierarchy, but erasing hierarchy‑related subspace cannot fully eliminate hierarchical‑platial performance. Is there empirical evidence to rule out that platial information is diffusely spread across many high‑rank directions with no compact low‑dimensional subspace? What implication does this one‑sided dissociation have for your co‑existence thesis?

  4. From your corpus mediation regression: what proportion of embedding‑distance variance can be uniquely attributed to transformer‑learned structure after partialling out the PPMI‑SVD corpus co‑occurrence distance signal?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ukTC

Part 1 of 2

Thanks for your four specific questions. We ran experiments to address each one on all 15 checkpoints of the main panel (re-extracted; Table 1 and every published Δ reproduced within 0.001).

Your points, as we understand them. W1/Q1: is the hierarchy claim bounded to admin1→country plus memorized pairs? W2/Q2: is Δ above its noise floor, and does the family trend survive? W3/Q4: how much geometry comes from corpus statistics versus the transformer? Q3: is the INLP dissociation one-sided because platial information is diffuse?

Reply to W2 & Q2 (noise floor). The noise floor is |Δ| ≈ 0.02, and the Qwen-2.5 contrast clears it in every model. Same 5,000 locations and 1M pairs as in the paper; location-level bootstrap, permutation null, split-half replication.

ModelΔ95% CI (location bootstrap)split halves same sign
Qwen-2.5-0.5B / 1.5B / 3B−0.051 / −0.042 / −0.048[−0.076, −0.027] / [−0.064, −0.018] / [−0.070, −0.023]100%
Qwen-2.5-7B / 14B−0.044 / −0.065[−0.064, −0.023] / [−0.088, −0.043]100%
Qwen-2.5-32B / 72B−0.090 / −0.066[−0.110, −0.066] / [−0.084, −0.042]100%
Qwen3-4B / 8B+0.004 / +0.004
−0.016,+0.028−0.016, +0.028

/

−0.019,+0.028−0.019, +0.028

| 20% / 30% | | Qwen3-14B / 32B | +0.031 / −0.030 |

+0.007,+0.051+0.007, +0.051

/ [−0.053, −0.010] | 100% | | Llama-3.2-3B / Llama-2-70B | +0.038 / +0.070 |

+0.014,+0.061+0.014, +0.061

/

+0.044,+0.091+0.044, +0.091

| 100% | | Llama-3.1-8B / 70B | +0.005 / +0.016 |

−0.018,+0.029−0.018, +0.029

/

−0.007,+0.038−0.007, +0.038

| 35% / 80% |

  • Permutation null: SD 0.010–0.013 and 95% |Δ| bound 0.021–0.027 in every model.
  • All seven Qwen-2.5 intervals exclude zero (100% split-half agreement), and in all 300 shared resamples every Qwen-2.5 model is more spatial than every Llama model.
  • Exact permutation test: p = 1/6435 that the seven Qwen-2.5 models are the seven most spatial of 15 by chance (1/330 against Llama).
  • Models inside the floor (Qwen3-4B/8B, which the paper calls near-balanced; Llama-3.1-8B/70B at +0.005/+0.016) carry no family claim, and Qwen3-32B confirms the paper's "Qwen3 is mixed". HGCD (318 places) gives wider intervals (about ±0.10), reported with tier-level comparisons in the appendix.

New Appendix "Δ noise floor"; the abstract now states that the family contrast is well above this floor.

Reply to Q3 (one-sided dissociation). With an exact eraser the dissociation is two-sided; the one-sidedness came from INLP's greedy removal. LEACE (Belrose et al., 2023), fit on half of the places and evaluated on the other half (15 models):

country acccoord R²within-country coord R²
no erasure (evaluation half)0.44 to 0.680.50 to 0.800.05 to 0.38
LEACE country (rank 224)0.09 to 0.18 (majority 0.08)−0.01 to 0.140.02 to 0.38 (median 94% kept)
LEACE coordinates (rank 2)0.43 to 0.67 (97.7 to 99.7% kept)−0.01 to 0.140.05 to 0.38
random rank 224unchangedunchangedunchanged
  • After erasing country, held-out accuracy is near the majority baseline, so little linearly decodable country information remains outside that subspace; a random subspace of the same rank removes nothing.
  • Raw coordinates fall with country because geography is nested (country identity alone predicts latitude/longitude with R² = 0.981). The within-country code survives (median 94% kept; 76–99% in every model except Qwen-2.5-0.5B, whose within-country code is weak).
  • Erasing coordinates leaves country identity intact.

The coordinates-country codes are linearly separable in both directions, which is direct evidence for spatial-platial coexistence (new Appendix "Two-sided erasure"; the abstract now states it).

AuthorsOpenReview ↗

Response to Reviewer ukTC

Part 2 of 2

Reply to W1 & Q1 (scope of the hierarchy claim). We claim partial relational knowledge, not a complete tree, and the paper states this. Contribution 3 reads "Hierarchy levels are not recovered uniformly"; Sec. 3.2 identifies admin1→country as the edge that transfers to unseen parents; Sec. 4 calls it "the strongest relational result". The claim has three tiers:

  1. every level is linearly decodable from the same place state (Table 1);
  2. every adjacent edge is recovered for held-out children (Table 2, Cond.@1);
  3. admin1→country also transfers to unseen parents, in every model (Table 2, Parent-out@1).

On memorization: in tier 2, every test child is unseen during training; only the parent label set is shared, as for any probe. Our phrase "parent-label memorization" invited the other reading and now reads "in-support recovery". The recovered structure is geographic: wrong admin1 predictions land in the correct country 55% of the time (0.7% by chance), and same-country errors lie 532 km from the true unit (750 km at random). For subregion and continent, holding out one of 24 or 6 parents removes a large share of the target space, so no linear map can produce it (Sec. 3.2). Extending transfer to further edges is future work.

Reply to W3 & Q4 (corpus versus transformer). A large share comes from the transformer, and it grows with model quality. We compare each model with its own static input-token embedding of the same name (same corpus, no transformer computation):

coord R²country acc
static input embedding (mean of name tokens)0.32 to 0.330.25 to 0.28
final-layer residual (same 15 models)0.51 to 0.810.44 to 0.67
  • Transformer computation supplies 37%–60% of the decodable coordinate variance and 41%–59% of the country accuracy.
  • In the distance regression you asked about (1,500 locations, 500K pairs, rank regression), PPMI's unique share is at most 0.001, and it leaves the unique spatial term unchanged (≤ 0.0001). Geography itself explains only R² = 0.001–0.047 of pairwise cosine distance, so the location-level probes are the stronger evidence.

We agree that a text model's knowledge ultimately originates in its corpus: our claim concerns what the residual stream represents and how the spatial-platial representation differs across transformer hidden layers, and tracing it to specific training data is future work (new Appendix "Corpus versus transformer contributions").

Thank you for the detailed response and additional experiments. The authors have addressed my main concerns by clarifying the scope of hierarchical representations, providing statistical results for the model-family differences, and adding analyses on representation separation and transformer contributions. I believe these results have strengthened the paper, and I will therefore raise my rating from 2 to 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to the Area Chair and all reviewers

We thank the reviewers for careful and constructive reviews. They value the benchmarks (GPH-30K: all three reviewers; HGCD: ukTC), the decoder-independent design and the partial causal evidence from erasure (ukTC), and the breadth of the experiments (a4SD, p5PA). We answered every testable concern with an executed analysis on all 15 checkpoints of the main panel (re-extracted; Table 1 and every published Δ reproduced within 0.001). The new analyses support the central claim: metric spatial and hierarchical platial structure coexist in the same residual stream, and their balance depends on model family and context.

New evidence

  1. The family contrast is far above noise (ukTC). Noise floor |Δ| ≈ 0.02; all seven Qwen-2.5 intervals exclude zero; exact permutation test p = 1/6435.
  2. The two codes dissociate in both directions (ukTC). LEACE erasure of country takes held-out accuracy to 0.09–0.18 (majority 0.08) while the within-country coordinate code survives (median 94% kept); erasing coordinates keeps 97.7–99.7% of country accuracy.
  3. The transformer builds the geography (ukTC). Each model's static input embedding of the name decodes coordinates at R² 0.32; the residual stream reaches 0.51–0.81. PPMI adds no unique variance (≤ 0.001).
  4. The late platial curve is not a metric artifact (a4SD). Under the reviewer's NIG, the model's own country prediction still rises late (Fig. 5A panel, three of four models; GPH-30K, four of four), and matched probes show platial information as early as spatial information.
  5. The filters do not drive the results (a4SD). Without them, country accuracy moves only from 0.599 to 0.589; chance-corrected accuracy is positive in every region; wrong admin1 predictions fall in the right country 55% of the time (0.7% by chance).
  6. The models go beyond spelling (a4SD). A spelling-only classifier recovers 43.7% of countries, against 0.59 for the LLM probe.
  7. Coexistence holds at the border (p5PA, a4SD). At the anchor cities of the predefined conflict pairs, proximity and membership effects are both significant in every model (15/15).

Clarifications

  • Hierarchy (ukTC). The paper claims every level is decodable and every edge recoverable for unseen children, with transfer to unseen parents for admin1→country, "the strongest relational result" (Sec. 4); it does not claim a complete relational tree.
  • Fig. 5A (a4SD). Its platial curve is the model's own country-label prediction (legend: "country label prediction"), contrasted with spatial information as Sec. 3.4 states. The figure is unchanged; a new appendix figure adds NIG curves. This is the only decoder readout; every representational claim rests on decoder-independent probes.
  • HGCD (p5PA). Δr describes the whole conflict set (50,403 pairs), as Sec. 3.4 now states; the appendix adds bootstrap intervals and the conflict-pair test.

Revisions. The main text stays within 10 pages: the abstract states the noise-floor and two-sided-erasure results, Sec. 2.1 gives the HGCD typology, and Secs. 3.1, 3.2 and 3.4 clarify wording. New appendices (with one new figure): Geographic breakdown; Name cues; Δ noise floor; Corpus versus transformer contributions; Two-sided erasure; Information versus label prediction; HGCD composition and conflict-pair test.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.