What's in a Place Name? Spelling as a Confounder in Coordinate Probes of Language Models

Advances in language models have enabled applications in geospatial fields, involving tasks ranging from geocoding to spatial question answering. These applications have raised the question of whether, and to what degree, language models encode spatial information about real-world entities. Language model probes are increasingly taken as evidence of learned representations, but the extent to which spatial information is available from the name alone has not been systematically measured. We show through a series of experiments that probes trained to predict coordinates from place names recover a large share of their signal from the name alone. This share rises with entity granularity, from under a tenth at metros to above two fifths at venues, in the same order across model families and does not diminish as model parameters scale, revealing a confounder in probe-based claims of learned spatial representations.