Hierarchical Urban Representation Pretraining from Large-Scale Geospatial Data

This paper studies how to encode cities with a Transformer for urban representation pretraining on large-scale geospatial data. Geospatial entities form the urban fabric of blocks, which in turn compose larger regions that collectively constitute cities. This entity–block–region–city hierarchy provides a natural basis for urban representation learning. Scaling it across thousands of cities requires modeling local functional interactions within blocks and spatial organization within and across regions. To this end, we propose HESTIA, a hierarchical framework for urban representation pretraining. Specifically, an Urban Fabric Tokenizer converts heterogeneous geospatial data into entity-category tokens arranged in a unified tensor reflecting the native city hierarchy. The Urban Interaction Transformer (UiT) applies attention in three nested stages, allowing urban semantics to emerge progressively through the hierarchy: first from functional interactions among entity categories within blocks, then from geometry-aware interactions among blocks within regions, and finally from city-scale contextualization across regions. Pretrained on a large-scale U.S. geospatial dataset of 8.18 million blocks and over 263 million entities, HESTIA achieves state-of-the-art performance on the multilevel benchmark, outperforming strong baselines including DeepMind's AlphaEarth, validating its hierarchical pretraining across spatial scales. Code is available at https://anonymous.4open.science/r/HESTIA-0D1C/.