acceptodds
Under review as a conference paper at ICLR 2027

LoCA: Enabling Multi-Modality in Geospatial Location Encoders

Abstract

Location encoders map geographic coordinates to high-dimensional descriptors through contrastive alignment with paired Earth Observation imagery. We prove that this construction is structurally limited: any Lipschitz location encoder produces an output of intrinsic dimension at most 2, regardless of architecture or bandwidth, which forces the contrastive loss to strictly exceed its asymptotic optimum. Empirically, joint training also collapses the encoder's embedding covariance to extremely low rank. Although dimension-limited, current location encoders sidestep this spectral collapse by training against a frozen image encoder, which we reinterpret as a covariance regularizer that supplies a high-rank target for the location encoder's output to fold onto. However, extending this approach to multimodal inputs is fundamentally challenging, as location encoders should simultaneously map a single location to multiple, potentially incompatible, descriptors. We identify two requirements a regularizer must satisfy for location encoding in multimodal regimes: co-alignment across modalities, since adapters alone cannot align independently pretrained encoders; and high effective rank, since the location encoder inherits its target's spectrum. We propose LoCA (Location Contrastive Alignment), a two-stage pipeline that satisfies both. Stage 1 jointly trains Sentinel-1 and Sentinel-2 encoders with contrastive learning, producing a co-aligned, high-rank multimodal target. Stage 2 freezes the image encoders and trains a location and a tabular metadata encoder as projection adapters with multi-view contrastive learning. Against five state-of-the-art encoders on 17 tasks ranging from land cover and wildfire forecasting to geophysical, socio-economic, biodiversity and climate applications, LoCA achieves the most task wins, is the only model to outperform every other on a majority of tasks, and wins 68.8% of all per-task comparisons.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Earth Observation, location encoder, multi-modal, representation learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VKJuOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies contrastive location encoders, which map coordinates to embeddings by aligning them with a paired image modality (e.g., satellite or ground-level imagery) and asks how to align a single location encoder to several modalities at once. There are several arguments. The authos first show that location encoders are low-dimensional, find that frozen pretrained image encoders help, and identify co-alignment and high effective rank as prerequisites for multimodality. Based on this, LoCA is proposed and shows nice performance across downstream tasks.

Critical strengths
  • The method is quite clear and simple. Contrastively pretrain S1-S2 encoders then reuse that pretrained image encoder and contrastively pretrain again with location and metadata. There is some resemblance with the LiT (Zhai et al., CVPR 2022) and Rethinking JEPA (Li et al., ICLR 2026).

  • The use case on 17 tasks is broad, which shows the wide applicability of the method. This is nice considering the simplicity of the method.

Critical weaknesses
  • Treating location encoders as adapters over a frozen target resembles LiT (Zhai et al., CVPR 2022) and Rethinking JEPA (Li et al., ICLR 2026), which show the effectiveness of learning against a fixed pretrained target. The frozen-target reframing is therefore not itself new, and the paper does not clearly explain what makes this specifically challenging (or non-trivial) in EO for contribution claims.

  • The rank requirement used to rule out masked-modelling targets seems weakly supported (I checked the references, but I do not see a concrete reference on why use CL over MAE in EO). There is a need for elaboration on this as it's not clear why not use multimodal MAE as stage 1 instead of CL since many EO foundation models for remote sensing are masked image modeling based. Both experiments and explanations are needed.

  • The theorem is interesting, but I do not think it fully explains the empirical results. The theorem shows a strict gap from the contrastive optimum, but does not show that this gap is large enough to explain the observed performance degradation. More importantly, it is about intrinsic dimension, whereas the empirical failure mode is low effective rank, which is a different property (and the paper does admit). Since the same intrinsic-dimension limit also applies to LoCA, the paper needs a clearer explanation of how the theorem actually motivates LoCA’s gains.

Questions for authors

q1 The paper argues that freezing the image encoders is essential. Did the authors try a Stage 2 variant where the Stage 1 encoders are finetuned together with the location encoder, as in LiT's locked versus unlocked comparison? Downstream results for the from-scratch SatCLIP variant in Table 1 would also help.

q2 Why is contrastive learning necessary for Stage 1, rather than a multimodal masked autoencoder? The claim that masked modelling produces low-rank embeddings rests on citations that do not appear to show this directly.

q3 What explains the negative R-squared values in the ablation without Stage 1? This is quite counterintuitive considering the tasks are location-based. The encoder's input already varies smoothly with location, targets such as air temperature are strongly tied to latitude, and the splits do not appear to be spatially blocked, which favors interpolation. Needs clarifications.

q4 Why not simply fusing representations instead of forcing them into a single contrastively aligned space? It is not clear why co-alignment via CL should be preferable to preserving separate S1, S2, and metadata representations and combining them through simple fusion techniques like ensembles, etc. A fusion baseline would help establish whether the shared contrastive space is necessary or not.

Minor

  • Check citations in Section 5.2 and in the TESSERA reference in Appendix F.1.
10 comments

W1 & Q1: Relation to literature, and the role of the frozen target

Indeed training against a frozen pretrained encoder has been an established technique for more than a decade. Providing a frozen encoder is therefore not a contribution of our paper. Our contribution is to formally show that location encoders do not work without a frozen target, which diverges from common knowledge of the community (also see response @rMgG on W2: How literature justified freezing the image encoder), which in turn rules out the natural multimodal extension of joint training of multiple encoders since frozen encoders must be used as evidenced from our new ablation (Table 17 NEW). In the domain of Earth Observation this limitation is even more pronounced due to the multimodal nature of the data. However, these frozen encoders are not compatible with each other, as shown in our new ablation in Table 15 NEW, where concatenating the embeddings of pretrained Sentinel-1 and Sentinel-2 encoders fails on most tasks (e.g., 8.7 against 80.3 for LoCA without metadata on Ecoregions) and loses to the contrastive Stage 2 on 12 of the 15 tasks, because the two encoders are not trained to be compatible (see response of Q4). This makes the co-alignment requirement necessary alongside the rank requirement, and this chain of arguments provides our methodological contribution, which is distinct from prior frozen-target methods such as LiT [1] and Rethinking JEPA [5]. In these methods, the frozen target is not a necessary condition, since training from scratch (LiT) or with an EMA teacher (Rethinking JEPA) also works, and freezing only improves performance and efficiency. A single target source is assumed, and the alignment of several targets does not arise.

Regarding Q1. Fine-tuning the Stage 1 encoders together with the location encoder in Stage 2 corresponds to training the image and location encoders jointly, which is the setting of Theorem 1. In this setting, the contrastive loss is kept strictly above its optimum, and the image embeddings are pulled towards the 2-dimensional surface of the location encoder. This is observed in Table 1, where the same SatCLIP [6] architecture is trained in both ways. The effective rank is 89 when the image encoder is pretrained and frozen, and it collapses to 8 when the image encoder is trained jointly from scratch (SatCLIP†). Since the effective rank is a standard proxy for linear-probe performance [7], a very poor downstream performance is expected for an embedding that uses only 8 of its 256 dimensions. In a similar direction, the authors of GeoCLIP [18] study the impact of the frozen backbone and report that finetuning the CLIP backbone results in worse performance than freezing (Suppl. Section 6 "Motivations for using Pretrained CLIP as Image encoder Backbone").

The downstream results of SatCLIP with an image encoder that is not frozen are now reported in Table 17 NEW. Without a frozen image encoder, the location encoder collapses, and the frozen SatCLIP is better on all 16 tasks. For example, with the MLP probe, the F1 on Countries is reduced from 95.5 to 2.4, the micro-F1 on reBEN from 57.3 to 1.6 and the R² on Air temperature from 93.9 to 0.2. The ranking accuracy on the five climate variables lies between 49.9% and 50.6%, i.e., at chance level.

==================>

W2 & Q2: Rank requirement and contrastive learning versus masked modelling in Stage 1

We thank the reviewer for this comment, which has helped us increase the clarity of our work. Both a direct measurement and a new experiment are now provided, together with more direct references.

  • New experiment with multimodal masked-modelling targets. Stage 1 is replaced by Copernicus-FM [3] or DOFA [4], two state-of-the-art EO foundation models that process both Sentinel-1 and Sentinel-2 and are trained with masked image modelling. With either of them as the Stage 2 target, LoCA performs better on 13 of the 15 tasks (Table 12 NEW, where the RGB model DINOv3 [2] is also tested; see also our response to Reviewer ZkCA, W3). An effective rank to available dimensionality ratio of 20.6% is measured for Copernicus-FM, against 62% for LoCA (Table 1).

  • Direct measurement of the rank. We compare two ViT-S/16 models pretrained on the same data (SSL4EO-S12 [8]) isolating the pretraining objective as the sole contributing factor. The effective rank of the CLS embedding is 93 of 384 dimensions (24.2%) with MAE [9], and 188 of 384 (49.0%) with MoCo-v3 [10] (Section 4.2 NEW). Under masked modelling, half as many directions are used.

  • References that show this directly. In Zhang et al. [11] (Section 3.3 and Figure 1b), it is shown that the features of MAE suffer from dimensional collapse, since they lie in a low-dimensional subspace with fewer large singular values, and that their effective rank decreases during training. In Earth observation, GAIR [12] reports that replacing its contrastive objective with an MAE objective reduces the burn-scar mIoU from 83.26% to 74.15%, and both GAIR-MAE and SatMAE [13] are among the weakest models in its benchmark.

We note that LoCA is proposed as a framework defined by two requirements on the target, namely co-alignment across modalities and high effective rank. Contrastive learning in Stage 1 is a controllable way to satisfy both. We do not claim that this is the only pretraining scheme that can achieve this. If a pretrained model that satisfies the two requirements becomes available, it can be introduced in LoCA in place of Stage 1, in the same way that a different location encoder can replace the SIREN [14]. So far, however, multimodal foundation models do not reliably make use of additional modalities. In GEO-Bench-2 [15] (Appendix 6.3.6), adding SAR to optical inputs reduces the test metrics of all models on PASTIS and of most models on BigEarthNet v2, with TerraMind [16] as the main exception.

To make the rank requirement better supported in the main text, the direct measurement and the reference to Zhang et al. [11] are now included in Section 4.2 NEW.

W3: How Theorem 1 motivates LoCA

We thank the reviewer for this comment.

Size of the gap. Theorem 1 is an impossibility result, and no claim on the magnitude of the gap is made. It is shown that the contrastive optimum cannot be reached by any aligned pair with a Lipschitz location encoder, and by Corollary 1 this holds uniformly over the whole SatCLIP family (Appendix B.5). The downstream degradation is not attributed to this gap. It is attributed to a separate empirical finding, which is the collapse of the embedding to an effective rank of 8 under joint training (Table 1), as stated in the abstract, in C1 and in Section 3.3. This collapse is measured in Table 1, and its effect on downstream performance is now measured for SatCLIP with an image encoder that is not frozen (Table 17 NEW, see Q1). Theorem 1 is used instead to explain why location encoders are trained against a frozen target in practice, which is consistent with the design of existing location encoders such as SatCLIP and GeoCLIP.

Intrinsic dimension and effective rank. Only the intrinsic dimension is bounded by the theorem, while the effective rank is inherited from the target (Section 3.3). The intrinsic dimension is 2 in either case, but the spectrum of the location embedding is changed by the choice of target. Without a fixed target, as in joint training, a low effective rank is observed. With a frozen high-rank target, the 2-dimensional surface is forced to oscillate, and a high effective rank is obtained (Table 1 and Figure 4 NEW).

Gains of LoCA under the same limit. The intrinsic dimension of LoCA is also bounded by 2, and its gains are obtained from the higher-rank, co-aligned target of Stage 1. In support of this, the fraction of dimensions used by LoCA (62%), SatCLIP (35%) and Copernicus-FM (21%) follows the same order as their pairwise win rates (68.1%, 57.4% and 33.0%, Tables 1 and 2 NEW). When the Stage 1 target is replaced by the masked-modelling targets of Copernicus-FM (effective rank 21%) or DOFA, the resulting model is outperformed by LoCA on 13 of the 15 tasks (Tables 1 and 12 NEW). When it is replaced by independently pretrained encoders, the performance collapses (Table 3, and Tables 14 and 15 NEW).

Q3: Negative R² without Stage 1

In Table 13 OLD (Table 14 NEW), all values are reported as metric × 100, so the R² of the variant without Stage 1 lies between −0.079 and −0.007. These values are close to zero, which is the R² of a trivial predictor that always outputs the mean of the targets.

Since targets such as air temperature are strongly related to latitude and longitude, an R² close to zero indicates that probably the location has not been modeled by the embedding. This is expected, since without Stage 1 the location encoder is trained to match two independently pretrained targets that do not agree with each other (cos = 0.50, lines 430–440 OLD). For each location, two different embeddings must therefore be fitted at the same time, which cannot be achieved by a single function of the coordinates (Section 4.1).

This holds for both types of split. The regression tasks follow the random splits of the SatCLIP benchmark, which favor interpolation, and an R² close to zero is still obtained. The climate-ranking tasks use spatially blocked splits, in which complete geographic clusters are held out (Appendix E.2, lines 1632–1639 OLD). These tasks are evaluated with pairwise ranking accuracy, for which 50% corresponds to a random ordering, and the accuracy of this variant lies between 49.6% and 50.6%.

Q4: Fusion versus a shared contrastive space

We thank the reviewer for this suggestion. The requested fusion baseline has been run (Table 15 NEW), in which the location encoder is aligned to the concatenated embeddings of independently pretrained Sentinel-1 and Sentinel-2 encoders. Compared with LoCA, both without metadata and evaluated with the same MLP probe, the location encoder collapses on most tasks, with an R² below 0.01 on Median income, Population, Air temperature and Elevation, and an F1 of 8.7 against 80.3 on Ecoregions. LoCA is best on 12 of the 15 tasks.

This is expected, since a location encoder can only represent information that is a function of the location. With concatenation, the variation that is specific to each sensor, such as acquisition conditions or noise, is kept in the target, although it cannot be predicted from the coordinates. With contrastive co-alignment, the information that is shared by the two sensors at the same location is retained, and the sensor-specific variation is suppressed [17]. The shared contrastive space of Stage 1 is therefore necessary.

Minor: Citations

We thank the reviewer for spotting these. The missing citation in Section 5.2 (Garrido et al. [7]) and the TESSERA citation in the extended related work (Appendix F.1 OLD, Appendix G.1 NEW) have been fixed.

References

[1] Zhai et al. "LiT: Zero-Shot Transfer with Locked-image Text Tuning", CVPR 2022.

[2] Siméoni et al. "DINOv3", arXiv 2025.

[3] Wang et al. "Towards a Unified Copernicus Foundation Model for Earth Vision", ICCV 2025.

[4] Xiong et al. "Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation", arXiv 2024.

[5] Li et al. "Rethinking JEPA: Compute-Efficient Video SSL with Frozen Teachers", ICLR 2026.

[6] Klemmer et al. "SatCLIP: Global, General-Purpose Location Embeddings with Satellite Imagery", AAAI 2025.

[7] Garrido et al. "RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank", ICML 2023.

[8] Wang et al. "SSL4EO-S12: A Large-Scale Multimodal, Multitemporal Dataset for Self-Supervised Learning in Earth Observation", IEEE GRSM 2023.

[9] He et al. "Masked Autoencoders Are Scalable Vision Learners", CVPR 2022.

[10] Chen et al. "An Empirical Study of Training Self-Supervised Vision Transformers", ICCV 2021.

[11] Zhang et al. "How Mask Matters: Towards Theoretical Understandings of Masked Autoencoders", NeurIPS 2022.

[12] Liu et al. "GAIR: Location-Aware Self-Supervised Contrastive Pre-Training with Geo-Aligned Implicit Representations", ISPRS JPRS 2026.

[13] Cong et al. "SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery", NeurIPS 2022.

[14] Sitzmann et al. "Implicit Neural Representations with Periodic Activation Functions", NeurIPS 2020.

[15] Simumba et al. "GEO-Bench-2: From Performance to Capability, Rethinking Evaluation in Geospatial AI", TMLR 2026.

[16] Jakubik et al. "TerraMind: Large-Scale Generative Multimodality for Earth Observation", ICCV 2025.

[17] Tian et al. "Contrastive Multiview Coding", ECCV 2020.

[18] Vivanco Cepeda et al. "GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization", NeurIPS 2023.

Reviewer_ZkCAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LoCA, a multimodal framework for learning geographic representations directly from coordinates. The authors first establish a theoretical limitation of contrastive location encoders by associating joint image–location training with low-rank representations. To address this, LoCA uses a two-stage approach: first, jointly aligning Sentinel-1 SAR and Sentinel-2 optical representations using DHEL, and subsequently freezing the image encoders while training location and metadata encoders using multi-view contrastive learning. Across 17 downstream tasks, LoCA achieves the best reported performance on eight tasks.

Critical strengths
  1. The paper provides a useful distinction between intrinsic dimensionality and covariance rank and how contrastively trained location encoders fail to capture dense information.
  2. Broad set of experiments are conducted in various settings including ecological, environmental and land cover modeling.
Critical weaknesses
  1. Limited baseline comparisons. I think the paper should include other fairer baselines which align location with multimodal observations as done in the paper like UNIGeoCLIP [1] and GAIR [2]. The paper also misses to compare against latest location encoders like Climplicit [3], Range [4] and TTE [5].
  2. It is not clear from the paper how theorem 1 relates to downstream task performance. How is the InfoNCE objective related to intrinsic dimensions and downstream task performance?
  3. The ablations should also compare what happens if other general purpose vision encoders are used such as DinoV3. Does the proposed framework hold strong with general purpose vision encoders? Can the first stage training be entirely removed with pretrained encoders?
  4. How does the model do in spatial generalization settings? Can the model generalize well when trained in one spatial region and then evaluated on a different spatial region?
  5. There are no visualizations provided in the paper that visualize location embeddings. How can we confirm they have learned meaningful representations with high intrinsic dimensionality.
  6. Why need separate encoders for Sentinel-1 and Sentinel-2 imagery? Why not just concatenate channel-wise and passed to a single encoder? This way the encoder can be trained along with the alignment objective. Table-4 has an experiment on BigEarthNet, but an ablation with the LoCA framework itself is missing.

References

[1] Astruc, Guillaume, et al. "UNIGEOCLIP: Unified Geospatial Contrastive Learning." arXiv preprint arXiv:2604.11668 (2026).

[2] Liu, Zeping, et al. "GAIR: Improving multimodal geo-foundation model with geo-aligned implicit representations." arXiv preprint arXiv:2503.16683 2 (2025).

[3] Dollinger, Johannes, et al. "Climplicit: Climatic implicit embeddings for global ecological tasks." arXiv preprint arXiv:2504.05089 (2025).

[4] Dhakal, Aayush, et al. "Range: Retrieval augmented neural fields for multi-resolution geo-embeddings." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

[5] Cher, Daniel, et al. "Tessellating the earth." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

Questions for authors
  1. How much of LoCA's improvement comes specifically from DHEL/MV-DHEL rather than multimodal training itself?
  2. Several metadata variables are using during training. How much gain is each multimodal input providing. It is unclear the gains coming from the input versus the proposed training objectives.
6 comments

W1: Additional baselines

LoCA is compared on 17 tasks against five state-of-the-art models [1–5], which are representative in the EO literature since they have been compared against a wide variety of more than 20 other methods, including CSP, GPS2Vec, MOSAIKS, Prithvi, Clay, SoftCon, CROMA, DOFA, PIGEON and Img2Loc in their respective papers.

We agree, however, that a comparison with the most recent location encoders strengthens the paper. The suggested methods are addressed as follows:

  • TTE [6]: TTE is now included in the benchmark, and the results over 10 seeds are reported in Table 2 NEW (mean ± std in Table 10 NEW). LoCA beats TTE in 11/17 tasks (Table 11 NEW).

  • GAIR [7]: GAIR is now included in the benchmark, and the results are reported in Table 18 NEW (one seed). LoCA beats GAIR in 14/16 tasks.

  • UniGeoCLIP [8]: it is trained only on metropolitan areas of the continental United States, and its pretrained weights and training data are not publicly available, so it cannot be evaluated on our global benchmark.

  • RANGE [9]: its embeddings are computed by retrieving from a database of satellite image features, so it does not fit the setting of coordinate-only location encoders considered here.

  • Climplicit [10]: it is trained by regressing climate variables from coordinates and month which is a completely different setup and use case.

The contribution of TTE is also complementary to LoCA, as the two methods address different problems of location encoders. TTE replaces the fixed spherical-harmonic basis of [11], which allocates representational capacity uniformly, with learnable spherical Voronoi partitions, which allocate capacity where it is needed in a data-driven fashion. However, our analysis is not affected by the exact architecture of the location encoder, as long as the assumptions of Theorem 1 are satisfied. In that case, the intrinsic dimension of its embeddings is bounded (Lemma 1 NEW), the encoder is fitted as an adapter over a frozen target (Section 3.3), and a co-aligned, high-rank target is required for its extension to multiple modalities (Section 4). In other words, TTE's location encoder improves how the location encoder fits the target, while LoCA defines what that target should be. The TTE location encoder architecture could therefore be used as the location encoder within LoCA seamlessly blending the contributions of the two works. We leave that as future work.

W2: Relation to downstream performance

Theorem 1 and downstream performance. Theorem 1 is a statement about the contrastive objective, and it is not intended to predict downstream accuracy. It is shown that, when the location and image encoders are trained jointly, the contrastive loss is kept strictly above its minimum. The link to downstream performance is provided by Wang & Isola [12], where the optimum of the contrastive loss is characterized by the alignment of positive pairs and the uniformity of the representations on the hypersphere, and where both properties are shown to be related to downstream performance. The uniform distribution on Sd-1 also has an isotropic covariance, so the contrastive optimum is the configuration in which all d dimensions are used equally, i.e., the one with maximal effective rank. Effective rank is in turn a standard proxy for linear-probe quality [13]. A loss that is kept away from its optimum therefore leaves the embedding without this guarantee, and under joint training the effective rank is indeed collapsed to 8 (Table 1).

Intrinsic dimension and downstream performance. The contrastive optimum has intrinsic dimension d−1, while the embeddings of a Lipschitz location encoder have intrinsic dimension at most 2 (Lemma 1 NEW). This is the reason the optimum cannot be reached (Theorem 1). The intrinsic dimension alone, however, does not determine downstream performance. As discussed in Section 3.3 and Figure 3 NEW (Figure 2 OLD), a 2-dimensional embedding can still fill a large part of the covariance spectrum when it is fitted against a frozen, high-rank target. The effective rank is then inherited from the target, and 158 of 255 dimensions are used by LoCA (Table 1). This is reflected in the downstream results, since the fraction of dimensions used by LoCA (62%), SatCLIP (35%) and Copernicus-FM (21%) follows the same order as their pairwise win rates (68.1%, 57.4% and 33.0%, Tables 1 and 2 NEW).

In summary, Theorem 1 explains why location encoders cannot be trained jointly, and the effective rank of the target determines how well a location encoder trained against a frozen target performs downstream.

==================>

W3: General-purpose vision encoders

We thank the reviewer for this recommendation. The suggested experiment has been run, in which Stage 1 is removed and only Stage 2 is trained against a frozen pretrained encoder. Three anchors are considered: DINOv3 [14], a general-purpose vision model trained on natural RGB images, and Copernicus-FM [5] and DOFA [15], two multimodal Earth observation foundation models pretrained on Sentinel-1 and Sentinel-2 among other sensors. The results are reported in Table 12 NEW.

Stage 1 cannot be removed. LoCA is best on 12 of the 15 tasks, and each of the three anchors is outperformed by LoCA on 13 of them. The largest drops are observed on the Biodiversity & Ecology and Geophysical & Socio-economic tasks, for example from 80.2 to at most 49.2 F1 on Ecoregions and from 90.3 to at most 22.9 R² on air temperature. The general-purpose DINOv3 anchor is outperformed on the same number of tasks as the two Earth observation anchors.

This is in line with the two requirements of Section 4. The targets of all three anchors are learned through masked image modeling or from natural RGB images, which are the two limiting cases discussed in Section 4.2, and the Sentinel-1 and Sentinel-2 embeddings of the same location are not explicitly aligned. In Stage 1, co-alignment and high effective rank are instead enforced by construction.

W4: Spatial generalization

This setting is already part of our evaluation. For the five climate-ranking tasks, a spatially blocked train/validation/test split is used. The coordinates are clustered on the sphere, and complete geographic clusters are assigned to each partition (Appendix E.2, lines 1632–1639 OLD). The downstream predictor is therefore trained on some regions and evaluated on regions that are not seen during training, which is the setting described by the reviewer.

For the remaining tasks, the official splits of established benchmarks are used, namely those of the SatCLIP benchmark [1], SatBird [16], reBEN [17] and Mesogeos [18] (Table 7). These splits are kept unchanged so that our results remain directly comparable with prior work on the same benchmarks.

We agree that this is not stated clearly enough in the initial manuscript, where the use of blocked splits is described only in the appendix. In the updated manuscript, the spatially blocked split is described in the main text (Section 5 NEW).

W5: Visualization of location embeddings

We thank the reviewer for this suggestion. Visualizations of the location embeddings are now provided in the main text (Figure 1 and Section 5.3 NEW) and in a new appendix (Appendix F NEW, Figures 4–11). In all of them, LoCA is compared with SatCLIP, which uses the same location-encoder architecture (degree-40 spherical harmonics and a SIREN, 256-dimensional output), and the embeddings are queried on a global 0.25° grid.

The visualizations show:

  • Embedding gradient (Figures 1a and 4): LoCA changes 1.3 times faster than SatCLIP over land, which is consistent with its higher effective rank. Its embedding varies little over homogeneous regions such as Siberia and the Amazon basin and fast along coastlines, whereas SatCLIP changes fastest over the ocean, where no training location exists.

  • Zonal wind (Figures 1b and 5): with linear probes on the LoCA embeddings, the tropical easterlies, the mid-latitude westerlies and the westerly jets are reproduced at 850 and 200 hPa. Only a faint trace of these belts is obtained from SatCLIP.

  • Annual precipitation (Figures 1c and 6): the dry interiors of Australia and Central Asia and the equatorial rain band over the oceans are recovered from LoCA, and the rain band is absent for SatCLIP.

  • Ocean regimes (Figures 1d and 7): when the ocean embeddings are clustered without labels, the north-to-south sequence of the reference climate regimes is followed by LoCA. The normalized mutual information with the reference is 0.37 for LoCA and 0.16 for SatCLIP, although neither model is trained on ocean locations.

  • Similarity maps (Figures 1e and 8): the cells most similar to an anchor are concentrated in coherent regions for LoCA. Over 918 anchors, the selection of SatCLIP is split into more separate patches than that of LoCA for 78% of the land anchors and 87% of the ocean anchors.

  • Land versus ocean (Figure 9): with a logistic-regression probe, 92.6% of the area north of 60° S is classified correctly from LoCA and 88.0% from SatCLIP, whose errors include patches of land predicted across the open ocean.

  • False color over land (Figure 10): the continents are divided into the same large units by the leading principal components of both encoders, as expected for two encoders fitted to Sentinel-2 targets on land.

  • False color over the ocean (Figure 11): the leading components of LoCA vary smoothly and mostly with latitude, whereas those of SatCLIP form a checkerboard with no counterpart in the ocean.

=============>

W6: Separate encoders versus channel-wise concatenation

With channel-wise concatenation, one embedding is obtained per location, and the image side of the Stage 2 target is reduced to a single view, as in SatCLIP. With separate co-aligned encoders, each modality provides its own view of the same location, and the location encoder is aligned with each of them through the multi-view objective. In this way, the information of every modality is ensured to be modeled by the location encoder.

This is supported by our results:

  • New ablation on concatenated embeddings (Table 15 NEW): We run another baseline requested by @VKJu (Table 15 NEW), in which the location encoder is aligned to the concatenated embeddings of independently pretrained Sentinel-1 and Sentinel-2 encoders. Compared with LoCA, both without metadata, the location encoder collapses on most tasks. LoCA is best on 12 of the 15 tasks.

  • Supervised BigEarthNet experiment (Table 4): the encoder trained on the concatenated input (S1+S2) obtains 78.1 mean F1, which is below the Sentinel-2 encoder alone (78.4). After Stage 2, 80.8 is obtained with this encoder, against 84.3 with the separate encoders.

  • New ablation on different rank anchors (Table 12 NEW): Stage 1 is replaced by a single frozen multi-sensor encoder, Copernicus-FM or DOFA, which encodes Sentinel-1 and Sentinel-2 with the same weights, and only Stage 2 is trained. These encoders use a dedicated architecture for multiple sensors, which is a stronger single-encoder design than channel-wise concatenation. With either of them as the Stage 2 target, LoCA performs better on 13 of the 15 tasks.

  • Training the image encoder with the alignment objective (Table 17 NEW): Training the image encoder together with the location encoder is the joint-training setting of Theorem 1. In this setting, the location encoder collapses, and the frozen SatCLIP is better on all 16 tasks.

Separate co-aligned encoders are therefore used, so that the location encoder is fitted to each modality.

Q1: Contribution of DHEL and MV-DHEL

In Table 16 NEW we report the ablation where DHEL [19] and MV-DHEL [20] are replaced by NT-Xent in stage 1 and a pairwise aggregation of NT-Xent in stage 2. The training with the DHEL family results in 12/15 wins. The reasons for choosing DHEL and MV-DHEL, which follow from the requirement of a high-rank target, are discussed in our response to Reviewer @rMgG (W3 & Q4).

Q2: Contribution of each input and of the training objective

Each modality improves the modeling of different semantics.

  • Metadata: results are provided in the appendix of the initial manuscript (Table 13 OLD, Table 14 NEW). The metadata improves 9 of the 17 tasks, with the largest gains on geophysical and climate tasks, such as air temperature (76.0 to 81.4) and elevation (57.1 to 64.4), while a higher score is obtained on land cover without it. A detailed discussion is given in our response to Reviewer @rMgG (W4 & Q3).

  • Sentinel-1 and Sentinel-2: in the BigEarthNet experiment (Table 4, and Table 12 OLD / Table 13 NEW), Sentinel-2 is the stronger single modality (78.4 mean F1, against 69.6 for Sentinel-1), and the highest score is obtained when both are combined with the location (84.3). The contribution of Sentinel-1 is also seen in the main benchmark. Without the metadata, LoCA obtains 59.1 F1 on reBEN with the MLP probe (Table 15 NEW), against 57.3 for SatCLIP (Table 2), which uses only Sentinel-2 and shares the same location-encoder architecture. This is consistent with the land-cover information of Sentinel-1 being used by LoCA.

Inputs and training objective. The training objective is what allows these inputs to be used together, since the modalities are co-aligned in Stage 1 and combined as separate views by MV-DHEL in Stage 2. The gains therefore come from both. The effect of the inputs is isolated in Table 4, where the same losses are used for every combination of inputs. The effect of the objective is isolated in the new ablation of Q1 in Table 16 NEW.

==================>

Reviewer_rMgGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LoCA, a contrastive location encoder aligned with several modalities. The authors prove that a Lipschitz location encoder produces embeddings of intrinsic dimension at most 2, and observe that joint image-location training collapses the effective rank. They argue that frozen pretrained image encoders avoid this collapse by providing a high-rank target. They then describe two requirements for multimodal location encoding: the modalities must first share a co-aligned space, and this space must have high effective rank. LoCA proposes to do this in two stages: Sentinel-1 and Sentinel-2 encoders are first trained jointly with DHEL, then frozen while a location encoder and a metadata encoder are aligned to them with MV-DHEL. On 17 downstream tasks, LoCA is best on 8 and wins most head-to-head comparisons. Ablations show that Stage 1 is essential, while metadata gives smaller, task-dependent gains.

Critical strengths

I agree with the authors that learning a location encoder with a frozen image model is equivalent to fitting a coordinate-indexed table of image embeddings.

Critical weaknesses
  1. Theorem 1 seems trivial and previously known: a Lipschitz function can’t change the dimensionality of its input. Maybe I’m missing something, but I’m under the impression that there is no contribution here. The authors seem to cite the original sources in the proof, but then state contribution 1 as being theirs.

  2. I’m under the impression that that part iii of Theorem 1 only applies if both encoders are learned simultaneously. This is never the case in the literature, from what I know, making 3.2 and 3.3 rather unnecessary.

  3. The proposed approach reads as the sequential application of Koromilas et al. 2024 and Koromilas et al. 2025. As the paper is written, it would seem that there is no methodological novelty.

  4. The metadata used in the alignment seems to include some concepts (topography, land-cover, demographics) that match some in the evaluation tasks. The baselines don’t seem to have access to these, possibly inflating the results of the proposed method.

  5. I am not sure it makes sense to make a clear-cut distinction between “observation-based” and “location encoders” when by “observation-based” the authors mean Earth embeddings. Earth embeddings are location encoders that use a location index as location representation, and are not more “observation-based” in any sense.

Questions for authors
  1. Are there any elements in Theorem 1 that the authors consider to be an original contribution?

  2. Are there existing location encoders train the image encoder jointly with the location encoder? If none, what does part (iii) of Theorem 1 imply for methods with a frozen image encoder, including LoCA?

  3. How does LoCA compare to the baselines on Elevation, Population and reBEN when the overlapping metadata variables (elevation, slope, population density, land-cover fractions) are removed from Stage 2?

  4. Are there elements in the proposed method, such as the specific choice of losses (e.g. DHEL+MV-SHEL instead of InfoNCE) that the authors consider a contribution? Have they studies the alternatives?

7 comments

W1 & Q1: Theoretical novelty

We thank the reviewer for this comment, which helps us clarify our theoretical contribution. In short, and in response to Q1, Theorem 1 is an original contribution, summarized below.

Our contribution in brief

Current theory in the literature. It is known that contrastive learning is optimized when positive pairs are aligned and representations are uniformly distributed on the hypersphere [1]. It is also a textbook fact that Lipschitz maps cannot increase intrinsic dimension [2].

Our theoretical contribution. Based on these two facts we prove that, when trained jointly, contrastive learning cannot reach its optimum for any Lipschitz encoder whose input has lower intrinsic dimension than the representation space. To the best of our knowledge, this result is new.

Impact of our theory. With this result, a structural limitation of contrastive learning is identified, which arises whenever one of the views has low intrinsic dimension i.e. the standard setup for learning location encoders. This enhances our understanding of learning location encoders and facilitates better design choices. To be more specific our theoretical results have three practical consequences:

  • First, they explain a design choice that has so far been made empirically (See response to W2). Since SatCLIP-style encoders [3] cannot reach the contrastive optimum under joint training (Corollary 1), fitting against a frozen image encoder becomes a necessity not a design choice.

  • Second, they separate the role of the location encoder architecture from that of the target encoder. Using a stronger location encoder, for example one with a larger bandwidth L, only changes the spatial granularity of the embeddings. It does not improve the contrastive optimization, since the gap holds for every L, and this provides a clear guide for the design of location encoders.

  • Third, explains the restrictions of existing location encoders in incorporating additional modalities. A location encoder cannot simply be added as a further contrastive view, and must instead be fitted against a target that is co-aligned across modalities and of high rank, which is the design principle behind LoCA (see response to W2). The same reasoning can be applied to other inputs of very low intrinsic dimension, such as the time encoder of GT-Loc [4], which is likewise trained against a frozen image encoder.

Our contribution in detail

  • Part (ii) OLD (part (i) NEW): States that the distribution of location embeddings, under joint training, has zero overlap with the uniform distribution on the sphere, which is the optimum of contrastive learning.

  • Part (iii) OLD (part (ii) NEW): as a consequence, the contrastive loss of any aligned Lipschitz pair, i.e., under joint training, strictly exceeds its optimum.

  • Corollary 1: this gap holds uniformly over the entire SatCLIP architecture class at any bandwidth, since the infimum over the class is attained and lies strictly above the optimum, which requires a compactness argument beyond applying the theorem to each encoder separately (Appendix B.5).

The role of these results in the rest of the paper, namely explaining why existing location encoders rely on a frozen image encoder and why they cannot be trivially extended to multiple modalities, is discussed in our response to W2.

Contribution 1. Contribution 1 states that "we prove that location encoders constrain contrastive learning away from its optimum", which corresponds to parts (ii) and (iii) OLD, and not to part (i) OLD.

What we change to increase clarity

We believe that the reviewer's impression comes from part (i), which is indeed a well-known textbook result, as already discussed in the Appendix of the initial manuscript (Section B.1 OLD). However, part (i) is not the contribution of Theorem 1, but the first step towards it. It is included as point (i) of the theorem in order to be stated explicitly as a fact, since it is a vital step for proving our theorem. Still, we understand that including it in the theorem may be misleading and may undermine the contribution of our own theoretical work. For this reason, in the updated manuscript, it has been moved to a separate lemma in the appendix (Lemma 1, Section B.1 NEW) and the theorem states the previous (ii) and (iii), which are now parts (i) and (ii).

==================>

W2 & Q2: Necessity of the theorem

We thank the reviewer for this comment, which has helped us improve the clarity of our manuscript. Sections 3.2 and 3.3 are necessary precisely because location encoders are not trained jointly in practice.

How literature justified freezing the image encoder: Existing works adopt frozen image encoders for practical rather than mechanistic reasons. In SatCLIP [3] (Section 2.2 and Appendix C.1) they justify this choice as ideal for the parameter imbalance between the image encoder and the location encoder. In GeoCLIP [6] (Sec. 6 of Suppl.) they attribute their choice to freeze the vision encoder to computational constraints, possible catastrophic forgetting of initial CLIP weights, and ethical implications of using publicly released weights. Similarly in CSP [5] (Sec. 3.2) the authors attribute this choice to parameter imbalance, claiming that joint training will yield overfitted image encoders and underfitted location encoders. GT-Loc [4] adopts the GeoCLIP framework without further discussion. To our knowledge, no prior work provides a theoretical explanation on this choice.

Sections 3.2 and 3.3 provide a formal explanation for a design choice the literature adopted empirically: As discussed in the previous paragraph, prior literature justified the training setup of location encoders on purely empirical observations. Sections 3.2 and 3.3 provide and justify a complete framework for location encoders which adds to the fundamentals of this line of research. In Section 3.2 it is established that location encoders cannot attain the contrastive optimum when trained from scratch (see our response to W1), and this is accompanied by the empirical result that joint training collapses the embedding to an effective rank of 8 (Table 1; lines 444–445 OLD). In Section 3.3 it is then discussed that existing location encoders are not trained from scratch (lines 270–283 OLD). Taken together, the two sections show that all location encoders are trained against a frozen image encoder precisely because they cannot be trained from scratch.

This becomes critical once it is combined with the fact that encoders of different modalities are not aligned (lines 108–113 and 294–308 OLD). As a result, a multimodal location encoder (i) cannot be constructed by extending the way location encoders are currently trained, as confirmed by the Stage 1 ablation, where freezing two independently pretrained encoders collapses performance (Table 3; lines 452–456 OLD), and (ii) cannot be trained the way multimodal encoders are trained with contrastive learning, since these are trained from scratch.

For methods with a frozen image encoder, including LoCA, part (ii) OLD of the theorem (part (i) NEW) still holds, so the location embeddings still cannot reach the uniform optimum. Instead, the location encoder fits the frozen target and inherits its spectrum (lines 313–316 OLD). This is why a co-aligned, high-rank target is required, and it is what makes the design of the LoCA pipeline principled.

It is true that part (iii) OLD (part (ii) NEW) concerns encoders trained jointly from scratch, which is not how location encoders are trained in practice, as already discussed in the initial manuscript (lines 99–106 and 270–283 OLD). To make this clearer, the following changes have been made in the updated manuscript:

  • Abstract and introduction: it is now stated explicitly that the theoretical result concerns location and image encoders trained jointly.

  • Section 3.2: a paragraph has been added after Corollary 1, which clarifies that the results concern training from scratch and explains why they are nevertheless necessary for the rest of our analysis.

  • Section 3.3: it is now stated that part (ii) OLD of the theorem (part (i) NEW) does not depend on alignment, so the location embeddings cannot reach the uniform optimum even when the image encoder is frozen.

Regarding Q2. To the best of our knowledge, the image encoder is kept frozen in all existing contrastive location encoders, such as CSP [5], GeoCLIP [6] and SatCLIP [3]. Based on the above discussion, the result for joint training is a vital step towards understanding how location encoders work and how they can be extended to multiple modalities. Beyond part (ii) OLD (part (i) NEW), which holds for any Lipschitz location encoder, Theorem 1 does not describe what happens when the image encoder is frozen, and this setting is outside the reach and scope of our theoretical results. The choice of a frozen image encoder is instead motivated by the working paradigm in the literature, and it is interpreted as a covariance regularizer that supplies a high-rank target (Section 3.3).

==================>

W3 & Q4: Methodological contributions and choice of losses

We thank the reviewer for this comment which helps us clarify our methodological contribution. The methodological contributions of LoCA are the following:

  • Two-stage pipeline: a co-aligned, high-rank target is first constructed by training the Sentinel-1 and Sentinel-2 encoders jointly (Stage 1), and the location encoder is then fitted against this frozen target (Stage 2).

  • Metadata encoder: to the best of our knowledge, a metadata encoder is introduced into a location encoder for the first time. We design it as an MLP that processes the 14 tabular metadata variables as a single feature vector. In existing Earth observation models, such variables are usually used as 2D rasters that follow the spatial structure of the Sentinel-2 images, as in MMEarth [7] and TerraMind [8]. In LoCA, they are instead used as point-wise tabular inputs.

  • Modality adapters and projection head: a trainable MLP adapter is attached to each modality, followed by a projection head that is shared by all modalities and used during training. This design is vital to enable embeddings from different modalities and is not used in existing location encoders. In SatCLIP, for example, only the last linear layer of the frozen image encoder is trained, and no projection head is used on the location side.

  • Pretraining data: a dataset is constructed in which Sentinel-1 and Sentinel-2 images are paired and aligned with the 14 metadata variables (Appendix C).

The results of the method are also the strongest among the compared encoders. LoCA is the only encoder that outperforms every other encoder on a majority of the shared tasks, with the highest pairwise win rate (68.1%, Table 11 NEW). For this reason, LoCA is the natural choice of location encoder for a wide array of applications, since it is not consistently outperformed by any other encoder.

Relation to DHEL and MV-DHEL. DHEL and MV-DHEL are the contrastive losses used in each stage, and they are not claimed as a contribution of this work. In principle, any contrastive loss that yields a high-rank target could be used in their place, and DHEL and MV-DHEL are chosen because they attain the highest effective rank [10, 11]. From the understanding built by our theory, two requirements are derived for multimodal location encoding (Section 4; lines 294–316 OLD), and each design choice of LoCA follows from them:

  • Stage 1, co-alignment across modalities → the Sentinel-1 and Sentinel-2 encoders are trained jointly.

  • Stage 1, high effective rank → DHEL is the contrastive loss that attains the highest effective rank [10], so it is the natural choice.

  • Stage 2, contrastive alignment of four modalities while keeping this high rank → MV-DHEL is the contrastive loss that handles many modalities by design while also keeping a high rank [11], so it is the natural choice.

Each step is also supported empirically:

  • Joint training collapses the embedding to an effective rank of 8 (Table 1).

  • Independently pretrained encoders are not co-aligned (lines 430–441 OLD).

  • Removing Stage 1 collapses performance, while Stage 2 improves every encoder (Tables 3 and 4; lines 452–463 OLD).

  • New: replacing Stage 1 with frozen state-of-the-art encoders, such as Copernicus-FM [12], DOFA [13] or DINOv3 [14], leads to worse results, and LoCA wins 13 of the 15 tasks against each of them (Table 12 NEW).

The two-stage pipeline, and the choice of DHEL and MV-DHEL within it, follow directly from this analysis (lines 116–121 and 355–369 OLD). Nevertheless, to examine the effect of this choice, the ablation in which both losses are replaced by InfoNCE (NT-Xent) has been run, and the results are reported in Table 16 of the updated manuscript. LoCA trained with DHEL and MV-DHEL is better on 12 of the 15 tasks, and the largest gains are observed on Air temperature (90.3 against 78.3 R²) and Wind speed (75.6 against 68.2 ranking accuracy).

Finally, the method is only one part of our contribution, since it is the direct application of the theory (see our response to W1) and of the understanding developed in Sections 3.3 and 4.

==================>

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Comment and Rebuttal Summary

Authors general comment: Our manuscript provides a principled view and understanding of location encoders and shows how to properly extend them to incorporate multiple modalities. Existing location encoders train against a frozen image encoder, a choice that prior work attributed to compute, parameter imbalance or forgetting. We ground this choice in theory by showing that location encoders cannot be trained from scratch (Theorem 1 and Table 17 NEW). Location encoders therefore do not perform contrastive learning in the usual sense, but instead fit a coordinate-indexed lookup table onto a frozen target. Making this explicit identifies two requirements that must be satisfied by the target to enable multiple modalities in location encoders, namely co-alignment and high effective rank. LoCA is designed to satisfy both, and combines Sentinel-1, Sentinel-2 and metadata in a single location encoder. It obtains the most task wins across 17 diverse Earth Observation tasks and beats each of the 6 (+1 in the Appendix) state-of-the-art models in pairwise comparisons. The new ablations requested by the reviewers show that each design choice is necessary, and their comments have helped us show this more clearly and clarify the scope of our theory. We therefore thank the reviewers for their truly constructive feedback.

The main concerns raised by the reviewers are addressed by the following results. Throughout the rebuttal, NEW refers to the updated manuscript and OLD to the initial manuscript (e.g., Table 12 NEW). Tables 1–11 keep their numbers in both versions, and all line numbers refer to the initial manuscript.

[Update] Following the suggestion of Reviewer ZkCA, who has acknowledged our responses and raised their score, all additions and modifications with respect to the initial submission are now marked in dark red in the revised PDF. Deleted text is not shown, as displaying it would push the main text beyond the 10-page limit.

New material

  • Additional baselines (Tables 2 and 18 NEW): LoCA beats TTE on 11/17 tasks and GAIR on 14/16.

  • Rank anchors (Table 12 NEW): Stage 1 cannot be removed. When Stage 2 is trained against frozen DINOv3, Copernicus-FM or DOFA targets, LoCA wins on 13/15 tasks against each (e.g., Ecoregions 80.2 vs. at most 49.2 F1).

  • Training without a frozen image encoder (Table 17 NEW): When the image encoder of SatCLIP is not frozen, the location encoder collapses, and the frozen SatCLIP is better on all 16 tasks (e.g., Countries 95.5 vs. 2.4 F1).

  • Visualizations (Figures 1 and 4–11 NEW): Compared with SatCLIP, LoCA recovers the wind belts and the rain band, which SatCLIP misses, and gives more coherent similarity maps, better land/ocean separation and better ocean-regime clustering (NMI 0.37 vs. 0.16).

  • Fusion ablation (Table 15 NEW): Fusing S1 and S2 by concatenation collapses the location encoder (R² < 0.01 on 4 tasks). LoCA wins 12/15.

  • Choice of contrastive loss (Table 16 NEW): Replacing DHEL and MV-DHEL with NT-Xent gives lower scores on 12/15 tasks.

Material already in the initial manuscript

  • Metadata overlap (Tables 3 and 13 OLD, Tables 3 and 14 NEW): LoCA was constructed to absorb an arbitrary number of modalities, unlike prior location encoders. Nevertheless, removing the metadata does not change the best model on any of the overlapping tasks, and LoCA even scores higher on reBEN without it.
  • Spatial generalization (Appendix E.2 OLD): Already evaluated in the initial manuscript. The 5 climate-ranking tasks use spatially blocked splits.

==============>

Acknowledged Contributions

Reviewers explicitly recognized our main contributions as well as the practicality of our approach:

ContributionRecognized byIn the reviewers' words
1. Strong performancerMgG, ZkCA, VKJu"wins most head-to-head comparisons" (rMgG), "best reported performance" (ZkCA), "nice performance across downstream tasks" (VKJu)
2. Broad evaluationZkCA, VKJu"broad set of experiments" across "ecological, environmental and land cover modeling" (ZkCA), "shows the wide applicability of the method" (VKJu)
3. Analysis of the geometry of location encodersZkCA"useful distinction between intrinsic dimensionality and covariance rank" (ZkCA)
4. Simplicity of the methodVKJu"quite clear and simple" (VKJu)
5. Reinterpretation of location encoders as lookup tablesrMgG"equivalent to fitting a coordinate-indexed table of image embeddings" (rMgG)
6. Necessity of the proposed pipelinerMgG"Stage 1 is essential" (rMgG)

Main Concerns and How We Address Them

Most remaining concerns focused on the role of the theoretical result and on the experimental comparison with additional baselines and alternative design choices.

  • Theorem 1 novelty and scope (rMgG, VKJu, ZkCA): Our theory explains a design choice that all prior work, to our knowledge, has made empirically, namely freezing the image encoder. Theorem 1 shows that under joint training, no Lipschitz location encoder can reach the contrastive optimum, uniformly over the SatCLIP class. Joint training pulls the image embeddings towards the low-dimensional surface of the location encoder, and the effective rank of the location embeddings collapses to 8 (Table 1). Consistent with this, SatCLIP performs worse on all 16 tasks when its image encoder is not frozen (Table 17 NEW). Our theorem also separates the roles of the two encoders: the bandwidth of the location encoder sets the spatial granularity, and the target encoder sets the quality of the representation space that the location encoder fits. Consistent with this, the effective rank inherited from the target follows the same order as the downstream win rates (LoCA 62%, SatCLIP 35%, Copernicus-FM 21% of the dimensions; Tables 1 and 11 NEW). Since the location encoder only fits its target, new modalities can be added neither as independent frozen targets, which are not co-aligned (Section 4.1, Table 3), nor as jointly trained contrastive views, which collapse. This is the core motivation behind LoCA, which sets out both the requirements for multimodal location encoders and a practical framework to achieve them.

  • Methodological novelty (rMgG, VKJu): In other ML methods (LiT, Rethinking JEPA), freezing is an optional choice that improves performance. For location encoders it is a necessity, and with several sensors the frozen targets must also be co-aligned and high-rank. Off-the-shelf encoders meet neither condition (Tables 12 and 15 NEW), and LoCA is built to meet both. Beyond the two-stage design, the pipeline introduces the first metadata encoder for location encoders, per-modality adapters with a shared projection head, and a new dataset that pairs Sentinel-1 and Sentinel-2 with 14 metadata variables.

  • Additional baselines (ZkCA): LoCA's advantage holds against the newest location encoders, TTE (11/17) and GAIR (14/16), which are now included in the benchmark (Tables 2 and 18 NEW).

  • Spatial generalization and visualizations (ZkCA): Spatial generalization was already evaluated with geographically blocked splits. The new visualizations show that LoCA recovers physically meaningful structure, such as wind belts and ocean regimes, which SatCLIP misses.

  • Ablations on the rank anchors, the modalities, the metadata and the loss (rMgG, ZkCA, VKJu): Stage 1 cannot be replaced by an off-the-shelf encoder, since LoCA wins 13/15 against frozen DINOv3, Copernicus-FM and DOFA targets (Table 12 NEW). Copernicus-FM and DOFA are trained with masked modelling, which halves the effective rank compared with contrastive learning on the same data (Section 4.2 NEW). Simple fusion fails, since concatenating S1 and S2 keeps sensor-specific variation that cannot be predicted from the coordinates, and the location encoder collapses (LoCA wins 12/15). The metadata does not drive the results, since it changes the best model on none of the overlapping tasks. Replacing DHEL and MV-DHEL, which are chosen for their high effective rank, with NT-Xent (InfoNCE) gives lower scores on 12 of the 15 tasks (Table 16 NEW).

In the rest of the rebuttal, we respond point-by-point to each reviewer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.