- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies contrastive location encoders, which map coordinates to embeddings by aligning them with a paired image modality (e.g., satellite or ground-level imagery) and asks how to align a single location encoder to several modalities at once. There are several arguments. The authos first show that location encoders are low-dimensional, find that frozen pretrained image encoders help, and identify co-alignment and high effective rank as prerequisites for multimodality. Based on this, LoCA is proposed and shows nice performance across downstream tasks.
Critical strengths
-
The method is quite clear and simple. Contrastively pretrain S1-S2 encoders then reuse that pretrained image encoder and contrastively pretrain again with location and metadata. There is some resemblance with the LiT (Zhai et al., CVPR 2022) and Rethinking JEPA (Li et al., ICLR 2026).
-
The use case on 17 tasks is broad, which shows the wide applicability of the method. This is nice considering the simplicity of the method.
Critical weaknesses
-
Treating location encoders as adapters over a frozen target resembles LiT (Zhai et al., CVPR 2022) and Rethinking JEPA (Li et al., ICLR 2026), which show the effectiveness of learning against a fixed pretrained target. The frozen-target reframing is therefore not itself new, and the paper does not clearly explain what makes this specifically challenging (or non-trivial) in EO for contribution claims.
-
The rank requirement used to rule out masked-modelling targets seems weakly supported (I checked the references, but I do not see a concrete reference on why use CL over MAE in EO). There is a need for elaboration on this as it's not clear why not use multimodal MAE as stage 1 instead of CL since many EO foundation models for remote sensing are masked image modeling based. Both experiments and explanations are needed.
-
The theorem is interesting, but I do not think it fully explains the empirical results. The theorem shows a strict gap from the contrastive optimum, but does not show that this gap is large enough to explain the observed performance degradation. More importantly, it is about intrinsic dimension, whereas the empirical failure mode is low effective rank, which is a different property (and the paper does admit). Since the same intrinsic-dimension limit also applies to LoCA, the paper needs a clearer explanation of how the theorem actually motivates LoCA’s gains.
Questions for authors
q1 The paper argues that freezing the image encoders is essential. Did the authors try a Stage 2 variant where the Stage 1 encoders are finetuned together with the location encoder, as in LiT's locked versus unlocked comparison? Downstream results for the from-scratch SatCLIP variant in Table 1 would also help.
q2 Why is contrastive learning necessary for Stage 1, rather than a multimodal masked autoencoder? The claim that masked modelling produces low-rank embeddings rests on citations that do not appear to show this directly.
q3 What explains the negative R-squared values in the ablation without Stage 1? This is quite counterintuitive considering the tasks are location-based. The encoder's input already varies smoothly with location, targets such as air temperature are strongly tied to latitude, and the splits do not appear to be spatially blocked, which favors interpolation. Needs clarifications.
q4 Why not simply fusing representations instead of forcing them into a single contrastively aligned space? It is not clear why co-alignment via CL should be preferable to preserving separate S1, S2, and metadata representations and combining them through simple fusion techniques like ensembles, etc. A fusion baseline would help establish whether the shared contrastive space is necessary or not.
Minor
- Check citations in Section 5.2 and in the TESSERA reference in Appendix F.1.
10 comments
W2 & Q2: Rank requirement and contrastive learning versus masked modelling in Stage 1
We thank the reviewer for this comment, which has helped us increase the clarity of our work. Both a direct measurement and a new experiment are now provided, together with more direct references.
-
New experiment with multimodal masked-modelling targets. Stage 1 is replaced by Copernicus-FM [3] or DOFA [4], two state-of-the-art EO foundation models that process both Sentinel-1 and Sentinel-2 and are trained with masked image modelling. With either of them as the Stage 2 target, LoCA performs better on 13 of the 15 tasks (Table 12 NEW, where the RGB model DINOv3 [2] is also tested; see also our response to Reviewer ZkCA, W3). An effective rank to available dimensionality ratio of 20.6% is measured for Copernicus-FM, against 62% for LoCA (Table 1).
-
Direct measurement of the rank. We compare two ViT-S/16 models pretrained on the same data (SSL4EO-S12 [8]) isolating the pretraining objective as the sole contributing factor. The effective rank of the CLS embedding is 93 of 384 dimensions (24.2%) with MAE [9], and 188 of 384 (49.0%) with MoCo-v3 [10] (Section 4.2 NEW). Under masked modelling, half as many directions are used.
-
References that show this directly. In Zhang et al. [11] (Section 3.3 and Figure 1b), it is shown that the features of MAE suffer from dimensional collapse, since they lie in a low-dimensional subspace with fewer large singular values, and that their effective rank decreases during training. In Earth observation, GAIR [12] reports that replacing its contrastive objective with an MAE objective reduces the burn-scar mIoU from 83.26% to 74.15%, and both GAIR-MAE and SatMAE [13] are among the weakest models in its benchmark.
We note that LoCA is proposed as a framework defined by two requirements on the target, namely co-alignment across modalities and high effective rank. Contrastive learning in Stage 1 is a controllable way to satisfy both. We do not claim that this is the only pretraining scheme that can achieve this. If a pretrained model that satisfies the two requirements becomes available, it can be introduced in LoCA in place of Stage 1, in the same way that a different location encoder can replace the SIREN [14]. So far, however, multimodal foundation models do not reliably make use of additional modalities. In GEO-Bench-2 [15] (Appendix 6.3.6), adding SAR to optical inputs reduces the test metrics of all models on PASTIS and of most models on BigEarthNet v2, with TerraMind [16] as the main exception.
To make the rank requirement better supported in the main text, the direct measurement and the reference to Zhang et al. [11] are now included in Section 4.2 NEW.
W3: How Theorem 1 motivates LoCA
We thank the reviewer for this comment.
Size of the gap. Theorem 1 is an impossibility result, and no claim on the magnitude of the gap is made. It is shown that the contrastive optimum cannot be reached by any aligned pair with a Lipschitz location encoder, and by Corollary 1 this holds uniformly over the whole SatCLIP family (Appendix B.5). The downstream degradation is not attributed to this gap. It is attributed to a separate empirical finding, which is the collapse of the embedding to an effective rank of 8 under joint training (Table 1), as stated in the abstract, in C1 and in Section 3.3. This collapse is measured in Table 1, and its effect on downstream performance is now measured for SatCLIP with an image encoder that is not frozen (Table 17 NEW, see Q1). Theorem 1 is used instead to explain why location encoders are trained against a frozen target in practice, which is consistent with the design of existing location encoders such as SatCLIP and GeoCLIP.
Intrinsic dimension and effective rank. Only the intrinsic dimension is bounded by the theorem, while the effective rank is inherited from the target (Section 3.3). The intrinsic dimension is 2 in either case, but the spectrum of the location embedding is changed by the choice of target. Without a fixed target, as in joint training, a low effective rank is observed. With a frozen high-rank target, the 2-dimensional surface is forced to oscillate, and a high effective rank is obtained (Table 1 and Figure 4 NEW).
Gains of LoCA under the same limit. The intrinsic dimension of LoCA is also bounded by 2, and its gains are obtained from the higher-rank, co-aligned target of Stage 1. In support of this, the fraction of dimensions used by LoCA (62%), SatCLIP (35%) and Copernicus-FM (21%) follows the same order as their pairwise win rates (68.1%, 57.4% and 33.0%, Tables 1 and 2 NEW). When the Stage 1 target is replaced by the masked-modelling targets of Copernicus-FM (effective rank 21%) or DOFA, the resulting model is outperformed by LoCA on 13 of the 15 tasks (Tables 1 and 12 NEW). When it is replaced by independently pretrained encoders, the performance collapses (Table 3, and Tables 14 and 15 NEW).
Q3: Negative R² without Stage 1
In Table 13 OLD (Table 14 NEW), all values are reported as metric × 100, so the R² of the variant without Stage 1 lies between −0.079 and −0.007. These values are close to zero, which is the R² of a trivial predictor that always outputs the mean of the targets.
Since targets such as air temperature are strongly related to latitude and longitude, an R² close to zero indicates that probably the location has not been modeled by the embedding. This is expected, since without Stage 1 the location encoder is trained to match two independently pretrained targets that do not agree with each other (cos = 0.50, lines 430–440 OLD). For each location, two different embeddings must therefore be fitted at the same time, which cannot be achieved by a single function of the coordinates (Section 4.1).
This holds for both types of split. The regression tasks follow the random splits of the SatCLIP benchmark, which favor interpolation, and an R² close to zero is still obtained. The climate-ranking tasks use spatially blocked splits, in which complete geographic clusters are held out (Appendix E.2, lines 1632–1639 OLD). These tasks are evaluated with pairwise ranking accuracy, for which 50% corresponds to a random ordering, and the accuracy of this variant lies between 49.6% and 50.6%.
Q4: Fusion versus a shared contrastive space
We thank the reviewer for this suggestion. The requested fusion baseline has been run (Table 15 NEW), in which the location encoder is aligned to the concatenated embeddings of independently pretrained Sentinel-1 and Sentinel-2 encoders. Compared with LoCA, both without metadata and evaluated with the same MLP probe, the location encoder collapses on most tasks, with an R² below 0.01 on Median income, Population, Air temperature and Elevation, and an F1 of 8.7 against 80.3 on Ecoregions. LoCA is best on 12 of the 15 tasks.
This is expected, since a location encoder can only represent information that is a function of the location. With concatenation, the variation that is specific to each sensor, such as acquisition conditions or noise, is kept in the target, although it cannot be predicted from the coordinates. With contrastive co-alignment, the information that is shared by the two sensors at the same location is retained, and the sensor-specific variation is suppressed [17]. The shared contrastive space of Stage 1 is therefore necessary.
Minor: Citations
We thank the reviewer for spotting these. The missing citation in Section 5.2 (Garrido et al. [7]) and the TESSERA citation in the extended related work (Appendix F.1 OLD, Appendix G.1 NEW) have been fixed.
References
[1] Zhai et al. "LiT: Zero-Shot Transfer with Locked-image Text Tuning", CVPR 2022.
[2] Siméoni et al. "DINOv3", arXiv 2025.
[3] Wang et al. "Towards a Unified Copernicus Foundation Model for Earth Vision", ICCV 2025.
[4] Xiong et al. "Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation", arXiv 2024.
[5] Li et al. "Rethinking JEPA: Compute-Efficient Video SSL with Frozen Teachers", ICLR 2026.
[6] Klemmer et al. "SatCLIP: Global, General-Purpose Location Embeddings with Satellite Imagery", AAAI 2025.
[7] Garrido et al. "RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank", ICML 2023.
[8] Wang et al. "SSL4EO-S12: A Large-Scale Multimodal, Multitemporal Dataset for Self-Supervised Learning in Earth Observation", IEEE GRSM 2023.
[9] He et al. "Masked Autoencoders Are Scalable Vision Learners", CVPR 2022.
[10] Chen et al. "An Empirical Study of Training Self-Supervised Vision Transformers", ICCV 2021.
[11] Zhang et al. "How Mask Matters: Towards Theoretical Understandings of Masked Autoencoders", NeurIPS 2022.
[12] Liu et al. "GAIR: Location-Aware Self-Supervised Contrastive Pre-Training with Geo-Aligned Implicit Representations", ISPRS JPRS 2026.
[13] Cong et al. "SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery", NeurIPS 2022.
[14] Sitzmann et al. "Implicit Neural Representations with Periodic Activation Functions", NeurIPS 2020.
[15] Simumba et al. "GEO-Bench-2: From Performance to Capability, Rethinking Evaluation in Geospatial AI", TMLR 2026.
[16] Jakubik et al. "TerraMind: Large-Scale Generative Multimodality for Earth Observation", ICCV 2025.
[17] Tian et al. "Contrastive Multiview Coding", ECCV 2020.
[18] Vivanco Cepeda et al. "GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization", NeurIPS 2023.
W1 & Q1: Relation to literature, and the role of the frozen target
Indeed training against a frozen pretrained encoder has been an established technique for more than a decade. Providing a frozen encoder is therefore not a contribution of our paper. Our contribution is to formally show that location encoders do not work without a frozen target, which diverges from common knowledge of the community (also see response @rMgG on W2: How literature justified freezing the image encoder), which in turn rules out the natural multimodal extension of joint training of multiple encoders since frozen encoders must be used as evidenced from our new ablation (Table 17 NEW). In the domain of Earth Observation this limitation is even more pronounced due to the multimodal nature of the data. However, these frozen encoders are not compatible with each other, as shown in our new ablation in Table 15 NEW, where concatenating the embeddings of pretrained Sentinel-1 and Sentinel-2 encoders fails on most tasks (e.g., 8.7 against 80.3 for LoCA without metadata on Ecoregions) and loses to the contrastive Stage 2 on 12 of the 15 tasks, because the two encoders are not trained to be compatible (see response of Q4). This makes the co-alignment requirement necessary alongside the rank requirement, and this chain of arguments provides our methodological contribution, which is distinct from prior frozen-target methods such as LiT [1] and Rethinking JEPA [5]. In these methods, the frozen target is not a necessary condition, since training from scratch (LiT) or with an EMA teacher (Rethinking JEPA) also works, and freezing only improves performance and efficiency. A single target source is assumed, and the alignment of several targets does not arise.
Regarding Q1. Fine-tuning the Stage 1 encoders together with the location encoder in Stage 2 corresponds to training the image and location encoders jointly, which is the setting of Theorem 1. In this setting, the contrastive loss is kept strictly above its optimum, and the image embeddings are pulled towards the 2-dimensional surface of the location encoder. This is observed in Table 1, where the same SatCLIP [6] architecture is trained in both ways. The effective rank is 89 when the image encoder is pretrained and frozen, and it collapses to 8 when the image encoder is trained jointly from scratch (SatCLIP†). Since the effective rank is a standard proxy for linear-probe performance [7], a very poor downstream performance is expected for an embedding that uses only 8 of its 256 dimensions. In a similar direction, the authors of GeoCLIP [18] study the impact of the frozen backbone and report that finetuning the CLIP backbone results in worse performance than freezing (Suppl. Section 6 "Motivations for using Pretrained CLIP as Image encoder Backbone").
The downstream results of SatCLIP with an image encoder that is not frozen are now reported in Table 17 NEW. Without a frozen image encoder, the location encoder collapses, and the frozen SatCLIP is better on all 16 tasks. For example, with the MLP probe, the F1 on Countries is reduced from 95.5 to 2.4, the micro-F1 on reBEN from 57.3 to 1.6 and the R² on Air temperature from 93.9 to 0.2. The ranking accuracy on the five climate variables lies between 49.9% and 50.6%, i.e., at chance level.
==================>