- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DOMAIN-SPECIFIC LANGUAGE ALIGNMENT OF MULTIMODAL EMBEDDINGS TOWARDS EXPLAINABLE PREDICTIONS explores the use of SOTA Earth embeddings for species distribution modelling (SDM). The authors benchmark existing SOTA models and find that AlphaEarth achieves results above previous SDM-specific models. They further propose a textual-alignment technique to improve the explainability and interpretability of these embeddings without requiring prior explicit textual captions or pairings. The technique could in principle be extended to other domains, though this is not explored in the paper.
Critical strengths
S1 - The paper proposes a new approach to interpret Earth embeddings and moves towards better explainability of their downstream predictions. This is a relevant research topic, and the obtained results provide a valuable contribution to the area. The proposed alignment method leverages a soft contrastive term to handle partial captions obtained from auxiliary data, which is a reasonable modification of InfoNCE for the problem at hand.
S2 - The prediction evaluations provide a useful benchmark comparing SOTA approaches for SDM and Earth-embedding probing, including an extensive ablation study to identify the best configurations for the latter.
Critical weaknesses
W1 - The alignment baselines do not provide a fully controlled comparison against AEF.
- Most baseline methods receive a single satellite image as input, from which some of the evaluated concepts may not be recoverable, while AEF incorporates multimodal Earth-observation information.
- AEF is aligned using captions generated from the same auxiliary variable groups underlying most of the in-distribution evaluation, putting it at an inherent advantage over pretrained baselines that were not optimized for these variables.
- AEF is evaluated using mixed and separate captions, but not against the same alignment strategy applied to another Earth embedding.
Therefore, we cannot reasonably infer whether the alignment gains observed in Tables 3 and 4 result from a better initial embedding, a better alignment strategy, closer correspondence between training and the evaluation protocol, or a combination of these factors.
W2 - What does language alignment buy?
- Training captions are constructed from auxiliary variables, most of which are subsequently used to evaluate alignment. These variables could in principle be aligned with the Earth embeddings directly, without passing through a textual embedding space. Figure 2 suggests that the introduction of language space may hinder some concepts: several bioclimatic variables are strongly encoded by AEF but poorly retrieved after alignment. This creates a cost to textual alignment that should ideally be justified by capabilities unavailable to direct auxiliary-variable alignment.
- Language should, in principle, provide stronger generalization to unseen concepts and enable open-vocabulary querying. Both experiments are present in the paper, but the evidence would benefit from further clarification:
- For OOD evaluation, most evaluated concepts are present during training. Of the two held-out S2BMS data groups, UKCEH is another land-cover dataset and may therefore be semantically close to the in-distribution CORINE variables, while PM2.5 consists of a single variable for which zero-shot GeoCLIP outperforms the proposed alignment. Furthermore, no OOD auxiliary groups are considered for SatBird. The results would be strengthened by considering a larger and more diverse set of OOD variables.
- Figure 3a shows that concept-caption similarities can recover relationships obtained from auxiliary data within the same geographic domain. It remains unclear whether these relationships would transfer to another region without access to the auxiliary data used to construct the alignment.
- The open-vocabulary experiment considers habitat descriptions constructed from in-distribution concepts that already achieved good retrieval performance. Therefore, it does not establish whether the proposed alignment can meaningfully answer OOD open-vocabulary queries.
Language alignment provides additional convenience and interpretability, which is itself relevant, but evidence of stronger OOD generalization would provide a significant additional motivation for introducing the textual space.
Questions for authors
Q1 - How would AEF alignment behave with a standard InfoNCE loss (one positive per data point)? The ablations vary sigma between 0.1 and 2.0 but never include the hard-contrastive case (sigma → 0) that the paper explicitly compares against.
Q2 - How would another embedding model (e.g., Tessera or SatCLIP, already computed for Table 1) perform with the proposed alignment strategy? This would help separate the contribution of the alignment method from that of AlphaEarth and support the claim that the framework applies to arbitrary embedding spaces.
Q3(clarification) - UKCEH, used for OOD evaluation, represents the same broad type of information as CORINE (land cover). How much do its classes overlap semantically with the CORINE classes used in the training captions, and does retrieval performance differ for UKCEH classes without a close CORINE counterpart?
Q4(clarification) - Why are GeoCLIP and CLIP absent from Table 4? GeoCLIP in particular is the only coordinate-based alignment baseline and achieved the best PM2.5 retrieval score in Table 3.
Rebuttal JCEE
Dear reviewer JCEE,
Thank you very much for your detailed review of our paper. We are happy that you found our paper to be a valuable contribution to the field, and we address your questions below:
Q1 InfoNCE loss comparison
We provide the requested comparisons. InfoNCE (sigma=0) results are now added to Figures S4-S7 (and mentioned in Experiments). This shows that this hard loss function in fact performs well, with the best hard loss configuration matching the best soft loss configuration (average score of 0.45 pm 0.00 for both). This is caused by a much better performance on the human footprint variables, which compensates for a worse performance on the bioclimatic and land cover variables. We have updated Tables 3 and S13, and have changed the “Alignment” paragraph in the Results section accordingly, to:
“Retrieval performance was best when AlphaEarth was projected to the text space on the larger, unlabelled dataset, using a large batch size of 512 (Table S13, Figs S4, S5). This is likely caused by the increased likelihood of encountering data points with similar environmental conditions for larger batch sizes, while the smaller, labelled data size performed best with a smaller batch size. Notably, this model achieved equal performance (maximum average score of 0.45 pm 0.00) for both sigma=0 and sigma=0.5, where sigma=0 models scored best for human footprint variables, and sigma>0 scored best for bioclimatic and land cover variables. Hence, a soft loss function that weighs positive pairs by their similarity improves the alignment of some, but not all, concepts.”
Thank you for suggesting this; this is certainly an important experiment to include.
Q2 Alignment of another embedding model
Thank you for this suggestion. We have now also trained a Tessera alignment model, results added to Table 3, using the S2BMS labelled configuration with a full hyperparameter sweep (Figure S8). The results in Table 3 confirm that the alignment framework is not specific to AlphaEarth: Tessera alignment achieves comparable performance, except for Human Footprint retrieval where it underperforms, demonstrating that the proposed method generalises to other embedding spaces. We have updated the text in experiments, results, and the appendix to reflect these new experiments.
Q3 Land cover concepts overlap
Yes, UKCEH and CORINE both represent land cover classes. They vary slightly in their semantic descriptions, but also in how land cover is mapped (i.e., the underlying auxiliary data values). We have now added a new analysis in Figure S9 that compares the retrieval scores and the AEF encoding strength between UKCEH and CORINE, by comparing the UKCEH classes to their equivalent in CORINE, for each of the 3 CORINE levels. Correlation values are positive but none are significant, though the number of data points is arguably too low to draw strong conclusions from.
Q4 Missing GeoCLIP and CLIP baselines in Table 4
Thank you for highlighting this: these results were accidentally left out in the original submission and are now restored in Table 4. Across all metrics in Table 4, neither surpasses the AlphaEarth model.