acceptodds
Under review as a conference paper at ICLR 2027

Domain-specific language alignment of multimodal embeddings towards explainable predictions

Abstract

Multimodal data are commonly fused into latent representations that improve downstream task performance but are not interpretable, limiting their application in domains that require explainable model decisions. Vision-language alignment has made image embeddings interpretable through language, but requires vast image-caption training data. However, captioned data are typically not available for multimodal embeddings, as relevant features cannot directly be observed and annotated from embeddings alone. Here, we propose a framework to language-align any embedding space that can be linked to interpretable auxiliary data. Captioning functions convert auxiliary data into partial captions; a soft contrastive loss weights pairs by the similarity of their properties that each caption describes; and a rank-based concept retrieval task evaluates alignment against the auxiliary data rather than the captions. We apply this framework to multimodal Earth embeddings to facilitate explainable predictions for two species distribution modelling tasks: S2BMS and SatBird. AlphaEarth embeddings beat previous state-of-the-art models on 6/7 metrics, while our framework language-aligns domain-specific concepts of the embedding space, outperforming previous Earth observation vision-language models on average concept retrieval for both datasets. Finally, aligned concepts recover species–habitat associations () and relate habitat descriptions to species predictions, showing that abstract embeddings can be interrogated in the language of domain experts.

Primary area:
interpretability and explainable AI
Keywords:
multimodal, language alignment, contrastive learning, open-vocabulary queries, Earth observation, species distribution models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JCEEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DOMAIN-SPECIFIC LANGUAGE ALIGNMENT OF MULTIMODAL EMBEDDINGS TOWARDS EXPLAINABLE PREDICTIONS explores the use of SOTA Earth embeddings for species distribution modelling (SDM). The authors benchmark existing SOTA models and find that AlphaEarth achieves results above previous SDM-specific models. They further propose a textual-alignment technique to improve the explainability and interpretability of these embeddings without requiring prior explicit textual captions or pairings. The technique could in principle be extended to other domains, though this is not explored in the paper.

Critical strengths

S1 - The paper proposes a new approach to interpret Earth embeddings and moves towards better explainability of their downstream predictions. This is a relevant research topic, and the obtained results provide a valuable contribution to the area. The proposed alignment method leverages a soft contrastive term to handle partial captions obtained from auxiliary data, which is a reasonable modification of InfoNCE for the problem at hand.

S2 - The prediction evaluations provide a useful benchmark comparing SOTA approaches for SDM and Earth-embedding probing, including an extensive ablation study to identify the best configurations for the latter.

Critical weaknesses

W1 - The alignment baselines do not provide a fully controlled comparison against AEF.

  1. Most baseline methods receive a single satellite image as input, from which some of the evaluated concepts may not be recoverable, while AEF incorporates multimodal Earth-observation information.
  2. AEF is aligned using captions generated from the same auxiliary variable groups underlying most of the in-distribution evaluation, putting it at an inherent advantage over pretrained baselines that were not optimized for these variables.
  3. AEF is evaluated using mixed and separate captions, but not against the same alignment strategy applied to another Earth embedding.

Therefore, we cannot reasonably infer whether the alignment gains observed in Tables 3 and 4 result from a better initial embedding, a better alignment strategy, closer correspondence between training and the evaluation protocol, or a combination of these factors.

W2 - What does language alignment buy?

  1. Training captions are constructed from auxiliary variables, most of which are subsequently used to evaluate alignment. These variables could in principle be aligned with the Earth embeddings directly, without passing through a textual embedding space. Figure 2 suggests that the introduction of language space may hinder some concepts: several bioclimatic variables are strongly encoded by AEF but poorly retrieved after alignment. This creates a cost to textual alignment that should ideally be justified by capabilities unavailable to direct auxiliary-variable alignment.
  2. Language should, in principle, provide stronger generalization to unseen concepts and enable open-vocabulary querying. Both experiments are present in the paper, but the evidence would benefit from further clarification:
    1. For OOD evaluation, most evaluated concepts are present during training. Of the two held-out S2BMS data groups, UKCEH is another land-cover dataset and may therefore be semantically close to the in-distribution CORINE variables, while PM2.5 consists of a single variable for which zero-shot GeoCLIP outperforms the proposed alignment. Furthermore, no OOD auxiliary groups are considered for SatBird. The results would be strengthened by considering a larger and more diverse set of OOD variables.
    2. Figure 3a shows that concept-caption similarities can recover relationships obtained from auxiliary data within the same geographic domain. It remains unclear whether these relationships would transfer to another region without access to the auxiliary data used to construct the alignment.
    3. The open-vocabulary experiment considers habitat descriptions constructed from in-distribution concepts that already achieved good retrieval performance. Therefore, it does not establish whether the proposed alignment can meaningfully answer OOD open-vocabulary queries.

Language alignment provides additional convenience and interpretability, which is itself relevant, but evidence of stronger OOD generalization would provide a significant additional motivation for introducing the textual space.

Questions for authors

Q1 - How would AEF alignment behave with a standard InfoNCE loss (one positive per data point)? The ablations vary sigma between 0.1 and 2.0 but never include the hard-contrastive case (sigma → 0) that the paper explicitly compares against.

Q2 - How would another embedding model (e.g., Tessera or SatCLIP, already computed for Table 1) perform with the proposed alignment strategy? This would help separate the contribution of the alignment method from that of AlphaEarth and support the claim that the framework applies to arbitrary embedding spaces.

Q3(clarification) - UKCEH, used for OOD evaluation, represents the same broad type of information as CORINE (land cover). How much do its classes overlap semantically with the CORINE classes used in the training captions, and does retrieval performance differ for UKCEH classes without a close CORINE counterpart?

Q4(clarification) - Why are GeoCLIP and CLIP absent from Table 4? GeoCLIP in particular is the only coordinate-based alignment baseline and achieved the best PM2.5 retrieval score in Table 3.

1 comment
AuthorsOpenReview ↗

Rebuttal JCEE

Dear reviewer JCEE,

Thank you very much for your detailed review of our paper. We are happy that you found our paper to be a valuable contribution to the field, and we address your questions below:

Q1 InfoNCE loss comparison

We provide the requested comparisons. InfoNCE (sigma=0) results are now added to Figures S4-S7 (and mentioned in Experiments). This shows that this hard loss function in fact performs well, with the best hard loss configuration matching the best soft loss configuration (average score of 0.45 pm 0.00 for both). This is caused by a much better performance on the human footprint variables, which compensates for a worse performance on the bioclimatic and land cover variables. We have updated Tables 3 and S13, and have changed the “Alignment” paragraph in the Results section accordingly, to:

“Retrieval performance was best when AlphaEarth was projected to the text space on the larger, unlabelled dataset, using a large batch size of 512 (Table S13, Figs S4, S5). This is likely caused by the increased likelihood of encountering data points with similar environmental conditions for larger batch sizes, while the smaller, labelled data size performed best with a smaller batch size. Notably, this model achieved equal performance (maximum average score of 0.45 pm 0.00) for both sigma=0 and sigma=0.5, where sigma=0 models scored best for human footprint variables, and sigma>0 scored best for bioclimatic and land cover variables. Hence, a soft loss function that weighs positive pairs by their similarity improves the alignment of some, but not all, concepts.”

Thank you for suggesting this; this is certainly an important experiment to include.

Q2 Alignment of another embedding model

Thank you for this suggestion. We have now also trained a Tessera alignment model, results added to Table 3, using the S2BMS labelled configuration with a full hyperparameter sweep (Figure S8). The results in Table 3 confirm that the alignment framework is not specific to AlphaEarth: Tessera alignment achieves comparable performance, except for Human Footprint retrieval where it underperforms, demonstrating that the proposed method generalises to other embedding spaces. We have updated the text in experiments, results, and the appendix to reflect these new experiments.

Q3 Land cover concepts overlap

Yes, UKCEH and CORINE both represent land cover classes. They vary slightly in their semantic descriptions, but also in how land cover is mapped (i.e., the underlying auxiliary data values). We have now added a new analysis in Figure S9 that compares the retrieval scores and the AEF encoding strength between UKCEH and CORINE, by comparing the UKCEH classes to their equivalent in CORINE, for each of the 3 CORINE levels. Correlation values are positive but none are significant, though the number of data points is arguably too low to draw strong conclusions from.

Q4 Missing GeoCLIP and CLIP baselines in Table 4

Thank you for highlighting this: these results were accidentally left out in the original submission and are now restored in Table 4. Across all metrics in Table 4, neither surpasses the AlphaEarth model.

Reviewer_7DBiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to improve the interpretability of abstract multimodal embeddings through post-hoc language alignment. The authors focus on Earth embeddings used for species distribution modelling, where strong downstream performance is obtained from latent multimodal representations that are difficult to interpret directly. To address this issue, the paper proposes to leverage sample-wise linked auxiliary data to generate partial text captions through predefined captioning functions. A projection network is then trained to align the multimodal embedding space with a pretrained text embedding space using a soft contrastive objective, where pairwise targets are determined by similarity in the auxiliary-variable space rather than standard one-hot positives. The authors further introduce a rank-based concept retrieval metric to evaluate whether language queries recover the corresponding auxiliary concepts.

Critical strengths
  1. The paper addresses a practically meaningful interpretability problem for abstract multimodal embeddings. The idea of leveraging linked auxiliary data to enable post-hoc language interaction with otherwise opaque embeddings is relevant, especially for domains such as Earth observation where the original multimodal representation is difficult to inspect directly.
  2. The overall framework is clearly structured and reasonably complete. The paper connects auxiliary-data-based caption generation, soft contrastive alignment, and a rank-based concept retrieval evaluation into a coherent pipeline, making the proposed setting and evaluation procedure relatively easy to follow.
  3. The experimental study is fairly extensive. The authors evaluate the framework on two species distribution modelling datasets, compare multiple Earth/vision-language baselines, and include ablations on factors such as projection strategy, captioning strategy, dataset size, batch size, and the soft contrastive parameter. The additional analysis on concept retrieval and species–habitat associations also provides some evidence for the intended interpretability use case.
Critical weaknesses
  1. The motivation may overstate the lack of supervision addressed by the proposed framework. While the paper emphasizes that captioned data are unavailable for multimodal embeddings, the proposed method still assumes that every embedding can be linked to rich, interpretable auxiliary variables at the sample level. These auxiliary variables are then used not only to generate captions but also to construct the soft contrastive targets. Therefore, the framework does not eliminate the need for paired supervision, but rather replaces paired text annotations with paired structured auxiliary annotations.
  2. The evaluation may be partially circular and does not fully disentangle language alignment from recovery of auxiliary-variable structure. The auxiliary variables are used to generate the training captions and, more importantly, to construct the soft contrastive targets through distances in the auxiliary-variable space. The proposed rank-based retrieval metric is then also evaluated against rankings derived from these auxiliary variables. As a result, strong retrieval performance may partly indicate that the projector preserves or recovers the structure of the auxiliary variables, rather than demonstrating that the embedding space has acquired general language semantics.
  3. The methodological novelty appears limited. The main alignment mechanism is a relatively standard projection-based vision/language alignment framework, where a small MLP maps pretrained multimodal embeddings into a pretrained text embedding space. The key modification is the use of a soft contrastive objective, but the paper itself notes that closely related soft contrastive formulations have already been explored in prior work.
Questions for authors

See the weaknesses above, especially point 1.

1 comment
AuthorsOpenReview ↗

Rebuttal 7DBi

Dear reviewer 7DBi,

Thank you very much for your detailed review. We are happy that you found our paper to be meaningful, clearly structured and extensive. We address your questions/raised weaknesses below:

W1. Overstating lack of supervision

We agree that our framework requires paired auxiliary variable supervision during training, and, respectfully, we did not claim that our framework works without supervision. Rather, we claim (in the introduction, lines 82-85 and 96-97, and the conclusion, line 489-491) that abstract embeddings cannot be directly captioned (because they are not interpretable as images are), which the use of auxiliary data alleviates. This is particularly relevant for Earth observation data, where many auxiliary data layers are publicly available and can be used to caption features that are not possible to identify/annotate from images alone (such as climate variables). At inference, no auxiliary variables are required; the aligned embedding space enables open-vocabulary concept retrieval freely.

W2. Evaluation of language generalisation

We acknowledge that auxiliary variable structure can be recoverable from the alignment, however this is by design and not a shortcoming: we specifically claim domain-specific language alignment, rather than general alignment. Critically, we evaluate retrieval on out-of-distribution (OOD) concepts that are absent from both training captions and auxiliary variables. Successful OOD retrieval cannot be explained by auxiliary-variable structure recovery alone and provides direct evidence of semantic generalisation.

Further, we disagree that strong retrieval performance of training (= in-distribution) concepts is due to only auxiliary-variable recovery and not language semantic understanding. During training we augment captions by using many linguistic and content variations (e.g., “location with predominantly forest”, “satellite image with mostly forest cover”, “Area with little pasture fields”), while in the evaluation stage the concept caption retrieval must bridge the gap for semantically related but linguistically distinct query concepts (e.g., “Forested area”). This generalisation requires semantic understanding, even if scoped to environmental concepts rather than general language.

Critically, we evaluate retrieval on out-of-distribution concepts that are absent from both training captions and auxiliary variables. Successful OOD retrieval cannot be explained by auxiliary-variable structure recovery alone and provides direct evidence of semantic generalisation.

W3. Limited methodological novelty.

We agree that the individual components of our method are not novel in isolation, but are based on existing work, which we cite in the paper as you mention. Instead, our contribution is the combination of these methods (captioning framework, alignment of pre-computed embeddings, concept retrieval evaluation) and their adoption for the use case of language-aligning pre-computed embeddings. This formulation is especially relevant for the geospatial domain, where such image/embeddings-text annotations are scarce for environmental drivers, and pre-computed (possibly closed-source) embeddings are becoming the new paradigm of using geospatial foundation models.

Reviewer_PATaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper mainly investigates how to map pre-trained embeddings into the language embedding space via interpretable auxiliary variables when natural language descriptions cannot be written for abstract multimodal embeddings, so as to enable concept querying and explanations for downstream predictions. The main contributions of this paper are as follows:

  1. Construct multiple partial captions using auxiliary contextual variables.
  2. Learn embedding-language alignment with a soft contrastive loss based on distances over auxiliary variables.
  3. Evaluate alignment quality through a rank-based concept retrieval metric grounded in auxiliary data rather than generated captions.
Critical strengths

This paper addresses the challenge of interpreting pre-trained multimodal embeddings that lack direct textual annotations. It proposes a framework that automatically generates partial captions from linked auxiliary data and achieves language alignment through soft contrastive learning. The framework does not require access to the internal architecture of the original multimodal model or manually annotated embedding–text pairs, making it practically valuable for foundation models that are closed-source or provide only precomputed embeddings.

Critical weaknesses

The paper primarily evaluates interpretability through environmental concept retrieval performance and correlations between concept similarity scores and species occurrence rates. However, the ability to retrieve environmental concepts from embeddings does not necessarily imply that the downstream prediction model actually relies on these concepts when making predictions. Since the language alignment module and the species prediction module are trained separately, the paper lacks experiments that directly establish the relationship between them, such as concept interventions, concept ablations, or explanation fidelity evaluations. Therefore, the current results mainly demonstrate the semantic retrievability of embeddings, rather than providing sufficient evidence to support the paper's central claim of explainable predictions.

First, the paper compares an alignment model trained with target-domain auxiliary variables against pre-trained vision-language models that do not receive equivalent auxiliary-variable supervision. Consequently, it is difficult to determine whether the performance improvements arise from the proposed soft contrastive loss or from the additional domain-specific supervision. Moreover, the paper lacks key baselines, such as standard InfoNCE and direct prediction of auxiliary variables from embeddings, making it difficult to establish the necessity of the proposed method. Second, Appendix A.2 indicates that the authors select environmental concepts for language alignment evaluation based on the linear probe R^2 scores measured on the test set. This makes the selection of evaluation concepts dependent on test-set information, compromising the independence of the evaluation protocol and the reliability of the results. Together, these issues substantially weaken the empirical evidence supporting the effectiveness and comparative advantages of the proposed method.

Questions for authors

Q1: The paper evaluates interpretability through concept retrieval and correlations between species occurrence rates and environmental concepts. However, the language alignment network and the species prediction network are trained independently. Could the authors provide further clarification or experimental evidence demonstrating that the retrieved environmental concepts are indeed features relied upon by the downstream prediction model, rather than merely environmental attributes correlated with its predictions? For example, are there any concept ablation, intervention, or explanation fidelity analyses that support this claim?

Q2: Could the authors provide comparisons with standard InfoNCE and direct auxiliary-variable prediction methods (e.g., Ridge Regression or MLP Regression), using the same AlphaEarth embeddings and auxiliary-data supervision? Since the proposed soft contrastive loss is a central methodological contribution, these comparisons are essential for determining whether it genuinely outperforms simpler alternatives.

Q3: According to Appendix A.2, the authors select concepts for language alignment evaluation based on linear probe R^2 scores measured on the test set. Could the authors clarify whether this selection procedure also affects the validation metric, model selection, and final test results? If so, could they provide concept retrieval results using a concept set determined exclusively from the training or validation data, to rule out potential influences of test-set information on the final evaluation?

Q4: Appendix A.4 presents several ablation studies, but many of these experiments primarily investigate hyperparameter sensitivity (e.g., batch size and Gaussian bandwidth) or compare different model configurations. Although these analyses are useful, they do not fully isolate the contribution of the proposed partial caption augmentation strategy. Could the authors provide a controlled ablation comparing the current approach with a simpler alternative that uses a single fixed caption per sample, while keeping the other training settings unchanged? Such a comparison would help clarify whether the proposed caption augmentation strategy is essential to the observed performance improvements.

2 comments
AuthorsOpenReview ↗

Rebuttal PATa

Dear reviewer PATa,

Thank you very much for your careful review and detailed suggestions for our paper. We are happy you found the paper to be valuable, and we address your specific points and questions point-by-point in the below:

Q1: Are retrieved environmental concepts used by the downstream prediction model?

Thank you for this great question. Indeed, the prediction and alignment modules are independent, and we cannot directly link the prediction results of Tables 1/2 to the alignment results of Tables 3/4. We do not claim this link, but instead motivate our work as enabling explainable systems grounded in embedding interpretability: through language alignment, the embedding space is structured such that environmentally meaningful concepts can be retrieved via open-vocabulary queries. This is why we subsequently conduct a task with the goal to retrieve locations with high species occurrence rates from captions that describe species habitat preferences (Figure 3). In this case, the retrieval directly depends on the concepts that are queried. This is a hard task, so the performance (quantified as the correlation between embedding similarity and occurrence rates) is variable across species, and is higher for predicted occurrence rates than true occurrence rates.

We did not perform concept ablations, because we do not know the true causal links between individual concepts and species occurrence. Our best estimate is that provided from the linear regression model, but this is outperformed by the embeddings. A more complex approach would be to identify the concepts in the embedding space and ablate them, which is a great suggestion but out of scope for this rebuttal.

Q2 Comparison with standard InfoNCE and direct auxiliary-variable prediction methods

We provide the requested comparisons. InfoNCE (σ=0) results are now added to Figures S4-S7 (and mentioned in Experiments). This shows that this hard loss function in fact performs well, with the best hard loss configuration matching the best soft loss configuration (average score of 0.45 pm 0.00 for both). This is caused by a much better performance on the human footprint variables, which compensates for a worse performance on the bioclimatic and land cover variables. We have updated Tables 3 and S13, and have changed the “Alignment” paragraph in the Results section accordingly, to:

“Retrieval performance was best when AlphaEarth was projected to the text space on the larger, unlabelled dataset, using a large batch size of 512 (Table S13, Figs S4, S5). This is likely caused by the increased likelihood of encountering data points with similar environmental conditions for larger batch sizes, while the smaller, labelled data size performed best with a smaller batch size. Notably, this model achieved equal performance (maximum average score of 0.45 pm 0.00) for both sigma=0 and sigma=0.5, where sigma=0 models scored best for human footprint variables, and sigma>0 scored best for bioclimatic and land cover variables. Hence, a soft loss function that weighs positive pairs by their similarity improves the alignment of some, but not all, concepts.”

Thank you for suggesting this; this is certainly an important experiment to include. We had already included models using auxiliary variables directly as input features, serving as a prediction baseline from inputs with known physical meaning; these results are present in Tables 1, 2, S3, and S9 (labelled “Linear regression” and “MLP”).

Q3. Concept selection procedure

This is a good point. We use test set performance to select auxiliary variables to assess, because the validation set was used to optimise the Ridge regression regularisation parameter. We have now tested what the results had been if we used the validation split: For S2BMS: 53/82 variables had R^2 > 0.5 for both the validation and the test set, 4/82 had R^2 > 0.5 for one, but not the other, and 25/82 variables had R^2 <= 0.5 for both. For SatBird: 31/36 variables had R^2 > 0.5 for both the validation and the test set, 2/36 had R^2 > 0.5 for one, but not the other, and 3/36 variables had R^2 <= 0.5 for both.

Because only 4/82 and 2/36 variables would have been different, respectively, we believe the effect of which dataset is used for selecting the concepts is negligible. None of the splits are ideal, as they all theoretically present a bias in some way. However this is a sharp point, we thank you for raising it, and we have now added this in appendix A.4

AuthorsOpenReview ↗

Rebuttal PATa Q4

Q4 Ablation of single fixed caption per sample

Thank you for this suggestion. We now conducted an ablation using a single simplified caption template for all locations, containing only the top land cover class, mean annual temperature, and precipitation: “location with <top_1_landcover> and influenced by <annual_mean_temp> and <annual_precipitation>.” Full caption augmentation outperforms this baseline by 0.07 in average retrieval performance (Table S13, AEF unlabelled, versus AEF proj., simple). We further compared mixed captions, where each caption covered multiple environmental aspects, against separate captions focused on a single aspect. Separate captions yielded better performance (Tables 3 and S13). Together, these ablations confirm that both caption volume and aspect specificity contribute meaningfully to representation quality for retrieval tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.