acceptodds
Under review as a conference paper at ICLR 2027

EarthAlign: Learning Language-Aligned Representations of Earth System Data

Abstract

Interpreting Earth system conditions is central to understanding environmental risks and informing decisions across weather- and climate-sensitive sectors. Existing multimodal weather systems primarily target task-specific prediction, generation, or question answering, while general-purpose alignment between Earth system data and language remains underexplored due to the limited availability of diverse language supervision grounded in meteorological conditions. To fill this gap, we introduce EarthAlign, a large-scale dataset with broad geographic coverage spanning 2000-2024 that combines 10 expert-curated natural event and disaster data sources with climatology-derived normal-weather samples. Each sample includes temporal and geospatial information and four grounded caption styles aligned with 63-variable ERA5 meteorological data. Building on EarthAlign, we introduce MeteoLIP, a dual-encoder framework for learning aligned meteorology-language representations. MeteoLIP incorporates MetTok, which uses a geolocation- and time-conditioned query to aggregate variable-specific patch embeddings into a compact sequence of meteorology tokens. To comprehensively assess Earth system multimodal representations, we introduce three downstream tasks: cross-modal retrieval, weather-condition classification, and physical variable estimation. Our experiments show that MeteoLIP improves upon the corresponding CLIP and SigLIP baselines across these tasks. Further analyses highlight the importance of meteorological input, temporal and geospatial conditioning, and multi-style supervision for meteorology-language alignment. Thus, EarthAlign and MeteoLIP provide a foundation for learning and evaluating language-aligned representations of Earth system data. We will publicly release the dataset & code.

Primary area:
datasets and benchmarks
Keywords:
Earth system, Weather and climate, Meteorology-language dataset, Multimodal alignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_46Z6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EarthAlign, a dataset of 553,738 samples pairing 63-variable ERA5 data with four styles of natural-language descriptions. It also proposes MeteoLIP, a framework for aligning meteorological and textual representations, with MetTok incorporating temporal and geographic information into variable aggregation. Experiments on cross-modal retrieval, weather-condition classification, and physical variable estimation show improvements over the corresponding CLIP and SigLIP baselines.

Critical strengths

EarthAlign addresses a meaningful need for large-scale meteorology–language data. Its combination of diverse event records, normal-weather samples, and multiple caption styles provides a useful resource for future research.

Critical weaknesses
  1. The differences between MetTok and ClimaX’s variable aggregation design need clarification.

  2. The WeatherQA experiment lacks a controlled comparison to isolate the benefit of EarthAlign pretraining for downstream transfer.

Questions for authors
  1. ClimaX [1] already encodes variables separately and aggregates their embeddings through cross-attention with a learnable query. Could the authors clarify how MetTok differs from this design and what additional benefits it provides?

  2. The authors have analyzed retrieval performance after removing temporal information from the captions. How does the model perform when candidate descriptions refer to similar times and nearby locations, so that distinguishing them requires meteorological information? Could the authors provide an evaluation under this setting to further clarify the contribution of meteorological content to retrieval?

  3. Appendix F.4 reports the transfer results of Meteo-CLIP on WeatherQA. Could the authors compare it with an encoder without EarthAlign pretraining while keeping the downstream language model and training settings unchanged? This would help establish the transfer benefits of EarthAlign pretraining.

[1] Nguyen, T., Brandstetter, J., Kapoor, A., et al. ClimaX: A Foundation Model for Weather and Climate. arXiv preprint arXiv:2301.10343, 2023.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 46Z6 (1/2)

We thank the reviewer for the constructive review and for recognizing the need for large-scale meteorology-language data that EarthAlign addresses. Below we address (1) the differences between MetTok and ClimaX (W1 & Q1; Sec. 4.1), (2) retrieval with candidates from similar times and nearby locations (Q2; Appendix F.4), and (3) a controlled WeatherQA comparison (W2 & Q3; Appendix F.7). In addition, we have uploaded a revised manuscript with all changes highlighted in gold.

Weakness 1 & Question 1: Differences between MetTok and ClimaX's variable aggregation

We agree that MetTok shares some similarities with ClimaX's variable aggregation. However, since ClimaX is designed for weather forecasting, MetTok differs from it in three aspects that suit meteorology-language alignment:

  1. Lead time vs. absolute time. ClimaX encodes the lead time Δt, which is the forecast horizon relative to the input time. In contrast, MetTok encodes the absolute time of the snapshot (year, day of year, and hour of day), which provides temporal context such as the diurnal and seasonal cycles.
  2. Implicit vs. explicit geolocation. In ClimaX, all samples of a dataset cover the same region (e.g., the entire globe or a specific region such as North America), so the positional embedding of each grid position also implicitly encodes its geolocation. However, EarthAlign samples cover regions of varying location and extent, standardized to a 56×56 grid, so the same grid position corresponds to different geographic locations across samples. To handle this, MetTok explicitly encodes the latitude and longitude of each patch, while the ViT's learnable positional embedding still encodes the patch's position within the grid.
  3. Injection of time and location information. ClimaX adds its lead-time and positional embeddings to the tokens after variable aggregation, so its aggregation query is shared across positions and samples. Since meteorological conditions vary with geographic location and temporal context, MetTok instead adds each patch's temporal-geospatial embedding to its aggregation query before cross-attention to obtain time- and location-aware queries.

These aforementioned differences arise from our different objective and inputs. We designed MetTok for meteorology-language alignment rather than forecasting, with regional inputs that vary in location and extent. We have now credited ClimaX and briefly stated these differences in Sec. 4.1 of the updated manuscript.

Question 2: Retrieval when candidates refer to similar times and nearby locations.

Based on your suggestion, we have now added an additional retrieval evaluation in Appendix F.4 in the revised manuscript. The evaluation is built on the time-removed captions, and thus the time cue in the descriptions is removed. For each query, the candidate pool contains only test samples within ±72h of the query (i.e. one synoptic period) whose ERA5 crop centers lie within 50 km of the query's, so all candidates refer to a nearby period and location. Here we use the improved time-removed retrieval benchmark of the revised manuscript (Appendix F.3). Following Reviewer iuG2's suggestion, we now identify for every sample all captions that correctly describe it, i.e. captions whose information is entirely contained in the sample's metadata record, and vice versa. These matches are obtained with an LLM judge followed by rule-based consistency checks, so that equally valid matches are no longer counted as errors. Within a pool, all valid matches of a query count as a single candidate, and only queries with at least four incorrect candidates are kept. This yields 1,335 queries with 8.3 candidates per pool on average. In addition, we report random guessing as a reference.

ModelM2T R@1T2M R@1
Random guess20.620.0
Metadata only (time + location)28.46 ± 1.0935.01 ± 0.71
Meteo-CLIP36.20 ± 0.5039.85 ± 0.40

In this setting, Meteo-CLIP is 15.6 (M2T) and 19.9 (T2M) points above random guessing, whereas the metadata-only model is only 7.9 and 15.0 points above random guessing. Meteo-CLIP roughly doubles the gain over random guessing in M2T (15.6 vs. 7.9) and increases it by about one third in T2M (19.9 vs. 15.0). Since the candidates come from similar times and nearby locations and their captions contain no time information, this additional gain further shows the importance of the meteorological data for distinguishing them.

AuthorsOpenReview ↗

Response to Reviewer 46Z6 (2/2)

Weakness 2 & Question 3: Benefit of EarthAlign pretraining on WeatherQA

Following your suggestion, we have now added this comparison to Appendix F.7. The baseline uses the same encoder architecture as Meteo-CLIP but without EarthAlign pretraining. The baseline ViT is initialized from OpenAI CLIP and its MetTok is randomly initialized, while the downstream language model and its training settings are kept the same. Since the baseline encoder has never been trained on meteorological data, we also fine-tune the ViT and MetTok end-to-end on WeatherQA, whereas Meteo-CLIP remains frozen. This should favor the baseline to some extent.

EncoderEncoder on WeatherQAArea Affected (%)Concerning (%)
Without EarthAlign pretrainingfine-tuned end-to-end34.91 ± 0.5748.86 ± 1.72
Meteo-CLIP (EarthAlign-pretrained)frozen49.22 ± 4.4551.73 ± 2.41

Despite being frozen, Meteo-CLIP achieves higher accuracy than the baseline on Area Affected (+14.31 points) and Concerning (+2.87 points). This supports the transfer benefit of EarthAlign pretraining to downstream severe weather reasoning.

Thank you for your detailed response. My concerns have been addressed, and I will maintain my current positive rating.

Thank you for your thoughtful review and for confirming that our clarifications and additional results have addressed your concerns. We also appreciate your constructive suggestions, which have helped us strengthen the work.

Reviewer_iuG2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces EarthAlign, an earth-data domain CLIP-style dataset of 553,738 samples (2000–2024) pairing a 63-variable ERA5 snapshot (standardized to a 56×56 patch over at least a 14°×14° box) with timestamp, lat/lon bounding box, and four Gemini-3.1-Pro-generated captions with different styles.

Based on the data, this paper proposes MeteoLIP, a CLIP-style dual encoder whose meteorology tokenizer (MetTok) patchifies each variable separately, adds variable-identity embeddings, and aggregates variables per patch via cross-attention from a query conditioned on time and geolocation, yielding 196 tokens for efficiency. Training samples one caption style per example. Three tasks are defined with a year-based split (2011–2013 held out): bidirectional retrieval, 11-class weather-condition classification (text prototypes or linear probe), and cyclone wind/pressure estimation via value-conditioned text prototypes or a linear probe.

The experimental results demonstrate effectiveness. Meteo-SigLIP reaches 66.1/66.8 R@1 vs 32.1/30.2 for a SigLIP baseline trained with one caption style and no time/geo input; text-prototype classification F1 50.2 vs 21.6; linear-probe F1 80.8 vs 76.3; cyclone wind R² 71.9 vs 70.5 (linear probe).

Critical strengths
  1. A large, carefully built, and reproducibly documented resource in an underserved domain. Existing meteorology–language datasets are task-specific QA sets; EarthAlign provides half a million time- and geo-referenced ERA5–caption pairs with global coverage, a documented curation pipeline, a published captioning prompt, climatology-based negatives, and a promised release. This is the main reason I lean toward acceptance: the dataset is useful independently of the MeteoLIP results.

  2. Honest, multi-seed analysis that exposes the dataset's own failure modes. Rather than hiding it, the paper quantifies the metadata shortcut (Tables 5–6), shows weak cross-source generalization (Table 7), compares against physical baselines (Table 15), and reports that scaling to ViT-L overfits. This makes the paper a credible benchmark paper even where the headline numbers are flattering.

Critical weaknesses
  1. The headline alignment results are confounded, and the retrieval task is dominated by metadata shortcuts. A model that sees only geolocation and time, with no meteorological input at all, reaches 50.4/58.5 R@1 against 58.8/60.1 for the full model (Table 5), and removing time from captions collapses full-model R@1 to 15.4 (Table 6). So the 20→59 jump in Table 1 mostly reflects matching dates and places that appear in the captions and are fed to the encoder, not meteorology–language alignment. The baselines, meanwhile, get neither time/geo inputs nor four-style training; when CLIP is given four styles it reaches 31.6 (Table 3), which is essentially the same as MetTok with meteorology but without time/geo conditioning (32.0, Table 5). In other words, the variable-wise tokenizer itself contributes almost nothing over multi-channel patchification, and the linear-probe gains on classification and cyclone intensity are 1–4 points. The paper's framing ("MeteoLIP improves upon CLIP and SigLIP", MetTok as a contribution) is therefore not supported by a controlled comparison, and the primary benchmark metric needs to be redefined (time- and location-stripped captions, or event-level pools) before it measures what the paper says it measures.

  2. Pairing and label validity are under-examined for a benchmark whose value rests on them. Many records span days to a full year (EM-DAT, FraDE droughts), yet each is paired with one uniformly sampled ERA5 hour, with no check that the snapshot exhibits the phenomenon. IBTrACS and similar sources yield thousands of near-identical captions ("a tropical cyclone was active in ... with maximum sustained winds of 20 knots and a central pressure of 1000 mbar", Fig. 7), which makes R@1 on a 63k pool partly arbitrary and inflates apparent errors or successes. The 93→11 class mapping and the treatment of geophysical events (earthquakes, volcanic ash in Fig. 5) in a "weather condition" task are undocumented, and the caption audit rests on a single annotator with no agreement statistics. These issues do not invalidate the dataset, but they need to be measured and reported for the benchmark numbers to be trusted.

Questions for authors
  1. Please report Table 1 under a controlled protocol: baselines with the same time/geo inputs and four-style training, and retrieval with time removed and with both time and location removed from captions. What is the full model's R@1 in the latter setting, and how does it compare to the 4-style CLIP baseline?
  2. How many test captions are exact or near duplicates (e.g., consecutive IBTrACS snapshots of one storm, PIKART snapshots)? Would an event-level retrieval evaluation (one query per event, candidates deduplicated) change the conclusions?
  3. For records with long time windows, what fraction of sampled ERA5 hours plausibly exhibit the labeled phenomenon? Even a small manual or rule-based check (e.g., for cyclones, wind/pressure in the patch; for heavy rain, accumulated precipitation) would help.
7 comments
AuthorsOpenReview ↗

Response to Reviewer iuG2 (1/4)

We thank the reviewer for the detailed review and for recognizing EarthAlign as a carefully documented resource and our analyses of its failure modes. Below we address (1) the 93->11 class mapping (W2; Appendix E), (2) the phenomenon support of sampled ERA5 hours (W2 & Q3; Appendix D.2), (3) duplicate captions and an evaluation with all valid matches (W2 & Q2; Appendix F.3 and updated Table 6), (4) the controlled comparison and retrieval without time and location (W1 & Q1; Appendix F.4), and (5) caption audit agreement (W2; revised Table 11 and Appendix C.2). In addition, we have uploaded a revised manuscript with all changes highlighted in gold.

Weakness 2: 93 -> 11 mapping

The 11 classes are derived from the phenomenon taxonomy in Figure 1. Both the taxonomy and the final class list were reviewed by a climate-science domain expert. The weather-condition classification task includes only weather conditions that can be reasonably characterized from meteorological variables. To avoid ambiguity in the labels, we also exclude conditions that commonly occur as part of another class’s events. For instance, heavy rainfall is excluded because it frequently accompanies atmospheric rivers and cyclones. The record-source labels that meet these criteria are then mapped to the 11 classes following the expert-reviewed taxonomy. We have now documented these criteria and the full mapping from record-source labels to the 11 classes in Appendix E of the revised manuscript.

Weakness 2 & Question 3: Phenomenon support of sampled ERA5 hours

We have now added an analysis in Appendix D.2 to estimate the fraction of sampled ERA5 hours that plausibly exhibit the labelled phenomenon. We apply a rule-based check to 9,863 samples from the test years (2011–2013) covering 10 phenomena, using the criteria and references in Table 14. The checked samples consist of 8,140 test records that span more than one day and 1,723 records spanning ≤ 1 d, stratified-sampled by 10 phenomena and 2 duration classes (≤ 6 h and 6–24 h). For each sample, we test whether the sampled hour meets the phenomenon's rule at the ERA5 grid point nearest the center of the reported area. For example, following your suggestion, floods require extreme accumulated precipitation over 24 h or 7 days (above the local 95th percentile). The tables below report the fraction of samples meeting the rule (Supported) by duration and by phenomenon. To verify that this support reflects the event rather than the local climate or season, we also report how often the same rule is met at the same location and calendar hour in other years (2000–2024), averaged over records (By chance).

DurationnSupported (%)By chance (%)
≤ 1 d1,72343.58.0
1–7 d4,11949.810.4
> 7 d4,02133.915.4
Phenomenon (> 1 d)nSupported (%)By chance (%)
Flood2,68627.08.7
Drought2,08847.221.4
Winter weather1,22965.017.7
Wildfire52020.65.8
Cyclone46251.97.5
Heat wave41070.09.6
Cold wave30838.68.7
High wind24434.04.5
Landslide13737.26.5
Frost5630.42.5

In records spanning more than one day, 41.9% of sampled hours plausibly exhibit the labelled phenomenon (Supported), which is 3.2x the By chance rate (12.9%), and every duration and phenomenon has a higher Supported rate than By chance. The sampled hours therefore reflect the recorded events much more often than the local climate alone would explain. However, an unsupported sample does not imply that the record or its caption is wrong. The check is designed to be specific: it tests a single ERA5 grid point at a single hour against a high local threshold, which keeps By chance rates low (2.5-21.4%) but also misses valid events for two main reasons. First, spatial mismatch: ERA5's 0.25° grid does not resolve localized phenomena such as convective flash floods, and an event may affect only part of the reported area rather than its center. This partly explains the low rates for floods (27.0%) and wildfires (20.6%), the latter also depending on fuels and ignition rather than weather alone. Second, temporal mismatch: the strongest meteorological forcing often occurs early in the reported window, e.g. heavy rain before a multi-day flood, whereas the snapshot is sampled uniformly within the window. In line with this, support does not drop from ≤ 1 d to 1-7 d records (43.5% vs. 49.8%) but declines beyond 7 d (33.9%). This temporal mismatch reflects the single-snapshot limitation we acknowledge in Appendix G. As EarthAlign retains each record's reported interval, incorporating multiple snapshots across the event window is a natural extension.

AuthorsOpenReview ↗

Response to Reviewer iuG2 (2/4)

Weakness 2 & Question 2: Duplicate captions and an evaluation with all valid matches

We have now quantified the duplicate captions and added an evaluation with all valid matches in Appendix F.3 of the revised manuscript, which also updates Table 6 in the main paper.

In the original 63,343 test captions, the described event or phenomenon, its date and time, and its available location information make every caption distinct and distinguishable from the others. After time is removed, only 53,020 distinct captions remain, and 21.5% of the samples share their exact caption with at least one other sample. Many of the remaining captions are near duplicates that state the same facts in different words, such as captions of consecutive IBTrACS snapshots that share the same cyclone type, minimum central pressure, and/or maximum sustained wind. In addition, please note that the IBTrACS examples in Fig. 7 are time-removed captions. For instance, one of them originally reads "A tropical cyclone was active in the Western Australia sub-basin of the South Indian Ocean on 27 February 2011 at 00 UTC, ...", while another one occurred at 06 UTC on 21 February 2013 (we have revised the caption of Fig. 7).

Rather than using one query per event, we now identify for every sample all captions that are valid matches for it, which we define as captions whose information is entirely contained in the sample's time-removed metadata record. Conversely, for every caption, we identify all samples/records for which it is a valid match. This reduces the number of valid matches counted as errors. Identical captions and identical records are matched directly. In addition, every other caption-record pair becomes a candidate if its caption-embedding similarity (Qwen3-Embedding-8B) is at least 0.70 and every number in the caption, such as a wind speed, central pressure, hail size, or count, matches a value in the record. Each candidate is then decided by a two-step LLM judge (Qwen3-8B for screening, Qwen3.5-9B for the final decision) followed by rule-based consistency checks and a second pass over pairs rejected at screening. The full pipeline reaches 92.0% precision and 95.2% recall on 187 manually checked pairs with a similarity of at least 0.70. Unlike event-level pooling, this also covers near duplicates across events. For instance, "A tropical cyclone or disturbance was moving through the Western North Pacific." is a valid match for 2,515 samples from many different cyclones. On average, a sample in the time-removed retrieval benchmark has 17.8 valid captions, and only 17.0% of the samples have their own caption as the only valid match.

Candidate poolCaptionEvaluationFull MetTokMetadata onlyRel. gap
All test samplesOriginalPaired caption58.79 ± 1.4350.41 ± 2.4314.3%
Time removedPaired caption15.41 ± 0.026.57 ± 0.1857.4%
Time removedAll valid matches26.02 ± 0.1712.75 ± 0.3151.0%
Same monthOriginalPaired caption61.84 ± 2.4853.02 ± 3.4114.3%
Time removedPaired caption38.65 ± 4.0623.26 ± 2.9939.8%
Time removedAll valid matches47.27 ± 4.8629.89 ± 3.8536.8%

M2T R@1. Paired caption: only the paired caption counts as correct. All valid matches: every valid match counts as correct.

Under this improved retrieval protocol, both evaluations lead to the same observation. With time removed from the captions, Full MetTok achieves 2.35× (paired caption) and 2.04× (all valid matches) higher R@1 than the metadata-only model on all test samples, and 1.66× and 1.58× higher R@1 within same-month pools, where seasonal cues are also controlled. These results further demonstrate the importance of meteorological data for retrieval.

AuthorsOpenReview ↗

Response to Reviewer iuG2 (3/4)

Weakness 1 & Question 1: Controlled comparison and retrieval without time and location

We agree that a controlled comparison is important for disentangling the contributions. In Table 1, our intention was to compare against CLIP and SigLIP in their original formulation, i.e. a single caption per image and no metadata input, as described in the Baselines paragraph (Section 5.1). MeteoLIP is designed as a framework that enables multi-style caption training for meteorology-language alignment and the injection of time and location metadata, along with a meteorology tokenizer (MetTok). Tables 3-5 analyze the effects of caption styles, temporal and geographic conditioning, and meteorological data, and Table 6 further examines the retrieval cues from time information in captions. Since multi-style caption training and metadata injection are independent of the tokenizer, the MeteoLIP formulation can also be combined with other tokenizers or encoder architectures. We agree that a baseline with the same inputs and caption styles helps isolate how much MetTok contributes on top of this formulation, and we are currently training a 4-style CLIP baseline with the same time and location inputs as Meteo-CLIP. We will share its results once they are available. Meanwhile, without time and location conditioning, MetTok reaches 32.44 / 32.71 (M2T / T2M) R@1 (Table 5), compared to 31.64 / 31.10 for the 4-style CLIP (i.e. baseline without time and location inputs; Table 3).

Regarding the metadata shortcut, we refer to our response to Weakness 2 & Question 2. With time removed from the captions and all identified valid matches counted as correct, Full MetTok achieves 2.04× the R@1 of the metadata-only model on all test samples and 1.58× within same-month pools.

As requested, we provide below the retrieval results with both time and location removed from the captions, where only the paired caption counts as correct.

ModelAll test M2T R@1All test T2M R@1Same month M2T R@1Same month T2M R@1
Metadata only0.15 ± 0.020.09 ± 0.020.76 ± 0.210.60 ± 0.15
4-style CLIP0.58 ± 0.030.36 ± 0.012.36 ± 0.531.95 ± 0.46
Full MetTok0.69 ± 0.040.40 ± 0.012.48 ± 0.592.04 ± 0.52

In this setting, R@1 drops below 3% for all models, and Full MetTok is only slightly higher than the 4-style CLIP baseline and the metadata-only model. This is mainly because removing both time and location makes the captions generic. Many source records contain little beyond the event type, place, and date, so only 35,906 distinct captions remain for 63,343 samples, and 49.5% of the samples share their exact caption with another sample. For instance, 7,000 samples share the caption "Normal weather conditions were observed.", and IndDE has only 60 distinct captions for its 3,835 samples. Paired-caption R@1 therefore counts most equally valid matches as errors.

We therefore control for time and location by restricting the candidate pool rather than stripping the captions. In our local-pool evaluation (Appendix F.4; see also our response to Reviewer 46Z6), all candidates of a query lie within ±72 h and 50 km of it, so they share similar time and location information, while their captions keep their event details. In this setting, Full MetTok reaches 36.20 (M2T) and 39.85 (T2M) R@1, compared with 28.46 and 35.01 for the metadata-only model (random guessing: about 20%). This shows that the meteorological input contributes beyond time and location cues.

In addition, time and location are informative in their own right, as they are part of a meteorological record and of realistic queries. For instance, EarthAlign includes the record "Between 15 April 2024 and 16 April 2024, heavy rains triggered flash flooding and storm conditions in Dubai, United Arab Emirates." Without its date and place, this description would match many other heavy rain events. Class-level matching without such context is better measured by the weather-condition classification and physical-variable estimation tasks in Table 1, whose class prompts and targets do not depend on time or location. We therefore consider the time-removed evaluation with all identified valid matches, together with the same-month and local candidate pools, as the controlled retrieval evaluation, which also covers near duplicates across events (Weakness 2 & Question 2).

Reviewer_scyaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EarthAlign, a meteorology-language dataset of 553,738 samples spanning 2000–2024. It combines 489,185 records from ten natural-event and disaster sources with 64,553 normal-weather samples selected relative to local climatology. Each sample pairs four metadata-grounded caption styles with one ERA5 snapshot containing 63 variables. The paper proposes MeteoLIP, a dual-encoder framework trained with CLIP or SigLIP objectives. Its MetTok module aggregates variable-specific patch embeddings using time and geolocation conditioned queries. Evaluation holds out 2011–2013 and covers bidirectional retrieval, 11-class weather-condition classification, and cyclone wind-speed and central-pressure estimation. MeteoLIP improves over the reported CLIP/SigLIP baselines. Additional analyses examine caption supervision, metadata dependence, initialization, data scaling, source-held-out performance, and transfer to WeatherQA.

Critical strengths

1.Integrating heterogeneous event sources with climatology-based normal weather provides a useful resource for meteorology–language research. Defining normality relative to location and season is sensible, and the four caption styles expose complementary descriptions of each record. The main text also reports a stratified human audit: 421 of 440 captions pass the metadata-grounding and style checks. This supports confidence in the text construction process, while leaving physical pair validity as a separate question.

2.Table 1 reports consistent gains across retrieval, classification, and intensity estimation. For example, Meteo-SigLIP improves linear-probe classification F1 from 76.30 to 80.75. The metadata controls are particularly valuable: Meteo-CLIP achieves classification F1 of 47.91 versus 32.64 for metadata only (Table 5), and M2T R@1 of 15.41 versus 6.57 after temporal expressions are removed from captions, using all test candidates (Table 6). These results support a contribution from meteorological content beyond event identity. Reporting means and standard deviations over three training runs further strengthens the analysis.

Critical weaknesses

1.The main comparison changes tokenization, explicit temporal/geographic inputs, and caption supervision together. Table 3 appropriately controls caption count, but the main text still does not establish the benefit of MetTok over a simple patch projection given the same metadata. Four-style CLIP reaches M2T/T2M R@1 of 31.64/31.10; MetTok without temporal/geographic conditioning reaches 32.44/32.71 (Tables 3 and 5). These cross-table results suggest that much of the full system’s retrieval gain accompanies the added conditioning. Metadata only already reaches 50.41/58.49, versus 58.79/60.10 for the full model. The useful controls in Tables 5–6 establish that meteorology contributes, but do not isolate the advantage of the conditioned aggregation design itself. An information-matched baseline across all three tasks would clarify the methodological contribution.

2.The source-held-out experiment is useful and candid, but it limits how broadly the aggregate results can be interpreted. When five sources are removed together, IndDE M2T R@1 falls from 69.6 to 13.1 over 3,835 test samples, and FraDE falls from 24.0 to 10.6 over 966 samples (Table 7). AusDE and CanDE are more stable, but have only 27 and 47 test samples. The current evidence therefore supports useful alignment within the benchmark more strongly than stable transfer across heterogeneous record sources. The authors acknowledge this limitation; what remains unclear is how much of the decline reflects reporting language, event composition, geographic coverage, or lost training data.

Questions for authors

1.How does a simple multichannel patch-projection baseline perform when given the same temporal/geographic inputs, four caption styles, initialization, and training budget as MeteoLIP?

2.Can the IndDE and FraDE declines be broken down by event type and geographic coverage, with retrieval candidate pools made explicit? Comparing the full model and metadata-only control under the same source-held-out protocol would help distinguish transferable meteorological information from source-specific matching cues.

3.For records spanning time intervals, how often does a sampled ERA5 timestamp support the meteorological content of the caption? Can the authors provide a targeted audit by event duration and type?

4 comments
AuthorsOpenReview ↗

Response to Reviewer scya (1/3)

We thank the reviewer for the constructive review and for recognizing EarthAlign's climatology-based normal-weather samples, multi-style captions, and the metadata controls in Tables 5-6. Below we address (1) how often the sampled ERA5 snapshot supports the caption (Q3; Appendix D.2), (2) the breakdown of the source-held-out declines (W2 & Q2; Appendix F.5), and (3) the information-matched baseline (W1 & Q1). In addition, we have uploaded a revised manuscript with all changes highlighted in gold.

Question 3: Snapshot support by event duration and type

We have now added an analysis in Appendix D.2 to estimate how often the sampled ERA5 snapshot supports the phenomenon in its caption. We apply a rule-based check to 9,863 samples from the test years (2011-2013) covering 10 phenomena, using the criteria and references in Table 14. The checked samples consist of 8,140 test records that span more than one day and 1,723 records spanning ≤ 1 d, stratified-sampled by 10 phenomena and 2 duration classes (≤ 6 h and 6–24 h). For each sample, we test whether the sampled hour meets the phenomenon's rule at the ERA5 grid point nearest the center of the reported area, e.g., heat waves require T2m ≥ the local 90th percentile. The tables below report the fraction of samples meeting the rule (Supported) by duration and by phenomenon. To verify that this support reflects the event rather than the local climate or season, we also report how often the same rule is met at the same location and calendar hour in other years (2000–2024), averaged over records (By chance).

DurationnSupported (%)By chance (%)
≤ 1 d1,72343.58.0
1–7 d4,11949.810.4
> 7 d4,02133.915.4
Phenomenon (> 1 d)nSupported (%)By chance (%)
Flood2,68627.08.7
Drought2,08847.221.4
Winter weather1,22965.017.7
Wildfire52020.65.8
Cyclone46251.97.5
Heat wave41070.09.6
Cold wave30838.68.7
High wind24434.04.5
Landslide13737.26.5
Frost5630.42.5

For records spanning more than one day, 41.9% of sampled hours are Supported, which is 3.2x the By chance rate (12.9%), and every duration and phenomenon has a higher Supported rate than By chance. The sampled hours therefore reflect the recorded events much more often than the local climate alone would explain. However, an unsupported sample does not imply that the record or its caption is wrong. The check is designed to be specific: it tests a single ERA5 grid point at a single hour against a high local threshold, which keeps By chance rates low (2.5-21.4%) but also misses valid events for two main reasons. First, spatial mismatch: ERA5's 0.25° grid does not resolve localized phenomena such as convective flash floods, and an event may affect only part of the reported area rather than its center. This partly explains the low rates for floods (27.0%) and wildfires (20.6%), the latter also depending on fuels and ignition rather than weather alone. Second, temporal mismatch: the strongest meteorological forcing often occurs early in the reported window, e.g. heavy rain before a multi-day flood, whereas the snapshot is sampled uniformly within the window. In line with this, support does not drop from ≤ 1 d to 1-7 d records (43.5% vs. 49.8%) but declines beyond 7 d (33.9%). This temporal mismatch reflects the single-snapshot limitation we acknowledge in Appendix G. As EarthAlign retains each record's reported interval, incorporating multiple snapshots across the event window is a natural extension.

AuthorsOpenReview ↗

Response to Reviewer scya (2/3)

Weakness 2 & Question 2: Breakdown of the source-held-out declines

Based on your suggestion, we have now added a breakdown of the source-held-out experiment in Appendix F.5. As in Table 7, retrieval for each held-out source uses the original captions, and each query is ranked against all test samples of the same source, i.e. 3,835 candidates for IndDE and 966 for FraDE. The table below reports M2T R@1 by event type (types with at least 20 test samples) for Full MetTok trained on all sources and trained without the five held-out sources.

SourceEvent typenAll sourcesSource held out
IndDEFlood1,41571.85 ± 3.5212.82 ± 4.02
Extreme weather1,22271.00 ± 4.599.03 ± 1.20
Landslide77263.26 ± 5.3915.89 ± 2.83
Drought19574.02 ± 0.5923.93 ± 4.11
Forest and land fires9659.03 ± 3.0120.49 ± 3.18
Tidal surge / coastal abrasion7558.22 ± 3.858.44 ± 3.85
Earthquake3890.35 ± 8.0425.44 ± 14.49
FraDEFlooding and/or mudflows39631.57 ± 1.3417.93 ± 0.00
Drought2759.21 ± 2.001.70 ± 1.28
Ground movement19932.66 ± 1.519.72 ± 4.18
Flooding due to rising groundwater6810.78 ± 2.254.90 ± 3.40
Mechanical impacts caused by wave action2233.33 ± 9.4616.67 ± 13.12

The decline is consistent across event types, suggesting that it is not driven by a particular event composition. We also examined whether the decline comes from missing training data in the same regions, which does not appear to be the case. Even after removing the five sources, the remaining sources (e.g. EM-DAT, COOLR, IBTrACS, and normal-weather samples) still provide training samples near Indonesia and France. Within 500 km of each IndDE and FraDE test sample, the median number of such training samples is 508 and 677 respectively. Grouping the test samples by this number, the held-out score does not go up where more training data from other sources is nearby.

Training samples from the other sources within 500 km (quantile)IndDE (source held out)FraDE (source held out)
q0–q2519.67 ± 3.3313.09 ± 3.10
q25–q5013.69 ± 1.206.78 ± 1.27
q50–q7510.19 ± 3.6810.37 ± 1.50
q75–q1008.76 ± 0.9412.12 ± 1.33

However, these sources report events differently from IndDE and FraDE. IndDE captions follow a fixed template that mainly differs in the event type, regency, and date, and mostly use Indonesian province names, whereas the other sources more often use English names and coarser or finer locations. Without the date, 72.4% of the IndDE test captions are identical to another IndDE caption, so retrieval within IndDE relies substantially on its own naming conventions, which are difficult to learn from the other sources. FraDE captions are less repetitive (14.3% identical without the date), and their relative decline is smaller (24.02 to 10.59, vs. 69.60 to 13.08 for IndDE). We will additionally include the metadata-only model trained under the same source-held-out protocol in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer scya (3/3)

Weakness 1 & Question 1: Information-matched baseline

We agree that a controlled comparison is important for disentangling the contributions of tokenization, temporal and geographic inputs, and caption supervision. In Table 1, our intention was to compare against CLIP and SigLIP in their original formulation, i.e., a single caption per image and no metadata input, as described in the Baselines paragraph (Section 5.1). MeteoLIP is designed as a framework that enables multi-style caption training and the injection of time and location metadata, along with a meteorology tokenizer (MetTok). Tables 3–5 analyze the effects of caption styles, temporal and geographic conditioning, and meteorological data, and Table 6 further examines the retrieval cues.

Since multi-style caption training and metadata injection are independent of the tokenizer, the MeteoLIP formulation can also be combined with other tokenizers or encoder architectures, and an information-matched baseline measures how much MetTok contributes on top of this formulation. We are currently training a 4-style CLIP baseline with the same time and location inputs, initialization, and training budget as Meteo-CLIP, and we will share its results on the three tasks once they are available. Meanwhile, without time and location conditioning, MetTok reaches 32.44 / 32.71 (M2T / T2M) R@1 (Table 5), compared to 31.64 / 31.10 for the 4-style CLIP baseline (i.e. without time and location inputs; Table 3).

Please note that we have improved the time-removed retrieval evaluation in Table 6 by counting all identified valid matches of each query as correct (Appendix F.3). Under this evaluation, Full MetTok achieves 2.04× the M2T R@1 of the metadata-only model on all test samples (26.02 vs. 12.75) and 1.58× within same-month pools (47.27 vs. 29.89).

AuthorsOpenReview ↗

Follow-up on Response to Reviewer scya (3/3)

Follow-up on Weakness 1 & Question 1: Information-matched comparison

We have now obtained the results of 4-style CLIP with the same time and location inputs, initialization, and training budget as Meteo-CLIP (Full MetTok). This model uses a simple patch projection followed by an additive time and location embedding, in addition to the ViT positional embedding. Since multi-style caption training and metadata injection form the MeteoLIP formulation, this model corresponds to MeteoLIP with a patch-projection tokenizer in place of MetTok.

ModelM2T R@1T2M R@1Cls F1Cls AccWind R²Wind MAE ↓Pres R²Pres MAE ↓
4-style CLIP + time/loc54.42 ± 0.2454.11 ± 0.5748.26 ± 3.0258.86 ± 2.4459.56 ± 7.059.31 ± 0.3671.52 ± 2.695.97 ± 0.22
Meteo-CLIP58.79 ± 1.4360.10 ± 0.2347.91 ± 2.1057.69 ± 1.4362.27 ± 4.759.35 ± 0.4267.64 ± 4.236.37 ± 0.22

With the same information, Meteo-CLIP improves retrieval by 4.37 (M2T) and 5.99 (T2M) R@1. Text-prototype classification and wind-speed estimation are on par, while the patch projection is slightly better on central-pressure estimation. With time removed from the captions, Meteo-CLIP remains consistently but modestly higher on all test samples and is comparable within same-month pools.

ModelTime removed, all test M2T R@1Time removed, all test T2M R@1Time removed, same month M2T R@1Time removed, same month T2M R@1
4-style CLIP + time/loc25.11 ± 0.2131.90 ± 0.1245.61 ± 4.8149.72 ± 4.75
Meteo-CLIP26.02 ± 0.1732.72 ± 0.3047.27 ± 4.8651.71 ± 4.81

On WeatherQA, Meteo-CLIP also achieves higher Area Affected accuracy (49.22 ± 4.45 vs. 43.19 ± 4.03), with similar conditional Concerning accuracy (51.73 ± 2.41 vs. 51.32 ± 0.98), and higher joint accuracy (25.50 ± 3.04 vs. 22.17 ± 2.15).

These results show that the MeteoLIP formulation also benefits a standard patch projection. It improves retrieval from 20.25 / 20.03 R@1 for CLIP to 54.42 / 54.11 R@1, and MetTok further improves retrieval and WeatherQA on top of it. This suggests that other tokenizer designs could also be explored within the MeteoLIP formulation. We have added these results in Appendix F.11 (Tables 25 and 26).

Comments on the paper

4 comments

Thank you for taking the time to write such a detailed response. Your clarifications and additional results have addressed my concerns. I will maintain my current score.

Thank you for your thoughtful review and for confirming that our clarifications and additional results have addressed your concerns. We also appreciate your constructive suggestions, which have helped us strengthen the work.

Dear Reviewers,

As the discussion period is coming to an end, I kindly ask all reviewers who have not yet responded to the authors' rebuttal to carefully review their responses and provide feedback or update their reviews accordingly.

Thank you for your time and efforts!

AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback and for confirming that our responses have addressed or largely resolved their concerns. We summarize the main additions below; all changes are highlighted in gold in the revised manuscript.

  • Retrieval beyond metadata cues (Reviewers iuG2, 46Z6). With time removed from the captions and all valid matches counted, Full MetTok achieves 2.04x the M2T R@1 of the metadata-only model (1.58x within same-month pools; Appendix F.3, Table 6). With candidates restricted to ±72 h and 50 km, it also outperforms the metadata-only model (36.20 vs. 28.46 M2T R@1; Appendix F.4).
  • Snapshot support (Reviewers scya, iuG2). For records spanning more than one day, the sampled ERA5 hour meets a phenomenon-specific rule 3.2x as often as by chance (41.9% vs. 12.9%). Misses reflect the coarse single-point, single-hour check and the single-snapshot pairing (Appendices D.2, G).
  • Source-held-out breakdown (Reviewer scya). The declines are consistent across event types and do not shrink with more nearby training data, pointing to source-specific reporting conventions (Appendix F.5).
  • Dataset documentation (Reviewer iuG2). We documented the class criteria and the full label mapping (Appendix E) and added a second caption annotator (93.2% agreement, Cohen's κ = 0.47, conservative pass rate 89.77%; Table 11).
  • ClimaX and WeatherQA (Reviewer 46Z6). We clarified MetTok's differences from ClimaX and credited it in Sec. 4.1. On WeatherQA, frozen Meteo-CLIP outperforms an end-to-end fine-tuned encoder without EarthAlign pretraining (49.22 vs. 34.91 Area Affected accuracy; Appendix F.7).
  • Information-matched baseline (Reviewers scya, iuG2). With MetTok replaced by a patch-projection tokenizer under the same inputs, captions, and training budget, the MeteoLIP formulation lifts retrieval from 20.25 (CLIP) to 54.42 M2T R@1. MetTok adds a further 4.37 (M2T) and 5.99 (T2M) R@1 and higher WeatherQA Area Affected accuracy (49.22 vs. 43.19), while classification and wind estimation are on par and the patch projection is slightly better on pressure (Appendix F.11).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.