acceptodds
Under review as a conference paper at ICLR 2027

SpectralEarth-FM: Bringing Hyperspectral Imagery into Multimodal Earth Observation Pretraining

39 views

Abstract

Earth observation (EO) foundation models (FMs) are increasingly trained on multisensor data, spanning multispectral imagery (MSI), synthetic aperture radar (SAR), and derived geospatial layers, but hyperspectral imagery (HSI) remains underrepresented. Conversely, most existing hyperspectral FMs are trained on HSI alone, leaving joint pretraining and fusion of HSI with co-located EO sensors underexplored. We introduce SpectralEarth-FM, a hierarchical transformer for multisensor EO input with heterogeneous spectral dimensionality. The architecture combines spectral tokenization for HSI, sensor-specific encoders, a cross-sensor fusion module, and a shared encoder. To pretrain SpectralEarth-FM, we curate SpectralEarth-MM, a multimodal dataset that co-locates HSI from three spaceborne sensors (EnMAP, EMIT, DESIS) with Sentinel-2, Landsat-8/9 optical imagery, Landsat land surface temperature, and Sentinel-1 SAR. It comprises approximately 2M globally distributed locations, 25M georeferenced patches, and over 40TB of data. Pretraining uses a Joint-Embedding Predictive Architecture (JEPA)-style objective that matches representations between global views and single-sensor local views from the same location. We evaluate SpectralEarth-FM on hyperspectral downstream tasks and standard EO benchmarks following the PANGAEA protocol, achieving results that are competitive with state-of-the-art models.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Earth observation, foundation models, hyperspectral imagery, multimodal learning, self-supervised learning, representation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8AoWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a large dataset pairing HSI with MSI, SAR, and LST, together with a hierarchical multisensor encoder trained using a JEPA-style objective and evaluated on hyperspectral and general earthobservation benchmarks.

Critical strengths

The large scale pairing of spaceborne HSI with complementary sensors is a good dataset contribution, supported by downstream evaluations.

Critical weaknesses

The central benefit of including HSI in multimodal pretraining is not isolated. The paper does not compare HSI-only, non-HSI, and joint pretraining under matched data and compute budgets. A direct comparison with an established multimodal JEPA framework is also missing. Consequently, the improvements cannot be clearly attributed to HSI complementarity or the proposed method. Optical matching permits a six month window, while SAR matching permits up to four years. For crops, floods, fires, and land-cover changes, these observations may represent different physical states. It remains unclear whether alignment learns complementary information or suppresses meaningful temporal variation. The strongest results conflate the proposed representation with multi branch ensemble benefits. Also the spatial independence between pretraining and downstream testing is not adequately documented.

Questions for authors

With matched data volume and training compute, what improvement comes specifically from adding HSI?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8AoW (1/2)

Part 1 of 2.

We thank the reviewer for the assessment and for recognizing the dataset contribution. We address the weaknesses and the question below.

1. Contribution of HSI to multimodal pretraining

Our goal is a single model that supports spaceborne HSI together with the most widely used multispectral, radar and thermal sensors, fuses them and runs on any subset of them. The submission did not study whether the modalities improve each other. We added the following comparison to Sec. 5.4 and App. E.2.

We pretrained HSI-only, non-HSI and joint variants on the same 200K locations as in Table 6 for the same number of epochs, with all other settings fixed. On hyperspectral inputs, joint pretraining scores 1.56 points above HSI-only pretraining, and 2.08 points above non-HSI pretraining, which can use these inputs only through its S2 branch.

Hyperspectral inputs. In the table below, joint pretraining with all seven sensors is the reference. Variants with HSI are evaluated on the eight hyperspectral tasks of Table 6 using the HSI branch. Non-HSI pretraining uses S2, Landsat optical, LST and S1, and is evaluated through its S2 branch after resampling the HSI input to the S2 bands (last row). With HSI-only global views, each local view contains HSI, S2 or Landsat optical, and S1 and LST are not used. The reference row reports absolute IoU and the remaining rows report the change in IoU points.

Pretraining sensorsCDLBD-FBNTDEuCropsD-CDLEO1-CDLGF5-WHOxHypMean
All seven sensors (Table 6 reference)72.269.944.461.765.065.743.275.962.25
HSI only−2.2−2.7−1.0−1.7−0.2−0.7−1.3−2.8−1.56
All except S1−0.2−0.3+0.10.0+1.3+1.1−0.3−1.7−0.02
All except LST−0.3−0.20.0+0.3+0.5+1.4−0.5−0.3+0.11
HSI, S2, Landsat optical−0.1−0.10.0−0.4+0.1−0.1−0.3−1.6−0.31
HSI-only global views−2.7−2.9−0.9−1.7−0.6−0.4−0.7−2.6−1.57
S2, Landsat optical, LST, S1 (S2 branch)−1.0−4.3−1.1−1.6−1.6+0.5−1.3−6.3−2.08

Sentinel-2 inputs. We also compared joint and non-HSI pretraining using S2 inputs, with HSI absent at inference. The comparison uses the colocated tasks of Table 4 with real S2 acquisitions as the only input. The table below reports micro IoU (%) as mean ± standard deviation over three downstream seeds.

Pretraining sensorsCDLBD-FEuCrops
All seven sensors70.8 ± 0.370.7 ± 0.167.9 ± 0.2
S2, Landsat optical, LST, S170.4 ± 0.270.0 ± 0.167.3 ± 0.2
Difference+0.38+0.73+0.52

Pretraining with HSI improves the S2-only result on all three tasks, by 0.38, 0.73 and 0.52 points, and in all nine pairs of task and seed.

Fusion. We also compared the two models on the colocated tasks of Table 4, where EnMAP, S2 and Landsat observe the same location. The table below reports test micro IoU (%).

Pretraining sensorsInputCDLBD-FEuCrops
S2, Landsat optical, LST, S1S2 + Landsat optical71.571.868.0
All seven sensorsEnMAP + S2 + Landsat optical72.6 (+1.10)74.3 (+2.56)70.4 (+2.44)

Without HSI, the model uses S2 and Landsat optical as inputs. With HSI in pretraining, it can also use the EnMAP acquisition, and it scores 1.1, 2.6 and 2.4 points higher on CDL, BD-Forêt and EuroCrops. We added this comparison to App. E.2 (new Table 15).

2. Comparison with multimodal JEPA

Table 2 includes a JEPA-based baseline. CARL (Baumann et al., 2026) combines I-JEPA spatial pretraining with a feature-based spectral objective, and SpectralEarth-FM-s scores above it on all ten tasks. In comparisons between published models, it is difficult to attribute differences to individual components, since the models usually also differ in pretraining data and compute. Corley et al. (2026) find that 94 of 126 surveyed papers use a pretraining configuration that no other paper uses. We therefore analyze the contribution of each component with controlled ablations. Table 6 varies the objective and the view construction while keeping the pretraining data fixed: removing the EMA teacher, SIGReg or the local views lowers the mean by 3.15, 3.40 and 3.77 points. The comparison in point 1 varies the pretraining sensors while keeping the method fixed. Tables 2 and 5 compare the resulting model with published hyperspectral and multimodal models under a common protocol.

AuthorsOpenReview ↗

Response to Reviewer 8AoW (2/2)

Part 2 of 2.

3. Temporal pairing and scene changes

For each HSI acquisition, we retrieved the closest observation of each additional sensor, with maximum offsets of six months for optical sensors and four years for S1. The table below (new Table 10, App. A) reports distances to the nearest retained observations.

SensorMedian (d)P90 (d)≤14 d≤30 d
S223085.6%90.1%
Landsat optical / LST34480.5%87.7%
S1226278.9%80.4%

Median distances are 2–3 days, with 78.9–85.6% within 14 days. The longer S1 window retains locations with coverage gaps: after Sentinel-1B failed in December 2021, Sentinel-1A operated alone until Sentinel-1C started regular acquisitions in 2025 (ESA, 2022; ASF, 2025). A shorter window would have removed these locations.

During pretraining, one retained date per sensor is drawn independently, so different dates are treated as views of the same scene and the objective encourages invariance to their differences. This invariance is desirable when the scene remains largely unchanged, for example under seasonal variation. Joint-embedding methods already match views differing through strong synthetic augmentations such as random resized cropping (Chen et al., 2020), and date differences add a natural augmentation, which can benefit contrastive pretraining (Mañas et al., 2021; Wang et al., 2023). A flood, a fire or a major land-cover change breaks this assumption, but such changes are rare (Mall et al., 2023), so we expect them in a small fraction of training views. Change-detection tasks need distinct representations before and after an event, which change-aware objectives can preserve by contrasting short- and long-interval pairs (Mall et al., 2023). Released acquisition times will enable stricter temporal filters or change-aware pretraining.

4. Representation gains and multi-branch inference

SpectralEarth-FM-a uses the same encoder as SpectralEarth-FM-s. It passes HSI inputs through all five optical branches and PANGAEA inputs through the S2 and Landsat optical branches. Their tokens are fused, then passed through the shared stage and a single prediction head. SpectralEarth-FM-s routes each input to a single optical branch and ranks ahead of all baselines in both settings, with average ranks of 2.30 against 3.00 for Spec. ViT-L (Table 2), and 4.43 against 5.29 for TerraMind-B (Table 5). Its head is the smallest in Table 2 (0.62M parameters against 0.74M/0.84M for Base/Large baselines), and its seven-sensor encoder has 156M parameters, about half the Large baselines' 306M–337M (new Table 13).

App. D (new Table 12) reports SpectralEarth-FM-a's higher inference cost. Its head receives all five branches' feature maps and has 0.91M parameters. To test whether its gain comes from the larger head, we reduced the width from 128 to 96, retaining all feature maps with 0.60M parameters, below the 0.62M of SpectralEarth-FM-s. The table below reports test IoU (%) as mean ± standard deviation over three downstream seeds.

ConfigurationHead widthHead params (M)CDLDESIS-CDLOxHyp
SpectralEarth-FM-s1280.6273.40 ± 0.0367.92 ± 0.2176.18 ± 0.19
SpectralEarth-FM-a, head width 96960.6074.74 ± 0.0968.18 ± 0.2377.64 ± 0.23
SpectralEarth-FM-a1280.9175.16 ± 0.0668.33 ± 0.1178.24 ± 0.09

With the smaller head, SpectralEarth-FM-a remains above SpectralEarth-FM-s in all nine task/seed pairs. On these tasks, the gain of SpectralEarth-FM-a is not explained by the number of head parameters alone.

5. Spatial independence

Before pretraining, we excluded all SpectralEarth-MM footprints that intersect a test patch of the EnMAP, EMIT and DESIS downstream tasks, so these tasks have no spatial overlap with the pretraining data. EO-1 Hyperion and GF-5 are not part of SpectralEarth-MM. App. A describes this filter.

References

Baumann et al. (2026). CARL: Camera-agnostic representation learning for spectral image analysis. ICLR.

Corley et al. (2026). No one knows the state of the art in geospatial foundation models. arXiv:2605.12678.

ASF (2025). Sentinel-1 mission. HyP3 documentation, Alaska Satellite Facility. https://hyp3-docs.asf.alaska.edu/sentinel1/

Chen et al. (2020). A simple framework for contrastive learning of visual representations. ICML.

ESA (2022). Mission ends for Copernicus Sentinel-1B satellite. https://www.esa.int/Applications/Observing_the_Earth/Copernicus/Sentinel-1/Mission_ends_for_Copernicus_Sentinel-1B_satellite

Mall et al. (2023). Change-aware sampling and contrastive learning for satellite images. CVPR.

Mañas et al. (2021). Seasonal contrast: Unsupervised pre-training from uncurated remote sensing data. ICCV.

Wang et al. (2023). SSL4EO-S12: A large-scale multimodal, multitemporal dataset for self-supervised learning in Earth observation. IEEE GRSM.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer,

As the discussion period ends on November 18, we would like to check whether our responses address your concerns. We added the comparison of HSI-only, non-HSI and joint pretraining (Tables 7, 8 and 15), the temporal offsets of the paired observations (Table 10), and multi-branch routing with a smaller head (Table 17). We also clarified the spatial independence of pretraining and test data (App. A). We would be glad to clarify any remaining point. If our responses resolve your concerns, we would appreciate it if you could take them into account in your assessment.

Thank you again for your review.

Reviewer_9WcxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SpectralEarth-MM, a large-scale Earth observation corpus that pairs hyperspectral imagery from EnMAP, EMIT, and DESIS with Sentinel-2, Landsat optical imagery, land surface temperature, and Sentinel-1 SAR. The reported dataset contains approximately two million locations and 25 million patches. The accompanying SpectralEarth-FM combines spectral tokenization, sensor-specific hierarchical branches, cross-sensor attention fusion, and a shared encoder, trained with a teacher-student latent alignment objective and SIGReg regularization. The evaluation covers ten hyperspectral tasks and seven PANGAEA tasks using frozen encoders. The model achieves competitive hyperspectral results and the best average rank among the evaluated models in both settings.

Critical strengths
  1. A potentially valuable dataset contribution Pairing three spaceborne hyperspectral sensors with multispectral, radar, and thermal observations at the reported scale addresses a useful gap identified in the paper. The corpus could support research on spectral representation learning and sensor complementarity.
  2. Competitive hyperspectral transfer across several sensors The evaluation spans ten hyperspectral tasks, including EO-1 Hyperion and GF-5 inputs that are absent from pretraining. Both single-branch and multiple-branch results are reported, and Table 4 separately examines co-located observations. These results provide encouraging evidence that the representations are useful.
Critical weaknesses

The reliability of the training pairs and the source of the gains need stronger evidence. Actual temporal offsets and relative registration errors of the paired observations are not quantified. Pretraining modality controls, inference-budget controls for multiple-branch routing, and an audit of pretraining and test-region overlap are also needed to interpret the reported gains. Uncertainty across repeated runs would strengthen the key comparisons.

Questions for authors
  1. Reconcile the backbone architecture and clarify spatial tokenization Figure2 depicts two shared Hiera stages, whereas AppendixB assigns only the last of three stages to the shared trunk. Please reconcile the stage definitions, pooling locations, and parameter-sharing boundaries. The common60m token stride should also be distinguished from the nominal ground span of the initial projection windows:90m for EnMAP/DESIS,70m for Sentinel-1/2, and60m for EMIT on the stated input grids.
  2. Clarify spectral mapping beyond the source sensor’s wavelength coverage In the hyperspectral multi-branch evaluation, DESIS inputs are routed to branches that expect SWIR bands, although DESIS covers approximately400–1000nm. AppendixD.1 describes linear interpolation but does not explain how target bands outside the observed range are handled. Please specify whether padding, extrapolation, masking, or another mechanism is used, and assess how this choice affects the results.
  3. Resolve the ambiguity in temporal sampling limits Figure6 specifies up to four dates per year, while AppendixA.4 specifies at most four timestamps per sensor and location without an annual qualifier. Please clarify the implemented annual and overall limits, together with the date-selection and de-duplication procedures.
  4. Quantify the temporal and spatial quality of paired observations The dataset uses nearest-date matching within search windows extending to six months for optical imagery and four years for SAR. Please report the actual acquisition-time offsets and relative registration errors, and evaluate sensitivity to stricter pairing criteria.
  5. Isolate the contribution of multimodal pretraining Table6 examines training components and view construction, while Table4 evaluates downstream fusion; neither directly isolates pretraining modality composition. The most informative control would compare HSI-only and selected multimodal pretraining settings using matched locations and comparable training budgets, followed by evaluation with the same HSI input and single-branch configuration. As a secondary clarification, please report the additional computation, feature dimensions, and head parameters of multi-branch routing, with a comparable-budget control where feasible.
  6. Establish evaluation independence and the stability of key results Please report scene-level and geographic overlap between pretraining and downstream test data, and provide repeated-run means and variability for the key comparisons.
6 comments
AuthorsOpenReview ↗

Response to Reviewer 9Wcx (1/3)

Part 1 of 3.

We thank the reviewer for the careful reading and for the detailed questions. We answer each point below.

1. Backbone stages, parameter sharing and spatial tokenization

App. B describes the implemented model. We adjusted Figure 2 and Sec. 4.1 accordingly.

Stages. The three Hiera stages have 2, 6 and 10 blocks. The first two stages (blocks 1–8) are sensor-specific, with widths 192 and 384 and mask-unit attention within 240×240 m windows. Stage 2 starts in block 3 with 2×2 query pooling, reducing the 64×64 grid to 32×32. Branch outputs are fused at each spatial position. Shared stage 3 (blocks 9–18) uses global attention and starts with a second 2×2 query-pooling transition, giving 16×16 tokens of width 768.

Parameter sharing. Tokenizers, stage-1 and stage-2 blocks, and fusion sensor embeddings are sensor-specific. The spatial position embedding, remaining fusion parameters, third stage and final normalization are shared. App. B now gives component parameter counts.

Token stride and receptive field. Patch strides are 2×2 pixels for EnMAP/DESIS/Landsat, 1×1 for EMIT and 6×6 for S1/S2, giving a common 60 m stride. Kernels of 3×3, 1×1 and 7×7 span 90, 60 and 70 m, respectively. With padding of 1, 0 and 3 pixels, window centers differ by at most 25 m per axis. These values refer to native evaluation grids. During pretraining, random resized crops give 30–60 m per token in global views and about 50–120 m in local views. Both quantities are now in App. B.

2. Spectral mapping beyond the input wavelength range

Gaps up to 80 nm within the input range are interpolated linearly. Outside the input range or inside a larger gap, a target band takes the nearest input band's value if within 80 nm. Otherwise, it takes the target band's mean reflectance in SpectralEarth-MM. We do not mask bands. App. D.1 now describes this procedure.

DESIS covers about 400–1000 nm, so 96 of 202 EnMAP bands and 153 of 244 EMIT bands are mean-filled. This affects the DESIS-CDL EnMAP/EMIT branch results (Table 3) and SpectralEarth-FM-a on DESIS-CDL (Table 2). The table below compares mean filling with setting uncovered bands to zero reflectance.

Filling of uncovered bandsEnMAP branchEMIT branch
Band mean (Table 3)64.360.8
Zero61.355.8

Zero filling lowers the scores by 3.0 and 5.0 points, supporting mean filling (App. E.5, new Table 18).

3. Date limits and sampling

We keep at most four dates per sensor and location over the full period (App. A.4), and corrected Figure 6. Some locations have up to a hundred acquisitions from one HSI sensor. The cap keeps the number of observations per location comparable. Before filtering, 18.9%, 22.6%, 25.1% and 25.1% of locations had more than four dates of S1, S2, Landsat optical and LST, respectively. HSI dates favor temporal proximity between HSI sensors and coverage across the year. Other sensors' dates are ranked by proximity to retained HSI dates. Same-sensor dates are at least 10 days apart to remove near-duplicates. App. A now describes these steps.

4. Temporal offsets and spatial registration

Temporal offsets. The six-month optical and four-year S1 windows are maximum offsets. For each HSI acquisition, we retrieved the closest observation of each additional sensor within these windows. New Table 10 in App. A gives the gap to the nearest retained observation:

SensorMedian (d)P90 (d)≤14 d≤30 d
S223085.6%90.1%
Landsat optical / LST34480.5%87.7%
S1226278.9%80.4%

Most offsets are small: for each additional sensor, 78.9–85.6% of HSI acquisitions have a retained observation within 14 days. The four-year S1 window retains locations with coverage gaps: after Sentinel-1B failed in December 2021, Sentinel-1A operated alone until Sentinel-1C started regular acquisitions in 2025 (ESA, 2022; ASF, 2025). Independently matched sensors may lie on either side of the anchor. The time span across an HSI acquisition and its four matches has a median of 11 days and a P90 of 342 days. It is within 14 days for 55.9% of anchors. HSI alignment is looser. At locations with two HSI sensors, the median minimum gaps are 131 days for EMIT–EnMAP, 396 for EMIT–DESIS and 593 for EnMAP–DESIS.

During pretraining, each sensor's date is drawn independently from its retained dates. Joint-embedding methods already match views differing through strong synthetic augmentations such as random resized cropping (Chen et al., 2020). Season or illumination differences provide natural augmentation, and seasonal augmentation can benefit contrastive pretraining (Mañas et al., 2021; Wang et al., 2023). This assumes a largely unchanged scene. Major land-cover changes break this assumption, but are rare and mainly concern change-detection tasks, for which change-aware objectives exist (Mall et al., 2023).

AuthorsOpenReview ↗

Response to Reviewer 9Wcx (2/3)

Part 2 of 3.

4. Temporal offsets and spatial registration (continued)

Registration. We discarded EnMAP and DESIS scenes with metadata orthorectification RMSE above 60 m (one token). Sentinel-2 L2A absolute geolocation accuracy is better than 12 m at 95.5% confidence (ESA). Landsat Collection 2 Tier 1 scenes are georegistered within 12 m radial RMSE (USGS). These values are in App. A.

Stricter pairing. Released acquisition times will allow stricter temporal filtering. In Table 4, we fuse acquisitions within 10 days for CDL and EuroCrops, and fusion exceeds the best single sensor on all three tasks.

5. Pretraining modalities and multi-branch inference cost

Modality composition. Each variant uses the same 200K locations as Table 6 for 50 epochs, with all other settings fixed, and is evaluated with the same HSI input and single-branch routing. The variant without HSI uses its S2 branch after resampling HSI to S2 bands (last row). With HSI-only global views, local views contain HSI, S2 or Landsat optical. S1 and LST are not used. The reference row reports absolute IoU. Other rows report changes in IoU points.

Pretraining sensorsCDLBD-FBNTDEuCropsD-CDLEO1-CDLGF5-WHOxHypMean
All seven sensors (Table 6 reference)72.269.944.461.765.065.743.275.962.25
HSI only−2.2−2.7−1.0−1.7−0.2−0.7−1.3−2.8−1.56
All except S1−0.2−0.3+0.10.0+1.3+1.1−0.3−1.7−0.02
All except LST−0.3−0.20.0+0.3+0.5+1.4−0.5−0.3+0.11
HSI, S2, Landsat optical−0.1−0.10.0−0.4+0.1−0.1−0.3−1.6−0.31
HSI-only global views−2.7−2.9−0.9−1.7−0.6−0.4−0.7−2.6−1.57
S2, Landsat optical, LST, S1 (S2 branch)−1.0−4.3−1.1−1.6−1.6+0.5−1.3−6.3−2.08

Pretraining on HSI alone lowers all eight task scores, by 1.56 points on average. Restricting the global views to HSI lowers all eight scores, by 1.57 points on average.

Removing S1 or LST changes the mean by only −0.02 and +0.11 points, respectively. Their inclusion has a negligible effect on HSI results while enabling SAR and thermal input. Without HSI pretraining, an HSI acquisition uses the S2 branch, lowering seven of eight scores and the mean by 2.08 points (new Table 7, Sec. 5.4). With real S2-only input, joint pretraining scores 0.4 to 0.7 points above non-HSI pretraining on Table 4's tasks (new Table 8). With EnMAP fused with S2 and Landsat optical, the joint model scores 1.1 to 2.6 points above the non-HSI model with S2 and Landsat optical (App. E.2, new Table 15).

Cost of multi-branch routing. New Table 12 in App. D reports encoder FLOPs per (3.84 km)² footprint, active parameters, feature dimensions and head size for each routing. Head parameters are for the 14-class CDL head.

RoutingEncoder GFLOPsActive encoder params (M)Head input channels (stages 1/2/3)Head params (M)
Single branch, EnMAP145.486.2192 / 384 / 7680.62
Single branch, DESIS122.886.2192 / 384 / 7680.62
Single branch, EMIT144.286.1192 / 384 / 7680.62
All HSI branches327.3110.4576 / 1152 / 7680.76
All optical branches (SpectralEarth-FM-a)395.7133.4960 / 1920 / 7680.91

For HSI inputs, SpectralEarth-FM-a runs five branches, fuses their tokens and runs the shared stage once. Its encoder cost is 2.7 times single-branch routing for EnMAP/EMIT and 3.2 times for DESIS. Its head has 1.5 times as many parameters. To control the head budget, we reduced the head width from 128 to 96. This retains all five branches' feature maps with 0.60M head parameters, below SpectralEarth-FM-s's 0.62M. The table below reports test IoU (%) as mean ± standard deviation over three downstream seeds.

ConfigurationHead widthHead params (M)CDLDESIS-CDLOxHyp
SpectralEarth-FM-s1280.6273.40 ± 0.0367.92 ± 0.2176.18 ± 0.19
SpectralEarth-FM-a, head width 96960.6074.74 ± 0.0968.18 ± 0.2377.64 ± 0.23
SpectralEarth-FM-a1280.9175.16 ± 0.0668.33 ± 0.1178.24 ± 0.09

With the smaller head, SpectralEarth-FM-a remains above SpectralEarth-FM-s in all nine pairs of task and seed, by 1.34, 0.26 and 1.46 points on average. Its scores are 0.42, 0.15 and 0.60 points below those with the original head. On these tasks, SpectralEarth-FM-a's gain is not explained by head parameter count alone. SpectralEarth-FM-s's head is smaller than the Base/Large baselines' (0.62M versus 0.74M/0.84M). Its seven-sensor encoder has 156M parameters, about half the Large baselines' 306M to 337M (new Table 13). SpectralEarth-FM-s alone ranks ahead of all baselines in Tables 2 and 5. Apps. D.2 and E.4 give these results.

AuthorsOpenReview ↗

Response to Reviewer 9Wcx (3/3)

Part 3 of 3.

6. Test overlap and repeated-run variability

Overlap. Before pretraining, we excluded all SpectralEarth-MM footprints that intersect a test patch of the EnMAP, EMIT and DESIS downstream tasks, so these tasks have no geographic overlap with the pretraining data. EO-1 Hyperion and GF-5 are not part of SpectralEarth-MM. We added a description of this filter to App. A.

Variability. Table 2 now reports the mean over five downstream seeds for all eleven models, and App. E.1 reports the standard deviations (new Table 14). For this evaluation, we reran all models under a common protocol, in which frozen encoders are kept in evaluation mode during head training. For SpecAware, we now also standardize each input band with the statistics of its sensor, and its average rank improves from 11.00 to 6.30. For the other baselines, the mean over the ten tasks changes by −0.2 to +1.1 points, and every score of SpectralEarth-FM-s and SpectralEarth-FM-a stays within 0.3 points of the original value.

The table below reports test IoU (%) as mean ± standard deviation over five downstream seeds. The best means are in bold and the second best in italics.

ModelCDLBD-FBNTDEuCropsNLCDTreeMapD-CDLEO1-CDLGF5-WHOxHypAvg. rank
CARL57.8 ± 0.3552.2 ± 0.2136.3 ± 0.0646.8 ± 0.3534.3 ± 0.0335.8 ± 0.0958.3 ± 0.1951.2 ± 0.3440.2 ± 0.1060.4 ± 0.568.30
DOFA-B57.4 ± 0.2947.9 ± 0.6035.6 ± 0.0645.2 ± 0.2133.2 ± 0.0434.2 ± 0.1155.8 ± 0.3257.5 ± 0.5842.1 ± 0.0861.2 ± 0.169.10
DOFA-L57.3 ± 0.2447.3 ± 0.2535.5 ± 0.0244.9 ± 0.2132.9 ± 0.0533.1 ± 0.0756.4 ± 0.2855.1 ± 0.5042.0 ± 0.0457.0 ± 0.2810.10
HyperSIGMA-B58.9 ± 0.3252.8 ± 0.3036.1 ± 0.1647.9 ± 0.2434.6 ± 0.0535.5 ± 0.1050.1 ± 0.2058.3 ± 0.2740.5 ± 0.0765.6 ± 0.407.50
HyperSIGMA-L58.9 ± 0.1551.0 ± 0.4335.2 ± 0.2446.1 ± 0.1934.1 ± 0.0734.6 ± 0.1248.7 ± 0.3357.5 ± 0.2140.4 ± 0.0964.8 ± 0.239.10
Panopticon61.4 ± 0.2560.7 ± 0.1938.6 ± 0.0449.7 ± 0.2138.6 ± 0.0438.6 ± 0.0957.9 ± 0.2163.6 ± 0.2744.2 ± 0.0362.7 ± 0.425.20
SpecAware61.7 ± 0.2655.9 ± 0.4139.2 ± 0.0751.9 ± 0.1836.2 ± 0.0435.8 ± 0.1057.6 ± 0.0963.6 ± 0.1342.2 ± 0.0756.6 ± 0.336.30
Spec. ViT-B71.1 ± 0.3469.0 ± 0.1441.1 ± 0.1961.8 ± 0.0441.0 ± 0.1139.3 ± 0.2263.4 ± 0.1664.9 ± 0.2843.1 ± 0.0469.9 ± 0.213.80
Spec. ViT-L72.2 ± 0.1871.6 ± 0.0642.1 ± 0.2762.3 ± 0.2243.0 ± 0.0841.4 ± 0.1165.5 ± 0.1963.7 ± 0.1143.3 ± 0.0670.2 ± 0.153.00
SpectralEarth-FM-s73.4 ± 0.0972.5 ± 0.1145.2 ± 0.0764.5 ± 0.1744.3 ± 0.0342.8 ± 0.0467.9 ± 0.1768.4 ± 0.3642.2 ± 0.1376.3 ± 0.182.30
SpectralEarth-FM-a75.1 ± 0.1073.4 ± 0.0845.7 ± 0.0566.3 ± 0.0644.9 ± 0.0543.2 ± 0.0568.5 ± 0.2969.2 ± 0.1043.0 ± 0.0778.2 ± 0.081.30

The largest standard deviations are 0.36 points for SpectralEarth-FM-s and 0.29 points for SpectralEarth-FM-a. SpectralEarth-FM-s scores above the best baseline on nine of the ten tasks in each of the five seeds, with mean margins of 0.9 to 6.1 points, and Panopticon remains best on GF-5 Wuhan. SpectralEarth-FM-a scores above SpectralEarth-FM-s on all ten tasks in each seed, with mean gains of 0.4 to 2.0 points.

References

ASF (2025). Sentinel-1 mission. HyP3 documentation, Alaska Satellite Facility. https://hyp3-docs.asf.alaska.edu/sentinel1/

Chen et al. (2020). A simple framework for contrastive learning of visual representations. ICML.

ESA (2022). Mission ends for Copernicus Sentinel-1B satellite. https://www.esa.int/Applications/Observing_the_Earth/Copernicus/Sentinel-1/Mission_ends_for_Copernicus_Sentinel-1B_satellite

Mall et al. (2023). Change-aware sampling and contrastive learning for satellite images. CVPR.

Mañas et al. (2021). Seasonal contrast: Unsupervised pre-training from uncurated remote sensing data. ICCV.

Wang et al. (2023). SSL4EO-S12: A large-scale multimodal, multitemporal dataset for self-supervised learning in Earth observation. IEEE GRSM.

ESA. Copernicus Sentinel-2 Collection 1 MSI Level-2A (L2A). https://sentinel.esa.int/web/sentinel/sentinel-data-access/sentinel-products/sentinel-2-data-products/collection-1-level-2a

USGS. Landsat Collection 2 Level-1 data. https://www.usgs.gov/landsat-missions/landsat-collection-2-level-1-data

Reviewer_mXxtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SpectralEarth-FM, a multimodal foundation model designed for Earth observation, alongside a large-scale dataset, SpectralEarth-MM, and a corresponding evaluation framework. SpectralEarth-MM integrates georegistered data from three spaceborne sensors (EnMAP, EMIT, and DESIS), comprising hyperspectral imagery (HSI), multispectral imagery (MSI), synthetic aperture radar (SAR), and land surface temperature (LST) data. All of these data types serve as multimodal inputs for the foundation model. The model architecture incorporates a hyperspectral-specific spectral tokenizer, sensor-specific branches, a cross-sensor fusion module, and a shared Hiera backbone. It is pre-trained via self-supervision using the Joint-Embedding Predictive Architecture (JEPA) paradigm. SpectralEarth-FM demonstrates strong performance on the PANGAEA benchmark, which covers tasks such as crop classification, flood monitoring, land cover mapping, and change detection. The model also achieves competitive results on various downstream hyperspectral applications. Overall, the work leans toward applied contributions centered on multimodal dataset and framework construction. I think its methodological novelty to be limited.

Critical strengths
  1. The paper constructs a large-scale multimodal dataset with hyperspectral data, SpectralEarth-MM. This substantial Earth observation dataset incorporates data from various sensors and carries practical value.
Critical weaknesses
  1. The method lacks sufficient innovation, the model architecture and the pre-training strategy are essentially a direct combination of existing, well-established modules.
  2. During the pre-training phase, the local view utilizes only optical modality data, excluding SAR and LST data. This introduces a distribution shift, which may lead to a decline in model performance during the inference phase when hyperspectral inputs are unavailable.
Questions for authors
  1. In the view construction strategy of SpectralEarth-MM, the local view relies exclusively on the optical modality, excluding SAR and LST. Consequently, the model’s representation capability degrades significantly in scenarios where hyperspectral data is entirely absent, leaving only SAR/LST. Does this imply that downstream tasks do not account for cases where the hyperspectral modality is missing? Can the model still perform valid inference when optical and hyperspectral data are unavailable?
  2. Furthermore, could the authors conduct an ablation study in which only hyperspectral imagery is used for the global representation, excluding MSI, SAR, and LST data? Such an experiment would help determine whether these modalities contribute useful information to the representation or merely introduce noise.
  3. The pre-training process defaults to a fixed view configuration consisting of 2 global views and 4 single-modality local views. The setup lacks more ablation studies. Could the authors provide further discussion on how the ratio of global to local views, as well as the sampling ratio of different modalities among the local views, influence the integration of multimodal information?
  4. The spatial positional encoding in the architecture suggests that the local views of each modality are only spatially aligned. However, downstream tasks are often temporally dependent. Could the authors include a discussion on how multimodal fusion is handled when observations are taken at different time points?
3 comments
AuthorsOpenReview ↗

Response to Reviewer mXxt (1/2)

Part 1 of 2.

We thank the reviewer for the positive assessment and for recognizing the practical value of SpectralEarth-MM. We address the weaknesses and questions below.

1. Technical novelty

We agree that the individual components are established, and we do not present them as novel. Our contribution lies in how they are adapted to a setting that existing models do not address. Multimodal EO foundation models rarely include HSI, while most hyperspectral foundation models are trained on hyperspectral data alone. SpectralEarth-FM encodes inputs from a single thermal band to 244 contiguous hyperspectral bands, remains usable when only one sensor is available, and fuses colocated observations from several sensors.

The pretraining objective is designed for these requirements. Global views fuse a random subset of the available sensors and define the target, while each local view contains a single sensor. Each sensor branch is therefore trained to predict the fused multisensor representation from its own input, which extends multi-crop training from spatial to sensor context. As a result, one set of weights supports single-sensor inference, multi-branch routing of one acquisition, and fusion of colocated observations (Tables 2 to 4). The other components follow from this objective. The EMA teacher keeps the fused target stable while the student learns from single-sensor views. Spectral tokenization encodes the full HSI spectrum before it is fused with sensors of much lower spectral dimensionality. The resulting model and SpectralEarth-MM are our main contributions.

The ablations in Table 6 support each of these choices. Removing the local views or the EMA teacher, or replacing the spectral tokenizers with linear encoders, reduces the mean by 3.77, 3.15 and 1.86 points and lowers every score. The new comparison of pretraining modalities in point 3 below isolates the multisensor target: restricting the global views to HSI lowers all eight hyperspectral scores, by 1.57 points on average, so fusing other sensors into the target improves the HSI representations themselves. Admitting SAR and LST as local views lowers every score in Table 6, which supports using only optical modalities in the local views for the hyperspectral tasks.

2. Local views and inference without HSI or optical inputs

When HSI is not available, multispectral imagery from Sentinel-2 or Landsat usually is, and our evaluation covers this case. None of the seven PANGAEA tasks contains HSI, and on these tasks SpectralEarth-FM-s ranks ahead of all baselines, with an average rank of 4.43 against 5.29 for TerraMind-B (Table 5). Table 4 also reports results with S2 or Landsat as the only input. HSI is also absent from part of the pretraining views. Each global view contains four modalities sampled at random from those available, so at a location observed by one HSI sensor, one in five global views contains no HSI. The S2 and Landsat local views are trained to predict the fused target from MSI alone.

As the reviewer notes, excluding SAR and LST from the local views affects inputs without any optical sensor, and it lowers S1-only performance. We compared two models of Table 6 with S1 as the only input: the reference, whose local views contain only optical modalities, and the variant that also admits SAR and LST as local views. Both were pretrained on the same 200K locations for 50 epochs and evaluated on PASTIS and Sen1Floods11, the two PANGAEA tasks of Table 5 with S1 data. We also evaluated the baselines of Table 5 that support S1 under the same frozen-encoder protocol. The table below reports test mIoU (%).

ModelPASTIS (S1 only)Sen1Floods11 (S1 only)
CROMA-L21.7979.18
DOFA-B16.3778.81
S12-MAE (S1)14.0378.38
TerraMind-B19.7280.37
THOR-B23.1480.61
Reference (optical local views)20.6980.52
SAR and LST as local views26.1081.49

With S1 as the only input, the reference ranks third of six models on PASTIS and is on par with the best baselines on Sen1Floods11. Admitting SAR and LST as local views improves its score by 5.4 points on PASTIS and 1.0 point on Sen1Floods11, which gives the best result on both tasks. On the hyperspectral tasks, however, this variant lowers all eight scores of Table 6, by 0.67 points on average. We deliberately chose to favor the optical modalities, since HSI is the focus of this work. We added this comparison to App. E.3 (new Table 16).

AuthorsOpenReview ↗

Response to Reviewer mXxt (2/2)

Part 2 of 2.

3. HSI-only global views

We ran this ablation on the same 200K locations as Table 6 for 50 epochs, with all other settings fixed. In the requested variant, global views contain only HSI, while local views contain HSI, S2 or Landsat optical. S1 and LST are not used. Models with HSI use the HSI branch on the eight Table 6 tasks. The model without HSI uses its S2 branch after resampling the HSI input to S2 bands (last row). The reference row gives absolute IoU. Other rows give changes in points.

Pretraining sensorsCDLBD-FBNTDEuCropsD-CDLEO1-CDLGF5-WHOxHypMean
All seven sensors (Table 6 reference)72.269.944.461.765.065.743.275.962.25
HSI only−2.2−2.7−1.0−1.7−0.2−0.7−1.3−2.8−1.56
All except S1−0.2−0.3+0.10.0+1.3+1.1−0.3−1.7−0.02
All except LST−0.3−0.20.0+0.3+0.5+1.4−0.5−0.3+0.11
HSI, S2, Landsat optical−0.1−0.10.0−0.4+0.1−0.1−0.3−1.6−0.31
HSI-only global views−2.7−2.9−0.9−1.7−0.6−0.4−0.7−2.6−1.57
S2, Landsat optical, LST, S1 (S2 branch)−1.0−4.3−1.1−1.6−1.6+0.5−1.3−6.3−2.08

Restricting global views to HSI and pretraining on HSI alone lower all eight scores, by 1.57 and 1.56 points on average. Additional sensors in the global views therefore contribute useful information to the HSI representations.

Although S1 and LST differ substantially from HSI, removing either changes the mean by −0.02 and +0.11 points, respectively. Including them has a negligible effect on the HSI results while allowing the model to process SAR and thermal inputs. These results are in Sec. 5.4 (new Table 7).

4. View counts and modality sampling

We use two global views mainly because of compute cost: each contains four modalities and is processed by both the student and EMA teacher. A local view contains one modality on a 16×16 rather than 64×64 token grid and is processed by the student only, adding little memory and compute. In Table 6, removing local views lowers the mean by 3.77 points, and using two instead of four lowers it by 0.51 points, supporting our view sampling strategy.

Each local view contains one optical modality, drawn uniformly from those in the global views (App. C). Admitting SAR and LST as local views lowers the mean by 0.67 points (Table 6), supporting optical local views for the HSI tasks.

5. Temporal alignment and time series

The position embedding is spatial only. The model has no temporal encoding. We added this discussion to App. A and the limitations.

Pretraining. Each sensor's date is drawn independently from its retained dates, so different dates at a location are treated as views of the same scene. The median distance from an HSI acquisition to the nearest retained S2, Landsat or S1 observation is 2 to 3 days (new Table 10). Remaining differences act as natural augmentation alongside synthetic ones such as random resized cropping (Chen et al., 2020). Seasonal augmentations can benefit contrastive pretraining (Mañas et al., 2021; Wang et al., 2023). This assumes a largely unchanged scene. Major land-cover changes break this assumption, but are rare and mainly concern change-detection tasks, for which change-aware objectives have been proposed (Mall et al., 2023).

Downstream fusion. Colocated observations are fused at each spatial position independently of date. When labels remain unchanged, as is typical for forest species or minerals and crop type within a season, the fused observations add information about the same scene. In Table 4, the colocated acquisitions are within 10 days for CDL and EuroCrops and 120 days for BD-Forêt. Fusion improves over the best single sensor on all three tasks.

Time series. Time-series modeling is not our focus. Like other single-date models, SpectralEarth-FM can encode each acquisition separately and aggregate features over time in the decoder. Our PASTIS evaluation uses the PANGAEA benchmark's L-TAE aggregation (Marsocci et al., 2026), and SpectralEarth-FM-a achieves the best score on this task (Table 5). Combining single HSI observations with MSI time series is a direction for future work.

References

Chen et al. (2020). A simple framework for contrastive learning of visual representations. ICML.

Mall et al. (2023). Change-aware sampling and contrastive learning for satellite images. CVPR.

Mañas et al. (2021). Seasonal contrast: Unsupervised pre-training from uncurated remote sensing data. ICCV.

Marsocci et al. (2026). PANGAEA: Assessing geospatial foundation models capabilities through a global and inclusive benchmark. IEEE GRSM.

Wang et al. (2023). SSL4EO-S12: A large-scale multimodal, multitemporal dataset for self-supervised learning in Earth observation. IEEE GRSM.

Thanks for the authors’ response, which addresses my concerns. I will retain my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their detailed assessments and for recognizing the value of SpectralEarth-MM and SpectralEarth-FM. The added experiments strengthen the evidence for joint hyperspectral and multispectral pretraining, while the revised manuscript clarifies the data alignment assumptions, inference costs and evaluation protocol. We uploaded a revised manuscript in which additions are highlighted in blue.

Pretraining modalities. The controlled ablations show that HSI and multispectral observations benefit each other during pretraining. Joint pretraining improves hyperspectral performance over HSI-only pretraining by 1.56 points on average, and provides small but consistent improvements on multispectral inputs when HSI is unavailable at inference. Fusing an EnMAP acquisition with S2 and Landsat optical improves over the non-HSI model by 1.1 to 2.6 points. Allowing multispectral observations in global views also improves the HSI representations. The contribution of SAR and thermal data to the hyperspectral scores is small.

Data alignment and test overlap. We clarified how observations are selected and paired, quantified temporal offsets, and documented the exclusion of downstream test areas. Most additional-sensor matches are close in time, but longer offsets remain, especially for SAR. The revision makes explicit that matching different dates assumes a largely unchanged scene. Released acquisition times will allow stricter temporal filtering.

Multi-branch routing. The single-branch model already ranks ahead of all baselines in both the HSI and PANGAEA evaluations. Multi-branch routing provides further gains at a higher inference cost, which we now report explicitly. On the tested tasks, these gains remain with a head smaller than that of the single-branch model, showing that they are not explained by head parameter count alone.

Evaluation variability. We repeated the hyperspectral evaluation for all models over five downstream seeds under a common protocol, with frozen encoders kept in evaluation mode. SpecAware now also uses per-band standardization for each sensor, which improves its average rank from 11.00 to 6.30. The average ranks of SpectralEarth-FM-s and SpectralEarth-FM-a remain 2.30 and 1.30, and the results now include standard deviations.

Further experiments and clarifications. Additional ablations show that including SAR and thermal local views improves SAR-only performance at some cost to hyperspectral performance, and support mean filling for uncovered spectral bands. We also adjusted the architecture and date-selection figures and clarified the implementation details.

The individual responses include the relevant results and explanations. We thank the reviewers for their constructive feedback, which helped us improve the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.