- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a large dataset pairing HSI with MSI, SAR, and LST, together with a hierarchical multisensor encoder trained using a JEPA-style objective and evaluated on hyperspectral and general earthobservation benchmarks.
Critical strengths
The large scale pairing of spaceborne HSI with complementary sensors is a good dataset contribution, supported by downstream evaluations.
Critical weaknesses
The central benefit of including HSI in multimodal pretraining is not isolated. The paper does not compare HSI-only, non-HSI, and joint pretraining under matched data and compute budgets. A direct comparison with an established multimodal JEPA framework is also missing. Consequently, the improvements cannot be clearly attributed to HSI complementarity or the proposed method. Optical matching permits a six month window, while SAR matching permits up to four years. For crops, floods, fires, and land-cover changes, these observations may represent different physical states. It remains unclear whether alignment learns complementary information or suppresses meaningful temporal variation. The strongest results conflate the proposed representation with multi branch ensemble benefits. Also the spatial independence between pretraining and downstream testing is not adequately documented.
Questions for authors
With matched data volume and training compute, what improvement comes specifically from adding HSI?
3 comments
Response to Reviewer 8AoW (2/2)
Part 2 of 2.
3. Temporal pairing and scene changes
For each HSI acquisition, we retrieved the closest observation of each additional sensor, with maximum offsets of six months for optical sensors and four years for S1. The table below (new Table 10, App. A) reports distances to the nearest retained observations.
| Sensor | Median (d) | P90 (d) | ≤14 d | ≤30 d |
|---|---|---|---|---|
| S2 | 2 | 30 | 85.6% | 90.1% |
| Landsat optical / LST | 3 | 44 | 80.5% | 87.7% |
| S1 | 2 | 262 | 78.9% | 80.4% |
Median distances are 2–3 days, with 78.9–85.6% within 14 days. The longer S1 window retains locations with coverage gaps: after Sentinel-1B failed in December 2021, Sentinel-1A operated alone until Sentinel-1C started regular acquisitions in 2025 (ESA, 2022; ASF, 2025). A shorter window would have removed these locations.
During pretraining, one retained date per sensor is drawn independently, so different dates are treated as views of the same scene and the objective encourages invariance to their differences. This invariance is desirable when the scene remains largely unchanged, for example under seasonal variation. Joint-embedding methods already match views differing through strong synthetic augmentations such as random resized cropping (Chen et al., 2020), and date differences add a natural augmentation, which can benefit contrastive pretraining (Mañas et al., 2021; Wang et al., 2023). A flood, a fire or a major land-cover change breaks this assumption, but such changes are rare (Mall et al., 2023), so we expect them in a small fraction of training views. Change-detection tasks need distinct representations before and after an event, which change-aware objectives can preserve by contrasting short- and long-interval pairs (Mall et al., 2023). Released acquisition times will enable stricter temporal filters or change-aware pretraining.
4. Representation gains and multi-branch inference
SpectralEarth-FM-a uses the same encoder as SpectralEarth-FM-s. It passes HSI inputs through all five optical branches and PANGAEA inputs through the S2 and Landsat optical branches. Their tokens are fused, then passed through the shared stage and a single prediction head. SpectralEarth-FM-s routes each input to a single optical branch and ranks ahead of all baselines in both settings, with average ranks of 2.30 against 3.00 for Spec. ViT-L (Table 2), and 4.43 against 5.29 for TerraMind-B (Table 5). Its head is the smallest in Table 2 (0.62M parameters against 0.74M/0.84M for Base/Large baselines), and its seven-sensor encoder has 156M parameters, about half the Large baselines' 306M–337M (new Table 13).
App. D (new Table 12) reports SpectralEarth-FM-a's higher inference cost. Its head receives all five branches' feature maps and has 0.91M parameters. To test whether its gain comes from the larger head, we reduced the width from 128 to 96, retaining all feature maps with 0.60M parameters, below the 0.62M of SpectralEarth-FM-s. The table below reports test IoU (%) as mean ± standard deviation over three downstream seeds.
| Configuration | Head width | Head params (M) | CDL | DESIS-CDL | OxHyp |
|---|---|---|---|---|---|
| SpectralEarth-FM-s | 128 | 0.62 | 73.40 ± 0.03 | 67.92 ± 0.21 | 76.18 ± 0.19 |
| SpectralEarth-FM-a, head width 96 | 96 | 0.60 | 74.74 ± 0.09 | 68.18 ± 0.23 | 77.64 ± 0.23 |
| SpectralEarth-FM-a | 128 | 0.91 | 75.16 ± 0.06 | 68.33 ± 0.11 | 78.24 ± 0.09 |
With the smaller head, SpectralEarth-FM-a remains above SpectralEarth-FM-s in all nine task/seed pairs. On these tasks, the gain of SpectralEarth-FM-a is not explained by the number of head parameters alone.
5. Spatial independence
Before pretraining, we excluded all SpectralEarth-MM footprints that intersect a test patch of the EnMAP, EMIT and DESIS downstream tasks, so these tasks have no spatial overlap with the pretraining data. EO-1 Hyperion and GF-5 are not part of SpectralEarth-MM. App. A describes this filter.
References
Baumann et al. (2026). CARL: Camera-agnostic representation learning for spectral image analysis. ICLR.
Corley et al. (2026). No one knows the state of the art in geospatial foundation models. arXiv:2605.12678.
ASF (2025). Sentinel-1 mission. HyP3 documentation, Alaska Satellite Facility. https://hyp3-docs.asf.alaska.edu/sentinel1/
Chen et al. (2020). A simple framework for contrastive learning of visual representations. ICML.
ESA (2022). Mission ends for Copernicus Sentinel-1B satellite. https://www.esa.int/Applications/Observing_the_Earth/Copernicus/Sentinel-1/Mission_ends_for_Copernicus_Sentinel-1B_satellite
Mall et al. (2023). Change-aware sampling and contrastive learning for satellite images. CVPR.
Mañas et al. (2021). Seasonal contrast: Unsupervised pre-training from uncurated remote sensing data. ICCV.
Wang et al. (2023). SSL4EO-S12: A large-scale multimodal, multitemporal dataset for self-supervised learning in Earth observation. IEEE GRSM.
Follow-up on our responses
Dear Reviewer,
As the discussion period ends on November 18, we would like to check whether our responses address your concerns. We added the comparison of HSI-only, non-HSI and joint pretraining (Tables 7, 8 and 15), the temporal offsets of the paired observations (Table 10), and multi-branch routing with a smaller head (Table 17). We also clarified the spatial independence of pretraining and test data (App. A). We would be glad to clarify any remaining point. If our responses resolve your concerns, we would appreciate it if you could take them into account in your assessment.
Thank you again for your review.
Response to Reviewer 8AoW (1/2)
Part 1 of 2.
We thank the reviewer for the assessment and for recognizing the dataset contribution. We address the weaknesses and the question below.
1. Contribution of HSI to multimodal pretraining
Our goal is a single model that supports spaceborne HSI together with the most widely used multispectral, radar and thermal sensors, fuses them and runs on any subset of them. The submission did not study whether the modalities improve each other. We added the following comparison to Sec. 5.4 and App. E.2.
We pretrained HSI-only, non-HSI and joint variants on the same 200K locations as in Table 6 for the same number of epochs, with all other settings fixed. On hyperspectral inputs, joint pretraining scores 1.56 points above HSI-only pretraining, and 2.08 points above non-HSI pretraining, which can use these inputs only through its S2 branch.
Hyperspectral inputs. In the table below, joint pretraining with all seven sensors is the reference. Variants with HSI are evaluated on the eight hyperspectral tasks of Table 6 using the HSI branch. Non-HSI pretraining uses S2, Landsat optical, LST and S1, and is evaluated through its S2 branch after resampling the HSI input to the S2 bands (last row). With HSI-only global views, each local view contains HSI, S2 or Landsat optical, and S1 and LST are not used. The reference row reports absolute IoU and the remaining rows report the change in IoU points.
Sentinel-2 inputs. We also compared joint and non-HSI pretraining using S2 inputs, with HSI absent at inference. The comparison uses the colocated tasks of Table 4 with real S2 acquisitions as the only input. The table below reports micro IoU (%) as mean ± standard deviation over three downstream seeds.
Pretraining with HSI improves the S2-only result on all three tasks, by 0.38, 0.73 and 0.52 points, and in all nine pairs of task and seed.
Fusion. We also compared the two models on the colocated tasks of Table 4, where EnMAP, S2 and Landsat observe the same location. The table below reports test micro IoU (%).
Without HSI, the model uses S2 and Landsat optical as inputs. With HSI in pretraining, it can also use the EnMAP acquisition, and it scores 1.1, 2.6 and 2.4 points higher on CDL, BD-Forêt and EuroCrops. We added this comparison to App. E.2 (new Table 15).
2. Comparison with multimodal JEPA
Table 2 includes a JEPA-based baseline. CARL (Baumann et al., 2026) combines I-JEPA spatial pretraining with a feature-based spectral objective, and SpectralEarth-FM-s scores above it on all ten tasks. In comparisons between published models, it is difficult to attribute differences to individual components, since the models usually also differ in pretraining data and compute. Corley et al. (2026) find that 94 of 126 surveyed papers use a pretraining configuration that no other paper uses. We therefore analyze the contribution of each component with controlled ablations. Table 6 varies the objective and the view construction while keeping the pretraining data fixed: removing the EMA teacher, SIGReg or the local views lowers the mean by 3.15, 3.40 and 3.77 points. The comparison in point 1 varies the pretraining sensors while keeping the method fixed. Tables 2 and 5 compare the resulting model with published hyperspectral and multimodal models under a common protocol.