acceptodds
Under review as a conference paper at ICLR 2027

hyperSMARTIES: Physics-Aware Spectral Representation Learning for Remote Sensing

61 views

Abstract

Unifying image representation learning is essential for enabling self-supervised pretraining in remote sensing to scale across diverse sensors. However, existing models are typically constrained to fixed, predefined band sets, or do not scale seamlessly to new generation hyperspectral sensors, where hundreds of spectrally narrow channels densely sample the electromagnetic spectrum. To address this, we introduce hyperSMARTIES, a unified foundation model for wavelength-continuous masked data modeling. hyperSMARTIES learns a unified spectral projection that embeds arbitrary bands using a single shared projection function, and employs a transformer block that adaptively aggregates per-band embeddings into a single spectral token conditioned by the central wavelengths of bands. This lifts the need for sensor/range-specific projection functions and enables unified projection across diverse spectral regimes. Furthermore, spectral consistency is enforced via flow matching over wavelengths: a wavelength-indexed velocity field is learned on band embeddings to enforce smooth, locally consistent evolution across neighboring wavelengths, hence reproducing the physical characteristics of hyperspectral signals. We pretrain hyperSMARTIES on hyperspectral imagery acquired by the EnMAP sensor, which provides fine-grained spectral sampling, and apply downstream transfer on multiple tasks with diverse sensors. hyperSMARTIES not only outperforms previous models with improved unseen-sensor generalization, but also shows wavelength-dependent consistency in the learned band embeddings.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, foundation models, remote sensing, hyperspectral imaging

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YJ26OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a new sensor-unifying remote sensing foundation model. Its main contribution is a wavelength-indexed flow matching module that enhances physical consistency across the band embeddings. Afterwards, these embeddings are forwarded to a spectral self-attention layer that compresses the spectral bands into one token. This token is then forwarded to the vision transformer for every spatial patch. The model is pretrained on Enmap data via masked reconstruction. Finally, Hypersmarties is evaluated on Enmap data, cross-sensor hyperspectral, and cross-sensor multispectral data. Furthermore, ablation studies on key design choices are conducted.

Critical strengths
  • HyperSMARTIES is one of the first works that attempt to embed the bands according to their wavelengths and introduce an extra loss component for this behavior.
  • The band embedding plots successfully demonstrate the increased band consistency of HyperSMARTIES.
  • The evaluation on hyperspectral data is comprehensive and convincing.
Critical weaknesses
  • The pretraining seems to be limited to hyperspectral data.
  • The cross-sensor evaluation, especially the evaluation on common Sentinel-2 datasets is very limited.
  • From the evaluation, it is not clear if the presented model is tied to hyperspectral data, or if it can also consistently outperform on multispectral or even RGB data.
  • The contribution relative to prior models (SMARTIES, Panopticon) seems to be limited to the wavelength-indexed flow matching module. So-called wavelength-continuous spectral projections are simply projections shared across all bands, commonly used in Panopticon, CARL, etc.
Questions for authors
  • Is the pretraining data limited to hyperspectral images? I think the derivatives of the wavelengths flow matching module are too coarse with multispectral (Sentinel-2) images. How does this behave when going from 25 to 50 bands as conducted in your pretraining?
  • Each band is reconstructed by a band-specific output layer in the pretraining. However, the authors mention to randomly subsample the bands in each mini batch. So, in each training step, the output layers reconstruct different bands. Does this make sense?
  • How can you explain the sharp drop in Sentinel-2 performance, when pretraining the model with all 202 bands? Wouldn't you good band embeddings as the derivatives are very accurate? Does this contradict the claimed cross-generalizability?
  • Did you adjust the image sizes when running the baseline experiments to enable fair comparison, as HyperSMARTIES has a very low patch size of 4?
Reviewer_kggZOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

hyperSMARTIES aims to learn transferable remote sensing representations through hyperspectral self-supervised pretraining. It combines shared band-wise linear projection, sinusoidal wavelength encoding, and a Transformer with a learnable spectral token to aggregate bands into one token per spatial patch. Alongside masked reconstruction, it introduces wavelength-indexed flow matching: a wavelength-conditioned network predicts the velocity along linear interpolations between band embeddings to encourage spectral consistency.

Critical strengths

Cross-sensor transfer is practically important given differences in band counts, central wavelengths, and spectral sampling densities. Pretrained solely on EnMAP, hyperSMARTIES transfers to other hyperspectral, multispectral, and high-resolution optical imagery with a frozen backbone and trained task heads. In Table 3, it achieves the best results on three of four unseen-sensor datasets, notably reaching 63.9% mIoU on EO-1 CDL versus 59.0% for the strongest baseline. These results provide meaningful evidence of practical cross-sensor utility.

Critical weaknesses

1.The substantive advance over prior work is insufficiently established, and the comparisons do not clearly isolate the proposed contributions. Panopticon already combines shared per-channel patch projection, sinusoidal central-wavelength encoding, and learned-query cross-channel attention to aggregate arbitrary numbers of bands into one spatial token. It also employs random channel sampling during pretraining. These components closely resemble the proposed wavelength-conditioned tokenization and band-sampling strategy. Although Panopticon is cited and evaluated, the additional architectural capability remains unclear.

SpectralEarth-FM likewise uses a spectral Transformer and learned-query aggregation to compress spectral group tokens into one spatial token. PhySwin incorporates spectral smoothness into pretraining and preserves spatial token counts through grouped feature concatenation. These methods differ from hyperSMARTIES: SpectralEarth-FM uses sensor-specific branches, while PhySwin regularizes reconstructed reflectance and aggregates features by concatenation. Nevertheless, efficient spectral aggregation and physically motivated spectral objectives are established ideas. The manuscript does not adequately position its contributions relative to these approaches or explain the significance of the remaining differences.

2.The connection between the flow-matching objective and the claimed physical consistency lacks sufficient justification and direct validation. Equations (4)–(5) train an auxiliary network to predict the secant velocity between two band embeddings. Accurately fitting this constructed velocity does not, by itself, establish physically meaningful local variation in actual band representations. The sampled endpoints need not be neighboring bands, and the interpolated path is not validated against actual intermediate-band embeddings. It therefore remains unclear why this objective preserves genuine spectral structure beyond improving predictability along artificial interpolation paths. The supporting evidence is indirect. PCC in Table 4 measures the correlation between embedding distances and wavelength distances, while Figures 2 and 4 visualize wavelength ordering after dimensionality reduction. Neither establishes more faithful preservation of spectral content: better wavelength ordering could coexist with suppression of discriminative spectral differences. Moreover, the appendix analyzes attention in downstream-finetuned models, which does not isolate the contribution of the pretraining flow-matching objective.

Questions for authors
  1. What specific capabilities does the proposed spectral aggregation add over the closest prior designs? Please clarify the differences from Panopticon’s shared projection, wavelength encoding, and single-query aggregation, and from SpectralEarth-FM’s spectral Transformer aggregation. Please also explain the relationship to spectral smoothness objectives such as those in PhySwin. Can comparisons using matched pretraining data and band-sampling protocols separately isolate the benefits of the aggregation mechanism and flow matching? These distinctions would directly affect my assessment of the methodological contribution.

  2. Where is the flow-matching loss applied, and how are gradients handled? Are the endpoint embeddings in Equation (4) taken after shared projection, wavelength encoding, or the spectral Transformer? Which branches retain gradients when constructing the interpolated embeddings and target velocities, and which use stop-gradient? Are the embeddings normalized? Please explain how these choices constrain spectral structure and distinguish improved velocity prediction from merely shrinking differences between embeddings.

Reviewer_USqSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Remote sensing sensors vary substantially in the number of bands, central wavelengths, bandwidths, and spatial resolutions. Models with fixed input channels are difficult to transfer directly across sensors, while assigning separate projections to discrete spectral intervals does not scale well to dense hyperspectral bands. This work aims to leverage EnMAP dense-spectral pre-training to learn an encoder capable of handling variable band sets and generalizing to novel sensors.

Critical strengths

The core innovation centers on combining wavelength-parameterized embeddings and velocity prediction with shared spectral tokenization for hyperspectral pre-training and cross-sensor transfer.

The work features broad task coverage, reports results for both frozen probing and fine-tuning, presents main results as the mean and standard deviation across three downstream runs, adopts FDR as an alternative regularization strategy, and openly discusses band subsampling and computational costs.

Critical weaknesses

Equation (3) describes an ODE flow generated by a velocity field, yet training proceeds along artificially constructed straight-line segments. The paper does not clarify what consistency relationship the resulting flow satisfies with respect to the true encoded spectral curves, nor does it demonstrate that the integration path can recover the true intermediate bands.

PCC measures the global correlation between embedding distance and wavelength distance. Physical spectra do not require such a distance to be globally monotonic: distant bands can exhibit similar reflectance, while local absorption features may also induce large variations. Thus, improving PCC does not necessarily preserve physical information. Furthermore, Figures 2 and 4 average patches before performing UMAP, which may dilute local absorption features and spatial variations, and the resulting 2D geometry is also subject to visualization parameter choices.

The main results use ViT-B/L, while the FM and projection ablation experiments use ViT-S with only 5% of the data. Table 4 provides valuable information, yet it cannot demonstrate that the 4.9 percentage point gain on the large-scale EO-1 CDL dataset is attributable to FM. Moreover, Table 4 reports GFLOPs of 22.2 and 69.9 for the discrete and continuous versions, respectively, suggesting that the changes extend beyond spectral parameterization to include substantially different tokenization computations.

Questions for authors

See above weaknesses.

1 comment
Reviewer_USqSOpenReview ↗

response

My first and third concerns have been well addressed, and the second concern has been mostly resolved; my main remaining concern is that the sensitivity of the UMAP visualizations to parameter choices has not been verified.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.