- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents a new sensor-unifying remote sensing foundation model. Its main contribution is a wavelength-indexed flow matching module that enhances physical consistency across the band embeddings. Afterwards, these embeddings are forwarded to a spectral self-attention layer that compresses the spectral bands into one token. This token is then forwarded to the vision transformer for every spatial patch. The model is pretrained on Enmap data via masked reconstruction. Finally, Hypersmarties is evaluated on Enmap data, cross-sensor hyperspectral, and cross-sensor multispectral data. Furthermore, ablation studies on key design choices are conducted.
Critical strengths
- HyperSMARTIES is one of the first works that attempt to embed the bands according to their wavelengths and introduce an extra loss component for this behavior.
- The band embedding plots successfully demonstrate the increased band consistency of HyperSMARTIES.
- The evaluation on hyperspectral data is comprehensive and convincing.
Critical weaknesses
- The pretraining seems to be limited to hyperspectral data.
- The cross-sensor evaluation, especially the evaluation on common Sentinel-2 datasets is very limited.
- From the evaluation, it is not clear if the presented model is tied to hyperspectral data, or if it can also consistently outperform on multispectral or even RGB data.
- The contribution relative to prior models (SMARTIES, Panopticon) seems to be limited to the wavelength-indexed flow matching module. So-called wavelength-continuous spectral projections are simply projections shared across all bands, commonly used in Panopticon, CARL, etc.
Questions for authors
- Is the pretraining data limited to hyperspectral images? I think the derivatives of the wavelengths flow matching module are too coarse with multispectral (Sentinel-2) images. How does this behave when going from 25 to 50 bands as conducted in your pretraining?
- Each band is reconstructed by a band-specific output layer in the pretraining. However, the authors mention to randomly subsample the bands in each mini batch. So, in each training step, the output layers reconstruct different bands. Does this make sense?
- How can you explain the sharp drop in Sentinel-2 performance, when pretraining the model with all 202 bands? Wouldn't you good band embeddings as the derivatives are very accurate? Does this contradict the claimed cross-generalizability?
- Did you adjust the image sizes when running the baseline experiments to enable fair comparison, as HyperSMARTIES has a very low patch size of 4?
response
My first and third concerns have been well addressed, and the second concern has been mostly resolved; my main remaining concern is that the sensitivity of the UMAP visualizations to parameter choices has not been verified.