- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents a training-free approach for hyperspectral image segmentation and classification with the use of pretrained multimodal (image + text) embedding models. The authors designed a pipeline to extract text- and RGB-image-based input data from the spectral image to forward it to the VLM. This input data includes mean spectral reflectance of image regions. The VLM then provides feature embeddings which can be used for few-shot classification. The prediction of a query image is deduced by the cosine similarity to feature embeddings of prototype classes. The model is evaluated on five spectral imaging tasks, including image classification, segmentation, and change detection.
Critical strengths
- The model efficiently leverages multi-scale regions to support dense prediction tasks such as segmentation.
- The RGB image generation from a median spectrum seems to work quite well.
Critical weaknesses
- The results in Tab. 1 are not completely convincing. The model outperforms in only 2 out of 10 datasets. I understand that that HyperMAG does not leverage any domain-specific or task-specific fine-tuning, but it is still not clear what the advantage of HyperMAG is.
- Questionable claim: "In supervised and few-shot settings (HC, HCD, and HOCC), where traditional deep learning architectures severely degrade due to label scarcity, HyperMAG consistently achieves superior performance". However, on those datasets, HyperMAG has the best performance in only 2 out of 6 cases.
- Limited Novelty: The contribution of this paper is mainly the extraction of VLM-suitable input data (text + RGB image) from a hyperspectral image. The way how the text prompt is formed is also not that complex.
Questions for authors
- What exactly is the advantage of HyperMAG in comparison to models such as HyperFree? Can you provide quantitative evidence for that?
- Is the limited context size of the CLIP/Siglip models a challenge for the text prompting? Do you always use all bands for the text prompt? Are they all really necessary or do they rather confuse the VLM?