acceptodds
Withdrawn from ICLR 2027

Translating Hyperspectral Images into Multimodal Semantics that Vision-Language Models can Better Understand

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Despite capturing unified spatial-spectral representations, hyperspectral foundation models incur prohibitive pre-training costs and require sensor/scene-specific fine-tuning, severely hindering real-world deployment. This raises a fundamental question: "Can we eliminate domain-specific pre-training and adaptation, directly leveraging pre-trained vision-language models (VLMs) for training-free hyperspectral interpretation?" To address this, we propose HyperMAG, a novel framework that rethinks hyperspectral image interpretation through multimodal semantic augmented generation. Our key insight is that pre-trained VLMs, despite having never seen hyperspectral data, can reliably interpret spectral information when effectively translated into their native modalities. Specifically, HyperMAG translates spectral signatures into complementary textual descriptors (band-indexed values and peak–valley trends) and visual strips (encoding magnitude and local contrast) across multi-scale homogeneous regions, enabling off-the-shelf VLMs to extract rich spatial-spectral semantics. Extensive experiments demonstrate that HyperMAG matches or surpasses both task-specific models and foundation models across diverse downstream tasks, including few-shot classification, single-class classification, target detection, anomaly detection, and change detection.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
hyperspectral image interpretation, pre-trained model, vision-language model, training-free interpretation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_hKCqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a training-free approach for hyperspectral image segmentation and classification with the use of pretrained multimodal (image + text) embedding models. The authors designed a pipeline to extract text- and RGB-image-based input data from the spectral image to forward it to the VLM. This input data includes mean spectral reflectance of image regions. The VLM then provides feature embeddings which can be used for few-shot classification. The prediction of a query image is deduced by the cosine similarity to feature embeddings of prototype classes. The model is evaluated on five spectral imaging tasks, including image classification, segmentation, and change detection.

Critical strengths
  • The model efficiently leverages multi-scale regions to support dense prediction tasks such as segmentation.
  • The RGB image generation from a median spectrum seems to work quite well.
Critical weaknesses
  • The results in Tab. 1 are not completely convincing. The model outperforms in only 2 out of 10 datasets. I understand that that HyperMAG does not leverage any domain-specific or task-specific fine-tuning, but it is still not clear what the advantage of HyperMAG is.
  • Questionable claim: "In supervised and few-shot settings (HC, HCD, and HOCC), where traditional deep learning architectures severely degrade due to label scarcity, HyperMAG consistently achieves superior performance". However, on those datasets, HyperMAG has the best performance in only 2 out of 6 cases.
  • Limited Novelty: The contribution of this paper is mainly the extraction of VLM-suitable input data (text + RGB image) from a hyperspectral image. The way how the text prompt is formed is also not that complex.
Questions for authors
  • What exactly is the advantage of HyperMAG in comparison to models such as HyperFree? Can you provide quantitative evidence for that?
  • Is the limited context size of the CLIP/Siglip models a challenge for the text prompting? Do you always use all bands for the text prompt? Are they all really necessary or do they rather confuse the VLM?
Reviewer_868UOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes HyperMAG for hyperspectral image analysis. It uses PCA and multi-scale superpixel segmentation to define regions. The mean spectrum of each region is converted into band-wise numerical text and dual-normalized color strips. A frozen Qwen3-VL-Embedding-2B model extracts features from both representations. These features are then concatenated.

Critical strengths

Applying current VLMs effectively to hyperspectral data remains challenging, making this a meaningful research direction. Reusing an existing general-purpose model avoids hyperspectral-specific pretraining and gradient updates for the target task, potentially offering an alternative for applications with limited annotation and development resources.

Critical weaknesses
  1. The novelty and scope of the work are insufficient to warrant publication at ICLR. The proposed approach primarily combines existing components into a processing pipeline. More importantly, the effectiveness and necessity of the individual processing steps have not been adequately validated. It is also unclear whether a consistent preprocessing protocol can be applied across different datasets, particularly regarding the choice of superpixel segmentation scales. The additional processing and parameter selection costs remain insufficiently characterized.

  2. The proposed method involves PCA, multi-scale segmentation, regional aggregation, text and spectral-strip construction, dual-branch encoding, and multi-scale fusion. All these operations incur computational costs, which should not be excluded merely because they do not involve gradient updates.

  3. The experimental design lacks rigor, and essential implementation details are missing. The experiments do not include representative methods designed for few-shot hyperspectral classification. Furthermore, substantial omissions in the experimental details limit reproducibility. Without publicly available code, the current description is insufficient to verify the method’s effectiveness or assess its full processing and deployment costs.

Questions for authors
  • What is the main methodological novelty, and which controlled ablations establish the necessity of each processing step?

  • What are the total preprocessing, adaptation, and inference costs compared with hyperspectral foundation models under matched hardware and supervision budgets?

  • How does the method compare with representative few-shot hyperspectral classification methods under the same 1–5-shot protocols?

  • Since superpixels may contain pixels from different semantic classes, how much accuracy is lost through regional averaging and uniform labeling, particularly for mixed-class regions, small targets, and class boundaries?

Reviewer_iPjDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents HyperMAG, a training-free framework for hyperspectral image (HSI) analysis using pretrained vision-language models (VLMs). The main idea is to connect the gap of modality between hyperspectral signals and the inputs of VLMs by encoding spectral signatures in two forms: structured textual descriptions of band values and trends, and visual representations of the magnitude and variations of spectra. The method uses multi-scale superpixel segmentation to extract regional spectra, which are then processed by a frozen VLM. The resulting text and image features are combined for downstream tasks without additional training. The authors evaluate HyperMAG on five HSI tasks: few-shot classification, one-class classification, target detection, anomaly detection and change detection. The paper addresses an important problem and presents an interesting approach to training-free HSI analysis.

Critical strengths

A practical and potentially useful perspective on training-free HSI interpretation. Broad empirical coverage with promising results in selected tasks.

Critical weaknesses

The empirical evidence does not adequately support the broad performance claims, and important reporting inconsistencies raise concerns about the reliability of the evaluation: The paper claims that HyperMAG performs better than existing methods across several tasks, but the results do not fully support this. Table 1, HyperMAG underperforms compared to the competing methods on both target-detection datasets and the anomaly-detection results reported as HyperMAG in Table 1 are same as in Table 9, despite using different detection strategies. Also multiple inconsistencies between some results in Tables 1 and 7. which raises questions about the validity of the experiments and comparisons.

The methodological novelty and the source of the reported improvements are not convincingly established. The method mainly combines existing techniques, including superpixel segmentation, spectral-to-text and image conversion, frozen VLM features, and feature concatenation and related studies have already explored such using vision-language models for hyperspectral analysis. Also more information on simpler spectral features and alternative encoders are missing, making hard to tell where improvements actually come from.

Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.