- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper tackles the problem of camera-agnostic representation learning via a proposed model that unifies rgb, multispectral, and hyperspectral images, which vary in their channel count a lot. The authors designed a model architecture with a dynamic and modality-adaptive tokenization before feeding the embedded tokens to a pretrained backbone such as Dinov3. For this, the model first extract channel-level embeddings, and then an image-level embedding by averaging the channel embeddings. The image descriptor then determines the representation capacity by predicting a cardinality integer K. Then, a dynamic token module maps these variable-cardinality components into fixed-dimensional patch tokens using a shared projection augmented by low-rank, component-conditioned residuals. The model is trained and evaluated on RGB-HSI segmentation task as well as on RGB-MSI tracking task. The model outperforms baseline models on these tasks.
Critical strengths
- Sophisticated workflow to adaptively embed RGB and HSI images with varying channel counts and information density.
- Strong results across the benchmark datasets.
Critical weaknesses
- Evaluation only on automotive domain. Do the results generalize?
- The model only supports supervised learning. Is the embedding module capable of learning reasonable cardinalities without labels too?
- Missing motivation of unnatural tokenization procedure. Is there related work that designed something similar? Furthermore, the mechanism is not clearly presented in the paper.
- Wavelength information is not incorporated into the model, which is essential for cross-sensor generalization (e.g., as shown in CARL).
Questions for authors
- How does the sole pretrained RGB backbone (Dinov3) compare to SpecUnify, where the backbone obtains the corresponding RGB images instead of the hyperspectral images?
- How can the RGB lifting be interpretated? Does a 4-channel image also require a lifting? This seems little arbitrary.
5 comments
W3. Motivation, mechanism, and related work for the token interface
The token interface connects the learned task representation to a pretrained backbone. Its components contain task features whose content is shaped by the input and cardinality. The interface therefore gives these components distinct mappings into token space while producing one fixed-dimensional token per spatial patch. Component-specific projection remains useful even when is fixed.
Conditional patch embedding has related precedents in spectral vision. DOFA (Xiong et al., 2024) uses a wavelength-conditioned hypernetwork to generate patch-embedding weights for heterogeneous input bands, mapping them into a common feature dimension. HyperFree (Li et al., 2025a) and HyperVision (Fu et al., 2026) construct channel-adaptive embeddings through wavelength-indexed weight dictionaries.
The connection is conditional construction of the input projection. The distinction is what is projected and what conditions the mapping: these methods use wavelength information to embed observed sensor bands, whereas SpecUnify projects generated task-representation components using the image descriptor, component coordinate, and learned cardinality. Its component-conditioned low-rank corrections augment a shared projection initialized from the pretrained RGB patch embedding. This couples representation generation with its mapping into backbone tokens.
The mechanism has three steps:
- Shared spatial projection. A common trainable patch projection, initialized from the RGB-pretrained patch embedding, maps the generated components into the backbone embedding space.
- Component-conditioned correction. A small network generates a low-rank correction for each component using the same image descriptor, component coordinate, and cardinality that condition its generation. The downstream objective can jointly shape the component content and its contribution to the token.
- Fixed-dimensional aggregation. The projected contributions are summed with the normalization in Eq. (12), producing one -dimensional token per spatial patch. At fixed resolution, both the backbone token count and embedding dimension remain independent of .
Experimental validation. At fixed , we compare the dynamic interface with a three-channel MLP readout followed by the original patch embedding. We also remove the low-rank residual from the full model while retaining the trainable shared projection.
Table E7. Token-interface ablations with DINOv3-L, LiDeRe, LoRA, and 50k joint-training updates. Accuracy is mIoU (%).
| Variant | Extra params (M) | HSI FLOPs (G) | Cityscapes | HSI-City v2 |
|---|---|---|---|---|
| B: Fixed + MLP projection | 0.4416 | 966.99 | 78.37 | 81.31 |
| C: Fixed + dynamic interface | 0.4372 | 749.31 | 78.68 | 85.44 |
| D: Without dynamic residual | 0.2945 | 749.21 | 78.55 | 81.43 |
| G: Full SpecUnify | 0.4416 | 749.31 | 79.01 | 86.88 |
Extra parameters exclude the common backend. B/C retain RGB lifting and representation generation and fix during training. D/G retain learned cardinality. FLOPs cover the complete inference model at 128 x 512 x 512 and controlled for all rows.
Results and interpretation. At fixed , the dynamic interface increases HSI mIoU from 81.31 to 85.44, with comparable trainable capacity. This supports directly mapping the generated representation into backbone tokens even without width learning. Removing the residual reduces HSI mIoU from 86.88 to 81.43, supporting the contribution of component-conditioned mappings beyond the shared projection. B/C compare complete readout designs; D/G form a component-removal ablation with the parameter difference shown explicitly. Together, these controls support both the direct token interface and its component-conditioned correction.
We have added the unification motivation and the connection to and distinction from DOFA before the equations in Section 2.3. The revised Figure 2 caption specifies the representation and token shapes and identifies exploration and neighboring-width probes as training-only operations.
References
Li, Jingtao, et al. "HyperFree: A channel-adaptive and tuning-free foundation model for hyperspectral remote sensing imagery." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.
Fu, Guanyiman, et al. "HyperVision: A Channel-Adaptive Ground-Based Hyperspectral Vision Pre-trained Backbone." arXiv preprint arXiv:2605.17286 (2026).
Xiong, Zhitong, et al. "Neural plasticity-inspired multimodal foundation model for earth observation." arXiv preprint arXiv:2403.15356 (2024).
W4. Wavelength information and cross-sensor transfer
SpecUnify uses observed channel content to construct task-oriented components and their token mappings, enabling a shared frontend to process heterogeneous spectral inputs without requiring wavelength metadata. Its intermediate components are learned task features and need not correspond to individual physical wavelengths. We evaluate whether this content-based interface can transfer from RGB training to HSI and MSI inputs.
Experimental validation. We have evaluated an RGB-only SpecUnify tracker directly on HOTC2023 and MUST and compared it with joint RGB-MSI-HSI training of the same architecture. The RGB-only model receives no HOTC2023 or MUST training data and no HSI or MSI task training.
Table E8. Tracking transfer without wavelength metadata (%), using SpecUnify with DINOv3-B/16 and LoRAT.
| Training source | HOTC2023 AUC | HOTC2023 DP20 | MUST AUC | MUST DP20 |
|---|---|---|---|---|
| RGB only | 57.10 | 76.60 | 47.23 | 68.98 |
| RGB + MSI + HSI | 67.70 | 86.70 | 63.20 | 80.00 |
Results and interpretation. The RGB-only model achieves 57.10/47.23 AUC on HOTC2023/MUST, demonstrating transfer to these spectral inputs without wavelength metadata. Joint spectral supervision increases AUC by 10.60/15.97 points. Thus, explicit wavelengths are not a prerequisite for the transfer demonstrated in these experiments.
The comparison evaluates training sources, rather than wavelength-aware and wavelength-free variants. It therefore does not determine the benefit of adding wavelengths or establish robustness to arbitrary unseen wavelength coverage or sensor responses. Wavelength information could complement the content-based representation. We have stated this scope explicitly in Appendix E.6, alongside the transfer results in Table 20. The limitations in Section 4 also distinguish channel-count compatibility from unseen-sensor generalization.
We have added a corresponding color-view baseline and an RGB-only tracking transfer comparison. Below, we explain the method's motivation and scope, present the relevant experimental evidence, and distinguish the demonstrated results from broader generalization claims.
W1. Applicability beyond automotive scenes
The evaluation covers two task settings: driving-scene segmentation and tracking across broader object and scene categories. Cityscapes and HSI-City v2 provide compatible semantic labels, allowing RGB-HSI segmentation with one label space and decoder. Tracking tests the shared frontend beyond that segmentation domain.
Evaluation evidence. One jointly trained tracker is evaluated on held-out sequences from LaSOT, HOTC2023, and MUST. LaSOT covers general objects; HOTC2023 includes indoor and outdoor sequences involving everyday objects and activities; MUST evaluates multispectral UAV tracking of targets including people, balls, and vehicles.
Interpretation. These evaluations support applicability beyond automotive scenes through tracking, while the segmentation evidence remains specific to driving datasets. They do not establish unrestricted domain generalization. We have clarified the shared segmentation label space and broader tracking coverage in Section 3.1. The limitations in Section 4 distinguish support for different channel counts from generalization to unseen sensors.
W2. Can cardinality be learned without task labels?
Our current scope is task-adaptive spectral representation learning. SpecUnify introduces task-adaptive representation cardinality across RGB, multispectral, and hyperspectral inputs: downstream supervision jointly shapes how many components are constructed, what they encode, and how they map into backbone tokens. For a given task, training dataset, and model configuration, the task objective guides the representation toward features and an operating width useful for prediction. The current formulation uses supervised segmentation and tracking objectives to provide this signal. The current experiments establish this capability under task supervision; learning useful cardinalities without labels remains to be evaluated.
Learning cardinality without task labels is an interesting extension. A self-supervised objective could provide a learning signal for jointly optimizing representation content and width. Exploring which objectives yield useful cardinalities and how the resulting representations transfer to downstream tasks is a promising direction for future work. We have clarified the supervised task-adaptive scope and identified self-supervised cardinality learning as a future extension in Section 4 (Discussion) of the revised manuscript.