acceptodds
Under review as a conference paper at ICLR 2027

SpecUnify: Learning Unified Spectral Representations for RGB, Multispectral, and Hyperspectral Vision

Abstract

RGB, multispectral, and hyperspectral images provide different spectral observations of the same visual world, yet modern vision models inherit the sensor's channel configuration as the basis for downstream representation. This couples how a scene is measured with how it is represented for a task, encouraging modality-specific projections, fixed spectral reductions, or separate models and limiting the sharing of representations and supervision across spectral modalities. We ask whether these roles can instead be separated: can heterogeneous spectral observations be mapped into a common task representation whose cardinality is learned from downstream supervision rather than prescribed by the sensor? We present SpecUnify, a unified framework for learning spectral representations across RGB, multispectral, and hyperspectral vision. SpecUnify learns a common spectral representation across modalities by lifting RGB observations into the shared space while directly incorporating multispectral and hyperspectral observations. Within this shared representation, it learns the cardinality of the task representation jointly with the downstream objective, rather than fixing it according to the sensor's channel configuration or a manually chosen spectral reduction. The resulting variable-cardinality representation is converted into fixed-dimensional tokens compatible with pretrained vision models, without reconstructing physical hyperspectral measurements or introducing separate cross-modal pretraining. We evaluate SpecUnify on semantic segmentation and visual tracking under within-modality, cross-modal, and joint RGB-hyperspectral training. The results show that different spectral observations can be processed through a common task representation while maintaining competitive performance across modalities and tasks, and that joint training can exploit complementary supervision from heterogeneous spectral observations. These findings suggest a broader principle for spectral vision: sensor dimensionality need not determine task representation, and heterogeneous spectral observations can instead be unified through a task-learned representation whose cardinality is determined by the downstream objective.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
spectral representation learning, multispectral vision, hyperspectral vision, cross-modal learning, adaptive representation learning, computer vision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2r8UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper tackles the problem of camera-agnostic representation learning via a proposed model that unifies rgb, multispectral, and hyperspectral images, which vary in their channel count a lot. The authors designed a model architecture with a dynamic and modality-adaptive tokenization before feeding the embedded tokens to a pretrained backbone such as Dinov3. For this, the model first extract channel-level embeddings, and then an image-level embedding by averaging the channel embeddings. The image descriptor then determines the representation capacity by predicting a cardinality integer K. Then, a dynamic token module maps these variable-cardinality components into fixed-dimensional patch tokens using a shared projection augmented by low-rank, component-conditioned residuals. The model is trained and evaluated on RGB-HSI segmentation task as well as on RGB-MSI tracking task. The model outperforms baseline models on these tasks.

Critical strengths
  • Sophisticated workflow to adaptively embed RGB and HSI images with varying channel counts and information density.
  • Strong results across the benchmark datasets.
Critical weaknesses
  • Evaluation only on automotive domain. Do the results generalize?
  • The model only supports supervised learning. Is the embedding module capable of learning reasonable cardinalities without labels too?
  • Missing motivation of unnatural tokenization procedure. Is there related work that designed something similar? Furthermore, the mechanism is not clearly presented in the paper.
  • Wavelength information is not incorporated into the model, which is essential for cross-sensor generalization (e.g., as shown in CARL).
Questions for authors
  • How does the sole pretrained RGB backbone (Dinov3) compare to SpecUnify, where the backbone obtains the corresponding RGB images instead of the hyperspectral images?
  • How can the RGB lifting be interpretated? Does a 4-channel image also require a lifting? This seems little arbitrary.
5 comments

We have added a corresponding color-view baseline and an RGB-only tracking transfer comparison. Below, we explain the method's motivation and scope, present the relevant experimental evidence, and distinguish the demonstrated results from broader generalization claims.

W1. Applicability beyond automotive scenes

The evaluation covers two task settings: driving-scene segmentation and tracking across broader object and scene categories. Cityscapes and HSI-City v2 provide compatible semantic labels, allowing RGB-HSI segmentation with one label space and decoder. Tracking tests the shared frontend beyond that segmentation domain.

Evaluation evidence. One jointly trained tracker is evaluated on held-out sequences from LaSOT, HOTC2023, and MUST. LaSOT covers general objects; HOTC2023 includes indoor and outdoor sequences involving everyday objects and activities; MUST evaluates multispectral UAV tracking of targets including people, balls, and vehicles.

Interpretation. These evaluations support applicability beyond automotive scenes through tracking, while the segmentation evidence remains specific to driving datasets. They do not establish unrestricted domain generalization. We have clarified the shared segmentation label space and broader tracking coverage in Section 3.1. The limitations in Section 4 distinguish support for different channel counts from generalization to unseen sensors.

W2. Can cardinality be learned without task labels?

Our current scope is task-adaptive spectral representation learning. SpecUnify introduces task-adaptive representation cardinality across RGB, multispectral, and hyperspectral inputs: downstream supervision jointly shapes how many components are constructed, what they encode, and how they map into backbone tokens. For a given task, training dataset, and model configuration, the task objective guides the representation toward features and an operating width useful for prediction. The current formulation uses supervised segmentation and tracking objectives to provide this signal. The current experiments establish this capability under task supervision; learning useful cardinalities without labels remains to be evaluated.

Learning cardinality without task labels is an interesting extension. A self-supervised objective could provide a learning signal for jointly optimizing representation content and width. Exploring which objectives yield useful cardinalities and how the resulting representations transfer to downstream tasks is a promising direction for future work. We have clarified the supervised task-adaptive scope and identified self-supervised cardinality learning as a future extension in Section 4 (Discussion) of the revised manuscript.

W3. Motivation, mechanism, and related work for the token interface

The token interface connects the learned task representation to a pretrained backbone. Its KK components contain task features whose content is shaped by the input and cardinality. The interface therefore gives these components distinct mappings into token space while producing one fixed-dimensional token per spatial patch. Component-specific projection remains useful even when KK is fixed.

Conditional patch embedding has related precedents in spectral vision. DOFA (Xiong et al., 2024) uses a wavelength-conditioned hypernetwork to generate patch-embedding weights for heterogeneous input bands, mapping them into a common feature dimension. HyperFree (Li et al., 2025a) and HyperVision (Fu et al., 2026) construct channel-adaptive embeddings through wavelength-indexed weight dictionaries.

The connection is conditional construction of the input projection. The distinction is what is projected and what conditions the mapping: these methods use wavelength information to embed observed sensor bands, whereas SpecUnify projects generated task-representation components using the image descriptor, component coordinate, and learned cardinality. Its component-conditioned low-rank corrections augment a shared projection initialized from the pretrained RGB patch embedding. This couples representation generation with its mapping into backbone tokens.

The mechanism has three steps:

  1. Shared spatial projection. A common trainable patch projection, initialized from the RGB-pretrained patch embedding, maps the generated components into the backbone embedding space.
  2. Component-conditioned correction. A small network generates a low-rank correction for each component using the same image descriptor, component coordinate, and cardinality that condition its generation. The downstream objective can jointly shape the component content and its contribution to the token.
  3. Fixed-dimensional aggregation. The projected contributions are summed with the normalization in Eq. (12), producing one DD-dimensional token per spatial patch. At fixed resolution, both the backbone token count NN and embedding dimension DD remain independent of KK.

Experimental validation. At fixed K=19K=19, we compare the dynamic interface with a three-channel MLP readout followed by the original patch embedding. We also remove the low-rank residual from the full model while retaining the trainable shared projection.

Table E7. Token-interface ablations with DINOv3-L, LiDeRe, LoRA, and 50k joint-training updates. Accuracy is mIoU (%).

VariantExtra params (M)HSI FLOPs (G)CityscapesHSI-City v2
B: Fixed KK + MLP projection0.4416966.9978.3781.31
C: Fixed KK + dynamic interface0.4372749.3178.6885.44
D: Without dynamic residual0.2945749.2178.5581.43
G: Full SpecUnify0.4416749.3179.0186.88

Extra parameters exclude the common backend. B/C retain RGB lifting and representation generation and fix K=19K=19 during training. D/G retain learned cardinality. FLOPs cover the complete inference model at 128 x 512 x 512 and controlled K=19K=19 for all rows.

Results and interpretation. At fixed KK, the dynamic interface increases HSI mIoU from 81.31 to 85.44, with comparable trainable capacity. This supports directly mapping the generated representation into backbone tokens even without width learning. Removing the residual reduces HSI mIoU from 86.88 to 81.43, supporting the contribution of component-conditioned mappings beyond the shared projection. B/C compare complete readout designs; D/G form a component-removal ablation with the parameter difference shown explicitly. Together, these controls support both the direct token interface and its component-conditioned correction.

We have added the unification motivation and the connection to and distinction from DOFA before the equations in Section 2.3. The revised Figure 2 caption specifies the representation and token shapes and identifies exploration and neighboring-width probes as training-only operations.

References

Li, Jingtao, et al. "HyperFree: A channel-adaptive and tuning-free foundation model for hyperspectral remote sensing imagery." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

Fu, Guanyiman, et al. "HyperVision: A Channel-Adaptive Ground-Based Hyperspectral Vision Pre-trained Backbone." arXiv preprint arXiv:2605.17286 (2026).

Xiong, Zhitong, et al. "Neural plasticity-inspired multimodal foundation model for earth observation." arXiv preprint arXiv:2403.15356 (2024).

W4. Wavelength information and cross-sensor transfer

SpecUnify uses observed channel content to construct task-oriented components and their token mappings, enabling a shared frontend to process heterogeneous spectral inputs without requiring wavelength metadata. Its intermediate components are learned task features and need not correspond to individual physical wavelengths. We evaluate whether this content-based interface can transfer from RGB training to HSI and MSI inputs.

Experimental validation. We have evaluated an RGB-only SpecUnify tracker directly on HOTC2023 and MUST and compared it with joint RGB-MSI-HSI training of the same architecture. The RGB-only model receives no HOTC2023 or MUST training data and no HSI or MSI task training.

Table E8. Tracking transfer without wavelength metadata (%), using SpecUnify with DINOv3-B/16 and LoRAT.

Training sourceHOTC2023 AUCHOTC2023 DP20MUST AUCMUST DP20
RGB only57.1076.6047.2368.98
RGB + MSI + HSI67.7086.7063.2080.00

Results and interpretation. The RGB-only model achieves 57.10/47.23 AUC on HOTC2023/MUST, demonstrating transfer to these spectral inputs without wavelength metadata. Joint spectral supervision increases AUC by 10.60/15.97 points. Thus, explicit wavelengths are not a prerequisite for the transfer demonstrated in these experiments.

The comparison evaluates training sources, rather than wavelength-aware and wavelength-free variants. It therefore does not determine the benefit of adding wavelengths or establish robustness to arbitrary unseen wavelength coverage or sensor responses. Wavelength information could complement the content-based representation. We have stated this scope explicitly in Appendix E.6, alongside the transfer results in Table 20. The limitations in Section 4 also distinguish channel-count compatibility from unseen-sensor generalization.

Reviewer_fFaVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents SpecUnify, a unified framework for adapting RGB, multispectral, and hyperspectral inputs to pretrained vision foundation models. Its main idea is to decouple the number of sensor channels from the cardinality of the task-oriented representation. SpecUnify first encodes an arbitrary set of spectral channels into a global descriptor, predicts an input-dependent representation cardinality (K), generates a set of latent spectral components, and maps them through a dynamic token interface into the token space of a frozen DINOv3 backbone. The framework is evaluated on semantic segmentation and object tracking using RGB, MSI, and HSI datasets.

Critical strengths
  • The paper introduces an interesting perspective on spectral representation learning.
  • The problem formulation is clear and addresses a meaningful practical challenge.
Critical weaknesses
  • The method-specific contribution is not sufficiently isolated from the strong pretrained backbone. The segmentation and tracking experiments are built on pretrained DINOv3 backbones together with LiDeRe and LoRAT, respectively. Table 4 includes an input-MLP baseline, and its improvement from 80.98 to 86.88 HSI mIoU indicates that the complete framework contributes beyond a minimal projection. However, the paper does not compare SpecUnify against sufficiently competitive, parameter-matched spectral adapters under the same backbone, task head, training data, and optimization budget. Comparisons with prior methods are also difficult to interpret when the backbone, pretraining, downstream framework, or training data differ. Consequently, the current results do not clearly establish how much of the reported performance is attributable specifically to SpecUnify. This is consequential because the paper’s main empirical claim concerns the effectiveness of its unified spectral interface, rather than that of the complete DINOv3-based system.

  • The experiments do not convincingly validate the paper’s central claims regarding adaptive cardinality and cross-modal joint training. In Table 4, the fixed-K=19K=19 variant with the dynamic token interface already achieves 85.44 HSI mIoU, compared with 86.88 for the full model. While the additional 1.44-point gain may be meaningful, the current evidence does not establish that it arises from input-dependent cardinality. The paper reports only the average evolution of KK, without showing its per-sample distribution or whether it varies meaningfully with scene content, spectral redundancy, or task difficulty. Moreover, the ablation does not disentangle the cardinality predictor from width exploration, which may itself provide substantial regularization. Key controls—such as random-width training without the predictor, a learned global KK, and comprehensively tuned fixed-KK baselines—are missing. The joint-training gain is similarly difficult to interpret: HSI performance improves from 85.26 to 86.88 mIoU after incorporating additional labeled RGB segmentation data, but the HSI-only and joint settings use different training schedules and data exposure. Thus, the gain may reflect additional supervision, longer training, or regularization rather than cross-modal knowledge transfer. Controlled comparisons with matched optimization steps, modality exposure, and compute are needed to substantiate both claims.

Questions for authors
  • How much of the reported performance can be attributed specifically to SpecUnify, rather than to the strong DINOv3 backbone and downstream framework? Please provide more controlled comparisons with competitive spectral adaptation baselines.

  • What evidence shows that the learned cardinality KK is genuinely input-adaptive and contributes meaningfully beyond fixed-cardinality or width-exploration alternatives? Does (K) vary systematically with scene content, spectral redundancy, sensor characteristics, or task difficulty?

7 comments

We have added matched frontend comparisons, separately trained fixed-width baselines, predictor/exploration controls, inference interventions, and a joint-supervision control.

Below, we explain the contribution being tested, describe the corresponding experiments, and interpret the results.

W1 / Q1. Contribution beyond the pretrained backbone and downstream framework

SpecUnify's frontend constructs task-representation components from heterogeneous spectral inputs and maps them directly into pretrained backbone tokens. Its contribution can be assessed by changing this frontend while holding the backbone, decoder, supervision, and training schedule constant. Complete-system benchmark comparisons provide context; the matched frontend experiments below provide the evidence for this specific contribution.

Experimental validation. We have added a nonlinear per-band adapter and a fixed-query spectral-attention adapter, both producing backbone tokens directly and using frontend parameter counts within 2% of SpecUnify's. Table E11 holds DINOv3-L/16, LiDeRe, LoRA, HSI-only training data, HS3 splits, 512 resolution, and the 30k-update schedule constant. The two adapters process all valid bands within each non-overlapping 16 x 16 spatial patch:

  • Nonlinear per-band projection. Each flattened band patch passes through a shared 256 -> 248 linear layer, GELU, and LayerNorm, followed by a residual 248 -> 248 -> 248 MLP with an intermediate GELU. The resulting embeddings are averaged over valid bands and linearly projected from 248 to 1024 dimensions to produce one backbone token per spatial patch.
  • Fixed-query spectral attention. Each band patch is embedded by a shared 256 -> 192 linear layer, GELU, and LayerNorm. Eight learned, input-independent queries attend across the band embeddings using eight attention heads and learned key/value projections; the attention weights depend on the input. A 192 -> 192 output projection and query residual are followed by a pre-normalized residual 192 -> 192 -> 192 MLP. The eight query features are averaged and linearly projected to 1024 dimensions.

These controls test nonlinear band encoding and content-dependent spectral aggregation without a three-channel bottleneck. Neither uses learned cardinality, wavelength embeddings, or channel-ID embeddings. Their linear layers and LayerNorm use default PyTorch initialization; the query parameters use truncated-normal initialization with standard deviation 0.02. Neither adapter loads pretrained frontend weights or copies the pretrained RGB patch kernel.

The added adapters use the common HSI reconstruction pipeline without additional raw-image normalization. Their training configuration uses global batch size 16, AdamW, frontend/LoRA learning rate 1e-4, decoder learning rate 1e-3, weight decay 1e-3, and 1,500 warmup updates. DINOv3 base weights remain frozen, with QKV LoRA rank 8 and scale 16. Table E11 also includes the input MLP and two CARL frontend adaptations, including one initialized with official CARL-SSL weights.

Table E11. Controlled spectral-frontend comparisons on HSI-City v2. Accuracy is mIoU (%).

FrontendExtra params (M)FLOPs (G)mIoU
Input MLP projection0.4602984.0580.98
Nonlinear per-band projection0.4427791.2678.05
Fixed-query spectral attention0.4346777.0880.32
CARL frontend adaptation14.69863139.2562.70
CARL-SSL frontend adaptation14.624910251.7168.04
SpecUnify0.4416749.3185.26

Extra parameters exclude the common backend. FLOPs cover a complete inference forward at 128 x 512 x 512, batch size one, with one multiply-add counted as two FLOPs. SpecUnify is profiled at controlled K=19 for architectural comparison; its mIoU uses the checkpoint's predicted widths. Both CARL adaptations use a nominal uniform wavelength grid. The CARL-SSL adaptation retains the pretrained patch-8 encoder and pools its outputs to the DINOv3 patch-16 grid.

Tracking frontend controls. We have also evaluated the input MLP, nonlinear per-band projection, and fixed-query spectral attention with the same DINOv3-B/16 backbone and LoRAT tracking framework as SpecUnify. All four models use the same joint training sources and sampling probabilities: 0.24 each for LaSOT, GOT-10k, COCO, and TrackingNet, 0.01 for HOTC2023, and 0.03 for MUST. They share the 200-epoch schedule with 98,304 image pairs per epoch, optimization and LoRA settings, and checkpoint-selection rule. Frontend parameter counts differ from SpecUnify by less than 0.14%.

Table E11a. Controlled tracking frontend comparisons with DINOv3-B/16 and LoRAT. Accuracy is reported in percent. Total trainable parameters include the frontend, backbone LoRA, and tracking-specific components.

FrontendFrontend params (M)Total params (M)LaSOT AUCLaSOT normalized precisionLaSOT DP20HOTC2023 AUCHOTC2023 DP20MUST AUCMUST DP20
Input MLP projection0.367013.352467.275.872.156.275.556.474.0
Nonlinear per-band projection0.366513.351965.373.669.859.178.457.274.8
Fixed-query spectral attention0.366313.351667.876.472.762.481.560.177.2
SpecUnify0.366513.351872.080.877.867.786.763.280.0

Table E11b. Tracking inference FLOPs (G) for the frontend controls in Table E11a.

FrontendRGBHSIMSI
Input MLP projection282.11284.62-286.72283.16
Nonlinear per-band projection161.67164.68-167.19162.93
Fixed-query spectral attention162.46164.64-166.45163.37
SpecUnify162.50162.02-162.35161.78

FLOPs count a complete forward pass for one 192 x 192 template and 384 x 384 search pair, including the backbone, unmerged LoRA, and tracking head. RGB/MSI inputs have 3/8 channels; HSI ranges cover 15-, 16-, and 25-channel inputs. SpecUnify uses controlled K=19 for both crops during cost profiling only; accuracy uses the evaluated checkpoints. One multiply-add counts as two FLOPs. Counts include convolution, matrix multiplication, and attention products, excluding normalization, interpolation, and elementwise operations.

To examine the token interface within the shared representation framework, Table E12 compares a three-channel MLP readout with the dynamic interface at fixed K=19 and separately removes the component-conditioned low-rank residual from the full model.

Table E12. Component controls with the same backend and 50k joint RGB-HSI training updates. Accuracy is mIoU (%).

VariantExtra params (M)CityscapesHSI-City v2
B: Fixed K + MLP projection0.441678.3781.31
C: Fixed K + dynamic interface0.437278.6885.44
D: Without dynamic residual0.294578.5581.43
G: Full SpecUnify0.441679.0186.88

B/C retain RGB lifting and representation generation and fix K=19. B uses an MLP to produce three channels followed by the original patch embedding; C uses the dynamic token interface. D removes only the low-rank residual while retaining the trainable shared projection. D/G retain learned cardinality.

Results and interpretation. Under matched HSI-only training, SpecUnify exceeds the nonlinear per-band adapter, fixed-query adapter, and input MLP by 7.21, 4.94, and 4.28 mIoU points, respectively. The relevant input-MLP comparison is 80.98 versus 85.26; 86.88 uses joint RGB-HSI supervision and is evaluated separately. These controls establish a frontend contribution beyond applying the same strong backend with the evaluated alternatives.

For tracking, SpecUnify exceeds fixed-query spectral attention, the strongest alternative in Table E11a, by 4.2, 5.3, and 3.1 AUC points on LaSOT, HOTC2023, and MUST, respectively. HOTC2023/MUST DP20 improves by 5.2/2.8 points. These gains hold with closely matched trainable capacity and the same backend, training sources, and optimization schedule. Together with the segmentation controls, they support the contribution of SpecUnify's frontend across both tasks and all three evaluated spectral modalities.

At fixed K, the dynamic interface improves HSI mIoU by 4.13 points over the three-channel readout with comparable trainable capacity. Removing the low-rank residual reduces HSI mIoU by 5.45 points. B/C compare complete readout designs, while D/G form a component-removal ablation with the parameter difference reported explicitly.

The CARL-SSL result provides an additional adaptation reference. Its nominal wavelengths and downstream integration limit the comparison to the evaluated frontend setting, rather than the complete CARL system. The tracking frontend controls in Tables E11a-E11b complement the RGB-only tracking control: the former compare input frontends under common joint supervision, while the latter tests the contribution of spectral supervision within SpecUnify.

We have added the matched segmentation frontend results to Section 3.3 (Table 5) and documented the segmentation adapter architectures and full comparison in Appendix D.3 (Table 10). We have also added the controlled tracking frontend comparisons, their training settings, FLOP-counting conventions, and result analysis to Appendix E.6 of the revised manuscript. Tables E11a-E11b above report these tracking results and computational costs.

W2(a) / Q2. Cardinality learning

SpecUnify learns task-adaptive representation cardinality through downstream supervision. For a given task, training dataset, and model configuration, K is optimized jointly with the representation and token mappings. Although the predictor is input-conditioned, this optimization can yield a stable operating width across inputs. The fixed-width and exploration controls support the benefit of this task-driven learning process.

Experimental validation. We evaluate three complementary aspects: the predictor's interaction with exploration, separately trained fixed-width alternatives, and the effect of width assignments after convergence.

First, Table E13 compares an input-conditioned predictor, a learned global scalar, and a fixed width, each with and without exploration. All variants retain trainable representation generation and the dynamic interface and use the same 50k joint-training updates.

Table E13. Width-learning controls under joint RGB-HSI supervision. All variants retain trainable representation generation and the dynamic interface and use 50k training updates. Accuracy is mIoU (%).

Width mechanismTraining explorationNon-exploratory widthCityscapesHSI-City v2
Input-conditioned predictorOnK(x)K(x)79.0186.88
Input-conditioned predictorOffK(x)K(x)78.8277.12
Learned global scalarOnKgK_g78.7581.65
Learned global scalarOffKgK_g78.6276.38
Fixed anchorOn1978.8677.78
Fixed widthOff1978.6885.44

KgK_g is learned from one scalar shared across inputs. Exploration uses the same decaying schedule and samples integer widths uniformly from 3 to 128. The fixed-anchor control has no trainable width predictor: it uses random widths on exploratory draws and 19 otherwise. Evaluation disables exploration.

Second, Table E14 evaluates whether selecting a different constant width can reproduce the full model's gain. We independently train models at K=19, 44, 66, and 88, covering the learned joint operating width and substantially wider alternatives.

Table E14. Separately trained fixed-width controls under joint RGB-HSI supervision. Accuracy is HSI-City v2 mIoU (%).

Width mechanismTraining explorationHSI-City v2
Fixed K=19Off85.44
Fixed K=44Off85.23
Fixed K=66Off85.16
Fixed K=88Off85.45
Full SpecUnify, learned K(x)K(x)On86.88

Each fixed-width model holds K constant throughout training and evaluation, with no width exploration or cardinality updates. All models retain the trainable generator and dynamic interface and use the same joint RGB-HSI data, DINOv3-L/16 backbone, LiDeRe decoder, LoRA configuration, 50k-update budget, and validation-based checkpoint-selection rule. The training comparisons use the same global batch size and input resolution, matching the number of backbone forward passes. K changes neither the backbone token count nor its embedding dimension; neighboring-width probes evaluate only the frontend and token interface, without additional backbone forward passes. Width-dependent computation is confined to the frontend and token interface.

Third, Tables E15 and E15a measure the executed width distributions and replace the predicted widths with a calibration-derived constant or shuffled assignments at inference.

Reviewer_y41QOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SpecUnify, a unified framework for handling spectral representations of RGB, multispectral (MSI), and hyperspectral (HSI) inputs. Instead of treating the number of sensor channels as the organization of downstream representations—for example, using a 3-channel patch embedding for RGB and a separate 128-channel front-end or compressing it into a 3-channel pseudo-RGB for HSI, the authors argue that the number of sensor channels, C_in, only describes "what was observed" and should not automatically determine "how many intermediate representation components the task requires." The relationship between the two is explicitly decoupled as C_in ≠ K, where K is the cardinality (number of components) of the task representation, jointly learned by the downstream task loss.

Critical strengths
  1. Spectral vision has long faced two paths: "one front-end per sensor" or "compressing into pseudo-RGB." This paper treats "sensor sampling format" and "task representation format" as two separate issues, offering an insightful perspective for spectral vision and aligning with the direction of using pre-trained RGB backbones to serve heterogeneous inputs.

  2. The methodology chain is complete and the documentation is of high quality. The paper not only proposes a variable cardinality representation but also provides a surrogate learning mechanism for discrete cardinality and an interface from variables to fixed tokens, achieving a closed loop.

  3. Ablation experiments provide compelling mechanistic evidence. With a fixed K=19, the dynamic token interface (85.44) has a 4.13 HSI mIoU higher than the parameter-matched three-channel MLP back-projection (81.31), directly supporting the principle of "not pushing the learned spectral representation back to the RGB image"; removing the low-rank residual loss of 5.45 points supports the principle that "different components require different token mappings." These two sets of control parameters match the backend, resulting in the shortest causal chain.

Critical weaknesses
  1. Methodology and Claims: The independent contribution of the learning base is not isolated. The variant with fixed K=19 has a C of 85.44 (86.88 for the full model), and Appendix E.3 also acknowledges that existing comparisons cannot distinguish the gains of learned predictors from random exploration. The most valuable addition is a budget-matched 2×2 comparison: with/without random exploration during training × with a fixed base of predictions/calibrated on the validation set during inference. It is also recommended to report the distribution of the prediction base on the test set (mean, variance, histogram); if the distribution is highly concentrated, per-sample prediction is approximately equivalent to learning a global constant, and the related claims should converge accordingly.

  2. "Full reconditioning is superior to truncation" lacks direct comparisons. This is a relatively distinctive design in the paper and should be compared with "fixed maximum component library + prefix/top-K truncation" under the same interface and budget. If it cannot be implemented, it should be described as an architectural design choice, not a proven advantage.

  3. The CARL adaptation does not use CARL-SSL pre-trained weights and replaces the calibration band center with a nominal uniform wavelength grid. The authors have disclosed these limitations; accordingly, the conclusions in the paper should be limited to "superior to the CARL adaptation under the settings presented in this paper," rather than generally superior to the CARL method.

  4. During joint tracking training, the combined sampling probability of the four RGB sources is 0.96, while HOTC2023 and MUST are only 0.01/0.03. This does not automatically mean the results are invalid, but it will affect the interpretation of spectral utilization capabilities. The most diagnostically valuable comparison with the R-row isomorphism in the segmentation is: zero-shot evaluation of the tracking model trained solely on RGB sources for HOTC/MUST—if it approaches SpecUnify's performance, the gain mainly comes from the strong RGB backbone; if it lags significantly, it supports the method. It is also recommended to explain the basis for the HOTC2023 training/test sequence partitioning.

Questions for authors

See weakness

3 comments

We have added cardinality-learning controls, separately trained fixed-width baselines, a prefix-bank comparison, a CARL-SSL frontend adaptation, and an RGB-only tracking control.

Below, we explain each design/comparison, present the corresponding validation, and interpret the results.

W1. Cardinality learning, random exploration, and inference widths

SpecUnify learns how many representation components to construct together with what those components encode, bringing representation-width selection into downstream task optimization. For a given task, training dataset, and model configuration, the task loss guides KK jointly with the component generator and token interface, so the representation width can adapt as its features are learned. Because KK conditions both whole-set generation and token projection, changing the width can reorganize the component content and its mappings into the backbone. This allows the model to learn an effective representation width within task training, without prescribing it from the sensor's measured channel count. Within this process, exploration trains the generator at alternative widths, while the task-gradient surrogate updates the width predictor using downstream supervision.

Experimental validation. We first compare three width mechanisms, each with and without exploration, under the same 50k joint RGB-HSI training updates. All retain input-conditioned representation generation and the dynamic token interface.

Table E1. Cardinality-learning controls under joint RGB-HSI supervision, with 50k training updates. Accuracy is mIoU (%).

Width mechanismTraining explorationInference widthCityscapesHSI-City v2
Input-conditioned predictor (G)OnPredicted K(x)K(x)79.0186.88
Input-conditioned predictor (F)OffPredicted K(x)K(x)78.8277.12
Learned global scalarOnLearned constant78.7581.65
Learned global scalarOffLearned constant78.6276.38
Fixed anchorOn1978.8677.78
Fixed width (C)Off1978.6885.44

The learned-global control optimizes one scalar shared across inputs. The fixed-anchor control has no trainable width predictor and uses K=19K=19 whenever exploration is inactive. Exploration replaces the mechanism's width with an integer sampled uniformly from 3 to 128 using the same decaying schedule; its probability reaches zero during training. Evaluation uses the inference rule listed in the table.

We also independently train fixed-width models at K=19, 44, 66, and 88 to test whether another constant width can reproduce the full model's performance.

Table E1a. Separately trained fixed-width controls under joint RGB-HSI supervision. Accuracy is HSI-City v2 mIoU (%).

Width mechanismTraining explorationHSI-City v2
Fixed K=19Off85.44
Fixed K=44Off85.23
Fixed K=66Off85.16
Fixed K=88Off85.45
Full SpecUnify, learned K(x)K(x)On86.88

Each fixed-width model holds K constant throughout training and evaluation, without width exploration or cardinality updates. All use the same joint RGB-HSI data, DINOv3-L/16 backbone, LiDeRe decoder, LoRA configuration, 50k-update budget, and validation-based checkpoint-selection rule. The generator and dynamic interface remain trainable. All variants use the same update budget, global batch size, and input resolution, with the same number of backbone forward passes. Neighboring-width probes operate only on the frontend and token interface.

Finally, we report the complete executed width distributions and evaluate predicted, calibration-constant, and shuffled widths for checkpoints trained with and without exploration.

Table E2. Executed width distributions. Exploration is disabled during evaluation.

DatasetTraining explorationMean KSD KVariance KK=19 (%)K=20 (%)K=88 (%)
CityscapesOn19.61670.48620.236438.3361.670.00
HSI-City v2On19.00950.09690.009499.050.950.00
CityscapesOff88.00000.00000.00000.000.00100.00
HSI-City v2Off88.00000.00000.00000.000.00100.00

Cityscapes statistics cover 1,500 sliding windows from 500 validation images; HSI-City v2 statistics cover 422 test images. SD and variance use population definitions. The frequency columns give the complete observed discrete distributions.

The predicted-width and calibration-constant evaluations under the two exploration settings form the requested 2x2 comparison. Shuffled-width inference provides an additional control.

Table E3. Inference interventions. Values are changes in mIoU percentage points relative to predicted-width inference.

Training explorationInference widthChange in CityscapesChange in HSI-City v2
OnPredicted K(x)K(x)0.00000.0000
OnCalibration-mean constant, K=19+0.0006-0.0003
OnWithin-dataset shuffled K(x)K(x)-0.0028-0.0001
OffPredicted K(x)K(x)0.00000.0000
OffCalibration-mean constant, K=880.00000.0000
OffWithin-dataset shuffled K(x)K(x)0.00000.0000

The constant is the rounded mean executed width on the 105-image HSI validation calibration set and is shared across both evaluation datasets. Shuffling preserves each dataset's width distribution. The complete representation and token interface are regenerated at the assigned width. Without exploration, all predictions equal 88, so the assignments are identical.

Results and interpretation. With exploration, the input-conditioned predictor exceeds the learned global scalar by 5.23 HSI mIoU points. Exploration without a predictor achieves 77.78, below fixed-K=19 training's 85.44. Random-width exposure alone therefore does not reproduce the full procedure's performance.

The four independently trained fixed-width models achieve 85.16-85.45 mIoU, a spread of 0.29 points. Full SpecUnify achieves 86.88, exceeding the highest score among these controls by 1.43 points. Increasing fixed K from 19 to 88 changes mIoU by only 0.01 points. The gain is therefore not recovered by selecting a larger constant width among the evaluated settings.

At inference, predicted widths concentrate at 19-20 with exploration and at 88 without it; replacing or shuffling them changes mIoU by less than 0.003 points. A calibrated constant width preserves the converged model's accuracy, while imposing K=19 from the start of training yields 85.44 rather than 86.88. Per-input width prediction provides no material accuracy benefit at inference for the evaluated checkpoints; the observed improvement concerns the joint width-and-representation training procedure. The generated component content and token mappings remain input-conditioned even when the integer width stabilizes.

We have added these controls to Section 3.3 and Appendix E.3 (Tables 15-18) and clarified the training-time interpretation in Section 2.2 and the Figure 2 caption. The revised Figure 3 caption distinguishes the fixed-checkpoint inference-width sweep from the separately trained fixed-width models in Table 16.

W2. Whole-set reconditioning versus truncation

Changing KK in SpecUnify changes both the normalized component coordinates and the cardinality embedding used by the generator and token interface. Each requested width therefore defines a newly conditioned set of components and mappings. A prefix construction instead keeps each bank entry's condition fixed and varies how many entries are retained.

Experimental validation. We have added a prefix-bank control that retains input conditioning, cardinality learning, exploration, and the dynamic token interface. In both representation generation and token projection, it fixes the component coordinates and cardinality embedding to a bank size of 128 and retains the first KK components. The bank is trained jointly on the same downstream data without additional pretraining.

Table E4. Whole-set reconditioning versus prefix selection. Both models use DINOv3-L/16, LiDeRe, LoRA, and 50k joint-training updates. Accuracy is mIoU (%).

Component constructionExtra trainable params (M)RGB FLOPs (G)HSI FLOPs (G)CityscapesHSI-City v2
Fixed bank + prefix selection0.4416753.71749.3178.0677.16
Whole-set reconditioning0.4416753.71749.3179.0186.88

Extra parameters are relative to the shared backend. FLOPs cover the complete inference model at controlled K=19, RGB/HSI input shapes 3/128 x 512 x 512, and batch size one. They compare architectures at a reference width, rather than each checkpoint's predicted-width cost. The prefix control computes only selected components without materializing unused bank entries.

Results and interpretation. Whole-set reconditioning improves RGB/HSI mIoU by 0.95/9.72 points at matched trainable capacity and training-update budget. This directly supports reconditioning the representation and token mappings for the requested width over the evaluated prefix-bank construction. The comparison tests the combined generation-and-interface design; it does not establish superiority over every truncation or top-K method. We have added the control as row H in Table 4, analyzed it in Section 3.3, and described its construction in Appendix E.4.

W3. Scope of the CARL comparison

The CARL comparison evaluates spectral frontends integrated with a common downstream backend. Its interpretation depends on the frontend initialization, wavelength inputs, and integration choices. We therefore distinguish this adaptation setting from the complete pretrained CARL system.

Experimental validation. We have added the official CARL-SSL spectral frontend alongside the DINOv2-initialized adaptation, holding DINOv3-L, LiDeRe, LoRA, HSI-only supervision, 512 resolution, and the 30k-update budget constant.

Table E5. HSI-City v2 segmentation (%), using 512 resolution and 30k training updates. Initialization refers to the spectral frontend.

MethodFrontend initializationTrainable params (M)FLOPs (G)Micro accuracyMacro accuracyMacro F1mIoU
CARL frontend + LiDeReDINOv215.823139.2596.0868.4871.0762.70
CARL frontend + LiDeReCARL-SSL15.7510251.7197.3372.3575.2968.04
SpecUnify + LiDeReNo separate spectral pretraining1.56749.3199.0988.2289.1185.26

Parameters include the frontend, backbone LoRA, and decoder. FLOPs cover a complete inference forward at 128 x 512 x 512 and batch size one; one multiply-add counts as two FLOPs. SpecUnify is profiled at controlled K=19, while its mIoU uses the checkpoint's predicted widths.

Both CARL adaptations use the same nominal grid of 128 uniformly spaced wavelength centers over 449-955 nm because calibrated per-band centers are unavailable. The CARL-SSL adaptation retains its pretrained patch-8 encoder and pools the outputs to the DINOv3 patch-16 grid.

Results and interpretation. The CARL-SSL-initialized adaptation achieves 68.04 mIoU, compared with 62.70 for the DINOv2-initialized adaptation. SpecUnify achieves 85.26 under the common downstream setting. This comparison supports SpecUnify relative to these CARL frontend adaptations with the specified backend and wavelength inputs. We have added the SSL result to Table 1 and specified the scope of the comparison in Section 3.2 and Appendix D.2.

W4. Contribution of spectral supervision and HOTC2023 partitions

Sampling probabilities determine how often each source is presented during training; they do not by themselves measure how much each source contributes to the learned tracker. To assess the contribution of HSI/MSI supervision, the relevant comparison uses the same SpecUnify architecture trained on RGB sources alone and evaluates it directly on the spectral benchmarks.

Experimental validation. We have trained an RGB-only SpecUnify tracker on the four RGB sources and evaluated it on HOTC2023 and MUST without HSI or MSI task training. It uses the same frontend, DINOv3-B/16 backbone, LoRAT head, and 192/384 template/search sizes as the jointly trained tracker. Both models follow the tracking schedule in Table 7: 200 epochs with 98,304 image pairs per epoch.

Table E6. Effect of training sources on tracking (%). R, M, and H denote RGB, MSI, and HSI supervision.

SourceLaSOT AUCLaSOT normalized precisionLaSOT DP20HOTC2023 AUCHOTC2023 DP20MUST AUCMUST DP20
R72.180.878.057.176.647.2368.98
R+M+H72.080.877.867.786.763.2080.00

For HOTC2023, we retain the sequence-level training and validation partitions from the official release. The released validation partition serves as our final evaluation set and contains 87 sequences: 46 VIS, 30 NIR, and 11 RedNIR.

Results and interpretation. Joint training improves HOTC2023/MUST AUC by 10.60/15.97 points and DP20 by 10.10/11.02 points, while LaSOT AUC remains similar. Spectral supervision therefore contributes substantial additional performance despite its combined 4% sampling probability. This comparison tests the training-source contribution within SpecUnify; it does not compare alternative tracking frontends.

We have summarized the source comparison in Section 3.2 and reported the full results in Appendix E.6 (Table 20). Appendix C.1 now specifies the official HOTC2023 sequence partitions and the role of the released validation partition as our final evaluation set.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of rebuttal and revision

Dear Area Chair,

Thank you for considering SpecUnify and for the reviewers’ detailed feedback.

Summary

SpecUnify learns the width of its spectral representation from downstream task supervision. With the same segmentation architecture and width exploration, the mean predicted width settled near 41 with HSI-only training, 66 with RGB-only training, and 19 with joint training. Learning the width during joint training improved HSI segmentation. At inference, fixing the width at 19 preserved HSI accuracy while the generated components remained image-specific. Under identical supervision, adding RGB lifting also improved HSI predictions, even though HSI inputs bypass that layer.

Strengths highlighted by the reviewers

  • Problem formulation. Reviewer y41Q described the distinction between the number of bands a sensor measures and the number of components a task learns to use as “an insightful perspective for spectral vision”. Reviewer fFaV highlighted the clarity and practical relevance of this formulation.
  • A unified method for heterogeneous inputs. Reviewer y41Q highlighted how the method connects discrete width learning, component generation, and projection to fixed-dimensional tokens. Reviewer 2r8U emphasized its ability to embed RGB and HSI images with different channel counts and information densities.
  • Ablation evidence and benchmark performance. Reviewer y41Q singled out the component-specific token mapping. Removing its low-rank correction lowered HSI-City v2 mIoU by 5.45 points (Table 4). Reviewer 2r8U highlighted the strong results across the benchmark datasets.

Key questions raised in review

Key questionEvidence and conclusion
When does learning the spectral width help? (y41Q, fFaV)On HSI-City v2, joint training reached 86.88 mIoU, versus 85.45 for the best of four fixed widths. With the same width-exploration schedule, a learned global width reached 81.65. Fixing K=19K=19 from the first update cost 1.44 points, while fixing it after training cost only 0.0003 points. The measured gain comes from learning width and representation together during training. Per-image width selection had negligible impact at inference in this setting (Tables 15–18).
Does RGB representation contribute beyond the additional labels? (fFaV, 2r8U)With identical labeled data and training schedules, enabling lifting only on the RGB branch raised HSI mIoU from 79.59 to 86.88. HSI images bypass lifting in both models. Separately, enabling the RGB task loss at matched updates and expected modality exposure raised HSI mIoU from 79.20 to 86.88. Thus, both RGB supervision and how the model encodes RGB affect HSI prediction (Tables 4 and 19).
What does SpecUnify add to DINOv3? (fFaV, 2r8U)With full HSI inputs and the same backbone, decoder, and training budget, SpecUnify reached 85.26 mIoU versus 80.98 for the strongest alternative frontend. Native DINOv3 on HSI-derived color views of the same scenes reached 85.83. The frontend advantage is established for the matched full-HSI comparison (Tables 5 and 11).
Do whole-set generation and direct tokenization each help? (y41Q, 2r8U)Generating all components at the requested width improved HSI mIoU by 9.72 points over truncating a fixed bank. Holding K=19K=19 fixed, mapping the components directly to backbone tokens improved HSI mIoU by 4.13 points over a three-channel readout (Table 4).
How far does the evidence extend across domains and sensors? (y41Q, 2r8U)Including spectral data in tracker training alongside RGB data improved AUC by 10.60 points on HOTC2023 and 15.97 on MUST, extending the evidence beyond driving-scene segmentation to general-object and UAV tracking. We documented the official HOTC2023 split and added the pretrained CARL-SSL frontend comparison using nominal wavelengths. Segmentation beyond driving scenes, transfer to unseen sensors, and learning width without labels remain open (Table 1, Appendix C.1, Table 20).

Discussion outcome

On 19 November, fFaV acknowledged the stronger empirical results and retained two concerns.

Our follow-up located the cardinality gain in training and showed that RGB encoding affects HSI accuracy under identical supervision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.