acceptodds
Under review as a conference paper at ICLR 2027

Mask Components, Not Bands: Rethinking Masked Autoencoding for Hyperspectral Imagery

19 views

Abstract

Hyperspectral imagery (HSI) captures hundreds of contiguous wavelengths per pixel, enabling material identification through spectral differences, but labeled data is scarce. Masked autoencoder (MAE) pretraining has emerged as the dominant self-supervised approach for learning HSI representations from unlabeled data. Existing HSI MAE methods adopt input-space masking from RGB imagery and video, a recipe designed around spatial redundancy. HSI distributes its redundancy differently. Neighboring pixels are often non-redundant, while adjacent wavelengths are so highly correlated that masked bands remain linearly recoverable from visible ones until most of the spectrum is removed. Input-space methods therefore rely on masking ratios of up to 90%, which make reconstruction difficult only by leaving the encoder a sparse view of each pixel's spectrum. We propose CoMask, which exploits this spectral correlation by projecting HSI pixels onto a small number of basis components that span all wavelengths. CoMask masks in this component basis, where zeroing a component removes its contribution from every wavelength. The encoder therefore sees every wavelength at reduced fidelity, with degradation controlled by how many components are masked. Reconstruction difficulty then depends on how much spectral signal is removed rather than on which wavelengths are hidden. That dependency aligns the task with the low-dimensional structure of HSI data. Under linear probing on Houston2018 at ViT-Tiny, CoMask exceeds every input-space masking configuration in our sweep, leading the strongest by 2.7 points of overall accuracy averaged over three pretraining seeds. On EnMAP-DFC the lead appears under finetuning rather than probing, at 2.0 points of macro accuracy over the strongest configuration in that sweep. Pretrained on the foundation-scale Spectral Earth corpus, CoMask outperforms an input-space baseline of the same architecture and recipe on all six downstream segmentation tasks, by 0.9 to 2.6 micro-IoU.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
hyperspectral imaging, masked autoencoders, self-supervised learning, representation learning, masking strategy, spectral redundancy, remote sensing, vision transformers, Earth observation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_brbZOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

CoMask addresses strong correlations between neighboring hyperspectral bands by moving self-supervised masking from input space to spectral components. It fits and fixes a spectral basis on unlabeled pretraining data, removes selected component coefficients at chosen spatial locations, and recombines the remaining components into a degraded input retaining all wavelength channels. An encoder–decoder reconstructs the original normalized spectra.

Critical strengths
  1. A well-motivated use of low-dimensional spectral structure for self-supervised learning. Figure 1 directly supports the motivation that masked hyperspectral bands remain readily predictable through simple cross-band relationships. CoMask uses spectral decomposition to redefine what information the pretext task removes while retaining wavelength coverage. The decomposition determines the input corruption, while the reconstruction target remains the original spectrum. This simple design translates domain knowledge into a meaningful pretraining strategy that integrates readily into existing masked autoencoding frameworks.

  2. Controlled experiments across data scales support the method’s practical value. The evaluation spans a single airborne scene, a satellite pretraining collection, and SpectralEarth with approximately 8.8 billion pixels. In Table 3, CoMask improves over the in-house MaskedSST-Tube baseline on all six tasks by 0.94–2.58 percentage points in micro-IoU, using the same ViT-Tiny backbone, pretraining data, training budget, and segmentation head. This controlled comparison is my primary evidence supporting acceptance. Table 4 further investigates the gains through matched input degradation and controls using random bases, band-permuted bases, and noise perturbations.

Critical weaknesses
  1. Dominant masking lacks a clearly specified treatment of basis scaling. The dominant strategy ranks components by their mean coefficient magnitude, interpreting larger coefficients as stronger spectral contributions. However, NMF permits reciprocal rescaling of a basis vector and its coefficients without changing the reconstruction, potentially changing this ranking. Unless basis scales are standardized, coefficient magnitude need not reflect actual spectral contribution. The paper describes input normalization but does not clearly specify corresponding normalization of the basis vectors. This leaves the interpretation and scaling sensitivity of dominant masking unresolved, particularly for the Houston and EnMAP experiments that use this strategy.

  2. The roles of spectral component learning and spatial completion remain insufficiently distinguished. CoMask shares one component mask across all corrupted locations within a sample while retaining intact spectra elsewhere. Appendix D shows that corrupting every location places most Houston configurations and all tested EnMAP configurations below the randomly initialized linear-probe baseline, suggesting that intact spectra are important to the current method. Although the authors report this dependence, its mechanism remains unclear. Because the same components are missing across corrupted locations, intact spectra may provide essential spatial cues for their recovery. The analysis does not yet distinguish learning relationships among spectral components from spatial completion using intact spectra, or establish their respective contributions. The standard MAE practice of retaining visible inputs does not, by itself, explain this dependence.

Questions for authors
  1. Are basis vectors normalized, with compensating coefficient rescaling, before dominant selection? Is selection invariant to reciprocal basis–coefficient rescaling? If not, why should mean coefficient magnitude represent spectral contribution?
  2. Spatial masking: Why share one component mask across spatial locations? Is there a comparison with independently sampled masks at comparable information retention that could clarify the role of intact spectra and the performance drop when every location is corrupted?
  3. Scope of transfer: How would the spectral basis, component count, and masking strategy adapt to different band counts, wavelength ranges, or missing bands? Would this require pretraining on target data, or could existing encoder weights be reused directly? Please distinguish the reusability of the masking strategy from cross-sensor transfer of pretrained representations.
  4. Suggested mechanism controls: Have you compared reconstructing the original spectra from full low-rank approximations without component masking, and reconstructing the full low-rank approximations from component-masked inputs? These controls could help separate the contributions of component completion and decomposition-residual recovery.
3 comments
AuthorsOpenReview ↗

Response to Reviewer brbZ (1/3): Q1

Thank you for the careful reading and for the questions. We ran the controls you proposed: a new selection rule for Q1, two masking arms for Q2, and both targets from Q4, all on Houston2018 at ViT-Tiny with 8 pretraining seeds at the paper's input NRMSE of 0.79. In every arm CoMask exceeds the input-space control.

Q1 (W1).

Are basis vectors normalized, with compensating coefficient rescaling, before dominant selection? Is selection invariant to reciprocal basis–coefficient rescaling? If not, why should mean coefficient magnitude represent spectral contribution?

No to both, and thank you for catching this. The finding changes how the paper describes dominant selection, and nothing else. CoMask's contribution is to mask in a fitted component basis instead of in input space, and every result in the paper was produced by one fixed rule on one released basis, so every number is reproducible as printed. The rule itself does not need to identify the components with the most physical signal. It needs to be consistent, so that the degradation is reproducible and its amount is set by the configuration (c, s), not by the rule. Dominant by magnitude is one rule the harness implements. Random selection is another (used for Spectral Earth), scale-invariant dominant is now a third, and further rules can be added for a given application without changing the method.

The gain does not depend on the rule. A scale-invariant version, ranking by |a_k| times the norm of basis vector k, exceeds the input-space control by 4.50 percentage points at the same input NRMSE (0.790 against 0.796):

Houston2018, ViT-Tiny, 8 pretraining seedsLP-OA
Dominant by coefficient magnitude (paper)56.16 ± 3.00
Dominant by contribution (scale invariant)53.95 ± 2.08
Input-space control49.46 ± 2.86

The two rules select the same components in only 13.4% of Houston tiles. The 2.21-point difference between them is smaller than the seed standard deviation (paired t = -1.4, 3 of 8 seeds positive), and the invariant rule exceeds the control at 7 of 8 seeds.

On the paper's wording, the NMF basis vectors are not normalised (norms span a factor of 9.5 on Houston2018 and 36 on EnMAP), so the paper's rule ranks by coefficient magnitude under the solver's scaling, and that ranking is not invariant to reciprocal rescaling. Coefficient magnitude therefore does not represent spectral contribution. The revision describes the rule as ranking by coefficient magnitude under the released basis, and reports the invariant variant beside it.

AuthorsOpenReview ↗

Response to Reviewer brbZ (2/3): Q2

Q2 (W2).

Spatial masking: Why share one component mask across spatial locations? Is there a comparison with independently sampled masks at comparable information retention that could clarify the role of intact spectra and the performance drop when every location is corrupted?

Sharing one mask within a sample is a design choice, not a CoMask requirement. For the results reported in the paper, every sample draws its own component mask, but within a sample, every location removes the same components. However, CoMask supports independent masking at each location too. We ran that version at matched input NRMSE (0.803 against 0.796). The difference is within seed noise, so CoMask works with either rule:

Houston2018, ViT-Tiny, 8 pretraining seedsLP-OA
Shared mask per sample (paper)56.16 ± 3.00
Independent mask per location55.43 ± 2.62
Input-space control49.46 ± 2.86

Independent masks are 0.73 points below the shared mask (3 of 8 seeds above it) and 5.97 points above the input-space control (7 of 8 seeds).

On the role of intact spectra, your weakness raises three points.

Intact spectra may provide essential spatial cues for their recovery.

They do. From a corrupted pixel alone, a linear predictor recovers CoMask's removed content at R² 0.66. With the mean of the tile's intact pixels and the nearest intact pixel, it reaches 0.94.

Because the same components are missing across corrupted locations.

This is not the reason. Independent masks remove different components at each location and score the same as the shared mask (table above), so the completion does not depend on neighbours missing the same components.

The analysis does not yet distinguish learning relationships among spectral components from spatial completion using intact spectra.

The full-corruption runs separate them. With every location corrupted (s = 1.0, 3 pretraining seeds), independent masks probe at 47.94 ± 2.04 and the shared mask at 40.96 ± 6.09, against a random-init floor of 46.45 and 58.22 ± 1.19 for the fixed configuration with intact locations. Without intact spectra, neither pattern learns a useful representation, so spatial completion is necessary for the pretext task. It is not what distinguishes CoMask from input-space masking. The input-space baselines keep the same fraction of locations intact, and at the same NRMSE CoMask exceeds them by 6 to 7 percentage points.

These runs rule out two explanations of the mechanism without establishing it. The dependence on intact spectra is real, it does not come from the shared mask pattern, and reconstruction is not the bottleneck, since the full-corruption models reconstruct well (validation loss 0.0047 and 0.0075, against 0.0027 for the fixed configuration) and their probe accuracy is at or below the floor. Why intact anchors are needed for the representation to transfer remains open, as you note, and the revision adds both tables to Appendix D and states this.

AuthorsOpenReview ↗

Response to Reviewer brbZ (3/3): Q3, Q4

Q3.

Scope of transfer: How would the spectral basis, component count, and masking strategy adapt to different band counts, wavelength ranges, or missing bands? Would this require pretraining on target data, or could existing encoder weights be reused directly? Please distinguish the reusability of the masking strategy from cross-sensor transfer of pretrained representations.

This is an important question for the next steps of CoMask. The goal of this paper was to show the concept and the harness for masking in component space, so every experiment was built around that. The scope of transfer splits into the two parts you name, and they have different answers.

The masking strategy transfers as it is. One configuration (NMF, K = 8, dominant, c = 0.75, s = 0.70) runs on Houston2018 (48 bands, visible and near infrared) and on EnMAP (200 bands, 420 to 2450 nm), and the same harness runs on Spectral Earth with K = 10 and random selection. What changes between sensors is the basis, because the basis is tied to the band grid. A new sensor needs a basis fitted on its own unlabeled pixels, which costs a small share of pretraining (Section 4), and K can be set from reconstruction error on those pixels, as we did for Spectral Earth. For missing bands, the basis can be restricted to the available bands and the coefficients recomputed. We have not tested that case.

Pretrained representations do not transfer across sensors, and that is a property of the encoder, not of the masking. The weights are tied to the tokenizer's band grid, as in any spectral ViT, so a new sensor needs pretraining on its own data. We do not test cross-sensor reuse of weights and make no claim about it. Within one sensor, the paper does test transfer of the pretrained representation to a new region (EnMAP, pretrained on 90 European scenes and finetuned on Mexico City), and CoMask exceeds the input-space baseline there at every encoder scale (Table 2). The revision states this distinction in Section 7.

Q4.

Suggested mechanism controls: Have you compared reconstructing the original spectra from full low-rank approximations without component masking, and reconstructing the full low-rank approximations from component-masked inputs? These controls could help separate the contributions of component completion and decomposition-residual recovery.

We ran both, and they separate the two contributions cleanly. Component completion accounts for nearly all of CoMask's gain, and recovery of the decomposition residual accounts for almost none of it:

Houston2018, ViT-Tiny, 8 pretraining seedsInputTargetLP-OA
CoMask-NMF (paper)component maskedoriginal spectrum56.16 ± 3.00
Low-rank targetcomponent maskedlow-rank approximation54.52 ± 2.20
Residual recovery onlyfull low-rank approximation, nothing maskedoriginal spectrum50.01 ± 3.14
Input-space controlzeroed tokensoriginal spectrum49.46 ± 2.86

The low-rank target removes the residual from the task. The encoder still has to fill in the masked components, and it probes at 54.52, which is 1.64 points below the paper's version and within seed noise (3 of 8 seeds above it). It is still 5.06 points above the input-space control at 8 of 8 seeds. The second control removes the masking and keeps only the residual. The encoder gets an almost intact spectrum (NRMSE 0.012) and has to reconstruct what the rank-8 approximation dropped. That gives 50.01, the same as the input-space control (+0.55, 6 of 8 seeds). So the residual on its own does not teach anything useful, and seeing a low-rank-filtered spectrum does not help either.

This confirms what we claim the pretext task does. The gain comes from removing components and training the encoder to reconstruct them. Whether the target has the residual in it or not changes little. Reviewer YJ18 asked for a related control, low-rank input with ordinary band masking. That one gives 48.82 ± 0.67, again level with the input-space control, so both sets of controls give the same result. We will add all three arms to Table 4.

Thank you again for the questions. They led to controls that make the attribution stronger than it was in the submission.

Reviewer_aSDkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoMask, a component-based masking strategy for self-supervised hyperspectral image (HSI) representation learning, addressing the high spectral redundancy that limits the effectiveness of conventional input-space masked autoencoding. Instead of directly masking spectral bands or spatial-spectral patches, CoMask decomposes pixel spectra into a low-dimensional component basis using methods such as PCA, NMF, and SPICE, selectively masks component coefficients, and reconstructs the original spectra from the partially recovered inputs. This design preserves full wavelength coverage while introducing structured spectral degradation, without modifying the underlying encoder-decoder architecture. The authors evaluate CoMask on Houston2018, EnMAP-DFC, and Spectral Earth using linear probing, finetuning, and downstream semantic segmentation tasks. Experimental results demonstrate improvements over conventional input-space masking baselines under several evaluation settings, particularly in low-label scenarios. Additional ablation studies investigate the effects of decomposition methods, masking configurations, spectral basis selection, and matched input degradation to further assess the effectiveness of the proposed strategy.

Critical strengths
  1. The paper identifies an important limitation of conventional masked autoencoding for hyperspectral imagery, namely that strong correlations between adjacent spectral bands can make masked information easily recoverable. The authors support this observation through linear recoverability analysis and propose component-level masking as an alternative.

  2. CoMask introduces component-based masking without requiring modifications to the underlying encoder-decoder architecture. The method can accommodate different spectral decomposition techniques, including PCA, NMF, and SPICE, making it relatively straightforward to integrate into existing masked autoencoding frameworks.

  3. The authors evaluate CoMask on three hyperspectral benchmarks, covering classification and semantic segmentation tasks under different evaluation protocols. The results demonstrate improvements over input-space masking baselines in several settings, particularly under limited-label supervision.

Critical weaknesses
  1. The core idea of CoMask relies on established spectral decomposition techniques, such as PCA, NMF, and SPICE, followed by coefficient masking and spectral reconstruction. Similar concepts involving transformed-domain masking have already been explored in previous studies.

  2. Direct performance comparisons with these methods are missing, making it difficult to assess whether CoMask provides a substantial improvement over existing transformed-domain masking approaches.

  3. Although the paper attributes CoMask's effectiveness to structured spectral component masking, the experimental evidence does not consistently support this explanation across datasets. In particular, the ablation results on EnMAP-DFC show that random non-negative bases and simple white-noise corruption achieve performance comparable to CoMask under matched input degradation.

  4. Matching input NRMSE does not necessarily ensure equivalent information loss across different corruption strategies.

  5. Although CoMask demonstrates improvements in several experimental settings, its advantages are not consistently maintained across evaluation protocols and model architectures. For example, on Houston2018 with ViT-Tiny, CoMask-NMF achieves lower full-label finetuning accuracy than the strongest input-space masking baseline. On EnMAP-DFC, its macro-accuracy advantage at ViT-Tiny also disappears under class-weighted cross-entropy.

  6. The Conv3D experiments show that CoMask does not consistently outperform supervised training from scratch, while cross-sensor generalization remains unexplored.

Questions for authors

Please responds to my concerns in the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer aSDk (1/3): W1 to W3

Thank you for the review. We respond to each weakness below. Four of the six (W3 to W6) concern limitations the paper itself reports, and for those we say what the paper reports and what we have added.

W1.

The core idea of CoMask relies on established spectral decomposition techniques, such as PCA, NMF, and SPICE, followed by coefficient masking and spectral reconstruction. Similar concepts involving transformed-domain masking have already been explored in previous studies.

W2.

Direct performance comparisons with these methods are missing, making it difficult to assess whether CoMask provides a substantial improvement over existing transformed-domain masking approaches.

Section 2 addresses this, and we restate the position here. CoMask uses standard decompositions on purpose. The contribution is not a new decomposition but what the mask removes, and that is what lets CoMask run inside an existing masked autoencoder without changing the encoder or decoder, as your second strength notes. Section 2 discusses seven transformed-domain masking methods (PMAE, MFM, WaveMAE, SSLSM, HSIMAE, KARMA, SFMIM) and states where CoMask differs from each: a basis fitted to the pretraining data, a mask on selected coefficients, and reconstruction of the original spectrum.

On direct comparison, we do compare the bases these methods mask in. SFMIM masks a fixed frequency basis, WaveMAE a fixed wavelet basis, and SSLSM and HSIMAE a PCA basis fitted to the spectrum. We ran each of these bases inside the same harness as CoMask, with the encoder, tokenizer, data, recipe and measured input degradation held fixed and only the basis swapped. CoMask's fitted NMF basis gives the best representation of all of them, at 8 pretraining seeds (Table R1 in the general response, and Table 1 for PCA). To our knowledge, this is the first comparison of masking bases under one harness at matched degradation for hyperspectral data.

The harness is released with the paper. It takes any basis as one more arm, with everything else held fixed, and that is the comparison that shows whether component masking works and in which basis. Running SSLSM or HSIMAE as published would not. Their encoder, input pipeline and pretraining data all differ from ours and from each other, so a difference in score could not be assigned to the masking. SFMIM has no public code.

W3.

Random non-negative bases and simple white-noise corruption achieve performance comparable to CoMask under matched input degradation.

This treats two of our control arms as competing methods, and they are not. We built the random non-negative basis and the white-noise corruption for the attribution experiment in Section 6. Both keep every wavelength, both are calibrated to CoMask's input NRMSE, and both remove the one property that differs between them and CoMask, a basis fitted to the data. We use them to measure how much of CoMask's gain comes from keeping the spectrum complete and how much from the fitted basis. On EnMAP they match CoMask, which means the fitted basis adds nothing there. It does not mean the method fails. No one pretrains a hyperspectral encoder with white noise at a tuned NRMSE.

The experimental evidence does not consistently support this explanation across datasets.

The paper's claim is that CoMask beats input-space masking at matched degradation, and it does on both datasets. The white-noise arm, the simplest spectrally complete corruption, exceeds the input-space control by 4.37 percentage points on Houston2018 (8 pretraining seeds) and by 3.08 on EnMAP-DFC. CoMask exceeds the same control by 6.71 and 3.37. Input-space masking is the only arm in Table 4 that removes whole wavelengths, and it is the lowest arm on both datasets. Keeping the spectrum complete produces the gain, on a single airborne scene under probing and on a satellite collection under cross-region finetuning.

The ablation results on EnMAP-DFC.

EnMAP is a different test from Houston. On Houston we fit the basis to the same scene we evaluate on, under linear probing, so the frozen encoder is scored on the data its basis describes. On EnMAP we fit the basis to 90 European scenes and then finetune the encoder on Mexico City, so the encoder adapts to a new region before it is scored. A basis fitted to Europe does not describe Mexico City, and finetuning lets the encoder learn the new region on its own. The fitted basis therefore adds 2.58 points on Houston and nothing measurable on EnMAP, which Section 6 already reports, and the revision scopes that contribution to Houston. The part of CoMask that does not depend on the basis, masking in component space so that every wavelength survives, is the part that wins on both datasets, and it is the part the paper is about.

AuthorsOpenReview ↗

Response to Reviewer aSDk (2/3): W4, W5

W4.

Matching input NRMSE does not necessarily ensure equivalent information loss.

We agree, and no quantity can. Information loss is defined relative to a predictor. Two corruptions can have equal loss under one predictor and unequal loss under another, so there is no predictor-free quantity to equalise. NRMSE has the same definition for every corruption and it measures the distance between the encoder's input and the original spectrum, which is the quantity the encoder is exposed to. For that reason we calibrate every arm to the same NRMSE. If you have a particular measure of information loss in mind, we are willing to add it.

We also measured information loss under one fixed predictor, at matched NRMSE on Houston2018. A ridge predictor, fitted on 3,200 training tiles and scored on 800 held-out tiles, reconstructs the original spectrum at band-masked locations with R² 0.95 and at CoMask-masked locations with R² 0.66. The two corruptions do differ in information loss at equal NRMSE, and the difference is larger for CoMask. Band masking removes content that a linear predictor explains almost fully from the visible bands (R² 0.95), which is the redundancy the paper starts from. CoMask removes content that a linear predictor explains only partly (R² 0.66), and the encoder trained on it probes 6.71 percentage points higher. Equal NRMSE is the harder condition for CoMask, and CoMask wins under it. The revision reports this R² next to NRMSE for every arm in Table 4.

W5.

Its advantages are not consistently maintained across evaluation protocols and model architectures.

This standard requires a method to lead in every cell of every table, and no pretraining method in the paper meets it. For instance, the strongest input-space baseline, MaskedSST-Tube, is below the random-init floor under probing at ViT-Small on Houston2018 (51.30 against 51.53, Table 1).

Both examples in the review are full-label finetuning results at ViT-Tiny, so the question is what finetuning measures there. On Houston2018 the finetuning data is the pretraining scene itself, with every label available, so the finetuning objective determines the final representation and the pretraining methods are within 2.7 percentage points of each other, against 6.0 under probing. Class-weighted cross-entropy on EnMAP changes the downstream objective once more, and the rows are closer together still. Neither setting measures the masking, which is why the paper's conclusions rest on linear probing for Houston2018 and finetuned macro accuracy for EnMAP-DFC, and states so.

On Houston2018 with ViT-Tiny, CoMask-NMF achieves lower full-label finetuning accuracy than the strongest input-space masking baseline.

By 0.47 points of overall accuracy (61.19 ± 0.45 against 61.66 ± 1.14).

On EnMAP-DFC, its macro-accuracy advantage at ViT-Tiny also disappears under class-weighted cross-entropy.

By 0.21 points (63.07 ± 0.53 against 63.28 ± 0.97), in a table where CoMask-NMF has the highest finetuned overall accuracy. Both differences are smaller than the standard deviations on either side, and both are reported in Section 7 and Appendix E. Under the protocols that do measure the masking, CoMask-NMF is above both input-space baselines at every encoder scale on Houston2018 (three pretraining seeds, our response to Reviewer YJ18, Q5) and on EnMAP-DFC (Table 2), above the MaskedSST-Tube baseline on all six Spectral Earth tasks (Table 3), and above the input-space control on a Conv3D backbone (W6). The two cells the review cites are the smallest differences in the paper, in the settings that measure the masking least.

AuthorsOpenReview ↗

Response to Reviewer aSDk (3/3): W6

W6.

The Conv3D experiments show that CoMask does not consistently outperform supervised training from scratch.

The Conv3D runs, like the class-weighted EnMAP runs in Appendix E, are completeness experiments. They test whether the masking strategy runs unchanged on a second architecture and under a second downstream objective. Both reuse the masking configuration and training settings from the ViT experiments, with nothing adjusted for the new backbone or the new objective, as the table captions in Appendices E and F state. They are not tuned comparisons against training from scratch, and the paper draws no conclusion from them about that comparison. What they do show is the one thing they were run for. Under probing, CoMask exceeds the input-space control by 7.98 percentage points at Conv3D-Tiny and 1.36 at Conv3D-Small (Appendix F), so the advantage over input-space masking holds on a second architecture. With the ViT settings carried over, both pretraining methods have lower finetuning accuracy than training from scratch. The input-space control is below training from scratch as well, so the difference comes from the Conv3D training recipe and not from the masking.

Cross-sensor generalization remains unexplored.

It is future work. This paper is about the masking strategy, and every experiment in it is built to isolate the effect of the mask: the same encoder, data and recipe, with only the masking changed. Reusing pretrained weights across sensors is a different question. It concerns the encoder and tokenizer, which are tied to the band grid in any spectral ViT, and it deserves its own controlled study rather than an addition to this one. The masking strategy itself already runs on two sensors with one configuration, Houston2018 with 48 bands and EnMAP with 200.

If these responses resolve your concerns, we ask that you reconsider your rating. We are happy to answer any remaining concern.

Comments on the paper

7 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for careful and detailed reviews. CoMask moves masked-autoencoder pretraining for hyperspectral imagery from input space to a fitted spectral component basis, so that every wavelength survives the mask. The paper's claim is that this produces a better representation than input-space masking at the same input degradation. The reviews ask whether that gain is real, what causes it, and whether it holds beyond the settings the paper reports, and each review named the controls that would settle its question. We ran all of them, on Houston2018 at ViT-Tiny with 8 pretraining seeds instead of 3, each calibrated to the paper's input NRMSE. Table R1 collects the arms, and the individual responses give the details.

Table R1. Houston2018, ViT-Tiny, overall accuracy (%) under linear probing (LP) and finetuning (FT), mean ± sd over 8 pretraining seeds. Random-init LP floor: 46.45.

ArmInput NRMSELP-OAFT-OA
CoMask-NMF, fixed configuration0.79656.16 ± 3.0059.53 ± 1.02
Input-space control (zero whole tokens)0.79149.46 ± 2.8660.26 ± 1.97
White noise0.79653.83 ± 1.6258.44 ± 1.13
Covariance noise0.79653.20 ± 1.9560.19 ± 1.42
Random non-negative basis0.79153.58 ± 1.3859.37 ± 1.09
Band-permuted NMF basis0.79755.63 ± 2.2858.94 ± 1.12
Fixed frequency (cosine) basis0.79952.51 ± 2.4659.14 ± 1.44
Wavelet basis0.80049.96 ± 2.2857.95 ± 1.25
New A. Low-rank input + token zeroing0.79148.82 ± 0.6759.23 ± 0.81
New B. Low-rank input, nothing masked0.01250.01 ± 3.1457.17 ± 1.14
New C. CoMask input, low-rank target0.79654.52 ± 2.2060.29 ± 1.05
New D. Dominant, independent per location0.80355.43 ± 2.6258.94 ± 1.65
New E. Dominant, scale-invariant ranking0.79053.95 ± 2.0859.60 ± 1.15

CoMask has the highest linear-probing accuracy of every arm in Table R1, 6.71 percentage points above the input-space control, and the margin is present under selection on validation, on test, and with no selection at all. The gain comes from two properties of the component mask, a complete spectrum and a fitted basis, and from nothing else we tested: not low-rank filtering of the input, not the selection rule, not the shared mask, and not the reconstruction target. With NMF fixed in advance, CoMask is above the input-space baseline at every encoder scale on Houston2018 and EnMAP-DFC, on all six Spectral Earth tasks, and on a Conv3D backbone. Two results limit the claims, and the revision states both: under full-label finetuning on Houston2018 the pretraining methods are within 2.7 points of one another, so the Houston conclusions rest on linear probing, and the fitted basis adds to the gain on Houston but not measurably on EnMAP.

Changes for the revised version:

  • Abstract and introduction state which protocol each dataset's conclusions rest on.
  • Table 1 reports three pretraining seeds at every scale. The Table 1 and 2 captions state the seed protocol.
  • Table 4 moves to 8 pretraining seeds and adds the five new arms. The appendix adds the selection analysis, the neighbour cells, the full-corruption runs, and linear recoverability next to NRMSE for every arm.
  • Section 3 describes dominant selection as ranking by coefficient magnitude under the released basis, and reports the scale-invariant variant.
  • The fitted-basis contribution is scoped to Houston.
  • Corrections: the Houston validation description (validation pixels come from the training region), and the EnMAP clamp figures in the appendix (64% to 14.7%, 1.55% to 0.9%).
AuthorsOpenReview ↗

Response to Reviewer YJ18 (1/6)

Thank you for a review that engages with the appendix in detail. You asked whether the gain is real, where it comes from, and how far it holds, and you named the controls that would settle each question. We ran all of them: the low-rank control you identified as missing, the decomposition in Q3, three pretraining seeds at every encoder scale, validation-only selection with equal trials, the recoverability measure in Q6, and the EnMAP clamp statistics. CoMask exceeds the input-space baselines in every one of them, at 8 pretraining seeds instead of 3. Two results limit a claim, and we say so where they appear. The general response above lists the changes we will make in the revised version.

AuthorsOpenReview ↗

Response to Reviewer YJ18 (2/6): Q1

Q1 (W2).

The main margin depends on a configuration chosen on the test set.

We think this is a misreading of how the Houston2018 comparison was set up, and we will make the setup explicit in the revision. The sentence implies that the configuration was selected on test for CoMask and not for the baseline. It conflates two things, the masking configuration of each method and the rule used to select it.

There are two standard ways to compare the methods here. One is to hold the input degradation fixed, with the input-space configuration calibrated to CoMask's input NRMSE and nothing selected. The other is to select each method's configuration on the validation split. We did both. At matched NRMSE, CoMask exceeds the input-space control by 6.71 percentage points (8 pretraining seeds, Table R1). With validation selection, CoMask exceeds MaskedSST by 4.55.

The test-based selection in Table 1 serves a different purpose and is the harder comparison. Houston2018 is small enough to pretrain at every masking ratio in both grids, so we did, and compared the best achievable performance of each method, selected by the same rule for both. That is the comparison Table 1 reports, and it is where CoMask's lead is smallest, 1.08 points (58.19 against 57.11). All 63 CoMask cells and 22 MaskedSST cells are in Appendix D.

What are the Houston and EnMAP margins when the CoMask configuration is chosen on validation only, with each method given the same number of test-blind trials?

CoMask is above MaskedSST on both datasets under validation-only selection. On EnMAP, where the validation tiles are disjoint from the training tiles and CoMask reuses the Houston configuration with no tuning, the baseline cell chosen on validation finetunes to 53.38% macro accuracy against 57.43% for CoMask (+4.05 percentage points). On Houston the validation-chosen margin is the +4.55 above. For equal trials, we drew k configurations from each grid and kept the one with the best validation accuracy, 10,000 times for each k from 1 to 22. CoMask is above MaskedSST at every k, by +1.12 at k = 1 and +4.35 at k = 22. Houston validation is weakly related to test for both methods (Spearman 0.15), so we also give the margin when the best of the k cells is chosen on test instead, which is the most any selection rule could reach, and CoMask is above MaskedSST there at every k as well, by +1.12 at k = 1 and +0.40 at k = 22.

Please also report the distribution over the 63 grid cells (median, quartiles)... since the median cell currently falls below the baseline.

The two medians are nearly equal, and neither compares the methods. A grid median measures how much of a grid is far from its optimum, and both grids were built to find the optimum, so most of each is far from it by design. The methods are compared at the cells the same selection rule picks, which is the comparison above.

Houston2018, ViT-Tiny, LP-OAMedianQuartilesBest cell
CoMask dominant (63 cells)51.8050.74 to 53.7258.29
MaskedSST (22 cells)51.6349.25 to 53.1057.11

The 3-seed value of the chosen and neighbouring cells.

The fixed cell is the best of its five-cell neighbourhood at three pretraining seeds, by 1.3 to 6.2 percentage points.

Houston2018, ViT-Tiny, 3 pretraining seedsLP-OA
c = 0.75, s = 0.70 (fixed)58.22 ± 1.19
c = 0.625, s = 0.7054.27 ± 2.22
c = 0.875, s = 0.7053.88 ± 1.38
c = 0.75, s = 0.6052.01 ± 1.50
c = 0.75, s = 0.8056.89 ± 5.89

The standard deviation at (c = 0.75, s = 0.80) comes from one pretraining seed at 63.57, the highest Houston probe we have measured in any run. These values use three downstream seeds per pretraining seed, so they differ slightly from Section 5.

W2 in the review reads the Houston selection as the basis of the paper's margin. Spectral Earth, the largest benchmark, involved no selection on test data for either method. Both configurations were fixed before any Spectral Earth downstream run, K was set by reconstruction error on unlabeled pixels, each method is one pretraining run, and CoMask is above the baseline on all six tasks (Table 3). The appendix adds the K analysis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.