acceptodds
Under review as a conference paper at ICLR 2027

Learning Discrete Semantic Invariance for Cross-Scene Hyperspectral Image Domain Generalization

Abstract

Cross-scene hyperspectral image (HSI) classification requires a model trained on one labeled scene to recognize shared classes in an unseen scene without using target data. Existing domain-generalization methods often synthesize source variations and enforce representation consistency across transformed views. However, a transformation that suppresses scene-specific variation for one material may also alter a narrow spectral cue that distinguishes another class. Uniformly enforcing consistency across these transformations can therefore remove class-relevant information. In this work, we propose a transformation-conditioned discrete self-supervision (TCDS) framework for cross-scene hyperspectral image domain generalization, which learns shared discrete semantics by weighting transformed views according to their semantic preservation. For each source sample, our TCDS constructs plausible virtual scene variations and estimates how much class-relevant evidence each transformation retains. Spectral and spatial-context views are then encoded with separate discrete codebooks whose semantic indices are aligned before forming a preservation-weighted consensus. A complementary residual branch retains class-discriminative information outside the shared semantic codes and contributes to prediction through a preservation-dependent gate. Extensive experiments on three datasets demonstrate the effectiveness and superiority of our proposed TCDS method. The code file is public at https://github.com/Sky-byte-box/TCDS_code.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Cross-scene domain generalization, Semantic invariance, Discrete representation learning, Self-supervised learning

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_a72qOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses single-source domain generalization for cross-scene hyperspectral image (HSI) classification and proposes a transformation-conditioned discrete self-supervision framework (TCDS). To overcome the potential loss of discriminative information caused by uniformly enforcing view consistency in conventional self-supervised learning, the authors compute a “semantic preservation score” for different transformed views based on the prediction divergence (JSD) of the source-domain classifier and the spectral angle distance (SAM). The model then encodes the spectral and spatial views separately through discrete codebooks, aligns them at the semantic-index level using the Sinkhorn algorithm, and performs preservation-weighted consensus. In addition, the framework designs a preservation-dependent residual gating branch to supplement features not captured by the discrete codebooks, and introduces Lagrangian constraints to align semantics and balance codebook usage.

Critical strengths
  1. The motivation—that uniform consistency may suppress class-specific discriminative information—has a clear physical grounding. Figure 1 provides an intuitive explanation through the different effects of adjacent-band mixing and spatial smoothing on two types of materials, and the proposed preservation-weighted consensus is well targeted.
  2. The basic rigor of the experimental setup is commendable. On three cross-scene HSI benchmarks, target-domain labels are strictly isolated and do not participate in hyperparameter tuning. The dual codebooks and the residual branch yield considerable ablation gains on the PaviaU dataset.
Critical weaknesses
  1. The hyperparameter matrix in this paper raises suspicions of dataset overfitting. In the Houston13→Houston18 task, the training-time parameters change abruptly during inference through an unjustified “source-only calibration” (e.g., γr\gamma_r changes from 1.15 to 1.80).
  2. The core “semantic preservation score” aima_{im} relies heavily on the JSD divergence of the source-domain classifier p0p_0. In domain generalization, p0p_0 inevitably carries source-domain-specific bias. If a transformation effectively removes source-specific noise and exposes truly cross-domain shared features, p0p_0 may instead assign a high JSD penalty because its prediction distribution changes, causing this high-quality transformation to be downweighted. This circular definition, at the underlying logic level, implicitly encourages the model to fit source-domain bias.
  3. The design of the residual gate has a structural disconnect in information flow. The gate βi\beta_i is determined jointly by the preservation scores of all transformed views, but the residual feature rˉi\bar{r}_i it controls is extracted only from the identity view. This means that when the preservation of transformed views is very low, the gate amplifies the non-shared features of the source image itself. Figure 6(d) also confirms that in the Q4Q_4 interval with the highest preservation (smallest βi\beta_i), removing the residual still causes a 5.25% increase in error, indicating that the gate fails to achieve the expected dynamic compensation.
  4. The Lagrangian constraints proposed in the paper lack empirical support and have extremely loose boundaries. Taking the codebook balance constraint H(cˉ)≥log⁡K−ηH(\bar{c}) \ge \log K - \eta as an example, the authors set K=96K=96 and η=2.0\eta=2.0, which in practice only requires the activation entropy to be greater than 2.56. This is equivalent to allowing the number of effective non-zero activation codes to shrink to about 13, and thus cannot substantially prevent codebook collapse.
Questions for authors
  1. In the Houston13→Houston18 task, why can the model abruptly change the training-time value γr=1.15\gamma_r=1.15 to 1.801.80 during inference through the so-called “source-only calibration”?
  2. According to Eq. (7), the gate βi\beta_i is determined by the preservation scores of all transformed views, but the residual rˉi\bar{r}_i it controls comes only from the identity view. Does this mean that when the transformed views are severely distorted, the gate instead amplifies the features of the source image itself?
  3. Figure 6(d) shows that in the Q4Q_4 interval with the highest preservation, removing the residual still leads to a 5.25% increase in error. Please explain why this phenomenon contradicts the dynamic compensation design you claim.
1 comment
AuthorsOpenReview ↗

Response to Reviewer a72q

Thank you so much for your positive and encouraging feedback. Below, we address your comments and suggestions in details:

Q1 & W1: Houston inference-time residual-scale change and source-only calibration.

A. The training-time value γr=1.15\gamma_r=1.15 is not fixed at inference because γr\gamma_r controls residual fusion rather than learned model weights. Its inference value of 1.80 is jointly selected with ρ\rho and TcT_c using held-out Houston13 source data, without accessing Houston18 or retraining the model (Appendix F, Table 7).

Q2 & W3: Transformed-view preservation and identity-view residual gating.

A. The preservation scores assess how reliably transformed views support the discrete semantic consensus, whereas the identity-view residual retains discriminative cues omitted by the discrete codes. When transformed views preserve less class evidence, the gate therefore increases reliance on the current input's untransformed residual to compensate for the less reliable consensus evidence. At inference, this residual is computed from the current target sample, not a stored training-source image. The residual-separation constraint discourages overlap with shared semantics, while clipping bounds the gate coefficient. The gate thus adjusts the contribution of complementary information without assuming that every residual feature is domain-invariant.

Q3 & W3: Residual removal in the highest-preservation quartile and dynamic compensation.

A. High transformation preservation does not imply that discrete codes capture all discriminative information. Moreover, the gate decreases with preservation but is not forced to zero (βmin⁡=0.03\beta_{\min}=0.03) on Pavia. Fig. 6(d) measures the effect of removing the entire residual branch within each preservation quartile, not the gate coefficient itself. The observed error increases upon removal are 15.49, 12.15, 6.14, and 5.25 percentage points across (Q1)(Q_1)–(Q4)(Q_4), respectively. Thus, the residual remains useful even in (Q4)(Q_4), whereas the observed benefit is larger in lower-preservation groups. This pattern is consistent with relative compensation, which does not require residual benefit to vanish at high preservation.

W2: Source-classifier JSD, beneficial transformations, and source-specific bias.

A. A change in source-classifier predictions may increase JSD and lower a transformation's preservation weight, but TCDS does not treat JSD alone as a rejection criterion. Eq. (2) combines prediction JSD with spectral-angle discrepancy to produce a continuous weight (aima_{im}), rather than discarding views outright. Fig. 6(a) evaluates this score with a separate source hold-out probe: balanced transformed-view error decreases from 14.73% in (Q1)(Q_1) to 1.40% in (Q4)(Q_4). On PaviaU$\rightarrow$Pavia Center, Full TCDS achieves 66.32% OA versus 65.3% with uniform consensus weights (Table 6). The score therefore has support from held-out discriminative evidence and target-domain performance, not merely from agreement with its source scoring classifier.

W4: Empirical support for Lagrangian constraints and codebook-collapse prevention under the relaxed code-balance bound.

A: The Lagrangian constraints target semantic agreement, residual separation, and code-use diversity, rather than uniform code activation. The code-balance bound (H(cˉ)≥log⁡K−ηH(\bar c)\ge\log K-\eta) imposes a minimum entropy on aggregate code usage. With (K=96K=96) and (η=2\eta=2), it requires (H(cˉ)≥2.56H(\bar c)\ge2.56), or an effective code count (exp⁡(H(cˉ))≳13\exp(H(\bar c))\gtrsim13) when satisfied. It therefore excludes single-code collapse under that condition while allowing nonuniform codebook usage. It does not ensure that all 96 codes remain active. Table 8 shows that removing code balance reduces mean OA by 2.85/2.20/0.94 percentage points on Houston/Pavia/HyRANK.

Table 8. Cross-task component ablations (target OA, %). Results are mean$\pm$SD over three runs. Each variant follows the corresponding task-specific configuration.

VariantHouston OA (%)Pavia OA (%)HyRANK OA (%)
Full TCDS74.40 ± 1.7866.32 ± 1.6756.09 ± 1.34
w/o Transformation Bank (TB)73.92 ± 0.7663.16 ± 2.0249.59 ± 0.29
w/o Preservation Weighting (PW)73.40 ± 1.2763.97 ± 1.1757.60 ± 0.94
w/o Dual Codebooks72.91 ± 1.6863.52 ± 0.8354.58 ± 2.66
w/o Semantic Index Alignment72.94 ± 0.6163.45 ± 0.3155.28 ± 3.81
w/o Residual Branch74.06 ± 0.8059.37 ± 10.0253.12 ± 0.59
w/o Code Balance71.55 ± 3.7664.12 ± 0.5655.15 ± 3.72
Reviewer_1dVSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes TCDS, a framework for cross-scene hyperspectral image domain generalization that combines transformation-conditioned semantic preservation, aligned discrete codebooks, and complementary residual representations. Experiments on three cross-scene benchmarks demonstrate improvements over existing methods, supported by ablation and sensitivity analyses.

Critical strengths

The observation that domain transformations may preserve class-discriminative information differently across samples provides a reasonable motivation for moving beyond uniform consistency learning.

Critical weaknesses
  1. Discrete codebook learning and semantic-preserving augmentation have already been explored in domain generalization. The proposed framework integrates several established techniques, but its fundamental methodological advance over these existing approaches is not sufficiently clear.

  2. Although the ablations demonstrate the usefulness of individual components, the largest performance degradation occurs when the residual branch is removed. It remains unclear how much of the improvement specifically comes from discrete semantic invariance rather than complementary feature preservation, particularly since the main ablation study is limited to one cross-scene benchmark.

Questions for authors
  1. Could the authors clarify the methodological distinction from existing discrete-codebook domain generalization methods, such as Domain Generalization via Discrete Codebook Learning, and provide comparisons with simpler preservation-weighted consistency or residual-based alternatives?

  2. In Eqs. (4) and (6), the spatial assignments are transported into the spectral index space, but the residual reconstruction appears to use the original spatial codebook. Could the authors clarify how the codebook and assignment indices remain consistent during this reconstruction?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 1dVS

Thank you for your comments and for taking the time to review our paper. We respond to your suggestions point by point as follows.

Q1 & W1 & W2: Methodological novelty and distinct contributions of semantic alignment and residual modeling.

A. Compared with existing discrete-codebook domain generalization approaches, TCDS explicitly models how reliably each source-derived HSI transformation preserves semantic evidence for each sample. Specifically, we establish a preservation-conditioned mechanism for discrete semantic invariance, in which sample-dependent transformation evidence governs semantic aggregation in an aligned spectral – spatial code space and regulates the use of complementary residual information. Eq. (2) estimates sample–transformation preservation from spectral-angle and predictive-JSD discrepancies. Eq. (4) establishes cross-view semantic correspondence by aligning spectral and spatial code indices using source-label code profiles. Eq. (5) uses preservation to weight the aligned semantic consensus. Eq. (7) conditions complementary residual information on the same preservation evidence. The methodological distinction is this linked preservation–alignment–residual mechanism: transformation reliability determines how corresponding discrete semantics are aggregated and how residual information is used, rather than treating codebook discretization or view consistency as the complete solution.

We also evaluate simpler consistency and prediction-level alternatives on PaviaU$\rightarrow$Pavia Center (OA, %; Appendices A and E). Uniform-JSD retains the transformation bank. Uniform-weight TCDS keeps semantic index alignment, the residual branch, and the preservation-dependent gate while changing only consensus weights. SPF and PLA retain the residual pathway but replace code-index alignment with prediction-level fusion or alignment.

Below is excerpted from expanded Table 2 and Table 6 of the revised manuscript:

Pavia controlOA (%)
Uniform-JSD consistency62.98
TCDS with uniform consensus weights65.30
Separate prediction fusion (SPF)64.67
Prediction-level alignment (PLA)65.37
Full TCDS66.3266.32

Full TCDS exceeds Uniform-JSD, uniform-weight TCDS, SPF, and PLA by 3.34, 1.02, 1.65, and 0.95 OA points, respectively. Because the matched uniform-weight and prediction-level controls retain the residual pathway, these gains cannot be attributed simply to its presence. Table 8 further evaluates semantic index alignment and residual modeling through separate component ablations across Houston, Pavia, and HyRANK: removing index alignment decreases mean OA by 1.46/2.87/0.81 points, whereas removing the residual branch decreases it by 0.34/6.95/2.97 points. Index alignment therefore contributes across all three tasks even when residual modeling is retained. The distinct cross-task effects show that TCDS gains do not arise solely from complementary residual features.

Q2: Consistency of spatial-codebook alignment in residual reconstruction.

A. In Eq. (4), AA maps spatial code indices into the spectral index order, yielding q~is,m=Aqis,m\tilde q_i^{s,m}=Aq_i^{s,m}. For residual reconstruction in Eq. (6), the spatial codebook is correspondingly aligned as E~s=AEs\tilde E^s=AE^s, so that ris,m=zis,m−(E~s)⊤q~is,mr_i^{s,m}=z_i^{s,m}-(\tilde E^s)^\top\tilde q_i^{s,m}. The spectral residual retains the original EλE^\lambda and qiλ,mq_i^{\lambda,m}. This is consistent with the implementation, where the spatial assignment and codebook are both mapped into the same index space before reconstruction. We have clarified this spatial-codebook convention immediately after Eq. (6).

Reviewer_fCqCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies single-source domain generalization for cross-scene hyperspectral image (HSI) classification. It starts from the observation that the same transformation can preserve class-discriminative cues for one material while obscuring them for another. On this basis, the authors propose TCDS, which weights augmented views according to how much class evidence they appear to retain.

For each input sample xix_i, TCDS generates a set of spectral and spatial transformations and scores each transformed view using spectral similarity and prediction consistency. It then extracts spectral and spatial-context embeddings and softly assigns each embedding to its corresponding codebook. The method aligns the two codebooks in a shared index space and combines their assignments into a consensus code weighted by the preservation scores. It also retains the part of each embedding that the codebook does not capture as a residual. The final classifier combines the consensus code and the residual, giving the residual more weight when the preservation score is low. During training, the method minimizes a robust classification risk over the transformations, subject to constraints on view agreement, residual separation, and codebook usage. The authors solve this problem using primal–dual Lagrangian updates.

With a shared MLP backbone, TCDS reports the highest mean overall accuracy among the eight methods compared. Its margins over the strongest competitor are 1.30 points on Houston13 → Houston18, 3.65 points on PaviaU → Pavia Center, and 5.21 points on Dioni → Loukia. The paper also presents an ablation study, a parameter-sensitivity analysis, and diagnostic visualizations, all on PaviaU → Pavia Center.

Critical strengths

Clearly defined setting and problem. The paper clearly distinguishes domain adaptation, where unlabeled target data can guide alignment, from the source-only domain generalization setting it studies. The protocol excludes target labels from training, early stopping, preservation scoring, and hyperparameter selection. At inference, the model uses no target statistics or parameter updates. Within this setting, the authors identify a plausible failure mode: enforcing the same consistency across augmented views may suppress class-discriminative cues when a transformation affects materials differently. The clear protocol makes this claim testable, although the paper would benefit from a targeted experiment and a stronger connection to prior work.

Reporting transparency and diagnostic analysis. The authors provide equations, an algorithm box, per-task hyperparameter values for all three benchmarks, and a link to the code. I also appreciate that the evaluation goes beyond overall accuracy. On PaviaU → Pavia Center, the authors use a preservation-quartile diagnostic, cross-view alignment matrices, and an analysis of how the residual’s benefit varies across preservation quartiles. These analyses give readers a way to examine whether the components behave as intended. With stronger controls, they could help establish how much of the reported improvement comes from preservation-aware weighting.

Critical weaknesses

1. The paper needs to clarify what is new and explain the method more clearly (Secs. 1–3).

The paper’s main motivation is that enforcing uniform consistency across transformed views can suppress class-discriminative information. This is a plausible concern, but the paper offers neither a citation nor a targeted experiment to establish when it arises in the HSI setting. Xiao et al. (ICLR 2021) and Tian et al. (NeurIPS 2020) provide relevant context. The discussion of FDGNet, S2AMSNet, and SPDDA would also be more useful if it explained how those methods address this concern and where TCDS differs, beyond describing their mechanisms.

I also found it difficult to separate the paper’s contribution from the established techniques it builds on. The codebook alignment in Sec. 3.3 appears to draw on optimal transport and Sinkhorn iterations, the scene reweighting in Eqs. 8 and 11 resembles KL-constrained distributionally robust optimization, and Eq. 10 uses a Lagrangian formulation for constrained learning. The paper should make these connections explicit and explain what it adds to them. The same applies to related work on per-sample augmentation weighting (MetaAugment and Soft Augmentation), Jensen–Shannon divergence for consistency across views (AugMix), discrete codebooks for domain generalization (DDG), and robust optimization over transformations (AdvST).

Section 3 would be easier to follow if it began with a walkthrough of the training and inference pipelines. As written, the reader encounters equations before learning what transformations are used during training, how the spatial-context operator works, how the codebooks and code-label distributions are learned, or how the warm-start classifier is obtained. The rationale for per-pixel ℓ2\ell_2 normalization and the choice of gating function is also unclear. Two equations need particular attention. Eq. 6 appears to combine the spatial codebook with assignments indexed in the spectral codebook, and Eq. 8 evaluates fθ(τmx)f_\theta(\tau_m x) even though fθf_\theta is already defined to aggregate across transformations. Clarifying these points would make the method easier to reproduce and help readers assess the novelty of the individual components and their combination.

2. The experiments do not yet establish an advantage over existing methods (Sec. 4).

My main concern is how to interpret the baseline comparisons. All baselines were reimplemented using a shared two-layer MLP, and several score substantially below their published results on the same task pairs. On the Pavia task, for example, S2AMSNet drops from 86.18 to 61.39 OA, and SDEnet drops from 81.94 to 61.51. TCDS reports OA values of 74.40, 66.32, and 56.09 across the three tasks, below the published results of all five HSI-specific baselines on each task. A shared backbone can help control for architecture, but the size of these gaps makes it difficult to tell whether the reported gains would hold under the baselines’ original protocols.

Three comparisons would help isolate what drives the gains.

  • A plain source-only model using the same backbone.
  • A uniform-consistency baseline using the same augmentations, which would directly test the paper’s central premise.
  • Baselines evaluated with the same multi-view test-time inference as TCDS, which would help separate gains from training and inference.

The Houston result also warrants clarification. Houston is the only task for which the paper uses different inference-time hyperparameters, selected through a “source-only calibration” procedure that is left unspecified. The OA gain is 1.30 points across three runs (within one standard deviation), while weighted F1 and κ\kappa are tied. Since one class accounts for roughly 61% of Houston18, OA and weighted F1 give limited insight into performance on minority classes. Per-class accuracy, average accuracy (AA), and macro-F1 would make the results easier to interpret.

Finally, all ablations, sensitivity analyses, and diagnostics beyond Table 1 are restricted to PaviaU → Pavia Center. With three runs, the reported ablation differences are statistically inconclusive. It remains unclear whether TCDS improves on existing methods under comparable conditions and which components account for the gains reported in Table 1.

Questions for authors

Clarifications and results that could change my assessment

  1. Baseline comparisons. Could the authors explain why the reimplemented baselines score substantially below their published results on the same task pairs? For example, S2AMSNet achieves 61.39 OA in Table 1 compared with 86.18 in the published Pavia results. It would help to see TCDS evaluated under the original studies’ protocols, using 13×1313 \times 13 input patches and the original backbones where applicable, with the published scores shown alongside for reference.

  2. Controls for training and inference. Could the authors report results for a plain source-only model with the same backbone and for a variant that uses the same transformation bank with uniform consistency across views (e.g., an AugMix-style Jensen–Shannon divergence loss)? The latter would directly test whether preservation weighting improves on uniform consistency. I would also like to see the baselines evaluated with the same multi-view inference as TCDS, and TCDS evaluated with only the identity transformation at inference. These comparisons would help separate the effects of training and inference.

  3. Hyperparameter selection. Could the authors explain why the “source-only calibration” changes (ρ,Tc,γr)(\rho, T_c, \gamma_r) at inference for Houston alone? How are the “deterministic source-derived pseudo-scenes” constructed, and what criterion is used to select the hyperparameters? Please also report the Houston results using the training-time values.

  4. Equations 6 and 8. How is the risk in Eq. 8 computed for each transformation, given that fθf_\theta in Eq. 7 already aggregates across transformations? In Eq. 6, what expression does the implementation use for the spatial view? As written, q~s,m\tilde q^{s,m} is indexed in the spectral codebook space, while EsE^s is indexed by spatial codes.

  5. Choice of task for follow-up analyses. Why are the ablations, sensitivity analyses, diagnostics, and classification maps beyond Table 1 all limited to PaviaU → Pavia Center? The other tasks offer informative checks. Houston13 → Houston18 shows the smallest gain and uses recalibrated inference settings, while Dioni → Loukia has 12 classes and uses a different codebook size. Do the conclusions from the Pavia analyses hold on either task?

  6. Ablation design and variability. Could the authors specify exactly what changes in the “w/o PW” and “w/o TB” variants? If removing preservation weighting means setting aim=1a_{im}=1, then the gate βi\beta_i appears to become zero, which would also disable the residual branch and make the effect of preservation weighting difficult to isolate. Given the large error bar for “w/o RB” in Fig. 4, please report the individual run scores for that variant.

  7. Class-level performance. Could the authors report per-class accuracy, average accuracy (AA), and macro-F1? These metrics would help readers assess performance on minority classes, particularly when class proportions differ between the source and target scenes. For example, a single class accounts for roughly 61% of Houston18.

5 comments
AuthorsOpenReview ↗

Response to Reviewer fCqC—- Part I

We sincerely thank you for your constructive feedback and expert insights. Based on your feedback, we have made the following detailed responses:

Q1 & W2: Fairness of baseline comparisons under matched protocols.

A. Table 1 standardizes source splits, inputs, and a two-layer MLP, not native backbones. S$^2$AMSNet/SDEnet obtain 61.39/61.51% OA versus published 86.18/81.94%. We have added Table 5 (Appendix D) to compare each native baseline with backbone-matched TCDS using identical PaviaU$\rightarrow$Pavia Center splits, preprocessing, and evaluation. Native S$^2$AMSNet/SDEnet obtain 85.31/80.81%, near published values. TCDS gains 1.92/1.95/1.79 OA points. Given the limited rebuttal period, we prioritized the two reviewer-cited methods and FDGNet.

Table 5. Matched-protocol OA (%) on PaviaU$\rightarrow$Pavia Center.

Protocol groupMethodFeature backboneOA (%)
S$^2$AMSNetS$^2$AMSNetNative S$^2$AMSNet85.31
TCDSMatched S$^2$AMSNet backbone87.23
SDEnetSDEnetNative SDEnet80.81
TCDSMatched SDEnet backbone82.76
FDGNetFDGNetNative FDGNet81.90
TCDSMatched FDGNet backbone83.69

Q2 & W1: Controls for training and inference.

A. Table 6 (Appendix E) separates training and inference. With multi-view, TCDS exceeds uniform-JSD/uniform-weight TCDS by 3.34/1.02 points; the latter alters only consensus weights. Multi-view adds 1.11 points. ERM+light augmentation: 62.63% (Appendix C).

Table 6. Identity-only vs. multi-view OA (%).

Training configurationIdentity-onlyMulti-view
Plain source-only ERM59.9461.45
Uniform-JSD consistency62.3262.98
TCDS with uniform consensus weights64.2165.30
Full TCDS65.2166.32

Q3 & W2. Hyperparameter selection.

A. Houston's source-only pseudo-scene search selected inference parameters different from training defaults. Pavia/HyRANK retain their training-time values. The same Houston checkpoints use (ρ,Tc,γr)=(0.30,0.25,1.15)(\rho,T_c,\gamma_r)=(0.30,0.25,1.15) before and (0.12,0.28,1.80)(0.12,0.28,1.80) after calibration. Three stratified Houston13 validation splits generate identity, spectral-smoothing, adjacent-band-mixing and spatial-mixing pseudo-scenes. Each scores 0.34 OA+0.38 weighted F1+0.28 κ0.34\,\mathrm{OA}+0.38\,\mathrm{weighted\ F1}+0.28\,\kappa. Selection uses 0.65×scene mean+0.35×scene minimum0.65\times\text{scene mean}+0.35\times\text{scene minimum} across three seeds, minus 0.15 SD0.15\,\mathrm{SD}. Table 7 (Appendix F) reports 73.29% vs. 74.40% OA (+1.11 points). No Houston18 data or model updates enter selection.

Q4 & W1: Equations 6 and 8.

A. For Eq. (6), the spatial assignment (q~is,m=Aqis,m)(\tilde q_i^{s,m}=Aq_i^{s,m}) and the aligned codebook (AEs)(AE^s) share the spectral index order. The residual is (ris,m=zis,m−(AEs)⊤q~is,m)(r_i^{s,m}=z_i^{s,m}-(AE^s)^\top\tilde q_i^{s,m}). For Eq. (8), each transformation's risk is the auxiliary-head CE on its spectral/spatial embeddings, computed before the consensus prediction in Eq. (7). These per-view risks are averaged. The consensus classifier uses a separate CE term in Eq. (11).

AuthorsOpenReview ↗

Response to Reviewer fCqC—-Part II (Continued)

Q5 & W2: Choice of task for follow-up analyses

A: We focus the detailed ablation, sensitivity, diagnostic, and classification-map analyses on PaviaU$\rightarrow$Pavia Center mainly for presentation convenience and consistent visual comparisons, rather than because the findings are specific to this task. To address the reviewer's concern, we extend all six component ablations to Houston13$\rightarrow$Houston18 and Dioni$\rightarrow$Loukia (HyRANK), which involve different performance margins, inference settings, class counts, and codebook configurations. Table 8 (Appendix G) reports the measured target OA (%) across the three tasks.

Table 8. Cross-task component ablations over three runs (target OA, %, mean$\pm$SD). Each variant follows the corresponding task-specific configuration.

VariantHoustonPaviaHyRANK
Full TCDS74.4066.3256.09
w/o TB73.9263.1649.59
w/o PW73.4063.9757.60
w/o Dual Codebooks72.9163.5254.58
w/o Semantic Index Alignment72.9463.4555.28
w/o Residual Branch74.0659.3753.12
w/o Code Balance71.5564.1255.15

The expanded ablations confirm that the transformation bank contributes positively across all three tasks, with OA differences of 0.48, 3.16, and 6.50 points, respectively. Semantic index alignment and code balance also improve mean OA on all three tasks. The effects of other components are task-dependent. Preservation weighting benefits Houston and Pavia but not HyRANK, while removing dual codebooks reduces mean OA by 1.49 points on Houston. These results support the main cross-task findings concerning transformation-based semantic preservation and alignment, while distinguishing them from task-specific component effects. The detailed sensitivity and visualization analyses remain focused on Pavia.

Q6 & W2: Ablation design and variability.

A. In “w/o PW”, setting all (aim=1)(a_{im}=1) makes consensus weighting uniform and reduces the clipped residual gate to (βmin⁡)(\beta_{\min}), without fully disabling the residual branch. “w/o TB” retains only the identity view during training and inference, likewise setting the gate to (βmin⁡)(\beta_{\min}). To isolate consensus weighting, Table 6 includes a matched uniform-weight TCDS variant that retains the preservation-dependent gate and all other modules, obtaining 65.3% versus 66.32% OA for Full TCDS. In “w/o RB”, the residual branch and its associated decorrelation penalty are disabled. Its three Pavia runs yield 47.89%, 63.87%, and 66.36% OA, explaining the large error bar.

Q7 & W2: Class-level performance.

A. To evaluate class-balanced performance on the imbalanced Houston13$\rightarrow$Houston18 task, we additionally report average accuracy (AA) and macro-F1 for TCDS and all eight baselines. Table 9 (Appendix H) summarizes the results alongside OA and weighted F1.

Table 9. Class-balanced performance on Houston13$\rightarrow$Houston18 (%).

MethodOA (%)Weighted F1 (%)AA (%)Macro-F1 (%)
SDEnet73.1071.0558.9354.07
FDGNet72.1569.1457.0653.22
S$^2$AMSNet72.4570.9460.3855.33
S$^2$ECNet71.9269.6958.2453.78
IRM64.5854.9346.8740.23
VREx63.4753.0046.0339.86
StyDeSty67.7861.1553.9448.18
SPDDA68.7163.4755.2849.93
TCDS74.4071.1661.2755.87

TCDS achieves the highest AA (61.27%) and macro-F1 (55.87%), exceeding the strongest baseline on these metrics, S$^2$AMSNet, by 0.89 and 0.54 percentage points, respectively. These results demonstrate that TCDS retains its advantage under class-balanced evaluation rather than only sample-weighted metrics. They do not imply that every individual minority class improves, which requires per-class recall comparisons.

Thank you for the detailed response and additional experiments. The shared MLP explains the gap with published baselines, and the matched-protocol comparison on Pavia shows gains of 1.8–2.0 points over three baselines (Table 5). The new source-only and uniform-JSD controls, identity-only versus multi-view comparison, cross-task ablations, and class-balanced Houston metrics are helpful. The response also clarifies Houston calibration, Eqs. 6 and 8, and the ablation definitions.

My main concern remains the evidence for preservation-aware weighting. It adds about one point on Pavia and Houston but lowers OA on HyRANK from 57.60 to 56.09. On Houston, uncalibrated TCDS leads the strongest baseline by 0.19 OA points, and the variant without the transformation bank still outperforms all baselines. This makes it difficult to attribute the Houston gain to the transformation-conditioned design. The large residual-branch effect on Pavia comes from one collapsed run. The matched comparison covers one task, and Tables 5 and 8 omit standard deviations.

The motivation still needs references, and the paper should clarify what TCDS adds to optimal transport with Sinkhorn iterations, distributionally robust optimization, and constrained learning, as well as how it relates to prior work outside HSI. Section 3 should also describe the auxiliary heads, separate consensus cross-entropy term, gate floor βmin⁡\beta_{\min}, and aligned codebook introduced in the response.

I stand by my overall assessment.

Reviewer_RHT7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TCDS (transformation-conditioned discrete self-supervision), a method for cross-scene hyper-spectral image (HSI) domain generalization. The main idea is to

  1. Weigh multiple transformations (augmentations) favoring to spectral and predictive consistency w.r.t the original sample.
  2. Extract spectral and spatial embedding into separate codebooks, and calculate a weighted consensus.
  3. Make prediction using the codebook consensus and the residuals outside the codebook consensus. Experiments show TCDS can outperform various domain generalization methods developed for hyper-spectral images on 3 benchmarks.
Critical strengths
  1. The HSI domain generalization task is described well.
  2. The spectral/spatial representation discretization and the alignment process is described clearly.
  3. Empirical results are in favor of the proposed method compared to various HSI domain generalization baselines.
Critical weaknesses
  1. For a method as complicated as TCDS, multiple parts aren't motivated well. In particular, the decision to discretize and align representation from the spectral and spatial-context view of augmented inputs is poorly justified.
  2. Since TCDS requires reweighing augmentations at test time, this is a test time method that adds computation. But this fact is not discussed at all in the paper. This method should potentially be compared to test time methods that can predict adaptively according to test time inputs.
  3. The comparison table is missing the simplest baseline of just training with augmentation on train sample and predict on test sample, without domain generalization objective.
  4. The current writing has some important architectural choices and design choices kept vague. This includes but is not limited to the actual architecture of the encoders, the hand designed spectral vs spatial view transformation, etc.
Questions for authors
  1. [see W1] Why align spectral and spatial codebooks at all? One can predict using the codebooks separately, and compare the predictions as a measure of alignment, but this simple ablation was not included in the method.
  2. [See W2] Please clarify if your method is more fitted as a test-time adaptation method, and how the computation compare to the baselines. Additionally, some empirical comparison to existing test time adaptation method can be informative. One potential baseline method is FATA (Cho et al., 2025).
  3. [See W3] How detrimental is the distribution shift in your experimental setup? A simple ERM baseline w/ light augmentation can answer this, and better quantify the strength of TCDS. Relatedly, for any method developed for robustness under domain shifts, it is crucial to report performance of the method on in-distribution hold-out set to see if in-distribution accuracy is harmed.
  4. [See W4] Please clarify the exact operation used to extract spectral view and the spatial-context view, and the architectures of the encoders used. Additionally, please clarify if the preservation-dependent gate is heuristics or theoretically derived.

Cho, Younggeol, et al. "Feature augmentation based test-time adaptation." 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2025.

3 comments
AuthorsOpenReview ↗

Response to Reviewer RHT7—Part I

Thanks for your insightful comments. We respond to your suggestions point by point below.

Q1 & W1: Why discrete spectral/spatial codebooks and alignment?

A. Spectral responses and spatial-context structures provide complementary class evidence but exhibit different scene-dependent variations. TCDS therefore employs separate codebooks to preserve view-specific representations, while discrete assignments provide reusable semantic coordinates. Since independently learned code indices are not inherently comparable, Eq. (4) establishes their correspondence through source-label distributions and Sinkhorn alignment. Unlike prediction-level alignment, TCDS establishes semantic-index correspondence before forming the preservation-weighted consensus in Eq. (5). To directly evaluate the reviewer's suggested alternative, we additionally compare separate prediction fusion (SPF) and prediction-level alignment (PLA) on PaviaU$\rightarrow$Pavia Center (Table 2 of Appendix A).

Table 2. Prediction-level alternatives and existing ablations on PaviaU$\rightarrow$Pavia Center (OA, %).

MethodOA (%)
w/o Semantic Index Alignment63.45
Shared Codebook63.52
Separate Prediction Fusion (SPF)64.67
Prediction-Level Alignment (PLA)65.37
Full TCDS66.3266.32

Ablations in Fig. 4 and the direct prediction-level controls in Appendix A support the proposed view-specific encoding and semantic-index alignment, with TCDS achieving the highest mean OA among the evaluated alternatives (0.95 points above PLA).

Q2 & W2: Clarification of test-time inference and computational cost.
A. TCDS is source-only domain generalization with fixed test-time augmentation, not parameter-updating test-time adaptation. Its transformation bank and model parameters are determined solely from source data. At inference, TCDS recomputes preservation scores and aggregates four fixed views per sample on Houston/Pavia (five on HyRANK), without gradient updates, target-batch alignment, or target-statistic estimation (Secs. 3.5 and 4.1). Table 3 (Appendix B) compares its measured inference cost with all eight DG baselines on PaviaU$\rightarrow$Pavia Center. TCDS requires 33.3 s, faster than four baselines and slower than the other four. Unlike TCDS, FATA (Cho et al., WACV 2025) performs target-domain model adaptation and thus follows a different evaluation protocol, which is not suitable for empricial comparison with TCDS.

Table 3. Measured inference cost (s) on PaviaU$\rightarrow$Pavia Center for TCDS and the eight DG baselines.

MethodSDEnetFDGNetS$^2$AMSNetS$^2$ECNetIRMVRExStyDeStySPDDATCDS
Inference time (s)25.633.749.8183.712.39.144.629.833.3

Q3 & W3: ERM and light-augmentation baselines, cross-scene degradation, and in-distribution accuracy.

A. We added ERM and ERM+light-augmentation controls on PaviaU$\rightarrow$Pavia Center to assess cross-scene degradation, the contribution of ordinary augmentation, and source-domain performance (Appendix C, Table 4).

Table 4. Source hold-out and target accuracy on PaviaU$\rightarrow$Pavia Center.

MethodSource OA (%)Target OA (%)OA Drop (p. p.)
ERM95.2059.9135.29
ERM + Light Aug.96.2362.6333.60
TCDS97.3866.3231.06

ERM loses 35.29 percentage points from the source hold-out to the unseen target scene. TCDS raises target OA by 6.41 points over ERM and by 3.69 points over ERM+light augmentation, while achieving the highest source hold-out OA. Thus, we observe no source validation-accuracy sacrifice on this task. The source hold-out split is also used for model selection (Sec. 4.1); these values are validation results, not an independent source test.

AuthorsOpenReview ↗

Response to Reviewer RHT7-Part II (Continued)

Q4 & W4: Clarification of spectral/spatial views, encoder architecture, and preservation-dependent gating.

A. Both views are deterministic and computed from the same normalized HSI cube. The spectral view is the central BB-band spectrum. The spatial-context view concatenates bandwise 3×33\times3 and 7×77\times7 reflection-padded neighbourhood means and the 13×1313\times13 minus 7×77\times7 mean (3B3B dimensions). No spectral projection is used. Two independent two-layer MLP encoders use LayerNorm and GELU after each linear layer, with 0.05 dropout between layers. Their intermediate widths are max⁡(128,d)\max(128,d) (spectral) and max⁡(192,d)\max(192,d) (spatial), with dd-dimensional outputs. On Pavia (B=102,d=128B=102,d=128), the dimensions are 102→128→128102\rightarrow128\rightarrow128 and 306→192→128306\rightarrow192\rightarrow128. The gate is a preservation-guided heuristic, not a theoretically optimal rule. Eq. (7) defines

βi=1−1M+1∑m=0Maim.\beta_i=1-\frac{1}{M+1}\sum_{m=0}^{M}a_{im}.

In implementation, the gate is clipped to [0.03,0.60][0.03,0.60] on Pavia. Eq. (2) obtains aima_{im} from spectral-angle and predictive discrepancies. Lower preservation increases the identity-view residual contribution within these bounds. Fig. 6(d) shows larger residual-branch error reductions at low than high preservation (15.49 vs. 5.25 p.p.), supporting the contribution of complementary residual information under low preservation.

I thank the authors for the detailed response. The revisions and the added baselines have helped clarify the method and better position the method against baselines. However, given the complexity of the method, the performance improvement is now small compared to the simple baselines. The ablations, especially on the discretize and align piece central to the proposed method, aren't consistent in its improvements. The new ERM w/ augmentation (during both train and test) baselines also need to be extended to all evaluated benchmarks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.