acceptodds
Withdrawn from ICLR 2027

FOTA: Forward-Only Test-Time Adaptation for Multispectral and Hyperspectral Image Fusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multispectral and hyperspectral image fusion (MHIF) is an important approach for obtaining high-resolution hyperspectral images. However, in real-world applications, out-of-distribution (OOD) shifts caused by unknown, target-specific physical degradations lead to severe performance degradation in existing pre-trained MHIF models. Most existing test-time adaptation (TTA) methods rely on proxy losses or explicit degradation loops that require computationally intensive iterative backpropagation, which incurs high computational overhead and makes models prone to overfitting in few-shot scenarios. To address this bottleneck, we propose a novel backpropagation-free, architecture-agnostic framework: Forward-Only Test-Time Adaptation (FOTA). Specifically, we first project high-dimensional features into a compact low-rank subspace to reduce the parameter scale. To overcome the slow convergence of zeroth-order optimization, we utilize the analytical closed-form solution of Bures-Wasserstein optimal transport under the multivariate Gaussian assumption for initialization, achieving rapid distribution alignment. Furthermore, we construct a structured prior from the physical observation differences of high-resolution multispectral images to directly guide the search direction. Finally, the algorithm iteratively refines the low-rank prompt via forward-only zeroth-order optimization, ensuring physical imaging consistency without backpropagation. Experiments on five benchmarks demonstrate that FOTA achieves leading adaptation performance. Our code will be released.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Test time adaptation, hyperspectral images, multispectral images, image fusion, image super-resolution

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hNpGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a forward-only TTA method for multispectral and hyperspectral image fusion. It adapts final-layer features by combining source-target distribution alignment, residual back-projection correction, and zeroth-order optimization. Extensive experiments validate its effectiveness.

Critical strengths

The use of Bures-Wasserstein optimal transport for closed-form affine feature alignment is a genuine methodological contribution. Although zeroth-order optimization and residual back-projection are not new, the integration of BW-based distribution alignment with a low-rank SVD subspace is novel and principled. This matters because it provides a geometrically grounded initialization for forward-only adaptation and reduces the search space and computational cost for high-dimensional hyperspectral features.

The empirical validation is broad and supports the effectiveness of the proposed framework. Experiments span multiple datasets, nine corruption types, cross-scene and cross-temporal remote sensing benchmarks, different backbones, and ablation studies. This is consequential because it shows the method is not limited to a single dataset or degradation setting, and it provides reasonable evidence that the combined adaptation strategy works in practice.

Critical weaknesses

The core assumption of a fixed and known spectral response function (SRF) is a major limitation for cross-dataset test-time adaptation. In realistic hyperspectral scenarios, different sensors have different SRFs, and the HR-MSI may also be degraded. The method’s physical consistency objective and residual back-projection both rely on the HR-MSI being clean and the SRF being unchanged. The paper’s own results show sub-optimal performance under SRF shift and MSI corruption, yet no solution is proposed. This is consequential because it undermines the claimed applicability to real cross-sensor adaptation, which is the primary motivation for TTA in this domain.

The ablation study does not isolate the contribution of the SVD-based low-rank projection. Low-rank reconstruction alone is a well-known denoising and completion operator and may already handle several tested corruptions such as Gaussian noise, stripe noise, and band dropout. Since all ablation variants include SVD, it is unclear whether the reported gains come from the proposed BW alignment, zeroth-order optimization, and observation prompt, or simply from low-rank feature reconstruction. This is consequential because it questions the necessity of the proposed components. Additionally, the manuscript contains duplicated sentences (lines 59–62) and near-duplicate passages between the introduction and later sections (lines 224–240), indicating insufficient proofreading.

Questions for authors

see the weakness

Reviewer_JDKbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes FOTA, a forward-only test-time adaptation framework for multispectral and hyperspectral image fusion. The method combines low-rank feature projection, Bures-Wasserstein optimal transport initialization, physics-informed observation prompts, and zeroth-order optimization to adapt pretrained fusion networks without backpropagation. Experiments are conducted on five datasets, including synthetic corruption scenarios and cross-domain remote sensing benchmarks. The authors report improvements in reconstruction quality and computational efficiency compared with several existing test-time adaptation methods.

Critical strengths

Practical motivation. The paper addresses test-time adaptation for hyperspectral image fusion under distribution shifts, which is a relevant problem for real-world remote sensing applications. Avoiding backpropagation could potentially benefit deployment in memory-constrained environments.

Integration of physical constraints. Incorporating multispectral observation consistency into the adaptation process is a reasonable design choice. The experiments across multiple datasets and corruption scenarios provide some evidence of the potential applicability of this approach.

Critical weaknesses

Limited methodological novelty and insufficient theoretical justification. The proposed framework primarily combines established techniques, including low-rank projection, Bures-Wasserstein alignment, zeroth-order optimization, and physical observation consistency. The paper does not sufficiently establish what fundamentally new optimization principle is introduced. Moreover, the Gaussian feature assumption and the effectiveness of global statistical alignment for preserving hyperspectral information are not adequately justified.

Insufficient experimental evidence and questionable practical efficiency. The experimental comparisons do not consistently demonstrate superiority over existing methods, particularly in spectral reconstruction accuracy. For example, Table 1 shows that the proposed method does not achieve the best average PSNR or SAM. Table 2 also contains several cases where competing methods perform better. Furthermore, the zeroth-order optimization requires multiple forward evaluations over 40 iterations, raising concerns about the claimed computational efficiency. The reliance on simulated degradations and known sensor response functions further limits the evidence for practical robustness.

Questions for authors

What is the fundamental methodological novelty beyond combining existing Bures-Wasserstein alignment, low-rank projection, and zeroth-order optimization techniques?

How is the multivariate Gaussian assumption validated for hyperspectral feature distributions? What happens when the target distribution is strongly non-Gaussian or multimodal?

Could the authors clarify the dimensional consistency of Eqs. (5) and (6), particularly the relationship between the spectral observation correction and the intermediate feature dimensions?

Why does the proposed method fail to consistently outperform competing approaches in SAM and several cross-domain experiments? How can spectral fidelity be guaranteed when the optimization objective primarily enforces multispectral observation consistency?

How many actual forward evaluations are required for each test sample? Please provide a fair end-to-end comparison of adaptation latency, peak GPU memory, and computational cost under identical hardware and backbone settings.

How robust is the proposed framework when the spectral response function is unknown or inaccurately estimated, or when both LR-HSI and HR-MSI observations are degraded?

Can the authors provide experiments involving authentic sensor degradations and independently acquired multispectral and hyperspectral observations to substantiate the practical applicability of FOTA?

Reviewer_VX6uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Forward-Only Test-Time Adaptation (FOTA) for multispectral and hyperspectral image fusion (MHIF) under out-of-distribution (OOD) shifts. The method first projects source and target features into a fixed low-rank subspace derived from source-domain features. It then employs a closed-form Bures-Wasserstein (BW) optimal transport mapping to align target-domain feature statistics with those of the source domain. To further refine the reconstruction, the authors introduce a physical observation correction based on HR-MSI consistency and optimize a low-rank residual transformation using zeroth-order (ZO) optimization without backpropagation. Experiments on five benchmarks demonstrate competitive adaptation performance with substantially reduced computational overhead.

Critical strengths

1 meaningful problem with practical relevance. Efficient test-time adaptation for MHIF is a relevant problem, particularly when target-specific degradations reduce the performance of pretrained fusion models. Avoiding repeated backpropagation can be beneficial in memory-constrained or inference-only deployment environments.

2.Promising computational efficiency. The computational results in Appendix Table 5 are encouraging. For example, FOTA reports an adaptation time of 1.2 seconds and peak memory consumption of 581 MB, compared with 250.4 seconds and 9,940 MB for UTAL. Although these comparisons involve different source models, they demonstrate the potential efficiency of the proposed framework.

Critical weaknesses
  1. Insufficient justification for zeroth-order optimization. The proposed method optimizes only a 16×1616\times16 residual matrix, with the backbone frozen and the prompt apparently injected near the output. In this setting, first-order optimization may require relatively little additional memory, particularly if preceding features can be cached. Meanwhile, the proposed ZO method also requires numerous forward evaluations. A controlled comparison with Adam or SGD under identical parameterization and initialization is necessary to establish whether ZO provides meaningful advantages in reconstruction quality, runtime, or memory consumption.

  2. Questionable environmental corruption settings and potential restrictions of the fixed low-rank subspace. In the main corruption benchmarks, degradations are applied exclusively to LR-HSI, while HR-MSI remains clean. While this is reasonable for sensor-specific degradations, the inclusion of environmental corruptions such as haze, brightness changes, and clouds is questionable, as these may affect both modalities. The authors should justify whether these settings represent realistic acquisition conditions. Otherwise, the Environment group should be revised or removed from the main comparison. Moreover, FOTA explicitly relies on clean HR-MSI observations for adaptation, whereas several competing methods may use different self-supervision objectives. This raises concerns about whether the observed advantages arise from the proposed optimization strategy or the availability of more reliable supervision. Additionally, the low-rank projection basis is fixed using source-domain features. Under substantial OOD shifts, the target feature subspace may differ from the source subspace, potentially limiting the effectiveness of the learnable correction. Have the authors considered adapting the low-rank basis itself during test-time optimization, and how would this affect reconstruction performance and computational efficiency?

  3. Poorly organized experimental tables that substantially hinder readability. Although the experimental results are promising, they are extremely difficult to interpret, even after careful reading. I strongly recommend that the authors reorganize the experimental tables to make the main findings and contributions more accessible. Table 1 lacks explicit row labels for PSNR and SAM and compares methods with different backbones while emphasizing relative improvements. For instance, SRTTA achieves a substantially higher clean-image PSNR than FOTA (48.98 vs. 46.74 dB). Why is FOTA not evaluated using this stronger pretrained source model? Moreover, FOTA reports a larger PSNR gain than UTAL (+2.09 vs. +0.53 dB), although UTAL achieves better absolute PSNR and SAM. Why should readers favor a larger relative improvement over better final reconstruction quality? Without explicitly presenting the corresponding computational costs, Table 1 fails to provide sufficient information for readers to evaluate the practical value of the proposed method. If feasible, using the same pretrained backbone and weights for different TTA methods, as partially done in Table 1, would provide a fairer comparison. Appendix Table 5 demonstrates FOTA's computational advantages, but these results are separated from the main comparisons, forcing readers to repeatedly cross-reference tables to understand the trade-off between reconstruction quality and efficiency. Table 2 repeatedly labels different configurations as "Ours" without clearly identifying their corresponding source models. Moreover, its boldface convention highlights the proposed method rather than consistently indicating the best-performing results, making comparisons less intuitive.

Questions for authors

Q1. Given that FOTA optimizes only a 16×1616\times16 residual matrix with a frozen backbone, what specific advantages does zeroth-order optimization offer over first-order methods such as Adam or SGD? Could the authors provide a controlled comparison under identical parameterization and initialization, particularly in terms of reconstruction quality, runtime, and peak memory consumption?

Q2. Appendix Table 9 shows negative PSNR adaptation gains when HR-MSI is corrupted. How do the authors justify the assumption of clean HR-MSI observations in the main environmental corruption benchmarks? To what extent would FOTA remain effective when both LR-HSI and HR-MSI are affected by environmental degradations?

Q3. Since the low-rank projection basis is fixed using source-domain features, how does the method handle substantial shifts in the target feature subspace? Would adapting the projection basis at test time improve reconstruction performance, and what additional computational cost would this introduce?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.