Hybrid Generalization-Aware Parameter Selection Test-Time Adaptation for All-in-One Image Restoration

All-in-one image restoration (AiOIR) methods achieve impressive performance across diverse degradations by learning a unified restoration model. However, they remain vulnerable to test-time distribution shifts. Existing test-time adaptation (TTA) methods stabilize online parameter updates via Fisher-based selective adaptation, yet this selection criterion is inherently one-sided: it only identifies which parameters to preserve from updates, but fails to distinguish which of the remaining parameters merit adaptation to the target degradation. To address this limitation, we propose Hybrid Generalization-aware Parameter Selection (HGPS), a budget-neutral, per-tensor selection principle. Specially designed to complement Fisher stability, HGPS employs cross-view gradient agreement as a proxy for target-relevant update utility. Within each tensor’s original update budget, HGPS hard-protects high-Fisher elements, assigns part of the update slots to parameters with consistent cross-view gradients, and fills the rest with low-Fisher elements. Furthermore, the mask is computed once before adaptation and can be plugged into Fisher-based TTA pipelines without altering their optimization or restoration schedule. Extensive experiments across multiple restoration tasks and distribution shifts demonstrate that HGPS consistently improves state-of-the-art restoration TTA, surpassing the strongest existing TTA method by 2.69 dB PSNR on Kodak24.