acceptodds
Under review as a conference paper at ICLR 2027

FROM LOW-RANK TO SKETCH: CHEBYSHEV-GUIDED UNFOLDING FOR EFFICIENT PANSHARPENING

Abstract

Pansharpening fuses a high-resolution panchromatic image with a low-resolution multispectral image. Existing methods repeatedly transform high-dimensional features, incurring cost that scales with feature dimensions and resolution. Motivated by the strong inter-band correlation and spatial-spectral redundancy of multispectral images, we design efficient sketch operators that confine the principal transformations to compact channel and spatial subspaces. To avoid the instability of repeatedly performing singular value decomposition (SVD) inside a deep network, we introduce a Chebyshev-Guided Approximation Mechanism that constructs adaptive, approximately row-orthogonal projections directly from Gram statistics. We further introduce a nullspace compensation mechanism to preserve information outside the selected subspaces. In the resulting multigrid-inspired unfolding architecture, termed OSSNet, CSS and CCS respectively provide dataadaptive spatial restriction with adjoint prolongation and compact channel routing at the coarsest level. Level-wise residual-correction blocks refine features before restriction, while coarse representations are prolonged and fused with complementary residuals during decoding. Experiments show that OSSNet achieves competitive reconstruction performance with low parameter count, measured latency, and computational cost.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
pansharpening, sketch, efficient, unfolding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7ZkMOpenReview ↗
Rating
3
Confidence
4(+2)
Informativeness
2
Presentation
3(+1)
Soundness
3
Read review
Paper summary

This paper proposes OSSNet, a pansharpening framework that formulates observation-guided feature refinement as a learned three-level multigrid-inspired V-cycle. It introduces an SVD-free Chebyshev-Guided Approximation Mechanism to construct data-adaptive, row-orthogonal projections, together with nullspace compensation to preserve complementary residuals. Experiments on multiple satellite datasets demonstrate competitive reconstruction quality, strong cross-sensor generalization, and substantially fewer parameters and FLOPs.

Critical strengths

1.OSSNet achieves competitive reconstruction quality while using significantly fewer parameters and FLOPs, demonstrating a favorable accuracy–efficiency trade-off.

2.The proposed SVD-free Chebyshev-Guided Approximation Mechanism constructs data-adaptive, approximately row-orthogonal projections without repeated SVD, avoiding the numerical instability of SVD when embedded in neural network training.

Critical weaknesses

The proposed method shows no clear advantage in full-resolution evaluation.

Questions for authors

In Section 3.1, the authors introduce a fixed random subspace to avoid constructing the full covariance matrix. However, Eq. (16) directly computes the channel covariance matrix. Could the authors further clarify the motivation for introducing the random projection and its role in the actual OSSNet implementation?

5 comments

We sincerely thank the reviewer for the positive assessment of our accuracy–efficiency trade-off and SVD-free Chebyshev mechanism. We address both points below.

W1. Full-Resolution Evaluation and Underlying Trade-offs (Table 2 & Appendix D, Tables 7–8)

We clarify OSSNet's baseline efficiency–accuracy standing, root-cause mechanism, and our targeted parameterization that achieves full-resolution SOTA across all benchmarks:

  • Baseline Pareto Optimality (Table 2): Under the default lightweight setting (K = 3, λ_orth = 10⁻⁴), OSSNet already achieves the overall 2nd-best HQNR across benchmarks (WV3: 0.952, QB: 0.905, GF2: 0.893) with the best/2nd-best spectral fidelity Dλ. The only method slightly ahead (ARConv) requires 28× more parameters (4.79M vs. 0.17M), 15× more FLOPs (61.57G vs. 4.10G), and 18× higher latency (92.56 ms vs. 4.96 ms).
  • Root-Cause Diagnosis (Appendix D, Figs. 7–8): CCS (d=12) routes 61.67% of the stable channel mean through the compact branch while leaving 86.7% of high-frequency energy (ρ_HF, Fig. 8) in the complementary nullspace residual N_c. At full resolution (4× larger grid with dense PAN details), a low Chebyshev order (K=3) has limited spectral transition sharpness, and a mild orthogonality weight (λ_orth = 10⁻⁴) leaves a small residual deviation (Q_c Q_cᵀ ≠ I_d), causing minor high-frequency leakage and slightly higher spatial distortion Dₛ.
  • Targeted Solution Achieving Full-Resolution SOTA (Appendix D, Tables 7–8): Increasing the Chebyshev order to K=6 (sharper spectral filtering) and strengthening orthogonality regularization to λ_orth = 10⁻³ (tighter subspace complementarity) resolves Dₛ with only a marginal increase in parameters (still 0.17M) and computational cost (+0.15G), outperforming previous SOTAs on all full-resolution benchmarks while simultaneously improving reduced-resolution and zero-shot WV2 performance:
Setting (Appendix D, Tables 7–8)WV3 Red. (PSNR/SAM)WV3 Full (HQNR/Dλ/Dₛ)QB Red. (PSNR/ERG)QB Full (HQNR/Dλ/Dₛ)GF2 Red. (PSNR/SAM)GF2 Full (HQNR/Dλ/Dₛ)WV2 ZS (PSNR)Cost (Params/FLOPs)
Prev. Best SOTA37.22 / 3.300.953 / 0.023 / 0.02438.42 / 3.570.915 / 0.052 / 0.03549.44 / 0.800.902 / 0.041 / 0.05929.000.49–4.79M / 13.6–61.6G
OSSNet (K = 3, 10⁻⁴)37.22 / 3.290.952 / 0.023 / 0.02738.67 / 3.470.905 / 0.045 / 0.05449.34 / 0.800.893 / 0.039 / 0.07030.240.17M / 4.10G
OSSNet (K = 6, 10⁻³)37.23 / 3.280.954 / 0.023 / 0.02338.70 / 3.450.917 / 0.044 / 0.04149.32 / 0.800.943 / 0.032 / 0.02630.330.17M / 4.25G

Q1. Random Projection Ω in Section 3.1 vs. Direct Covariance in Eq. (16)

CSS and CCS address different compression scales, and the code and equations are fully consistent. Section 3.1 (Eqs. 1–4) gives the general projected-SVD reference rather than requiring every sketch to use random projection:

  • CSS Uses Ωₛ (Sec. 3.3): Each 2× 2 patch X_patch ∈ ℝ^(C×4) has 4 tokens, making a full C× C covariance rank-deficient and costly over all patches. CSS uses a fixed Ωₛ ∈ ℝ^(16×C) solely to estimate the 4× 4 spatial covariance in a 16-D space before applying Qₛ to the C-channel patch, mirroring landmark/proxy sketching in efficient transformers (Chen et al., 2021; Lee et al., 2024).
  • CCS Omits Ω by Design (Eq. 16, Lines 147–149, 286–290): CCS computes the global C× C channel covariance once per sample at reduced resolution. Since C=48 is compact, direct computation is inexpensive and avoids random-projection bias, keeping the nullspace residual N_c = (I_C − Q_cᵀ Q_c)F accurate while remaining consistent with learnable sketching (Indyk et al., 2019; Liu et al., 2021). We have revised Secs. 3.1 and 3.3 to state this explicitly.

Thank you for the detailed rebuttal. To better validate the performance trade-offs and practical efficiency of the proposed method, I have two remaining requests:

  1. Ablation on Chebyshev Order KK: Please evaluate different KK values (with λorth\lambda_{\text{orth}} fixed) in terms of reconstruction performance, parameters, FLOPs, training time, and inference latency.
  2. Practical Runtime: Beyond parameters and FLOPs, please report the training time and inference latency across all ablation variants (especially Chebyshev CSS vs. Average Pooling).

Thank you for the follow-up and for specifying these two additional evaluations. We address them separately below.

For computational benchmarking in both evaluations, we use an NVIDIA GeForce RTX 5060 Ti GPU with a batch size of 1. The network inputs are an upsampled four-band multispectral (LMS) tensor of size 1×4×256×2561\times4\times256\times256 and a panchromatic (PAN) tensor of size 1×1×256×2561\times1\times256\times256 (batch ×\times channels ×\times height ×\times width). Both inputs are therefore represented at the PAN spatial resolution; at a scale factor of 4, the corresponding original low-resolution multispectral image has a spatial size of 64×6464\times64. These settings apply to the computational benchmarks; the WV3 and WV2 reconstruction results are reported separately below.

1. Ablation on Chebyshev order KK with fixed λorth\lambda_{\mathrm{orth}}

Our previous comparison varied KK and λorth\lambda_{\mathrm{orth}} jointly and therefore did not isolate the effect of polynomial order. We now evaluate K=1,…,8K=1,\ldots,8 with λorth\lambda_{\mathrm{orth}} fixed at 1e-4. Table 4 provides the complementary sensitivity analysis for λorth\lambda_{\mathrm{orth}} at fixed K=3K=3.

KKWV3 PSNR ↑ERGAS ↓SAM ↓WV2 PSNR ↑HQNR ↑DλD_\lambda ↓DsD_s ↓
137.092.273.3229.460.94500.0230.032
237.112.263.3129.870.94760.0230.030
337.222.233.2930.240.95230.0230.027
437.222.243.2830.100.94990.0240.027
537.232.243.2829.980.95110.0250.024
637.242.273.3130.150.94990.0250.026
737.222.243.2929.840.95050.0250.026
837.152.253.3130.080.94830.0250.028

Reconstruction quality generally improves from low to moderate orders, while further increases yield smaller and non-monotonic changes. K=6K=6 achieves the highest WV3 PSNR in this sweep, whereas K=3K=3 achieves the highest HQNR and WV2 PSNR. Increasing KK from 1 to 5 reduces DsD_s from 0.032 to 0.024, and increases DλD_\lambda from 0.023 to 0.025. The sweep characterizes the spatial–spectral trade-off under a fixed orthogonality weight.

KKParameters (M)FLOPs (G)Latency (ms)Peak memory (MB)
10.173.896.6673.66
20.173.956.8173.67
30.174.106.8773.68
40.174.157.0073.69
50.174.207.2173.70
60.174.267.3373.70
70.184.307.5473.71
80.184.357.7173.72

Across the sweep, FLOPs increase from 3.89 to 4.35 G, while parameter counts remain within 0.17–0.18 M at the reported precision. The measured latency ranges from 6.66 to 7.71 ms, and peak memory changes only slightly.

Reviewer_y5aVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

OSSNet is a depth-unfolded panchromatic sharpening method whose architecture borrows from multigrid approaches. It achieves observation-guided feature refinement through a learned three-level V-loop and introduces a Chebyshev-guided approximation mechanism to avoid repetitive singular value decomposition (SVD). This mechanism utilizes Gram statistics to generate input-related spatial and channel projections with approximately orthogonal rows. The authors report state-of-the-art reconstruction results while significantly reducing the number of parameters and floating-point operations, and improving the mobility across satellite sensors.

Critical strengths
  1. Experimental results strongly demonstrate that subspace sketching improves computational efficiency. The comparative results reported in this paper help illustrate the contribution of sketching operations to the overall design.

  2. Analysis of CSS reveals its behavior beyond reconstruction scores more clearly. In particular, energy preservation and PCA distance measurements help characterize the information preserved in the resulting subspace.

  3. The evaluation covers a wide range of competing methods. This coverage strengthens the empirical comparison and comprehensively demonstrates OSSNet's position relative to existing methods.

Critical weaknesses
  1. Visual comparisons show little difference between the methods. Based on the results presented so far, these comparisons offer limited insight into which reconstruction errors OSSNet can address and in which areas it performs poorly. Qualitative evidence would be more valuable if magnified regions with finer structures were provided, and error maps were available with reference images.

  2. OSSNet performs poorly in full-resolution evaluations, but this result is not fully explored. The authors should investigate why the advantages observed in other settings do not consistently persist and identify the conditions for performance degradation. This analysis is crucial for determining the practical application limits of the method.

  3. The current evaluation results do not fully support the rationale for replacing Singular Value Decomposition (SVD) with Chebyshev constructions. Higher reconstruction scores alone cannot explain the numerical and computational trade-offs of this substitution. A comparative study should measure approximation error, row orthogonality bias, runtime, and memory usage under identical conditions. Examining how these quantities vary with approximation order will help further clarify in which conditions the proposed mechanism outperforms SVD.

Questions for authors
  1. Table 2 shows that WFA-Net suffers from memory shortage in terms of latency and peak memory. What hardware configuration, input dimension, batch size, and numerical precision led to this result? Additionally, please explain whether all methods used the same evaluation settings, and whether WFA-Net can run with smaller batch sizes.

  2. GCM ablation and zero-shot evaluation did not demonstrate the advantages of the V-loop structure. Could the authors compare their proposed design with versions that remove coarse mesh correction or replace the V-loop with a simpler refinement sequence, while maintaining consistency in parameter count and computational cost as much as possible?

12 comments

W1. Visual comparisons offer limited insight

We appreciate this valuable feedback. At modern pansharpening fidelity (36–49 dB36\text{--}49\text{ dB} PSNR), full-scene RGB differences naturally approach human perceptual limits. To clearly diagnose where OSSNet excels and where it trades off, we have replaced Figs. 2 and 3 in the main text with magnified, structure-rich patches and uniform-scale absolute error maps (with additional scenes in the revised Appendix):

  • Reduced-Resolution Advantage (Significantly Lower Spectral Distortion): The updated Fig. 2 and reference error maps show that OSSNet's core visual strength is suppressing spectral distortion. Competing methods exhibit noticeable color shifts and bright error residuals along building edges and vegetation, whereas OSSNet yields darker, more uniform error maps—corroborating its best SAM (3.293.29 on WV3, 4.164.16 on QB) and ERGAS in Table 1.
  • Full-Resolution Behavior & Subspace Routing Analysis: In the updated Fig. 3 (see yellow arrows on roof boundaries and surrounding vegetation), competing methods (e.g., DISPNet, WFA-Net, DAPNet) suffer from severe red/blue color fringing and saturation artifacts. Compared to sharp SOTA baselines like DCINN and ARConv, OSSNet eliminates these chromatic artifacts and outperforms most competitors, though its internal roof textures appear slightly smoother. As shown in Appendices C & D (Figs. 5–8), CSS retains 95.89%95.89\% of local patch energy, whereas CCS (d=12d=12) retains 35.53%35.53\% of total channel energy—prioritizing stable channel means (61.67%61.67\%) over centered spatial variations (13.46%13.46\%)—while routing 64.5%64.5\% of total energy and 86.7%86.7\% of high-frequency energy (ρHF\rho_{\mathrm{HF}}, Fig. 8) into the complementary nullspace residual Nc\mathbf{N}_{\mathrm{c}}. Under default compression, this conservative routing suppresses chromatic noise at the cost of mild spatial smoothing.
  • Targeted Solution (Full-Resolution SOTA Achieved): Guided by Appendix D, this spatial–spectral trade-off is directly controllable via the Chebyshev order KK and orthogonality weight λorth\lambda_{\mathrm{orth}}. Adjusting these parameters recovers sharp high-frequency details while preserving spectral purity, enabling OSSNet to achieve SOTA full-resolution metrics across all benchmarks (see Response to W2 and Appendix D, Tables 7–8).

W2. Full-resolution behavior and underlying trade-offs

We clarify OSSNet's baseline standing, root-cause mechanism, and our targeted parameterization that achieves full-resolution SOTA across all benchmarks (added to Appendix D, Tables 7–8):

  • Baseline Efficiency–Accuracy Pareto Optimality (Table 2): Even under the default lightweight setting (K=3, λ_orth=10⁻⁴), OSSNet does not perform poorly: it achieves the overall 2nd-best HQNR across benchmarks (WV3: 0.952, QB: 0.905, GF2: 0.893) with the best/2nd-best spectral fidelity Dλ. Crucially, the only method slightly ahead (ARConv) requires 28× more parameters (4.79M vs. 0.17M), 15× more FLOPs (61.57G vs. 4.10G), and 18× higher latency (92.56 ms vs. 4.96 ms).

  • Root Cause of Baseline Spatial Distortion (Dₛ): As diagnosed in Appendix D (Figs. 7–8), CCS routes 61.67% of the channel mean through the compact branch while leaving 86.7% of high-frequency energy in the nullspace residual N_c. At full resolution (4× larger grid with dense PAN details), a low Chebyshev order (K=3) has limited spectral transition sharpness, and a mild orthogonality penalty (λ_orth=10⁻⁴) leaves a small residual deviation (Q_c Q_cᵀ ≠ I_d), causing minor high-frequency leakage between the compact subspace and N_c.

  • Targeted Solution Achieving Full-Resolution SOTA (Appendix D, Tables 7–8): Increasing the Chebyshev order to K=6 (sharper high-frequency spectral filtering) and strengthening orthogonality regularization to λ_orth=10⁻³ (tighter subspace complementarity) resolves the Dₛ bottleneck with only a marginal increase in parameters (still 0.17M) and computational cost (+0.15G), outperforming previous SOTAs on all full-resolution benchmarks while simultaneously improving reduced-resolution and zero-shot WV2 performance:

Setting (Tables 7–8)WV3 Red. (PSNR/SAM)WV3 Full (HQNR/Dλ/Dₛ)QB Red. (PSNR/ERG)QB Full (HQNR/Dλ/Dₛ)GF2 Red. (PSNR/SAM)GF2 Full (HQNR/Dλ/Dₛ)WV2 ZS (PSNR)Cost (Params/FLOPs)
Prev. Best SOTA37.22 / 3.300.953 / 0.023 / 0.02438.42 / 3.570.915 / 0.052 / 0.03549.44 / 0.800.902 / 0.041 / 0.05929.000.49–4.79M / 13.6–61.6G
OSSNet (K=3, 10⁻⁴)37.22 / 3.290.952 / 0.023 / 0.02738.67 / 3.470.905 / 0.045 / 0.05449.34 / 0.800.893 / 0.039 / 0.07030.240.17M / 4.10G
OSSNet (K=6, 10⁻³)37.23 / 3.280.954 / 0.023 / 0.02338.70 / 3.450.917 / 0.044 / 0.04149.32 / 0.800.943 / 0.032 / 0.02630.330.17M / 4.25G

W3. Rationale and Trade-offs of Replacing SVD with Chebyshev Constructions

Chebyshev is not a general-purpose numerical SVD solver, but a hardware-friendly, differentiable spectral filter. Together with Appendices B–E and the benchmark in Appendix F.3 (Table 11), we evaluate all requested metrics under identical conditions:

Method (Appendix F.3, Table 11)Forward (ms) ↓Backward (ms) ↓Total (ms) ↓Peak Mem (MB) ↓WV3 / WV2 (dB) ↑
Stabilized SVD (w/o Cheb.)26.1016.7042.8071.1937.04 / 29.34
Chebyshev (K=3, Ours)6.0517.5123.5671.2637.22 / 30.24

1. Runtime, Memory & Why Avoid Stabilized SVD (Appendix F.3, Table 11):

  • Efficiency: Using pure GPU GEMMs instead of iterative cuSOLVER decompositions, Chebyshev cuts forward latency by 76.8% (4.31× faster) and total step time by 45.0% (23.56 vs. 42.80 ms) at virtually identical peak memory (71.26 vs. 71.19 MB).
  • Gradient Fidelity vs. Clamped SVD: Modern torch.linalg.svd avoids NaN crashes and maintains fast backward passes (16.70 ms) via heuristic eigen-gap clamping on (σ_i² − σ_j²)⁻¹ (Ionescu et al., 2015). However, when non-dominant singular values cluster (σ_i ≈ σ_j), clamping zeroes out or biases true gradients, while hard top-d truncation induces discrete basis switching when singular values swap ranks. Chebyshev polynomials provide exact, smooth analytical gradients everywhere and enable task-adaptive filtering, outperforming SVD by +0.18 / +0.90 dB on WV3/WV2 (+0.64 dB in patch-wise CSS, Appendix E.2, Table 10) and surpassing Power/Bernstein/Jacobi/Legendre bases (Appendix B, Table 6).

2. Approximation Error, Orthogonality Bias & Order K (Appendices C, D, F.2):

  • Subspace Proximity (δ_sub, Appendices C & D, Figs. 5–7): Against exact top-d SVD, CSS closely tracks local SVD (95.89% vs. 96.90% energy; δ_sub=0.119; alignment 0.988–0.989), whereas CCS intentionally learns a task-adaptive channel subspace distinct from PCA (δ_sub=1.317).
  • Row-Orthogonality Bias (Appendix F.2): CSS (Qₛ ∈ ℝ^(1×4)) is exact (Qₛ Qₛᵀ = 1) via ℓ₂ normalization. For CCS (d=12), 5 Newton–Schulz steps locally approximate Q_c Q_cᵀ ≈ I_d, while L_orth penalizes the terminal deviation during training (+0.63 dB on WV2, Table 4).
  • Order K (Appendix D, Tables 7–8): Increasing K=3 → 6 (λ_orth=10⁻³) sharpens spectral selectivity at negligible cost (+0.15G FLOPs, still 0.17M params) and achieves full-resolution SOTA across all benchmarks.
Reviewer_6xZmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OSSNet, a lightweight pansharpening network organized as a learned three-level multigrid V-cycle. To compress spatial-spectral features without unstable SVD backpropagation, the authors introduce a Chebyshev-Guided Approximation Mechanism that constructs near-row-orthogonal spatial (CSS) and channel (CCS) projections via Chebyshev matrix polynomials and Newton–Schulz iterations. A nullspace compensation branch retains the complementary residual outside the projected subspace. Experiments on WV3, QB, GF2, and zero-shot WV2/QB benchmarks show that OSSNet achieves SOTA-comparable fidelity and strong cross-sensor generalization with only 0.17M parameters, 4.10G FLOPs.

Critical strengths
  1. Approximating spectral truncation via Chebyshev polynomials and Newton–Schulz iterations elegantly bypasses SVD gradient instability, while explicit nullspace compensation prevents the loss of high-frequency details during dimensionality reduction.

  2. The proposed OSSNet delivers top-tier accuracy and remarkable zero-shot cross-sensor transfer at minimal computational cost.

Critical weaknesses
  1. Chebyshev polynomials Tj(x)T_j(x) diverge outside [−1,1][-1, 1], and Newton–Schulz iterations require ∥I−Qraw(Qraw)⊤∥2<1\|\mathbf{I} - \mathbf{Q}^{\text{raw}}(\mathbf{Q}^{\text{raw}})^\top\|_2 < 1 to converge. However, Eqs. (16)–(18) neither normalize the spectral radius of G\mathbf{G} into [−1,1][-1, 1] nor specify how Qraw\mathbf{Q}^{\text{raw}} is scaled before Newton–Schulz orthogonalization.

  2. Section 3.1 introduces random projection Ω\mathbf{\Omega} to avoid full covariance computation, yet CCS (Eq. 16) drops Ω\mathbf{\Omega} entirely.

  3. OSSNet runs only one V-cycle without multi-stage unfolding iterations, omits observation consistency (GCM) in the decoder (Fig. 1), and replaces additive nullspace completion (Eq. 14) with 1×11\times 1 conv fusion. It seems to behave more like an enhanced U-Net than a true iterative unfolding solver.

  4. WV3 HQNR jumps from 0.952 (Table 2) to 0.970 (Table 4); and removing CSS or Nullspace barely affects in-distribution WV3 (Table 3).

Questions for authors
  1. How are G\mathbf{G} and Qraw\mathbf{Q}^{\text{raw}} normalized to guarantee bounded Chebyshev polynomials and Newton–Schulz convergence? If 5 Newton–Schulz steps already orthogonalize Q\mathbf{Q} (and CSS achieves QsQs⊤=1\mathbf{Q}_{\text{s}}\mathbf{Q}_{\text{s}}^\top=1 as a 1×41\times 4 vector), why is Lorth\mathcal{L}_{\text{orth}} (Eq. 20) still needed for CCS?

  2. Why use Newton–Schulz and Chebyshev polynomials per patch? What happens in Table 3 if CSS is replaced by simple 2×2 average pooling + complementary residual?

4 comments

We thank the reviewer for the careful reading. We have revised the manuscript (highlighted in blue) and Appendix to make the spectral normalization, random projection, multigrid design, and evaluation consistency explicit.

W1. Chebyshev Boundedness and Newton–Schulz Convergence

  • Chebyshev Spectral Boundedness (Lines 289–292, Appendix F.1): To guarantee the Chebyshev recurrence operates within [−1,1][-1, 1] without iterative eigenvalue estimation, we normalize the PSD covariance G⪰0G \succeq 0 using its Schatten-4 norm upper bound: u4(G)=(tr(G4))1/4=∥G2∥F1/2≥λmax⁡(G)u_4(G) = (\text{tr}(G^4))^{1/4} = \lVert G^2 \rVert_F^{1/2} \ge \lambda_{\max}(G). With u=(1+δ)u4(G)+ϵu = (1 + \delta)u_4(G) + \epsilon (δ=10−6,ϵ=10−8\delta = 10^{-6}, \epsilon = 10^{-8}), the affine mapping Gcheb=2(G/u)−ICG_{\text{cheb}} = 2(G/u) - I_C guarantees λi(Gcheb)∈[−1,1]\lambda_i(G_{\text{cheb}}) \in [-1, 1] and ∥Tj(Gcheb)∥2≤1\lVert T_j(G_{\text{cheb}}) \rVert_2 \le 1 in exact arithmetic. Appendix F.1 details the entry-scaled implementation (B=G/sB = G/s) that prevents squaring overflow and stabilizes gradients near zero covariance.
  • CSS vs. CCS Orthogonalization: CSS outputs a single row Qs∈R1×4Q_s \in \mathbb{R}^{1 \times 4}, so exact row orthogonality (QsQs⊤=1Q_s Q_s^\top = 1) is achieved by direct ℓ2\ell_2 normalization without Newton–Schulz. For CCS (Qc∈Rd×C,d>1Q_c \in \mathbb{R}^{d \times C}, d > 1), QrawQ^{\text{raw}} is cast to FP32, divided by max⁡(∥Qraw∥F,10−8)\max(\lVert Q^{\text{raw}} \rVert_F, 10^{-8}) so its initial spectral norm is ≤1\le 1, and refined via five Newton–Schulz iterations.
  • Role of Eq. (20) (Appendix F.2): Chebyshev polynomials construct an adaptive compression matrix rather than performing an SVD, and do not automatically ensure row orthogonality. We approximate QQ⊤≈IdQ Q^\top \approx I_d via five forward Newton–Schulz steps, while Eq. (20) (Lorth\mathcal{L}_{\text{orth}}, on the four channel sketches only) penalizes the remaining deviation during training so QrawQ^{\text{raw}} stays within the local convergence basin.

W2. Random Projection Ω\Omega in Section 3.1 vs. Direct Covariance in Eq. (16)

CSS and CCS address different compression scales, and the code and equations are consistent. Sec. 3.1 (Eqs. 1–4) gives the general projected-SVD reference rather than requiring every sketch to use random projection:

  • CSS Uses Ωs\Omega_s (Sec. 3.3): Each 2×2 patch Xpatch∈RC×4X_{\text{patch}} \in \mathbb{R}^{C \times 4} has 4 tokens, making a full C×CC \times C covariance rank-deficient and costly over all patches. CSS uses a fixed Ωs∈R16×C\Omega_s \in \mathbb{R}^{16 \times C} solely to estimate the 4×4 spatial covariance in a 16-D space before applying QsQ_s to the CC-channel patch.
  • CCS Omits Ω\Omega by Design (Eq. 16, Lines 147–149, 286–290): CCS computes the global C×CC \times C channel covariance once per sample at reduced resolution. Since C=48C = 48 is compact, direct computation is inexpensive and avoids random-projection bias, keeping the nullspace residual Nc=(IC−Qc⊤Qc)FN_c = (I_C - Q_c^\top Q_c)F accurate while remaining consistent with learnable sketching (Indyk et al., 2019; Liu et al., 2021).

W3. Single V-Cycle, Asymmetric Smoothing, and Learned Fusion (Lines 165–168, 252–256, 333–337; Appendix Sec. E)

  • Multigrid V-Cycle: OSSNet unfolds a single nonlinear Multigrid (FAS) V-cycle rather than multi-stage single-grid iterations (Briggs et al., 2000). One V-cycle damps multi-scale errors; stacking dual V-cycles nearly doubles cost (7.95G FLOPs) for +0.06 dB on WV3 while dropping WV2 by 0.22 dB.

  • Asymmetric Smoothing (No Decoder GCM): Multigrid theory allows asymmetric pre-/post-smoothing (Trottenberg et al., 2001; He & Xu, 2019). Encoder GCMs enforce observation consistency before restriction; prolonged decoder features inherit this consistency and only need LPM post-smoothing to remove interpolation artifacts.

  • Learned 1×11\times 1 Conv Fusion (Eq. 15) & U-Net Distinction: Because bottleneck features undergo nonlinear refinement B(Zc)\mathcal{B}(\mathbf{Z}_{\mathrm{c}}) and AlAl∗≈I\mathcal{A}_l\mathcal{A}_l^{\ast}\approx\mathbf{I}, a 1×11\times 1 conv reconciles scale shifts. Unlike U-Nets, OSSNet uses orthogonal restriction/prolongation and routes only nullspace residuals Nl=(Il−Al∗Al)Fl\mathbf{N}_l=(\mathbf{I}_l-\mathcal{A}_l^{\ast}\mathcal{A}_l)\mathbf{F}_l:

Topology Variant (Appendix Sec. E)WV3 / WV2 PSNR (dB)FLOPs / Params
Single-scale sequential36.99 / 27.9010.14G / 0.16M
Rigid Additive Fusion (Eq. 14)37.06 / 29.533.72G / 0.16M
Symmetric Decoder GCM37.10 / 30.205.41G / 0.21M
Dual Cascaded V-Cycles37.28 / 30.027.95G / 0.31M
AvgPool + 1×11\times 1 Conv (U-Net-like)37.15 / 28.053.77G / 0.13M
OSSNet (Single V-Cycle, Ours)37.22 / 30.244.10G / 0.17M

W4. HQNR Consistency (Table 2 vs. Table 4) and Cross-Sensor Gains

  • HQNR Unification & Reproducibility: Table 4 originally noted (original ablation protocol) because it used an early PyTorch HQNR script. We have updated Table 4 in the revision using the unified GRSM MATLAB benchmark (0.952 at λorth=10−4\lambda_{\text{orth}} = 10^{-4}, matching Table 2); the sensitivity trend is unchanged. Full code, pre-trained weights, evaluation scripts, and datasets are provided at Anonymous Code.
  • In-Distribution vs. Cross-Sensor Role (Sec. 4.3, Table 3): Removing CSS or nullspace barely affects in-distribution WV3 (-0.03 and 0.00 dB) because unconstrained convs easily memorize same-sensor statistics. Instead, they act as geometric regularizers against sensor shift: removing CSS or nullspace drops zero-shot WV2 PSNR by 1.47 dB and 0.47 dB, while removing CSS also raises cost (4.83G vs. 4.10G).

Q1. Normalization of GG and QrawQ^{\text{raw}}, and Need for Eq. (20)

As detailed in W1 and Appendix F.1, GG is normalized via its entry-scaled Schatten-4 bound u4(G)=∥G2∥F1/2u_4(G) = \lVert G^2 \rVert_F^{1/2} into Gcheb=2(G/u)−ICG_{\text{cheb}} = 2(G/u) - I_C, confining eigenvalues to [−1,1][-1, 1] (Lines 290–292). QrawQ^{\text{raw}} is Frobenius-normalized before five Newton–Schulz steps. Since five fixed steps only locally approximate QQ⊤≈IdQ Q^\top \approx I_d, Eq. (20) (Appendix F.2) penalizes the remaining terminal deviation during training, boosting zero-shot WV2 by 0.63 dB (Table 4).

Q2. Per-Patch Chebyshev in CSS and Controlled Pooling Ablations (Appendix E.2, Table 10)

CSS normalizes Qs∈R1×4Q_s \in \mathbb{R}^{1 \times 4} directly without Newton–Schulz. Per-patch Chebyshev adapts projection directions to local geometry (departing from uniform 1214⊤\frac{1}{2}\mathbf{1}_4^\top on textured patches, Appendix C, Fig. 6). Controlled ablations in Appendix E.2 (Table 10) confirm its necessity:

Spatial Restriction Variant (Appendix E.2, Table 10)WV3 / WV2 PSNR (dB) ↑FLOPs / Params ↓
AvgPool / MaxPool + Comp. Residual37.19 / 30.11 ; 37.02 / 29.994.08G / 0.16M
Static Learnable 1×4 Projection37.20 / 29.724.08G / 0.16M
Explicit Per-Patch SVD/PCA37.17 / 29.604.08G / 0.16M
CSS (Adaptive Chebyshev, Ours)37.22 / 30.244.10G / 0.17M

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.