acceptodds
Under review as a conference paper at ICLR 2027

Symmetric Attention Improves Optimization Geometry for Image Restoration

Abstract

Self-attention has become the workhorse of Transformer-based image restoration, yet its optimization behavior remains largely underexplored. We identify two structural issues arising from the standard asymmetric score matrix : (i) the gradients with respect to and are governed by structurally different operators, allowing the two projections to drift along uncoupled directions and creating flat valleys in the loss landscape; (ii) the asymmetric score matrix may admit complex eigenvalues, whose imaginary components induce rotational dynamics during gradient descent. Together, these effects lead to ill-conditioned optimization and unstable training. Our main contribution, the Balanced Kernel (BK), resolves both issues through a lightweight symmetric score construction that couples the gradient dynamics of and and guarantees a real eigenvalue spectrum. BK introduces negligible additional FLOPs while achieving faster convergence, oscillation reduction, and a substantially more compact optimization basin. When integrated into existing architectures such as Restormer and Uformer, BK consistently improves performance, demonstrating largely architecture-agnostic optimization benefits. We further introduce ChannelRouter, which generates instance-adaptive head weights from degradation-aware channel statistics, and Gated Feature Modulation (GFM), which enables spatially adaptive processing through multiplicative gating. Integrating these components, our Balformer achieves state-of-the-art performance on deraining, deblurring, and denoising benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image Restoration, Self-Attention, Optimization, Transformer, Balformer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5CgAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a lightweight symmetric Balanced Kernel to resolve the optimization bottlenecks of standard asymmetric channel-wise self-attention.

Critical strengths

The paper identifies optimization bottlenecks in standard asymmetric self-attention and resolves them via the Balanced Kernel (BK), a parameter-free, one-line symmetric score construction with negligible computational overhead.

Critical weaknesses
  1. Line 302 claims that Balformer is evaluated on 'single-image motion deblurring,' yet no motion deblurring experiments are presented in the paper, only defocus deblurring comparisons (on the DPDD dataset) are reported in Section 4.2.
  2. 'Balanced Kernel Attention' is essentially symmetric attention rather than a spatial convolutional kernel, so why do the authors use such a potentially misleading name as 'Balanced Kernel'?
Questions for authors

In comparison to mainstream restoration methods (e.g., Restormer), comparative experiments on motion deblurring and Gaussian denoising are missing. Could the authors provide results on these standard tasks to more comprehensively validate the effectiveness of Balformer?

Reviewer_8LLMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a symmetric attention mechanism for image restoration. The authors argue that conventional attention is inherently asymmetric, which can lead to ill-conditioned optimization and unstable training. To address this issue, they introduce a Balanced Kernel that constructs symmetric attention scores and claim that it enables faster convergence. In addition, they propose a ChannelRouter and a Gated Feature Modulation (GFM) module. Experimental results on deraining, deblurring, and denoising benchmarks show that the proposed method outperforms existing approaches.

Critical strengths

The paper proposes the Balanced Kernel (BK) and provides a derivation of its gradient computation. A notable practical advantage is that symmetric attention scores can be obtained with only minor modifications to the standard attention computation, which makes the method easy to implement and integrate into existing architectures without substantial engineering overhead.

The authors further demonstrate the generality of the proposed kernel by applying BK to well-established image restoration models, such as Uformer and Restormer. In both cases, BK yields modest but consistent performance improvements, suggesting that the method can serve as a plug-in component for a range of transformer-based restoration backbones rather than being tied to a specific architecture.

In addition, the ablation study compares the proposed kernel form against several alternative kernel functions, and the results show that the proposed formulation achieves better performance. This comparison helps justify the specific design choice of the kernel and provides empirical support for the authors’ argument that a balanced, symmetric formulation is beneficial for image restoration.

Critical weaknesses
  • Comparison with recent state-of-the-art methods is insufficient. Image restoration has advanced quickly, and PSNR/SSIM on standard benchmarks improves every year. The proposed method mainly shows an advantage over methods published before 2023 (few papers in 2024 and 2025). However, the manuscript omits several recent methods that report higher performance on the same restoration tasks and the same datasets. The authors should include comparisons with the following methods.

    • Deraining (Rain200L). The authors should compare their method with MPHM (AAAI 2026), FADformer (ECCV 2024), MSDT (AAAI 2024), and NeRD-Rain (CVPR 2024). The authors should also confirm that their evaluation protocol matches these methods.

    • Single-image defocus deblurring (DPDD). The authors should compare their method with ErA (NeurIPS 2025 Workshop), PPTformer (AAAI 2025), SFHformer (ECCV 2024), P2IKT (AAAI 2024), and FSNet (IEEE TPAMI 2024). Since DPDD results are reported with either single-image or dual-pixel inputs across papers, the comparison should include only methods that use the same single-image setting.

    • Real-world image denoising (SIDD). The authors should compare their method with CGNet (TMLR 2024) and NAFNet (ECCV 2022).

  • Figure 1(a) compares the training behavior of the baseline and BK in terms of convergence and oscillation. I partially agree that BK reduces oscillation during training. However, the claim of faster convergence is not convincing, because the difference in loss between the two curves is small. In addition, BK changes the loss function itself, so the loss values of the two methods are not directly comparable. It is therefore unclear whether comparing how quickly each method reaches a given loss value is meaningful.

Questions for authors
  1. How does the proposed method perform against more recent state-of-the-art methods on the same tasks and datasets (Rain200L, DPDD, and SIDD), evaluated under the same protocol (e.g., the color space used for PSNR/SSIM and the single-image setting for DPDD)?
  2. In Figure 1(a), the baseline and BK use different loss functions, and the difference between the loss curves appears small. On what basis do the authors claim faster convergence and reduced oscillation? Could they support this claim with a metric shared by both methods, such as validation PSNR/SSIM over training iterations?
Reviewer_N5HROpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors investigate the advantages of a symmetric attention score over an asymmetric one in the image restoration domain. Their experiments show that the symmetric attention matrix leads to faster convergence speeds, and different levels of improvement across various restoration tasks. Together with several architectural designs such as ChannelRouter and Gate Feature Modulation, the proposed Balformer demonstrates state-of-the-art performance on multiple image restoration datasets.

Critical strengths
  1. The motivation of this paper is relatively clear. Inspired by previous works on Kernels (scaled dot-product), such as Primal-Attention overcoming asymmetry by dual-space decomposition, the authors propose a Balanced Kernel method (symmetric formulation: it seems to be one matrix transpose operation and one add operation) to construct a symmetric attention matrix. Figure 1 (a) clearly shows that the training convergence speed of Restormer and Balformer with the proposed kernel is notably faster than the baselines without that kernel.

  2. Quantitative comparisons with existing works show that the proposed network Balformer achieves state-of-the-art performance on defocus deblurring, denoising, and deraining tasks. Ablation studies also verify that the proposed symmetric formulation clearly improves the deraining performance under various kernel functions.

Critical weaknesses
  1. Insufficient experiments in the main paper: First, the authors provide ablation studies of the architectural designs (component-wise studies) in the appendix, which should be moved to the main paper, as these might be considered as the part of contributions improving the network's performance up to state-of-the-art. Second, as the symmetric formulation is the core contribution, the authors are expected to provide more ablation results or observations on the other restoration tasks and datasets; otherwise, it might be hard for the reviewer to tell whether or not the performance gain of symmetric formulation is domain-dependent or not. Any observation will be appreciated.

  2. Concerns about the symmetricity: The reviewer is confused about how SsymS_{sym} is implemented, and when the softmax function is applied to the scaled dot products. The authors should clarify it in the main paper if they have not.

Questions for authors

Please see the weaknesses.

Comments on the paper

1 comment
Area_Chair_N8WLOpenReview ↗

AC Reminder: Please Follow Up on the Authors’ Responses

Dear Reviewers,

Thank you for your time and thoughtful reviews. We are now about halfway through the discussion period, and the authors have posted responses to your questions and concerns.

If you have not yet done so, please read their responses and any relevant revisions, and leave a follow-up comment visible to the authors indicating which concerns have been addressed and which remain unresolved. Please update your review and rating if warranted. If your assessment remains unchanged, a brief explanation would still be helpful.

Thank you for your continued engagement in the discussion.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.