Adaptive Frequency Attention: Frequency-domain Context Modeling for Medical Image Segmentation

U-Net and its variants have become the cornerstone of medical image segmentation by effectively leveraging semantic context to distinguish regions with similar local features. However, convolution-based modules are inherently limited in modeling long-range contextual dependencies. To address this, recent works incorporate Transformer modules to capture global context via self-attention mechanisms. In this work, we explore an alternative direction based on frequency-domain learning, where global dependencies can be captured implicitly and efficiently. We propose Adaptive Frequency Attention (AFA), a lightweight plug-and-play module for U-Net-based architectures that adaptively captures informative frequency components to enhance contextual representations. The AFA module is integrated into the encoder with a residual connection, enabling simultaneous modeling of global context and local details. By applying the Discrete Fourier Transform, spatial information is decomposed into multi-scale frequency components, allowing AFA to adaptively emphasize discriminative spectral responses with a learnable mask. To encourage diverse frequency representations, feature channels are partitioned into groups with an orthogonality regularization to promote complementary spectral focus. In addition, a spectral preservation constraint dynamically regulates suppression strength to prevent the loss of critical frequency information. Extensive experiments on multiple medical image segmentation benchmarks demonstrate that AFA consistently improves U-Net variants and achieves state-of-the-art performance.