acceptodds
Under review as a conference paper at ICLR 2027

Low-Frequency Shortcuts in Texture-Driven Visual Learning

Abstract

Neural networks suffer from shortcut learning, where learned features generalize well to the training set but not to in-distribution (ID) or out-of-distribution (OOD) test sets. Existing studies are all based on a few standard benchmarks, which are shape-driven. Numerous application domains, however, are texture-driven. In this work, we present shortcut learning analysis for texture-driven domains, and compare it with that of a standard benchmark. We show that texture-driven domains suffer from low-frequency shortcuts. They make the majority of their decisions based on a few low-frequency components (LFCs) with a skewed spectral behavior, despite that higher-frequency components (HFCs) have a higher predictive power. Pruning LFCs from training and test sets mitigates the shortcut and provides a more balanced spectral behavior, improving the ID accuracy by up to 10% and OOD accuracy by up to 40% under algorithmic and real-world domain shifts. We show that general-purpose and domain-specific foundation models can also suffer from low-frequency shortcuts. While large models can mitigate the shortcuts, they incur a high computational cost and may result in a significantly lower accuracy than shortcut-pruned from-scratch trained small models. We show that reduced image resolutions amplify the degree of shortcuts; large frequency-transformation block sizes capture low-frequency shortcuts better than small block sizes; and, low-frequency shortcuts persist across different color spaces. Our findings provide valuable insights, which we hope will be useful for practitioners working on new, understudied domains.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
spectral bias, shortcut learning, generalization, robustness, signal processing, image compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_CDzKOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends frequency analysis to texture-driven domains and identifies low-frequency shortcuts. The main claim is that neural network predictions are dominated by low-frequency components even though higher-frequency components have higher predictive power. The paper introduces a pruning strategy that eliminates low-frequency components from training and test sets, and empirically validates its effectiveness on both in-domain and out-of-domain data. The paper further studies the impact of large-scale training and image resolutions on low-frequency shortcuts.

Critical strengths
  • The paper extends frequency analysis to texture-driven domains and identifies low-frequency shortcuts in neural networks trained on these domains.
  • The main claim is supported by extensive empirical validation across various datasets and model architectures under well-controlled validation settings.
Critical weaknesses
  • W1: The paper focuses on identifying low-frequency shortcuts in texture-driven domains but provides limited explanation of why these arise, particularly in texture-driven domains. As a result, it remains unclear whether low-frequency shortcuts arise from texture-driven domains or from the biases of neural networks. Although the paper claims that texture/shape bias of neural network architectures is orthogonal to the goal of this work, further discussion or analysis of the underlying causes would help clarify the phenomenon.
  • W2: The paper lacks of both conceptual and technical novelty. Prior works have investigated frequency shortcuts and effectiveness of high-frequency components, leaving domain extension as the main novelty. Also, frequency analysis relies on commonly used methods with an additional pruning strategy, which is closer to an engineering approach than a novel algorithm.

That said, I am willing to revise my rating based on the discussion.

Questions for authors
  • Q1: The finding that pruning low-frequency components leads to better generalization seems task-specific, as texture-driven tasks require texture-specific information, and results with non-texture-domain dataset (e.g., CIFAR10) do not exhibit a similar phenomenon. Is this observation generalizable across different domains?
  • Q2: What is the purpose of the paragraph `Prior Theoretical Work.' in Section 5? Current version of the paper does not provide why this discussion is relevant.
  • Q3: In Section 6, the paper states that `generalization performance decreases if the corruption corrupts the HFCs that the model has started to rely on'. However, in some datasets performance drop on OOD data occurs after pruning far fewer components than ID data. Isn't it possible to say that the model also relied on some part of the HFCs rather than dominant LFCs?
  • Q4: In Section 7, frequency analysis on GalaxyMNIST and EuroSAT is said to have a skewed spectral behavior towards LFCs, but pruning LFCs results in moderate drops in in-domain accuracy unlike other texture-driven datasets. This raises the question of whether low-frequency shortcuts and improvements from pruning are consistently observed across texture-driven datasets.
  • Q5: Section 4 states that validation accuracies are reported in the main paper. Is there a reason why test accuracies are not used for the analysis?
  • Q6: How exactly is `accuracy contribution' measured?
11 comments
AuthorsOpenReview ↗

Response to Reviewer CDzK – Part 1

W1: The paper focuses on identifying low-frequency shortcuts in texture-driven domains but provides limited explanation of why these arise, particularly in texture-driven domains. As a result, it remains unclear whether low-frequency shortcuts arise from texture-driven domains or from the biases of neural networks. Although the paper claims that texture/shape bias of neural network architectures is orthogonal to the goal of this work, further discussion or analysis of the underlying causes would help clarify the phenomenon.

Response Part 1: Neural Network vs. Texture-Driven Domains. Neural networks are biased towards learning from low-frequency components (LFCs) (our [22; 31; 51; 64; 67; 70; 72; 81; 98; 99; 105]). Texture-driven domains perform classification using fine-grained higher-frequency details. Hence, neural networks’ bias towards LFCs forms an easy shortcut for texture-driven classification tasks, resulting in suboptimal accuracy. Removing the shortcuts from the training and test sets improves the accuracy, sometimes as high as by up to 10%.

So, the shortcuts are due to both the low-frequency bias of neural networks and the high-frequency dependency of texture-driven domains.

Response Part 2: Texture/Shape Bias vs. Low-Frequency Shortcuts. Yes, these two dimensions are completely independent. Looking for high-frequency information to differentiate different types of textiles or fake-vs-real images is different than looking for low-frequency information when images are corrupted with random textures. Geirhos et al. (our [35]) change the texture of images in shape-driven tasks. This is not what we are doing. We analyze classification tasks that look for high-frequency information. None of the existing work has looked at such datasets. Furthermore, what level of texture corruption changes what part of the frequency spectrum and how much are open research questions, as we also say in our related work.

W2: The paper lacks both conceptual and technical novelty. Prior works have investigated frequency shortcuts and the effectiveness of high-frequency components, leaving domain extension as the main novelty. Also, frequency analysis relies on commonly used methods with an additional pruning strategy, which is closer to an engineering approach than a novel algorithm.

Response: Our work does not investigate the effectiveness of high-frequency components. It investigates the harm of low-frequency components, which none of the existing work has done. Furthermore, none of the prior work has performed an interventional analysis, where they change the training set. They only perform observational analysis, where they modify only the test set. Almost every single experiment in our work requires re-training the model from scratch, identifying a new shortcut based on how neural networks work. It has implications for interpretability and predictability, as we are investigating spectral behavior in connection with ID/OOD accuracies.

Yes, new domains are a great strength of our work. It is not the only strength, however. We also show groundbreaking results. These domains are already being studied by numerous groups using sophisticated network architectures or training regimes [1, 2]. We show that you can achieve similar and even better results by simply studying the data. This opens a whole new line of research. Now that the datasets are free of low-frequency shortcuts, what kind of network architectures, data augmentation algorithms, and training regimes should we use to achieve an even higher accuracy? How can we create a pre-training set for a new foundation model while studying shortcuts side-by-side? These are exciting and groundbreaking research questions. As we have AI in our daily lives more and more, identifying shortcuts and transparently improving model quality by studying its learning behavior, e.g., spectral behavior, is key to making AI models more reliable and predictable.

Our results are even more interesting, as we show that we can achieve a higher accuracy with less data. Particularly, for Tiny-Gen-Images, we can remove 50% of the data and achieve 10% higher accuracy. This is unique. The conventional approach to neural networks and data compression is that reducing data typically causes loss of quality, or at best, preserves the quality. For the first time, we are showing that you can actually achieve a higher quality with less data. This is groundbreaking, as the cost of storing, moving, and processing data is ever increasing.

[1] Jia Xue, Hang Zhang, Kristin Dana. Deep Texture Manifold for Ground Terrain Recognition. CVPR 2018. https://arxiv.org/abs/1803.10896.

[2] Mingjian Zhu, Hanting Chen, Mouxiao Huang, Wei Li, Hailin Hu, Jie Hu, Yunhe Wang. GenDet: Towards Good Generalizations for AI-Generated Image Detection. https://arxiv.org/abs/2312.08880.

AuthorsOpenReview ↗

Response to Reviewer CDzK – Part 2

Q1: The finding that pruning low-frequency components leads to better generalization seems task-specific, as texture-driven tasks require texture-specific information, and results with a non-texture-domain dataset (e.g., CIFAR10) do not exhibit a similar phenomenon. Is this observation generalizable across different domains?

Response: We are not sure what you mean by “different domains”. As we show, out of 12 different combinations of four domains and four models, we achieve improved ID and/or OOD accuracy for 10 cases. This shows that, yes, our results do appear in multiple domains again and again.

If you mean whether it generalizes to all the domains, the answer is no. Low-frequency shortcuts do not appear in all domains, all the time. This is almost impossible for any real-life phenomenon, unfortunately. We study a large number of domains and use cases and report successful results on a large number of domains while also reporting the failures across different use cases.

It has been repeatedly shown that neural networks are biased towards low-frequency components (our [22; 31; 51; 64; 67; 70; 72; 81; 98; 99; 105]). We are showing that this LFC bias is a significant problem for numerous real-world domains that require information at higher frequencies.

[1] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report No. CNS-TR-2011-001, 2011. https://www.vision.caltech.edu/datasets/cub_200_2011.

Q2: What is the purpose of the paragraph `Prior Theoretical Work.' in Section 5? Current version of the paper does not provide why this discussion is relevant.

Response: Simplicity bias is fundamentally a theoretical concept. We touch on the theoretical aspect and cite the relevant work for the interested reader.

Q3: In Section 6, the paper states that `generalization performance decreases if the corruption corrupts the HFCs that the model has started to rely on'. However, in some datasets performance drop on OOD data occurs after pruning far fewer components than ID data. Isn't it possible to say that the model also relied on some part of the HFCs rather than dominant LFCs?

Response: It is certainly possible that the model relies on some part of the HFCs. LFCs are dominant, yes. That doesn’t mean they are the only source of information.

Q4: In Section 7, frequency analysis on GalaxyMNIST and EuroSAT is said to have a skewed spectral behavior towards LFCs, but pruning LFCs results in moderate drops in in-domain accuracy unlike other texture-driven datasets. This raises the question of whether low-frequency shortcuts and improvements from pruning are consistently observed across texture-driven datasets.

Response: These two datasets, particularly GalaxyMNIST, look for different levels of shape cues. Nevertheless, we agree that low-frequency shortcuts might not surface in all use cases. It is an important vulnerability of neural networks trained with gradient descent that can surface in numerous important use cases, as we have extensively shown.

Q5: Section 4 states that validation accuracies are reported in the main paper. Is there a reason why test accuracies are not used for the analysis?

Response: Yes, having both validation and test set results shows that a system designer or deep learning engineer can look for low-frequency shortcuts by using his/her dataset and deploy it in real time and expect similar behavior. Having both validation and test-set results makes our work practically more relevant, showing that, by pruning LFCs, we are not overfitting to the validation set, but safely generalizing to any unseen images.

Q6: How exactly is `accuracy contribution' measured?

Response: We fix the training set to a specific number of LFCs pruned. We then measure the validation accuracy by using validation images having the exact same number of pruned LFCs as the training set. Let’s call this value A. We then modify the validation set images by pruning the same number of LFCs and also one more specific diagonal, say, the 123rd diagonal. We then re-measure the validation accuracy by using these modified validation images. Let’s call this value B. The difference between A and B, i.e., A-B, gives the accuracy contribution of the 123rd diagonal. We repeat this for all diagonals from 1 to 447, and obtain a spectral graph, as one of those in Figure 5. One diagonal is one frequency component. We usually report diagonals from 1 to 224, as after the 224th diagonal, the contributions are usually near zero.

Thank you for your detailed response to my review. I summarize my follow-up comments below.

Revised W1

My original concern was that the paper did not sufficiently explain why low-frequency shortcuts arise in new texture-driven domains.

From the authors’ response, I understand that (1) the shortcuts are due to both the low-frequency bias of neural networks and the high-frequency dependency of texture-driven domains and (2) they distinguish their research question from those addressed by prior work on texture/shape bias.

The wording “texture-driven domains suffer from low-frequency shortcuts” could be read as attributing the phenomenon to the domains themselves. I suggest that the authors clarify in the paper that the observed shortcuts arise from the interaction described in their response.

Revised W2

The authors’ response helped me understand that investigating the value of high-frequency components (HFCs) and showing that reliance on low-frequency components (LFCs) can limit performance are distinct research questions. However, I still have concerns about novelty, and I would appreciate clarification of the claim about data reduction.

(1) Conceptual novelty of the explanation for low-frequency shortcuts

The authors cite prior work for the premise that neural networks favor LFCs:

“Neural networks are biased towards learning from low-frequency components (LFCs) (our [22; 31; 51; 64; 67; 70; 72; 81; 98; 99; 105]).”

They then explain the observed shortcuts as follows:

“So, the shortcuts are due to both the low-frequency bias of neural networks and the high-frequency dependency of texture-driven domains.”

I agree that extending the analysis to new domains is one of the paper’s strengths. The experiments also show that the models rely on LFCs in texture-driven domains and that pruning LFCs can improve performance. However, it remains unclear what new insights the paper offers into low-frequency shortcuts in texture-driven domains beyond showing how the known preference for LFCs manifests in texture-driven tasks.

For example, how does the texture-driven nature of these domains contribute to shortcuts? What does the paper add beyond applying existing theory to explain why the model’s preference for LFCs gives rise to shortcuts in these domains? Based on the evidence presented so far, the main contribution is limited to an empirical extension of prior work to new domains, and I remain uncertain about the conceptual novelty. Please clarify what new insight the paper offers into how the shortcuts arise.

(2) Distinction from prior work that uses frequency analysis to improve performance

The authors highlight that “We show that you can achieve similar and even better results by simply studying the data.” Prior work has also used frequency analysis to understand differences in model performance and to guide improvements. Section 2 describes DCT as a commonly used method, and Appendix N cites prior work that prunes frequency coefficients by diagonal. Bai et al. [1], for example, use DFT to separate the low- and high-frequency components of images, analyze the performance of ViTs and CNNs on those components, and further investigate how existing data augmentation methods affect ViTs’ use of HFCs. In their own words:

… we make a hypothesis that ViT models are less effective in capturing the high-frequency components of images than CNN models, and verify it by a frequency analysis. Inspired by this finding, we first investigate the effects of existing techniques for improving ViT models from a new frequency perspective, and find that the success of some techniques (e.g., RandAugment) can be attributed to the better usage of the high-frequency components.

Given this prior work, the claim that studying the frequency content of the data can improve performance leaves the distinction unclear. Please explain how LFC pruning in texture-driven tasks differs from prior approaches and what new empirical findings this analysis adds.

(3) Meaning of “50% less data”

The authors state, “Particularly, for Tiny-Gen-Images, we can remove 50% of the data and achieve 10% higher accuracy.” The response leaves it unclear which experiment supports these findings. Figure 5 labels the setting for TinyGenImgs as “224 LFCs pruned,” which appears to correspond to setting approximately half of the DCT coefficients in each image to zero. Section 2 (p. 3, line 123) states that the images are then transformed back into RGB for training and inference. Please identify the experiment behind this claim and specify whether “50% of the data” refers to the fraction of frequency coefficients set to zero, a reduction in the number of training images, or a measured reduction in the amount of data stored, transferred, or processed.

Reviewer_cekFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates low-frequency shortcut learning in texture-driven image classification, particularly how low-frequency shortcut learning negatively impacts ID and OOD generalisation. Using DCT-based frequency pruning and sensitivity analysis, it finds that models often rely heavily on low-frequency components. Moreover, removing these components during training and inference can improve in-distribution accuracy and robustness to distribution shifts. Experiments span several application domains, architectures, and pretrained models, with additional analyses of image resolution, transform block size, and colour space.

Critical strengths

The strengths of this paper are:

  1. Extensive and comprehensive empirical evaluation across multiple texture-driven classification tasks, model architectures, and pretrained models. Additional analyses of hyperparameters, image resolution, transform block size, and colour space strengthen the evidence beyond individual experimental configurations.

  2. The paper also proposed a method (low-frequency pruning) that can improve in-distribution accuracy and robustness to selected distribution shifts. Its diagnostic framework provides a straightforward approach to investigating frequency dependence in relatively under-studied texture domains.

Critical weaknesses

The experiments included in the paper show the benefits of low-frequency pruning, but they do not sufficiently distinguish the proposed shortcut-removal mechanism from conventional appearance normalisation. The paper does not establish the effect of different per-image preprocessing, such as colour/stain normalisation or illumination correction, on the frequency components, as well as how much of the benefit these preprocessing methods could provide. Without these control experiments, it remains unclear whether the gains reveal a distinctive failure mode in texture-driven learning or largely reflect suppression of familiar appearance variations. Given prior work on frequency shortcuts, this ambiguity weakens the central interpretation and leaves the significance of the contribution insufficiently established.

In addition, for texture domains such as histopathology, fabrics, and terrain, which are the main focus of the paper, the paper tests robustness only using synthetic corruptions such as fog, blur and elastic transformation. It does not test real differences caused by scanners, staining methods, or cameras, which could have been captured by different datasets, so it is unclear whether the improvements would hold in practice.

Also, it is intuitive that images with smaller resolutions would have more LFCs than those with larger resolutions, as the image gets smaller, more details are removed.

Questions for authors

How much of the generalisation improvement is attributable to changing the learned representation, as comapared to directly attenuating the corruption at inference time? Both corruption pipelines in Appendix J end with pruning, so their comparison does not isolate these explanations. Can the existing results distinguish these contributions, particularly for the large improvements under fog?

Section 2 defines a low-frequency shortcut through both concentrated low-frequency sensitivity and improved ID accuracy after pruning. However, pruning reduces accuracy for TextileNet with ViTs and for CLIP/CONCH. What evidence distinguishes harmful shortcut reliance from reliance on genuinely useful low-frequency information in these cases, especially for VITs? This distinction is necessary to assess the claimed generality of the phenomenon.

Comments on the paper

10 comments

Thank you for the detailed clarifications and additional normalization experiments. My main concerns are cutoff selection. I understand that each model is trained and evaluated using the same pruning cutoff; I am not requesting mismatched train/test preprocessing. The question is how one cutoff is selected among the separately trained configurations without using target-domain outcomes. Please select a cutoff using source-domain validation data alone, then report the held-out target results for that fixed configuration, including NICO++. Target-domain sweeps remain valuable diagnostic analyses, but their best points should be distinguished from performance achievable using source-only model selection.

The normalization comparisons strengthen the empirical study. However, “Best-LFC pruning” should specify the data used to select the best cutoff and whether the reported accuracy is evaluated on a separate test set. These comparisons also do not yet replace the matched-energy control. By matched energy, I mean comparing removal from different frequency regions at comparable removed squared-coefficient energy under an orthonormal DCT, using a rule fixed without target labels. If some energy levels cannot be matched across bands, reporting the feasible range would suffice.

Dear Reviewer B3w5,

Thank you for your response.

Part 1: Cutoff: We hope we understand your comment on the cutoff correctly.

We do not choose any cutoff for any of the experiments. We simply report ID or OOD numbers for validation-set images across a number of pruned LFCs. We then also report test-set images in Appendix B, Figures 11 and 13, both for ID and OOD results, and show that they follow similar trends.

So, if you would optimize the number of pruned LFCs based on the validation set images, you would get similar test set results. This is consistent with existing hyperparameter tuning studies. We do not optimize anything for the test set. The test set is always completely unseen during any of the experiments.

We are not sure what you mean by “please select a cut-off”, as none of our experiments choose a cutoff. Do you ask us to actually manually choose the best validation-accuracy-giving LFC, and report its test-set results? We can do that. This will be a single data point for each OOD corruption, rather than a sweep, though. And it will basically be the difference between Figures 4 and 11.

Regarding ”Target-domain sweeps remain valuable diagnostic analyses, but their best points should be distinguished from performance achievable using source-only model selection.”, do you refer to our test-set results? We report test set results only to show that they follow the same trends as validation set results, not to actually use them to optimize anything.

Please let us know if we are misunderstanding your comments.

NICO++’s test-set results are indeed missing, and we are working on it now.

Part 2: Normalizations: Yes, all the results in normalization are based on the validation set. Sure, we are now working on getting the same table with the test set results. We will optimize based on the validation set (get the configuration with maximum validation accuracy) and report its test set results.

Regarding the matched energy, sure, we can report matched-energy results using the rule we already have, by giving a range when it cannot be matched. Thanks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.