acceptodds
Under review as a conference paper at ICLR 2027

On High-Frequency Collapse in Spectral Vision Transformers

10 views

Abstract

Modern vision architectures often emphasize low-frequency representations as depth increases. In spectral token mixers, this tendency can suppress high-frequency (HF) information—edges, textures, and fine local structure—that is discriminative for detail-dependent recognition. We study this architecture-dependent failure mode, which we call high-frequency collapse, and introduce HF-Skip: a frequency-selective pathway that applies the learned spectral gate to low-frequency coefficients while routing the upper-frequency band around it. Controlled band-filtered datasets isolate frequency content from dataset size, labels, and image identity. On ImageNet-100 at split radius r = 0.12, HF-Skip improves the HF-only arm by +12.21 ± 0.25 points but the matched LF-only arm by only +0.67 ± 0.70. Parameter-matched controls on DTD further show that HF-Skip gains +8.90 points, compared with +6.78 for a full-spectrum bypass, +4.15 for a random-bin bypass, +0.53 for an inverted frequency mask, and +0.00 for a generic spatial residual. On standard benchmarks, HF-Skip improves SpectFormer by +1.27 points on ImageNet-100 and +8.83 on DTD, generalizes to GFNet, and adds only 0.009% FLOPs. Full ImageNet-1K accuracy is tied (79.674 vs. 79.678), delimiting the benefit to regimes where fine detail matters rather than claiming a universal gain. Together, these results identify HF collapse as a task-dependent bottleneck and show that frequency-selective routing preserves useful fine detail with negligible computational overhead.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
spectral vision transformers, frequency-domain learning, high-frequency representations, architectural inductive bias, Fourier neural networks, fine-grained visual recognition, texture recognition, image classification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EPq5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper identifies and names "high-frequency (HF) collapse": the phenomenon by which learned spectral gating in spectral-mixer Vision Transformers (e.g., SpectFormer) progressively attenuates high-frequency image content as a side effect of the gate's parameterization, which is harmful specifically on tasks where fine-scale texture/edge information is discriminative. The proposed fix, HF-Skip, is a minimal architectural change: it splits the spectral mixer's frequency domain at a threshold τ, continues to apply the learned multiplicative gate K only to low-frequency (LF) coefficients, and routes high-frequency (HF) coefficients around the gate through an additive identity pathway scaled by a single learnable scalar α. The paper evaluates this change against (a) controlled band-filtered datasets that isolate which frequency band is actually responsible for an observed accuracy gain, (b) parameter-matched architectural controls (full-spectrum bypass, random-bin bypass, inverted mask, generic spatial residual) that test whether the frequency-specific framing is actually doing the work, (c) standard classification benchmarks (DTD, ImageNet-100, Imagenette, full ImageNet-1K), (d) a cross-architecture transfer to GFNet, (e) a frozen-backbone segmentation probe, and (f) a corruption-robustness analysis.

Headline results: HF-Skip improves SpectFormer by +8.83 points on DTD, +1.27 on ImageNet-100, +3.44 on Imagenette, transfers to GFNet (+2.46 on ImageNet-100), and adds 0.009% FLOPs. On full ImageNet-1K at 300 epochs, however, SpectFormer-S and HF-SpectFormer-S are statistically tied (79.674 vs. 79.678 top-1, McNemar p = 0.987), which the authors use to explicitly delimit the method's benefit to HF-dependent regimes rather than claim a universal gain.

Critical strengths

The input parameter matching experiment is straightforward and trustworthy. Table 3 has applied the HF skip with four control groups. These control groups have exactly excluded the debate that it is not just some more identical paths whereever you protect the path, it will be effective. Especially the reverse mask hyperparameter just protected the error frequency band with little gain, which verified that the frequency band selection itself is making a change rather than the extra path.

It has honestly reported an important zero result. The authors compared ImageNet 1000 using 3 epochs, which showed that the specformer-S and the HF version have the same performance. It is good to report this in the abstract and conclusion and further constrain the approach to be effective only on fine-grained tasks, which strengthened the claim of the paper.

Critical weaknesses

The robustness cost is not revealed sufficiently in the abstract. The proposed high-frequency skip protects the high-frequency information but also simultaneously makes the model more fragile under high-frequency intervention. As reviewed in Table 5, in the DTD dataset, the corruption has a negative impact on the model over baseline. This trade-off will make the identical path indifferently protect high-frequency structure and the injected noise.

The threshold of the routing will be dependent on the dataset and cannot be learned end-to-end. Figure 4 has reported a different optimized threshold on different datasets, where the default threshold value is not optimal for both datasets. More importantly, the author wants it to be differentiable and learn the threshold in Section 6. However, it cannot reliably find the optimal value out of the box, which means that we need extra validation scanning and introduce extra cost.

Questions for authors
  1. Given that the robustness cost associated with HF-corruption is so significant (a 19.7-point degradation on DTD), could this trade-off be explicitly mentioned in the abstract rather than being relegated to Section 6.4? This is crucial information for readers deciding whether to deploy HF-Skip in a production environment.

  2. The optimal value of τ varies widely between DTD (0.2) and ImageNet-100 (0.7). Is this variation linked to factors such as image resolution, the ratio of patch size to texture scale, or the dataset's overall frequency-domain energy distribution (given that the paper already calculates HF-content quintiles in Appendix G)? Could the τ sweep results be integrated with the existing HF-content stratification analysis to provide a heuristic for selecting τ based on dataset statistics, rather than requiring readers to perform a new sweep each time?

1 comment

Dear Reviewer EPq5, thank you again for your constructive review. As the discussion period closes soon, we wanted to check whether our responses address your concerns. The revision states the robustness trade-off explicitly with absolute accuracies (§6.4, Table 5) and adds the spectral-content heuristic and practical recipe for choosing τ (§6.3, Appendix E). If these address your concerns, we would be grateful if you would consider updating your assessment; we are happy to clarify anything further.

Reviewer_FA1bOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates high-frequency information loss in spectral vision models and proposes HF-Skip, which routes upper-frequency features around a learned spectral gate. Its main contributions are: 1) examining a task-dependent spectral bottleneck through feature statistics and matched high-/low-frequency input experiments; 2) introducing a lightweight frequency-selective bypass and evaluating it against routing and initialization controls; and 3) demonstrating gains in detail-sensitive settings and transfer to GFNet, while reporting no aggregate improvement on ImageNet-1K and a tradeoff under high-frequency corruption.

Critical strengths
  • The matched HF-only and LF-only experiments preserve image identities, labels, and sample counts, helping separate frequency dependence from differences between datasets. Full-spectrum, random-bin, inverted-mask, spatial-residual, and initialization controls further examine plausible alternative explanations for the gains. Together, these experiments provide more informative evidence than benchmark improvements or spectral statistics alone.
  • HF-Skip makes a small change to the spectral mixer, with negligible reported computational overhead, and shows improvements on DTD and ImageNet-100 as well as transfer to GFNet. The paper also reports the ImageNet-1K null result, threshold dependence, and sensitivity to high-frequency corruption, helping identify the conditions under which the intervention may be useful.
Critical weaknesses
  • The matched band-filtered experiments and routing controls provide meaningful evidence that the benefit depends on frequency content. However, the connection between preserving high-frequency energy and preserving task-useful information remains under-specified. The reported HF fraction can increase through changes in either HF or LF energy, and bypassing a complex gate also changes phase processing, signal scale, and the effective degrees of freedom of the HF pathway. The full-spectrum bypass already provides substantial gains. Could the authors clarify which evidence specifically links the additional benefit of selective routing to useful information retained at the gate, beyond the descriptive energy statistics and general benefits of an identity route?
  • The advantage of a persistent bypass over a better-conditioned gate, and the scope of the resulting insight, need more precise characterization. Identity initialization already closes much of the reported gap, leaving a 0.30-point advantage on ImageNet-100, but Tables 4/11 do not clearly report paired-run uncertainty for this comparison. The GFNet results are useful, although several transfer and scale comparisons are single-run. Moreover, SpectFormer and GFNet both use closely related per-frequency multiplicative filters. These experiments support transfer within that operator family, but provide limited guidance on which gate properties or task conditions predict a benefit. Clarifying this boundary would strengthen the generalizable insight without requiring coverage of every vision architecture.
Questions for authors
  • Several configuration and metric ambiguities limit the auditability of the supporting experiments. Table 7 reports 79.86% for HF-SpectFormer on ImageNet-100 with 100 epochs and seed 42, whereas Table 15 reports 81.48% for the hard-mask default under the same stated dataset, schedule, and seed. The Gaussian-noise results in Tables 21 and 24 are also difficult to reconcile without additional protocol details. Finally, Table 5 does not define its degradation metric sufficiently: the DTD LF-family value of 36.2 cannot be interpreted as an absolute accuracy drop from the reported clean accuracy of approximately 21.44%. These may be reporting or protocol differences, but they should be resolved before the corresponding ablation and robustness conclusions can be assessed confidently.
  • Could the authors supplement the HF-fraction measurements with absolute band energies and a task-relevant probe of representations before and after the learned gate? This would help distinguish more HF energy from more predictive information. For Table 2, please also clarify the ordering of band filtering, mean/variance restoration, cropping, and CutMix/RandomErasing, and how the intended frequency separation is preserved at the actual model input.
3 comments

Dear Reviewer FA1b, thank you again for your careful audit of the tables. As the discussion period closes soon, we wanted to check whether our responses address your concerns. The revision corrects the issues you identified (original Tables 7 and 24), defines the Table 5 metric with absolute accuracies, adds absolute band energies and three-seed matched runs (Appendix L), states the band-filter protocol (Appendix P), and makes the scope explicit (Conclusion). If these address your concerns, we would be grateful if you would consider updating your assessment; we are happy to clarify anything further.

Thank you for the detailed rebuttal and the revised manuscript. The added equations clarify the distinction between the block-level gated shortcut and the full-spectrum bypass. The configuration corrections, removal of the non-comparable noise table, definition of the degradation metric, and matched identity-initialization runs address my main concerns. One mechanistic point remains. Relative to HF-Skip, the full-spectrum bypass both adds an LF identity route and retains the learned HF-gated component. Therefore, its lower performance does not isolate the effect of adding the LF route alone.

Dear Reviewer FA1b,

Thank you for the follow-up, and for confirming that the revision addresses your main concerns.

You are right: relative to HF-Skip, the full-spectrum bypass changes two things at once. It adds an LF identity route, and it keeps the learned gate on the HF bins. The HF-Skip vs. full-bypass comparison alone therefore does not isolate the LF route. To separate the two, we trained the two missing cells of the 2×2 factorial on DTD, with the same recipe and the same five seeds (42, 123, 7, 1, 2024) as the routing controls in the paper:

  • HF-Skip + LF route only: Y = IFFT(K⊙M_LF⊙X̂ + σ(α)·M_HF⊙X̂ + σ(α)·M_LF⊙X̂)
  • HF-Skip + HF gate only: Y = IFFT(K⊙M_LF⊙X̂ + σ(α)·M_HF⊙X̂ + K⊙M_HF⊙X̂)

At the operator level, (full bypass − HF-Skip) = (LF-route-only − HF-Skip) + (HF-gate-only − HF-Skip) exactly. All four models have the same parameter count (19,661,619).

DTD test top-1, 5 seedsLF binsHF binsMean ± stdPaired Δ vs HF-Skip [95% CI]Seeds below HF-Skip
HF-Skipgateroute30.04 ± 0.46——
+ LF route onlygate + routeroute27.64 ± 0.61−2.40 [−3.22, −1.59]5/5
+ HF gate onlygategate + route30.14 ± 0.86+0.10 [−1.15, +1.34]2/5
Full bypassgate + routegate + route27.92 ± 0.69−2.13 [−3.19, −1.07]5/5

The two effects separate clearly. Adding the LF route alone costs 2.40 points and is lower in all five seeds. Keeping the learned HF gate is neutral (+0.10, CI includes zero). The interaction is small (+0.18, 95% CI [−1.72, +2.09]). The full bypass's deficit therefore comes from the LF identity route, not from the retained HF-gated component.

This also bears on your earlier point about phase processing and degrees of freedom. The HF-gate-only model keeps the gate's phase processing and parameters on the HF bins and still matches HF-Skip. The benefit comes from the HF route itself, and the same route on the LF bins is harmful.

Alongside these runs, we re-ran HF-Skip and the full bypass for seed 42. Both reproduce the original results exactly (29.68 and 26.60), so the new runs are paired with the existing ones. (The exact full-bypass gap is −2.13; the "2.12" in the text was the difference of the rounded means.) We will add this factorial to Appendix O and make the §6.1 wording precise: the 2.13-point cost of the full bypass is due to the LF route.

Reviewer_CECLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HF-Skip adds an identity bypass for high-frequency coefficients around learned spectral gates. The paper studies whether this alleviates high-frequency attenuation in spectral vision Transformers, using standard and band-filtered datasets, parameter-matched controls and initialization ablations. It reports gains in several from-scratch settings, negligible extra computation, a robustness trade-off and no gain on ImageNet-1K.

Critical strengths
  1. The intervention is minimal and essentially cost-free, making it easy to adopt if its advantage over simpler residual or initialization changes is established. Multi-seed main comparisons and parameter-matched controls strengthen the empirical design.

  2. The scope is presented candidly: the experiments include a null ImageNet-1K result, initialization controls, unsuccessful threshold learning and a corruption-robustness trade-off. The band-filtered datasets also offer a controlled way to probe frequency dependence.

Critical weaknesses
  1. The key residual control is not yet interpretable. At zero cutoff, the full-spectrum bypass appears equivalent to adding a scaled spatial residual around the gate, but Table 3 reports gains of +6.78 versus +0.00 for the generic residual. The insertion point, scaling, normalization and initialization must be specified to explain why these controls differ. Since the full bypass explains much of the reported DTD gain, this issue directly affects attribution to frequency selectivity.

  2. The evidence for a learned high-frequency collapse remains incomplete. Identity initialization closes much of the performance gap, and an untrained-activation analysis partly reflects the chosen initial scales. Converged-model spectral analysis with matched initialization is needed to distinguish persistent frequency suppression from an initialization problem. The strong from-scratch DTD gain and null ImageNet-1K result also suggest a limited regime of benefit rather than a broadly established mechanism. I currently lean toward weak rejection, but a convincing resolution of the residual control could change my assessment.

Questions for authors
  1. Please give the exact equations and insertion points for the full-spectrum bypass and generic spatial residual, including scaling, normalization and initialization. Under which conditions are they equivalent, and what explains their very different Table 3 results?

  2. Do converged baseline models still exhibit the proposed collapse with identity initialization? Can spectral measurements and matched-initialization comparisons isolate the remaining benefit of selective high-frequency routing?

  3. Which data selected the threshold in the robustness experiments? Please separate prespecified-cutoff results from tuned results, and report uncertainty without unstable gain ratios. Does the mechanism persist when fine-tuning a pretrained model rather than training from scratch on a small dataset?

2 comments

Dear Reviewer CECL, thank you again for your detailed review. As the discussion period closes soon, we wanted to check whether our responses address your concerns. Your review notes that you lean toward weak rejection and that a convincing resolution of the residual control could change your assessment. The revision now gives exact equations for every control (Appendix O) and shows that the "generic spatial residual" is a block-level gated shortcut, while a spatial residual around the gate is identical to the full-spectrum bypass, which HF-Skip outperforms by +2.12 points (95% CI [+1.07, +3.19], 5/5 seeds; §6.1). On the regime of benefit, we agree and now state the scope explicitly: the benefit is expected when labels depend on HF content, which the matched band-filtered test diagnoses directly (Conclusion). We would be grateful if you would reconsider your score in light of these; we are happy to clarify anything further.

AuthorsOpenReview ↗

ImageNet-100 routing control: third seed

Dear Reviewer CECL, a short update on your residual-control question (Q1). Table 3 reports the ImageNet-100 routing controls over two seeds, with HF-Skip ahead of the full-spectrum bypass by +0.43. The third seed has now finished for these two arms with the Table 3 recipe (150 epochs, pre-specified τ = 0.5, not the tuned value), and both arms use the same data order and augmentation within each seed. Seeds 42 and 123 are the runs already in Table 3.

SeedHF-SkipFull-spectrum bypassHF-Skip − bypass
4283.6083.24+0.36
12384.1483.64+0.50
783.9283.54+0.38
Mean ± population std83.89 ± 0.2283.47 ± 0.17+0.41 ± 0.06

HF-Skip is ahead in all three paired seeds, and the paired gap varies much less across seeds than either arm does. As in Appendix L, with three seeds we report this as a consistent sign rather than a CI-backed effect size. The ordering of the selective route over the non-selective full bypass therefore holds on both datasets, not only on DTD. We will add the third seed to Table 3 and §6.1.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised manuscript uploaded

We thank the reviewers for their thoughtful and constructive feedback, which has helped us strengthen the paper. We have uploaded a revised manuscript that incorporates the clarifications, corrections, and additional analyses from our responses. All new and revised text is highlighted in blue.

Key improvements:

  • Sharper routing evidence (§6.1, Table 3, new Appendix O): exact equations for every control, a clearer name for the block-level gated shortcut, and an explicit routing decomposition: every route that includes HF helps, LF-only routing does not, and HF-Skip beats the full-spectrum bypass (+2.12, 95% CI [+1.07, +3.19]).
  • Stronger statistics for the band-filtered test (Table 2, new Appendix P): seed-paired Δ ± std, paired HF − LF differences in place of point-estimate ratios, and the full data-processing protocol.
  • More complete robustness reporting (§6.4, Table 5): a precise metric definition, absolute accuracies alongside relative degradation, and clear labels for default vs. tuned thresholds.
  • New multi-seed evidence (§6.2, Appendix L): three-seed matched DTD runs in which HF-Skip outperforms identity initialization in every seed on both SpectFormer and GFNet, plus absolute band energies.
  • Practical guidance on τ (§6.3, Appendix E): a spectral-content heuristic, a safe default, and a three-point validation recipe.
  • Clearer scope (Conclusion): the operator and task conditions under which HF-Skip applies.
  • Improved consistency (Appendix A, J): the original Tables 7 and 24 were removed, with a short note in each place explaining why. All other results are unchanged.

Table numbers and appendix letters A–N are preserved, so all references in our responses still apply. We are happy to address any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.