- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper identifies and names "high-frequency (HF) collapse": the phenomenon by which learned spectral gating in spectral-mixer Vision Transformers (e.g., SpectFormer) progressively attenuates high-frequency image content as a side effect of the gate's parameterization, which is harmful specifically on tasks where fine-scale texture/edge information is discriminative. The proposed fix, HF-Skip, is a minimal architectural change: it splits the spectral mixer's frequency domain at a threshold τ, continues to apply the learned multiplicative gate K only to low-frequency (LF) coefficients, and routes high-frequency (HF) coefficients around the gate through an additive identity pathway scaled by a single learnable scalar α. The paper evaluates this change against (a) controlled band-filtered datasets that isolate which frequency band is actually responsible for an observed accuracy gain, (b) parameter-matched architectural controls (full-spectrum bypass, random-bin bypass, inverted mask, generic spatial residual) that test whether the frequency-specific framing is actually doing the work, (c) standard classification benchmarks (DTD, ImageNet-100, Imagenette, full ImageNet-1K), (d) a cross-architecture transfer to GFNet, (e) a frozen-backbone segmentation probe, and (f) a corruption-robustness analysis.
Headline results: HF-Skip improves SpectFormer by +8.83 points on DTD, +1.27 on ImageNet-100, +3.44 on Imagenette, transfers to GFNet (+2.46 on ImageNet-100), and adds 0.009% FLOPs. On full ImageNet-1K at 300 epochs, however, SpectFormer-S and HF-SpectFormer-S are statistically tied (79.674 vs. 79.678 top-1, McNemar p = 0.987), which the authors use to explicitly delimit the method's benefit to HF-dependent regimes rather than claim a universal gain.
Critical strengths
The input parameter matching experiment is straightforward and trustworthy. Table 3 has applied the HF skip with four control groups. These control groups have exactly excluded the debate that it is not just some more identical paths whereever you protect the path, it will be effective. Especially the reverse mask hyperparameter just protected the error frequency band with little gain, which verified that the frequency band selection itself is making a change rather than the extra path.
It has honestly reported an important zero result. The authors compared ImageNet 1000 using 3 epochs, which showed that the specformer-S and the HF version have the same performance. It is good to report this in the abstract and conclusion and further constrain the approach to be effective only on fine-grained tasks, which strengthened the claim of the paper.
Critical weaknesses
The robustness cost is not revealed sufficiently in the abstract. The proposed high-frequency skip protects the high-frequency information but also simultaneously makes the model more fragile under high-frequency intervention. As reviewed in Table 5, in the DTD dataset, the corruption has a negative impact on the model over baseline. This trade-off will make the identical path indifferently protect high-frequency structure and the injected noise.
The threshold of the routing will be dependent on the dataset and cannot be learned end-to-end. Figure 4 has reported a different optimized threshold on different datasets, where the default threshold value is not optimal for both datasets. More importantly, the author wants it to be differentiable and learn the threshold in Section 6. However, it cannot reliably find the optimal value out of the box, which means that we need extra validation scanning and introduce extra cost.
Questions for authors
-
Given that the robustness cost associated with HF-corruption is so significant (a 19.7-point degradation on DTD), could this trade-off be explicitly mentioned in the abstract rather than being relegated to Section 6.4? This is crucial information for readers deciding whether to deploy HF-Skip in a production environment.
-
The optimal value of τ varies widely between DTD (0.2) and ImageNet-100 (0.7). Is this variation linked to factors such as image resolution, the ratio of patch size to texture scale, or the dataset's overall frequency-domain energy distribution (given that the paper already calculates HF-content quintiles in Appendix G)? Could the τ sweep results be integrated with the existing HF-content stratification analysis to provide a heuristic for selecting τ based on dataset statistics, rather than requiring readers to perform a new sweep each time?
Dear Reviewer EPq5, thank you again for your constructive review. As the discussion period closes soon, we wanted to check whether our responses address your concerns. The revision states the robustness trade-off explicitly with absolute accuracies (§6.4, Table 5) and adds the spectral-content heuristic and practical recipe for choosing τ (§6.3, Appendix E). If these address your concerns, we would be grateful if you would consider updating your assessment; we are happy to clarify anything further.