- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SC-ViT, a spectral tokenizer that transforms images into blockwise DCT coefficients, retains selected frequencies, and constructs tokens using separate DC and AC processing paths. The design combines coefficient normalization, spatial resampling, frequency-dependent projections, and learned weighting. It also modifies the transformer encoder with query/key normalization and GEGLU layers. Evaluation covers ImageNet classification, a Swin variant, progressive bandwidth/depth training, JPEG robustness, and image-quality assessment on KADID-10k.
Critical strengths
-
The same-resolution comparison provides promising evidence under the evaluated training budget. At 448×448 with 196 tokens, SC-ViT achieves 77.36% versus 74.70% for RGB ViT-B/32 after 50 epochs. This comparison is more informative than comparing only with a lower-resolution RGB model. The SC-Swin result also improves over Swin-B under a common training recipe without the unequal JPEG augmentation used in the flat-ViT comparisons.
-
The representation offers an explicit way to control spectral information at a fixed token count. Separating fine spatial DC information from grouped AC components is an interesting design direction. The authors also clearly acknowledge several limitations, including differences between experimental configurations and the distinction between retained scalar count and actual compression.
Critical weaknesses
-
The main gains are against short-budget baselines, while the fully trained result is uncompetitive with established RGB models. The principal comparisons use 50 training epochs, so the gains could reflect favorable early optimization or regularization rather than an improved final accuracy–compute tradeoff. This concern is particularly relevant because the spectral model also changes encoder components and augmentation. After 300 epochs, SC-ViT achieves 78.44%. In comparison, non-distilled DeiT-B achieves 81.8% using ImageNet-1K alone, comparable model capacity, and 224×224 inputs; even DeiT-S achieves 79.8% with approximately 22M parameters. These are published results from the DeiT paper. Although the recipes differ, the comparison demonstrates that the reported system has not established competitive final accuracy. A matched long-training comparison is necessary to determine whether spectral tokenization provides an architectural benefit. Alternatively, a claim of faster training would require accuracy-versus-training-cost measurements.
-
The narrow and confounded evaluation does not sufficiently justify the architectural complexity. The evidence is concentrated on ImageNet, JPEG perturbations of ImageNet, and the relatively small KADID-10k dataset. The stem introduces numerous components, but the appendix’s design studies use different configurations and do not isolate their contributions to the principal model. The JPEG comparison additionally gives spectral models JPEG augmentation while withholding it from RGB baselines. The efficiency evidence is also incomplete: the reported MAC values use different profilers with potentially different operation coverage, and no end-to-end latency or memory measurements establish a practical advantage. The progressive model’s substantial compute reduction accompanies a reduction in encoder depth, without depth-only controls demonstrating the contribution of progressive bandwidth. Given the weak fully trained classification result, stronger controlled ablations and broader validation are needed to establish the value of the added complexity.
Questions for authors
-
Does SC-ViT outperform a strong RGB baseline after 300 epochs when encoder components, augmentation, regularization, and training budget are matched? How do the methods compare in accuracy versus cumulative training compute or wall-clock time?
-
Under the principal experimental configuration, how much improvement comes from the elaborate spectral stem over a simple truncated-DCT projection? Which components provide reproducible gains?
-
Can the authors provide consistent MAC accounting and end-to-end runtime measurements, including spectral preprocessing? Does progressive bandwidth training outperform fixed-bandwidth models using the same depth schedule and exit losses?
-
Appendix G describes correlations computed separately on validation shards and distinct ranks assigned to tied values. Was this evaluator used for Table 4? If so, please report SRCC/PLCC computed over the complete validation set with standard tie handling.
3 comments
4. Consistent compute accounting and measured inference cost.
Consistent profiling, including both attention matrix products, gives 17.65 GMACs for SC-ViT versus 17.58 for ViT-B/16. We have therefore withdrawn the reduced-MAC claim. Appendix B.3 gives the profiling convention and distinguishes it from direct runtime measurement.
The benchmark uses a fixed-exit SC-ViT without fine DC fusion or LayerScale, distinct from the principal fine-DC accuracy model in Table 1. These timings therefore do not measure the Table 1 accuracy configuration. On one A100 at batch 64 with FP16, SDPA attention, TF32 disabled, and the same torch.compile(mode="reduce-overhead") setting for every model, the median GPU RGB-tensor-to-logits timings are (Appendix B.4, Table 8):
| Model | Compiled latency per batch |
|---|---|
| SC-ViT, fixed exit, no fine DC, 448 | 18.456 ms |
| ViT-B/32, 448 | 18.793 ms |
| ViT-B/16, 224 | 15.940 ms |
SC-ViT differs from ViT-B/32 by 0.337 ms (about 1.8%), so we treat the two implementations as similar in latency rather than claim a robust speed advantage. SC-ViT remains 15.8% slower than ViT-B/16. The measurements include spectral preprocessing but exclude image decoding, data loading and host-to-device transfers; they do not establish an end-to-end pipeline or training-speed advantage. We used 20 warmup and 100 measured forwards with CUDA-event timing.
The measured SC-ViT implementation retains only the 16 active coefficients immediately after the DCT, avoiding unnecessary downstream processing; implementation and numerical-validation details are provided in Appendix B.4.
Separate eager diagnostics of the no-fine-DC variant measured spectral preprocessing at 7.68 ms and full-forward peak allocated memory at 1,573 MiB for SC-ViT versus 969 MiB for ViT-B/32. These are uncompiled measurements, not the memory costs of the compiled implementations. We retain them to explain implementation overhead and distinguish allocated from reserved memory. For JPEG inputs already at the required resolution and crop alignment, the forward-DCT component could be avoided by reading the stored block-DCT coefficients directly, without first reconstructing RGB pixels. Dequantization, coefficient normalization and compatible chroma handling would still be required. This JPEG-native possibility is discussed separately in Appendix B.4 and is not used to claim a measured speedup.
5. Global IQA evaluation and the spatial control.
Yes, the evaluator described in Appendix G was used for Table 4. We have corrected the evaluation to compute SRCC/PLCC over the complete validation set of 1,125 images, using standard averaged ranks for SRCC. The updated correlations are:
| Backbone | NR SRCC | NR PLCC | FR SRCC | FR PLCC |
|---|---|---|---|---|
| ViT-B/16, 224, from scratch | 0.3867 | 0.3812 | 0.8915 | 0.8921 |
| ViT-B/16, 384, from scratch | 0.5040 | 0.4994 | 0.8934 | 0.8944 |
| SC-ViT, 448, K=64, from scratch | 0.7022 | 0.6614 | 0.9040 | 0.9018 |
| ViT-B/16, 224, initialized | 0.7745 | 0.7675 | 0.9161 | 0.9154 |
| ViT-B/32, 448, initialized | 0.7094 | 0.6965 | 0.8746 | 0.8697 |
| SC-ViT, 448, K=16, initialized | 0.7976 | 0.7707 | 0.9356 | 0.9384 |
The initialized comparison includes the added ViT-B/32-at-448 control under the same IQA recipe. Section 4.4 and Table 4 have been updated; Appendices G.1 and G.2 document the global evaluation.
6. Progressive bandwidth versus depth and remaining limits.
Section 4.2, Table 2, and Appendix C now clarify that the progressive experiment does not yet isolate bandwidth scheduling from the reduced-depth schedule. This requires a fixed-K=16 control with the same exit depths, stage schedule and exit losses. We do not claim that progressive bandwidth outperforms depth-only exits on the current evidence.
Contribution and motivation.
Our motivation is to give a ViT explicit control over how a fixed input budget is allocated between raster resolution and retained spectral detail. Related choices between spatial sampling and frequency fidelity are familiar in transform coding and adaptive-resolution streaming. A standard RGB patch tokenizer does not expose a separate retained-frequency budget; SC-ViT makes that choice explicit before token embedding while keeping the encoder's token count fixed. Here the budget is retained coefficients and tokens, rather than coded bitrate (Section 3.3 and Appendix E). The controlled stem comparison above provides evidence for this learned realization of the allocation principle. We see this input control as complementary to advances in transformer backbones and training recipes.
Upon acceptance, we will release the training and profiling code, run configurations, sweep definitions and available logs in our GitHub repository.
We thank the reviewer for the specific questions. The revision adds a completed long-training experiment, encoder-matched controls, direct runtime measurements, and corrected full-validation IQA results. We address the questions below; section and table references refer to the revised manuscript.
1. Long-training performance and recipe dependence.
We completed an additional 300-epoch SC-ViT run using a DeiT III-based recipe, reaching 79.84% top-1 and 92.26% top-5, 1.40 percentage points above our original 300-epoch result (78.44%). Under the original recipe, the 300-epoch encoder-matched RGB control reached 76.74%, versus 78.44% for SC-ViT (+1.70 points). Appendix B.2, Table 6 reports these as separate comparisons: the original matched control supports the tokenizer's benefit, while the additional run shows sensitivity to the training recipe. The new result remains below the published DeiT-B result (81.8%), whose training setup differs. A matched RGB run under the DeiT III recipe and an accuracy-versus-cumulative-training-cost comparison remain to be completed; we make no claim of faster training.
The result also motivates studying how SC-ViT should be trained. DeiT and DeiT III were developed for RGB-based ViTs, whereas our tokenizer reduces the spatial resolution and projection capacity of chrominance AC features. Some fine-scale colour perturbations may therefore be attenuated after tokenization, while global colour and luminance changes remain relevant. DCT- and bandwidth-aware augmentation is a promising direction, but we have not isolated the effects of individual augmentations or attributed the recipe gain to colour sensitivity.
2. JPEG-augmentation correction.
We verified the archived training configurations: JPEG augmentation was disabled in every reported 50-epoch RGB and spectral run. The contradictory markers and prose described an earlier setup and have been corrected. JPEG augmentation was used only in the original direct and progressive 300-epoch experiments, not the additional DeiT III-based run. Sections 4.1 and 4.3, Table 3, and Appendix B.1 now state the correct training conditions.
3. What the tokenizer contributes.
SC-ViT lets input raster resolution and retained spectral bandwidth be selected separately at a fixed token count (Sections 3.1–3.3). At K=16, it retains 150,528 coefficients from a 448×448 raster, equal to the scalar count of a 224×224 RGB image, while producing 196 tokens. Equal scalar counts do not imply equal information.
To test what the proposed stem adds beyond encoder changes and a simple DCT input, we completed the following four-way 50-epoch comparison under one recipe and environment:
The plain DCT projection changes accuracy by only +0.02 points relative to RGB patches in the same encoder; the proposed stem adds +1.73 points over that projection. These are successive substitutions, not an orthogonal decomposition: they support the full stem's contribution but do not identify which individual stem component carries the gain. Repeating the same SC-ViT experiment with two seeds gave 77.22% and 77.12%; the control arms remain single runs.