acceptodds
Under review as a conference paper at ICLR 2027

Spectral Tokenization for Scalable Vision Transformers

Abstract

At , a standard ViT converts RGB patches into tokens; processing a finer image with similar encoder cost requires a different way to construct those tokens. Inspired by the blockwise discrete cosine transform (DCT) used in JPEG compression, we introduce SC-ViT, which tokenizes selected coefficients from a dense grid of image blocks instead of embedding RGB patches directly. It retains a limited spectral bandwidth while sampling the image at : with coefficients per channel and block, the retained tensor has the same number of scalars as a RGB image, and both models form image tokens. Separate luminance and chrominance projections and a finer path for DC coefficients form the token embeddings. After epochs on ImageNet-1K, SC-ViT reaches top 1 accuracy at GMACs, versus at GMACs for ViT-B/16 trained with the same recipe. A separate direct epoch run reaches . A progressive configuration yields three inference budgets from one checkpoint, reaching at GMACs. On KADID-10k, no reference quality prediction reaches Spearman correlation, versus for a spatial ViT-B/16 at . The spectral tokenizer also fits a hierarchical Swin encoder, where SC-Swin reaches on ImageNet-1K after training epochs.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision Transformers, Frequency-Domain Learning, DCT Tokenization, Spectral Representation Learning, Efficient Deep Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3u5NOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SC-ViT, a spectral tokenizer that transforms images into blockwise DCT coefficients, retains selected frequencies, and constructs tokens using separate DC and AC processing paths. The design combines coefficient normalization, spatial resampling, frequency-dependent projections, and learned weighting. It also modifies the transformer encoder with query/key normalization and GEGLU layers. Evaluation covers ImageNet classification, a Swin variant, progressive bandwidth/depth training, JPEG robustness, and image-quality assessment on KADID-10k.

Critical strengths
  1. The same-resolution comparison provides promising evidence under the evaluated training budget. At 448×448 with 196 tokens, SC-ViT achieves 77.36% versus 74.70% for RGB ViT-B/32 after 50 epochs. This comparison is more informative than comparing only with a lower-resolution RGB model. The SC-Swin result also improves over Swin-B under a common training recipe without the unequal JPEG augmentation used in the flat-ViT comparisons.

  2. The representation offers an explicit way to control spectral information at a fixed token count. Separating fine spatial DC information from grouped AC components is an interesting design direction. The authors also clearly acknowledge several limitations, including differences between experimental configurations and the distinction between retained scalar count and actual compression.

Critical weaknesses
  1. The main gains are against short-budget baselines, while the fully trained result is uncompetitive with established RGB models. The principal comparisons use 50 training epochs, so the gains could reflect favorable early optimization or regularization rather than an improved final accuracy–compute tradeoff. This concern is particularly relevant because the spectral model also changes encoder components and augmentation. After 300 epochs, SC-ViT achieves 78.44%. In comparison, non-distilled DeiT-B achieves 81.8% using ImageNet-1K alone, comparable model capacity, and 224×224 inputs; even DeiT-S achieves 79.8% with approximately 22M parameters. These are published results from the DeiT paper. Although the recipes differ, the comparison demonstrates that the reported system has not established competitive final accuracy. A matched long-training comparison is necessary to determine whether spectral tokenization provides an architectural benefit. Alternatively, a claim of faster training would require accuracy-versus-training-cost measurements.

  2. The narrow and confounded evaluation does not sufficiently justify the architectural complexity. The evidence is concentrated on ImageNet, JPEG perturbations of ImageNet, and the relatively small KADID-10k dataset. The stem introduces numerous components, but the appendix’s design studies use different configurations and do not isolate their contributions to the principal model. The JPEG comparison additionally gives spectral models JPEG augmentation while withholding it from RGB baselines. The efficiency evidence is also incomplete: the reported MAC values use different profilers with potentially different operation coverage, and no end-to-end latency or memory measurements establish a practical advantage. The progressive model’s substantial compute reduction accompanies a reduction in encoder depth, without depth-only controls demonstrating the contribution of progressive bandwidth. Given the weak fully trained classification result, stronger controlled ablations and broader validation are needed to establish the value of the added complexity.

Questions for authors
  1. Does SC-ViT outperform a strong RGB baseline after 300 epochs when encoder components, augmentation, regularization, and training budget are matched? How do the methods compare in accuracy versus cumulative training compute or wall-clock time?

  2. Under the principal experimental configuration, how much improvement comes from the elaborate spectral stem over a simple truncated-DCT projection? Which components provide reproducible gains?

  3. Can the authors provide consistent MAC accounting and end-to-end runtime measurements, including spectral preprocessing? Does progressive bandwidth training outperform fixed-bandwidth models using the same depth schedule and exit losses?

  4. Appendix G describes correlations computed separately on validation shards and distinct ranks assigned to tied values. Was this evaluator used for Table 4? If so, please report SRCC/PLCC computed over the complete validation set with standard tie handling.

3 comments

We thank the reviewer for the specific questions. The revision adds a completed long-training experiment, encoder-matched controls, direct runtime measurements, and corrected full-validation IQA results. We address the questions below; section and table references refer to the revised manuscript.

1. Long-training performance and recipe dependence.

We completed an additional 300-epoch SC-ViT run using a DeiT III-based recipe, reaching 79.84% top-1 and 92.26% top-5, 1.40 percentage points above our original 300-epoch result (78.44%). Under the original recipe, the 300-epoch encoder-matched RGB control reached 76.74%, versus 78.44% for SC-ViT (+1.70 points). Appendix B.2, Table 6 reports these as separate comparisons: the original matched control supports the tokenizer's benefit, while the additional run shows sensitivity to the training recipe. The new result remains below the published DeiT-B result (81.8%), whose training setup differs. A matched RGB run under the DeiT III recipe and an accuracy-versus-cumulative-training-cost comparison remain to be completed; we make no claim of faster training.

The result also motivates studying how SC-ViT should be trained. DeiT and DeiT III were developed for RGB-based ViTs, whereas our tokenizer reduces the spatial resolution and projection capacity of chrominance AC features. Some fine-scale colour perturbations may therefore be attenuated after tokenization, while global colour and luminance changes remain relevant. DCT- and bandwidth-aware augmentation is a promising direction, but we have not isolated the effects of individual augmentations or attributed the recipe gain to colour sensitivity.

2. JPEG-augmentation correction.

We verified the archived training configurations: JPEG augmentation was disabled in every reported 50-epoch RGB and spectral run. The contradictory markers and prose described an earlier setup and have been corrected. JPEG augmentation was used only in the original direct and progressive 300-epoch experiments, not the additional DeiT III-based run. Sections 4.1 and 4.3, Table 3, and Appendix B.1 now state the correct training conditions.

3. What the tokenizer contributes.

SC-ViT lets input raster resolution and retained spectral bandwidth be selected separately at a fixed token count (Sections 3.1–3.3). At K=16, it retains 150,528 coefficients from a 448×448 raster, equal to the scalar count of a 224×224 RGB image, while producing 196 tokens. Equal scalar counts do not imply equal information.

To test what the proposed stem adds beyond encoder changes and a simple DCT input, we completed the following four-way 50-epoch comparison under one recipe and environment:

Configuration at 448Top-1
ViT-B/3274.70%
Our encoder with RGB patches75.47%
Our encoder with a plain DCT projection75.49%
Full SC-ViT77.22%

The plain DCT projection changes accuracy by only +0.02 points relative to RGB patches in the same encoder; the proposed stem adds +1.73 points over that projection. These are successive substitutions, not an orthogonal decomposition: they support the full stem's contribution but do not identify which individual stem component carries the gain. Repeating the same SC-ViT experiment with two seeds gave 77.22% and 77.12%; the control arms remain single runs.

4. Consistent compute accounting and measured inference cost.

Consistent profiling, including both attention matrix products, gives 17.65 GMACs for SC-ViT versus 17.58 for ViT-B/16. We have therefore withdrawn the reduced-MAC claim. Appendix B.3 gives the profiling convention and distinguishes it from direct runtime measurement.

The benchmark uses a fixed-exit SC-ViT without fine DC fusion or LayerScale, distinct from the principal fine-DC accuracy model in Table 1. These timings therefore do not measure the Table 1 accuracy configuration. On one A100 at batch 64 with FP16, SDPA attention, TF32 disabled, and the same torch.compile(mode="reduce-overhead") setting for every model, the median GPU RGB-tensor-to-logits timings are (Appendix B.4, Table 8):

ModelCompiled latency per batch
SC-ViT, fixed exit, no fine DC, 44818.456 ms
ViT-B/32, 44818.793 ms
ViT-B/16, 22415.940 ms

SC-ViT differs from ViT-B/32 by 0.337 ms (about 1.8%), so we treat the two implementations as similar in latency rather than claim a robust speed advantage. SC-ViT remains 15.8% slower than ViT-B/16. The measurements include spectral preprocessing but exclude image decoding, data loading and host-to-device transfers; they do not establish an end-to-end pipeline or training-speed advantage. We used 20 warmup and 100 measured forwards with CUDA-event timing.

The measured SC-ViT implementation retains only the 16 active coefficients immediately after the DCT, avoiding unnecessary downstream processing; implementation and numerical-validation details are provided in Appendix B.4.

Separate eager diagnostics of the no-fine-DC variant measured spectral preprocessing at 7.68 ms and full-forward peak allocated memory at 1,573 MiB for SC-ViT versus 969 MiB for ViT-B/32. These are uncompiled measurements, not the memory costs of the compiled implementations. We retain them to explain implementation overhead and distinguish allocated from reserved memory. For JPEG inputs already at the required resolution and crop alignment, the forward-DCT component could be avoided by reading the stored block-DCT coefficients directly, without first reconstructing RGB pixels. Dequantization, coefficient normalization and compatible chroma handling would still be required. This JPEG-native possibility is discussed separately in Appendix B.4 and is not used to claim a measured speedup.

5. Global IQA evaluation and the spatial control.

Yes, the evaluator described in Appendix G was used for Table 4. We have corrected the evaluation to compute SRCC/PLCC over the complete validation set of 1,125 images, using standard averaged ranks for SRCC. The updated correlations are:

BackboneNR SRCCNR PLCCFR SRCCFR PLCC
ViT-B/16, 224, from scratch0.38670.38120.89150.8921
ViT-B/16, 384, from scratch0.50400.49940.89340.8944
SC-ViT, 448, K=64, from scratch0.70220.66140.90400.9018
ViT-B/16, 224, initialized0.77450.76750.91610.9154
ViT-B/32, 448, initialized0.70940.69650.87460.8697
SC-ViT, 448, K=16, initialized0.79760.77070.93560.9384

The initialized comparison includes the added ViT-B/32-at-448 control under the same IQA recipe. Section 4.4 and Table 4 have been updated; Appendices G.1 and G.2 document the global evaluation.

6. Progressive bandwidth versus depth and remaining limits.

Section 4.2, Table 2, and Appendix C now clarify that the progressive experiment does not yet isolate bandwidth scheduling from the reduced-depth schedule. This requires a fixed-K=16 control with the same exit depths, stage schedule and exit losses. We do not claim that progressive bandwidth outperforms depth-only exits on the current evidence.

Contribution and motivation.

Our motivation is to give a ViT explicit control over how a fixed input budget is allocated between raster resolution and retained spectral detail. Related choices between spatial sampling and frequency fidelity are familiar in transform coding and adaptive-resolution streaming. A standard RGB patch tokenizer does not expose a separate retained-frequency budget; SC-ViT makes that choice explicit before token embedding while keeping the encoder's token count fixed. Here the budget is retained coefficients and tokens, rather than coded bitrate (Section 3.3 and Appendix E). The controlled stem comparison above provides evidence for this learned realization of the allocation principle. We see this input control as complementary to advances in transformer backbones and training recipes.

Upon acceptance, we will release the training and profiling code, run configurations, sweep definitions and available logs in our GitHub repository.

Reviewer_UYCpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SC-ViT, which builds a fixed number of vision tokens from local DCT coefficients. The tokenizer uses nonlinear coefficient processing, separate luminance and chrominance projections, and a finer DC path. The paper studies classification, multi-budget inference, adaptation to Swin, and image quality assessment. In the main experiments with 50-epoch ImageNet-1K training, the full model with 448 input improves over ViT-B/32 at the same resolution by 2.66 points, at a similar reported cost.

Critical strengths

Clear problem and a well-targeted comparison. The paper asks how to use a denser input raster while keeping the encoder at 196 tokens. This is a clean problem formulation for the trade-off between spatial sampling and spectral bandwidth.

Substantial gains and extensions across architectures and inference budgets. In the paper's 50-epoch setting, SC-ViT improves by 2.94 points over ViT-B/16 at 224 and by 2.66 points over ViT-B/32 at 448, which is large for a fixed token budget.

Careful statement of limits. The paper acknowledges prior DCT work, states that equal scalar counts do not imply equal information, and notes that the main comparison does not isolate the tokenizer.

Critical weaknesses

W1. The added value over prior frequency-domain methods is not shown. The paper states that its contribution is not the use of DCT inputs or spectral truncation, but the learned construction of tokens (p. 2), and that this distinction "motivates comparison with prior DCT tokenizers" (p. 5). However, no such comparison is reported. In addition, Xu et al. (2020), cited only as frequency learning for convolutional models, already transform 448×448 images into a 56×56 DCT input and select channels so that the input size matches 224 RGB, which is close to the main argument of the paper. Park & Johnson (CVPR 2023), who train ViTs on JPEG DCT coefficients by changing only the patch embedding, are not cited. A controlled comparison with DCFormer or a simple DCT linear stem, under the same encoder and training, is needed to show what the proposed construction adds.

W2. The training conditions in Table 1 are not described consistently. The abstract states that ViT-B/16 is "trained with the same recipe", and p. 5 (l. 251–252) states that "Every result in Table 1 uses the common recipe". However, Table 3 marks the SC-ViT models from Table 1 as trained with JPEG augmentation, while the ViT baselines are unmarked, and p. 8 (l. 410–411) confirms that the flat SC-ViT models used this augmentation. Since JPEG augmentation can affect both clean accuracy in Table 1 and the robustness results in Table 3, the configuration of each row should be stated, and the spatial baselines should ideally be trained with the same augmentation.

W3. The value of progressive spectral budgets is not separated from depth. Progressive bandwidth and depth selection is listed as part of the contribution (p. 2), and the paper states that it "offers a different inference control from changing the patch size or exit depth alone" (p. 7). In Table 2, however, the first three stages increase K and depth together, and there is no control with fixed K and the same exits. The progressive model also omits the fine DC path, so it differs from the principal model. A depth-only early-exit model with K = 16 under the same schedule would show whether varying the spectral budget adds value.

W4. The compute comparison may not use a consistent count. The abstract reports 16.95 GMACs for SC-ViT versus 17.47 for ViT-B/16. By my count from the architecture described in Section 3, the SC-ViT encoder alone requires about 17.36 GMACs when the attention matrix products are included, which exceeds the reported total. The value for ViT-B/16 is reproduced only when these products are counted, whereas the increments in Table 2 match a count that excludes them. With a consistent count, SC-ViT would be about 1% more expensive than ViT-B/16, not 3% cheaper. All rows should be counted with the same tool and coverage.

Questions for authors

Q1. How does SC-ViT compare with DCFormer, or with a simple DCT linear stem, when both use the same encoder and training recipe? This would show how much of the gain comes from the proposed token construction.

Q2. Which rows in Table 1 were trained with JPEG augmentation? How do ViT-B/16 and ViT-B/32 perform when trained with the same augmentation?

Q3. With K fixed at 16 and the same exits and schedule, how does a depth-only early-exit model compare with the progressive model in Table 2?

Q4. Were all GMAC values in Tables 1 and 2 computed with the same tool and coverage? In particular, are the attention matrix products included for SC-ViT?

3 comments

We thank the reviewer for the careful assessment and constructive suggestions. In particular, your analysis of the computational cost was correct and helped us identify an inconsistency in our profiling. We address each concern below.

W1/Q1 — Contribution beyond prior spectral methods. We agree that the original comparison did not sufficiently isolate the contribution of our tokenizer. We have expanded the related work to acknowledge Xu et al. (2020), whose DCT representation and input-size matching closely relate to our motivation, and Park & Johnson (2023), who train ViTs on minimally decoded JPEG coefficients.

More importantly, we conducted an additional controlled comparison under the same training recipe and evaluation setup, holding our encoder fixed across the three tokenizer variants:

ModelTop-1 (%)Params (M)GMACs
ViT-B/32, 448 (timm)74.7088.3417.93
Our encoder, RGB patches75.4788.3517.84
Our encoder, linear DCT stem75.4987.7617.72
SC-ViT77.2287.4317.66

The encoder modifications improve accuracy by 0.77 points over ViT-B/32. Replacing RGB patches with a simple DCT linear stem provides only 0.02 additional points, whereas the proposed tokenizer improves accuracy by a further 1.73 points. These successive substitutions suggest that the gain comes primarily from learned token construction rather than the DCT representation alone. SC-ViT also has the lowest parameter count and reported computational cost among these configurations. The revised paper now summarizes this experiment in Section 4.1 and reports the full comparison in Appendix B.5, Table 9.

These single-run comparisons do not isolate individual tokenizer components, and we have not completed a validated DCFormer comparison. We therefore limit our claims to the evaluated settings.

W2/Q2 — JPEG augmentation consistency. The reviewer correctly identified an inconsistency between the manuscript and the reported experiments. We verified the archived training configurations and confirmed that JPEG augmentation was disabled for every reported 50-epoch model, including SC-ViT, ViT-B/16, ViT-B/32, SC-Swin, and Swin-B. The spectral models had been retrained without JPEG augmentation before submission to match the RGB baselines, but the manuscript retained outdated descriptions and Table 3 markers.

We have corrected Sections 4.1 and 4.3 and Appendix B accordingly. JPEG augmentation was used only in the original 300-epoch direct and progressive runs, not in the additional DeiT III-based experiment. Thus, the 50-epoch classification and JPEG robustness comparisons use matched augmentation, while the progressive results are explicitly distinguished.

W3/Q3 — Separating spectral bandwidth from depth. We agree that the progressive experiment does not isolate the benefit of increasing spectral bandwidth. The first three stages increase both K and encoder depth, and the progressive architecture omits the fine DC branch used in the principal model.

We have clarified this limitation in Section 4.2. The experiment demonstrates that spectral bandwidth and depth can be combined to provide multiple inference budgets from one checkpoint, with the best exit reaching 77.2% at 8.98 GMACs, approximately half the compute of direct full-depth training (78.44%). The deeper exits do not improve accuracy, potentially reflecting the allocation of training updates across stages; this experiment does not establish the cause.

W4/Q4 — Consistent computational accounting. Your analysis was correct. Our original SC-ViT counts excluded the attention matrix products, while the ViT-B/16 counts included them. This arose from mixing ptflops, which missed functional attention operations in our implementation, with fvcore. Fused attention in the baseline implementations also required explicit handling for consistent profiling.

We have now recomputed every GMAC value using fvcore, with both attention matrix products included for all models. Our analytical encoder count agrees with your estimate of approximately 17.36 GMACs, including 0.708 GMACs from the attention products.

With consistent accounting, SC-ViT requires 17.65 GMACs versus 17.58 for ViT-B/16. We therefore withdraw the reduced-compute claim and instead report the improvement from 74.42% to 77.36% at approximately equal computation (within 0.5%). The abstract, Tables 1–2, and corresponding discussion have been updated, with the profiling convention documented in Appendix B.3.

Additional revision. In response to the other reviewers, we also completed a separate 300-epoch SC-ViT run with a DeiT III-based recipe, reaching 79.84% top-1 and 92.26% top-5, a 1.40-point improvement over the original 78.44% result. Appendix B.2 reports this separately from the original encoder-matched comparison (76.74% RGB versus 78.44% SC-ViT); the additional run does not establish a matched RGB advantage under the new recipe.

We appreciate the reviewer's detailed feedback, which helped us correct important reporting issues and better distinguish the contribution of spectral token construction from the broader model design.

Thanks for the thorough rebuttal. The added controlled comparison and the unified GMAC accounting have addressed my main concerns.And the revised manuscript appropriately narrows the claims to what the current experiments support, so I maintain my rating of 3.

Reviewer_67F2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SC-ViT, which constructs visual tokens from selected blockwise DCT coefficients. Its tokenizer combines coefficient normalization, separate luminance and chrominance projections, and a finer spatial path for DC coefficients. The paper also explores a Swin extension, progressive inference budgets, JPEG robustness, and image quality assessment. The results suggest that spectral tokenization can improve recognition at a fixed token count. However, controlled comparisons are needed to identify which components produce the gains and establish practical efficiency.

Critical strengths
  1. Motivation is clear. Separating spatial resolution from retained frequency bandwidth provides an interpretable way to control information entering the transformer.

  2. Experimental results are promising. Under the common 50-epoch recipe, SC-ViT reaches 77.36% ImageNet accuracy, compared with 74.70% for ViT-B/32 at the same 448 resolution and 196-token count.

Critical weaknesses
  1. The tokenizer’s contribution remains unclear because architecture and training settings also differ. Please compare RGB and spectral tokenizers using identical encoders, augmentation, and verified stochastic-depth implementations.

  2. Direct comparisons with closely related spectral methods are missing. Please evaluate DCFormer and DCT-ViT under matched training and compute settings.

  3. The longer-training result lacks a matched baseline, and all results are single runs. Please provide a 300-epoch RGB baseline and repeated runs for the main comparisons.

  4. The progressive experiment does not isolate the benefit of increasing spectral bandwidth. Please compare with depth-only exits and independently trained models at matched budgets, particularly since deeper exits currently reduce accuracy.

  5. Practical efficiency is not established by the reported MAC estimates. Appendix B notes mixed profiler coverage. Please use consistent profiling and report end-to-end latency and memory, including spectral preprocessing.

  6. The IQA evaluation requires verification. Appendix G reports correlations computed per worker and distinct ranks assigned to tied values. Please compute correlations globally using standard tie handling, and repeat comparisons with matched resolutions and training schedules.

  7. The JPEG robustness results use unequal training-time JPEG augmentation. Please provide an augmentation-matched comparison.

Questions for authors

See Weaknesses.

2 comments

We thank the reviewer for the constructive feedback. We have corrected the evaluation and reporting issues, conducted additional controlled experiments, and clarified the limitations of the remaining comparisons.

1. Isolating the tokenizer's contribution. We conducted an additional comparison under one training recipe and evaluation setup, holding our encoder implementation fixed across the three tokenizer variants:

ModelTop-1 (%)Params (M)GMACs
ViT-B/32, 448 (timm)74.7088.3417.93
Our encoder, RGB patches75.4788.3517.84
Our encoder, linear DCT stem75.4987.7617.72
SC-ViT77.2287.4317.66

The encoder modifications improve accuracy by 0.77 points over the standard ViT-B/32. Replacing RGB patches with a simple DCT projection changes accuracy by only 0.02 points, while the proposed spectral tokenizer improves it by a further 1.73 points. These single-run substitutions suggest that the improvement comes primarily from learned token construction rather than the DCT representation alone, although they do not isolate individual tokenizer components.

We also verified that the original stochastic-depth helper inadvertently disabled branch dropping. This is now explicitly documented in the revision.

2. Comparison with prior spectral methods. We agree that matched evaluations against DCFormer and DCT-ViT would strengthen the paper. We have not completed reliable reproductions under our training protocol; in particular, we could not obtain a usable DCFormer implementation. We therefore do not claim superiority over these methods. Instead, we have expanded the related-work discussion and added the controlled linear-DCT comparison above to better identify our contribution.

3. Longer training and repeated runs. Under the original 300-epoch recipe, our encoder-matched RGB baseline reaches 76.74%, compared with 78.44% for SC-ViT (+1.70 points). We also completed a separate 300-epoch run with a DeiT III-based recipe, reaching 79.84% top-1 and 92.26% top-5, improving the original spectral result by 1.40 points. This is a separate training protocol, not an extension of the controlled RGB comparison.

DeiT III's augmentation was designed for RGB-based ViTs and may not be optimal for our retained spectral bandwidths. Exploring DCT-aware augmentation and representation-specific training remains a promising direction.

Repeating the same SC-ViT experiment with two random seeds yielded 77.22% and 77.12% top-1 accuracy. The control arms remain single runs.

4. Progressive inference. We agree that the current experiment cannot isolate bandwidth selection from depth adaptation, since the first three stages increase both. The best progressive exit reaches 77.2% at 8.98 GMACs, approximately half the compute of direct full-depth training (78.44%). Deeper exits do not improve accuracy, despite retaining the same spectral bandwidth. This may reflect the progressive training schedule, which gives earlier blocks more updates, but the experiment does not establish the cause. We have clarified that our result demonstrates multiple inference budgets from one checkpoint, not an advantage over depth-only adaptation.

5. Computational cost and practical efficiency. We corrected the mixed profiling conventions and recomputed all GMACs using fvcore, including both attention matrix products. SC-ViT now measures 17.65 GMACs versus 17.58 for ViT-B/16. We therefore withdraw the lower-compute claim and describe the 77.36% versus 74.42% comparison as approximately iso-compute.

We also measured inference on an A100 at batch 64 using FP16 and compilation for all models. With early retention of the 16 active coefficients after the unchanged DCT, this variant takes 18.456 ms, versus 18.793 ms for ViT-B/32 at 448 and 15.940 ms for ViT-B/16 at 224. These timings do not measure the Table 1 accuracy configuration. The 1.8% difference against ViT-B/32 is too small to establish a reliable speed advantage. A separate eager audit of the no-fine-DC variant also shows 1.62× peak allocated memory relative to ViT-B/32; this is not a compiled-memory measurement.

The timings include GPU spectral preprocessing but exclude decoding and host-to-device transfer. We therefore make no end-to-end speedup claim. Appendix B.4 provides implementation and numerical-validation details.

6. IQA evaluation. The reviewer correctly identified a problem in our evaluator. We have recomputed SRCC and PLCC globally over all 1,125 validation images, using standard averaged ranks for SRCC ties, and updated Table 4 accordingly.

SC-ViT retains its advantage in the evaluated settings. For example, in the initialized 60-epoch comparison, SC-ViT achieves NR/FR SRCC of 0.7976/0.9356. We also added an initialized ViT-B/32 baseline at the same 448 resolution and token count, which reaches 0.7094/0.8746 under the same IQA training schedule. The revision documents the original evaluation issue and the corrected procedure.

7. JPEG augmentation and robustness. The reviewer identified an inconsistency in our description, for which we apologize. We checked the archived configurations and confirmed that all reported 50-epoch models, both spectral and RGB, were trained with JPEG augmentation disabled. The earlier text and table markers incorrectly described a previous configuration and have been corrected.

JPEG augmentation was enabled only in the original 300-epoch direct and progressive runs, not in the additional DeiT III experiment. Consequently, the 50-epoch JPEG robustness comparisons use matched augmentation, while the progressive exits are explicitly identified as a separate, unmatched setting.

We appreciate these comments, which helped us identify reporting errors and clarify what the experiments establish. The revised manuscript includes the corrected evaluation protocols, computational accounting, additional long-training results, and a more explicit discussion of the remaining limitations.

Comments on the paper

2 comments

Overall rebuttal summary

We thank the reviewers for their constructive feedback. We have revised the manuscript through additional controlled experiments, corrected evaluation and computational accounting, and clearer statements of what the results establish. The main changes are summarized below.

1. Controlled tokenizer comparison. We added a four-model, 50-epoch comparison under one training recipe and evaluation setup. ViT-B/32 reaches 74.70%, our encoder with RGB patches 75.47%, our encoder with a linear DCT stem 75.49%, and SC-ViT 77.22%. Holding our encoder fixed, the proposed tokenizer improves accuracy by 1.73 points over the linear DCT stem, supporting learned token construction beyond the use of DCT inputs alone. Two seeds of the same SC-ViT experiment achieved 77.22% and 77.12%. The paper now includes the controlled comparison and distinguishes it from the original Table 1 results.

2. JPEG augmentation correction. We verified the archived training configurations and confirmed that JPEG augmentation was disabled in every reported 50-epoch RGB and spectral run. The contradictory text and table markers described an earlier configuration and have been corrected. JPEG augmentation was enabled only in the original 300-epoch direct and progressive runs, not in the additional DeiT III-based experiment. Thus, the 50-epoch classification and JPEG robustness comparisons use matched augmentation; the progressive robustness results are explicitly distinguished.

3. Additional long-training results. Under the original 300-epoch protocol, the encoder-matched RGB control reaches 76.74%, compared with 78.44% for SC-ViT, a 1.70-point improvement. A separate 300-epoch experiment using a DeiT III-based recipe reaches 79.84% top-1 and 92.26% top-5, improving the original spectral result by 1.40 points. We keep this experiment separate because it uses a different training protocol. The result motivates further study of DCT- and bandwidth-aware augmentation, without attributing the improvement to any individual augmentation.

4. Corrected computational accounting. We recomputed GMACs consistently using fvcore, including both attention matrix products for all models. The principal SC-ViT configuration requires 17.65 GMACs, versus 17.58 for ViT-B/16. We have withdrawn the reduced-compute claim and report the accuracy comparison at approximately equal computation. The corresponding tables, figures and discussion have been corrected, and the profiling convention is documented.

5. Measured inference cost. On an A100 at batch 64 with FP16 and the same compilation setting for all models, the optimized fixed-exit SC-ViT takes 18.456 ms per batch, versus 18.793 ms for ViT-B/32 at 448 and 15.940 ms for ViT-B/16 at 224. The benchmarked spectral variant omits fine DC fusion and LayerScale, so these timings do not measure the principal Table 1 accuracy configuration. The small difference against ViT-B/32 does not establish a reliable speed advantage. Separate eager diagnostics measured preprocessing overhead and a 1.62× peak allocated-memory ratio against ViT-B/32; these are uncompiled measurements. Timings include GPU spectral preprocessing but exclude decoding, data loading and host-to-device transfer, so we make no end-to-end pipeline or training-speed claim.

6. Corrected global IQA evaluation. We recomputed SRCC and PLCC over all 1,125 validation images, using standard averaged ranks for SRCC ties, and updated the reported results. In the initialized 60-epoch comparison, SC-ViT achieves 0.7976 NR SRCC and 0.9356 FR SRCC. We also added the ViT-B/32-at-448 control under the same IQA training schedule, obtaining 0.7094/0.8746 NR/FR SRCC. The revised paper documents the corrected evaluation procedure.

7. Clearer scope and limitations. We expanded the discussion of related spectral methods, documented the original stochastic-depth implementation issue, and clarified that the progressive experiment demonstrates joint bandwidth and depth control rather than isolating bandwidth scheduling. The controlled substitutions do not isolate individual tokenizer components, and we do not claim superiority over spectral methods without matched evaluations.

The motivation remains to give a transformer explicit control over raster resolution and retained spectral detail at a fixed token count. Such allocation choices are familiar in transform coding and adaptive-resolution streaming. SC-ViT exposes a retained-frequency budget before token embedding, complementing advances in transformer backbones and training recipes; the budget studied here is retained coefficients and tokens, rather than coded bitrate.

Area_Chair_jTY5OpenReview ↗

Please read the rebuttal and engage in discussion

Dear Reviewers,

The authors have posted their rebuttal. Please read it and engage in discussion, in particular:

  • Whether each of your questions and concerns has been addressed.
  • Whether your rating and recommendation have changed, and why.
  • Any remaining concerns that must be settled before the meta-review.

Thanks for your time and contribution.

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.