Spectral Tokenization for Scalable Vision Transformers

At $224\times 224$, a standard ViT converts RGB patches into tokens; processing a finer image with similar encoder cost requires a different way to construct those tokens. Inspired by the blockwise discrete cosine transform (DCT) used in JPEG compression, we introduce SC-ViT, which tokenizes selected coefficients from a dense grid of image blocks instead of embedding RGB patches directly. It retains a limited spectral bandwidth while sampling the image at $448\times 448$: with $K=16$ coefficients per channel and block, the retained tensor has the same number of scalars as a $224\times 224$ RGB image, and both models form $196$ image tokens. Separate luminance and chrominance projections and a finer path for DC coefficients form the token embeddings. After $50$ epochs on ImageNet-1K, SC-ViT reaches $77.36\%$ top 1 accuracy at $16.95$ GMACs, versus $74.42\%$ at $17.47$ GMACs for ViT-B/16 trained with the same recipe. A separate direct $300$ epoch run reaches $78.44\%$. A progressive configuration yields three inference budgets from one checkpoint, reaching $77.2\%$ at $8.62$ GMACs. On KADID-10k, no reference quality prediction reaches $0.7064$ Spearman correlation, versus $0.5353$ for a spatial ViT-B/16 at $384\times 384$. The spectral tokenizer also fits a hierarchical Swin encoder, where SC-Swin reaches $79.44\%$ on ImageNet-1K after $50$ training epochs.