acceptodds
Under review as a conference paper at ICLR 2027

Texture as Signal: Spectral Adaptive Tokenization for Vision Transformers

Abstract

Vision Transformers (ViTs) struggle with quadratic computational costs, a bottleneck often reduced by adaptive tokenization. Current methods predominantly rely on spatial sparsity (using edge detection or region homogeneity) to reduce token counts. However, this spatial tokenization fails in texture-dominated regimes (such as high-resolution aerial and microscopic imagery) where high-frequency gradients are dense. In these environments, edge-driven models either over-fragment the image or aggressively merge critical details, destroying both compression and performance. To resolve this, we introduce texture-aware adaptive tokenization. Rather than relying on spatial discontinuities, we formulate token saliency through spectral statistics to capture local frequency content and redundancy. We propose a general framework that dynamically partitions tokens based on texture complexity, requiring no modifications to the underlying ViT architecture. Supported by a theoretical analysis of spectral density failure modes, this approach is extensively evaluated across SAM, Swin, and ViT architectures on diverse, texture-rich datasets. The proposed texture-driven framework achieves up to a 41.5 speedup of training wall-clock time for SAM on wild blueberry aerial imagery, while simultaneously improving convergence dice scores (from 63.1% to 71.2%). Furthermore, under severe image perturbations, the texture partitioning mechanism maintains greater than 91% Structural Intersection over Union (SIoU) under heavy Gaussian noise and greater than 84% under severe Gaussian blur, preventing the catastrophic structural collapse observed in conventional edge-based tokenization (as low as 50.90%). Ultimately, these findings establish that spectral properties (rather than spatial boundaries) are the governors of token saliency in texture-dominated regimes, enabling the efficient deployment of heavy foundation models in complex visual domains.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Token Merging, High-Resolution Image Processing, Image Segmentation, Quadtree Partitioning, Preprocessing, Computational Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_gbMDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper observes that adaptive tokenization driven by spatial discontinuities degrades on texture-dominated imagery, where dense high-frequency gradients mislead edge-based quadtree allocators. It replaces the allocation signal with texture and spectral responses, which are thresholded into a binary map that drives budgeted quadtree splitting, without changing the backbone architecture. The paper also formalizes when edge-count allocation degenerates to area-based splitting and gives stability conditions for the split sequence. Evaluation covers three texture-rich segmentation datasets, five backbones, and several adaptive-tokenization baselines, with time-to-target-Dice as the efficiency metric.

Critical strengths
  1. The evaluation is careful, with three seeds, statistical tests, matched token budgets, and honest wall-clock accounting that includes preprocessing overhead.
  2. The analysis separates what is proven from what is only observed empirically, which raises confidence in the reported results.
Critical weaknesses
  1. The experiments use only three niche datasets, namely aerial wild blueberry, tree plantation, and EM mitochondria, and none of them is a mainstream vision benchmark, which is inconsistent with the general claim suggested by the title.
  2. The algorithmic novelty is limited, as the method only swaps the allocation signal inside existing quadtree tokenizers and keeps the splitting mechanism and architecture unchanged.
  3. The paper has no method overview figure, and Figure 1 illustrates the failure of edge-based tokenization rather than the proposed method.
Questions for authors
  1. Please report how much of the main-table advantage remains when the allocation threshold is selected without access to ground-truth masks.
  2. Please test whether texture-guided tokenization hurts on non-texture imagery such as natural-image segmentation.
  3. Please add a figure illustrating the proposed pipeline.
1 comment

We thank the reviewer for the constructive feedback and recognition of the evaluation and theoretical analysis.

Experimental scope and non-texture imagery: We have added experiments on Crack-seg and Package-seg, which have lower average edge densities than the original datasets. The completed comparisons show that texture-guided tokenization can reduce accuracy and increase time to target Dice on these images at 30% token allocation. We will clarify that the benefits are specific to the evaluated texture-rich settings.

Algorithmic novelty: Our contribution is the analysis and design of the allocation signal for texture-dominated imagery. Retaining the existing splitting mechanism and backbone isolates the effect of texture guidance under matched token budgets. The theoretical analysis identifies when edge-count allocation becomes non-selective and provides conditions for split-sequence stability. We will make this scope clearer.

Method overview: We have added a pipeline figure in “Texture-Guided Budgeted Tokenization”, illustrating texture-response extraction, thresholding, and budgeted quadtree allocation, alongside a comparison with Canny-guided partitions.

Label-free performance: Only Texture-AP uses ground-truth threshold calibration; the Texture-SHF results already require no mask calibration. For Texture-AP at 30% allocation, label-free thresholds reduce mean time to target in three of six comparisons, with changes ranging from a 15.4% reduction to a 62.8% increase. We will add label-free (LF) results to Tables 1 for direct comparison. The token compression analysis takes the most time to run, and therefore wasnt able to be completed by the time the discussion period was complete but should be complete by publication if accepted.

Reviewer_PeLXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a texture-guided adaptive tokenization framework for Vision Transformers, replacing edge-based partitioning criteria with morphological and frequency-based texture descriptors. The approach uses budget-constrained quadtree partitioning to reduce token counts and is evaluated across several transformer architectures and segmentation datasets. While the method shows promising results in certain settings, the mismatch between its spectral motivation and actual empirical findings, together with inconsistencies in experimental reporting and evaluation fairness, raises substantial concerns about the validity of its central claims.

Critical strengths
  1. The paper addresses an important problem in adaptive tokenization, highlighting the limitations of edge-based partitioning in texture-dominated images.
  2. The method is evaluated across multiple transformer architectures and datasets, with some promising improvements in segmentation accuracy and training efficiency.
Critical weaknesses
  1. Although the paper emphasizes spectral adaptive tokenization, Table 3 shows that the best-performing BTH operator is a spatial-domain morphological method, while spectral PSD performs substantially worse. This raises fundamental questions about the actual novelty and validity of the central spectral motivation.
  2. Section 6.3 indicates that Texture-AP thresholds are optimized using ground-truth segmentation masks, while comparable supervision is not clearly provided to baseline methods. The paper also does not clarify which data splits are used for calibration, raising concerns about evaluation fairness and possible information leakage.
  3. Conclusion attributes a runtime of 375.83 seconds to Texture-SHF-SAM on Mitochondria, whereas Table 3 reports 1,120.31 seconds for BTH-SHF-SAM under the same stated 15% token budget. Furthermore, Table 10 reveals that texture-based tokenization substantially slows down Swin inference, challenging the claimed general efficiency benefits.
  4. In Table 7, the signed threshold difference for Tree Plantation–Wavelet should be −9 rather than +9. Appendix A.6.2 also incorrectly states that label-free Wavelet thresholding increases Mitochondria training time by 15.4%, whereas Table 8 actually shows a 15.4% reduction.
  5. The Introduction claims experiments on aerial, microscopic, telescopic, and synthetic textures, but Section 6.1 and Table 6 only report aerial and microscopy datasets. The absence of telescopic and synthetic experiments undermines the claimed comprehensiveness and generalizability.
Questions for authors

Please see the Weaknesses.

1 comment

We thank the reviewer for the constructive feedback. We address the framing, calibration protocol, runtime reporting, and experimental scope below.

Texture guidance and spectral framing: We agree that the spectral framing is too narrow. BTH’s stronger performance supports texture-guided allocation, which can use spatial morphology as well as spectral descriptors. We will revise the manuscript to emphasize texture-guided adaptive tokenization, retaining the spectral analysis to explain why edge-based guidance can become less selective in highly textured images.

Calibration and fairness: Only Texture-AP variants use ground-truth threshold calibration, performed exclusively on the training split defined by each run’s random seed, without validation or test masks. Texture-SHF and all baselines were evaluated without this calibration. We will clarify this protocol and add label-free (LF) Texture-AP results to Table 1 for direct comparison. The token compression analysis takes the most time to run, and therefore wasnt able to be completed by the time the discussion period was complete but should be complete by publication if accepted.

Runtime and Swin: after reviewing the inconsistent runtimes, we believe it is the difference between cumulative and average run times creating the significant gap in the Conclusion and Table 3. The manuscript already reports the increased Swin inference time; we will make this limitation more explicit in the efficiency claims and Conclusion.

Numerical corrections: We have corrected +9 to −9 in Table 7 and changed the stated 15.4% training-time increase to a 15.4% reduction in Appendix A.6.2.

Experimental scope: We have removed the unsupported references to telescopic and synthetic experiments and restricted the experimental and generalizability claims to the evaluated aerial and microscopy datasets.

Reviewer_bRVLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at how to save computing power when AI vision models process large images. Instead of giving more attention to edges and object boundaries, a texture-guided budgeted tokenization method was proposed. Across aerial, microscopic, and synthetic datasets with ViT, SAM, Swin, DINOv3, and RF-DETR backbones, texture-guided allocation improves accuracy-efficiency trade-offs across the board.

Critical strengths

The manuscript provided a simple, general and architecture agnostic method to handle texture dominated images, and the experimental study is of good quality.

Critical weaknesses

The texture guidance significance can not be easily pre-determined, therefore, how to balance texture guidance vs edge/shape would become a challenge.

Questions for authors

Would it be possible to provide a rigrous statement on the applicability of the texture guided tokenization to general imaging analysis? Or provide recommendations on cases where both texture and non-texture information are important, how do you actual implement this methods, such as determine the threshold etc.

1 comment

Thank you for highlighting the practical question of when to use texture guidance. Texture-guided tokenization is most promising when local appearance distinguishes the target and edge detection also responds strongly to background detail. When object contours reliably delineate the target, edge guidance may be sufficient. For images containing both cues, Section 5 now recommends comparing the partitions at the same token budget to identify which structures receive finer patches, then using early validation performance with the same backbone to guide the choice. Both methods embed the original image content, so texture-guided allocation still retains shape information within each patch.

Sections 4.1 and 6.3 also clarify threshold selection. The threshold determines which texture responses guide subdivision, while the token budget sets the target sequence length. Texture-SHF samples the threshold during training without mask calibration. Texture-AP uses a fixed threshold for each dataset and texture response, selected in the primary experiments by maximizing overlap with annotated calibration masks. Appendix A.6 evaluates a label-free alternative for Texture-AP.

Reviewer_CtqYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles the problem of efficient tokenization for ViTs. Most works in this domain fail on images with very high textured appearance (e.g. high-resolution aerial images or microscopuc images) as they tend to over-fragment or over-merge the image. The proposed method augments a quadtree tokenizer of AP and SHF from previous methods and replaces the Canny edge map with a thresholded texture response.

Critical strengths
  • Table 1 experiments include both accuracy and actual latency results
  • Table 1 covers three datasets and five backbones with a notion of variance across runs
  • Table 2 sweeps token allocation from 75% to 15% which is an interesting ablation
  • Table 8 reports a label-free threshold variant.
Critical weaknesses
  • The paper heavily discusses the negative of "edge driven" tokenizers, but a lot of efficient ViT tokenizers are not edge driven but e.g. learned. Actually the quadtree tokenizers the work is based on are themselve not edge drive. Rather they split the image into grid cells of various size and learn the adequate patch size for different image regions.
  • The potential performance improvement in Table 1 are really inconsistent across datasets and model. This suggest analyzing the Pareto curve of the results would be more insightful than a table. This also means that the "consistent improvements" claim (intro, Sec. 8) is not supported in Table 1.
  • The headline number of "41.5× speedup" mentioned in the abstract does not appear in any table.
  • The proposed method uses ground-truth calibrated threshold. Appendix A.6 also proposes label-free results but only for a small setting; It would be interesting to get a more in-depth understanding
  • The paper lacks qualitative analysis of how the proposed tokenization differs from the base quadree tokenizations
  • Minor on formatting: the template of the paper looks very crowded (very little space between sections, smaller margins)
Questions for authors
  • How does the proposed method and the baseline quadtree tokenization differ in terms of qualitative tokenization ?
  • What does the "41.5x" speedup mentioned in the abstract refer to ?
  • Which Table 1 and Table 2 numbers use ground-truth-calibrated thresholds, and how much of the gain over SHF remains with the label-free rule at other allocations?
1 comment

We thank the reviewer for the constructive feedback. We have added a qualitative analysis subsection and clarify below the reference used for the reported speedup and the scope of ground-truth calibration. additionally, the heading margins have been increased as per your preference.

  • Qualitative differences between the proposed and baseline tokenization. We have added “Qualitative Comparison of Quadtree Partitions”, with a figure comparing BTH, wavelet detail energy, and the Canny-guided SHF baseline at an identical budget of 1,228 leaves per image (30% token allocation). The comparison shows how the guidance signal changes the location of refinement while holding the token budget fixed. In the wild blueberry image, BTH concentrates responses within the targeted fescue clusters, whereas wavelet energy emphasizes a broader vegetation region and Canny detects edges throughout the surrounding vegetation. In the tree plantation image, both texture operators concentrate responses around individual crowns, whereas Canny responds densely across crowns and ground. In the microscopy image, only BTH clearly distinguishes mitochondrial profiles from the surrounding tissue; wavelet energy and Canny respond more broadly to tissue texture. The analysis also reports the number of leaves with identical bounds across all three methods: 503, 850, and 841 for the wild blueberry, tree plantation, and microscopy examples, respectively. These examples illustrate that the allocation signal can alter refinement around target structures even when a substantial proportion of the partition remains shared.
  • Interpretation of the 41.5× speedup. The reported 41.5× speedup refers specifically to SAM, using the corresponding unmodified SAM baseline as the reference. In contrast, the speedup values presented in the table use ViT-Base as the reference. The discrepancy therefore arises from different reference models. We will state the SAM reference explicitly alongside the headline value and clarify the table’s reference to make this distinction unambiguous.
  • Ground-truth calibration and label-free results. Ground-truth threshold calibration applies only to the Texture-AP variants, which are the only evaluated methods with this calibration procedure. All other methods, including the Texture-SHF variants and the baseline methods, were evaluated without ground-truth threshold calibration as they cant have the same technique leveraged for them. Consequently, the reported improvements of Texture-SHF over SHF do not depend on ground-truth calibration. We will add label-free (LF) Texture-AP results to Table 1 for direct comparison. The token compression analysis takes the most time to run, and therefore wasn't able to be completed by the time the discussion period was complete but should be complete by publication if accepted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.