acceptodds
Withdrawn from ICLR 2027

Exact Native Vertices for Continuous Adaptive Tokenization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Choosing where to use fine patches is central to adaptive image tokenization, but each choice changes the length and structure of the token sequence. We introduce Differentiable Virtual-Slot Partitioning (DVSP), which places coarse and fine representations in a shared, fixed-length computation graph. Each image region is represented by four virtual slots whose content, position, and attention mass vary together with a learned resolution coordinate. At the binary endpoints, these slots reproduce either one coarse token or four fine tokens. We prove that every resulting binary partition exactly matches its native variable-length execution under compatible deterministic transformer operations. This allows resolution to be learned through task gradients, with projection evaluated separately from exact execution. On ImageNet, DVSP reduces visual-token counts by 30.4% on ViT-Small with a Top-1 accuracy decrease of 0.53 percentage points. Controlled spatial comparisons show that learned partitions outperform entropy selection at identical per-image token counts and fixed task weights. DVSP makes adaptive input resolution learnable within a continuous representation whose discrete states retain their native meaning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Adaptive Image Tokenization, Vision Transformers, Continuous Relaxation, Differentiable Partitioning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eVU1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Differentiable Virtual-Slot Partitioning (DVSP), a continuous representation for learning adaptive coarse-to-fine image tokenization. Each parent region is represented by four virtual slots during training. A learned resolution coordinate jointly controls their content, positional embeddings, and attention masses. At the coarse endpoint, the four slots become identical copies whose attention masses sum to that of one native coarse token; at the fine endpoint, they recover four native fine tokens.

The main theoretical result establishes that, under compatible deterministic transformer operations and output readouts, every binary mixture of coarse and fine regions produces the same output in the virtual-slot representation as in native variable-length execution. An image-conditioned router learns continuous resolution coordinates through task gradients and an L1 penalty. At inference, a fixed half-threshold projects the coordinates to a binary partition, which is then executed natively.

On ImageNet, DVSP reduces visual-token counts by 21.7% for ViT-Tiny and 30.4% for ViT-Small, with respective Top-1 accuracy decreases of 0.18 and 0.53 percentage points relative to the original dense classifiers. Matched-count spatial controls show that learned partitions outperform entropy-based selection within the adapted task models. Additional analyses examine projection effects, coordinate distributions, and the relationship between continuous gradients and measured native discrete-operation utility. Supplementary experiments extend spatial-selection evaluation to segmentation and detection, with explicitly different execution protocols.

Critical strengths
  1. DVSP couples content, position, and attention mass so that every binary partition has a well-defined native execution with the same output under the stated compatibility assumptions. The contribution is not merely a differentiable routing score: it is a representation-level construction that preserves the semantics of both coarse and fine choices. The accompanying proof and independent-tokenizer numerical checks make this a convincing technical contribution.

  2. The paper distinguishes adaptation, continuous representation, binary projection, and native execution instead of conflating them. The matched per-image token-count comparisons isolate spatial membership within fixed task weights, and the native single-region flip probes test gradients against measured discrete outcomes. Although these probes do not replace complete training baselines, they provide substantive evidence that the learned coordinates carry useful task information.

Critical weaknesses
  1. Have you trained an otherwise identical model using binary forward routing and identity-STE backward gradients, with matched initialization, trainable modules, training budget, and comparable final token counts? If so, how do its native accuracy–token trade-offs compare with DVSP?

  2. What are the latency, throughput, and peak-memory changes relative to the dense model when including MobileNet routing, mixed-patch embedding, partition construction, and variable-length execution? Please distinguish single-image latency from batched throughput and specify how variable sequence lengths are batched.

  3. What fractions of learned coordinates are below zero, above one, or in the attention-mass-floor regime? Does constraining coordinates to [0, 1] substantially change performance at matched token counts?

  4. Can the main ViT-Small evaluation include an independently adapted entropy baseline and a strong feature-based selector? Given the supplementary result in which the Quadformer feature scorer matches DVSP in accuracy, how should the scope of the claimed allocation advantage be interpreted?

  5. Have you numerically compared the detector’s binary full-slot outputs with its compact native outputs under rotary positions? For segmentation, which decoder or spatial reconstruction operations must be modified or verified before partition-implied token reductions can become actual compact execution?

  6. How many independent runs contribute to the ViT-Tiny means, and what is their variability? Can you report additional regularization settings to show whether the observed accuracy–token trade-off is stable rather than specific to the selected L1 coefficient?

Questions for authors

See the weakness.

Reviewer_tXrfOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes differentiable virtual-slot partitioning (DVSP), which learns directly from the task loss whether each region in adaptive image tokenization should remain a coarse patch or be split into fine patches. During training, each 32×32 region is always represented by 4 virtual slots, and a scalar α\alpha jointly interpolates the content, position, and attention mass of these slots. With α\alpha, DVSP determines whether four slots are exactly equivalent to one coarse token, or to four fine tokens, following ToMe. Authors also show that for every binary partition, the output of the fixed length training model matches that of the actual variable length execution. α\alpha is predicted by a small router based on frozen auxiliary model features.

Critical strengths
  1. The motivation to replace APT's hand-crafted entropy rule and MSViT's STE approximation with continuous learning driven by the task loss is clear.
  2. The idea of continuously connecting coarse and fine tokens by jointly interpolating content, position, and attention mass with a single coordinate is simple.
  3. The property that the training model and the actual variable length execution match exactly for every binary partition narrows the deployment error, which makes the analysis clear. The authors also provide numerical verification against an actual implementation.
  4. The structure of the manuscript are clear, making the paper easy to read.
Critical weaknesses
  1. Although the effort to address the weaknesses of recent work is commendable, the paper does not demonstrate a clear improvement that would warrant acceptance. Only ViT-Tiny and ViT-S are evaluated, and the evidence that task gradients help discrete decisions is weak. These line of adaptive tokenizer works usually report actual throughput gains on ViT-B/L/H.
  2. The paper reports only token counts, not compute time or FLOPs. It also does not address how variable length sequences are batched or the cost of running the router. Since training always processes 196 slots, the training cost is presumably not reduced.
  3. While the theoretical contribution of linking learning to task gradients through the router is notable, the resulting conclusion appears to be the same as the proportional attention of prior work, ToMe.
  4. Only two levels (32×32 / 16×16) are allowed, and each region can have only one or four tokens, which limits expressiveness. The representational range is limited compared to that of the recent works, and this should be addressed.
  5. Many figures show "illustrative DVSP successes" but tables not. The opposite cases should also be presented.
Questions for authors
  1. Does DVSP still show a clear improvement on larger models (ViT-B/L/H) and when compared with existing methods (APT, MSViT, ToMe, etc.) at the same token count? Can the authors show in broader settings that task gradients help discrete decisions?
  2. What are the actual compute clock time and FLOPs, including the router cost and the batching of variable length sequences? How does the training cost compare with dense training?
  3. What does the exactness theorem show beyond the proportional attention property of ToMe?
  4. Can DVSP be extended to more than two patch sizes or to content-adaptive regions? Does the exactness theorem still hold in that case?
Reviewer_E8dfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DVSP learns coarse/fine image partitions through a fixed-length representation containing four virtual slots per parent region. A continuous coordinate jointly interpolates content, position, and attention mass; thresholding then selects a native variable-length partition. The paper proves binary-slot/native output equivalence under compatible deterministic transformer operations and evaluates projection separately. On ImageNet, ViT-S reduces average visual tokens from 196 to 136.45 with Top-1 changing from 81.39% to 80.86%, while matched-count spatial controls and local gradient probes assess the learned allocation.

Critical strengths
  1. The endpoint correspondence is precise, useful, and supported by a convincing proof: Eq. (5) and Appendix A.1 correctly use mass sharing to preserve attention contributions of duplicate coarse slots, with explicit positional/readout conditions; Appendix B additionally checks against APT’s independent tokenizer, reporting maximum FP32 logit discrepancies of 2.93×10⁻⁵ and 8.30×10⁻⁵.
  2. The spatial-selection experiments control important confounders: Table 4 fixes task weights and each image’s token count, improving Top-1 over entropy by 1.03 percentage points on both backbones; Section 4.5 supplies paired image-bootstrap intervals excluding zero.
  3. The paper carefully separates representation exactness from empirical projection behavior: Table 2 distinguishes adaptation, continuous inference, binary slots, and native execution, while Appendix B reports both harmful and beneficial per-image projection changes rather than claiming a loss-preserving projection.
Critical weaknesses
  1. The benefit of the continuous training construction is not established against a trained task-aware routing alternative. Table 1’s adapted comparator is entropy-based; Section 4.6 evaluates identity-STE gradients only inside models trained continuously. This informative probe cannot establish how an STE-trained router would perform, and Appendix C’s feature scorer is statistically tied with DVSP in frozen-task classification (Table 5). The closely related learned approaches—MSViT, ICCV Workshops 2023, DART, and Differentiable Hierarchical Visual Tokenization, NeurIPS 2025—are cited in Section 2 but not quantitatively compared. At minimum, train the same router and task modules with hard-forward/identity-STE routing at matched achieved token budgets. A soft-masked explicit coarse/fine construction would also clarify the value of this particular interpolation.
  2. Token reduction has not been translated into a practical efficiency frontier. Table 1 demonstrates fewer tokens, but Sections 4.1–4.2 and Appendix B provide no end-to-end latency, throughput, memory, or FLOPs including MobileNet routing and variable-length batching. Main results also use one final regularization weight, λ = 0.3, rather than a budget sweep. Report several accuracy–budget points and complete inference costs. This limits the practical assessment; it does not contradict the narrower token-count claim.
  3. Training variability is insufficiently documented. Table 1 calls Tiny results “means” without specifying run count, seeds, or dispersion. Appendix B explicitly conditions bootstrap intervals on trained checkpoints, so these intervals do not measure training variability. Report the constituent runs and repeated-training uncertainty, particularly for the small Top-1 differences in Tables 1–2.
Questions for authors

See the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.