- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Differentiable Virtual-Slot Partitioning (DVSP), a continuous representation for learning adaptive coarse-to-fine image tokenization. Each parent region is represented by four virtual slots during training. A learned resolution coordinate jointly controls their content, positional embeddings, and attention masses. At the coarse endpoint, the four slots become identical copies whose attention masses sum to that of one native coarse token; at the fine endpoint, they recover four native fine tokens.
The main theoretical result establishes that, under compatible deterministic transformer operations and output readouts, every binary mixture of coarse and fine regions produces the same output in the virtual-slot representation as in native variable-length execution. An image-conditioned router learns continuous resolution coordinates through task gradients and an L1 penalty. At inference, a fixed half-threshold projects the coordinates to a binary partition, which is then executed natively.
On ImageNet, DVSP reduces visual-token counts by 21.7% for ViT-Tiny and 30.4% for ViT-Small, with respective Top-1 accuracy decreases of 0.18 and 0.53 percentage points relative to the original dense classifiers. Matched-count spatial controls show that learned partitions outperform entropy-based selection within the adapted task models. Additional analyses examine projection effects, coordinate distributions, and the relationship between continuous gradients and measured native discrete-operation utility. Supplementary experiments extend spatial-selection evaluation to segmentation and detection, with explicitly different execution protocols.
Critical strengths
-
DVSP couples content, position, and attention mass so that every binary partition has a well-defined native execution with the same output under the stated compatibility assumptions. The contribution is not merely a differentiable routing score: it is a representation-level construction that preserves the semantics of both coarse and fine choices. The accompanying proof and independent-tokenizer numerical checks make this a convincing technical contribution.
-
The paper distinguishes adaptation, continuous representation, binary projection, and native execution instead of conflating them. The matched per-image token-count comparisons isolate spatial membership within fixed task weights, and the native single-region flip probes test gradients against measured discrete outcomes. Although these probes do not replace complete training baselines, they provide substantive evidence that the learned coordinates carry useful task information.
Critical weaknesses
-
Have you trained an otherwise identical model using binary forward routing and identity-STE backward gradients, with matched initialization, trainable modules, training budget, and comparable final token counts? If so, how do its native accuracy–token trade-offs compare with DVSP?
-
What are the latency, throughput, and peak-memory changes relative to the dense model when including MobileNet routing, mixed-patch embedding, partition construction, and variable-length execution? Please distinguish single-image latency from batched throughput and specify how variable sequence lengths are batched.
-
What fractions of learned coordinates are below zero, above one, or in the attention-mass-floor regime? Does constraining coordinates to [0, 1] substantially change performance at matched token counts?
-
Can the main ViT-Small evaluation include an independently adapted entropy baseline and a strong feature-based selector? Given the supplementary result in which the Quadformer feature scorer matches DVSP in accuracy, how should the scope of the claimed allocation advantage be interpreted?
-
Have you numerically compared the detector’s binary full-slot outputs with its compact native outputs under rotary positions? For segmentation, which decoder or spatial reconstruction operations must be modified or verified before partition-implied token reductions can become actual compact execution?
-
How many independent runs contribute to the ViT-Tiny means, and what is their variability? Can you report additional regularization settings to show whether the observed accuracy–token trade-off is stable rather than specific to the selected L1 coefficient?
Questions for authors
See the weakness.