acceptodds
Withdrawn from ICLR 2027

MergeNet: Spatial Routing and Execution Trade-offs in Learned Token Compression

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision transformers preserve fine image detail but pay to process every patch through global blocks. Learned token merging must decide where information flows while producing a shorter sequence for later computation. We introduce MergeNet, a local-to-latent transformer that combines differentiable spatial mass transport with an exact-budget gather in one forward pass. Routing redistributes features and mass on the original patch grid; selected carriers then read full-grid features through cross-attention before latent encoding. Mass-aware latent attention retains the transported token mass, and an exact query/key augmentation implements its bias in an unmodified FlashAttention call. On ImageNet-1K with DeiT-S/8, MergeNet reaches 81.360% top-1 with 392 final patches and 82.618% after 512-pixel fine- tuning. Its 224-pixel configuration halves the latent patch sequence and reduces estimated multiply–accumulate operations by 37.0% relative to dense DeiT-S/8. In a separate synthetic A100 benchmark, compiled inference takes 27.24 ms at batch 64, compared with 27.22 ms for compiled ToMe. Evaluations across token budgets and image resolutions characterize the accuracy and implementation cost of this trainable physical bottleneck.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
token merging, learned token compression, vision transformers, spatial routing, efficient deep learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pMEWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

MergeNet combines six full-grid local transformer blocks with learned spatial token transport, an exact-budget carrier selection, full-grid cross-attention recovery, and six latent transformer blocks. Routing redistributes features and token mass within spatial neighborhoods. A hard gather physically reduces the sequence, while a straight-through weighting supplies a gradient path to continuous selection scores. Latent attention incorporates token mass through a query/key augmentation compatible with FlashAttention. The paper evaluates ImageNet-1K classification across token budgets and resolutions and reports synthetic latency and memory benchmarks. At 224 pixels, MergeNet achieves 81.36% accuracy with 392 retained patches, compared with 81.93% for post-training ToMe at the same final patch count.

Critical strengths
  1. The physical bottleneck is concretely implemented during training and inference. The supplied code gathers exactly the selected carriers before recovery and latent processing. It also implements the stated straight-through multiplier while leaving index selection discrete. This makes the proposed execution model concrete, rather than merely masking a full-length sequence.

  2. The paper reports actual runtime and memory measurements, including unfavorable outcomes. The distinction between arithmetic savings and execution cost is valuable. The query/key augmentation for the log-mass bias is algebraically correct, and the implementation retains the original attention scale while padding the augmented dimensions. At 384 pixels, the reported H20 benchmark demonstrates a genuine operating-point tradeoff: lower training and inference latency than dense DeiT-S/8, accompanied by higher memory use.

Critical weaknesses
  1. The reported results do not establish a competitive accuracy–efficiency advantage. At the default 224-pixel budget, MergeNet achieves 81.360%, versus 81.932% for ToMe applied to the trained dense checkpoint—a deficit of 0.572 percentage points. ToMe remains more accurate at every budget in Table 5, with the deficit increasing to 1.524 points at 196 patches. The runtime measurements do not provide a compensating advantage at the principal operating point. Compiled batch-64 latency is effectively identical: 27.24 ms for MergeNet versus 27.22 ms for ToMe. MergeNet is slower at batch 1 and has slightly higher peak allocated memory at batch 64. In eager execution, MergeNet is also slower than dense DeiT-S/8 despite its lower estimated MAC count. The high-resolution speedup over dense DeiT is encouraging, but the corresponding benchmark omits ToMe and PiToMe, and MergeNet uses substantially more memory. A compiled dense baseline is also absent from the A100 comparison. Consequently, the paper has not established where the proposed trainable method offers a preferable tradeoff over available alternatives.

  2. The experiments do not isolate the value of learned spatial routing or establish sufficient additional contribution. MergeNet simultaneously changes the early attention pattern, introduces spatial transport and hard selection, adds full-grid recovery, uses mass-aware attention, and introduces a training-only auxiliary readout. Yet the paper provides no controlled component ablations. Budget and resolution sweeps characterize the complete system but do not show whether learned transport is necessary. Particularly important controls are the same local-to-latent architecture with simple pooling or non-learned selection, removal of recovery cross-attention, and removal of the straight-through selection mechanism or auxiliary readout. The supplied code already exposes a no-recovery variant, but no corresponding result is reported. Since recovery reads all pre-transport features, it could compensate for weak carrier selection; its contribution needs to be separated from routing.

Questions for authors
  1. At what matched measured latency or compute budget does MergeNet outperform ToMe or PiToMe? Please include a compiled dense baseline and high-resolution comparisons with the token-merging baselines.

  2. How does learned spatial transport compare with simple pooling, fixed selection, or non-learned merging inside the same local-to-latent architecture, retaining the same recovery and training recipe?

  3. What are the separate contributions of recovery cross-attention, mass bias, straight-through selection, and the training-only auxiliary readout? Does routing still provide a meaningful improvement without recovery?

  4. Since the motivation emphasizes physical compression during training, how does total training time and peak memory compare with end-to-end ToMe or DTEM at comparable final accuracy?

Reviewer_3RFXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MergeNet, a Vision Transformer architecture that introduces a physical token bottleneck by connecting differentiable spatial mass routing with an exact-budget token gathering mechanism. The method performs local feature encoding, learns spatial information transport through soft routing, selects a fixed number of carrier tokens based on transported mass, and applies latent transformer blocks on the compressed token sequence. The paper further introduces mass-aware attention and a FlashAttention-compatible implementation of the mass bias. Experiments on ImageNet-1K with DeiT-S/8 evaluate accuracy under different token budgets and resolutions, as well as computational cost and latency.

Critical strengths
  1. The paper identifies an interesting problem gap between differentiable token reduction and physically shorter token sequences during training. The proposed physical bottleneck provides a coherent architectural perspective for integrating token routing and compression.
  2. The implementation is carefully developed and the evaluation includes token-budget analysis, resolution scaling, and efficiency measurements.
Critical weaknesses
  1. The empirical evidence does not demonstrate a clear advantage over existing token merging methods. In the main ImageNet experiments, MergeNet achieves lower accuracy than ToMe and PiToMe under the same token budget, despite requiring a substantially more complex architecture and training procedure. The paper therefore does not convincingly show why the proposed physical token bottleneck is preferable to existing token reduction approaches.
  2. The paper introduces multiple components, including spatial mass routing, differentiable token selection, cross-attention recovery, and mass-aware attention. However, the paper lacks sufficient ablation studies to isolate the contribution of each component. It remains unclear whether the observed performance is mainly due to the central physical bottleneck idea or the combination of several engineering modules.
  3. The efficiency claims are not fully convincing. Although the paper reports reduced MACs, the actual end-to-end latency improvement is limited. For example, the reported compiled inference latency is almost identical to ToMe, suggesting that the additional local encoding, routing, and recovery operations introduce substantial overhead. The practical acceleration benefit therefore needs further justification.
  4. The experimental validation is relatively narrow for an ICLR submission. The evaluation is mainly limited to ImageNet classification with a single backbone (DeiT-S/8). More diverse tasks, architectures, or transfer settings would be needed to demonstrate that the proposed bottleneck is a generally useful principle rather than a task-specific architecture design.
  5. The paper's presentation, especially the abstract, does not sufficiently highlight the central scientific message. The abstract contains many numerical performance results (accuracy, FLOPs reduction, latency) but lacks a clear logical explanation of the motivation, key insight, and why the proposed physical bottleneck matters. The current presentation gives a benchmark-report impression rather than emphasizing the conceptual contribution of the work.
Questions for authors
  1. Can the authors provide more comprehensive ablation studies to quantify the individual contributions of spatial routing, mass-aware attention, cross-attention recovery, and differentiable selection?
  2. Since ToMe achieves higher accuracy with comparable runtime, what specific advantages does MergeNet provide over existing token merging approaches beyond enabling training-time compression?
  3. Can the authors clarify whether the proposed physical bottleneck is expected to provide benefits on tasks beyond ImageNet classification, such as dense prediction or transfer learning?
Reviewer_ai32OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MergeNet, a bottleneck method that combines local encoding, differentiable spatial mass transport, fixed-budget carrier selection, and full-grid cross-attention before latent processing. A query/key augmentation implements the log-mass attention bias through FlashAttention. On ImageNet-1K, it reports 81.360% accuracy with 392 patches at 224 resolution and 82.618% after higher-resolution fine-tuning. Separate synthetic benchmarks characterize execution time and memory.

Critical strengths
  1. The architecture combines trainable soft routing with a physically shorter downstream sequence within one forward pass, while clearly acknowledging that gather indices remain discrete.
  2. The query/key augmentation exactly represents the per-key log-mass bias while retaining the original attention scale.
  3. Budget sweeps, resolution experiments, and measured latency and memory provide useful evidence beyond arithmetic-cost estimates.
Critical weaknesses
  1. The cited DTEM [1] and LookupViT [2] already provide soft mass-weighted merging and compressed-token access to full-grid features, respectively. The combination here is distinct, but its practical advantage over these closely related designs is not established.
  2. FlashBias [3] is not discussed. It already implements low-rank attention biases by augmenting Q/K while preserving the original scale; the proposed per-key log-mass construction is a rank-one special case, limiting the novelty of this implementation.
  3. No component ablation isolates the benefit of learned spatial routing from the surrounding local-to-latent architecture and recovery mechanism.
  4. At 224 pixels and 392 final patches, MergeNet reports lower accuracy than ToMe (81.360% versus 81.932%). The separate synthetic A100 benchmark shows essentially equal compiled batch-64 latency (27.24 versus 27.22 ms), so these results do not establish a better accuracy–latency trade-off at this operating point.

References

[1] Lee and Hong. Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers. 2024. Paper.
[2] Koner et al. LookupViT: Compressing Visual Information to a Limited Number of Tokens. 2024. Paper.
[3] Wu et al. FlashBias: Fast Computation of Attention with Bias. NeurIPS 2025. Paper.

Questions for authors
  1. Are there existing controlled results isolating learned routing while retaining the same carrier budget, recovery module, and training setup?
  2. What are the per-layer token counts and estimated MACs for the matched-final-count baselines?
  3. Is a compiled dense DeiT measurement available under the same A100 protocol?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.