- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MergeNet combines six full-grid local transformer blocks with learned spatial token transport, an exact-budget carrier selection, full-grid cross-attention recovery, and six latent transformer blocks. Routing redistributes features and token mass within spatial neighborhoods. A hard gather physically reduces the sequence, while a straight-through weighting supplies a gradient path to continuous selection scores. Latent attention incorporates token mass through a query/key augmentation compatible with FlashAttention. The paper evaluates ImageNet-1K classification across token budgets and resolutions and reports synthetic latency and memory benchmarks. At 224 pixels, MergeNet achieves 81.36% accuracy with 392 retained patches, compared with 81.93% for post-training ToMe at the same final patch count.
Critical strengths
-
The physical bottleneck is concretely implemented during training and inference. The supplied code gathers exactly the selected carriers before recovery and latent processing. It also implements the stated straight-through multiplier while leaving index selection discrete. This makes the proposed execution model concrete, rather than merely masking a full-length sequence.
-
The paper reports actual runtime and memory measurements, including unfavorable outcomes. The distinction between arithmetic savings and execution cost is valuable. The query/key augmentation for the log-mass bias is algebraically correct, and the implementation retains the original attention scale while padding the augmented dimensions. At 384 pixels, the reported H20 benchmark demonstrates a genuine operating-point tradeoff: lower training and inference latency than dense DeiT-S/8, accompanied by higher memory use.
Critical weaknesses
-
The reported results do not establish a competitive accuracy–efficiency advantage. At the default 224-pixel budget, MergeNet achieves 81.360%, versus 81.932% for ToMe applied to the trained dense checkpoint—a deficit of 0.572 percentage points. ToMe remains more accurate at every budget in Table 5, with the deficit increasing to 1.524 points at 196 patches. The runtime measurements do not provide a compensating advantage at the principal operating point. Compiled batch-64 latency is effectively identical: 27.24 ms for MergeNet versus 27.22 ms for ToMe. MergeNet is slower at batch 1 and has slightly higher peak allocated memory at batch 64. In eager execution, MergeNet is also slower than dense DeiT-S/8 despite its lower estimated MAC count. The high-resolution speedup over dense DeiT is encouraging, but the corresponding benchmark omits ToMe and PiToMe, and MergeNet uses substantially more memory. A compiled dense baseline is also absent from the A100 comparison. Consequently, the paper has not established where the proposed trainable method offers a preferable tradeoff over available alternatives.
-
The experiments do not isolate the value of learned spatial routing or establish sufficient additional contribution. MergeNet simultaneously changes the early attention pattern, introduces spatial transport and hard selection, adds full-grid recovery, uses mass-aware attention, and introduces a training-only auxiliary readout. Yet the paper provides no controlled component ablations. Budget and resolution sweeps characterize the complete system but do not show whether learned transport is necessary. Particularly important controls are the same local-to-latent architecture with simple pooling or non-learned selection, removal of recovery cross-attention, and removal of the straight-through selection mechanism or auxiliary readout. The supplied code already exposes a no-recovery variant, but no corresponding result is reported. Since recovery reads all pre-transport features, it could compensate for weak carrier selection; its contribution needs to be separated from routing.
Questions for authors
-
At what matched measured latency or compute budget does MergeNet outperform ToMe or PiToMe? Please include a compiled dense baseline and high-resolution comparisons with the token-merging baselines.
-
How does learned spatial transport compare with simple pooling, fixed selection, or non-learned merging inside the same local-to-latent architecture, retaining the same recovery and training recipe?
-
What are the separate contributions of recovery cross-attention, mass bias, straight-through selection, and the training-only auxiliary readout? Does routing still provide a meaningful improvement without recovery?
-
Since the motivation emphasizes physical compression during training, how does total training time and peak memory compare with end-to-end ToMe or DTEM at comparable final accuracy?