- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Monotone Adaptive Tokenization (MAT), an efficient approach to hierarchical adaptive tokenization that replaces sequential Top-1 refinement with one-shot global Top-K selection. The key insight is that when importance scores are monotone along ancestor–descendant paths, the hierarchy satisfies the heap-order property, allowing global Top-K selection to exactly reproduce the token allocation of sequential greedy refinement. For non-monotone importance metrics, the authors introduce bottom-up sum or max reductions to construct monotone scores, enabling the same selection strategy under the adapted scores.
The method is evaluated across high-resolution 2D image segmentation, large-scale 3D point-cloud segmentation, and weather downscaling, with additional ImageNet experiments examining score adaptation. The results demonstrate improved end-to-end training and inference efficiency while maintaining comparable downstream task quality, including up to 3.73× training and 2.56× inference speedups over sequential refinement.
Critical strengths
-
A clear and effective algorithmic insight. The paper presents a simple and principled reformulation of hierarchical adaptive tokenization, exploiting score monotonicity to replace sequential refinement with global Top-K selection. Under the stated assumptions, this reformulation preserves the exact token allocation without introducing approximation errors. Although the underlying heap-order property is well established, I consider its application to eliminating the sequential dependency in adaptive tokenization a meaningful and elegant insight.
-
Meaningful system-level optimization with convincing empirical evidence. The paper identifies a concrete computational bottleneck in existing sequential refinement and demonstrates substantial end-to-end efficiency improvements while maintaining comparable downstream performance. The evaluation across multiple data modalities, together with detailed runtime profiling and scaling experiments, provides convincing evidence of the practical value of the proposed optimization.
Critical weaknesses
I didn't recognize main flaws that may justify rejection. Below are my major concerns:
-
Unclear practical advantages over competitive parallel baselines. While MAT achieves substantial speedups over sequential Top-1 refinement, its additional efficiency gains over existing parallel alternatives, such as Batch-M and APT, are relatively modest in several settings. MAT preserves exact greedy allocation and provides explicit token-budget control, but the paper does not sufficiently establish when these properties translate into meaningful practical advantages. A more systematic comparison of accuracy–efficiency trade-offs against strong parallel baselines would better substantiate its practical significance.
-
Insufficient characterization of applicability and computational trade-offs. MAT relies on importance scores being available for all candidate regions before selection and satisfying monotonicity, potentially through score adaptation. This assumption may limit its applicability to methods with dynamically computed or feature-dependent importance scores. Furthermore, computing scores across the full hierarchy may introduce unnecessary overhead when only a small fraction of regions requires refinement. The paper lacks a clear characterization of these applicability boundaries and the conditions under which global selection remains beneficial.
Questions for authors
-
Practical advantages over strong parallel baselines. Table 1 shows that MAT's efficiency gains over Batch-M and APT are relatively modest in several settings, despite substantial improvements over sequential Top-1 refinement. Could the authors clarify under which conditions MAT provides meaningful advantages over these competitive parallel methods? In particular, can the authors further substantiate the practical benefits of exact greedy allocation and fixed token-budget control, ideally through matched-budget accuracy–efficiency comparisons?
-
Applicability and computational trade-offs. MAT assumes that importance scores for all candidate regions can be computed before refinement. As I am not fully familiar with the broader adaptive tokenization literature, could the authors clarify which representative existing methods satisfy this assumption, and how broadly applicable it is in practice? Additionally, the Top-1 and MAT implementations share the same score-construction pipeline. Could the authors further discuss the computational trade-off between precomputing scores for the full hierarchy and computing them on demand during refinement?
2 comments
Thank you for your careful review and constructive feedback. We appreciate your positive assessment and the time you took to engage with our work throughout the discussion.
Thank you for the detailed rebuttal and additional clarification. The matched-budget comparison with APT provides useful evidence for MAT's practical advantages beyond sequential Top-1 refinement. I also appreciate the clearer characterization of the applicable scoring methods and the explicit discussion of the method's limitations.
I think these responses substantially address my main concerns. While the additional efficiency gains over strong parallel baselines remain somewhat setting-dependent, the combination of exact greedy allocation, explicit token-budget control, and competitive efficiency is a meaningful practical advantage.
Overall, I maintain my positive assessment.