acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Token–Block Mixing for Efficient Vision Encoders

Abstract

Efficient visual representation learning requires fine-grained token selection and access to long-range context. Local attention restricts direct cross-region interaction, whereas global associative aggregation can limit contextual diversity. We introduce Adaptive Token-Block Mixing (ATBM), a visual mixing operator that combines within-block softmax attention with structured communication over block-specific key-value states. A head-specific Toeplitz mixer propagates these states using learned functions of relative block-index distance, after which token queries retrieve cross-block context. Content-dependent gates fuse the local and cross-block outputs for each token and attention head. This design preserves explicit token interaction within blocks while avoiding a dense global token-to-token attention matrix. On ImageNet-1K, the tiny version ATBM-T achieves 77.2% top-1 accuracy with 6M parameters and 1.2 GFLOPs, improving over DeiT-T by 5.0 percentage points at the same reported model size. At resolution, ATBM-T achieves a inference speedup over DeiT-T while reducing peak GPU memory usage by 82.3%. Evaluations in plain and hierarchical encoders, image generators, together with transfer experiments on COCO and ADE20K, further demonstrate the applicability of the proposed operator.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Adaptive Token--Block Mixing, Linear Attention, Local Attention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aHQ6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ATBM combines softmax attention within contiguous token blocks with associative communication between blocks. A head-specific Toeplitz kernel mixes blockwise key–value states, and a token/head-dependent gate combines the two outputs. The paper evaluates the operator in plain and hierarchical encoders, dense prediction and diffusion models. It delivers improved accuracy and high-resolution efficiency.

Critical strengths

The two pathways have a sensible division of labor, and Table 6 supports their complementarity. The local-only and state-only models achieve 75.5% and 76.1%; fixed fusion reaches 76.5%, and the complete model reaches 77.2%. Tables 7 and 8 examine the mixer and fusion choices in more detail.

The gains are reported in both recognition and generation. Improvements over DiT at three sizes, together with COCO and ADE20K results, make the approach worth considering beyond ImageNet classification.

Critical weaknesses
  1. The stated architecture does not account for the reported model size. Table 9 gives ATBM-B 12 blocks, width 768 and MLP ratio 3.6, versus ratio 4 in DeiT-B. Assuming the shared QKV in Eq. (1) and an unchanged output projection, the router, kernel and path gains add approximately 0.113M parameters, while the smaller MLP removes approximately 5.677M weights. The described changes predict a decrease of about 5.56M, yet Table 2 reports an increase from 87M to 93M. Rounding and class-token removal cannot explain this gap. Appendix B.2 also enables an undefined local positional branch. A module-level count and the actual classification configurations are needed to determine whether the reported gains belong to the described operator.

  2. The main comparison does not isolate the mixer from the surrounding architecture. ATBM-T changes DeiT-T from three to six heads and replaces class-token readout with average pooling. At fixed width, the stated state-mixing cost is proportional to M2D2/HM^2D^2/H, so head count also affects efficiency. Tables 6–8 provide internal ablations, but no matched comparison to MHLA or the related mixed-chunk/MC-TFM++ construction. One Tiny experiment with identical heads, pooling, MLP width, positional modules and training recipe would be sufficient to test the mixer against softmax and one close alternative. Fig. 3 should specify the actual fused backend, precision and whether the neck/decoder are timed.

Table 8(b) has a related attribution issue: at 224×224, M=4M=4, and division by M−1M-1 can be absorbed into the learned global-path gain by gammaG′=γG−log⁡3gamma'_G=\gamma_G-\log 3. If calibration remains enabled, the two variants have the same function class at this length. Their accuracy difference may reflect initial path balance or gain regularization, consistent with the distinction between parameterization and optimization discussed by Salimans and Kingma (2016). Matching the initial effective gain would clarify this secondary ablation.

Questions for authors

With the Tiny backbone fixed, how much of the gain remains when only the mixer changes? Please compare against matched softmax; one adapted MHLA or MC-TFM++ variant would also clarify the specific benefit of the state mixer. Use the same training budget and report accuracy alongside latency and memory.

Which backend, dtype and modules are included in Fig. 3? In Table 8(b), are the path gains enabled, and how are their initialization and weight decay handled? A matched-initial-gain check would resolve the scale ambiguity.

2 comments
AuthorsOpenReview ↗

Author Response (1/2)

aHQ6-1 Correction of Base Model Statistics in the Main Text

We thank the reviewer for the careful check. ATBM-B was actually trained with MLP ratio 3.6, and the 82.4% result corresponds to this configuration. We updated the configuration in the appendix at the time but missed the resource statistics in the main text. With MLP ratio 3.6 and path scales enabled, the model has 88,285,624 parameters (about 88M) and 17.57 GFLOPs. In the revision, we correct 93M / 18.4G in Table 2 to 88M / 17.57G and update the related comparisons in the main text accordingly. The module-level parameter count is as follows:

ModuleParameters
Patch embedding590,592
Positional components (CPE / LePE)331,776
Attention (QKV, distance kernel, fusion gate, output gate, output projection)35,548,704
MLP50,988,432
Normalization and path scales57,120
Classification head769,000
Total88,285,624

aHQ6-2 Mixer Comparison with a Fixed Tiny Framework and Comparison with MHLA

We thank the reviewer for this concrete suggestion. We add comparisons among softmax attention, the adapted MHLA core, and ATBM in a unified Tiny framework. All three share the same architecture (12 layers, width 192, 6 heads, MLP ratio 4, average pooling, and identical positional and output modules) and are trained for 300 epochs.

When adapting MHLA, we retain its feature map, multi-window KV states, state mixing, and retrieval normalization. The two methods partition tokens differently. MHLA divides the 14×14 token grid into four 7×7 square windows, whereas ATBM flattens the grid in raster-scan order and groups every 49 consecutive tokens into a block. Latency and memory are measured at 224×224 with batch size 8, also on an RTX 3090 in FP32 with TF32 disabled and the math SDPA attention backend.

MethodParamsTop-1Latency (ms) ↓Peak memory (MiB) ↓
Adapted MHLA core6.2M76.5%19.0734.88
ATBM6.2M77.2%20.4335.73

When only the mixer is replaced, ATBM outperforms the adapted MHLA core by 0.7 points, with slightly higher latency and memory (+1.36 ms, +0.85 MiB). In the same framework, softmax attention reaches 77.1%. ATBM outperforms it by 0.1 points while being faster and using less memory at high resolution (see our response to C5yA-3).

aHQ6-3 Backend and Timing Scope of Figure 3

Figure 3 times only the inference of the two backbones and excludes the neck and decoder. The inputs follow the semantic segmentation setting, and we vary the resolution to compare how latency and memory grow. The original phrase “under the same UPerNet configuration” could easily be read as an end-to-end benchmark of the full UPerNet; we have corrected it in the revision.

The measurement setup is as follows:

  • Hardware and precision: RTX 3090, FP32.
  • Attention backend: naive math attention.
  • Batch size: 16.
  • Timing and memory: After 15 warm-up iterations, we perform 50 synchronized measurements and report the median latency and the PyTorch peak allocated CUDA memory.
AuthorsOpenReview ↗

Author Response (2/2)

aHQ6-4 Equivalence Between Normalization and Path Scales

We agree with the reviewer’s analysis. For a 224×224224\times224 input, N=196N=196 and C=49C=49, so M=4M=4. Let UU denote the unnormalized cross-block output. Then exp⁡(γG)⋅U/3=exp⁡(γG−ln⁡3)⋅U\exp(\gamma_G)\cdot U/3=\exp(\gamma_G-\ln 3)\cdot U. As long as the path scale is learnable and the number of blocks is fixed, division by M−1M-1 can be absorbed by the path scale, and the function class does not change.

The original explanation of Table 8(b) was therefore not accurate. When both settings are initialized with γG=0\gamma_G=0, their initial effective cross-block path weights differ by a factor of 3. The difference between 76.8% and 77.2% should therefore be understood as an effect of parameterization and initialization on optimization, rather than a difference in expressivity.

Note that this equivalence holds only when the number of blocks is fixed. With row-wise ℓ1\ell_1 normalization, whose row sums differ across blocks, or when MM varies with resolution, a single per-head path scale generally cannot perform the same absorption.

aHQ6-5 Use and Regularization of Path Scales

All normalization variants in Table 8(b) retain learnable per-head path scales and differ only in the cross-block normalization. The specific settings are as follows:

  • Parameterization: sL=exp⁡(γL)s_L=\exp(\gamma_L), sG=exp⁡(γG)s_G=\exp(\gamma_G).
  • Initialization: Both γL\gamma_L and γG\gamma_G are initialized to 0, so both pathways start with unit scale.
  • Weight decay: The same as for ordinary linear-layer weights (0.05), with no separate setting.
Reviewer_C5yAOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Adaptive Token–Block Mixing (ATBM), combining within-block softmax attention with cross-block communication through associative key–value states. A head-specific Toeplitz kernel propagates these states, while token-head gates and learned path scales combine local and cross-block outputs. Experiments cover classification, detection, segmentation, and diffusion generation. The largest classification gain is on DeiT-T, while a high-resolution Tiny/UPerNet benchmark demonstrates substantial latency and memory savings.

Critical strengths
  1. The Tiny-scale classification improvement is substantial: ATBM-T achieves 77.2% versus DeiT-T’s 72.2% at similar reported parameters and FLOPs. Improvements on detection, segmentation, and generation provide evidence that the approach is useful beyond image classification.
  2. Combining the local and state pathways improves over either pathway alone. The high-resolution Tiny/UPerNet measurements demonstrate a setting where ATBM delivers substantial latency and memory savings. These measurements are more informative than theoretical complexity alone.
Critical weaknesses
  1. ATBM introduces local attention, associative state construction, a parameterized Toeplitz mixer, adaptive token-head fusion, and learned path calibration. Yet classification improvements decrease from +5.0 points at Tiny scale to +1.6 at Small and +0.6 at Base. At Base scale, parameters increase from 87M to 93M and FLOPs from 17.6G to 18.4G. The complete design therefore offers a relatively small accuracy gain while increasing both reported resource measures. The ablations do not establish that all this machinery is necessary for a favorable practical trade-off: fixed fusion already achieves 76.5%, versus 77.2% for the complete model, and corresponding component comparisons are not reported at larger scales or in wall-clock time. Comparisons against closely related methods such as MHLA are also missing. Furthermore, the plain-backbone comparison does not cleanly isolate the mixer. Appendix A changes pooling, head count, and the Base MLP ratio, while Appendix B mentions an insufficiently specified local positional branch. As emphasized by the paper "How to Train Your ViT," augmentation, regularization, and training choices can materially affect architectural comparisons. A matched softmax baseline and an equally tuned simpler mixing baseline are needed to establish the incremental contribution.

  2. The strongest efficiency result is not connected to an demonstrated accuracy–runtime advantage across model scales. The headline 5.73× speedup and 82.3% memory reduction are measured for Tiny at 1280×1280, but the paper does not report task accuracy at this particular operating point. Consequently, it is unclear whether the largest speedup corresponds to a useful accuracy–latency trade-off. This distinction matters because increasing resolution and increasing model width have different computational effects. Pairwise attention grows quadratically with token count, making high-resolution Tiny a favorable setting for ATBM. Increasing width, however, raises the relative cost of the retained MLPs and projections, reducing the advantage of replacing pairwise attention. The Tiny resolution-scaling curve therefore cannot establish comparable benefits for Small or Base, precisely where the accuracy gains diminish. The paper reports no corresponding latency or memory measurements for these larger backbones, standard-resolution classification, or diffusion generation. Indeed, classification FLOPs are unchanged at Tiny scale and increase at Small and Base scales. Joint accuracy, latency, and memory comparisons across model sizes and resolutions are necessary to justify the broader efficiency contribution.

Questions for authors
  1. Can you report full-softmax and simpler mixing baselines with exactly the same pooling, head counts, MLP ratios, positional components, and training settings as ATBM? Which reported baselines were retrained, and what tuning budget was allocated to each?
  2. What task accuracy is achieved by both models at the resolutions in Figure 3, particularly 1280×1280? Can you report accuracy–latency curves rather than latency versus resolution alone?
  3. Can you provide end-to-end latency, throughput, and peak memory for Small and Base, including standard-resolution classification and representative high-resolution settings? How does the complete model compare with fixed fusion and simpler state mixers on these measures?
2 comments
AuthorsOpenReview ↗

Author Response (1/2)

C5yA-1a Controlled Comparison with Strictly Matched Peripheral Components

We agree with this point. The original comparison with DeiT includes differences in overall model configuration, so the 5.0-point gain cannot be attributed entirely to the mixer. We therefore add controlled comparisons in a unified Tiny framework in which only the mixer is replaced. The remaining settings are as follows:

  • Architecture: 224×224 input, patch size 16, 12 layers, width 192, 6 heads, MLP ratio 4, and average pooling.
  • Shared modules: CPE, LePE, QKV projection, output normalization, output gate, and output projection are kept identical.
  • Training: The same training recipe is used, and every model has about 6M parameters.

Here, softmax attention performs global interaction over all tokens, and the adapted MHLA retains its multi-state mixing core. “Static fusion” follows the learned global scalar of Table 8(a), i.e., a single input-independent learnable fusion coefficient shared across the network, and retains the path scales of both pathways. This differs from the setting in Table 6, which uses a fixed α=0.5 without path scales.

Mixer and fusionTop-1
Softmax attention77.1%
Adapted MHLA core76.5%
Uniform + static fusion76.4%
Toeplitz + static fusion76.7%
ATBM (Toeplitz + adaptive fusion)77.2%

With fully matched peripheral components, ATBM outperforms softmax attention by 0.1 points and the adapted MHLA by 0.7 points. Distance-dependent propagation and adaptive fusion each contribute:

  • Distance kernel: Under static fusion, Toeplitz mixing outperforms uniform mixing by 0.3 points.
  • Adaptive fusion: Keeping the Toeplitz kernel and path scales unchanged, replacing static fusion with adaptive fusion adds another 0.5 points (76.7% → 77.2%).

At the same time, softmax attention already reaches 77.1% in this framework. The value of the mixer itself lies in matching and slightly exceeding the accuracy of softmax attention at a lower cost than global attention, with complexity linear in sequence length. We have revised the relevant statements accordingly.

C5yA-1b Baseline Retraining and Tuning Budget

The results of the compared methods in Tables 1 and 2 are taken from their original papers and were not retrained. Their training recipes are not identical to ours, although we kept the comparisons as consistent and comparable as possible. We now state this in the table captions of the revision. All ablations in the paper are conducted in the same framework with the same training recipe.

The newly added softmax attention, adapted MHLA, and simplified mixing controls are all trained from scratch for 300 epochs with the same configuration:

  • Data and input: ImageNet-1K, 224×224.
  • Optimization: batch size 1024, AdamW, weight decay 0.05, peak learning rate 1e-3, 20-epoch warm-up, and cosine decay.
  • Other: The same data augmentation is used, without EMA or distillation.

Regarding the tuning budget, we did not perform a separate hyperparameter search for ATBM or any compared model. All models use the same training recipe and the same random seed.

C5yA-2 Joint Comparison of Accuracy and Latency at High Resolution

We thank the reviewer for pointing out this ambiguity. Figure 3 is a backbone-only inference benchmark. It varies the resolution under the semantic segmentation input setting and measures the latency and memory of the two Tiny backbones, excluding the neck and decoder. Segmentation accuracy is not evaluated at each resolution, so the points in the figure have no corresponding mIoU.

Segmentation accuracy is reported in Table 4. Under the standard ADE20K evaluation setting, DeiT-T and ATBM-T achieve 39.2 and 40.7 mIoU, respectively. Figure 3 shows how backbone cost grows with resolution, while Table 4 reports segmentation accuracy. The two cannot be combined into a per-resolution accuracy-latency curve.

In the revision, we explicitly label Figure 3 as a backbone-only inference benchmark. We also state that the 5.73× speedup and 82.3% memory reduction at 1280×1280 apply only to backbone inference and do not represent the accuracy or speed of the full UPerNet at that resolution. We agree that an accuracy-latency curve would better reflect the trade-off in practical deployment, and we list it as future work.

AuthorsOpenReview ↗

Author Response (2/2)

C5yA-3 Actual Cost at Small/Base Scales and Necessity of Components

We agree that the Tiny backbone benchmark in Figure 3 does not directly generalize to Small/Base or to full task networks. We therefore measure the forward cost of complete Small and Base classifiers at standard and high resolution, with three additional controls.

Setup. All methods share the same peripheral components (positional components, normalization, output gating, average pooling). Small: 12 layers, width 384, 6 heads, MLP ratio 4. Base: 12 layers, width 768, 12 heads, MLP ratio 3.6. We use an RTX 3090 in FP32 with TF32 disabled and math SDPA. We report the median of 50 runs after 15 warm-ups, covering the full classifier forward pass and excluding data loading and CPU-GPU transfer. Latency is measured at batch size 8 and throughput at batch size 4. Memory is PyTorch peak allocated memory at both batch sizes.

All ATBM variants retain learnable per-head path scales, unlike fixed fusion in Table 6. Fixed: α=0.5. Static: one learned, input-independent α shared across the network (Table 8(a)). Adaptive: α predicted per token and head. Uniform: uniform cross-block weights.

Table A: 224×224

ScaleMethodLat. B8 (ms)↓Thr. B4 (img/s)↑Mem. B8 (MiB)↓Mem. B4 (MiB)↓
SmallSoftmax10.52471.74146.08123.49
SmallToeplitz+fixed23.92202.43150.60126.65
SmallToeplitz+static23.80204.21150.60126.65
SmallUniform+adaptive21.64222.82150.72126.75
SmallATBM24.20200.24150.74126.77
BaseSoftmax25.97282.28438.02394.50
BaseToeplitz+fixed30.06198.45445.88400.15
BaseToeplitz+static30.21199.45445.88400.15
BaseUniform+adaptive28.92219.29446.36400.59
BaseATBM29.84186.55446.38400.61

Table B: 1024×1024

ScaleMethodLat. B8 (ms)↓Thr. B4 (img/s)↑Mem. B8 (MiB)↓Mem. B4 (MiB)↓
SmallSoftmax592.9413.5710563.895331.79
SmallToeplitz+fixed191.6640.681141.55620.73
SmallToeplitz+static191.5740.641141.55620.73
SmallUniform+adaptive191.2740.811142.38621.19
SmallATBM191.7440.601142.40621.21
BaseSoftmax1346.206.0021181.4210765.23
BaseToeplitz+fixed527.6915.112332.691343.05
BaseToeplitz+static527.5115.122332.691343.05
BaseUniform+adaptive526.9715.172334.631344.21
BaseATBM528.0515.122334.651344.22

224×224. ATBM is slower than softmax attention and uses slightly more memory. With only 196 tokens, the quadratic cost is small, while ATBM has more small operators. This overhead is amortized as width grows: ATBM’s latency is about 2.3× that of softmax attention for Small and about 1.15× for Base.

1024×1024. For Small/Base, ATBM is 3.09×/2.55× faster with 2.99×/2.52× higher throughput. Its peak memory is 89.2%/89.0% lower at batch size 8 and 88.3%/87.5% lower at batch size 4.

Component overhead. At 1024×1024, the ATBM variants differ by less than 0.3% in latency (191.27–191.74 ms for Small, 526.97–528.05 ms for Base), with nearly identical memory. At 224×224, the Toeplitz kernel adds about 2.6 ms (Small) and 0.9 ms (Base) over uniform mixing, and adaptive and static fusion differ by at most 0.4 ms. Both components therefore add little cost.

Reviewer_55giOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Adaptive Token-Block Mixing (ATBM), an efficient vision encoder that pairs local softmax attention with cross-block communication via associative key-value states. A head-specific Toeplitz mixer carries information across blocks, and content-dependent gating fuses local and global features. Image classification, object detection, semantic segmentation, and image generation experiments show that it is effective and efficient, and extensive ablation studies support these results.

Critical strengths

Comprehensive experimental evaluation. The paper tests ATBM on several vision tasks and on both plain and hierarchical backbones. Systematic ablations examine what the proposed components and design choices contribute.

Critical weaknesses
  1. Limited methodological novelty. ATBM builds largely on techniques that already exist: local-global attention (FLASH), multiple KV states (MHLA), Toeplitz mixing (TNN), and adaptive fusion (FocalNet). The integration works, but the experiments mainly check individual components against simplified variants and do not sufficiently establish the distinctive advantages of the proposed combination over existing alternatives.

  2. The distance-based cross-block mixing is not well justified. ATBM uses a monotonically decaying Toeplitz kernel built on one-dimensional block-index distance; such distance need not correspond to spatial proximity or semantic relatedness in a two-dimensional image. The ablations show that it works empirically, but the suitability of this distance prior for visual representation learning remains insufficiently justified.

Questions for authors
  1. The key components of ATBM have been explored in prior work (e.g., FLASH, MHLA, TNN, and FocalNet). Beyond combining them, what is the main methodological novelty? Could the authors provide more direct comparisons with existing alternatives to establish the distinctive benefits of the proposed design?

  2. Why is one-dimensional relative block-index distance a suitable way to model cross-block interactions in two-dimensional images? Could the imposed distance-decay prior restrict interactions between regions that are spatially far apart but semantically related?

  3. Why are the improvements larger on plain backbones than on hierarchical ones? How does progressive spatial downsampling affect the usefulness of cross-block state mixing and adaptive fusion?

3 comments
AuthorsOpenReview ↗

Author Response (1/2)

55gi-1 Novelty and Direct Comparisons

We thank the reviewer for this question. Each component used in ATBM does have a counterpart in prior work. Our contribution is not a new individual component, but an explicit division of labor between local interaction and cross-block communication. Specifically, ATBM works in three steps:

  • Within blocks: Local attention handles fine-grained local interaction.
  • Across blocks: Only KV states are propagated, weighted by a per-head symmetric distance kernel with a zero diagonal. The zero diagonal ensures that the cross-block pathway handles only communication between different blocks and does not duplicate the within-block modeling of the local pathway.
  • Retrieval and fusion: Each token reads the propagated states with its own query. The fusion ratio of the two pathways is then determined dynamically per token and per head.

In the revision, we clarify how ATBM relates to and differs from prior methods, so that these components are not read as our own proposals.

For direct comparisons, Table 6 in the original paper analyzes the two pathways and their fusion, and Table 7 analyzes different state propagation schemes. Following the reviewer’s suggestion, we add comparisons with softmax attention and an adapted MHLA core in a unified Tiny framework. All models share the same projections, positional components, output modules, and classification structure. Only the mixer is replaced, and every model is trained for 300 epochs.

MethodTop-1
Softmax attention77.1%
Adapted MHLA core76.5%
Uniform + static fusion76.4%
Toeplitz + static fusion76.7%
ATBM (Toeplitz + adaptive fusion)77.2%

Under matched settings, ATBM outperforms the adapted MHLA core by 0.7 points and slightly outperforms softmax attention (+0.1). In terms of efficiency, the cross-block pathway operates only on block-level states, so its cost is far lower than that of token-level global attention (see our response to C5yA-3). The two static-fusion controls further show that Toeplitz mixing outperforms uniform mixing by 0.3 points even without input-dependent fusion. Adding adaptive fusion on top of it improves accuracy by another 0.5 points (76.7% → 77.2%).

55gi-2 One-Dimensional Distance Prior and Long-Range Semantic Interaction

We agree that block-index distance in raster order is only an approximation of 2D spatial distance. We adopt this design to give cross-block communication an ordering structure with very few parameters. Parameters are shared across equal distances, so each head needs only 2R+1 parameters, and the parameter count does not grow with the number of blocks.

Experimentally, in Table 7 of the original paper, uniform mixing, shared Toeplitz, and per-head exponential Toeplitz achieve 76.5%, 76.9%, and 77.2%, respectively. To exclude the effect of adaptive fusion, we fix the fusion to the learned global scalar of Table 8(a), i.e., a single input-independent learnable fusion coefficient shared across the network, while retaining the path scales. Under this setting, uniform mixing reaches 76.4% and Toeplitz mixing 76.7%. Distance-dependent propagation still brings a 0.3-point gain, so its benefit does not depend on adaptive fusion. However, these results only show that the 1D distance kernel is an effective lightweight ordering prior; they do not show that it is better than a 2D geometric design. In the revision, we narrow this claim and list 2D distance kernels as future work.

As for regions that are spatially distant but semantically related, the distance kernel does not cut off their connection, for two reasons:

  • Long-range weights are not zero. As the distance increases, κh(δ)\kappa_h(\delta) approaches the learnable value σ(bh)\sigma(b_h) rather than being forced to zero.
  • The retrieved content is still determined by queries and keys. When expanded, the contribution coefficient of a cross-block token is Tmjh⟨ϕ(qih),ϕ(kj,th)⟩(M−1)Cjd\frac{T_{mj}^{h}\langle\phi(q_i^{h}),\phi(k_{j,t}^{h})\rangle}{(M-1)C_j\sqrt{d}} (M>1M>1). Distance only modulates the weight. Which information is retrieved is still determined by query-key similarity.
AuthorsOpenReview ↗

Author Response (2/2)

55gi-3 Different Gains in Plain and Hierarchical Backbones

We thank the reviewer for noticing this. The difference between plain and hierarchical gains appears mainly at the Tiny scale: +5.0 points over DeiT-T versus +2.3 points over Swin-T. At the Small scale, the gains are 1.6 and 1.5 points, which are already very close (Tables 1 and 2).

We attribute this to two factors.

First, the baselines differ. DeiT-T lacks a locality inductive bias and is relatively weak at small model sizes. Swin-T already has local modeling within windows and cross-window communication through shifted windows, so ATBM has relatively less to add.

Second, hierarchical architectures downsample progressively, so both the number of tokens and the number of blocks shrink in later stages. With fewer blocks, cross-block state mixing provides less additional communication, and adaptive fusion has less room to adjust. Moreover, by the later stages, downsampling has already enlarged the receptive field, so the local pathway covers most of the context. In contrast, plain architectures keep the same resolution in every layer, so cross-block communication is effective throughout the network.

Thank you for the detailed response and additional experiments. However, I remain concerned about whether the use of one-dimensional block-index distance as a prior for visual representation learning is appropriate. Therefore, I will maintain my original rating. I look forward to seeing the future development of this work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all three reviewers for their careful and constructive feedback. In response, we made the following main changes.

(1) Controlled mixer comparison. We add a comparison in a unified Tiny framework where only the mixer is replaced. ATBM reaches 77.2%, compared with 77.1% for softmax attention and 76.5% for an adapted MHLA core. Under static fusion, Toeplitz mixing still outperforms uniform mixing (76.7% vs. 76.4%), and adaptive fusion adds another 0.5 points. We also clarify that the 5.0-point gain over DeiT-T partly reflects differences in overall model configuration (55gi-1, C5yA-1a, aHQ6-2).

(2) Efficiency. We add forward-cost measurements of complete Small and Base classifiers. At 224×224, ATBM is slower than softmax attention. At 1024×1024, it is 3.09× (Small) and 2.55× (Base) faster and uses about 89% less peak memory, and the distance kernel and adaptive fusion add little cost. We also relabel Figure 3 as a backbone-only inference benchmark and correct its measurement details (C5yA-2, C5yA-3, aHQ6-3).

(3) Corrections and narrowed claims. We correct ATBM-B in Table 2 to 88M / 17.57G (MLP ratio 3.6). We revise the explanation of Table 8(b): for a fixed number of blocks, the 1/(M−1) factor can be absorbed by the learnable path scale, so the gap reflects parameterization and initialization rather than expressivity. We also present the 1D distance kernel as a lightweight ordering prior and list 2D kernels as future work (aHQ6-1, aHQ6-4, 55gi-2).

(4) Positioning and reproducibility. We state that ATBM’s components have counterparts in prior work and that its contribution is the explicit division of labor between local and cross-block interaction. We also note that baseline results are taken from the original papers, and document the training recipe, the path-scale settings, and the absence of model-specific tuning (55gi-1, C5yA-1b, aHQ6-5). All new results and details are included in the revised paper and supplementary material, and we hope these changes address the reviewers’ concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.