- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ATBM combines softmax attention within contiguous token blocks with associative communication between blocks. A head-specific Toeplitz kernel mixes blockwise key–value states, and a token/head-dependent gate combines the two outputs. The paper evaluates the operator in plain and hierarchical encoders, dense prediction and diffusion models. It delivers improved accuracy and high-resolution efficiency.
Critical strengths
The two pathways have a sensible division of labor, and Table 6 supports their complementarity. The local-only and state-only models achieve 75.5% and 76.1%; fixed fusion reaches 76.5%, and the complete model reaches 77.2%. Tables 7 and 8 examine the mixer and fusion choices in more detail.
The gains are reported in both recognition and generation. Improvements over DiT at three sizes, together with COCO and ADE20K results, make the approach worth considering beyond ImageNet classification.
Critical weaknesses
-
The stated architecture does not account for the reported model size. Table 9 gives ATBM-B 12 blocks, width 768 and MLP ratio 3.6, versus ratio 4 in DeiT-B. Assuming the shared QKV in Eq. (1) and an unchanged output projection, the router, kernel and path gains add approximately 0.113M parameters, while the smaller MLP removes approximately 5.677M weights. The described changes predict a decrease of about 5.56M, yet Table 2 reports an increase from 87M to 93M. Rounding and class-token removal cannot explain this gap. Appendix B.2 also enables an undefined local positional branch. A module-level count and the actual classification configurations are needed to determine whether the reported gains belong to the described operator.
-
The main comparison does not isolate the mixer from the surrounding architecture. ATBM-T changes DeiT-T from three to six heads and replaces class-token readout with average pooling. At fixed width, the stated state-mixing cost is proportional to , so head count also affects efficiency. Tables 6–8 provide internal ablations, but no matched comparison to MHLA or the related mixed-chunk/MC-TFM++ construction. One Tiny experiment with identical heads, pooling, MLP width, positional modules and training recipe would be sufficient to test the mixer against softmax and one close alternative. Fig. 3 should specify the actual fused backend, precision and whether the neck/decoder are timed.
Table 8(b) has a related attribution issue: at 224×224, , and division by can be absorbed into the learned global-path gain by . If calibration remains enabled, the two variants have the same function class at this length. Their accuracy difference may reflect initial path balance or gain regularization, consistent with the distinction between parameterization and optimization discussed by Salimans and Kingma (2016). Matching the initial effective gain would clarify this secondary ablation.
Questions for authors
With the Tiny backbone fixed, how much of the gain remains when only the mixer changes? Please compare against matched softmax; one adapted MHLA or MC-TFM++ variant would also clarify the specific benefit of the state mixer. Use the same training budget and report accuracy alongside latency and memory.
Which backend, dtype and modules are included in Fig. 3? In Table 8(b), are the path gains enabled, and how are their initialization and weight decay handled? A matched-initial-gain check would resolve the scale ambiguity.
2 comments
Author Response (2/2)
aHQ6-4 Equivalence Between Normalization and Path Scales
We agree with the reviewer’s analysis. For a input, and , so . Let denote the unnormalized cross-block output. Then . As long as the path scale is learnable and the number of blocks is fixed, division by can be absorbed by the path scale, and the function class does not change.
The original explanation of Table 8(b) was therefore not accurate. When both settings are initialized with , their initial effective cross-block path weights differ by a factor of 3. The difference between 76.8% and 77.2% should therefore be understood as an effect of parameterization and initialization on optimization, rather than a difference in expressivity.
Note that this equivalence holds only when the number of blocks is fixed. With row-wise normalization, whose row sums differ across blocks, or when varies with resolution, a single per-head path scale generally cannot perform the same absorption.
aHQ6-5 Use and Regularization of Path Scales
All normalization variants in Table 8(b) retain learnable per-head path scales and differ only in the cross-block normalization. The specific settings are as follows:
- Parameterization: , .
- Initialization: Both and are initialized to 0, so both pathways start with unit scale.
- Weight decay: The same as for ordinary linear-layer weights (0.05), with no separate setting.
Author Response (1/2)
aHQ6-1 Correction of Base Model Statistics in the Main Text
We thank the reviewer for the careful check. ATBM-B was actually trained with MLP ratio 3.6, and the 82.4% result corresponds to this configuration. We updated the configuration in the appendix at the time but missed the resource statistics in the main text. With MLP ratio 3.6 and path scales enabled, the model has 88,285,624 parameters (about 88M) and 17.57 GFLOPs. In the revision, we correct 93M / 18.4G in Table 2 to 88M / 17.57G and update the related comparisons in the main text accordingly. The module-level parameter count is as follows:
aHQ6-2 Mixer Comparison with a Fixed Tiny Framework and Comparison with MHLA
We thank the reviewer for this concrete suggestion. We add comparisons among softmax attention, the adapted MHLA core, and ATBM in a unified Tiny framework. All three share the same architecture (12 layers, width 192, 6 heads, MLP ratio 4, average pooling, and identical positional and output modules) and are trained for 300 epochs.
When adapting MHLA, we retain its feature map, multi-window KV states, state mixing, and retrieval normalization. The two methods partition tokens differently. MHLA divides the 14×14 token grid into four 7×7 square windows, whereas ATBM flattens the grid in raster-scan order and groups every 49 consecutive tokens into a block. Latency and memory are measured at 224×224 with batch size 8, also on an RTX 3090 in FP32 with TF32 disabled and the math SDPA attention backend.
When only the mixer is replaced, ATBM outperforms the adapted MHLA core by 0.7 points, with slightly higher latency and memory (+1.36 ms, +0.85 MiB). In the same framework, softmax attention reaches 77.1%. ATBM outperforms it by 0.1 points while being faster and using less memory at high resolution (see our response to C5yA-3).
aHQ6-3 Backend and Timing Scope of Figure 3
Figure 3 times only the inference of the two backbones and excludes the neck and decoder. The inputs follow the semantic segmentation setting, and we vary the resolution to compare how latency and memory grow. The original phrase “under the same UPerNet configuration” could easily be read as an end-to-end benchmark of the full UPerNet; we have corrected it in the revision.
The measurement setup is as follows: