- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes VisionAlphabet, a complex-valued compact visual backbone that represents and propagates spatial context through learned Laplace poles. Each pole has a learned damping and phase, giving a stable discrete transition a = e^{−d+iθ}. A rank-2 complex reader produces pole-specific excitations, which are propagated by four directional separable product scans (a row recurrence followed by a column recurrence) with reference-gain normalization. At stage transitions, direction-aware endpoint coarsening of the memory states is combined with a carry branch, followed by a gated widely-linear refinement. The classifier reads only the log of the spatially averaged, gain-normalized directional pole energies of the terminal scan. The paper also gives a product-Poisson characterization of population pole energies under second-order stationarity, adapted from ALPHABET (Ko et al., 2026).
The method is validated on ImageNet-1K under a shared 100-epoch protocol without distillation, where K96 reaches 72.35% Top-1 with 3.25M parameters and K128 reaches 75.02% with 5.08M parameters. Additional experiments cover the ImageNet-1K 10% subset, Food-101, COCO detection and instance segmentation with a common Mask R-CNN/FPN head (39.60 box AP and 36.64 mask AP for K128), and ImageNet-C. Finally, the authors apply frozen-checkpoint interventions on ImageNet-1K (periodic state resets, removal of phase and damping, and neutralization of selected poles) and use them to argue that the trained model relies on pole-state transport and phase.
Critical strengths
1.Transparent and reproducible experimental reporting. Main results report three seeds with sample SD and use final-epoch rather than best-epoch accuracy. The ImageNet-1K 10% subset is a pinned public list. One explicit FLOP-counting rule is applied to all models, and latency, throughput and peak memory are measured on fixed hardware (Table 4). This makes the reported numbers credible and makes the accuracy–efficiency trade-off assessable.
2.An interpretable operator with consistent gains in data-limited and transfer settings. Each pole has two explicit parameters (damping and phase), which enables targeted frozen-checkpoint interventions (Section 6). Among the compared models, K128 leads on: ImageNet-1K 10%: 58.21% vs. 55.46% for ParC-Net-S; Food-101: 87.42% vs. 85.23%; COCO: 39.60 vs. 37.99 box AP over TinyViM-S. These are the most interesting empirical signals in the paper.
Critical weaknesses
1.The efficiency comparison is unfair: the parameter-compact claim does not translate into compute efficiency. All headline comparisons (Abstract, Fig. 2a, Table 1) are parameter-matched. However, Table 4, under the paper’s own counting rule, shows that VisionAlphabet uses up to an order of magnitude more FLOPs than the baselines it is compared against:
- At similar FLOPs, VA is far behind. K32 uses 1.61 GFLOPs and reaches ≈54.3% (75.02 − 20.70), whereas ConvNeXtV2-Atto reaches 72.01% with 1.14 GFLOPs and TinyViM-S reaches 74.42% with 1.92 GFLOPs.
- K128 vs. TinyViM-S: 19.51 vs. 1.92 GFLOPs (≈10.2×) for a +0.60 Top-1 gain; batch-16 throughput is 2,665 vs. 5,210 img/s.
- K96 vs. TinyViM-S: K96 has lower accuracy (72.35 vs. 74.42), ≈6× the FLOPs, and lower throughput (3,589 vs. 5,210 img/s).
2.The “Laplace pole = spatial memory” insight has limited novelty and lacks methodological and experimental comparison with the closest prior work. RetNet (Sun et al., 2023) derives its retention mechanism from a recurrence with complex decay γe^{iθ}. This is the same mechanism as here: damping controls persistence, and per-step phase rotation controls how contributions combine. RMT (Fan et al., CVPR 2024) brings this to vision with an explicit spatial decay prior, γ^{|Δx|+|Δy|} (Manhattan distance), and axis-decomposed attention, which is conceptually close to the separable spatial damping used here. Neither work is cited or discussed, and RMT is not included as a baseline. The paper should clarify its conceptual contribution relative to RetNet, RMT and ALPHABET, and compare against RMT (or retention-based token mixers) under the same protocol and matched FLOPs.
3.Unclear motivation and research question. The introduction describes how CNNs, ViTs and visual SSMs aggregate spatial information, then directly proposes the method. It never identifies a specific limitation of existing compact backbones that VisionAlphabet addresses. It is therefore unclear whether the target is accuracy, efficiency, data efficiency or interpretability, and which experiment demonstrates that the targeted problem is solved.
4.Missing ADE20K semantic segmentation. ADE20K is a standard benchmark for backbone papers, and dense prediction is where spatial context propagation should matter most. Only COCO with a 12-epoch Mask R-CNN schedule is reported, and backbone FLOPs and latency at the detection resolution are not given.
Questions for authors
1.Can you provide FLOPs- or latency-matched comparisons, e.g., a VisionAlphabet variant at 1–2 GFLOPs against the baselines, or baselines scaled to 12–20 GFLOPs? Can you add accuracy–FLOPs and accuracy–throughput plots to the main text? 2.How does the pole memory differ from RetNet’s decayed-rotation retention and RMT’s Manhattan spatial decay, conceptually and empirically under the same training protocol? 3.How does VisionAlphabet compare with EfficientVMamba, MobileNetV4 and RepViT under the same protocol? 4.Which specific limitation of existing compact backbones does VisionAlphabet target, and which experiment demonstrates that it is resolved?
3 comments
Q2: Relationship to RetNet and RMT, and the contribution of spatial pole-energy representations
Thank you for highlighting RetNet and RMT. We have added both references to Related Work and clarified the differences in spatial propagation, state representation, and readout. However, describing these methods as using “the same mechanism” overlooks differences in the operators, state representations, and mathematical quantities being measured. Sharing exponential decay or rotation does not make their resulting representations equivalent.
RetNet uses content-dependent key–value accumulation and query-based retrieval:
Here, the query and key terms include the positional rotations used in retention.
RMT’s Manhattan Self-Attention (MaSA) combines content-dependent self-attention with an explicit spatial prior based on distance decay. In RMT’s early stages, this operation is decomposed along the two spatial axes:
VisionAlphabet instead propagates pole-specific excitations from local readers through learned complex spatial recurrences:
The terminal readout directly computes normalized energies from the directional states, without intervening GatedWL mixing. The resulting representation consists of pole-energy descriptors with an explicit spectral characterization, rather than content-dependent retention or attention outputs. Thus, the distinction concerns both how spatial information is propagated and the mathematical form of the features used for classification—not merely terminology or a reinterpretation of decay.
Building on ALPHABET’s pole-energy measurement principle, VisionAlphabet extends this construction to two spatial axes and a hierarchical visual backbone. Our contribution is the resulting spatial representation and its evaluation. We have clarified this relationship and the distinction from retention and MaSA in the revised Related Work section.
Q3: Comparison with EfficientVMamba, MobileNetV4, and RepViT
Thank you for suggesting these additional compact backbones. We evaluated EfficientVMamba-T, MobileNetV4-Small, and RepViT-M0.9 alongside VisionAlphabet under the same 100-epoch ImageNet-1K training protocol. The table reports mean ± SD across three training seeds.
| Architecture / variant | Top-1 mean ± SD (%) | Top-5 mean ± SD (%) | Parameters (M) |
|---|---|---|---|
| VisionAlphabet-K128 | 75.019 ± 0.050 | 92.575 ± 0.057 | 5.083 |
| VisionAlphabet-K96 | 72.352 ± 0.058 | 91.337 ± 0.074 | 3.253 |
| RepViT-M0.9 | 74.600 ± 0.123 | 92.282 ± 0.078 | 5.104 |
| EfficientVMamba-T | 69.726 ± 0.184 | 89.768 ± 0.113 | 6.494 |
| MobileNetV4-Small | 65.044 ± 0.147 | 85.790 ± 0.096 | 3.774 |
K128 achieves higher mean Top-1 and Top-5 accuracy than the three additional baselines, exceeding RepViT-M0.9 at a nearly identical parameter count and EfficientVMamba-T with fewer parameters. K96 also exceeds MobileNetV4-Small and EfficientVMamba-T with fewer parameters.
Q4: Research objective and experimental evidence
Our work does not posit a specific limitation shared by existing compact backbones. Instead, it asks whether Laplace-pole states and energy measurements offer an effective alternative for compressing spatial context while retaining recognition-relevant information. The contribution is the construction and validation of this representation, assessed through its accuracy–parameter trade-off, transferability, and the role of its core components.
The experiments test three aspects of this objective. First, the ImageNet-1K comparisons evaluate recognition performance relative to parameter count, while the low-data, Food-101, and COCO evaluations test usefulness across data conditions and tasks. Second, the mechanistic analyses test whether the trained classifier actually uses the proposed representation: state resets interrupt accumulated spatial context, terminal-phase interventions change the final scan while holding upstream features fixed, and selected-versus-random pole interventions assess the predictive relevance of the identified pole evidence.
Third, retrained ablations evaluate whether the design choices remain useful when alternatives can adapt during training. We trained VA-K64 and three variants from scratch on ImageNet-1K under the same 100-epoch protocol using seeds 0, 1, and 2. P fixes scan phases to zero while retaining learned damping and the remaining complex-valued components; A retains all four scans but averages their terminal energies across directions; G replaces energy pooling with spatial average pooling of the signed real and imaginary state components. Results are mean ± SD.
| Variant | Top-1 mean ± SD (%) | Top-5 mean ± SD (%) | Parameters |
|---|---|---|---|
| Original VA-K64 (B) | 66.871 ± 0.095 | 88.105 ± 0.004 | 1,562,760 |
| Phase-free scan (P) | 65.594 ± 0.014 | 87.180 ± 0.091 | 1,562,760 |
| Direction-averaged energy (A) | 65.647 ± 0.007 | 87.409 ± 0.148 | 1,322,760 |
| Feature pooling (G) | 66.900 ± 0.130 | 88.158 ± 0.288 | 1,882,760 |
Removing phase reduces mean Top-1 by 1.277 percentage points at the same parameter count (B − P). Retaining directional energies improves mean Top-1 by 1.224 points (B − A), with an accompanying head-capacity difference. The energy readout achieves similar accuracy to feature pooling with 17.0% fewer total parameters than G. Together, these results support the stated objective: pole-based spatial representations retain useful recognition information, and the energy descriptor provides a compact way to use it.
Thank you for the detailed comments on related work, motivation, and computational cost, which helped us sharpen the presentation of the contribution and its trade-offs.
Thank you for recognizing the transparent experimental reporting and the results in low-data and transfer settings. Our central question is whether learned Laplace-pole states can compress spatial context into an effective visual representation under a limited parameter budget. We address computational cost, the relationship to prior work, the research motivation, and additional evaluation below.
Q1: Parameter compactness and computational efficiency
Our initial focus was on evaluating Laplace-pole representations for visual recognition rather than aggressively compressing the surrounding layers. We appreciate your feedback, which prompted us to examine VisionAlphabet’s computational efficiency more closely and identify additional savings from its existing factorization. Instead of composing the learned factors into a dense convolution kernel, we now evaluate the same operation through channel projection followed by spatial filtering:
Up(x)=∑i,Δ(∑r=1RDp,r,ΔAp,r,i)Ei(x+Δ)=∑r=1R∑ΔDp,r,Δ(∑iAp,r,iEi(x+Δ)).
This algebraic reassociation preserves the architecture, learned parameters, and checkpoint without approximation or retraining. It reduces whole-model FLOPs by 44.8–52.4%, with the following measured execution performance:
Following your suggestion, K48 now falls within the proposed 1–2 GFLOP range at 1.64G, while equivalent execution reduces K96/K128 to 5.63/9.29G without scaling the baselines to the original 12–20 GFLOP range.
Figure 7 shows how the optimized execution changes the cited comparisons. K128’s FLOP ratio to TinyViM-S decreases from approximately 10.2× to 4.84×, while its batch-16 throughput increases to 4,095 images/s. For K96, the FLOP ratio decreases from approximately 6.1× to 2.93×, and throughput reaches 5,299 images/s, comparable to TinyViM-S’s 5,210 images/s. K32 now requires 0.889 GFLOPs; its accuracy result is unchanged.
The FLOP disadvantage remains, but the measured latency and throughput reveal operating-point-specific trade-offs. In particular, K128 achieves substantially lower batch-1 latency than TinyViM-S (0.514 vs. 2.90 ms). At comparable batch-16 throughput, K96 achieves 72.35% Top-1 versus 72.01% for ParC-Net-S (5,299 vs. 5,161 images/s). These results distinguish FLOP efficiency from latency and throughput under the reported measurement settings.
Figure 8 provides context for the remaining cost. Our study prioritizes evaluating Laplace-pole spatial representations without aggressively compressing the surrounding fully connected layers and linear projections. K128 nevertheless achieves competitive recognition with fewer parameters, while pole scans themselves account for only 1.32% of total FLOPs. We would appreciate it if these results were assessed in light of this design scope, recognizing that most arithmetic arises in the surrounding feature transformations rather than pole propagation.
We have updated Table 4 and added Figures 7–8 in Appendix A, together with the derivation of the equivalent Reader computation and operation breakdown.