acceptodds
Under review as a conference paper at ICLR 2027

VisionAlphabet: Compressing Spatial Context with Learned Laplace Poles

Abstract

VisionAlphabet introduces a distinct approach to compact visual backbones, whereby spatial context is represented and propagated through learned Laplace poles. Each pole provides a stable complex-valued state that carries spatial context: damping controls how far contributions persist across space, while phase modulates how contributions from different locations combine. Four directional scans propagate spatial evidence across the image; the terminal scan's gain-normalized pole energies are spatially averaged to form the image-level classifier descriptor. Under a shared 100-epoch ImageNet-1K protocol without distillation, the 3.25M- and 5.08M-parameter models attain 72.35% and 75.02% Top-1 accuracy, respectively, while using 10–12% fewer parameters than their corresponding compact baselines. The larger variant also outperforms the compared baselines with 10% of the ImageNet-1K training data and, when pretrained on full ImageNet-1K, transfers to COCO detection and instance segmentation. Controlled interventions on a frozen checkpoint show that recognition changes when pole-state transport is disrupted or phase is removed, supporting the interpretation of pole states as functional spatial memory.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
compact vision models, Laplace poles, spatial memory, state-space models, hierarchical vision backbone, efficient image recognition, interpretable vision models, ImageNet classification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oSLLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes VisionAlphabet, a complex-valued compact visual backbone that represents and propagates spatial context through learned Laplace poles. Each pole has a learned damping and phase, giving a stable discrete transition a = e^{−d+iθ}. A rank-2 complex reader produces pole-specific excitations, which are propagated by four directional separable product scans (a row recurrence followed by a column recurrence) with reference-gain normalization. At stage transitions, direction-aware endpoint coarsening of the memory states is combined with a carry branch, followed by a gated widely-linear refinement. The classifier reads only the log of the spatially averaged, gain-normalized directional pole energies of the terminal scan. The paper also gives a product-Poisson characterization of population pole energies under second-order stationarity, adapted from ALPHABET (Ko et al., 2026).

The method is validated on ImageNet-1K under a shared 100-epoch protocol without distillation, where K96 reaches 72.35% Top-1 with 3.25M parameters and K128 reaches 75.02% with 5.08M parameters. Additional experiments cover the ImageNet-1K 10% subset, Food-101, COCO detection and instance segmentation with a common Mask R-CNN/FPN head (39.60 box AP and 36.64 mask AP for K128), and ImageNet-C. Finally, the authors apply frozen-checkpoint interventions on ImageNet-1K (periodic state resets, removal of phase and damping, and neutralization of selected poles) and use them to argue that the trained model relies on pole-state transport and phase.

Critical strengths

1.Transparent and reproducible experimental reporting. Main results report three seeds with sample SD and use final-epoch rather than best-epoch accuracy. The ImageNet-1K 10% subset is a pinned public list. One explicit FLOP-counting rule is applied to all models, and latency, throughput and peak memory are measured on fixed hardware (Table 4). This makes the reported numbers credible and makes the accuracy–efficiency trade-off assessable.

2.An interpretable operator with consistent gains in data-limited and transfer settings. Each pole has two explicit parameters (damping and phase), which enables targeted frozen-checkpoint interventions (Section 6). Among the compared models, K128 leads on: ImageNet-1K 10%: 58.21% vs. 55.46% for ParC-Net-S; Food-101: 87.42% vs. 85.23%; COCO: 39.60 vs. 37.99 box AP over TinyViM-S. These are the most interesting empirical signals in the paper.

Critical weaknesses

1.The efficiency comparison is unfair: the parameter-compact claim does not translate into compute efficiency. All headline comparisons (Abstract, Fig. 2a, Table 1) are parameter-matched. However, Table 4, under the paper’s own counting rule, shows that VisionAlphabet uses up to an order of magnitude more FLOPs than the baselines it is compared against:

  • At similar FLOPs, VA is far behind. K32 uses 1.61 GFLOPs and reaches ≈54.3% (75.02 − 20.70), whereas ConvNeXtV2-Atto reaches 72.01% with 1.14 GFLOPs and TinyViM-S reaches 74.42% with 1.92 GFLOPs.
  • K128 vs. TinyViM-S: 19.51 vs. 1.92 GFLOPs (≈10.2×) for a +0.60 Top-1 gain; batch-16 throughput is 2,665 vs. 5,210 img/s.
  • K96 vs. TinyViM-S: K96 has lower accuracy (72.35 vs. 74.42), ≈6× the FLOPs, and lower throughput (3,589 vs. 5,210 img/s).

2.The “Laplace pole = spatial memory” insight has limited novelty and lacks methodological and experimental comparison with the closest prior work. RetNet (Sun et al., 2023) derives its retention mechanism from a recurrence with complex decay γe^{iθ}. This is the same mechanism as here: damping controls persistence, and per-step phase rotation controls how contributions combine. RMT (Fan et al., CVPR 2024) brings this to vision with an explicit spatial decay prior, γ^{|Δx|+|Δy|} (Manhattan distance), and axis-decomposed attention, which is conceptually close to the separable spatial damping used here. Neither work is cited or discussed, and RMT is not included as a baseline. The paper should clarify its conceptual contribution relative to RetNet, RMT and ALPHABET, and compare against RMT (or retention-based token mixers) under the same protocol and matched FLOPs.

3.Unclear motivation and research question. The introduction describes how CNNs, ViTs and visual SSMs aggregate spatial information, then directly proposes the method. It never identifies a specific limitation of existing compact backbones that VisionAlphabet addresses. It is therefore unclear whether the target is accuracy, efficiency, data efficiency or interpretability, and which experiment demonstrates that the targeted problem is solved.

4.Missing ADE20K semantic segmentation. ADE20K is a standard benchmark for backbone papers, and dense prediction is where spatial context propagation should matter most. Only COCO with a 12-epoch Mask R-CNN schedule is reported, and backbone FLOPs and latency at the detection resolution are not given.

Questions for authors

1.Can you provide FLOPs- or latency-matched comparisons, e.g., a VisionAlphabet variant at 1–2 GFLOPs against the baselines, or baselines scaled to 12–20 GFLOPs? Can you add accuracy–FLOPs and accuracy–throughput plots to the main text? 2.How does the pole memory differ from RetNet’s decayed-rotation retention and RMT’s Manhattan spatial decay, conceptually and empirically under the same training protocol? 3.How does VisionAlphabet compare with EfficientVMamba, MobileNetV4 and RepViT under the same protocol? 4.Which specific limitation of existing compact backbones does VisionAlphabet target, and which experiment demonstrates that it is resolved?

3 comments

Thank you for recognizing the transparent experimental reporting and the results in low-data and transfer settings. Our central question is whether learned Laplace-pole states can compress spatial context into an effective visual representation under a limited parameter budget. We address computational cost, the relationship to prior work, the research motivation, and additional evaluation below.

Q1: Parameter compactness and computational efficiency

Our initial focus was on evaluating Laplace-pole representations for visual recognition rather than aggressively compressing the surrounding layers. We appreciate your feedback, which prompted us to examine VisionAlphabet’s computational efficiency more closely and identify additional savings from its existing factorization. Instead of composing the learned factors into a dense convolution kernel, we now evaluate the same operation through channel projection followed by spatial filtering:

Up(x)=∑i,Δ(∑r=1RDp,r,ΔAp,r,i)Ei(x+Δ)=∑r=1R∑ΔDp,r,Δ(∑iAp,r,iEi(x+Δ)).U_p(x)=\sum_{i,\Delta}\left(\sum_{r=1}^{R}D_{p,r,\Delta}A_{p,r,i}\right)E_i(x+\Delta)=\sum_{r=1}^{R}\sum_{\Delta}D_{p,r,\Delta}\left(\sum_i A_{p,r,i}E_i(x+\Delta)\right).

This algebraic reassociation preserves the architecture, learned parameters, and checkpoint without approximation or retraining. It reduces whole-model FLOPs by 44.8–52.4%, with the following measured execution performance:

ModelParameters (M)Original GFLOPsOptimized GFLOPsReductionB1 latency (ms)B16 throughput (img/s)B16 peak (MiB)
K320.5731.6100.88944.8%0.32013,450138.9
K481.0053.1221.63847.5%0.34010,189141.5
K641.5635.1262.61748.9%0.3688,320159.1
K963.25311.7325.62852.0%0.4485,299198.2
K1285.08319.5089.28752.4%0.5144,095257.1

Following your suggestion, K48 now falls within the proposed 1–2 GFLOP range at 1.64G, while equivalent execution reduces K96/K128 to 5.63/9.29G without scaling the baselines to the original 12–20 GFLOP range.

Figure 7 shows how the optimized execution changes the cited comparisons. K128’s FLOP ratio to TinyViM-S decreases from approximately 10.2× to 4.84×, while its batch-16 throughput increases to 4,095 images/s. For K96, the FLOP ratio decreases from approximately 6.1× to 2.93×, and throughput reaches 5,299 images/s, comparable to TinyViM-S’s 5,210 images/s. K32 now requires 0.889 GFLOPs; its accuracy result is unchanged.

The FLOP disadvantage remains, but the measured latency and throughput reveal operating-point-specific trade-offs. In particular, K128 achieves substantially lower batch-1 latency than TinyViM-S (0.514 vs. 2.90 ms). At comparable batch-16 throughput, K96 achieves 72.35% Top-1 versus 72.01% for ParC-Net-S (5,299 vs. 5,161 images/s). These results distinguish FLOP efficiency from latency and throughput under the reported measurement settings.

Figure 8 provides context for the remaining cost. Our study prioritizes evaluating Laplace-pole spatial representations without aggressively compressing the surrounding fully connected layers and linear projections. K128 nevertheless achieves competitive recognition with fewer parameters, while pole scans themselves account for only 1.32% of total FLOPs. We would appreciate it if these results were assessed in light of this design scope, recognizing that most arithmetic arises in the surrounding feature transformations rather than pole propagation.

We have updated Table 4 and added Figures 7–8 in Appendix A, together with the derivation of the equivalent Reader computation and operation breakdown.

Q2: Relationship to RetNet and RMT, and the contribution of spatial pole-energy representations

Thank you for highlighting RetNet and RMT. We have added both references to Related Work and clarified the differences in spatial propagation, state representation, and readout. However, describing these methods as using “the same mechanism” overlooks differences in the operators, state representations, and mathematical quantities being measured. Sharing exponential decay or rotation does not make their resulting representations equivalent.

RetNet uses content-dependent key–value accumulation and query-based retrieval:

Sn=γSn−1+Kn⊤Vn,on=QnSn.S_n=\gamma S_{n-1}+K_n^\top V_n,\qquad o_n=Q_nS_n.

Here, the query and key terms include the positional rotations used in retention.

RMT’s Manhattan Self-Attention (MaSA) combines content-dependent self-attention with an explicit spatial prior based on distance decay. In RMT’s early stages, this operation is decomposed along the two spatial axes:

MaSA⁡(X)=(Softmax⁡(QK⊤)⊙D2d)V,Dnm2d=γ∣xn−xm∣+∣yn−ym∣.\operatorname{MaSA}(X)=\bigl(\operatorname{Softmax}(QK^\top)\odot D^{2d}\bigr)V,\qquad D^{2d}_{nm}=\gamma^{|x_n-x_m|+|y_n-y_m|}.

VisionAlphabet instead propagates pole-specific excitations from local readers through learned complex spatial recurrences:

zt,p=apzt−1,p+γput,p.z_{t,p}=a_pz_{t-1,p}+\gamma_pu_{t,p}.

The terminal readout directly computes normalized energies from the directional states, without intervening GatedWL mixing. The resulting representation consists of pole-energy descriptors with an explicit spectral characterization, rather than content-dependent retention or attention outputs. Thus, the distinction concerns both how spatial information is propagated and the mathematical form of the features used for classification—not merely terminology or a reinterpretation of decay.

Building on ALPHABET’s pole-energy measurement principle, VisionAlphabet extends this construction to two spatial axes and a hierarchical visual backbone. Our contribution is the resulting spatial representation and its evaluation. We have clarified this relationship and the distinction from retention and MaSA in the revised Related Work section.

Q3: Comparison with EfficientVMamba, MobileNetV4, and RepViT

Thank you for suggesting these additional compact backbones. We evaluated EfficientVMamba-T, MobileNetV4-Small, and RepViT-M0.9 alongside VisionAlphabet under the same 100-epoch ImageNet-1K training protocol. The table reports mean ± SD across three training seeds.

Architecture / variantTop-1 mean ± SD (%)Top-5 mean ± SD (%)Parameters (M)
VisionAlphabet-K12875.019 ± 0.05092.575 ± 0.0575.083
VisionAlphabet-K9672.352 ± 0.05891.337 ± 0.0743.253
RepViT-M0.974.600 ± 0.12392.282 ± 0.0785.104
EfficientVMamba-T69.726 ± 0.18489.768 ± 0.1136.494
MobileNetV4-Small65.044 ± 0.14785.790 ± 0.0963.774

K128 achieves higher mean Top-1 and Top-5 accuracy than the three additional baselines, exceeding RepViT-M0.9 at a nearly identical parameter count and EfficientVMamba-T with fewer parameters. K96 also exceeds MobileNetV4-Small and EfficientVMamba-T with fewer parameters.

Q4: Research objective and experimental evidence

Our work does not posit a specific limitation shared by existing compact backbones. Instead, it asks whether Laplace-pole states and energy measurements offer an effective alternative for compressing spatial context while retaining recognition-relevant information. The contribution is the construction and validation of this representation, assessed through its accuracy–parameter trade-off, transferability, and the role of its core components.

The experiments test three aspects of this objective. First, the ImageNet-1K comparisons evaluate recognition performance relative to parameter count, while the low-data, Food-101, and COCO evaluations test usefulness across data conditions and tasks. Second, the mechanistic analyses test whether the trained classifier actually uses the proposed representation: state resets interrupt accumulated spatial context, terminal-phase interventions change the final scan while holding upstream features fixed, and selected-versus-random pole interventions assess the predictive relevance of the identified pole evidence.

Third, retrained ablations evaluate whether the design choices remain useful when alternatives can adapt during training. We trained VA-K64 and three variants from scratch on ImageNet-1K under the same 100-epoch protocol using seeds 0, 1, and 2. P fixes scan phases to zero while retaining learned damping and the remaining complex-valued components; A retains all four scans but averages their terminal energies across directions; G replaces energy pooling with spatial average pooling of the signed real and imaginary state components. Results are mean ± SD.

VariantTop-1 mean ± SD (%)Top-5 mean ± SD (%)Parameters
Original VA-K64 (B)66.871 ± 0.09588.105 ± 0.0041,562,760
Phase-free scan (P)65.594 ± 0.01487.180 ± 0.0911,562,760
Direction-averaged energy (A)65.647 ± 0.00787.409 ± 0.1481,322,760
Feature pooling (G)66.900 ± 0.13088.158 ± 0.2881,882,760

Removing phase reduces mean Top-1 by 1.277 percentage points at the same parameter count (B − P). Retaining directional energies improves mean Top-1 by 1.224 points (B − A), with an accompanying head-capacity difference. The energy readout achieves similar accuracy to feature pooling with 17.0% fewer total parameters than G. Together, these results support the stated objective: pole-based spatial representations retain useful recognition information, and the energy descriptor provides a compact way to use it.

Thank you for the detailed comments on related work, motivation, and computational cost, which helped us sharpen the presentation of the contribution and its trade-offs.

Reviewer_uychOpenReview ↗
Rating
1(−1)
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a compact visual backbone that stores spatial context in learned complex Laplace poles. Each pole is propagated by four directional scans with learned damping and phase, and the classifier reads gain-normalized terminal energies rather than a pooled feature vector.

Critical strengths

A learned per-axis damping and phase set the multiplier on the previous state, reverse scans use conjugate coefficients, a factorized reader supplies the excitation, and a widely-linear block mixes the four memories with a local carry.

Spectral separation is argued for a fixed second-order stationary excitation, and the paper says intermediate-stage energies do not enter the classifier. The training recipe, the COCO head, and the FLOP counting rule are written down for the compared models.

The authors have conducted experiments on different datasets and vision tasks, and some empirical analyses are interesting.

Critical weaknesses

Parameter count overstates compactness. The larger model is listed at 19.51 GFLOPs, against about one GFLOP for the compact convolutional baseline it is compared with, and its batch throughput is lower.

The mechanism tests edit one trained checkpoint. They do not retrain a model without phase, without some scan directions, or with a different readout. Zeroing phase or damping at inference removes the transition the optimizer used, so a collapse does not show that a Laplace pole is a better bias than a real recurrence or a convolution at similar cost.

The separation argument assumes a stationary product-pole population. The trained network uses input-dependent features, nonlinear mixing, and stage-wise coarsening, which sit outside that hypothesis.

The smaller model does not clearly separate from a compact convolutional baseline on full ImageNet under the shared recipe. On the corruption benchmark, the larger model does not lead. Detection reads intermediate excitation fields, not the terminal energies used for classification, and one baseline needs a special precision and normalization fix.

The authors mainly evaluate performance under short training schedules and with relatively small models. However, this is far from sufficient for a vision backbone paper. Specifically, the evaluation should include models of different sizes (e.g., 20M, 50M, and 100M parameters), sufficient training epochs (e.g., the standard 300-epoch training on ImageNet-1K), and additional experiments (e.g., semantic segmentation and image generation).

Questions for authors

My questions are given in the Critical Weaknesses.

5 comments

Thank you for recognizing the interpretable architecture, the evaluation across multiple tasks, and the transparent experimental reporting.

W1: Parameter compactness and computational efficiency

Our initial focus was on evaluating Laplace-pole representations for visual recognition rather than aggressively compressing the surrounding layers. We appreciate your feedback, which prompted us to examine VisionAlphabet’s computational efficiency more closely and identify additional savings from its existing factorization. Instead of composing the learned factors into a dense convolution kernel, we now evaluate the same operation through channel projection followed by spatial filtering:

Up(x)=∑i,Δ(∑r=1RDp,r,ΔAp,r,i)Ei(x+Δ)=∑r=1R∑ΔDp,r,Δ(∑iAp,r,iEi(x+Δ)).U_p(x)=\sum_{i,\Delta}\left(\sum_{r=1}^{R}D_{p,r,\Delta}A_{p,r,i}\right)E_i(x+\Delta)=\sum_{r=1}^{R}\sum_{\Delta}D_{p,r,\Delta}\left(\sum_i A_{p,r,i}E_i(x+\Delta)\right).

This algebraic reassociation preserves the architecture, learned parameters, and checkpoint without approximation or retraining. It reduces whole-model FLOPs by 44.8–52.4%, with the following measured execution performance:

ModelParameters (M)Original GFLOPsOptimized GFLOPsReductionB1 latency (ms)B16 throughput (img/s)B16 peak (MiB)
K320.5731.6100.88944.8%0.32013,450138.9
K481.0053.1221.63847.5%0.34010,189141.5
K641.5635.1262.61748.9%0.3688,320159.1
K963.25311.7325.62852.0%0.4485,299198.2
K1285.08319.5089.28752.4%0.5144,095257.1

Figure 7 distinguishes arithmetic cost from measured execution performance. Under the reported measurement settings, optimized K128 requires more FLOPs than TinyViM-S but achieves lower batch-1 latency (0.514 vs. 2.90 ms); its batch-16 throughput remains lower (4,095 vs. 5,210 images/s). At similar batch-16 throughput, K96 achieves 72.35% Top-1 at 5,299 images/s, compared with 72.01% at 5,161 images/s for ParC-Net-S. These results show that the observed accuracy–execution trade-off depends on the operating condition, rather than FLOPs alone.

Figure 8 provides important context for these trade-offs. Unlike approaches primarily designed to minimize execution cost, our design prioritizes evaluating Laplace-pole spatial representations without aggressively compressing the surrounding fully connected and linear projections. K128 nevertheless achieves competitive recognition with fewer parameters, while pole scans themselves account for only 1.32% of total FLOPs. We would appreciate it if these results were assessed in light of this design scope, recognizing that most arithmetic arises in the surrounding feature transformations rather than pole propagation.

We have updated Table 4 and added Figures 7–8 in Appendix A, together with the equivalent Reader derivation and operation breakdown.

W2: Frozen interventions and component contributions

We agree that frozen-checkpoint interventions establish dependence on learned components, whereas retrained comparisons assess their contribution relative to alternative designs. We therefore trained the original VisionAlphabet-K64 (VA-K64) and three variants from scratch on ImageNet-1K under the same 100-epoch training protocol: P fixes the horizontal and vertical scan phases to zero throughout the backbone, retaining learned damping and the remaining complex-valued components; A retains all four scans but averages their gain-normalized energies across directions at the terminal readout; G replaces energy pooling with global spatial average pooling of the signed real and imaginary components of the gain-normalized states, concatenated across all four directions. All other architecture and training settings are unchanged. Each variant was trained with three seeds (0, 1, and 2); the table reports mean ± SD across these runs.

VariantTop-1 mean ± SD (%)Top-5 mean ± SD (%)Parameters
Original VA-K64 (B)66.871 ± 0.09588.105 ± 0.0041,562,760
Phase-free scan (P)65.594 ± 0.01487.180 ± 0.0911,562,760
Direction-averaged energy (A)65.647 ± 0.00787.409 ± 0.1481,322,760
Feature pooling (G)66.900 ± 0.13088.158 ± 0.2881,882,760

These results support the design rationale motivated by our spectral analysis: learned phase controls the spectral selectivity of pole-energy measurements, while the energy readout retains those measurements as compact recognition features. The phase-free comparison shows that this learned selectivity contributes to recognition after retraining. The pooling comparison shows that retaining pole energies provides similar observed accuracy with 17.0% fewer total parameters than G, supporting their utility as a compact descriptor. Thus, the retrained ablations provide empirical support for the theory-motivated design, complementing the original frozen-checkpoint interventions.

Retrained convolution control. To address the comparison with a convolutional alternative, we also trained C on ImageNet-1K under the same 100-epoch protocol with seeds 0, 1, and 2. C replaces all Laplace scan operators with four mode-wise complex depthwise 3×3 convolution outputs at each site, preserving the complex mode widths, four-path interface, and remaining backbone modules. This changes spatial propagation to local convolution without introducing a channel bottleneck.

VariantTop-1 mean ± SD (%)Parameters
Original VA-K64 (B)66.871 ± 0.0951,562,760
Convolution control (C)58.578 ± 0.0381,628,040

C uses 4.18% more parameters, yet its mean Top-1 is 8.293 percentage points lower (B − C). This supports the utility of the Laplace-based propagation relative to this retrained local-convolution alternative, beyond the dependence established by frozen-checkpoint interventions.

W3: Scope of the theory and its connection to the architecture

The theory analyzes how pole scans and energy measurements characterize the spatial structure of input features under the stated assumptions. Within the backbone, nonlinear mixing, residual carry paths, and coarsening construct features across resolutions; the theory concerns the pole-energy measurement applied to those features, rather than every preceding transformation. To retain this connection explicitly, the terminal readout forms an energy descriptor directly from the directional pole states, without intervening GatedWL mixing, and passes it to an affine classifier. This follows the theoretical rationale that distinct population energy descriptors can be separated by affine logits.

W4: Task-specific results and the purpose of dense prediction evaluation

Thank you for highlighting where the interpretation of our results could be clearer.

We would appreciate considering the overall pattern across ImageNet-1K, low-data recognition, Food-101, and COCO, where VisionAlphabet shows recurring gains against comparably sized baselines. Several of these gains are substantial, supporting the representation’s broader utility beyond the modest full-data accuracy difference noted here. Although the larger variant does not rank first on ImageNet-C, its performance is close to the leading baselines. This indicates competitive performance under corruption, complementing the gains observed in clean-image recognition.

The COCO experiment evaluates whether the learned spatial backbone transfers to detection. It uses intermediate spatial features for this purpose. The results show that ImageNet-1K pretraining with the energy-based classification readout produces intermediate representations useful for detection transfer. ParC-Net-S required precision and normalization adjustments for stable training; these baseline-specific settings are disclosed in the appendix.

W5: Training duration, model scale, and additional tasks

Thank you for these suggestions. Our shared 100-epoch evaluation shows gains across compact-model classification, low-data recognition, and transfer, providing evidence for the representation’s utility across multiple settings. Limited GPU resources led us to prioritize this breadth of controlled evaluation over longer schedules and larger-scale training. The results motivate further scaling studies, while additional tasks would broaden the assessment of the proposed representation.

Thank you for the thoughtful feedback on the theoretical scope and experimental evaluation; it helped us explain more precisely what the analysis and results establish.

I thank the authors for their response. Unfortunately, my concerns have not been addressed. First, the comparisons are quite limited. A large number of lightweight backbones and models of comparable size are missing, such as the EfficientFormer series and MetaFormer. In addition, as mentioned in my original comment, a paper on a general backbone requires comprehensive evaluation across diverse tasks. Although the authors provide detection evaluations, I do not see experiments with other detection frameworks, such as Mask R-CNN 3x, RetinaNet, and Cascade R-CNN. Note that these are widely used in backbone-related papers. Third, the authors train on ImageNet-1k for only 100 epochs, which differs substantially from the commonly used 300-epoch training schedule. Therefore, these results are also unconvincing to me. Finally, the scale is also quite concerning. The current results do not demonstrate whether this work can be scaled up or extended by follow-up work. Overall, my impression is that the authors do not have a strong understanding of vision backbones. I do not think the paper in its current form is suitable for publication at a top-tier conference like ICLR.

Reviewer_tmcTOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes VisionAlphabet, a compact visual backbone that models spatial context using learned complex Laplace-pole states. It uses rank-factorized local readers to generate pole-specific excitations, four-directional spatial scans to propagate them, and gain-normalized pole energies for classification. The paper also provides a spectral analysis and uses frozen-checkpoint interventions to examine the role of learned memory. Experiments on ImageNet-1K, Food-101, and COCO show competitive classification and dense prediction results with relatively few parameters.

Critical strengths
  1. The method combines complex-valued memory states, four-directional scanning, and energy-based readout in a unified architecture. The explicit damping and phase parameters provide a clear interpretation of spatial information propagation (Section 3, Figure 1).

  2. The paper evaluates the method on both classification and dense prediction tasks. VisionAlphabet-K128 achieves 75.02% Top-1 accuracy on ImageNet-1K with 5.08M parameters, outperforming TinyViM-S with fewer parameters (Table 1). The frozen-checkpoint experiments also show that learned memory and phase affect model predictions (Figures 3–5).

Critical weaknesses
  1. The contribution of individual components is not sufficiently isolated. Section 6 mainly uses frozen-checkpoint interventions, including memory resets, phase removal, and pole neutralization. These experiments show that the trained model depends on these components, but not whether they outperform simpler alternatives after retraining. Ablations with retrained phase-free or real-valued recurrences and conventional feature-pooling heads would help establish their individual contributions.

  2. The justification for separate directional-energy features is incomplete. In Appendix B (Eqs. 8–10), reverse scans use conjugate transition coefficients, giving forward and reverse scans identical squared frequency responses under the stationary assumptions in Section 4. Thus, the four directions of a given pole have identical population energies in this setting. Finite boundaries and nonstationary features may produce differences in practice, but their discriminative value is not demonstrated. A comparison with direction-averaged or single-direction energy readout is needed to justify the four-direction descriptor.

  3. The parameter savings come with substantial computational overhead. VisionAlphabet-K128 requires 19.51G FLOPs, compared with 1.92G for TinyViM-S (Table 4), for a 0.60 percentage-point improvement in ImageNet-1K Top-1 accuracy (Table 1). Its batch-16 throughput is also lower (2,665 vs. 5,210 images/s), although its batch-1 latency is better. The parameter advantage therefore does not consistently translate into computational efficiency. Comparisons at matched FLOP or throughput budgets would better establish the practical trade-off.

Questions for authors

please refer to critical weaknesses

2 comments

Thank you for recognizing the unified architecture, the interpretation of damping and phase, and the evaluation across classification. Your comments helped us strengthen the evidence for individual design choices and clarify the relationship between the population analysis and the directional readout.

W1: Component contributions through retrained ablations

We agree that frozen-checkpoint interventions establish dependence on learned components, whereas retrained comparisons assess their contribution relative to alternative designs. We therefore trained the original VisionAlphabet-K64 (VA-K64) and three variants from scratch on ImageNet-1K under the same 100-epoch training protocol: P fixes the horizontal and vertical scan phases to zero throughout the backbone, retaining learned damping and the remaining complex-valued components; A retains all four scans but averages their gain-normalized energies across directions at the terminal readout; G replaces energy pooling with global spatial average pooling of the signed real and imaginary components of the gain-normalized states, concatenated across all four directions. All other architecture and training settings are unchanged. Each variant was trained with three seeds (0, 1, and 2); the table reports mean ± SD across these runs.

VariantTop-1 mean ± SD (%)Top-5 mean ± SD (%)Parameters
Original VA-K64 (B)66.871 ± 0.09588.105 ± 0.0041,562,760
Phase-free scan (P)65.594 ± 0.01487.180 ± 0.0911,562,760
Direction-averaged energy (A)65.647 ± 0.00787.409 ± 0.1481,322,760
Feature pooling (G)66.900 ± 0.13088.158 ± 0.2881,882,760

These results support the design rationale motivated by our spectral analysis: learned phase controls the spectral selectivity of pole-energy measurements, while the energy readout retains those measurements as compact recognition features. The phase-free comparison shows that this learned selectivity contributes to recognition after retraining. The pooling comparison shows that retaining pole energies provides similar observed accuracy with 17.0% fewer total parameters than G, supporting their utility as a compact descriptor. Thus, the retrained ablations provide empirical support for the theory-motivated design, complementing the original frozen-checkpoint interventions.

W2: Population-energy equality and finite-image directional readout

We sincerely appreciate your careful examination of the theory and readout design. We agree that the stationary population analysis alone does not justify retaining separate directional-energy coordinates.

For each spatial axis, the conjugate reverse construction gives H−(ω)=H+(ω)‾H_-(\omega)=\overline{H_+(\omega)}, hence ∣H−(ω)∣2=∣H+(ω)∣2|H_-(\omega)|^2=|H_+(\omega)|^2. Applying this relation along both axes yields equal directional population energies under the stated stationary, infinite-domain assumptions.

However, the classifier uses directional energies computed over each finite image. The identity above equates the energy responses of the infinite-domain filters; it does not establish equality after direction-dependent boundary truncation and position-dependent gain normalization. Consequently, the finite-image energy sums used by the classifier need not coincide, even though their idealized stationary population counterparts do.

Intuitively, scans starting from different boundaries accumulate different spatial contexts before their energies are pooled over the image. Retaining separate coordinates allows the classifier to weight these directional energies individually, whereas averaging removes that distinction.

We directly tested this choice through the retrained direction-averaged variant in W1. Across seeds 0, 1, and 2, the original readout achieves 1.224 percentage points higher mean Top-1 accuracy (B − A).

W3: Parameter compactness and computational cost

Our initial focus was on evaluating Laplace-pole representations for visual recognition rather than aggressively compressing the surrounding layers. We appreciate your feedback, which prompted us to examine VisionAlphabet’s computational efficiency more closely and identify additional savings from its existing factorization. Instead of composing the learned factors into a dense convolution kernel, we now evaluate the same operation through channel projection followed by spatial filtering:

Up(x)=∑i,Δ(∑r=1RDp,r,ΔAp,r,i)Ei(x+Δ)=∑r=1R∑ΔDp,r,Δ(∑iAp,r,iEi(x+Δ)).U_p(x)=\sum_{i,\Delta}\left(\sum_{r=1}^{R}D_{p,r,\Delta}A_{p,r,i}\right)E_i(x+\Delta)=\sum_{r=1}^{R}\sum_{\Delta}D_{p,r,\Delta}\left(\sum_i A_{p,r,i}E_i(x+\Delta)\right).

This algebraic reassociation preserves the architecture, learned parameters, and checkpoint without approximation or retraining. It reduces whole-model FLOPs by 44.8–52.4%, with the following measured execution performance:

ModelParameters (M)Original GFLOPsOptimized GFLOPsReductionB1 latency (ms)B16 throughput (img/s)B16 peak (MiB)
K320.5731.6100.88944.8%0.32013,450138.9
K481.0053.1221.63847.5%0.34010,189141.5
K641.5635.1262.61748.9%0.3688,320159.1
K963.25311.7325.62852.0%0.4485,299198.2
K1285.08319.5089.28752.4%0.5144,095257.1

Figure 7 distinguishes arithmetic cost from measured execution performance. Under the reported measurement settings, optimized K128 requires more FLOPs than TinyViM-S but achieves lower batch-1 latency (0.514 vs. 2.90 ms); its batch-16 throughput remains lower (4,095 vs. 5,210 images/s). At similar batch-16 throughput, K96 achieves 72.35% Top-1 at 5,299 images/s, compared with 72.01% at 5,161 images/s for ParC-Net-S. These results show that the observed accuracy–execution trade-off depends on the operating condition, rather than FLOPs alone.

Figure 8 provides important context for these trade-offs. Unlike approaches primarily designed to minimize execution cost, our design prioritizes evaluating Laplace-pole spatial representations without aggressively compressing the surrounding fully connected and linear projections. K128 nevertheless achieves competitive recognition with fewer parameters, while pole scans themselves account for only 1.32% of total FLOPs. We would appreciate it if these results were assessed in light of this design scope, recognizing that most arithmetic arises in the surrounding feature transformations rather than pole propagation.

We have updated Table 4 and added Figures 7–8 in Appendix A, together with the equivalent Reader derivation and operation breakdown.

Thank you again for the careful reading and constructive suggestions, which helped us clarify the design rationale and strengthen its empirical support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.