acceptodds
Under review as a conference paper at ICLR 2027

Linear-Time Global Visual Modeling without Explicit Attention

Abstract

Existing research largely attributes the global sequence modeling capability of Transformers to the explicit computation of attention weights, a process that inherently incurs quadratic computational complexity. In this work, we offer a different perspective: we demonstrate that attention can be mathematically reframed as a Multi-Layer Perceptron (MLP) equipped with dynamically predicted parameters. Through this lens, we explain attention's global modeling power not as explicit token-wise aggregation, but as an implicit process where dynamically generated parameters act as a compressed representation of the global context. Inspired by this insight, we investigate a fundamental question: can we achieve Transformer-level sequence global modeling entirely through dynamic parameterization while maintaining linear complexity, effectively replacing explicit attention? To explore this, we design various dynamic parameter prediction strategies and integrate them into standard network layers. Extensive empirical studies on vision models demonstrate that dynamic parameterization can indeed serve as a highly effective, linear-complexity alternative to explicit attention, offering a promising paradigm for efficient sequence modeling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
attention, vision transformer, dynamic network

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oWs7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WeightFormer and H-WeightFormer, which replace explicit self-attention with input-conditioned dynamic parameterization for global visual modeling. The authors reinterpret attention as a dynamic MLP whose parameters are generated from the input sequence, and use this perspective to design dynamic linear layers and dynamic depthwise convolutions. The proposed methods predict weights from compressed global representations, including pooled features, feature correlations, and spatially adaptive descriptors, avoiding explicit construction of an N × N attention matrix. Experiments on ImageNet-1K classification, COCO object detection and instance segmentation, ADE20K semantic segmentation, and class-conditional ImageNet generation show competitive performance against several Transformer, ConvNet, and state-space-model baselines. The results suggest that input-conditioned parameterization is a promising alternative for efficient global visual modeling, although the novelty of the formulation and the evidence supporting its generality and efficiency warrant further scrutiny.

Critical strengths

The paper reframes attention as a dynamic MLP with input-conditioned parameters and uses this perspective to explore a family of dynamic linear and depthwise convolution operators. This provides a coherent design framework for investigating global context modeling without explicitly constructing a dense attention matrix. The proposed architectures are evaluated on image classification, object detection, instance segmentation, semantic segmentation, and class-conditional image generation. The results are encouraging: for example, H-WeightFormer-T achieves 83.3% ImageNet top-1 accuracy at 4.8G FLOPs, compared with 82.5% for VMamba-T at 4.9G FLOPs. These results suggest that the proposed approach merits further investigation as an efficient visual backbone.

Critical weaknesses

Rewriting attention as a dynamic MLP is mathematically valid, but appears to be primarily an alternative interpretation of the standard attention computation. The proposed replacement then uses compressed global descriptors to generate a fixed-size set of parameters, which is a different mechanism from attention's token-dependent pairwise interactions. The paper does not sufficiently characterize when this compression preserves useful global information, what expressive capabilities may be lost, or how the approach differs fundamentally from existing hypernetworks and dynamic convolution methods. Consequently, the central claim that dynamic parameterization can substitute for attention is supported mainly by empirical performance rather than a deeper theoretical account.

Although the proposed operators avoid an explicit N × N attention matrix, linear complexity alone does not establish superior practical efficiency. For example, H-WeightFormer-B reports 25.3G FLOPs, compared with 15.4G for VMamba-B, while WeightFormer-B reports 17.7G FLOPs compared with 17.2G for Vim-B. The efficiency advantage is therefore not uniform across model scales and baselines. In addition, the effective receptive field visualization demonstrates broad input sensitivity, but does not establish that the model captures the same kinds of task-relevant long-range dependencies as attention. More systematic comparisons of actual throughput, memory consumption, and performance under matched computational budgets would strengthen the main claims.

Questions for authors

What is the precise technical distinction between the proposed dynamic parameterization and existing hypernetworks, dynamic filter networks, and conditional convolutions? Beyond the algebraic reinterpretation of attention as a dynamic MLP, is there a specific theoretical property or capability established by the proposed framework that prior dynamic networks do not provide?

The proposed operators generate parameters from compressed global descriptors, including pooled features and feature correlations. How much task-relevant spatial information is retained by this compression? Can the authors provide controlled experiments that distinguish genuine long-range relational reasoning from global conditioning or increased effective receptive fields alone?

Can the authors provide more systematic comparisons of end-to-end inference latency, peak GPU memory, and accuracy under matched parameter, FLOP, and input-resolution settings? In particular, how does the method scale with increasing image resolution, and does its practical advantage persist across model sizes and hardware configurations?

Since the proposed architectures also incorporate depthwise convolutions, conditional positional encoding, and sparsely placed dynamic blocks, can the authors isolate the contribution of dynamic weight prediction through controlled ablations against otherwise identical static backbones and strong convolutional baselines?

4 comments

We thank all reviewers for their questions and constructive comments. Below we provide direct responses to each question raised.

1. Relationship to existing dynamic networks

  1. WeightFormer's innovation lies in two specific generators for global visual modeling. Bilateral predicts channel weights from nonlinear token co-occurrences. Spatially Adaptive DWC maps an ordered 3×33\times3 pooled grid to depthwise kernels. These descriptors capture information beyond global channel means.

  2. The differences concern the concrete generator designs. Dynamic Filter Networks provides a general filter-generation framework. HyperNetworks uses layer embeddings or recurrent context. CondConv mixes expert kernels through GAP-based routing. WeightFormer develops nonlinear correlation and spatial-grid parameterizations.

  3. These generators produce fixed-size weight updates with O(N)O(N) cost at fixed channel widths and kernel size. Bilateral retains its Q(K⊤V)Q(K^\top V) factorization, including Equation (15)'s column scaling. The dynamic depthwise branch uses global context to condition local spatial filtering.

2. Compression and nonlocal relations

  1. The two generators retain complementary information. Bilateral preserves nonlinear feature co-occurrences and is invariant to pooled-token order. Spatially Adaptive DWC retains a coarse 3×33\times3 layout. Pooling removes spatial detail within each region.

  2. We tested which compressed information matters in frozen models. On 128 ImageNet validation images at 224×224224\times224, we modified only generator inputs and kept the feature maps transformed by the generated weights intact. Permutations preserve channel means, and permutation results average three seeds.

    Generator inputTiny Top-1 (%)Small Top-1 (%)
    Normal73.4478.91
    Bilateral with permuted pooled tokens73.4478.91
    Bilateral with broadcast channel means63.2867.97
    DWC with permuted 3×33\times3 regions72.4076.82
    DWC with broadcast channel means72.6678.12
  3. We also tested a specific nonlocal relation with matched input means. A width-32 Bilateral layer classifies whether a query and a marked target at least eight tokens away share a binary attribute. Paired examples swap target/decoy attributes, flipping the label while preserving all input channel means and the query. Only the query output is classified. Bilateral and the mean-only generator each have 3,218 parameters.

  4. After 2,000 training steps, Bilateral reached 100% on 8,192 held-out examples in all three training seeds. The static transform and mean-only generator each reached 50%. Together with the frozen-model interventions, this supports the use of information beyond channel means and demonstrates this specific nonlocal relation through generated weights.

3. Controlled comparison with the static backbone

  1. The static CNN and dynamic variants in Table 2 share the same backbone layout. Both contain 3×33\times3 depthwise convolutions and use no separate positional-encoding modules. Spatial structure is handled by the convolutional layers.

  2. Table 2 then varies dynamic weight prediction within that common backbone. Accuracy rises from 73.3% for the static CNN to 76.4% with Bilateral Linear 1, 74.6% with Adaptive DWC and 76.8% with both. The separate channel, depthwise and combined ablations show their contributions with the same DWC and positional-encoding settings.

  3. Table 3 compares our models with strong CNN baselines, including ConvNeXt, FasterNet, and InceptionNeXt.

4. Inference efficiency

  1. The new comparison uses optimized attention under common settings. Original Figure 5 used naive DeiT attention. We now use verified PyTorch SDPA FlashAttention for DeiT on an H100 (80 GB), with batch 64 and BF16 autocast over FP32 parameters. DeiT represents an isotropic backbone with global self-attention and uses random weights for runtime measurements.

  2. We measure throughput, wall time and peak memory with the same protocol. After 10 warm-ups, throughput and wall time are medians of three CUDA-synchronized host-timed repetitions. Each repetition runs two forwards of the same synthetic GPU-resident batch, totaling 128 image-inferences. Transfers and preprocessing are excluded. Peak memory includes the model and whole batch.

    InputModelThroughput (img/s)Peak memory (GiB)Wall time (s)
    1024 × 1024DeiT-T (Flash)928.12.370.138
    1024 × 1024WeightFormer-T891.32.720.144
    2048 × 2048DeiT-T (Flash)100.79.321.272
    2048 × 2048WeightFormer-T232.410.620.551
    3072 × 3072DeiT-T (Flash)22.720.335.634
    3072 × 3072WeightFormer-T104.123.771.229
    1024 × 1024DeiT-S (Flash)490.73.870.261
    1024 × 1024WeightFormer-S511.64.790.250
    2048 × 2048DeiT-S (Flash)51.015.142.508
    2048 × 2048WeightFormer-S132.418.320.966
    3072 × 3072DeiT-S (Flash)11.533.9211.178
    3072 × 3072WeightFormer-S59.040.882.171
  3. The throughput advantage grows at high resolution. At 3072, WeightFormer achieves 4.58× and 5.15× DeiT throughput for Tiny and Small, respectively. Flash-DeiT uses less peak memory at every tested point.

5. Input resolutions in Table 3 and Figure 1

  1. Figure 1 uses the same results and input resolutions as main-paper Table 3. The input sizes are 448×448 for WeightFormer-B, 384×384 for H-WeightFormer-B, and 224×224 for all other entries.

  2. Within each family, B shares the Small architecture and parameter count.

  3. Other experimental tables use consistent input resolutions within each comparison. We will annotate Figure 1 and Table 3 and clarify this naming in Section 5.1.

Thank you for the detailed responses and additional experiments. The authors have addressed several of my concerns, particularly regarding the contribution of dynamic weight prediction through controlled ablations, the preservation of nonlocal information, and the inference efficiency comparison at different resolutions. The newly added experiments provide useful evidence supporting the effectiveness of the proposed dynamic parameterization. However, some fundamental concerns remain. In particular, the distinction from existing dynamic networks and hypernetwork-based approaches is still primarily established through specific architectural designs rather than a sufficiently clear conceptual or theoretical contribution. Moreover, the efficiency advantage is not consistent in terms of peak memory consumption, and the current comparisons do not fully establish a general practical advantage across different architectures and hardware settings. Overall, I appreciate the authors' efforts and believe the rebuttal has improved the paper, but the remaining concerns are not sufficiently resolved to change my original assessment. Therefore, I maintain my original score.

WeightFormer's distinct starting point is to view softmax attention as a dynamic MLP and use that view to design a visual backbone. For each query, K⊤K^\top and VV serve as two input-dependent weight matrices with softmax between them (Section 3.2). HyperNetworks studies weight generation as a form of relaxed sharing, Dynamic Filter Networks generates filters for adaptive transformations, and CondConv combines expert kernels per input. The attention viewpoint leads us to ask how generated weights can carry global context as the token count grows. WeightFormer uses weight updates of fixed shape for both channel mixing and spatial filtering. HyperNetworks · Dynamic Filter Networks · CondConv

This viewpoint guides what information each generator retains. Section 4 uses pooling and correlation as global summaries motivated by attention. Bilateral aggregates nonlinear feature correlations for channel mixing. Adaptive DWC retains the order of the pooled grid to set the weights at different kernel offsets. In our frozen-model diagnostic on 128 ImageNet images, permuting Bilateral tokens preserved accuracy. Permuting the DWC grid preserved channel means and reduced WeightFormer-S top-1 from 78.91% to 76.82%.

WeightFormer achieves higher reported ImageNet-1K top-1 than these prior dynamic models at similar compute or parameter counts. All results use 224×224224\times224 inputs and the training recipes of the respective papers. Compute refers to FLOPs or multiply-adds as reported by each source. Our results are from Table 3. CondConv, Table 1 · ODConv, Table 2 · Involution, Table 1 · DynaMixer, Table 2

Prior dynamic modelCompute (G)Top-1 (%)Our modelCompute (G)Top-1 (%)Gain (pp)
CondConv ResNet-50 (8 experts)4.2178.6WeightFormer-S4.481.3+2.7
ODConv ResNet-50 (4 kernels)4.0878.52WeightFormer-S4.481.3+2.8
RedNet-101 (Involution, 25.6M params)4.779.1WeightFormer-S (27M params)4.481.3+2.2
DynaMixer-M (57M params)17.083.7H-WeightFormer-S (58M params)8.784.3+0.6

When the token count increases 9×, runtime grows 8.6–8.7× for WeightFormer and 41–43× for FlashAttention DeiT. We measured this scaling from 1024×10241024\times1024 to 3072×30723072\times3072 at batch 64 on the same GPU. At 3072×30723072\times3072, WeightFormer-T and WeightFormer-S achieve 4.58× and 5.15× the throughput with 17% and 21% higher peak allocated memory, respectively. This scaling addresses the title's emphasis on linear time in the token count. We would appreciate your reconsidering the paper in light of the operator design and its measured scaling against optimized softmax attention.

Reviewer_tXRnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether global visual modeling can be achieved without explicitly computing attention. The authors first reinterpret self-attention as a dynamic MLP with input-dependent weights, and use this observation to motivate dynamic linear and depthwise convolution layers. They explore several ways of generating these weights and propose a Bilateral Activation mechanism for the linear layers, along with a spatially adaptive approach for depthwise convolutions. These components are used to build WeightFormer and its hierarchical variant, H-WeightFormer, with linear computational complexity in the number of tokens.

The authors evaluate their models on image classification, object detection, semantic segmentation, and image generation. The results show competitive performance against several Transformer and CNN baselines. They also report better throughput and lower GPU memory usage compared to DeiT at higher image resolutions. The paper includes ablation studies on different weight prediction strategies and the number of dynamic blocks used in the network

Critical strengths
  1. The main strength of the paper is that the proposed method achieves competitive results with relatively low computational cost. For example, H-WeightFormer-T achieves 83.3% on ImageNet with a similar computational budget to VMamba-T, which achieves 82.5%. The approach is also fairly simple and maintains linear complexity with respect to the number of tokens. This make it a potentially useful alternative to explicit attention in vision models.

  2. I also find the experimental study of different dynamic weight prediction strategies useful. The authors explore several designs for both linear and depthwise convolution layers, and the ablations help understand which components contribute to the performance. The results on detection, segmentation, and image generation are also encouraging. Although stronger comparisons are still needed, these experiments suggest that the approach is not limited to image classification.

Critical weaknesses
  1. My main concern is the novelty of the proposed method. The interpretation of attention as a dynamic MLP is interesting, but it is mostly a mathematical reformulation. More importantly, the Bilateral Activation mechanism seems closely related to existing linear attention and fast-weight approaches. In fact, Eq. 9 can be rearranged as (XΔW(X)=Q(K⊤V)W4)(X\Delta W(X)=Q(K^\top V)W_4), before applying the update normalization. This is essentially an unnormalized linear attention computation over pooled tokens. I think the paper needs to better explain what is fundamentally different from these existing methods. This is important since the paper positions dynamic parameterization as a new alternative to attention.

  2. The experimental comparisons are not fully convincing to me. While the overall results are good, the Bilateral Activation only gives around 0.2% improvement over simpler dynamic weight prediction strategies, and there is no variance reported across runs. I would like to see comparisons with simpler input-conditioned low-rank updates to understand whether the proposed mechanism is really necessary. Also, the stronger H-WeightFormer models are not evaluated on detection or segmentation, and the efficiency comparisons mainly focus on DeiT rather than optimized attention or other efficient architectures. The lack of these comparisons make it difficult to judge how much improvement actually comes from the proposed method.

Questions for authors
  1. My main question is about the connection between Bilateral Activation and existing linear attention or fast-weight methods. Eq. 9 can be rearranged as XΔW(X)=Q(K⊤V)W4X\Delta W(X)=Q(K^\top V)W_4, before the update normalization, which looks similar to unnormalized linear attention over pooled tokens. Could the authors clarify what the main difference is between their formulation and these existing approaches? I think this is important for understanding the actual novelty of the method.

  2. In Table 2, Bilateral Activation improves accuracy by only around 0.2% compared to some simpler dynamic weight prediction strategies. Is this improvement consistent across different runs? I would also like to understand what specific benefit the bilateral formulation provides over the simpler alternatives already evaluated in the paper. The differences seem relatively small, so it is not entirely clear how important this particular design choice is.

  3. Regarding the efficiency results in Figure 5, could the authors clarify which attention implementation was used for DeiT? In particular, was an optimized implementation such as FlashAttention considered, and were the throughput and memory measurements performed under the same precision and batch size? This would help me better understand how much of the reported efficiency improvement comes from the proposed architecture rather than differences in implementation.

4 comments

We thank all reviewers for their questions and constructive comments. Below we provide direct responses to each question raised.

1. Connection to linear attention and fast weights

  1. Dynamic parameterization includes linear attention. Softmax attention can be viewed as a two-layer dynamic MLP whose weights come from keys and values. In kernel feature space, normalized linear attention applies the dynamic matrix M(X)=ϕ(K)⊤VM(X)=\phi(K)^\top V to ϕ(qi)\phi(q_i) and divides by ϕ(qi)∑jϕ(kj)⊤\phi(q_i)\sum_j\phi(k_j)^\top. This is a dynamic linear layer followed by a query-dependent denominator. We study the broader design space that contains these operators.

  2. Dynamic DWC provides an example beyond the standard global linear-attention form. A pooled spatial grid predicts an image-specific 3×33\times3 kernel for each channel. That kernel is shared across positions and applied to local neighborhoods. Global context therefore controls local spatial filtering, extending the design beyond global QKV aggregation.

  3. Equations (9) and (15) have a precise connection to linear attention. For pooled tokens XpX_p and σ=SiLU⁡\sigma=\operatorname{SiLU}, define Q=XW1Q=XW_1, K=σ(XpW2⊤)K=\sigma(X_pW_2^\top) and V=σ(XpW3)V=\sigma(X_pW_3). Then

    D_{jj}=(1+\|\Delta W_{:,j}\|_2)^{-1}.$$ Equation (9) shares the unnormalized QKV product and its [fast-weight interpretation](https://proceedings.mlr.press/v139/schlag21a.html). Pooling compresses keys and values while queries retain all original tokens. Equation (15) preserves this factorization and scales weight columns, rather than using the query-dependent denominator above. The complete layer applies $W_0+\widehat{\Delta W}(X)$ and uses no head partitioning.

2. The 0.2-point gain and simpler generators

  1. Bilateral changes when nonlinear feature interactions are formed. It transforms each pooled token before global aggregation. GAP uses a mean descriptor, while Linear, Nonlinear and Deep predict weights from the already aggregated Xp⊤XpX_p^\top X_p. Bilateral can therefore use nonlinear feature co-occurrences beyond this raw correlation matrix, with linear token complexity and the same reported parameter/FLOP budget as Deep (6.9M/1.2G).

  2. Table 2 reports single ImageNet runs. Bilateral achieves 76.4% with 6.9M parameters, compared with GAP's 74.7% with 10.1M and pooled Linear/Deep's 76.2%.

  3. We added a controlled three-seed generator comparison on Tiny ImageNet. Following exploration on a 90k/10k split of training images, we evaluated a shared width-64, 12-block backbone with patch size 4. Each generator was trained from scratch for 50 epochs on all 100,000 training images, using three fresh paired seeds, AdamW, batch 256 and 64×64 inputs. Blocks 2, 5, 8 and 11 use dynamic channel weights, while all depthwise convolutions remain static. Normalization and initial backbone functions are identical. Low-rank uses Adiag⁡(g(GAP⁡(Xp)))BA\operatorname{diag}(g(\operatorname{GAP}(X_p)))B with rank 16 and a two-layer SiLU gate. The table reports final accuracy on the 10,000-image official validation split as mean ± sample SD.

    GeneratorParametersValidation Top-1 (%)
    Linear505,09647.23 ± 0.94
    Deep513,28847.67 ± 0.53
    Bilateral513,28848.37 ± 0.40
    Low-rank513,41648.26 ± 0.31
  4. Bilateral exceeds Linear and Deep by 1.14 and 0.70 points on average, with gains in all three paired seeds. Its 0.11-point mean gain over Low-rank varies in sign across seeds.

3. H-WeightFormer on downstream tasks

  1. The existing detection and segmentation experiments evaluate isotropic WeightFormer.

  2. Given the available time and computational resources, we focused these experiments on that backbone. Evaluation of H-WeightFormer on these tasks remains future work.

4. Inference efficiency

  1. The new comparison uses optimized attention under common settings. Original Figure 5 used naive DeiT attention. We now use verified PyTorch SDPA FlashAttention for DeiT on an H100 (80 GB), with batch 64 and BF16 autocast over FP32 parameters. DeiT represents an isotropic backbone with global self-attention and uses random weights for runtime measurements.

  2. We measure throughput, wall time and peak memory with the same protocol. After 10 warm-ups, throughput and wall time are medians of three CUDA-synchronized host-timed repetitions. Each repetition runs two forwards of the same synthetic GPU-resident batch, totaling 128 image-inferences. Transfers and preprocessing are excluded. Peak memory includes the model and whole batch.

    InputModelThroughput (img/s)Peak memory (GiB)Wall time (s)
    1024 × 1024DeiT-T (Flash)928.12.370.138
    1024 × 1024WeightFormer-T891.32.720.144
    2048 × 2048DeiT-T (Flash)100.79.321.272
    2048 × 2048WeightFormer-T232.410.620.551
    3072 × 3072DeiT-T (Flash)22.720.335.634
    3072 × 3072WeightFormer-T104.123.771.229
    1024 × 1024DeiT-S (Flash)490.73.870.261
    1024 × 1024WeightFormer-S511.64.790.250
    2048 × 2048DeiT-S (Flash)51.015.142.508
    2048 × 2048WeightFormer-S132.418.320.966
    3072 × 3072DeiT-S (Flash)11.533.9211.178
    3072 × 3072WeightFormer-S59.040.882.171
  3. The throughput advantage grows at high resolution. At 3072, WeightFormer achieves 4.58× and 5.15× DeiT throughput for Tiny and Small, respectively. Flash-DeiT uses less peak memory at every tested point.

5. Input resolutions in Table 3 and Figure 1

  1. Figure 1 uses the same results and input resolutions as main-paper Table 3. The input sizes are 448×448 for WeightFormer-B, 384×384 for H-WeightFormer-B, and 224×224 for all other entries.

  2. Within each family, B shares the Small architecture and parameter count.

  3. Other experimental tables use consistent input resolutions within each comparison. We will annotate Figure 1 and Table 3 and clarify this naming in Section 5.1.

Thank you for the additional experiments. Your response confirms my observation that the dynamic output from Eq. (9) admits a pooled, unnormalized QKV factorization, preserved after normalization. This confirms the connection to linear attention underlying my novelty concern, but does not resolve that concern.

The three-seed results favor Bilateral over Linear and Deep, but show no consistent advantage over the simpler low-rank generator; the original ImageNet gain remains based on single runs. The FlashAttention comparison supports high-resolution throughput gains, but shows higher peak memory.

I therefore maintain my rating.

Vanilla linear attention can lose substantial information when it compresses many tokens into a narrow feature space. The rank of the token mixing matrix in each head is bounded by its feature width. Standard query normalization also cancels the magnitude of the query feature for fixed keys and values. Eq. (9) shares the pooled, unnormalized QKV product. Eq. (15) scales the generated weight columns, so query magnitude can still affect the readout when the generated weights are fixed. Han et al., §4.2 · Fan et al., §3.2

WeightFormer-T outperforms the vanilla linear attention baseline by 5.8 top-1 points at the same reported 1.1G FLOPs. These are reported ImageNet-1K results at 224×224224\times224 from our Table 3 and the published DeiT-T comparison. FLatten, Table 4

ModelParametersFLOPsTop-1 (%)
WeightFormer-T7M1.1G76.3
FLatten-DeiT-T with focused linear attention and DWC6.1M1.1G74.1
DeiT-T with softmax5.7M1.2G72.2
DeiT-T with vanilla linear attention5.7M1.1G70.5

Adding Adaptive DWC to Bilateral raises top-1 from 76.4% to 76.8% in Table 2. Adaptive DWC generates depthwise kernels from an ordered 3×33\times3 image summary and mixes nearby features directly. This suggests pairing global linear attention with image-conditioned local filters so local detail has a separate path. The three-seed study keeps DWC static and shows that low-rank is competitive for channel mixing.

At 3072×30723072\times3072, WeightFormer-T and WeightFormer-S achieve 4.58× and 5.15× the throughput of FlashAttention DeiT. Peak allocated memory is 17% and 21% higher, respectively. This throughput gain addresses the title's emphasis on linear time in the token count. We would appreciate your reconsidering the full architecture in light of its accuracy and measured throughput.

Reviewer_9hggOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies global visual modeling through input-conditioned network weights. Its starting observation is that a softmax attention head can be written as a two-layer dynamic MLP: the key and value matrices become the two weight matrices, while softmax supplies the intermediate nonlinearity. The proposed architectures then replace this sequence-dependent hidden width with generators of fixed-size channel-mixing matrices and depthwise kernels. The design study considers pooled descriptors, channel correlations, several nonlinear predictors, and spatially adaptive kernel prediction. The preferred combination uses bilateral activation for the first channel-MLP projection and adaptive pooling for dynamic depthwise convolution, inserted periodically among static blocks.

Critical strengths
  1. Table 2 compares multiple generators, the placement of dynamic channel transforms, dynamic depthwise convolution, and their combination. Reporting parameters, FLOPs, throughput, and accuracy together makes the practical compromises visible.

  2. In Table 2, the combined dynamic design reaches 76.8% versus 73.3% for the static baseline. H-WeightFormer-T obtains 83.3% at 4.8G FLOPs in Table 3. The segmentation improvements over the corresponding DeiT backbones are also encouraging.

  3. Detection, segmentation, and generation offer complementary evidence that the proposed modules are usable beyond a classifier.

Critical weaknesses
  1. calling the keys and values MLP weights does not remove the computation of each query's softmax vector or its weighted combination of values. The hidden units still correspond to individual tokens. Consequently, the statement at the end of that section that global dependencies are integrated without constructing or applying attention weights needs qualification: avoiding storage of a full matrix is different from eliminating those interactions.

  2. Ignoring the subsequent update normalization, Equation (9) gives a dynamic output contribution of the form X W1 SiLU(W2 Xp^T) SiLU(Xp W3) W4, where Xp is the pooled token map. This can be regrouped into a projected query, a nonlinear key transpose, and nonlinear values: an unnormalized attention-style product with pooled keys and values.This does not invalidate the method, but it makes the sharp division between dynamic parameterization and feature recombination less convincing.

  3. Table 8 shows that WeightFormer-S and WeightFormer-B have the same width, depth, and 27M parameter count, i.e., the change is from 224 to 448 input resolution. Table 9 similarly makes H-WeightFormer-B the higher-resolution version of the Small architecture, with the same 58M parameters. These are legitimate operating points, but Table 3 and Figure 1 do not display input resolution, and Section 5.1's language about matched scale can suggest architecture scaling. Please annotate resolution and distinguish the benefit of more image tokens from a larger model. A comparison at a common resolution would make the efficiency and accuracy claims easier to interpret.

Questions for authors
  1. Can you characterize the relationship between the bilateral linear update and a reassociated query-key-value product, including pooling and Equation (15)? Which part of the proposed operator provides the principal distinction?

  2. What happens with a parameter- and normalization-matched static backbone, and when generated updates are replaced by averages or updates from another image? These would be particularly informative tests of input-specific global conditioning.

2 comments

We thank all reviewers for their questions and constructive comments. Below we provide direct responses to each question raised.

1. Attention as a dynamic MLP

  1. The reformulation preserves the original attention computation. Softmax and weighted value aggregation remain, together with quadratic complexity.

  2. Its role is to motivate fixed-shape parameter generators in Section 4. These generators give WeightFormer linear complexity in token count. We will make this distinction explicit in Sections 3.2–3.3.

2. Relationship to linear attention

  1. Dynamic parameterization includes linear attention. Softmax attention can be viewed as a two-layer dynamic MLP whose weights come from keys and values. In kernel feature space, normalized linear attention applies the dynamic matrix M(X)=ϕ(K)⊤VM(X)=\phi(K)^\top V to ϕ(qi)\phi(q_i) and divides by ϕ(qi)∑jϕ(kj)⊤\phi(q_i)\sum_j\phi(k_j)^\top. This is a dynamic linear layer followed by a query-dependent denominator. We study the broader design space that contains these operators.

  2. Dynamic DWC provides an example beyond the standard global linear-attention form. A pooled spatial grid predicts an image-specific 3×33\times3 kernel for each channel. That kernel is shared across positions and applied to local neighborhoods. Global context therefore controls local spatial filtering, extending the design beyond global QKV aggregation.

  3. Equations (9) and (15) have a precise connection to linear attention. For pooled tokens XpX_p and σ=SiLU⁡\sigma=\operatorname{SiLU}, define Q=XW1Q=XW_1, K=σ(XpW2⊤)K=\sigma(X_pW_2^\top) and V=σ(XpW3)V=\sigma(X_pW_3). Then

    D_{jj}=(1+\|\Delta W_{:,j}\|_2)^{-1}.$$ Equation (9) shares the unnormalized QKV product and its [fast-weight interpretation](https://proceedings.mlr.press/v139/schlag21a.html). Pooling compresses keys and values while queries retain all original tokens. Equation (15) preserves this factorization and scales weight columns, rather than using the query-dependent denominator above. The complete layer applies $W_0+\widehat{\Delta W}(X)$ and uses no head partitioning. In the complete-backbone comparison at $224\times224$, WeightFormer-S achieves 81.3% at 4.4G FLOPs, compared with multi-head DeiT-S's 79.8% at 4.6G.

3. Static comparisons and input-specific conditioning

  1. Table 2 compares complete blocks on a common static-CNN backbone. Generators add parameters and the normalization described in Appendix A. A parameter- and normalization-matched static ImageNet control is not included. The final models are configured against DeiT and other benchmark backbones.

  2. We tested input-specific conditioning with frozen WeightFormer-T/S. On 128 random ImageNet validation images at 224×224224\times224 in FP32, we replaced normalized updates in both dynamic layer types with either their mean over 64 separate training images or updates from another image's clean forward pass. Parameters and normalization stayed fixed. Cross-image results are mean ± sample SD over three donor permutations without self-matches.

    Update in both dynamic layer typesWeightFormer-T Top-1 (%)WeightFormer-S Top-1 (%)
    Normal73.4478.91
    Calibration-set mean23.4450.00
    Cross-image11.20 ± 2.5138.02 ± 1.19
  3. Replacing the input-specific updates substantially reduces accuracy. These interventions support the role of input-specific conditioning in the trained models.

4. Inference efficiency

  1. The new comparison uses optimized attention under common settings. Original Figure 5 used naive DeiT attention. We now use verified PyTorch SDPA FlashAttention for DeiT on an H100 (80 GB), with batch 64 and BF16 autocast over FP32 parameters. DeiT represents an isotropic backbone with global self-attention and uses random weights for runtime measurements.

  2. We measure throughput, wall time and peak memory with the same protocol. After 10 warm-ups, throughput and wall time are medians of three CUDA-synchronized host-timed repetitions. Each repetition runs two forwards of the same synthetic GPU-resident batch, totaling 128 image-inferences. Transfers and preprocessing are excluded. Peak memory includes the model and whole batch.

    InputModelThroughput (img/s)Peak memory (GiB)Wall time (s)
    1024 × 1024DeiT-T (Flash)928.12.370.138
    1024 × 1024WeightFormer-T891.32.720.144
    2048 × 2048DeiT-T (Flash)100.79.321.272
    2048 × 2048WeightFormer-T232.410.620.551
    3072 × 3072DeiT-T (Flash)22.720.335.634
    3072 × 3072WeightFormer-T104.123.771.229
    1024 × 1024DeiT-S (Flash)490.73.870.261
    1024 × 1024WeightFormer-S511.64.790.250
    2048 × 2048DeiT-S (Flash)51.015.142.508
    2048 × 2048WeightFormer-S132.418.320.966
    3072 × 3072DeiT-S (Flash)11.533.9211.178
    3072 × 3072WeightFormer-S59.040.882.171
  3. The throughput advantage grows at high resolution. At 3072, WeightFormer achieves 4.58× and 5.15× DeiT throughput for Tiny and Small, respectively. Flash-DeiT uses less peak memory at every tested point.

5. Input resolutions in Table 3 and Figure 1

  1. Figure 1 uses the same results and input resolutions as main-paper Table 3. The input sizes are 448×448 for WeightFormer-B, 384×384 for H-WeightFormer-B, and 224×224 for all other entries.

  2. Within each family, B shares the Small architecture and parameter count.

  3. Other experimental tables use consistent input resolutions within each comparison. We will annotate Figure 1 and Table 3 and clarify this naming in Section 5.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.