- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes WeightFormer and H-WeightFormer, which replace explicit self-attention with input-conditioned dynamic parameterization for global visual modeling. The authors reinterpret attention as a dynamic MLP whose parameters are generated from the input sequence, and use this perspective to design dynamic linear layers and dynamic depthwise convolutions. The proposed methods predict weights from compressed global representations, including pooled features, feature correlations, and spatially adaptive descriptors, avoiding explicit construction of an N × N attention matrix. Experiments on ImageNet-1K classification, COCO object detection and instance segmentation, ADE20K semantic segmentation, and class-conditional ImageNet generation show competitive performance against several Transformer, ConvNet, and state-space-model baselines. The results suggest that input-conditioned parameterization is a promising alternative for efficient global visual modeling, although the novelty of the formulation and the evidence supporting its generality and efficiency warrant further scrutiny.
Critical strengths
The paper reframes attention as a dynamic MLP with input-conditioned parameters and uses this perspective to explore a family of dynamic linear and depthwise convolution operators. This provides a coherent design framework for investigating global context modeling without explicitly constructing a dense attention matrix. The proposed architectures are evaluated on image classification, object detection, instance segmentation, semantic segmentation, and class-conditional image generation. The results are encouraging: for example, H-WeightFormer-T achieves 83.3% ImageNet top-1 accuracy at 4.8G FLOPs, compared with 82.5% for VMamba-T at 4.9G FLOPs. These results suggest that the proposed approach merits further investigation as an efficient visual backbone.
Critical weaknesses
Rewriting attention as a dynamic MLP is mathematically valid, but appears to be primarily an alternative interpretation of the standard attention computation. The proposed replacement then uses compressed global descriptors to generate a fixed-size set of parameters, which is a different mechanism from attention's token-dependent pairwise interactions. The paper does not sufficiently characterize when this compression preserves useful global information, what expressive capabilities may be lost, or how the approach differs fundamentally from existing hypernetworks and dynamic convolution methods. Consequently, the central claim that dynamic parameterization can substitute for attention is supported mainly by empirical performance rather than a deeper theoretical account.
Although the proposed operators avoid an explicit N × N attention matrix, linear complexity alone does not establish superior practical efficiency. For example, H-WeightFormer-B reports 25.3G FLOPs, compared with 15.4G for VMamba-B, while WeightFormer-B reports 17.7G FLOPs compared with 17.2G for Vim-B. The efficiency advantage is therefore not uniform across model scales and baselines. In addition, the effective receptive field visualization demonstrates broad input sensitivity, but does not establish that the model captures the same kinds of task-relevant long-range dependencies as attention. More systematic comparisons of actual throughput, memory consumption, and performance under matched computational budgets would strengthen the main claims.
Questions for authors
What is the precise technical distinction between the proposed dynamic parameterization and existing hypernetworks, dynamic filter networks, and conditional convolutions? Beyond the algebraic reinterpretation of attention as a dynamic MLP, is there a specific theoretical property or capability established by the proposed framework that prior dynamic networks do not provide?
The proposed operators generate parameters from compressed global descriptors, including pooled features and feature correlations. How much task-relevant spatial information is retained by this compression? Can the authors provide controlled experiments that distinguish genuine long-range relational reasoning from global conditioning or increased effective receptive fields alone?
Can the authors provide more systematic comparisons of end-to-end inference latency, peak GPU memory, and accuracy under matched parameter, FLOP, and input-resolution settings? In particular, how does the method scale with increasing image resolution, and does its practical advantage persist across model sizes and hardware configurations?
Since the proposed architectures also incorporate depthwise convolutions, conditional positional encoding, and sparsely placed dynamic blocks, can the authors isolate the contribution of dynamic weight prediction through controlled ablations against otherwise identical static backbones and strong convolutional baselines?
4 comments
4. Inference efficiency
-
The new comparison uses optimized attention under common settings. Original Figure 5 used naive DeiT attention. We now use verified PyTorch SDPA FlashAttention for DeiT on an H100 (80 GB), with batch 64 and BF16 autocast over FP32 parameters. DeiT represents an isotropic backbone with global self-attention and uses random weights for runtime measurements.
-
We measure throughput, wall time and peak memory with the same protocol. After 10 warm-ups, throughput and wall time are medians of three CUDA-synchronized host-timed repetitions. Each repetition runs two forwards of the same synthetic GPU-resident batch, totaling 128 image-inferences. Transfers and preprocessing are excluded. Peak memory includes the model and whole batch.
Input Model Throughput (img/s) Peak memory (GiB) Wall time (s) 1024 × 1024 DeiT-T (Flash) 928.1 2.37 0.138 1024 × 1024 WeightFormer-T 891.3 2.72 0.144 2048 × 2048 DeiT-T (Flash) 100.7 9.32 1.272 2048 × 2048 WeightFormer-T 232.4 10.62 0.551 3072 × 3072 DeiT-T (Flash) 22.7 20.33 5.634 3072 × 3072 WeightFormer-T 104.1 23.77 1.229 1024 × 1024 DeiT-S (Flash) 490.7 3.87 0.261 1024 × 1024 WeightFormer-S 511.6 4.79 0.250 2048 × 2048 DeiT-S (Flash) 51.0 15.14 2.508 2048 × 2048 WeightFormer-S 132.4 18.32 0.966 3072 × 3072 DeiT-S (Flash) 11.5 33.92 11.178 3072 × 3072 WeightFormer-S 59.0 40.88 2.171 -
The throughput advantage grows at high resolution. At 3072, WeightFormer achieves 4.58× and 5.15× DeiT throughput for Tiny and Small, respectively. Flash-DeiT uses less peak memory at every tested point.
5. Input resolutions in Table 3 and Figure 1
-
Figure 1 uses the same results and input resolutions as main-paper Table 3. The input sizes are 448×448 for WeightFormer-B, 384×384 for H-WeightFormer-B, and 224×224 for all other entries.
-
Within each family, B shares the Small architecture and parameter count.
-
Other experimental tables use consistent input resolutions within each comparison. We will annotate Figure 1 and Table 3 and clarify this naming in Section 5.1.
Thank you for the detailed responses and additional experiments. The authors have addressed several of my concerns, particularly regarding the contribution of dynamic weight prediction through controlled ablations, the preservation of nonlocal information, and the inference efficiency comparison at different resolutions. The newly added experiments provide useful evidence supporting the effectiveness of the proposed dynamic parameterization. However, some fundamental concerns remain. In particular, the distinction from existing dynamic networks and hypernetwork-based approaches is still primarily established through specific architectural designs rather than a sufficiently clear conceptual or theoretical contribution. Moreover, the efficiency advantage is not consistent in terms of peak memory consumption, and the current comparisons do not fully establish a general practical advantage across different architectures and hardware settings. Overall, I appreciate the authors' efforts and believe the rebuttal has improved the paper, but the remaining concerns are not sufficiently resolved to change my original assessment. Therefore, I maintain my original score.
WeightFormer's distinct starting point is to view softmax attention as a dynamic MLP and use that view to design a visual backbone. For each query, and serve as two input-dependent weight matrices with softmax between them (Section 3.2). HyperNetworks studies weight generation as a form of relaxed sharing, Dynamic Filter Networks generates filters for adaptive transformations, and CondConv combines expert kernels per input. The attention viewpoint leads us to ask how generated weights can carry global context as the token count grows. WeightFormer uses weight updates of fixed shape for both channel mixing and spatial filtering. HyperNetworks · Dynamic Filter Networks · CondConv
This viewpoint guides what information each generator retains. Section 4 uses pooling and correlation as global summaries motivated by attention. Bilateral aggregates nonlinear feature correlations for channel mixing. Adaptive DWC retains the order of the pooled grid to set the weights at different kernel offsets. In our frozen-model diagnostic on 128 ImageNet images, permuting Bilateral tokens preserved accuracy. Permuting the DWC grid preserved channel means and reduced WeightFormer-S top-1 from 78.91% to 76.82%.
WeightFormer achieves higher reported ImageNet-1K top-1 than these prior dynamic models at similar compute or parameter counts. All results use inputs and the training recipes of the respective papers. Compute refers to FLOPs or multiply-adds as reported by each source. Our results are from Table 3. CondConv, Table 1 · ODConv, Table 2 · Involution, Table 1 · DynaMixer, Table 2
| Prior dynamic model | Compute (G) | Top-1 (%) | Our model | Compute (G) | Top-1 (%) | Gain (pp) |
|---|---|---|---|---|---|---|
| CondConv ResNet-50 (8 experts) | 4.21 | 78.6 | WeightFormer-S | 4.4 | 81.3 | +2.7 |
| ODConv ResNet-50 (4 kernels) | 4.08 | 78.52 | WeightFormer-S | 4.4 | 81.3 | +2.8 |
| RedNet-101 (Involution, 25.6M params) | 4.7 | 79.1 | WeightFormer-S (27M params) | 4.4 | 81.3 | +2.2 |
| DynaMixer-M (57M params) | 17.0 | 83.7 | H-WeightFormer-S (58M params) | 8.7 | 84.3 | +0.6 |
When the token count increases 9×, runtime grows 8.6–8.7× for WeightFormer and 41–43× for FlashAttention DeiT. We measured this scaling from to at batch 64 on the same GPU. At , WeightFormer-T and WeightFormer-S achieve 4.58× and 5.15× the throughput with 17% and 21% higher peak allocated memory, respectively. This scaling addresses the title's emphasis on linear time in the token count. We would appreciate your reconsidering the paper in light of the operator design and its measured scaling against optimized softmax attention.
We thank all reviewers for their questions and constructive comments. Below we provide direct responses to each question raised.
1. Relationship to existing dynamic networks
WeightFormer's innovation lies in two specific generators for global visual modeling. Bilateral predicts channel weights from nonlinear token co-occurrences. Spatially Adaptive DWC maps an ordered 3×3 pooled grid to depthwise kernels. These descriptors capture information beyond global channel means.
The differences concern the concrete generator designs. Dynamic Filter Networks provides a general filter-generation framework. HyperNetworks uses layer embeddings or recurrent context. CondConv mixes expert kernels through GAP-based routing. WeightFormer develops nonlinear correlation and spatial-grid parameterizations.
These generators produce fixed-size weight updates with O(N) cost at fixed channel widths and kernel size. Bilateral retains its Q(K⊤V) factorization, including Equation (15)'s column scaling. The dynamic depthwise branch uses global context to condition local spatial filtering.
2. Compression and nonlocal relations
The two generators retain complementary information. Bilateral preserves nonlinear feature co-occurrences and is invariant to pooled-token order. Spatially Adaptive DWC retains a coarse 3×3 layout. Pooling removes spatial detail within each region.
We tested which compressed information matters in frozen models. On 128 ImageNet validation images at 224×224, we modified only generator inputs and kept the feature maps transformed by the generated weights intact. Permutations preserve channel means, and permutation results average three seeds.
We also tested a specific nonlocal relation with matched input means. A width-32 Bilateral layer classifies whether a query and a marked target at least eight tokens away share a binary attribute. Paired examples swap target/decoy attributes, flipping the label while preserving all input channel means and the query. Only the query output is classified. Bilateral and the mean-only generator each have 3,218 parameters.
After 2,000 training steps, Bilateral reached 100% on 8,192 held-out examples in all three training seeds. The static transform and mean-only generator each reached 50%. Together with the frozen-model interventions, this supports the use of information beyond channel means and demonstrates this specific nonlocal relation through generated weights.
3. Controlled comparison with the static backbone
The static CNN and dynamic variants in Table 2 share the same backbone layout. Both contain 3×3 depthwise convolutions and use no separate positional-encoding modules. Spatial structure is handled by the convolutional layers.
Table 2 then varies dynamic weight prediction within that common backbone. Accuracy rises from 73.3% for the static CNN to 76.4% with Bilateral Linear 1, 74.6% with Adaptive DWC and 76.8% with both. The separate channel, depthwise and combined ablations show their contributions with the same DWC and positional-encoding settings.
Table 3 compares our models with strong CNN baselines, including ConvNeXt, FasterNet, and InceptionNeXt.