- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MoC, which replaces static short depthwise convolution with a token-dependent mixture of learned kernels. Experiments on Transformer and Gated DeltaNet models show improved language-modeling perplexity, accompanied by a study of the interaction between convolution initialization, learning rates, and residual connections.
Critical strengths
-
MoC improves FineWeb-Edu and WikiText-103 perplexity across both backbones and model sizes. The parameter-matched mlp baseline and fixed-routing ablation provide useful evidence that the gains involve token adaptivity rather than simply additional parameters.
-
The initialization and learning-rate experiments provide practical guidance for training short convolutions. The residual-connection ablation also shows that conclusions about local architectural choices can change with the optimization setting.
Critical weaknesses
-
The input-dependent kernel mixture is closely related to established constructions in CondConv and dynamic Convolution, while the token-adaptive QKV setting is also examined in the concurrent LRDC work. The distinctive contribution is therefore relatively narrow, and the routing and optimization refinements do not, on the present evidence, amount to a sufficiently substantial advance.
-
At 460M parameters, the FineWeb-Edu improvements over LRDC are only 0.05 and 0.06 perplexity, without reported seed variability for these comparisons. The three-seed study and improved LRDC initialization are evaluated at 110M, leaving it unclear whether the smaller main-setting advantages persist across repeated runs and comparable baseline tuning.
-
Average downstream accuracy exceeds static convolution in only one of the four main configurations, while the reported 460M Transformer benchmark shows approximately 4.1% lower training throughput than static convolution. The fixed-token comparisons therefore leave the practical benefit unresolved: the paper does not establish that MoC is preferable when training time or compute is held constant.
I recommend weak rejection. The empirical study contains useful controls, but the limited methodological advance and insufficient evidence of a practically meaningful advantage leave the contribution below the acceptance threshold in my assessment.
Questions for authors
-
Could the authors report variability across training seeds for a representative 190M or 460M comparison, particularly where the FineWeb-Edu margin over LRDC is only 0.05–0.06 perplexity?
-
Do the main-setting gains persist when LRDC uses the improved initialization from Table 8 and receives comparable convolution-specific learning-rate tuning?
-
At an existing model scale, does MoC retain its perplexity advantage over static convolution when the methods receive the same training time or compute budget?
-
Appendix B describes the dynamic low-rank projections in LRDC as zero-initialized, whereas the original LRDC paper specifies zero initialization only for the second projection. What initialization was used for each factor in these experiments, and is this discrepancy purely a wording issue?
5 comments
We thank the reviewer for their careful evaluation of our work and for recognizing the value of our empirical and optimization studies. We appreciate the opportunity to clarify our contributions and address the concerns raised.
Contemporaneous work and novelty. We would first like to emphasize that LRDC is contemporaneous work developed independently of ours. ICLR's 2027 Reviewer Guidelines explicitly define work published within two months as contemporaneous, and state that authors are not required to compare against such work and that the absence of these comparisons cannot be grounds for rejection. LRDC's NeurIPS 2026 acceptance was announced on September 24, 2026, just one day before the ICLR submission deadline. We voluntarily included comparisons with LRDC for scientific transparency, not to present MoC as an incremental improvement over LRDC. In light of ICLR's contemporaneous-work policy, we respectfully ask that our submission be evaluated on its own merits, rather than treating LRDC as established prior work against which MoC must demonstrate novelty or superiority.
Scope of the LRDC comparisons. We agree that more comprehensive comparisons across random seeds, model scales, architectures, and optimization settings would be valuable, but emphasize that establishing general superiority over LRDC is not a central claim of our paper. Our three-seed experiment at 110M parameters favors MoC across all seeds, including with the adjusted LRDC initialization, but does not establish the robustness of the smaller margins observed at 460M or the relative performance under other training configurations. We have further revised the manuscript to clarify that these comparisons are insufficient to establish general superiority of either method across model scales and architectures.
Clarifications regarding LRDC (Q2, Q4). Neither LRDC nor MoC received method-specific learning-rate tuning. All main experiments use the same fixed optimization recipe: the low-rank projection in LRDC (corresponding to the router in MoC) is optimized with NorMuon at a learning rate of 1e-2, while the U projection in LRDC (corresponding to the convolutional bases in MoC) is optimized with AdamW at 1e-3. Our separate learning-rate sweeps were supplementary and conducted only after the main experiments had been completed, and therefore did not inform any of the main experimental configurations.
The reduced initialization of LRDC's static kernel component was investigated after the main experiments. At 110M parameters, it yielded a small improvement in validation perplexity (24.20 to 24.15), and we used this adjusted initialization in our three-seed comparison. We have not established whether this adjustment improves LRDC at larger scales. Lastly, we confirm that LRDC's first low-rank projection is initialized with standard deviation 0.02, while the subsequent (U) projection is zero-initialized, consistent with the original formulation. We have ensured that this is correctly described in the manuscript.
Computational efficiency (Q3). Our original efficiency experiments were primarily intended to demonstrate that token-adaptive short convolution can be implemented with practical overhead, rather than to establish the efficiency limit of the operator. Avoiding materialization of token-specific convolution kernels is essential for practical efficiency, and our fused Triton implementation demonstrates this feasibility. Section 4.7 identifies several opportunities for further optimization.
To directly evaluate the efficiency of MoC relative to LRDC, we have now adapted MoC to LRDC's publicly released Triton implementation and benchmarked both methods under identical conditions on an NVIDIA H100, using our 460M-scale Transformer.
| Method | Forward (k tok/s) ↑ | Forward + backward (k tok/s) ↑ |
|---|---|---|
| LRDC | 273.7 | 94.1 |
| MoC (softmax QKV) | 273.6 | 94.3 |
| MoC (exp Q/K, softmax V) | 274.6 | 94.5 |
MoC and LRDC achieve effectively identical throughput under this shared implementation, demonstrating that MoC's routing parameterization introduces negligible additional overhead relative to LRDC. We have included these results in the revised manuscript's appendix.
To assess the effectiveness of the additional computation independently of current implementation efficiency, we include a static-convolution baseline with an expanded MLP. Rather than allocating additional matrix-multiplication work to token-adaptive convolution, this baseline allocates approximately the same additional parameter and FLOP budget to wider MLP projections while retaining static convolution. We increase the MLP expansion only slightly (from 3.0× to 3.1× for Transformer and 3.075× for Gated DeltaNet), while keeping the initialization of all other parameters the same. The additional MLP capacity provides substantially smaller improvements in FineWeb-Edu perplexity than MoC:
| Model (190M) | Static conv. | Static + MLP | MoC |
|---|---|---|---|
| Transformer | 18.91 | 18.87 | 18.72 |
| Gated DeltaNet | 19.45 | 19.44 | 19.19 |
This demonstrates that allocating the additional computation to adaptive local mixing is considerably more effective than allocating it to MLP width in these experiments. We have updated the manuscript to clarify that the expanded-MLP baseline is approximately matched to MoC in both parameter count and FLOPs, whereas only parameter matching was explicitly noted in the original submission.
Finally, the practical viability of a sequence-modeling primitive depends not only on training efficiency, but also on prefill and autoregressive decoding. Even with our original implementation, MoC incurs only 1.9% additional overhead relative to static convolution in our full-sequence forward-pass benchmark, compared with 4.1% for forward and backward execution. During autoregressive decoding, where memory bandwidth is often the primary bottleneck, MoC retains the same minimal historical convolution state as static short convolution, negligible compared with the KV cache in full-attention Transformers, while introducing only modest additional matrix-multiplication work and lightweight routing operations. This distinction is particularly relevant to our expanded-MLP comparison: the substantially smaller gains from allocating equivalent compute to MLP expansion suggest that matching MoC's improvements through MLP scaling alone would require considerably more parameters, increasing weight-memory traffic during decoding, even if training throughput were competitive in a compute-bound regime. While a full characterization of inference-time trade-offs requires dedicated decoding implementations and benchmarks, these considerations highlight why training throughput alone does not fully characterize the practical efficiency of token-adaptive short convolution.
After rereading the paper and the other reviews, I am lowering my rating from 2 to 1. I find the optimization study useful, but the evidence for a substantial advance over existing dynamic convolutions is not convincing enough. At 460M, the FineWeb-Edu gains over LRDC are only 0.05–0.06 perplexity. The three-seed comparison with improved LRDC initialization supports the results at 110M, but does not establish whether these much smaller margins at 460M are robust. I also share Drux’s concern that Table 6 omits LRDC, and there is no comparison at equal training time or compute to establish a practical advantage. On reconsideration, these gaps carry more weight in my assessment because the methodological contribution is narrow and the case for acceptance rests largely on the empirical gains.