- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
- The authors state that there is a trilemma between selectivity, normalization, and linear complexity for state-space models (SSMs) and linear attention. They argue that previous architectures cannot satisfy all three constraints at once.
- The authors propose FieldNet, which maintains a signal state and a companion mass under the same recurrent transition (one operator, two sources). Dividing the signal by the mass gives a normalized weighted average when the weights are nonnegative. They argue that this allows selectivity, normalization, and linear complexity to coexist.
- Experiments cover PG19 and WikiText-103 perplexity and seven zero-shot tasks, including normalization ablations at 200M parameters and baseline comparisons at 400M parameters. At 400M, the "NORM" variant outperforms the Llama-2 baseline on all datasets (both perplexity and average zero-shot accuracy).
Critical strengths
- Table 3 provides evidence that the companion mass helps WITHIN the tested 200M architecture. It improves both perplexity and downstream mean accuracy.
Critical weaknesses
-
I do not think the paper establishes the claimed trilemma. First, omitting normalization does not imply that an architecture cannot be normalized. The paper does not identify prior work explicitly stating that normalization is impossible while preserving selectivity and linear complexity. Second, Mamba-2 [1], Section 7.3, already describes computing a denominator () by applying the sequence transformation to an all-ones value channel, and Section 9.4.3 evaluates adding it. Although this normalizes different weights from FieldNet's per-channel decay-mass normalization, it demonstrates that selectivity and efficient normalization can coexist.
-
More importantly, the evidence is insufficient to establish a general benefit from normalization. First, the FieldNet configurations in Table 1 differ in query projection and transition parameterization, so the perplexity improvement cannot be attributed specifically to the companion mass. Second, Mamba-2 Section 9.4.3 evaluates adding normalization and reports that it introduces instabilities in most variants, while slightly improving performance with the ReLU activation function. Table 3 provides evidence that normalization helps within the proposed 200M architecture, but it does not establish whether these benefits extend to other architectures or larger scales.
[1] Dao, T., and Gu, A. (2024). Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. ICML 2024. https://arxiv.org/abs/2405.21060
Questions for authors
- Could the authors identify prior work that explicitly states the claimed trilemma? How does FieldNet differ from the normalization construction in Mamba-2 Section 7.3?
- Do any reported comparisons isolate the companion mass while keeping other setting fixed?
- Is there evidence that the benefits in Table 3 extend beyond the proposed 200M architecture, given the mixed normalization results in Mamba-2 Section 9.4.3?