acceptodds
Withdrawn from ICLR 2027

FieldNet: A Source-Free Normalizer for Any Transition, and Closed-Form Under Time Homogeneity

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Selectivity, bounded normalization and linear complexity are treated as a trilemma, reached in three steps: an input-dependent decay leaves the kernel lag-dependent, and a lag-dependent kernel has no closed-form mass, so the state cannot be normalized. The first two steps are right. The third holds only if the readout is unnormalized: under a posterior-mean readout (A3) the normalizer is the kernel's own mass, driven by the constant source , and it exists under any causal transition, content gating included. The object is a coupled dual recursion—one operator at two sources, not two recurrences—and the field locates the absence in the readout itself: removing the denominator degrades vision accuracy and destabilises token norms (Han et al., 2024). As stated, the trilemma is a statement about unnormalized readouts; under A3 it does not hold. The paper forms a closed loop. Derived: under A3 the normaliser is forced to be the kernel's own mass, and the coupled dual recursion it forces holds under any causal transition. Measured: a companion mass and weak content dependence compose, and that composition is the best model here—the 400M dual arm leads the matched baseline on WikiText-103 ( against ), seven-task zero-shot ( against ) and PG19 ( against ). Direction: the strong-gating end is supported by the same implementation; it opens the effect-size question.

Primary area:
foundation or frontier models, including LLMs
Keywords:
source-free normalizer, companion mass, posterior-mean readout, coupled dual recursion, time homogeneity, closed-form normalizer, selectivity and bounded normalization, long-context extrapolation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qKmQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary
  • The authors state that there is a trilemma between selectivity, normalization, and linear complexity for state-space models (SSMs) and linear attention. They argue that previous architectures cannot satisfy all three constraints at once.
  • The authors propose FieldNet, which maintains a signal state and a companion mass under the same recurrent transition (one operator, two sources). Dividing the signal by the mass gives a normalized weighted average when the weights are nonnegative. They argue that this allows selectivity, normalization, and linear complexity to coexist.
  • Experiments cover PG19 and WikiText-103 perplexity and seven zero-shot tasks, including normalization ablations at 200M parameters and baseline comparisons at 400M parameters. At 400M, the "NORM" variant outperforms the Llama-2 baseline on all datasets (both perplexity and average zero-shot accuracy).
Critical strengths
  • Table 3 provides evidence that the companion mass helps WITHIN the tested 200M architecture. It improves both perplexity and downstream mean accuracy.
Critical weaknesses
  • I do not think the paper establishes the claimed trilemma. First, omitting normalization does not imply that an architecture cannot be normalized. The paper does not identify prior work explicitly stating that normalization is impossible while preserving selectivity and linear complexity. Second, Mamba-2 [1], Section 7.3, already describes computing a denominator (W1W\mathbf{1}) by applying the sequence transformation to an all-ones value channel, and Section 9.4.3 evaluates adding it. Although this normalizes different weights from FieldNet's per-channel decay-mass normalization, it demonstrates that selectivity and efficient normalization can coexist.

  • More importantly, the evidence is insufficient to establish a general benefit from normalization. First, the FieldNet configurations in Table 1 differ in query projection and transition parameterization, so the perplexity improvement cannot be attributed specifically to the companion mass. Second, Mamba-2 Section 9.4.3 evaluates adding normalization and reports that it introduces instabilities in most variants, while slightly improving performance with the ReLU activation function. Table 3 provides evidence that normalization helps within the proposed 200M architecture, but it does not establish whether these benefits extend to other architectures or larger scales.

[1] Dao, T., and Gu, A. (2024). Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. ICML 2024. https://arxiv.org/abs/2405.21060

Questions for authors
  1. Could the authors identify prior work that explicitly states the claimed trilemma? How does FieldNet differ from the normalization construction in Mamba-2 Section 7.3?
  2. Do any reported comparisons isolate the companion mass while keeping other setting fixed?
  3. Is there evidence that the benefits in Table 3 extend beyond the proposed 200M architecture, given the mixed normalization results in Mamba-2 Section 9.4.3?
Reviewer_psA4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FieldNet studies a simple normalization for recurrent sequence mixers. Alongside a signal state that accumulates token-dependent inputs under a decay, it carries a second state driven by the constant input 1 and the same decay; the readout divides the signal state by this companion mass. For positive decay weights, the resulting per-channel output is a convex combination of past inputs and is therefore bounded by their range. The paper emphasizes that this construction works even when the decay depends on the current input. Time-homogeneous decay is a separate choice: it makes the mass a closed-form function of the learned rate spectrum and allows the authors to analyze position rescaling and horizon behavior. The implementation has position-only, weakly content-dependent “permeability,” and stronger content-gating settings, although the strongest gate is not trained in the reported experiments. The main measured system is a roughly 400M-parameter weak-permeability model with companion normalization.

Critical strengths

The practical normalization is simple and interpretable. Given a nonnegative recurrence, carrying the same transition on a constant source produces exactly the denominator needed for a convex weighted average, without sacrificing linear sequence complexity or requiring a fixed, position-only transition. The paper distinguishes that structural fact from the extra closed form available under time homogeneity. I find that separation useful, regardless of whether the Bayesian language ultimately survives revision.

The empirical record has more care than the broad abstract might suggest. The 200M same-lineage divisor comparison isolates a real failure mode: omitting the mass is not catastrophic at length 1,024 but deteriorates sharply at 32K, while an RMS divisor is bounded yet performs much worse. The anchored-region protocol compares the same text regions across lengths, and the paper reports paired uncertainty rather than just a collection of perplexity curves. It also states plainly that the 1.3B baseline has better in-distribution quality and that a registered prediction about which layers dominate rate-rescaling sensitivity was refuted. Those disclosures make the narrow experimental findings easier to trust.

Critical weaknesses
  1. The claimed derivation from a Gaussian posterior mean appears incorrect. Appendix 17.3 says the posterior mean solves a covariance linear system, but then writes a normalized kernel average in Eq. (19) and says it is exact after taking a large-noise limit. For a Gaussian prior with Gaussian observation noise, the ordinary posterior mean has the form kx⊤(K+σ2I)−1yk_x^\top(K+\sigma^2 I)^{-1}y; it does not generally normalize to reproduce a constant signal, and its weights need not be nonnegative or sum to one. Multiplying the large-noise expression by an extra factor to make its weights sum to one is a new estimator, not an equivalent posterior mean. Proposition 9 then defines “posterior mean” as any source-independent convex weighted average, making the uniqueness of the mass largely true by construction. The companion normalization remains sensible, but the paper's main assertion that A3 forces it under the stated probabilistic model needs either a corrected model and proof or substantially narrower terminology.

  2. The strongest architectural conclusions outrun the isolated comparisons. The 400M normalized arm uses weak permeability and a different code version from the unnormalized 400M arm, so their 8.32-point PG19 gap cannot be attributed to the mass alone; the authors acknowledge this. The clean 200M comparison is valuable but does not resolve the result at 400M or 1.3B, where no dual model is trained. Nor is strong content gating with the companion mass evaluated, despite the paper's prominent claim about combining selectivity and bounded normalization. The structural construction is clear, but whether the proposed combination is a competitive selective language model remains an open effect-size question. Finally, the paper describes a “released implementation” while its reproducibility statement says code will be released upon acceptance, so the present availability of the full training and evaluation setup is unclear.

Questions for authors

Could the authors derive Eq. (19) directly from the stated Gaussian prior and Gaussian noise model, without inserting a normalization after taking a limit? If the intended result is instead a Nadaraya-Watson estimator or a normalized filtering rule, can the authors restate A3 and the “forced” and “posterior mean” claims accordingly? This would materially affect my assessment of the main theoretical contribution.

Can the authors provide a same-code-version, same-transition 400M comparison that changes only the companion-mass readout, and ideally a trained strong-content-gate-plus-mass arm? The former would identify how much of the reported 400M gain is due to normalization; the latter would test the combination that motivates the trilemma claim. Please also clarify which code and evaluation scripts are currently available versus promised for release.

Reviewer_BC3iOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors propose a normalized recurrent sequence model based on exponentially decayed signal and mass states. The paper presents a theoretical derivation and evaluates language-modeling performance and length extrapolation up to 1.3B parameters.

Critical strengths

The paper tackles an interesting problem.

The empirical results are somewhat interesting, especially the normalization ablation at 200M parameters, and the promising language-modeling results for the 400M model/

Critical weaknesses
  1. Limited novelty and questionable theoretical foundation. The central normalized dual recurrence is already contained, as a special case, in the original RWKV formulation (Peng et al., 2023, Appendix D). The uniqueness and boundedness results are elementary consequences of normalized weighted averaging. Moreover, the Gaussian-posterior derivation underlying A3 is mathematically unjustified. I therefore find the main theoretical and conceptual contributions substantially overstated.

Furthermore, this is aggravated by the presentation. The paper is excessively verbose, repetitive, and unnecessarily self-justifying, repeatedly elevating rather elementary observations into elaborate theoretical claims. The writing has a strongly AI-polished feel, which might be a matter of personal preference but makes the reading tedious. More importantly, this rhetorical overengineering seems to obscure the limited novelty rather than provide genuine scientific insight.

  1. Insufficient empirical validation. The strongest 400M comparison changes several architectural and implementation factors simultaneously, preventing attribution of the improvements to the proposed normalization. The dual model is not evaluated at 1.3B, and length-invariant perplexity does not establish effective long-context utilization. Comparisons with other competitive recurrent architectures are also insufficient.
Questions for authors
  1. What is genuinely new relative to RWKV's existing numerator/denominator recurrence?

  2. Can the authors rigorously justify the Gaussian-posterior interpretation?

  3. Can the 400M improvements be reproduced when changing only the normalization mechanism?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.