acceptodds
Withdrawn from ICLR 2027

Think Neuron: Belief Revision as a Sequence Transition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Most sequence architectures transform point representations. Probabilistic extensions usually attach uncertainty to parameters, latent variables, layer outputs, embeddings, or attention operators. We instead place the probabilistic object at the neural unit and make local belief revision the sequence transition itself. Elemental Variational Expanse (EVE), the variational distributional neuron used here, defines an input-conditioned local posterior. Affine-EVE turns prior–posterior revision into an analytical vector-valued transition. For diagonal Gaussian prior and evidence, the transition is exactly . These affine transitions compose associatively; we call this property Variational Chain Closure (VCC). EVER is the resulting sequential architecture. Across language modeling, semantic classification, audio, biological sequences, and structured composition, EVER achieves the lowest recorded headline value on seven of eight generalist tasks. We then use two WikiText-2 controls to isolate the source of the gain. In a frozen 6.6514M-parameter system benchmark, analytical prior–evidence precision fusion outperforms both a parameter-matched Transformer and a causal cross-attention model on all five paired seeds in NLL. In a stricter control with exactly matched architecture and parameter count, the precision-fusion model retains lower NLL: versus for a direct deterministic transition (5/5 paired seeds; paired -test ). Brier score, accuracy, and top-5 accuracy also favor precision fusion on all five pairs. A frozen FineWeb scaling suite from 10M to 120M parameters further shows EVER outperforming the matched Transformer at every tested size, with Mamba-2 as a strong sequential reference. Same-checkpoint interventions show direct use of prior–posterior quantities, while retrained ablations show that optimization can also reach effective alternative solutions. Together, these results support local belief revision as an explicit, reusable inductive bias for sequence computation.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
variational inference, belief revision, probabilistic neural computation, variational neurons, sequence modeling, associative computation, uncertainty modeling, local inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_A8h1OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes EVER, a sequence model in which each step performs Gaussian prior–evidence fusion. The previous state serves as the prior mean with a learned, time-invariant variance Σp\Sigma_p, the input yields a learned Gaussian evidence factor, and a sigmoid gate controls how much of the posterior mean is adopted. The resulting update is a diagonal affine recurrence st=Atst−1+Bts_t = A_t s_{t-1} + B_t, and the authors call its associative closure VCC and compute it with a parallel prefix scan. Experiments report gains over parameter-matched Transformer and cross-attention baselines on small-scale tasks, but EVER underperforms Mamba-2 on FineWeb language modeling.

Critical strengths
  1. The formulation is simple and analytically transparent. The fusion step yields closed-form coefficients that guarantee 0<At<10 < A_t < 1, and the Triton scan is verified against the sequential recurrence to floating-point precision.
  2. The experimental protocol is careful in places and reports unfavorable results openly. The WikiText-2 control matches architecture and parameter count exactly, uses paired seeds and a frozen test split, and the paper does not hide its losses to Mamba-2 or the retrained ablations that match or beat the full model.
Critical weaknesses
  1. The abstract is very poorly organized. It packs in experimental details such as exact parameter counts, p-values, and seed counts, and it frames a clear loss to Mamba-2 as "an architectural diagnosis". The core idea and contribution are hard to extract from it.

  2. Large parts of the appendix read as unedited text. Around L1080–1100, sentences are broken into isolated one-line fragments such as "evidence value, and relative credibility of the evidence". The same pattern appears in F.1 and G.2, where single numbers sit on their own lines, and in standalone filler sentences such as "This result quantifies how well the variants compensate after retraining." The text also refers to "the notebook implementation". Together these suggest the manuscript was largely AI-generated and not carefully reviewed by the authors before submission.

  3. The proposed transition reduces to a standard gated linear RNN. Because Σp\Sigma_p is time-invariant and the posterior variance is never propagated, βt=σ(log⁡Σp−log⁡Σe,t)\beta_t = \sigma(\log\Sigma_p - \log\Sigma_{e,t}) is simply an input-dependent sigmoid. The gate feature rt=−log⁡(1−βt)r_t = -\log(1-\beta_t) is a deterministic function of βt\beta_t. The update st=(1−kt)st−1+ktμe,ts_t = (1-k_t)s_{t-1} + k_t\mu_{e,t} therefore has the same form as minGRU, differing only in how the gate is parameterized. VCC is the well-known associativity of affine maps that underlies existing parallel scans. Closely related work such as minGRU, S5, LRU, and HGRN is not cited.

  4. The paper's own ablations undermine the claimed mechanism. On PTB, a constant-variance model retrained from scratch reaches 4.9902, better than the full model at 5.0127. The variant with ut=1u_t = 1 in H.4, which removes the adoption gate the authors call central, outperforms canonical EVER by over 0.4 nats. The only architecture-matched gain is 0.03 nats, which is attributable to gate parameterization rather than to belief revision.

  5. The baselines are weak. Mamba-2 beats EVER by about 1.5 nats at 120M, and EVER's NLL worsens as model size grows. The Transformer baseline is non-monotonic in size and trails Mamba-2 by nearly 2 nats, which suggests it is undertrained. The generalist suite includes no RNN or SSM baseline. The core-budget control uses a 10× smaller learning rate with persistent gradient clipping for the widened model, so it cannot support the conclusion about state geometry.

Questions for authors
  1. Beyond gate parameterization, how does Affine-EVE differ from minGRU or other gated linear RNNs? Since Σp\Sigma_p is fixed and posterior variance is not propagated, in what sense does sts_t represent a belief? A direct comparison with minGRU, LRU, or S5 under the same setup would be needed to support the claimed contribution.

  2. The deterministic control NLL in Table 3 on WikiText-2 and the direct deterministic transition NLL in Table 8 on PTB are both 5.0540. Please clarify why two experiments on different datasets report the same value.

Reviewer_PxpwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EVER constructs sequence transitions by analytically combining a learned diagonal-Gaussian evidence factor with a Gaussian prior centered on the preceding state. A separate adoption gate produces an input-dependent affine recurrence, enabling associative prefix-scan execution. Experiments span several sequence domains, with architecture-matched WikiText-2 controls, checkpoint interventions, retrained ablations, and fixed-token language-model comparisons. The paper reports benefits from the precision-fusion parameterization and proposes richer state organization as a route to increased capacity.

Critical strengths
  1. A transparent and mathematically consistent transition. The Gaussian-product moments, adoption update, and associative affine composition are sound. Appendix B.4 explicitly identifies the static prior covariance that enables precomputed transition coefficients. The scan implementation is checked numerically against the causal recurrence and yields measured acceleration, rather than relying on associativity alone. [Sections 3–4; Appendices B–C]

  2. Meaningful controls and informative negative results. The architecture-matched WikiText-2 comparison uses equal active parameter counts, symmetric learning-rate selection, and five paired seeds. Precision fusion improves NLL from 5.05401 to 5.02237, with all pairs favorable. The paper also reports that retrained fixed-variance PTB models can outperform the full model and appropriately distinguishes checkpoint dependence from mechanism necessity. [Table 3; Appendices F and H.2]

Critical weaknesses
  1. The decisive mechanism comparison is insufficiently specified. For coordinatewise prior and evidence variances vp,ve,tv_p,v_{e,t}, the proposed transition can be written as

    r_t=-\log(1-\beta_t),\qquad K_t=\beta_t\sigma(g_t+b+\lambda_r r_t).$$ The canonical model therefore implements a deterministic coupled gate; the contribution to assess is its particular parameterization and inductive bias. Appendix H.2 says the comparator learns the coefficient directly but does not give its defining function, the mapping of nonshared parameters, or the initialization of the differing branches. The reported matching and symmetric tuning are valuable, but without these details the comparison's precise scope cannot be assessed: gate nonlinearity, initial retention, and optimization conditioning may all differ. The positive WikiText-2 result warrants credit, but its attribution needs a reproducible definition of the competing transition. [Section 3.1; Appendix H.2, pp. 28–29]
  2. The capacity diagnosis is stronger than the evidence, and the practical advantage remains unclear. The core-budget experiment widens EVER from approximately 120M to 208M total parameters under the same 100M-token budget. The wider model remains persistently gradient-clipped, while canonical EVER exits that regime. Its worse NLL establishes a result for this training configuration; it does not establish that additional capacity is productive only through different state geometry or covariance structure. Separately, NLL 3.9628 in 536.2 seconds for Mamba-2 versus 4.7881 in 366.8 seconds for EVER gives two fixed-token operating points, without establishing an advantage at matched time or quality. These gaps limit the broader architectural diagnosis and practical case. [Section 5.3; Appendices H.3.2 and H.4, pp. 30–32]

Questions for authors
  1. What are the exact coefficient equations, parameter mappings, and branch initializations for the deterministic control? Please provide the existing implementation and initial retention distributions, and clarify which differences the comparison isolates. A reproducible definition substantiating the claimed control would resolve the principal attribution concern.

  2. Do existing optimization logs distinguish the effect of widening from persistent clipping and insufficient training? Please support the stronger capacity-allocation interpretation with controlled evidence, or restrict it to the tested configuration and budget.

  3. Can existing checkpoints establish NLL-versus-wall-clock curves against Mamba-2 under the same hardware and evaluation protocol? Evidence of a useful advantage at matched time or quality, together with a fully specified transition comparison, would support raising the rating.

Reviewer_asU8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a sequence model called EVER where the recurrent update is framed as a Gaussian belief update. At each step, the previous state gives a prior, the current input gives a Gaussian "evidence" term, and these are fused to get a posterior. A seperate gate then decides how much of the posterior goes into the new state. With diagonal Gaussians and a fixed prior variance, the update is affine (st=Atst−1+Bts_t = A_t s_{t-1} + B_t), so it can be computed with a parallel prefix scan which they call VCC and they give a Triton kernel for it.

They evaluate on quite a few tasks against transformer/attention models where EVER is best on 7 of 8. There is also a WikiText-2 control where the same architecture learns the gate directly instead from the precision ratio, as well as a FineWeb sweep from 10M to 120M params that includes Mamba-2.

Critical strengths
  1. The authors have put a lot of effort into fair comparisons, e.g. paired seeds, matched parameter counts, learning rates picked on validation, and the test set locked before evaluation. The matched deterministic control on WikiText-2 (Table 3) is nice, it is the right way to check if the precision-based gate actually helps. They also report results where they lose (Mamba-2 in Table 13) which I appreciate.
  2. Treating the gate as a ratio of prior and evidence uncertainty is easy to understand and gives a clear meaning to the gate values.
Critical weaknesses
  1. It seems to me that the core update is a standard gated linear RNN and closely related work is missing. Using the definitions in Sec. 3.1 (all operations elementwise) the fusion weight simplifies to βt=Σq,tΣe,t−1=ΣpΣp+Σe,t=σ(log⁡Σp−log⁡Σe,t).\beta_t = \Sigma_{q,t} \Sigma_{e,t}^{-1} = \frac{\Sigma_p}{\Sigma_p + \Sigma_{e,t}} = \sigma(\log \Sigma_p - \log \Sigma_{e,t}).

In App. B.4 the prior variance Σp\Sigma_p is said to be a fixed learned vector and Σe,t\Sigma_{e,t} is computed from the current input so βt\beta_t is just a sigmoid gate on the input. The contraction term is also a function of βt\beta_t as rt=log⁡(Σp/Σq,t)=−log⁡(1−βt)r_t = \log(\Sigma_p / \Sigma_{q,t}) = -\log(1 - \beta_t) so the adoption gate utu_t is an input gate too. The update is therefore st=(1−Kt)st−1+Ktμe,t,Kt=utβt∈(0,1),s_t = (1 - K_t) s_{t-1} + K_t \mu_{e,t}, \qquad K_t = u_t \beta_t \in (0,1), where neither the gate or μe,t\mu_{e,t} depends on st−1s_{t-1}. This is the same as gated linear RNNs such as GILR [Martin & Cundy, 2018] and minGRU [Feng et al., 2024]. The difference seems to only be how the gate is parameterised. In filtering terms this is a Kalman update where the prior variance is reset to a constant at each step instead of being carried forward.

The paper notes that carrying the posterior covariance forward would mean its current scan no longer applies (App. B.4). However, prior work, Kalman Linear Attention (KLA) [Shaj et al., ICML 2026] studies this setting. KLA frames sequence mixing as diagonal Gaussian filtering with token dependent observation precision, carries the precision forward, and still computes the model with parallel scans. Its posterior-mean recursion (Cor. 2.2, with a=1a=1, k=1k=1) has the same form as this paper's posterior-mean variant, except that here the prior precision is fixed. This work is not cited or discussed. Given the above, I do not think the novelty claims (Affine-EVE, VCC, and the "local belief revision" framing) are supported.

  1. The experiments do not seem to show a clear benefit over recurrent baselines. The main results in Table 2 compare against small transformer/attention models but not modern recurrent models with a similar structure. Mamba-2 is included inTable 13 and it seems to be much better (3.90 vs. 5.43 NLL at 120M). Also, EVER's NLL gets worse as model size grows while Mamba-2's improves. Finally, there are no state-tracking or recall benchmarks (e.g., MQAR, MAD), where recurrent models are usually compared.

  2. I found the paper quite hard to follow.

  • There is a lot of new terminology. EVE, Affine-EVE, canonical Affine-EVER, VCC, EVER, "adoption", and "relation-neutral gate" are introduced for fairly standard ideas.
  • There are a lot of missing details. The model is described only briefly. For example, it is not clear how gtg_t is computed and the "hierarchical hourglass topology" and other task-specific variants used for Logical Arguments and BBH-hard are not described.
  • The results are all over the place. The paper evaluates in many experiment suites and the numbers do not always line up. For example, WikiText-2 NLL for the proposed model is 5.187 in Table 2 but 5.022 in Tables 3 and 10, and the Transformer is 5.206 in Table 2 but 5.244 in Table 10. It is not clear whether these are the same models.

A shorter method section and one main results table for one clearly defined model would make the paper much easier to read and assess.

Questions for authors
  1. Relation to gated RNNs. Do you agree that Affine-EVE is just a gated linear RNN where the gate Kt=utβtK_t = u_t \beta_t depends only on the current input? If not, what does your probabilistic view compute that this recurrence does not? If so, what does your "variational neuron" framing contribute other than a different way of parameterising the gate?
  2. Fixed prior variance. Why is the posterior covariance not carried forward as the next prior and how does EVER relate to Kalman Linear Attention which does do this whilst remaining scan parallel?
  3. Baselines and benchmarks. Please compare against modern recurrent models (e.g., Mamba-2, GLA, Gated DeltaNet) at matched size, and include recall and state-tracking benchmarks (e.g., MQAR, MAD).
  4. Scaling. In Table 13, EVER's NLL gets worse from 10M to 120M parameters, while Mamba-2's improves. Do you have any results where EVER improves with more parameters?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.