acceptodds
Under review as a conference paper at ICLR 2027

Voltic: Distinguishing Volatility from Stochasticity in Recurrent Memory

Abstract

Recurrent sequence models must decide how strongly to overwrite their memory at each token. Read as Bayesian filtering, this write is the gain of a Kalman update, set by uncertainty from two sources that pull it in opposite directions: volatility, how quickly the underlying associations change, and stochasticity, how noisy each observation of them is. First, we show that the update of gated delta-rule memories is the form this filter takes under isotropic uncertainty. Next, we introduce Voltic, a recurrent memory that keeps the covariance anisotropic and makes both noise variances input-dependent, so the write is vector-valued and carries uncertainty accumulated over the sequence. A dense covariance would have to be propagated token by token, ruling out the parallel training these models depend on. We therefore give two assumed-density approximations, diagonal and quasi-diagonal, both of which leave the memory update in delta-rule form and reuse its chunked kernels. On controlled recall tasks in which associations change and observations are corrupted, Voltic leads all baselines. On the task combining volatility and stochasticity, its margin over the strongest baseline is larger at both extrapolation sizes than at the training sizes. In 45M-parameter language models it leads an eight-task reasoning average and achieves higher retrieval accuracy beyond the training context length than gated baselines, at throughput close to those baselines. Deriving the write from an uncertainty recursion therefore makes memory more responsive to change.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian filtering, Kalman filter, assumed-density filtering, uncertainty, volatility, stochasticity, recurrent memory, linear attention, associative recall

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yxFtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Voltic derives recurrent memory writes from a conditional linear-Gaussian filtering model. Input-dependent process variance and observation variance enter the prediction and observation steps separately, while a covariance state determines the direction and strength of a rank-one innovation update. Voltic-D retains diagonal covariance with learned coordinate-wise decay. Voltic-Q retains correlations within short token blocks and projects back to diagonal covariance at block boundaries, using identity decay. The uncertainty calculation is sequential, but the resulting mean-memory recurrence can reuse chunked delta-rule kernels.

The paper evaluates these designs on four controlled associative-recall conditions, 45M-parameter language models trained on approximately 2.52B FineWeb-Edu tokens, downstream and retrieval tasks, and dedicated 45M/180M throughput configurations. It reports strong altered-MQAR performance and improved retrieval beyond the training context, alongside task-dependent language-model quality and computational trade-offs.

Critical strengths

1. A coherent and implementable covariance-aware memory design.

The paper turns a classical filtering perspective into a concrete recurrent layer whose write direction depends on accumulated uncertainty. The diagonal and short-block approximations offer explicit compromises between retaining covariance information and limiting sequential work. Separating gain computation from mean-memory updates allows reuse of efficient delta-rule machinery. The appendix carefully specifies the projection direction and the limited isotropic correspondence to scalar delta writes. This is a meaningful architectural contribution even though the underlying filtering identities are classical.

2. Useful recall results supported by substantial experimental disclosure.

Voltic performs well across the four controlled conditions, with five reported training runs. In the mixed condition, Voltic-Q scores 72.89% versus KDA's 70.88%; at the unseen K=128 setting, the corresponding scores are 61.23% and 49.63%. The language-model experiments also show a substantial relative RULER extrapolation improvement, albeit at low absolute accuracy and with considerable seed variation.

The shared data streams, parameter/state accounting, learning-rate searches, three fresh LM reporting seeds, and optimized baseline timing implementations are valuable. The Transformer curriculum experiment is particularly informative: it recovers performance on tasks where direct training fails, correctly identifying an optimization issue rather than claiming an attention capacity limitation. These controls and candid limitations strengthen the paper.

Critical weaknesses

1. The experiments do not establish the headline separation mechanism, and one central ablation interpretation might be mathematically incorrect.

Section 4.2, p.7, lines 343–346 states that fixing the volatility vector to 0.5 makes predicted covariance isotropic and the gain a scalar multiple of the key. This does not follow from the recurrence: �0��0� can retain anisotropy from both the previous covariance and coordinate-wise decay. Even starting with isotropic covariance and identity decay, an observation generally breaks isotropy. Thus Table 9 does not remove gain anisotropy as claimed.

Moreover, these interventions replace learned variance outputs only at evaluation time. They test sensitivity to an altered numerical parameterization, rather than whether the proposed distinction is necessary. Appendix E.7 correctly acknowledges this limitation. For Voltic-Q, fixing the stochasticity branch changes mixed-task accuracy from 72.89% to 72.98%, which provides little positive evidence that an adaptive stochasticity branch is needed in this setting.

The chosen mixed-task parameters further weaken the mechanism test. Under Appendix E.1, returning the value at each key's last designated observation-round position has exactly 75% expected accuracy because the final observation is independently corrupted with probability 0.25. An independent single-key calculation with the stated 4096-value alphabet, three rounds, and known h=0.30,e=0.25h=0.30,e=0.25 finds that the latest value is a MAP choice for every observation-equality pattern, including a,a,ba,a,b. This calculation ignores cross-key dependence from initial sampling without replacement; it is not a global Bayes ceiling. It is also a diagnostic decision rule, not a state-matched neural baseline. Nevertheless, success on this condition does not require choosing accumulated agreement over recency. Separately training models for each condition further limits the inference about adaptive discrimination.

The existing results demonstrate useful memory behavior, but a genuine isotropy control, a retrained constrained-variance comparison, and a more diagnostic mixed condition are needed to attribute that behavior to the proposed separation. I am not requiring learned feature-space variances to numerically match categorical generator probabilities.

2. The reported quality and efficiency have not been established jointly for the same model configuration.

The Voltic quality models use eight layers, four heads, and key/value width 128, whereas the corresponding dedicated 45M Voltic timing models use seven layers, sixteen heads, and width 64, with fixed untrained weights and different feed-forward widths (Table 10 versus Appendix H.1). These are legitimate matched-budget timing comparisons, but they do not measure the cost of the architecture that produced the quality results. The distinction matters especially for Voltic's sequential uncertainty calculation.

The quality gains are also selective: Voltic-Q leads the eight-task mean, 42.79% versus KDA's 42.08%, but their nine-task means are essentially tied, 39.98% versus 39.99%. KDA has better reported FineWeb NLL and WikiText perplexity. The RULER extrapolation result is promising, but it does not establish a general quality advantage.

Finally, Appendix H.3 discloses failed final-state API oracle checks for both 180M Voltic variants at 8K/batch 8 without reporting the tolerances, error magnitudes, or diagnosis. The separate cache check in H.5 covers 64 teacher-forced continuation positions with a relative logit-error threshold of 0.05; it does not require greedy-token agreement. This does not invalidate the full-prefix-verified retrieval results or the output/gradient-checked training timings, but it leaves the scope of reliable recurrent state reuse unclear. Timing the actual quality architectures and clarifying the state discrepancy would materially strengthen the practical contribution.

Questions for authors

1. Mechanism controls — correction and targeted additional evidence. Do you agree that fixing process variance does not force isotropic predicted covariance? Please correct the interpretation of Table 9. Can you provide a genuine isotropic-gain control and a retrained constrained-variance variant, with otherwise matched training, to distinguish covariance geometry from dependence on the learned numerical scales?

2. Synthetic-task interpretation — clarification and targeted additional evidence. Please include latest-observation and appropriate Bayesian decision-rule references for the mixed condition. Given the 75% latest-observation reference, what existing result requires distinguishing change from corruption? A mixed parameter setting where older agreement can change the optimal decision would provide a more diagnostic test. Testing one trained model across multiple regimes would additionally support a claim of adaptive discrimination, but is not required merely to establish a useful task-specific recurrence.

3. Quality and speed — additional measurement. What are training throughput and cached decoding latency for the actual trained 45M quality architectures, at representative training and extrapolation lengths? Can you report cached/full-prefix greedy agreement or downstream quality for those same checkpoints, so that a quality/efficiency operating point is directly observable?

4. Returned-state correctness — clarification and validation. Which state components and APIs fail the H.3 oracle checks, by how much, and at what tolerance? Please map these paths to the training and decoding results and explain whether the discrepancy affects any reused state. Quantitative continuation errors and token agreement over at least the timed 128-step interval would clarify the remaining limitation.

7 comments
AuthorsOpenReview ↗

Response to Reviewer yxFt (1/6) — Covariance geometry and the design of the MQAR controls

We thank reviewer yxFt for their detailed and actionable feedback, including the covariance derivation and decision-rule calculation. New material in this response:

  • We correct the interpretation of Section 4.2 and test isotropy directly. On updated Vol & Sto MQAR, Voltic-D scores 58.61% with its usual covariance update versus 3.81% with isotropic covariance. For Voltic-Q the comparison is 58.71% versus 52.34%, at the same retraining budget (W1, Q1).
  • Copying the latest observation and the single-key Bayesian rule both give 75% on the submitted mixed task. On the updated task, earlier observations improve the Bayesian decision, and paired histories show when Voltic retains an old value or adopts a change (W1, Q2).
  • At 45M, both Voltic variants improve on Kalman Linear Attention (KLA) on both task averages, NLL and perplexity. At 130M, Voltic-D exceeds KDA on the eight- and nine-task means across three seeds (W2).
  • On the trained 45M checkpoints, Voltic-D and Voltic-Q reduce time per cached token by 23% and 17% relative to KDA at 8K. We report the 8K prefill cost and training throughput at both 2K and 8K (W2, Q3).
  • With FP32 mean computation, the returned mean state passes the FP64 check in all four sampled H.3 cases. For the separate decoding path, we report next-token agreement after each cached update and exact agreement over 128-token greedy continuations (W2, Q3, Q4).

1. Mechanism controls (W1, Q1, Q2)

1.1 Correction to Section 4.2 and a genuine isotropic-gain control

W1: "Thus Table 9 does not remove gain anisotropy as claimed."
Q1: "Do you agree that fixing process variance does not force isotropic predicted covariance? Please correct the interpretation of Table 9."

We agree, and the reviewer's derivation is right. In Pt=AtUt−1At⊤+diag⁡(vt)P_t = A_t U_{t-1} A_t^\top + \operatorname{diag}(v_t), a constant vtv_t does not remove anisotropy inherited from the previous covariance, from the coordinate-wise decay, or from the rank-one observation updates themselves. Table 9 therefore does not remove gain anisotropy; we will correct both the text and the interpretation in the revision.

Q1: "Can you provide a genuine isotropic-gain control… to distinguish covariance geometry from dependence on the learned numerical scales?"

In the updated MQAR setting each sequence has 16 keys and 64 rounds. Each active change or corruption probability switches independently between 20% and 60%, with a 1% switch chance per key per round. Standard MQAR has neither source, Volatile MQAR has changes only, Stochastic MQAR has corruption only, and Vol & Sto MQAR has both. Each model is trained on its own task. A model trained on Vol & Sto MQAR can encounter all four rate combinations within a sequence, with no rates or event labels in the input.

We query every key after every round, train on all query positions and score rounds 33–64. This gives a prediction after each observation, including the first conflict and the next observation that can corroborate a change.

Projecting the covariance to a scalar matrix lowers mean accuracy on Vol & Sto MQAR. Voltic-Q scores 52.34 ± 1.71% with the projection and 58.71 ± 0.04% with its usual covariance update. For Voltic-D the corresponding scores are 3.81 ± 2.42% and 58.61 ± 0.09%. We project the initial covariance and the predicted and posterior covariances at every step to scalar matrices of the same trace. Then Pt=ρtIP_t = \rho_t I gives gt=ρtkt/(st+ρt∥kt∥2)g_t = \rho_t k_t / (s_t + \rho_t \lVert k_t \rVert^2), which is parallel to the key. Both controls retain input-dependent variances and the original decay. The projected control carries scalar uncertainty from step to step. We will relabel the original Table 9 as sensitivity to variance replacement and use this projection to test covariance geometry.

1.2 Retrained variance and gain controls

W1: "these interventions replace learned variance outputs only at evaluation time."
Q1: "Can you provide… a retrained constrained-variance variant, with otherwise matched training?"

Keeping the other trained weights fixed, replacing the covariance update with either a directly learned vector gain or isotropic covariance lowers mean accuracy on all four tasks for each Voltic variant. The starting checkpoints use the submitted paper’s model initialization and 10,000 updates on the respective task. For each architecture and task, we chose the base learning rate from 0.003, 0.01, and 0.03 by final validation accuracy on 256 sequences using tuning seed 710, with ties broken by lower validation NLL and then the lower rate. Every reporting seed used that rate, with 50 warmup updates and cosine decay to 10% of the base rate.

AuthorsOpenReview ↗

Response to Reviewer yxFt (2/6) — Retrained MQAR controls and the 75% copying reference

For branch and gain retraining, we reset the parameters producing sts_t and vtv_t at each starting checkpoint so their initial outputs are 0.5. We train only the permitted parameters for 10,000 additional updates, keeping embeddings, output weights, key/value projections, decay, and all other weights fixed. The reference retrains both variance branches. Fixing st=0.5s_t=0.5 leaves vtv_t free to retrain, and fixing vt=0.5v_t=0.5 leaves sts_t free. When both outputs are fixed, no permitted parameters remain, so we evaluate the starting checkpoint with both outputs replaced.

The two-branch reference and the direct-gain and isotropic controls each train 33,540 parameters. Within each reporting seed they use identical fresh examples, AdamW, the prespecified learning rate 0.003, and batch size two. The direct-gain control reuses the two variance projections to produce a step size and a vector direction from the token features. Its gain can point away from the key, with bounded components along and perpendicular to it and a fixed perpendicular bound coefficient of 0.5. It carries no covariance across tokens.

Every row uses reporting seeds 711–714 and the same 256 test sequences for its task. We score all 131,072 queries in rounds 33–64 per seed and report mean accuracy (%) ± SEM over all four seeds.

ModelTraining or evaluation procedureStandardVolatileStochasticVol & Sto
Voltic-DFull model before branch retraining74.37 ± 0.91100.00 ± 0.0096.94 ± 0.1358.66 ± 0.06
Voltic-DBoth variance branches retrained74.46 ± 0.8790.14 ± 9.8695.64 ± 1.8058.61 ± 0.09
Voltic-Dst=0.5s_t=0.5, vtv_t retrained63.59 ± 8.0676.24 ± 23.1280.39 ± 4.2852.38 ± 5.67
Voltic-Dvt=0.5v_t=0.5, sts_t retrained20.45 ± 13.920.54 ± 0.4648.93 ± 9.481.65 ± 0.97
Voltic-Dst=vt=0.5s_t=v_t=0.5 (evaluation only)19.06 ± 13.570.32 ± 0.2726.17 ± 6.751.62 ± 0.96
Voltic-DDirectly learned vector gain38.53 ± 10.624.30 ± 2.0027.66 ± 11.696.68 ± 1.23
Voltic-DIsotropic covariance21.66 ± 11.310.58 ± 0.543.23 ± 2.163.81 ± 2.42
Voltic-QFull model before branch retraining75.18 ± 1.5099.99 ± 0.0072.30 ± 24.1058.66 ± 0.07
Voltic-QBoth variance branches retrained75.35 ± 1.4399.99 ± 0.0070.91 ± 23.6658.71 ± 0.04
Voltic-Qst=0.5s_t=0.5, vtv_t retrained65.25 ± 6.1199.96 ± 0.0268.94 ± 22.9958.70 ± 0.03
Voltic-Qvt=0.5v_t=0.5, sts_t retrained5.49 ± 1.341.23 ± 0.5533.36 ± 17.757.82 ± 4.19
Voltic-Qst=vt=0.5s_t=v_t=0.5 (evaluation only)5.18 ± 1.340.85 ± 0.3429.94 ± 16.365.94 ± 3.86
Voltic-QDirectly learned vector gain10.29 ± 2.5727.11 ± 8.0147.29 ± 21.9355.97 ± 1.08
Voltic-QIsotropic covariance7.50 ± 2.6820.08 ± 7.4241.10 ± 22.6452.34 ± 1.71

The small effect of fixing observation variance for Voltic-Q on submitted Vol & Sto MQAR (72.89% versus 72.98%) persists after retraining on the updated task (58.71% versus 58.70% with st=0.5s_t=0.5). The gain can still vary with the predicted covariance. The choice of process-variance scale also matters. Fixing each vtv_t coordinate at its own calibration mean and retraining sts_t gives 58.62% for Voltic-D and 58.77% for Voltic-Q, close to their two-branch references. The projection in §1.1 tests what happens when the gain must stay parallel to the key.

1.3 Decision-rule references, and what requires distinguishing change from corruption

W1: "success on this condition does not require choosing accumulated agreement over recency."
Q2: "Please include latest-observation and appropriate Bayesian decision-rule references for the mixed condition. Given the 75% latest-observation reference, what existing result requires distinguishing change from corruption? A mixed parameter setting where older agreement can change the optimal decision would provide a more diagnostic test."

We confirm the reviewer’s calculation for the submitted Vol & Sto MQAR. Copying the latest observation and selecting the most probable value under the known-rate, single-key model both give 75% expected accuracy. The calculation assumes perfect key binding and considers one key’s observations at a time.

AuthorsOpenReview ↗

Response to Reviewer yxFt (3/6) — Bayesian decisions and paired histories on updated MQAR

On the 256 updated Vol & Sto MQAR test sequences, a Bayesian rule that uses each key's history scores 61.69%, versus 59.69% for copying the latest observation. The rule knows the generator, observes only the sequence and updates beliefs about the true value and both hidden rates. Earlier observations can therefore improve the decision in this setting. Both full Voltic models score 58.66% overall.

To test when the models use earlier evidence, we score the same checkpoints before branch retraining on two types of queries. In the first, the latest observation is wrong and the true value is unchanged. In the second, the latest observation correctly reflects a change. The model columns report mean accuracy (%) ± SEM over the four seeds. The copying rule is deterministic.

Query typeQueries per seedVoltic-DVoltic-QLatest observation
True value unchanged, latest observation incorrect31,43016.81 ± 0.5616.96 ± 0.240.00
True value changed, latest observation correct31,45979.75 ± 0.5779.84 ± 0.39100.00

These two query types are about equally frequent. Voltic recovers about 17% of copying's errors in the first group and misses about 20% of the changes copying gets right in the second. Across these two groups, the lost correct switches outweigh the gains from recovering earlier values.

We also tested 256 pairs of histories per seed on the same Vol & Sto checkpoints. Four correct observations of A are followed by B. In one history B is an isolated corruption and the next observation is A. In the other the true value changes to B and the next observation is another B. The models see identical inputs through the first B. Each cell is mean accuracy (%) ± SEM over the same four seeds.

Model and updateRetain A at the isolated BFollow the change at the first BFollow the change at the second B
Voltic-D, full model98.34 ± 1.281.37 ± 0.9999.90 ± 0.10
Voltic-D, directly learned vector gain3.12 ± 1.453.03 ± 0.1014.26 ± 0.65
Voltic-Q, full model98.34 ± 0.491.56 ± 0.5398.14 ± 0.77
Voltic-Q, directly learned vector gain66.89 ± 6.8628.12 ± 5.7096.88 ± 0.83

Both full models retain A on 98.34% of isolated first conflicts. Once a second B corroborates the change, Voltic-D adopts B on 99.90% of cases and Voltic-Q on 98.14%. The identical history through the first B explains why the same resistance to corruption also delays a genuine switch.

The submitted Stochastic MQAR also tests accumulation under corruption. Voltic-D and Voltic-Q score 88.33% and 87.61% there, above its 75% expected accuracy for copying the latest observation at a fixed 0.25 corruption probability. In updated Vol & Sto MQAR, the change and corruption probabilities also switch within sequences. Its event-conditioned and paired-history results show how a model trained on that single task responds to conflicting observations.

1.4 What we now claim

We distinguish the model's construction from what the controls establish. Voltic separately parameterizes process variance vtv_t and observation variance sts_t and propagates uncertainty through the sequence. Our empirical claim is history-dependent memory updates with separately modeled process and observation uncertainty.

The controls test three changes to a trained Voltic model. We fix a variance output, remove covariance anisotropy while retaining scalar uncertainty history, or replace covariance propagation with a directly learned gain. The branch comparisons measure reliance on the learned update, and the paired histories show how the model responds when evidence for a new value accumulates. Neither measurement estimates the categorical change or corruption probability from a learned variance.

The abstract did not make this sense of "distinguishing" explicit, which we accept invited the stronger reading. We will revise it to read: "Next, we introduce Voltic, a recurrent memory that keeps the covariance anisotropic and makes both noise variances input-dependent, so the write is vector-valued and carries uncertainty accumulated over the sequence. The two variances enter the filter at the prediction and observation steps respectively, so the write responds to the two sources in opposite directions without inference over either." Here “opposite directions” refers to the write strength along the current key. With the history and key held fixed, this strength is u/(u+st)u/(u+s_t), where u=kt⊤Ptktu=k_t^\top P_t k_t. More observation noise reduces it, and more process noise in a direction sampled by the key increases it.


Reviewer_sFpbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies recurrent memory updates through Bayesian filtering, distinguishing latent-state volatility from observation stochasticity. It proposes Voltic, which derives memory write strengths from a kalman style uncertainty with input-dependent noise parameters. Two variants maintain diagonal or short-block covariance approximations while preserving efficient delta-rule computation. Experiments evaluate controlled associative recall under changing and corrupted observations, language modeling, long-context retrieval, and computational throughput.

Critical strengths

The paper presents a principled connection between Bayesian filtering and recurrent associative memory, extending prior filtering-based approaches with input-dependent uncertainty and anisotropic gains. The controlled recall experiments directly target the proposed motivation and demonstrate substantial improvements over several baselines. The evaluation is relatively comprehensive, including multiple seeds, length extrapolation, natural-language benchmarks, and efficiency measurements.

Critical weaknesses
  1. Insufficient mechanistic validation of volatility–stochasticity separation
  • The central motivation is to distinguish volatility from stochasticity and adapt memory updates accordingly. However, the experiments primarily demonstrate improved task performance rather than whether the learned uncertainty branches actually capture and respond to these distinct sources.
  • Notably, fixing the stochasticity branch in Voltic-Q leaves performance on the combined task largely unchanged (72.89 vs. 72.98). Moreover, fixing volatility does not necessarily eliminate covariance anisotropy, contrary to the interpretation in Sec 4.2, since accumulated uncertainty and coordinate-dependent decay can preserve anisotropic covariance.
  • paper lacks direct analysis of how the learned uncertainty estimates and gains respond to actual latent changes versus observation corruption. Consequently, it remains unclear whether the improvements arise from the proposed uncertainty decomposition or simply from increased expressivity in the memory update.
  1. Limited generalizability of empirical improvements.
  • Although the controlled recall results are encouraging, the improvements on natural language tasks are less consistent. The language modeling gains are relatively small, and Voltic does not achieve the strongest language-modeling capacity. The RULER advantage of Voltic-Q also exhibits variability across training seeds, while improvements on other retrieval benchmarks depend on the selected variant.
  • Quality evaluation is limited to 45M-parameter models, and training efficiency becomes substantially less competitive at longer sequences with small batches. These limitations make it difficult to establish whether the proposed benefits persist at practically relevant regimes.
Questions for authors
  1. Please directly examine whether the learned volatility and stochasticity branches respond selectively to their corresponding uncertainty sources? The controlled MQAR setting provides ground-truth change and corruption indicators, enabling analyses of learned uncertainty and effective gain responses under independently varied noise conditions.

  2. Could authors provide retrained, matched-capacity ablations that independently constrain the two noise branches, together with a directly learned vector-valued gain baseline? This would help distinguish the contribution of the Bayesian uncertainty recursion from additional gating expressivity. Further validation across model scales and retrieval settings would also clarify the practical significance of the proposed mechanism.

5 comments
AuthorsOpenReview ↗

Response to Reviewer sFpb (1/5) — MQAR design and matched-capacity retraining

We thank the reviewer for a careful review that identified the right gap. New material in this response:

  • With the same 33,540 trainable parameters, Voltic's covariance update outperforms both a directly learned vector gain and an isotropic covariance update on mean accuracy for all four MQAR tasks in both variants (W1, Q2).
  • The mean variance outputs show no consistent selectivity for change or corruption. On paired histories, both variants usually retain the old value after one conflicting observation and adopt the new value after a second observation agrees with it (W1, Q1).
  • At 45M, both Voltic variants improve on Kalman Linear Attention (KLA) on all four aggregate language-modeling measures, and Voltic-Q leads KDA on RULER in every seed. At 130M, Voltic-D exceeds Transformer and GDN-2 on both task averages in each of three seeds (W2, Q2).
  • On the trained 45M models at 2K with batch 8, Voltic-D and Voltic-Q process 2.0–2.5 times as many tokens per second as KLA. Profiling explains why this ordering reverses at 8K with batch 1 (W2).

1. Mechanistic validation of the volatility–stochasticity separation (W1, Q1, Q2)

W1: "the experiments primarily demonstrate improved task performance rather than whether the learned uncertainty branches actually capture and respond to these distinct sources… it remains unclear whether the improvements arise from the proposed uncertainty decomposition or simply from increased expressivity in the memory update."
Q1: "Please directly examine whether the learned volatility and stochasticity branches respond selectively to their corresponding uncertainty sources?"
Q2: "Could authors provide retrained, matched-capacity ablations that independently constrain the two noise branches, together with a directly learned vector-valued gain baseline?"

1.1 Retrained ablations constraining each branch independently

Retraining measures how much each branch contributes when the other can adapt. The submitted tasks use three observations per key at 16 and 64 keys. The updated setting uses 16 keys and 64 rounds. Each active change or corruption probability switches independently between 20% and 60%, with a 1% switch chance per key per round. Standard MQAR has neither source, Volatile MQAR has changes only, Stochastic MQAR has corruption only, and Vol & Sto MQAR has both. The model observes the key–value pairs, with the rates and events hidden. We query every key after every round and score rounds 33–64.

We trained each variant on each task for 10,000 updates from the paper initialization. A separate tuning seed (710) selected the base learning rate from 0.003, 0.01 and 0.03 by final validation accuracy, breaking ties by lower NLL and then lower learning rate. For each reporting seed, we trained a checkpoint at the selected rate, then reset both variance branches to produce 0.5 and retrained the permitted parameters for 10,000 additional updates. Embeddings, output weights, key/value projections and decay stayed frozen. The reference retrains both branches. Fixing one variance at 0.5 allows the other branch to adapt. Fixing both leaves no trainable parameters, so we score that intervention directly on the checkpoint. The two-branch reference, direct-gain and isotropic controls each train 33,540 parameters using AdamW, a base learning rate of 0.003 and two sequences per update.

AuthorsOpenReview ↗

Response to Reviewer sFpb (2/5) — Branch ablations and covariance geometry

All rows use reporting seeds 711–714 and the same 256 test sequences for each task, scoring 131,072 queries per seed. Values are mean accuracy (%) ± SEM.

ModelConditionStandardVolatileStochasticVol & Sto
Voltic-DBoth branches retrained74.46 ± 0.8790.14 ± 9.8695.64 ± 1.8058.61 ± 0.09
Voltic-Dst=0.5s_t=0.5, retrain vtv_t63.59 ± 8.0676.24 ± 23.1280.39 ± 4.2852.38 ± 5.67
Voltic-Dvt=0.5v_t=0.5, retrain sts_t20.45 ± 13.920.54 ± 0.4648.93 ± 9.481.65 ± 0.97
Voltic-Dst=vt=0.5s_t=v_t=0.5 (evaluation only)19.06 ± 13.570.32 ± 0.2726.17 ± 6.751.62 ± 0.96
Voltic-DDirectly learned vector gain38.53 ± 10.624.30 ± 2.0027.66 ± 11.696.68 ± 1.23
Voltic-DIsotropic covariance21.66 ± 11.310.58 ± 0.543.23 ± 2.163.81 ± 2.42
Voltic-QBoth branches retrained75.35 ± 1.4399.99 ± 0.0070.91 ± 23.6658.71 ± 0.04
Voltic-Qst=0.5s_t=0.5, retrain vtv_t65.25 ± 6.1199.96 ± 0.0268.94 ± 22.9958.70 ± 0.03
Voltic-Qvt=0.5v_t=0.5, retrain sts_t5.49 ± 1.341.23 ± 0.5533.36 ± 17.757.82 ± 4.19
Voltic-Qst=vt=0.5s_t=v_t=0.5 (evaluation only)5.18 ± 1.340.85 ± 0.3429.94 ± 16.365.94 ± 3.86
Voltic-QDirectly learned vector gain10.29 ± 2.5727.11 ± 8.0147.29 ± 21.9355.97 ± 1.08
Voltic-QIsotropic covariance7.50 ± 2.6820.08 ± 7.4241.10 ± 22.6452.34 ± 1.71

Fixing vtv_t at 0.5 lowers mean accuracy on all four tasks in both variants. Fixing sts_t costs Voltic-D 6.23–15.25 percentage points, with the largest decrease on Stochastic MQAR. For Voltic-Q the largest decrease is on Standard (10.10 points), followed by Stochastic (1.98 points).

1.2 What the submitted ablations did and did not show

W1: "Notably, fixing the stochasticity branch in Voltic-Q leaves performance on the combined task largely unchanged (72.89 vs. 72.98). Moreover, fixing volatility does not necessarily eliminate covariance anisotropy, contrary to the interpretation in Sec 4.2, since accumulated uncertainty and coordinate-dependent decay can preserve anisotropic covariance."

The reviewer's observation about Voltic-Q holds after retraining. On Vol & Sto MQAR it scores 58.70% with st=0.5s_t=0.5 and 58.71% when both branches retrain. A fixed observation variance still lets the gain adapt through vtv_t and the accumulated covariance. The choice of process-variance scale matters too. Fixing each coordinate at its own calibration mean and retraining sts_t gives 58.62% for Voltic-D and 58.77% for Voltic-Q, both within 0.1 points of their two-branch references.

We agree with the correction to Section 4.2. Accumulated covariance, coordinate-wise decay and rank-one observation updates can maintain anisotropy even with constant process noise. Table 9 measures sensitivity to replacing the variance outputs. We will correct the interpretation in the text and caption.

We therefore test covariance geometry directly in the updated setting. At initialization and after every prediction and observation update, we replace the covariance with a scalar matrix of the same trace. This forces the gain to remain parallel to the key while retaining scalar uncertainty history and input-dependent variances. Under the branch-retraining procedure in §1.1, Vol & Sto accuracy is 52.34 ± 1.71% for Voltic-Q and 3.81 ± 2.42% for Voltic-D, compared with 58.71 ± 0.04% and 58.61 ± 0.09% when both branches retrain. This measures the contribution of covariance geometry within each trained model.

AuthorsOpenReview ↗

Response to Reviewer sFpb (3/5) — Learned gains and responses to change and corruption

1.3 Uncertainty recursion versus gating expressivity

The directly learned gain tests the contribution of covariance propagation while matching the number of trainable gain parameters. It keeps the memory update

Mt=(I−gtkt⊤)AtMt−1+gtxt⊤.M_t=(I-g_tk_t^\top)A_tM_{t-1}+g_tx_t^\top.

The original two variance projections are repurposed to produce the gain directly from token features. The scalar branch sets its step size, and the vector branch adjusts its direction away from the key. With zero perpendicular adjustment, this is a scalar-gated delta update along the key. The vector gate also permits a perpendicular adjustment bounded by a fixed coefficient of 0.5. It computes the write directly from the token features without propagating covariance. Both projections are reset and trained under §1.1's procedure, with the other trained weights held unchanged.

On Vol & Sto MQAR, Voltic-D scores 58.61 ± 0.09% with its covariance update versus 6.68 ± 1.23% with the directly learned gain. For Voltic-Q the comparison is 58.71 ± 0.04% versus 55.97 ± 1.08%. The covariance update also has higher mean accuracy on the other three tasks in both variants under the same retraining budget.

1.4 In what sense Voltic distinguishes the two sources

Our empirical claim is history-dependent memory updates with separately modeled process and observation uncertainty. The branch constraints show where each branch contributes within the trained models. The two branches also differ in dimension by design, with a process-variance value for each latent coordinate and a shared scalar observation variance.

We measured both variance outputs directly on the combined task before branch retraining. For each layer, we compared the observation variance sts_t and the coordinate mean of vtv_t between the 20% and 60% change rates, averaging equally over the two corruption rates, then repeated the comparison with corruption varied. In both layers of both variants, every four-seed mean difference was below 0.00045 in magnitude. These readouts show no consistent selective response to the corresponding source.

The write also depends on the accumulated covariance. We tested the same full models on 256 pairs of histories per seed. After four observations of A, the first B looks identical whether it is an isolated corruption or the start of a lasting change. Both variants retained A in 98.34% of isolated-corruption cases. When a lasting change produced a second B, Voltic-D adopted B in 99.90% of cases and Voltic-Q in 98.14%. These four-seed means show how corroborating evidence changes the model's decision.

The abstract did not make this sense of "distinguishing" explicit, which we accept invited the stronger reading. We propose revising it to read: "Next, we introduce Voltic, a recurrent memory that keeps the covariance anisotropic and makes both noise variances input-dependent, so the write is vector-valued and carries uncertainty accumulated over the sequence. The two variances enter the filter at the prediction and observation steps respectively, so the write responds to the two sources in opposite directions without inference over either." Here “opposite directions” refers to the write strength along the current key. With the history and key held fixed, this strength is u/(u+st)u/(u+s_t), where u=kt⊤Ptktu=k_t^\top P_t k_t. More observation noise reduces it, and more process noise in a direction sampled by the key increases it.


Reviewer_wd61OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces Voltic, a recurrent memory module derived from Bayesian‑filtering principles. It generalizes scalar‑valued write strength in gated delta‑rule memory to history‑dependent, vector‑valued update gains, with input‑dependent noise modelling for volatility and stochasticity. Two assumed‑density approximations (diagonal, quasi‑diagonal) are proposed to preserve delta‑rule updates and enable parallel training, alongside careful kernel engineering. Experiments are conducted on synthetic recall benchmarks and a 45M‑parameter language model across eight reasoning tasks. However, key mechanisms lack clean isolating ablations; performance gains are uneven across tasks, and quality‑efficiency trade‑offs are not validated under unified model settings.

Critical strengths
  1. Voltic provides a principled filtering-based memory update whose history-dependent, vector-valued gain extends the scalar write strength of gated delta-rule models.
  2. The paper combines substantial kernel engineering with detailed derivations, multiple-seed experiments, and transparent reporting of implementation choices and limitations.
Critical weaknesses
  1. The central mechanism is insufficiently isolated. The evaluation-only ablations do not establish the benefit of input-dependent noise modeling, and constant process noise does not imply isotropic covariance, contrary to the interpretation in Section 4.2. Moreover, perfect retrieval of the latest observation yields 75% expected accuracy on the combined task, so task accuracy alone does not demonstrate volatility–stochasticity discrimination.
  2. The eight-task average advantage over KDA is largely driven by BoolQ, while NLL and perplexity are worse than KDA and retrieval gains vary across tasks. Quality evaluation is also restricted to approximately 45M-parameter models, leaving the broader applicability of the approach uncertain.
  3. The quality–efficiency trade-off is not established for a single model configuration: quality and throughput evaluations use different configurations, while sequential uncertainty propagation leads to substantial slowdowns at long sequence lengths and small batch sizes.
Questions for authors
  1. Can you provide isolating ablations to quantify the individual benefit of input‑dependent noise modelling, and clarify why constant process noise does not equate to isotropic covariance as stated in Section 4.2? Given the 75% expected accuracy ceiling from retrieving only the latest observation on the combined synthetic task, how do results explicitly verify that Voltic truly disentangles volatility and stochasticity rather than only fitting task patterns?
  2. Since the eight‑task average improvement against KDA is heavily contributed by BoolQ, while NLL/perplexity are inferior and retrieval gains fluctuate across individual tasks, could you break down per‑task results and explain the factors driving this uneven performance?
  3. Could you report quality metrics and throughput under identical single‑model configuration? Please also characterise the practical slowdown incurred by sequential uncertainty propagation under long sequences and small‑batch conditions.
6 comments
AuthorsOpenReview ↗

Response to Reviewer wd61 (1/5) — Retrained MQAR ablations

We thank the reviewer for a concrete, actionable review. New material in this response:

  • A thorough ablation analysis addressing mechanistic validation of the model. The ablations were performed with retraining rather than evaluation-only substitution, across four MQAR tasks. The pattern presented in Table 9 holds. This will be presented as a new table in the main text, with further revision of the text explaining the rationale behind the design choices (W1, Q1).
  • A larger-model comparison at 130M between Voltic and KDA has been run, showing an edge for Voltic over KDA. More generally we scope the claim, with the two tied at 45M (W2).
  • A per-task breakdown across the eight tasks, and a clarification of which baseline comparison is central to our claim: not KDA but KLA (Kalman linear attention), the only baseline that also computes uncertainty (W2, Q2).
  • An account of why retrieval gains vary across the five in-context groups, following from the difference between long-range verbatim recovery and the short-context subsets (W2, Q2).
  • Quality and throughput on the same 45M checkpoint, with profiling and an overlap control: KDA processes 3.88×/6.06× as many tokens/s as Voltic-D/Q at 8K and batch 1, 1.83×/3.20× at batch 8 (W3, Q3).

1. Isolating the central mechanism (W1, Q1)

Bottom line: the conclusions of the evaluation-only Table 9 survive retraining.

W1: "The central mechanism is insufficiently isolated. The evaluation-only ablations do not establish the benefit of input-dependent noise modeling… task accuracy alone does not demonstrate volatility–stochasticity discrimination." Q1: "Can you provide isolating ablations to quantify the individual benefit of input dependent noise modelling… how do results explicitly verify that Voltic truly disentangles volatility and stochasticity rather than only fitting task patterns?"

1.1 Table 9, rerun with retraining

The reviewer's objection is that our ablations substituted values at evaluation time, so a degradation could reflect sensitivity to an altered parameterization rather than the necessity of a channel. We therefore shut each channel down and let the model adapt: starting from each model trained on its own task with the paper initialization and a learning rate selected on validation seed 710, we reset SS and VV to 0.5 and retrained the permitted branches for 10,000 updates with all other weights frozen. Each sequence has 16 keys and 64 rounds; in the three new tasks an active probability — the chance that a key's true value changes, or that its observation is wrong — switches between 20% and 60% without announcement. All rows use seeds 711–714 and the same 256 test sequences for each task, scoring rounds 33–64: 131,072 queries per seed. Values are mean accuracy (%) ± SEM:

MQAR taskModelBoth branches retrainedS=0.5S=0.5V=0.5V=0.5S=V=0.5S=V=0.5 (eval only)
StandardVoltic-D74.46 ± 0.8763.59 ± 8.0620.45 ± 13.9219.06 ± 13.57
Voltic-Q75.35 ± 1.4365.25 ± 6.115.49 ± 1.345.18 ± 1.34
VolatileVoltic-D90.14 ± 9.8676.24 ± 23.120.54 ± 0.460.32 ± 0.27
Voltic-Q99.99 ± 0.0099.96 ± 0.021.23 ± 0.550.85 ± 0.34
StochasticVoltic-D95.64 ± 1.8080.39 ± 4.2848.93 ± 9.4826.17 ± 6.75
Voltic-Q70.91 ± 23.6668.94 ± 22.9933.36 ± 17.7529.94 ± 16.36
Vol & StoVoltic-D58.61 ± 0.0952.38 ± 5.671.65 ± 0.971.62 ± 0.96
Voltic-Q58.71 ± 0.0458.70 ± 0.037.82 ± 4.195.94 ± 3.86

Moreover, the pattern across channels is asymmetric, and informatively so. Fixing the volatility channel at 0.5 reduces mean accuracy on every task, including Standard, where no probability is manipulated. Fixing the stochasticity channel at 0.5 costs Voltic-D 6–15 points throughout, while for Voltic-Q the cost is largest on Standard (10.10 points), followed by Stochastic (1.98 points). This is the clearest evidence we have that a channel does the job its derivation assigns it. §1.4 gives the reason for the asymmetry, which is a design choice rather than a property of volatility and stochasticity as such.

AuthorsOpenReview ↗

Response to Reviewer wd61 (2/5) — Input-dependent variance and the copying baseline

1.2 Why both noise variances are input-dependent, and what we claim

The ablations establish that both channels are necessary in the sense they test, and that fixing the stochasticity channel at 0.5 produces the largest drop in mean accuracy for Voltic-D on Stochastic MQAR (15.25 percentage points).

We should also address where the input-dependence comes from, since the submission did not make this clear and the MQAR results are not what motivates it. Two independent lines of reasoning do, and the revision states both.

The first is inferential. Estimating the two online is a long-standing problem in adaptive filtering (Särkkä & Nummenmaa, 2009), and normative accounts of learning treat their dissociation as central, showing that failing to make it produces characteristic failures of adaptation (Piray & Daw, 2021). These accounts infer both noise parameters jointly with the latent state, using a Rao-Blackwellized particle filter or a deterministic quantile grid in place of the particles (Doucet et al., 2000; Chen & Piray, 2026). Such methods are designed for behavioral tasks, a few static parameters inferred from hundreds of observations. Their representation of the noise parameters grows exponentially with the number of parameters, so they do not extend to the input-dependent, per-coordinate rates a high-dimensional memory trained on billions of tokens would require. That the variances depend on the input is what inference over them would produce: a learner that treats vv and ss as unknown maintains a posterior, and the rates entering its state update at time tt are posterior-weighted and therefore functions of x1:tx_{1:t}, as particle-filter and grid-based treatments compute explicitly. We instead amortize that dependence, producing the rates directly as learned functions of the input and training them end-to-end through the sequence-prediction loss, which treats them as parameters rather than as generative random variables.

The second is empirical, and internal to sequence modeling. Making the parameters of a recurrent update input-dependent rather than fixed is the change that separated Mamba from its fixed-parameter predecessors, and it is what brought state-space models to competitive language-model performance. That precedent was established on natural text at language-model scale, not on synthetic recall.

Both lines point the same way, so the design question is not whether the rates should depend on the input but whether the amortized form recovers enough of that dependence to be useful.

1.3 In what sense Voltic distinguishes the two sources

We do not claim that Voltic infers volatility and stochasticity; inference over the two is what the normative accounts do, and what does not scale here. We claim that the amortized form yields a functional separation, with the two channels needed in different circumstances.

The evidence for that is uneven, and we state where it holds. In Voltic-Q the stochasticity channel at 0.5 has its largest effect on Standard MQAR and almost none in the combined condition, where this ablation supplies no evidence of discrimination for this variant. In Voltic-D it lowers mean accuracy on all four tasks, with its largest effect on Stochastic MQAR. Fixing the volatility channel at 0.5 lowers mean accuracy on every task in both variants, including Standard, where nothing changes; we read that as following from the design choice described in §1.4 rather than from a property of volatility, and making the two symmetric would test it.

The tasks themselves limit how far this can be fitting rather than discrimination: the active probability switches between 20% and 60% within a sequence and without announcement.

The abstract did not make this sense of "distinguishing" explicit, which we accept invited the stronger reading. We have revised it to read: "Next, we introduce Voltic, a recurrent memory that keeps the covariance anisotropic and makes both noise variances input-dependent, so the write is vector-valued and carries uncertainty accumulated over the sequence. The two variances enter the filter at the prediction and observation steps respectively, so the write responds to the two sources in opposite directions without inference over either."

Also, to test whether Voltic uses earlier evidence instead of simply returning the latest observation, we separately score queries where the true value is unchanged but the latest observation is incorrect. On the updated Vol & Sto test, the Voltic-D and Voltic-Q original checkpoints before branch retraining recover the correct value in 16.81% and 16.96% of these cases, versus 0% for copying the latest observation. This demonstrates the use of earlier evidence when the latest observation is misleading.

AuthorsOpenReview ↗

Response to Reviewer wd61 (3/5) — Covariance correction and the 45M and 130M language results

1.4 Correction to Section 4.2

The reviewer is right that constant process noise does not imply isotropic covariance: anisotropy survives through the accumulated covariance, the coordinate-wise decay, and the rank-one observation updates themselves. We have corrected the text.

The claim we should have made concerns dimensionality rather than geometry. Volatility is a vector because the latent state is, while stochasticity is a scalar because each observation value is a single number per step. Shutting down the volatility channel therefore removes a degree of freedom per coordinate, whereas shutting down the stochasticity channel removes a single one. The revision makes clear that this asymmetry, not isotropy, is what the pattern in §1.1 reflects: volatility ablations degrade every task, stochasticity ablations degrade the task that targets them. Table 9 is relabelled accordingly.


2. Per-task breakdown and the factors driving uneven performance (W2, Q2)

W2: "The eight-task average advantage over KDA is largely driven by BoolQ, while NLL and perplexity are worse than KDA and retrieval gains vary across tasks. Quality evaluation is also restricted to approximately 45M-parameter models…" Q2: "…could you break down per task results and explain the factors driving this uneven performance?"

2.1 Per-task breakdown

We would like to highlight two points.

First, in the 45M comparison across eight tasks and three seeds, KDA is ahead on four (HellaSwag −0.96, PIQA −0.05, ARC-E −1.30, WinoGrande −0.87) and Voltic-Q on the other four (ARC-C +0.57, OpenBookQA +1.40, Social IQa +0.26, BoolQ +6.64). Among these, BoolQ shows by far the largest separation — 61.74% against 55.11%, where the next largest difference in either direction is 1.40 points — and it favours Voltic-Q.

We have nonetheless scoped the interpretation rather than resting on the mean. KDA is slightly ahead on language comprehension, which is what perplexity, NLL and LAMBADA load on. Voltic-Q is ahead on binary choice under a skewed prior (BoolQ) and on retrieval beyond the training context (RULER, 15.15 against 2.03). BoolQ's labels are skewed toward "yes", suggesting part of the difference is better modelling of the task distribution (decision making under uncertainty), which is consistent with maintaining an explicit uncertainty state.

Second, the baseline comparison central to our claim is not KDA but KLA, the only baseline that, like Voltic, carries an explicit uncertainty computation and filtering. Our claim separates into three parts: on downstream language tasks we are on par with the uncertainty-neutral models (such as KDA or GDN-2); on retrieval we are substantially ahead; and against KLA we are ahead throughout. §2.2 covers retrieval and §2.3 the KLA comparison. We will revise the manuscript to make this three-way distinction explicit.

Finally, to address applicability beyond 45M, we evaluated 130M models at 4,800 updates (2.52B input tokens, matching the 45M token count). At this intermediate checkpoint, both Voltic variants exceed KDA and Transformer on the eight-task mean. On the nine-task mean both exceed Transformer, and only Voltic-D exceeds KDA. Values are means ± sample SD across training seeds 0, 1 and 2.

130M modelEight-task accuracy (%)Nine-task accuracy (%)
Voltic-D43.70 ± 0.1341.52 ± 0.32
Voltic-Q43.43 ± 0.2640.92 ± 0.27
KDA43.21 ± 1.0541.41 ± 1.05
Transformer42.45 ± 0.6340.52 ± 0.54
GDN-242.32 ± 0.7440.35 ± 0.65
Mamba-3 SISO43.68 ± 0.4341.56 ± 0.39
Mamba-3 MIMO43.83 ± 0.3041.80 ± 0.17

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.