- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Voltic derives recurrent memory writes from a conditional linear-Gaussian filtering model. Input-dependent process variance and observation variance enter the prediction and observation steps separately, while a covariance state determines the direction and strength of a rank-one innovation update. Voltic-D retains diagonal covariance with learned coordinate-wise decay. Voltic-Q retains correlations within short token blocks and projects back to diagonal covariance at block boundaries, using identity decay. The uncertainty calculation is sequential, but the resulting mean-memory recurrence can reuse chunked delta-rule kernels.
The paper evaluates these designs on four controlled associative-recall conditions, 45M-parameter language models trained on approximately 2.52B FineWeb-Edu tokens, downstream and retrieval tasks, and dedicated 45M/180M throughput configurations. It reports strong altered-MQAR performance and improved retrieval beyond the training context, alongside task-dependent language-model quality and computational trade-offs.
Critical strengths
1. A coherent and implementable covariance-aware memory design.
The paper turns a classical filtering perspective into a concrete recurrent layer whose write direction depends on accumulated uncertainty. The diagonal and short-block approximations offer explicit compromises between retaining covariance information and limiting sequential work. Separating gain computation from mean-memory updates allows reuse of efficient delta-rule machinery. The appendix carefully specifies the projection direction and the limited isotropic correspondence to scalar delta writes. This is a meaningful architectural contribution even though the underlying filtering identities are classical.
2. Useful recall results supported by substantial experimental disclosure.
Voltic performs well across the four controlled conditions, with five reported training runs. In the mixed condition, Voltic-Q scores 72.89% versus KDA's 70.88%; at the unseen K=128 setting, the corresponding scores are 61.23% and 49.63%. The language-model experiments also show a substantial relative RULER extrapolation improvement, albeit at low absolute accuracy and with considerable seed variation.
The shared data streams, parameter/state accounting, learning-rate searches, three fresh LM reporting seeds, and optimized baseline timing implementations are valuable. The Transformer curriculum experiment is particularly informative: it recovers performance on tasks where direct training fails, correctly identifying an optimization issue rather than claiming an attention capacity limitation. These controls and candid limitations strengthen the paper.
Critical weaknesses
1. The experiments do not establish the headline separation mechanism, and one central ablation interpretation might be mathematically incorrect.
Section 4.2, p.7, lines 343–346 states that fixing the volatility vector to 0.5 makes predicted covariance isotropic and the gain a scalar multiple of the key. This does not follow from the recurrence: can retain anisotropy from both the previous covariance and coordinate-wise decay. Even starting with isotropic covariance and identity decay, an observation generally breaks isotropy. Thus Table 9 does not remove gain anisotropy as claimed.
Moreover, these interventions replace learned variance outputs only at evaluation time. They test sensitivity to an altered numerical parameterization, rather than whether the proposed distinction is necessary. Appendix E.7 correctly acknowledges this limitation. For Voltic-Q, fixing the stochasticity branch changes mixed-task accuracy from 72.89% to 72.98%, which provides little positive evidence that an adaptive stochasticity branch is needed in this setting.
The chosen mixed-task parameters further weaken the mechanism test. Under Appendix E.1, returning the value at each key's last designated observation-round position has exactly 75% expected accuracy because the final observation is independently corrupted with probability 0.25. An independent single-key calculation with the stated 4096-value alphabet, three rounds, and known finds that the latest value is a MAP choice for every observation-equality pattern, including . This calculation ignores cross-key dependence from initial sampling without replacement; it is not a global Bayes ceiling. It is also a diagnostic decision rule, not a state-matched neural baseline. Nevertheless, success on this condition does not require choosing accumulated agreement over recency. Separately training models for each condition further limits the inference about adaptive discrimination.
The existing results demonstrate useful memory behavior, but a genuine isotropy control, a retrained constrained-variance comparison, and a more diagnostic mixed condition are needed to attribute that behavior to the proposed separation. I am not requiring learned feature-space variances to numerically match categorical generator probabilities.
2. The reported quality and efficiency have not been established jointly for the same model configuration.
The Voltic quality models use eight layers, four heads, and key/value width 128, whereas the corresponding dedicated 45M Voltic timing models use seven layers, sixteen heads, and width 64, with fixed untrained weights and different feed-forward widths (Table 10 versus Appendix H.1). These are legitimate matched-budget timing comparisons, but they do not measure the cost of the architecture that produced the quality results. The distinction matters especially for Voltic's sequential uncertainty calculation.
The quality gains are also selective: Voltic-Q leads the eight-task mean, 42.79% versus KDA's 42.08%, but their nine-task means are essentially tied, 39.98% versus 39.99%. KDA has better reported FineWeb NLL and WikiText perplexity. The RULER extrapolation result is promising, but it does not establish a general quality advantage.
Finally, Appendix H.3 discloses failed final-state API oracle checks for both 180M Voltic variants at 8K/batch 8 without reporting the tolerances, error magnitudes, or diagnosis. The separate cache check in H.5 covers 64 teacher-forced continuation positions with a relative logit-error threshold of 0.05; it does not require greedy-token agreement. This does not invalidate the full-prefix-verified retrieval results or the output/gradient-checked training timings, but it leaves the scope of reliable recurrent state reuse unclear. Timing the actual quality architectures and clarifying the state discrepancy would materially strengthen the practical contribution.
Questions for authors
1. Mechanism controls — correction and targeted additional evidence. Do you agree that fixing process variance does not force isotropic predicted covariance? Please correct the interpretation of Table 9. Can you provide a genuine isotropic-gain control and a retrained constrained-variance variant, with otherwise matched training, to distinguish covariance geometry from dependence on the learned numerical scales?
2. Synthetic-task interpretation — clarification and targeted additional evidence. Please include latest-observation and appropriate Bayesian decision-rule references for the mixed condition. Given the 75% latest-observation reference, what existing result requires distinguishing change from corruption? A mixed parameter setting where older agreement can change the optimal decision would provide a more diagnostic test. Testing one trained model across multiple regimes would additionally support a claim of adaptive discrimination, but is not required merely to establish a useful task-specific recurrence.
3. Quality and speed — additional measurement. What are training throughput and cached decoding latency for the actual trained 45M quality architectures, at representative training and extrapolation lengths? Can you report cached/full-prefix greedy agreement or downstream quality for those same checkpoints, so that a quality/efficiency operating point is directly observable?
4. Returned-state correctness — clarification and validation. Which state components and APIs fail the H.3 oracle checks, by how much, and at what tolerance? Please map these paths to the training and decoding results and explain whether the discrepancy affects any reused state. Quantitative continuation errors and token agreement over at least the timed 128-step interval would clarify the remaining limitation.
7 comments
Response to Reviewer yxFt (2/6) — Retrained MQAR controls and the 75% copying reference
For branch and gain retraining, we reset the parameters producing and at each starting checkpoint so their initial outputs are 0.5. We train only the permitted parameters for 10,000 additional updates, keeping embeddings, output weights, key/value projections, decay, and all other weights fixed. The reference retrains both variance branches. Fixing leaves free to retrain, and fixing leaves free. When both outputs are fixed, no permitted parameters remain, so we evaluate the starting checkpoint with both outputs replaced.
The two-branch reference and the direct-gain and isotropic controls each train 33,540 parameters. Within each reporting seed they use identical fresh examples, AdamW, the prespecified learning rate 0.003, and batch size two. The direct-gain control reuses the two variance projections to produce a step size and a vector direction from the token features. Its gain can point away from the key, with bounded components along and perpendicular to it and a fixed perpendicular bound coefficient of 0.5. It carries no covariance across tokens.
Every row uses reporting seeds 711–714 and the same 256 test sequences for its task. We score all 131,072 queries in rounds 33–64 per seed and report mean accuracy (%) ± SEM over all four seeds.
| Model | Training or evaluation procedure | Standard | Volatile | Stochastic | Vol & Sto |
|---|---|---|---|---|---|
| Voltic-D | Full model before branch retraining | 74.37 ± 0.91 | 100.00 ± 0.00 | 96.94 ± 0.13 | 58.66 ± 0.06 |
| Voltic-D | Both variance branches retrained | 74.46 ± 0.87 | 90.14 ± 9.86 | 95.64 ± 1.80 | 58.61 ± 0.09 |
| Voltic-D | , retrained | 63.59 ± 8.06 | 76.24 ± 23.12 | 80.39 ± 4.28 | 52.38 ± 5.67 |
| Voltic-D | , retrained | 20.45 ± 13.92 | 0.54 ± 0.46 | 48.93 ± 9.48 | 1.65 ± 0.97 |
| Voltic-D | (evaluation only) | 19.06 ± 13.57 | 0.32 ± 0.27 | 26.17 ± 6.75 | 1.62 ± 0.96 |
| Voltic-D | Directly learned vector gain | 38.53 ± 10.62 | 4.30 ± 2.00 | 27.66 ± 11.69 | 6.68 ± 1.23 |
| Voltic-D | Isotropic covariance | 21.66 ± 11.31 | 0.58 ± 0.54 | 3.23 ± 2.16 | 3.81 ± 2.42 |
| Voltic-Q | Full model before branch retraining | 75.18 ± 1.50 | 99.99 ± 0.00 | 72.30 ± 24.10 | 58.66 ± 0.07 |
| Voltic-Q | Both variance branches retrained | 75.35 ± 1.43 | 99.99 ± 0.00 | 70.91 ± 23.66 | 58.71 ± 0.04 |
| Voltic-Q | , retrained | 65.25 ± 6.11 | 99.96 ± 0.02 | 68.94 ± 22.99 | 58.70 ± 0.03 |
| Voltic-Q | , retrained | 5.49 ± 1.34 | 1.23 ± 0.55 | 33.36 ± 17.75 | 7.82 ± 4.19 |
| Voltic-Q | (evaluation only) | 5.18 ± 1.34 | 0.85 ± 0.34 | 29.94 ± 16.36 | 5.94 ± 3.86 |
| Voltic-Q | Directly learned vector gain | 10.29 ± 2.57 | 27.11 ± 8.01 | 47.29 ± 21.93 | 55.97 ± 1.08 |
| Voltic-Q | Isotropic covariance | 7.50 ± 2.68 | 20.08 ± 7.42 | 41.10 ± 22.64 | 52.34 ± 1.71 |
The small effect of fixing observation variance for Voltic-Q on submitted Vol & Sto MQAR (72.89% versus 72.98%) persists after retraining on the updated task (58.71% versus 58.70% with ). The gain can still vary with the predicted covariance. The choice of process-variance scale also matters. Fixing each coordinate at its own calibration mean and retraining gives 58.62% for Voltic-D and 58.77% for Voltic-Q, close to their two-branch references. The projection in §1.1 tests what happens when the gain must stay parallel to the key.
1.3 Decision-rule references, and what requires distinguishing change from corruption
W1: "success on this condition does not require choosing accumulated agreement over recency."
Q2: "Please include latest-observation and appropriate Bayesian decision-rule references for the mixed condition. Given the 75% latest-observation reference, what existing result requires distinguishing change from corruption? A mixed parameter setting where older agreement can change the optimal decision would provide a more diagnostic test."
We confirm the reviewer’s calculation for the submitted Vol & Sto MQAR. Copying the latest observation and selecting the most probable value under the known-rate, single-key model both give 75% expected accuracy. The calculation assumes perfect key binding and considers one key’s observations at a time.
Response to Reviewer yxFt (3/6) — Bayesian decisions and paired histories on updated MQAR
On the 256 updated Vol & Sto MQAR test sequences, a Bayesian rule that uses each key's history scores 61.69%, versus 59.69% for copying the latest observation. The rule knows the generator, observes only the sequence and updates beliefs about the true value and both hidden rates. Earlier observations can therefore improve the decision in this setting. Both full Voltic models score 58.66% overall.
To test when the models use earlier evidence, we score the same checkpoints before branch retraining on two types of queries. In the first, the latest observation is wrong and the true value is unchanged. In the second, the latest observation correctly reflects a change. The model columns report mean accuracy (%) ± SEM over the four seeds. The copying rule is deterministic.
| Query type | Queries per seed | Voltic-D | Voltic-Q | Latest observation |
|---|---|---|---|---|
| True value unchanged, latest observation incorrect | 31,430 | 16.81 ± 0.56 | 16.96 ± 0.24 | 0.00 |
| True value changed, latest observation correct | 31,459 | 79.75 ± 0.57 | 79.84 ± 0.39 | 100.00 |
These two query types are about equally frequent. Voltic recovers about 17% of copying's errors in the first group and misses about 20% of the changes copying gets right in the second. Across these two groups, the lost correct switches outweigh the gains from recovering earlier values.
We also tested 256 pairs of histories per seed on the same Vol & Sto checkpoints. Four correct observations of A are followed by B. In one history B is an isolated corruption and the next observation is A. In the other the true value changes to B and the next observation is another B. The models see identical inputs through the first B. Each cell is mean accuracy (%) ± SEM over the same four seeds.
| Model and update | Retain A at the isolated B | Follow the change at the first B | Follow the change at the second B |
|---|---|---|---|
| Voltic-D, full model | 98.34 ± 1.28 | 1.37 ± 0.99 | 99.90 ± 0.10 |
| Voltic-D, directly learned vector gain | 3.12 ± 1.45 | 3.03 ± 0.10 | 14.26 ± 0.65 |
| Voltic-Q, full model | 98.34 ± 0.49 | 1.56 ± 0.53 | 98.14 ± 0.77 |
| Voltic-Q, directly learned vector gain | 66.89 ± 6.86 | 28.12 ± 5.70 | 96.88 ± 0.83 |
Both full models retain A on 98.34% of isolated first conflicts. Once a second B corroborates the change, Voltic-D adopts B on 99.90% of cases and Voltic-Q on 98.14%. The identical history through the first B explains why the same resistance to corruption also delays a genuine switch.
The submitted Stochastic MQAR also tests accumulation under corruption. Voltic-D and Voltic-Q score 88.33% and 87.61% there, above its 75% expected accuracy for copying the latest observation at a fixed 0.25 corruption probability. In updated Vol & Sto MQAR, the change and corruption probabilities also switch within sequences. Its event-conditioned and paired-history results show how a model trained on that single task responds to conflicting observations.
1.4 What we now claim
We distinguish the model's construction from what the controls establish. Voltic separately parameterizes process variance and observation variance and propagates uncertainty through the sequence. Our empirical claim is history-dependent memory updates with separately modeled process and observation uncertainty.
The controls test three changes to a trained Voltic model. We fix a variance output, remove covariance anisotropy while retaining scalar uncertainty history, or replace covariance propagation with a directly learned gain. The branch comparisons measure reliance on the learned update, and the paired histories show how the model responds when evidence for a new value accumulates. Neither measurement estimates the categorical change or corruption probability from a learned variance.
The abstract did not make this sense of "distinguishing" explicit, which we accept invited the stronger reading. We will revise it to read: "Next, we introduce Voltic, a recurrent memory that keeps the covariance anisotropic and makes both noise variances input-dependent, so the write is vector-valued and carries uncertainty accumulated over the sequence. The two variances enter the filter at the prediction and observation steps respectively, so the write responds to the two sources in opposite directions without inference over either." Here “opposite directions” refers to the write strength along the current key. With the history and key held fixed, this strength is , where . More observation noise reduces it, and more process noise in a direction sampled by the key increases it.
Response to Reviewer yxFt (1/6) — Covariance geometry and the design of the MQAR controls
We thank reviewer yxFt for their detailed and actionable feedback, including the covariance derivation and decision-rule calculation. New material in this response:
1. Mechanism controls (W1, Q1, Q2)
1.1 Correction to Section 4.2 and a genuine isotropic-gain control
We agree, and the reviewer's derivation is right. In Pt=AtUt−1At⊤+diag(vt), a constant vt does not remove anisotropy inherited from the previous covariance, from the coordinate-wise decay, or from the rank-one observation updates themselves. Table 9 therefore does not remove gain anisotropy; we will correct both the text and the interpretation in the revision.
In the updated MQAR setting each sequence has 16 keys and 64 rounds. Each active change or corruption probability switches independently between 20% and 60%, with a 1% switch chance per key per round. Standard MQAR has neither source, Volatile MQAR has changes only, Stochastic MQAR has corruption only, and Vol & Sto MQAR has both. Each model is trained on its own task. A model trained on Vol & Sto MQAR can encounter all four rate combinations within a sequence, with no rates or event labels in the input.
We query every key after every round, train on all query positions and score rounds 33–64. This gives a prediction after each observation, including the first conflict and the next observation that can corroborate a change.
Projecting the covariance to a scalar matrix lowers mean accuracy on Vol & Sto MQAR. Voltic-Q scores 52.34 ± 1.71% with the projection and 58.71 ± 0.04% with its usual covariance update. For Voltic-D the corresponding scores are 3.81 ± 2.42% and 58.61 ± 0.09%. We project the initial covariance and the predicted and posterior covariances at every step to scalar matrices of the same trace. Then Pt=ρtI gives gt=ρtkt/(st+ρt∥kt∥2), which is parallel to the key. Both controls retain input-dependent variances and the original decay. The projected control carries scalar uncertainty from step to step. We will relabel the original Table 9 as sensitivity to variance replacement and use this projection to test covariance geometry.
1.2 Retrained variance and gain controls
Keeping the other trained weights fixed, replacing the covariance update with either a directly learned vector gain or isotropic covariance lowers mean accuracy on all four tasks for each Voltic variant. The starting checkpoints use the submitted paper’s model initialization and 10,000 updates on the respective task. For each architecture and task, we chose the base learning rate from 0.003, 0.01, and 0.03 by final validation accuracy on 256 sequences using tuning seed 710, with ties broken by lower validation NLL and then the lower rate. Every reporting seed used that rate, with 50 warmup updates and cosine decay to 10% of the base rate.