- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FOLD, a neural architecture designed to support both conventional offline training and forward-only online adaptation. Its OnlineTSSA operator treats the recurrent state as a correction to learned dictionaries, updates that correction using incoming-token reconstruction residuals, and consolidates part of it into the model weights across sequences. FOLD uses dense and sparse OnlineTSSA modules in place of the usual attention and MLP/MoE components. Experiments show competitive offline language-modeling results and improved learning-and-retention performance on a controlled synthetic associative-recall stream, although the broader applicability of the online-learning mechanism remains to be established.
Critical strengths
Strength 1:
The paper identifies a meaningful problem and proposes a coherent, conceptually novel mechanism for addressing it. By framing the recurrent state as a correction to a learned dictionary and introducing a consolidation rule to transfer this correction into the base weights, the authors offer a substantive architectural idea. This is a principled and distinct perspective compared to existing Test-Time Training (TTT) or Fast Weight Programmer methods, which typically treat states as separate, transient memory.
Strength 2:
The online MQAR experiment provides direct, clear evidence for the proposed mechanism in a controlled setting. The results showing that FOLD with forward consolidation learns new groups on first exposure and retains 94.4% of old associations strongly support the claim that this specific consolidation design can alleviate the write/retain trade-off in associative recall.
Critical weaknesses
Weakness 1:
The online evaluation is limited to a synthetic associative-recall task, while the language-modeling benchmarks evaluate models trained offline. The current experiments therefore do not establish whether forward consolidation supports continual learning from natural-language streams or preserves a language-model-pretrained checkpoint's existing capabilities. In addition, the online comparisons do not include a method that updates persistent model parameters through a comparable mechanism. These limitations leave the broader continual-learning claims insufficiently substantiated.
Weakness 2:
Equation (10) is described as a gradient step on the objective in Eq. (9), but the residual is evaluated using the decayed previous correction. This appears to implement a decay-then-update procedure, which requires a more explicit derivation. Moreover, the paper does not report measured throughput or peak-memory comparisons that account for the cost of online updates and sequence-level consolidation. These omissions limit the mathematical clarity and practical efficiency claims of the proposed method.
Questions for authors
Question 1—Update-rule clarification:
In Section 3.1, Eq. (10) is described as a gradient step on the objective in Eq. (9), but its residual uses the decayed previous correction, , rather than directly evaluating the residual at . Is this intended to be a decay-then-update rule? Please clarify the derivation and the exact objective or update procedure being optimized.
Question 2— Continual-learning capability and scope:
Have the authors evaluated whether a language-model-pretrained FOLD checkpoint retains its language-modeling performance after processing a stream of new data through forward consolidation? If not, how should the broader claims about continual learning and avoiding catastrophic forgetting be scoped in light of the current evaluation on synthetic associative recall?
Question 3 — Scope of the online-learning comparison:
The current online-learning comparison focuses on state-clearing and state-carrying baselines. Are there applicable online or continual-learning methods that also incorporate new information into persistent model parameters and could serve as meaningful comparisons in this setting? If not, what makes such a comparison unsuitable?
Question 4 — Measured efficiency:
Can the authors report measured throughput and peak memory usage under a consistent hardware and evaluation setup, distinguishing standard inference from forward online learning and including the overhead of dictionary updates and sequence-level consolidation? This would help establish whether the proposed efficiency advantages hold in practice.