acceptodds
Withdrawn from ICLR 2027

Forward Online Learning Deep Networks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern deep neural networks are typically trained offline via computationally intensive large-scale backpropagation on massive datasets. In deployment scenarios, adapting the model weights to new data streams again requires costly backpropagation. In this paper, we derive from first principles a novel deep architecture, which we call FOLD, that supports both efficient *offline training via backward propagation* and *online learning via forward propagation*. Each layer consists of *a novel linear-time operator* that can replace both the attention and the MLP operators in a Transformer. The weights of each such operator are essentially a learned dictionary for its input data; in this way online adaptation to incoming new data becomes an online dictionary learning problem. As a result, the model weights can be updated at test time through *forward propagation as an optimization step* to better align the dictionaries with every new sample sequence. Experiments on synthetic and real-world benchmarks show that this new architecture performs competitively against the state-of-the-art linear-time recurrent deep architectures such as Gated DeltaNet with back propagation training while demonstrating qualitatively superior online learning capabilities with its forward propagation mechanism. We also show that incorporating explicit forward propagation into the architecture has side benefits, such as strong long-context performance with only subquadratic operations. Our results call into question the necessity of solely using back propagation to adapt pretrained models.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Neural network architecture, representation learning, online learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vJXrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FOLD, a neural architecture designed to support both conventional offline training and forward-only online adaptation. Its OnlineTSSA operator treats the recurrent state as a correction to learned dictionaries, updates that correction using incoming-token reconstruction residuals, and consolidates part of it into the model weights across sequences. FOLD uses dense and sparse OnlineTSSA modules in place of the usual attention and MLP/MoE components. Experiments show competitive offline language-modeling results and improved learning-and-retention performance on a controlled synthetic associative-recall stream, although the broader applicability of the online-learning mechanism remains to be established.

Critical strengths

Strength 1:

The paper identifies a meaningful problem and proposes a coherent, conceptually novel mechanism for addressing it. By framing the recurrent state as a correction to a learned dictionary and introducing a consolidation rule to transfer this correction into the base weights, the authors offer a substantive architectural idea. This is a principled and distinct perspective compared to existing Test-Time Training (TTT) or Fast Weight Programmer methods, which typically treat states as separate, transient memory.

Strength 2:

The online MQAR experiment provides direct, clear evidence for the proposed mechanism in a controlled setting. The results showing that FOLD with forward consolidation learns new groups on first exposure and retains 94.4% of old associations strongly support the claim that this specific consolidation design can alleviate the write/retain trade-off in associative recall.

Critical weaknesses

Weakness 1:

The online evaluation is limited to a synthetic associative-recall task, while the language-modeling benchmarks evaluate models trained offline. The current experiments therefore do not establish whether forward consolidation supports continual learning from natural-language streams or preserves a language-model-pretrained checkpoint's existing capabilities. In addition, the online comparisons do not include a method that updates persistent model parameters through a comparable mechanism. These limitations leave the broader continual-learning claims insufficiently substantiated.

Weakness 2:

Equation (10) is described as a gradient step on the objective in Eq. (9), but the residual is evaluated using the decayed previous correction. This appears to implement a decay-then-update procedure, which requires a more explicit derivation. Moreover, the paper does not report measured throughput or peak-memory comparisons that account for the cost of online updates and sequence-level consolidation. These omissions limit the mathematical clarity and practical efficiency claims of the proposed method.

Questions for authors

Question 1—Update-rule clarification:

In Section 3.1, Eq. (10) is described as a gradient step on the objective in Eq. (9), but its residual uses the decayed previous correction, �0��0�, rather than directly evaluating the residual at �1��1�. Is this intended to be a decay-then-update rule? Please clarify the derivation and the exact objective or update procedure being optimized.

Question 2— Continual-learning capability and scope:

Have the authors evaluated whether a language-model-pretrained FOLD checkpoint retains its language-modeling performance after processing a stream of new data through forward consolidation? If not, how should the broader claims about continual learning and avoiding catastrophic forgetting be scoped in light of the current evaluation on synthetic associative recall?

Question 3 — Scope of the online-learning comparison:

The current online-learning comparison focuses on state-clearing and state-carrying baselines. Are there applicable online or continual-learning methods that also incorporate new information into persistent model parameters and could serve as meaningful comparisons in this setting? If not, what makes such a comparison unsuitable?

Question 4 — Measured efficiency:

Can the authors report measured throughput and peak memory usage under a consistent hardware and evaluation setup, distinguishing standard inference from forward online learning and including the overhead of dictionary updates and sequence-level consolidation? This would help establish whether the proposed efficiency advantages hold in practice.

Reviewer_AK9cOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes FOLD, a novel deep architecture that supports efficient offline training via backward propagation and online adaptation via forward propagation. Building on Token Statistics Self-Attention (TSSA), the proposed architecture introduces online dictionary updates that can be consolidated into the model weights, enabling continual adaptation to newly observed sequences without backprop. Experiments show competitive performance compared to SOTA linear-time recurrent deep architectures such as Gated DeltaNet and demonstrate its online learning capabilities with its forward propagation mechanism while largely retaining previously learned associations on a continuous associative recall task.

Critical strengths
  • The proposed architecture is mathematically well motivated, derived from a principled online dictionary-learning objective.
  • The OnlineTSSA operator is compared to conceptually related methods, providing useful insights into the proposed learning mechanism.
  • The experiments demonstrate that the proposed forward learning mechanism enables the network to adapt to new input sequences while largely retaining previously learned associations, without requiring backpropagation.
Critical weaknesses
  • Experiments illustrating online learning capabilities of FOLD lack additional comparisons with alternative approaches to continual learning. Gated DeltaNet does not support persistent adaptation of its model weights. Comparisons with other online adaptation or memory consolidation approaches would help establish the practical advantages of FOLD.

  • Online-learning experiments are restricted to a synthetic associative recall task, but do not yet demonstrate that FOLD can adapt to more general distribution shifts.

  • The experiments lack comparisons of computational efficiency, e.g., training and inference throughput, memory requirements, or the computational cost of online adaptation. This would be valuable to assess the practical computational efficiency of FOLD, particularly the computational savings of its forward learning mechanism compared to conventional gradient-based online adaptation.

  • FOLD introduces additional hyperparameters, particularly the consolidation rate η\eta and the carry-over fraction ρ\rho, which may be difficult to tune in practical online-learning settings. An analysis of the sensitivity to these parameters would be valuable, especially since suitable hyperparameters may depend on the characteristics of the incoming data stream.

Questions for authors
  • Why does FOLD-MLP perform slightly better than pure FOLD after offline training? How would FOLD-MLP perform in the online learning experiments, where only the OnlineTSSA block adapts online while the MLP weights remain fixed? Such a comparison would help clarify whether replacing the MLP with a second OnlineTSSA operator provides additional benefits for online learning, given that it does not appear to improve performance in the offline experiments.
Reviewer_BAFjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper solves the problem of expensive back propagation during online learning. Thus the paper proposes Forward Online Learning Deep Networks (FOLD) which performs backpropagation for offline learning and forward propagation during online learning. FOLD is based on OnlineTSSA which extends the existing Token Statistics Self-Attention (TSSA) method by adding a fast correction to each learned dictionary. The paper provides theoretical justification to support the method.

Critical strengths
  • The fast correction is done directly to the existing weights of the model. This is done without any backpropagation. Baseline models can’t do this and this is the Novelty of FOLD.
  • Fold demonstrated evidence of learning retention without forgetting on one dataset. FOLD retains 94.4% of old pairs against 48.9% for baseline (Gated DeltaNet).
Critical weaknesses
  • The title, opening of the abstract, and conclusion frame the contribution in terms of deep networks and pretrained models in general. But the method is a specific Transformer-style sequence architecture (FOLD), and forward learning works only in networks built from its OnlineTSSA layers, not in standard Transformers or other existing pretrained models. The authors need to position their paper fairly without overclaiming.
  • The main novelty of the paper is forward propagation, however, it is tested only on a single small-scale synthetic experiment (MQAR, 2-block width-256 networks). The full FOLD architecture is never tested online, and there is no language-modeling or distribution-shift experiment.
  • The baselines in the paper are not sufficient. The paper compares against Gated DeltaNet. Gated DeltaNet does not support storing the learned information and thus showing FOLD performance of 94.4% vs. 48.9% of Gated DeltaNet is unfair. It does show FOLD’s retention of information without forgetting. But it does not quantify that FOLD is a superior online learning. The authors should compare against methods like test-time fine-tuning with BP or LoRA, replay, etc.
  • No experiment reports multiple seeds, variances, confidence intervals or significance tests. This matters most for Table 1, where averages span only 50.1–51.4 and per-task differences are very small (1-2), so the claim that FOLD “achieves comparable or superior performance to the state-of-the-art TTT models” (Sec. 4.1) is not established. Moreover, the full FOLD underperforms Gated DeltaNet on 5 of 9 tasks, has the lowest WinoGrande score of all models (49.6), and is best only on SciQ. The best average comes from FOLD-MLP, which retains a standard MLP as opposed to FOLD’s claim of “OnlineTSSA operator is the only basic operator needed to build a Forward Online Learning Deep (FOLD) architecture, replacing not only the attention operator in a Transformer, but also the multi-layer perceptron (MLP)”.
Questions for authors

Clarifications:

  • Please address the points raised in Critical Weaknesses.
  • The text says the updated U₀ is used only in the encoder, while codes, residuals and routing still use the pretrained U₀. Does the model therefore keep two copies of U₀? Algorithm 1 does not reflect this.
  • Each OnlineTSSA head holds a 1536×256 state, about 12× a DeltaNet head’s, and the sparse layer has 45 states. Can you report throughput and memory against Gated DeltaNet to support the claim of “same complexity as TTT models”?
  • Why are Gated DeltaNet-2 and Gated KalmaNet, both cited in the paper, not included as baselines?
  • The paper doesn't discuss any limitations or future works. The only acknowledgment is that the forward-learning mechanism is “still rather primitive” in the conclusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.