acceptodds
Under review as a conference paper at ICLR 2027

Forward-Forward Learning for Regression

Abstract

The Forward-Forward (FF) algorithm offers a computationally efficient and biologically plausible alternative to backpropagation (BP) by training neural networks through purely local, layer-wise optimization. However, FF is inherently designed for classification via contrastive positive-negative sample pairs, and extending it to regression poses fundamental challenges: continuous target space lacks natural “opposites” for contrastive learning, and the standard goodness function carries no information about target magnitude or ordering. We propose FFR (Forward-Forward for Regression), to our knowledge, the first framework to extend FF to real-world regression and demonstrate competitive performance across diverse real-world datasets. FFR introduces three key innovations: (1) an ordinal competitive goodness function that replaces contrastive pairs with competitive learning between partitioned neuron groups under distance-aware ordinal supervision; (2) a stratified ladder architecture where shallow layers learn coarse ordinal discrimination and deeper layers refine into fine-grained regression, with multi-scale feature aggregation for inter-layer collaboration; and (3) hierarchical prediction with uncertainty estimation, where multi-scale predictors jointly provide robust predictions and a single-pass uncertainty score. Extensive experimental results show FFR recovers on average of BP's accuracy across five real-world regression benchmarks while reducing peak training memory to only of BP's at depth and at depth , with per-iteration time around of BP's, and substantially outperforms all BP-free competitors.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Forward-Forward Learning, Backpropagation-Free Learning, Local Learning, Regression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SppcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends Forward-Forward learning to regression through grouped-neuron competition with Gaussian soft ordinal targets. A coarse-to-fine group schedule and a terminal head aggregate information across layers without propagating gradients between them. The layer-wise predictions are also combined to produce a final prediction and a disagreement-based uncertainty score.

Critical strengths
  1. Extending local FF learning to practical regression tasks is useful. The evaluation covers six real-world datasets and multiple backbone types, and the component ablations support the proposed design.

  2. The measured memory savings are substantial. In Table 13, FFR uses 1253 MB versus 4687 MB for BP-UR, with iteration time decreasing from 186.10 to 131.81 ms.

Critical weaknesses
  1. The advantage over existing local-learning and distributional-regression approaches is not sufficiently established. Channel-wise competition without negative samples already appears in Papachristodoulou et al. (AAAI 2024), while Gaussian-target histogram regression was studied by Imani and White (ICML 2018), which is not discussed. The coarse-to-fine schedule and aggregation may be useful, but the FF-MSE baseline uses direct activation averaging rather than a learned local regression head. Its poor performance therefore does not establish that local MSE regression generally suffers from representation collapse. A stronger comparison with learned local MSE and distributional heads is missing.

  2. The comparison with BP needs stronger evidence. The reported 98.5% figure averages BP-to-FFR error ratios rather than measuring accuracy, and it masks larger gaps on individual tasks: Tool Wear MAE increases from 7.259 to 8.388. No run-to-run variability is reported. Moreover, Table 9 measures convergence relative to each method's own minimum training criterion, not the time needed to reach the same prediction error. The reported iteration-level savings are useful, but the practical training advantage at matched accuracy, including comparison with activation-checkpointed BP, remains unclear.

Questions for authors
  1. Could the authors compare FFR with learned per-layer MSE and Gaussian-target distributional heads, keeping the backbone, aggregation, and training budget matched?

  2. Could the authors report repeated-run results and wall-clock time to a common validation-error target, including a memory-efficient BP baseline?

  3. Goodness is computed after group normalization in Section 3.2. Without the learned affine transformation, each group's mean squared normalized activation is approximately one. How important are the affine parameters and their initialization to learning discriminative goodness scores?

  4. Does the uncertainty score reliably rank prediction errors on the real-world datasets? Table 12 only evaluates synthetic tasks, and the ambiguity decomposition alone does not establish an epistemic interpretation.

Reviewer_PT5ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the challenge of extending the Forward-Forward Algorithm (FFA) to regression tasks, where the conventional goodness function is not well suited to continuous target spaces. To this end, they propose Forward-Forward Regression (FFR), a framework consisting of three components: an ordinal competitive goodness function, a stratified ladder architecture, and hierarchical prediction with uncertainty estimation. Experiments on both synthetic and real-world datasets demonstrate that FFR achieves performance comparable to backpropagation while offering lower computational and memory costs.

Critical strengths
  1. Clear research problem. Extending the FFA from classification to regression is a meaningful research direction. The authors also clearly identify two key limitations of conventional FFA when applied to regression tasks.
  2. Broad experimental coverage and competitive results. The experiments cover a wide range of datasets and demonstrate competitive performance. Furthermore, the computational efficiency advantages are relatively thoroughly validated.
Critical weaknesses
  1. Appears more like an application of ordinal regression within the FFA framework. Although this paper aims to address the limitations of conventional FFA in regression tasks, the proposed method does not appear to fundamentally rely on FFA. In my view, it resembles an application of ordinal regression within the FFA framework. Moreover, the paper overlooks important related work, such as label distribution learning [1]. In addition, coarse-to-fine prediction and multi-resolution supervision have been widely explored in regression tasks, including age estimation [2]. Hierarchical prediction through the aggregation of multiple predictors is also a well-established strategy.
  2. Unclear distinction from conventional layer-wise supervised learning. The paper emphasizes that FFR extends the positive-negative goodness learning mechanism of conventional FFA to continuous regression tasks. However, as shown in Eq. (2)–(4), FFR essentially transforms each hidden layer into a bin-based supervised learning module optimized using distance-aware soft labels and cross-entropy loss. Therefore, following W1, further clarification is needed regarding whether FFR's performance improvements primarily arise from goodness-based competition or from the discretization of continuous targets and layer-wise supervision.
  3. Limited justification for uncertainty estimation. The proposed uncertainty score is derived from prediction disagreement across different network layers. However, since these predictors share the same network and training data, their disagreement does not necessarily provide a reliable estimate of epistemic uncertainty. Moreover, the theoretical analysis mainly relies on a standard ensemble ambiguity decomposition, which does not establish uncertainty calibration. Although the authors provide empirical evaluations on synthetic datasets, additional calibration experiments on real-world datasets would strengthen this claim.
  4. Lack of statistical reliability analysis. The main experimental results do not report means, standard deviations, or confidence intervals across multiple random seeds, making it difficult to assess the stability and statistical significance of the reported performance improvements.

[1] Geng X, et al. Facial age estimation by learning from label distributions. IEEE TPAMI. 2013.
[2] Yang T Y, et al. Ssr-net: A compact soft stagewise regression network for age estimation. IJCAI. 2018.

Questions for authors
  1. How does the proposed ordinal competitive goodness fundamentally differ from conventional layer-wise ordinal classification or distributional regression? Would replacing the squared-activation goodness with standard learnable logits, while keeping the same ordinal supervision and hierarchical architecture, yield comparable performance?
  2. Since predictions from different layers share the same network and training data, what justifies interpreting their disagreement as epistemic uncertainty? Could the authors provide quantitative uncertainty calibration results on real-world datasets?
Reviewer_L8SGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FFR, which extends Forward-Forward (FF) layer-local training to regression. FF has been proposed for classification problems, and it learns by contrasting positive and negative samples, but regression targets have no natural negatives. FF's goodness score ignores target size and order. This work on FFR turns each layer into an ordinal classifier: neuron groups compete over bins of the target range and are trained with Gaussian soft labels. Bins get finer with depth, and the layer predictions are averaged into a final estimate, with their spread used as an uncertainty score. On six real-world datasets, FFR reaches about 98.5% of backpropagation's accuracy, beats all backprop-free baselines, and uses 27% of backpropagation's training memory at depth 8.

Critical strengths
  1. Tackles an important and relatively unexplored problem of using local training for regression.
  2. The problem analysis and motivation are clearly established. It identifies why naive per-layer MSE loss collapses.
  3. The evaluation is clear and done across a wide range of modalities (tabular, image, text) with different architectures.
Critical weaknesses
  1. Baselines used for comparisons could be improved. Local-loss supervised training is an alternative approach that is not discussed here.
  2. Hierarchical ordinal regression is a well-explored approach that has not been cited in the paper.
  3. The accuracy improvements in some places are unclear given the small improvements and the lack of standard deviation over multiple runs.
Questions for authors
  1. The paper cites Nøkland & Eidnes (2019) only for update locking [Sec. 1, App. A.1], but FFR's per-layer supervised cross-entropy [Eq. 4, Alg. 1] is itself a local-loss method. How does FFR differ from local-loss training (Nøkland & Eidnes 2019; DGL, Belilovsky et al. 2020), and how does it compare against them using the same per-layer loss? How does it compare with backprop using gradient checkpointing for memory?

  2. There have been many works on soft-label ordinal regression. How do the ordinal goodness [Eqs. 3–4] and readout [Eq. 8] differ from HL-Gauss (Imani & White 2018; Farebrother et al. 2024), DLDL/DEX, and the coarse-to-fine bins of SSR-Net? It is important to cite prior works to differentiate between existing approaches and new ideas proposed in this work.

  3. How many seeds were run? Please report the mean and standard deviation for Tables 1–4. Are the wins over BP-UR [Table 2] and the "best in every column" ablation claim [Table 3] significant?

  4. The main backprop reference, BP-UR, trains FFR's own architecture with only a global MSE loss and no layer averaging [App. A.5]. MSE is often not the best regression loss: cross-entropy over soft bins (HL-Gauss) beats MSE in Farebrother et al. (ICML 2024). Can you report, for each benchmark:

a. A standard MLP/CNN/GRU trained by backprop; b. Backprop with stronger losses (e.g., HL-Gauss or Huber); c. The best published result, in the standard task metric?

Without these, "98.5% of BP" and "surpasses BP" [Sec. 4.2] are measured against a possibly weak reference.

  1. What happens if the test example has an out-of-range value?

  2. Each target receives only K_ℓ/D groups of at least about 4 units each [App. A.2, A.7.1]. So layer width seems to grow with the number of targets D and with the required precision relative to the target range. Can you evaluate problems with many targets or high required precision against backprop at matched width? Several regression problems, such as robotics, deal with many targets (grows with robot DOF) and precision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.