acceptodds
Under review as a conference paper at ICLR 2027

Rescuing Points Trapped Inside Incorrect Decision Regions: A target for Evaluating and Improving Forward-Forward Learning

Abstract

Forward-Forward learning replaces end-to-end backpropagation with local, layer-wise training objectives. Its variants are evaluated primarily by classification accuracy, often measured with linear probes trained on their representations. We show that this evaluation can over-credit the learning method: a linear probe on the representations of the same convolutional architecture with random, untrained weights already reaches accuracy well above chance. Moreover, a single accuracy number does not reveal which datapoints benefit from training and which remain misclassified. To examine what learning contributes beyond this random baseline, we present an analysis methodology built on the Boundary Proximity Index (BPI), a multiclass margin normalized by a class-centroid reference. We apply our analysis to seven Forward-Forward variants and related layer-local methods and find that initially misclassified points near the probe’s decision boundary generally have higher rescue rates than those farther inside incorrect decision regions. Rescue rates vary substantially across methods, with some correcting a substantial fraction of these distant points. Together, these analyses offer a way to evaluate representation learning beyond accuracy and identify the rescue of points far inside incorrect decision regions as a target for improving local learning methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
forward-forward, local learning, backpropagation-free training, representation learning, linear probes, random features, architectural prior, evaluation, diagnostics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_STgNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The authors propose a new way, called BPI, to evaluate the representation learned by various forward-forward algorithms/backpropagation. In comparison to linear probing which only provides accuracy, BPI quantifies the geometry of misclassified points for a linear probe. A more negative BPI means that the point is more on the side of misclassified class centroid than the side of true class centroid. The authors primarily analyze the distribution of rescue rate (i.e. the proportion of misclassified points that become properly classified after training) at different BPI. The authors find (1) training mostly rescues misclassified points with less negative BPI; (2) local methods that gain more accuracy compared to the untrained network also rescue more; (3) BP rescues gradually over depth, whereas local methods rescue mostly at early layers.

Critical strengths

The rescue rate distribution at different BPI offers a direct description on how representation changes over training and across layers. Authors make an extensive null hypothesis check to isolate the effect of learning algorithms.

Critical weaknesses

The authors claim that they develop a new method to evaluate representation learning and a rescue rate measure as a target to improve local learning. But the benefit of this method is not clear. Specifically:

  1. It is unclear when using BPI rescue/break rate would be more beneficial than simply using linear probes. As the author points out (line 261-263), rescue rates are consistent with linear probing performance gain. Also, by linearly probing each layer, we could also observe the same conclusion in 2.5: local learning is known to have better decoding accuracy in early layers but struggle to stack.
  2. It is unclear how the proposed BPI rescue rate could guide better design for improving local learning methods. In particular, BP also shares the trend of better rescuing less negative BPI inputs. And rescue rate for low BPI is not a differentiable target. For example: Yerxa et al., (NIPS 2023) uses manifold capacity (which quantifies the number of object categories that may be represented in a linearly separable fashion) as an objective function to train a network.
Questions for authors

Questions:

  1. Hyperplanes defined by the linear probes might be heavily influenced by hyperparameters like weight decay and learning rate. How do they influence the result?
  2. What is the motivation of evaluating pretrained VGG16? What can we conclude from evaluating this model?
  3. All experiments are performed on convnets and forward-forward local learning methods. Do the conclusions hold when evaluated on other architectures like simple MLP? And other layer-local learning methods, such as greedy layerwise learning (Belilovsky et al., ICML 2020)? I think it is more meaningful to compare different types of local learning instead of different forward-forward algorithms.
  4. SCFF is fully self-supervised, whereas methods like ASGE is supervised training. There are also other local self-supervised learning: CLAPP (Illing et al., NIPS 2021), LPL (Havagal and Zenke, Nature. Neuro 2023). Are there differences between supervised and self-supervised learning?
5 comments

We thank Reviewer STgN for the constructive feedback. New experiments and revisions are in blue; details and limitations are in Apps. J–M.

W1. What does the protocol add beyond linear-probe accuracy? The protocol uses linear probes, but adds three things that probe accuracy, at one layer or at every layer, does not give.

(1) A layer's own contribution (Fig. 1). Probing each layer of a trained network credits that layer with everything it inherits from the trained layers before it. Our layer-wise ablation instead trains only the first mm layers and keeps the rest at their initialization, isolating what training each layer adds beyond its trained predecessors. To our knowledge, such an ablation has not been used to evaluate layer-wise learning methods.

(2) A pointwise answer. Accuracy says how many errors training fixes, not which ones. Conditioning rescue on each error's initial BPI shows what makes a point hard to fix: how deep it lies inside an incorrect region.

(3) Changes that accuracy nets out. Net rescue is exactly the accuracy change, and we agree that Section 2.5 is therefore an accuracy view; we have revised it accordingly. Separating rescue from breakage shows what the net hides: our HFF re-implementation rescues 14.9% and breaks 12.5% of the CIFAR-10 test set at layer 3, a net gain of only 2.4 points. At matched probe accuracy (App. J), BP and DGL rescue fewer errors with BPI <−0.5<-0.5 than SCFF and break fewer correct examples with BPI ≥0.5\geq0.5, which accuracy cannot show.

W2. How can it guide method development? Rescue is an evaluation target, not a proposed differentiable training loss. The diagnostic can identify concrete goals and assess whether a design change achieves them. We have not demonstrated a new algorithm derived from this diagnostic. The cited manifold-capacity objective is complementary: our protocol can evaluate the representations it produces. We also agree that preferential rescue near the boundary is generic, occurring with BP and untrained controls; the contribution is to quantify what training adds beyond those controls, not to interpret the shared trend as a local-learning-specific failure.

Q1. Sensitivity to hyperparameters (App. K, Table A.16). On SCFF's untrained networks, we tested nine probe variants spanning weight decay, learning rate, dropout, and training duration. The qualitative BPI–rescue relationship persists under every tested variant: near-boundary errors are rescued more often than deeply misclassified errors, although band membership and rescue rates change. App. K gives the details and coverage limits.

Q2. Motivation for pretrained VGG16. It provides a reference trained by end-to-end BP on a large dataset and anchors the ImageNet-1K comparison. It tests whether patterns in BP trained on smaller datasets extend to pretraining and supports the null controls (Section 2.4). On CIFAR-10, its deep rescue does not differ significantly from VGG16 trained from scratch (Table A.10).

Q3. (Apps. L and J). We added a three-hidden-layer ReLU MLP on CIFAR-10, trained from the same initialization per seed with BP, DGL (Belilovsky et al., 2019; 2020), and supervised FF over three seeds, under a protocol committed before the runs. For BP and DGL, near-boundary errors are rescued more often than those with BPI <−0.5<-0.5 at every layer, as under a second untrained initialization, and both rescue more than that null in every band and layer. At matched accuracy, BP vs. DGL on deep errors (BPI <−0.5<-0.5) is inconclusive with three seeds, with estimates favoring DGL; FF failed the protocol's minimum-gain criterion and is not analyzed (App. L). On SCFF's architecture, DGL rescues fewer deep errors than SCFF at matched accuracy (App. J). We agree that comparing types of local learning is more informative: the study now covers goodness-based FF, layer-wise cross-entropy, greedy auxiliary-classifier training (DGL), and two self-supervised local methods (App. M), without establishing universality.

Q4. (Apps. J and M). We added CLAPP-s and LPL on STL-10, with SCFF as a reference (App. M). In all three, as for BP and DGL (App. L), rescue rises toward the boundary at every layer, as it does under their untrained nulls, so this trend does not separate supervised from self-supervised learning. At matched accuracy the result is as on CIFAR-10 (App. J): BP rescues fewer deep errors (BPI <−0.5<-0.5) and breaks fewer correct examples far from the boundary than SCFF or our LPL instance (native 58.1–59.5% vs. 63.2% published), though some comparisons are inconclusive; CLAPP-s could not be matched at every seed. DGL, a supervised local learner, also rescues fewer than LPL, and than SCFF on CIFAR-10, so locality alone does not explain this. Objectives, recipes and data budgets also differ, so these comparisons do not isolate supervision; we report method-specific differences, not a general claim about supervised versus self-supervised learning. App. M gives every outcome and deviation.

Reviewer_STgNOpenReview ↗

Further comments on weakness

I thank the authors for their detailed response and extensive additional simulations. My concerns for the questions are properly addressed. Meanwhile, I still have a few comments regarding weakness and writing of the paper:

Weakness 1: I agree that the per-layer contribution ablation is useful for isolate the contribution of local learning algorithm from architecture. This is meaningful because the analysis could change the benchmark comparisons of different local learning algorithms, especially when they use different archictectures. For BPI, which is claimed as an evaluation method, it offers limited insights on local algorithms. I acknowledge that it provides pointwise information and breaks down net accuracy, but it does not tell me about strengths or limitations of particular local algorithms or local learning in general (at least among the algorithms analyzed in the paper). Specifically, for local learning practitioners, why should they also examine BPI in addition to accuracy? For local learning researchers, can BPI be used to justify one method over the other ones? The impact of BPI would be clearer if the authors could use it to make evaluations on the algorithms studied the papers.

Weakness 2: The authors also agree that the paper does not demonstrate how BPI could be used as a target to improve FF/local algorithms. To me, it is neither clear nor promising that BPI could help to design better algorithms. Rather, I would consider it as a potentially useful evaluation method for analyzing the FF/local algorithms in classification tasks. As a result, it would be misleading to define BPI as a target for improvement in the title or abstract, which could give the readers a wrong expectation before reading in detail.

Appendix Writing: The current appendix is difficult to read. I understand the discussion period is too short. If the paper is accepted, I would kindly ask the authors to clean the appendix by (1) careful proofreading to remove weird phrases like "(November 9, 11:50 UTC)" in line 4292; (2) instead of directly flushing paragraphs of details, first summarizing in a few sentences the primary take-away messages for each table and appendix experiment.

W1 (what BPI says about the algorithms). We agree that the usefulness of the analysis should be demonstrated through concrete evaluations of the studied methods. Our results identify differences that aggregate accuracy does not reveal:

  1. Different local learners at matched accuracy (App. J, Tables A.13–A.14). On the same architecture and from the same initialization per seed, SCFF and DGL have layer-2 probe accuracies of 76.7% and 76.6%, but rescue 14.7% and 9.1% of the untrained network’s errors with BPI < −0.5. Conversely, DGL breaks fewer initially correct predictions far from the boundary (BPI ≥ 0.5: 1.2% vs. 2.2%). Both differences have the same sign at layers 1–3, with the 90% intervals for deep-rescue differences and the 95% intervals for far-boundary breakage differences excluding zero. As both methods are layer-local, locality alone does not explain these differences. At these matched checkpoints, SCFF corrects more deep errors, while DGL better preserves initially correct predictions far from the boundary; neither dominates on both criteria.
  2. Correction and preservation differ across methods (Apps. J, M). Matched-accuracy comparisons with BP likewise show greater deep-error rescue and greater breakage far from the boundary for SCFF on CIFAR-10, with a similar pattern for SCFF and our LPL instance on STL-10, although some individual comparisons are inconclusive. These are differences between the tested methods and training recipes; they do not establish a general distinction between supervised and self-supervised learning.
  3. A small accuracy gain can conceal substantial changes (App. J, Table A.12). At layer 3, our HFF re-implementation rescues 14.9% of the CIFAR-10 test set while breaking 12.5%, yielding only a 2.4-point net gain. Separating rescue and breakage reveals this cancellation; conditioning them on BPI further identifies where corrections and failures occur relative to the initial boundaries.

For practitioners, these analyses are useful when the evaluation goal includes correcting errors far inside the untrained network’s incorrect decision regions or preserving its correct predictions. For researchers, they distinguish methods that achieve similar accuracy through different patterns of correction and breakage. They support comparisons against a stated criterion, rather than an unconditional ranking or a claim that greater deep rescue is always preferable.

We have foregrounded these method-specific findings in Section 2.6 and clarified the distinct roles of the analyses: the layer-wise ablation measures what training each layer contributes, rescue and breakage separate changes that net accuracy combines, and BPI locates those changes relative to the reference probe’s decision boundaries.

W2 (title and abstract). We agree that presenting BPI as a target for improving local learning suggests an algorithm-development contribution that we have not demonstrated. We have revised the title and abstract to present BPI as an evaluation tool. The revised title is “Rescuing Points Trapped Inside Incorrect Decision Regions: Evaluating Forward-Forward Learning Beyond Accuracy.”

Appendix Thank you also for the suggestions on the appendix. We have removed operational timestamps, clarified the prose, and added takeaway summaries before experimental details and in table captions.

Reviewer_jT1dOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an evaluation framework for Forward-Forward learning beyond classification accuracy. It introduces the Boundary Proximity Index, a centroid-normalized multiclass margin, to analyze samples that change from incorrect to correct classification (rescue) or vice versa (breakage), both across training and network depth.

The authors evaluate seven layer-local learning methods and backpropagation on CIFAR-10, CIFAR-100, Tiny ImageNet, and exploratory ImageNet-1K experiments. They find that untrained CNNs already provide substantial linear-probe accuracy, near-boundary samples are more frequently rescued, and several FF methods benefit less from additional depth than backpropagation. Validation includes random-network baselines, layer-wise ablations, and three null controls.

Critical strengths
  1. Comprehensive evaluation with meaningful controls. The paper evaluates seven layer-local learning methods against random-network baselines and backpropagation. Layer-wise ablations and null controls help distinguish learning-induced improvements from random representations and probe variability.

  2. Informative sample-level analysis. Rescue and breakage rates reveal how individual classification outcomes change beyond aggregate accuracy. The authors acknowledge that the relationship between boundary proximity and rescue probability exists without learning.

Critical weaknesses
  1. Limited methodological novelty and insufficient validation of BPI. Several components have established precedents: "On Random Weights and Unsupervised Feature Learning" studies random-network baselines; "An Empirical Study of Example Forgetting During Deep Neural Network Learning" examines correctness transitions and classification margins; and "Deep Learning Through the Lens of Example Difficulty" investigates sample difficulty across depth. "Sample Margin-Aware Recalibration of Temperature Scaling" and "Geometry-Lite: Interpretable Safety Probing via Layer-Wise Margin Geometry", study margin-based difficulty and layer-wise geometry. Although BPI's normalization and its application to FF rescue analysis appear distinctive, comparisons against standard signed margins, geometric margins, and alternative normalizations are missing, leaving its added value unclear.

  2. Limited novelty and interpretation of the main findings. The higher rescue probability of near-boundary errors is consistent with standard margin reasoning and occurs in the authors' untrained controls. Although trained-versus-random comparisons are informative, the paper does not establish whether rescue differences reflect difficulty-specific behavior beyond overall accuracy differences. Moreover, "The Price of Locality: Why Forward-Forward Underperforms Backpropagation?" investigates FF's depth-related limitations. The additional insights beyond established margin behavior and local-learning limitations remain insufficiently demonstrated.

Questions for authors
  1. What advantage does BPI provide over standard classification margins? Could the authors compare BPI with the classification margin used in "An Empirical Study of Example Forgetting During Deep Neural Network Learning" under the same rescue-and-breakage analysis to demonstrate the benefit of centroid normalization?

  2. Do rescue differences reflect difficulty-specific behavior beyond overall accuracy? Could the authors compare methods at similar accuracy levels or control for overall performance to determine whether certain methods have distinctive difficulty-specific rescue behavior?

1 comment

We thank Reviewer jT1d for the constructive feedback. The additional analyses and revisions are highlighted in blue.

W1/Q1.

Our contribution is an evaluation protocol for forward layer-wise learning built around established ingredients; we do not claim that BPI is superior to existing margins. The protocol examines what local learning adds beyond matched random representations and how these gains are distributed across examples and depth. Although prior work has shown that random filters can provide useful features, this baseline is often overlooked when evaluating forward layer-wise learning methods. Our protocol addresses this gap. Furthermore, the ablation in Fig. 1 compares training layer ℓ\ell against leaving it randomly initialized while layers 1,…,ℓ−11,\ldots,\ell-1 are trained. This isolates the contribution of training that layer from benefits inherited from earlier layers. To our knowledge, this component has been missing from the forward layer-wise learning literature.

We repeated the rescue-and-breakage analysis using Toneva et al.'s margin, m=fy−max⁡k≠yfkm=f_y-\max_{k\neq y}f_k, and the geometric margin m/∥w∥2m/\lVert w\rVert_2, keeping the refitted probe fixed within each comparison (App. I, Table A.7). BPI provides no average predictive advantage over these alternatives. However, the main qualitative finding is unchanged: near-boundary errors are rescued more often than deeply misclassified errors in every evaluated condition, across training and depth. On CIFAR-10 at layer 3, equally sized margin bands preserve the network ranking by near-band rescue, and Table A.2's near-band rates move by −0.2 to +1.2 pp. The deep band is index-specific: it overlaps the equally large set of most negative margins only partly (Jaccard 0.27–0.45), and 3 of ten deep-band rates are significantly lower under margin bands (HFF, layer 3: −5.6 pp). Thus, standard classification margins can also be used within our protocol. We have clarified in the manuscript that we do not claim predictive superiority for BPI.

BPI provides an interpretable reference scale: BPI =−1=-1 places an error at the same distance from the selected hyperplane as the predicted class's refined centroid, on the incorrect side. A margin threshold would have to be re-chosen per condition: the median ∣m∣|m| of correct points ranges from 0.40 to 13.4 logits across our CIFAR-10 conditions. This relative scale facilitates comparisons across depth and before and after training, although equal BPI does not imply equal difficulty.

App. G now clarifies the connections to the cited work (Saxe et al.; Toneva et al.; Baldock et al.; SMART; and Geometry-Lite).

W2/Q2.

We agree that preferential rescue near the boundary is generic, as our untrained controls demonstrate. Deep-error rescue strongly tracks accuracy gain across network–layer observations (r=0.92r=0.92, 0.940.94, and 0.970.97 on CIFAR-10, CIFAR-100, and Tiny ImageNet; these observations are dependent). A pooled probit model controlling for this gain finds no robust BP effect. Among the original networks matched within one pp in both initial and final accuracy, the four CIFAR-10 local–BP pairings show no significant deep-rescue differences, although their intervals do not establish equality. We therefore narrow claims about method-specific behavior (App. J).

A controlled comparison on SCFF's architecture nevertheless shows that similar accuracy can conceal different prediction changes. Across three seeds, BP variants and greedy layer-wise DGL rescue 4.1–7.4 pp fewer errors with BPI <−0.5<-0.5 than SCFF, while breaking 1.0–1.7 pp fewer correct examples with BPI ≥0.5\geq0.5. Checkpoints were matched to SCFF's probe test accuracy within one pp; under our protocol committed before the runs, the verdict is "fewer" at layer 1 and inconclusive at layers 2–3. Two BP controls matched at the end of training agree in direction but were added during the runs, one after a first look at early results. However, absolute between-band contrasts do not consistently establish that the deficit is specific to far-boundary errors; three-seed uncertainty remains, and DGL's locality rules out attributing the difference to locality alone. App. J distinguishes prespecified analyses from exploratory controls.

Beyond margin behavior, our analysis exposes offsetting rescue and breakage that accuracy hides. For example, our HFF re-implementation at CIFAR-10 layer 3 rescues 14.9% and breaks 12.5% of the test set: a net gain of only 2.4 pp, with breakage 4.0 pp above a second random initialization. On depth, our finding agrees with the concurrent The Price of Locality, now cited. Section 2.5 is now an accuracy view (net rescue equals the change in probe accuracy), and its depth conclusion is restricted to ASGE's architecture, where local and BP training share architecture and initialization. The abstract, findings, and conclusion have been qualified accordingly.

Reviewer_JsLBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates what Forward-Forward and related layer-local learning methods contribute beyond the representations already supplied by randomly initialized convolutional networks. It proposes an evaluation methodology combining architecture-matched untrained baselines, layer-wise training ablations, and pointwise analyses of classification changes. Experiments cover seven local-learning methods, backpropagation baselines, and primarily CIFAR-10, CIFAR-100, and Tiny ImageNet, with an exploratory ImageNet-1K analysis.

Critical strengths

The paper is well written. The core idea is well motivated, and easy to follow. The experiments are comprehensive and results are promising.

Critical weaknesses
  • The quantitative conclusions depend on a probe and normalization protocol whose robustness is only partially validated. BPI uses centroids estimated from correctly classified test examples, so its scale depends on evaluation-set composition and the number of correct examples per class.
  • The main analysis concatenates features from all preceding layers, potentially concealing deterioration in later representations. Appendix F usefully checks individual-layer probes, but only for one seed and the first six layers, with some configurations missing.
Questions for authors
  • Do the rescue curves remain stable when centroids are estimated from training or held-out validation data? How are classes with no correctly classified examples, or very small centroid margins, handled?
  • Apart from the explicitly qualified ImageNet HFF experiment, how closely do the reproduced models match their originally reported native-readout performance?
1 comment

Thank you for your comments. The revisions are highlighted in blue.

Q1/W1.

We repeated the analysis using training-set and held-out validation centroids (App. I, Tables A.8–A.9, Fig. A.43). The main qualitative relationship persists in the tested settings: errors near the boundary are rescued more often than those farther into the incorrect region. Held-out centroids produce small changes in band-level rescue rates on CIFAR-10 and Tiny ImageNet. Training-set centroids also have little effect on CIFAR-10 (deep-band rescue changes ≤1.9 pp over five networks), but on SCFF's CIFAR-100 and Tiny ImageNet networks they shrink the deep-error band and lower rescue in every band, consistent with probe overfitting there (train−test accuracy gaps of 28–74 pp vs. 9–18 pp on CIFAR-10). Deep-band membership and rescue rates, however, depend on the centroid source.

Held-out centroids, the direct test, were run on SCFF's untrained CIFAR-10 and Tiny ImageNet networks, which are also the starting point of BP on SCFF's architecture; no other network was tested this way. Stable rescue rates do not imply stable band membership: on Tiny ImageNet, the held-out and test-centroid deep bands contain substantially different examples. We also varied probe hyperparameters (App. K); near-boundary errors remained more frequently rescued under every tested variant.

For classes with no correctly classified reference examples, the implementation falls back to the mean of all reference examples of that class; App. A now states this explicitly. This fallback is never needed on CIFAR-10; on CIFAR-100 and Tiny ImageNet it affects at most 18 of 100 and 107 of 200 classes, both at layer 1 of the untrained VGG16. The denominator is clipped at 10−810^{-8}, although this bound was not reached in the refitted probes we could audit. Small centroid margins can nevertheless inflate ∣BPI∣|\mathrm{BPI}| and affect deep-band membership. App. I now quantifies this sensitivity: removing examples associated with the smallest centroid margins generally lowers deep-band rescue in the tested networks.

W2.

We agree that the original individual-layer check had limited coverage. App. F now extends the rescue analysis (not breakage) to three seeds for five CIFAR-10 networks, adds a three-seed accuracy comparison for SCFF, and evaluates all layers of ASGE and pretrained VGG16 individually on CIFAR-10; Fig. A.42 maps the coverage of every method–dataset–layer cell. The qualitative increase in rescue toward the boundary persists throughout these comparisons. We find no statistically significant increase in deep-error rescue from concatenation, although the intervals are wide and each read-out defines its own initial errors.

Concatenation can nevertheless conceal weaker individual-layer performance. For pretrained VGG16 evaluated on CIFAR-10, late-layer accuracy declines when layers are read individually while concatenated accuracy continues to improve. BP on SCFF's architecture also has a weaker final layer when read alone, but its native classifier uses concatenated features, so this comparison evaluates a read-out it was not trained to use. Other networks show no clear decline, or one under both read-outs. We now distinguish cumulative predictive information from the quality of an individual layer and restrict the depth conclusion in Section 2.5 to the concatenated read-out.

Q2.

New App. H (Tables A.5–A.6) reports native-readout performance against the closest published results and documents implementation and training differences. On CIFAR-10, our SCFF, HCL-FF, Mono-Forward, and ASGE models fall approximately 0.8–3.5 pp below the published values. Elsewhere gaps are larger: Mono-Forward on Tiny ImageNet (−3.9 pp) and our 100-epoch ASGE on CIFAR-100 (−6.5 pp). Comparable native-readout measurements remain unavailable for DeeperForward and HFF on CIFAR-10.

For ASGE, rerunning the authors' longer training recipe brings native accuracy within 0.5 pp of the published results on CIFAR-10 and CIFAR-100 and substantially increases deep-error rescue. Our original shorter run therefore understates ASGE's rescue capability. Its cumulative gain over the untrained baseline still plateaus after the early layers under the concatenated read-out. For HFF, our probe at the last layer reaches 66.4% against the published 83.1%, so we restrict conclusions to our simplified re-implementation, including on CIFAR-10.

The revision also documents SCFF's released-script defaults differing from published hyperparameters, Trifecta's omitted overlapping updates and test-based checkpoint selection (worth +0.30 pp on CIFAR-10 in validation-selected re-runs), validation holdouts used by several networks, and Mono-Forward's MLP-Mixer architecture on Tiny ImageNet. These disclosures clarify which results characterize close reproductions and which characterize modified trained instances.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their constructive feedback. New analyses and revisions are highlighted in blue (new Apps. G–M; expanded Apps. A and F).

The gap we address. Evaluations of forward layer-wise learning often emphasize linear-probe accuracy after training. Yet probes fitted to the same architectures with random weights already achieve accuracy far above chance—58–74% on CIFAR-10 in our experiments. Although random-feature baselines are well established (Jarrett et al., 2009; Saxe et al., 2011), they are often overlooked in this literature, so reported accuracy mixes learning with what the architecture already provides. Our protocol therefore (i) compares trained and architecture-matched untrained networks under the same probe-fitting protocol, and (ii) isolates the contribution of training layer ℓ\ell by comparing it with its random initialization while all preceding layers are trained (Fig. 1).

Which datapoints learning changes. Accuracy reports the net balance of corrected and newly introduced errors, but not which examples change. We use the Boundary Proximity Index (BPI) to express an example's signed margin to the selected class-pair hyperplane relative to a class-centroid reference. We then measure rescue and breakage as functions of initial BPI, alongside untrained and probe-noise controls.

Case study: SCFF on CIFAR-10. At layer 3, the untrained network achieves 74.2% probe accuracy, increasing to 78.4% after training. Evaluated at this depth, training layer 1 contributes 3.2 points, layer 2 adds 1.0, and layer 3 adds essentially none (Fig. 1). The rescue analysis shows which errors are corrected: training rescues 50.7% of near-boundary errors, compared with 33.6% under a second random initialization, but only 7.1% of errors with BPI <−1<-1 (null: 0.9%). Thus, random features supply most of the accuracy, later layers add little, and near-boundary errors are far more likely to be corrected than deep ones. This does not mean that representation changes are confined to these examples or that the pattern reflects locality: at similar accuracy, BP and DGL rescue even fewer errors with BPI <−0.5<-0.5, while also breaking fewer correct examples far from the boundary (App. J).

What the new analyses add.

  • Margins (App. I). BPI provides no average predictive advantage over raw or geometric margins. Its role is an interpretable reference scale. Near-boundary errors remain more frequently rescued under the alternative margins, although deep-band membership and rescue rates depend on the index.
  • Robustness (Apps. F, I, K). The near-boundary rescue relationship persists under the tested probe-hyperparameter variants, alternative centroid sources, and individual-layer probes. Numerical rates and band membership can change. Each appendix states the scope and limitations of its checks.
  • Beyond aggregate accuracy (App. J). Deep-error rescue strongly tracks accuracy gains across methods. Nevertheless, matched-accuracy comparisons reveal different patterns of rescue and breakage, as in the SCFF comparison above. Separating these transitions also exposes offsetting changes: our HFF re-implementation rescues 14.9% and breaks 12.5% of the CIFAR-10 test set at layer 3, yielding only a 2.4-point net gain. App. J documents test-accuracy matching, uncertainty, and exploratory controls.
  • Broader coverage (Apps. J, L, M). We added BP and DGL on a plain MLP, DGL on SCFF's architecture, and CLAPP-s and LPL on STL-10. The near-boundary relationship persists in the analyzed settings. Supervised FF on the MLP failed the minimum-gain criterion and was not analyzed. On STL-10, matched BP rescues fewer deep errors and breaks fewer correct examples far from the boundary than SCFF or our LPL instance, although some comparisons remain inconclusive under the prespecified criterion. CLAPP-s could not be matched across all seeds. These comparisons do not isolate supervision from other methodological differences.
  • Reproduction fidelity (App. H). New tables compare native read-outs with published results and list every deviation. ASGE's longer training recipe approaches its published accuracy and increases rescue. Conclusions concerning imperfect reproductions, including HFF and LPL, are explicitly qualified.

Revised claims. We revised the abstract, findings, Section 2.5, and conclusion (App. G). Section 2.5 is explicitly an accuracy view: net rescue equals accuracy change, and its depth conclusion is restricted to ASGE's architecture and the concatenated read-out. Rescue is an evaluation criterion, not a proposed training loss. Our contribution is a protocol for identifying what learning adds, where it helps, and where corrections are offset by deterioration—not a claim that BPI is predictively superior or that the observed relationship establishes a difficulty-specific limitation unique to local learning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.