acceptodds
Withdrawn from ICLR 2027

Local-Window Contrastive Forward-Forward Learning with Masked Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

End-to-end Vision Transformer training retains a backward graph across the whole network and can become memory-prohibitive on resource-constrained hardware. Restricting backpropagation to a local part of the network reduces this burden, but contrastive Forward–Forward (CFF) mainly supervises pooled representations rather than patch-level visual structure. We introduce contrastive Forward–Forward with masked autoencoding (CFFMAE), which augments local CFF training with masked reconstruction without restoring a full-depth backward graph. At each stage, CFFMAE optimizes a small local window of encoder blocks together with a lightweight decoder, while preceding blocks are evaluated without autograd. We consider layer-wise one-block and pair-wise two-block windows. Unmasked views provide the CFF objective, and masked views provide patch-level reconstruction to the same active block. We separate the encoder input mask from the reconstruction-loss mask, transfer the decoder checkpoint across successive local windows, and backpropagate the CFF and MAE branches separately to limit temporary memory. We evaluate CFF+MAE on CIFAR-10, ImageNet-1K, and the STB hand-pose benchmark, with additional Jetson profiling against CFF-only, full backpropagation, and activation-checkpointed BP. Masked reconstruction consistently improves CFF-only training in the controlled classification experiments. On Jetson, CFF+MAE substantially reduces peak memory relative to uncheckpointed full backpropagation and requires lower per-batch latency and energy than checkpointed BP at comparable peak-memory footprints.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision Transformers, Forward–Forward Learning, Masked Autoencoding, Block-wise Local Learning, Memory-Efficient Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NSo8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses a practical problem of training under memory constraints, using a local gradient update strategy through contrastive learning. The paper shows solid motivation and technical soundness, but also has a few issues in its evaluation.

Critical strengths
  • The local training problem is real - edge-deployable models are hardly trainable on edge devices, because full BP requires too much memory. So the motivation of paper is realistic and strong.
  • Overall the proposed method has shown promising results in the evaluation, with competitive accuracy, and high memory efficiency for training. Obtaining competitive results on ImageNet-1K with CFF-based learning is a commendable achievement.
  • Sequential execution of the contrastive and reconstruction branches further contributes to the enabling of memory-efficient progressive training.
Critical weaknesses
  • The reported peak memory for stage 2 needs further clarification - the paper states "gradients must pass through the frozen suffix blocks to reach the active window", but the calculation of M_{stage2} seems to be a theoretical formulation rather than empirically recorded peak memory. Please report the actual peak memory for stage 2, and compare it against full BP.
  • Although the authors claim that the paper specifically targets CFF training of ViTs, the lack of broader comparison with other baselines such as BIM, local contrastive reconstruction, LocalMIM, makes it difficult to assess the impact of the proposed method in the local training field of research.
  • Some external baselines are taken from paper rather than reproduced (e.g. Table 2). This is not ideal.
Questions for authors

What ImageNet performance is obtained with head-only evaluation after Stage 1?

Reviewer_LZtyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper adds an MAE-style masked pixel reconstruction branch to Contrastive Forward-Forward local training. Training proceeds window by window from shallow to deep, where a window is one block (layer-wise) or two consecutive overlapping blocks (pair-wise); only the active window and its local decoder receive gradients. Three accompanying designs: decoupled input/loss masking with a depth-scheduled loss subset, decoder checkpoint transfer across windows, and sequential rather than joint backward through the two branches. A final-window attention-distillation term from a frozen MAE teacher is used at ImageNet scale.

Critical strengths
  • The gains over CFF are consistent across tasks: ImageNet accuracy improves from 45.90% to 49.17% for layer-wise training and from 63.30% to 64.75% for paired windows; STB AUC improves from 83.03% to 84.20% (p. 7, Tables 1–2). These controlled ImageNet improvements are the strongest evidence for the method's usefulness.

  • Measured peak memory changes only from 785.9 to 788.3 MiB and from 1301.4 to 1303.8 MiB after adding reconstruction (p. 9, Table 5). Preserving this low-memory footprint while improving accuracy supports a practical contribution for memory-constrained training.

Critical weaknesses
  • The increment over nearby methods is insufficiently established. Local reconstruction and its combination with local contrastive learning have precedents acknowledged in Appendix E (p. 22). Table 3 uses heterogeneous backbones, locality definitions and protocols (p. 8), limiting comparative conclusions. Table 4's additional mask/transfer gains are 0.60/0.51 points on CIFAR-10 without repeat statistics. The CFF improvement is useful, but the significance of the added design remains uncertain.

  • The joint accuracy–memory–cost case remains incomplete. Most memory savings already exist in CFF-only. Against sequential-view BP at 1839.1 MiB, the advantage is 2.33x/1.41x for layer-wise/paired training, versus the headline 4.3x against joint-view BP (p. 17, Table 8; p. 9, Table 5). Reconstruction adds 25.8%/22.2% local-update latency; complete schedules are projected at 81.40/86.84 hours versus 62.26 hours for BP (pp. 20–21, Appendix D). Table 2 supplies external accuracy references, not the authors' own BP accuracy paired with that cost comparison (p. 7). The disclosed trade-off therefore leaves practical significance uncertain. Separately, I could not find the frozen MAE teacher's identity or enabled distillation weight (pp. 5–6, 13–14). This limits reproduction of the distilled extension and Table 6's quantization results, which all use distillation; the main 49.17%/64.75% results do not (pp. 7 and 9).

Questions for authors
  • Do existing runs provide ImageNet BP accuracy paired with the reported resource measurements, or other controlled evidence showing a compelling accuracy–memory–cost advantage over CFF-only and nearby alternatives? Please distinguish this evidence from Table 3's contextual comparisons.

  • Please fully disclose the final-window MAE teacher: architecture, parameter count, pretraining data and epochs, total GPU-hours, and the value of α_t. If it was trained with full backpropagation, how should readers position the 50.20% / 64.98% results and all of Table 6 relative to the paper's memory-efficiency premise? Please also report Table 6 without any distillation.

Reviewer_4AbPOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CFF+MAE, a local window training method for ViTs to reduce peak GPU memory on edge devices. Standard CFF only applies contrastive loss on pooled features, so it misses patch level spatial details. To fix this without full backpropagation, the authors add a local MAE reconstruction branch to each training window of 1 or 2 encoder blocks. They use three main tricks to make it work well: separating the fixed input mask ratio from a depth scheduled loss mask ratio, transferring local decoder weights to the next window, and running the CFF and MAE backward passes sequentially so the two computation graphs do not stay in memory at the same time. After Stage 1 local pretraining, Stage 2 uses either linear head tuning or reverse order window fine tuning. Experiments on CIFAR-10, STB hand pose, and ImageNet-1K show consistent improvements over CFF only, and profiling on Jetson Orin Nano confirms the memory savings.

Critical strengths

First, the memory optimization is well engineered and tested on real edge hardware. Unlike many local learning papers that only give theoretical numbers, the authors profile their method on a Jetson Orin Nano and compare it against activation checkpointing (SPR). Running the CFF and MAE backward passes sequentially is a very practical idea, as Table 5 shows it adds patch reconstruction with less than 0.3 percent extra peak memory over CFF only.

Second, the accuracy gain on ImageNet-1K is solid for fine grained local ViT training. Most prior Forward-Forward methods only work on small datasets or CNNs, while here pair-wise training with window size 2 reaches 64.75 percent top-1 accuracy on a 6 block ViT, beating the end to end DeiT-6 baseline of 63.4 percent and improving over CFF only.

Critical weaknesses

First, Stage 2 on ImageNet-1K still relies on full depth gradient propagation, which weakens the local learning claim. On CIFAR-10 the encoder is frozen in Stage 2, which cleanly tests Stage 1 features. However, on ImageNet-1K, Stage 2 uses reverse order fine tuning where the loss is computed at the final output and gradients backpropagate through all the frozen suffix blocks using checkpointing. When the first window is active, gradients actually pass through the whole network. Because ImageNet results are only reported after Stage 2, it is unclear how much of the 64.75 percent accuracy comes from Stage 1 local learning versus Stage 2 end to end gradients.

Second, the 4.3x peak memory reduction is compared against a very heavy baseline, and the total training time is quite high. The 3373 MiB Full BP baseline processes two views jointly. However, Table 8 in the appendix shows that simply doing backward passes for the two views sequentially drops Full BP memory almost in half to 1839 MiB with no speed penalty. Also, standard supervised training like DeiT-6 only needs 1 view per image, which would bring Full BP memory even closer to the pair-wise CFF+MAE memory of 1303 MiB. In addition, training all the windows sequentially takes 2250 to 2700 total epochs across Stage 1 and 2, which takes about 81 to 87 hours on H200 GPUs compared to 62 hours for 800-epoch Full BP.

Questions for authors
  1. On ImageNet-1K, what is the top-1 accuracy if you only use the head-only evaluation protocol after Stage 1 (freezing the encoder like you did on CIFAR-10)? Also, what accuracy do you get if you skip Stage 1 and run your Stage 2 reverse order fine tuning directly from random initialization? This would help clarify the real contribution of Stage 1 vs Stage 2.

  2. Since DeiT-6 is a 1-view supervised baseline, what is the peak GPU memory, latency, and energy of standard 1-view Full BP and 1-view SPR on the Jetson Orin Nano at batch size 32 and 64?

  3. In Section 3.2, the CFF loss is supervised and defines positive pairs using ground truth class labels. For the STB 3D hand pose regression task, there are no class labels. How do you define positive pairs in the CFF branch during Stage 1 on STB? Does it just use the two augmented views of the same image as positives?

  4. In pair-wise training with window size 2, overlapping windows mean the middle blocks are trained twice, getting double the epochs compared to layer-wise training with window size 1. Have you tested non-overlapping 2-block windows on ImageNet-1K to see if the big jump from 49.17 percent to 64.75 percent comes from cross-block gradients or mostly from window overlap and extra training epochs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.