- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses a practical problem of training under memory constraints, using a local gradient update strategy through contrastive learning. The paper shows solid motivation and technical soundness, but also has a few issues in its evaluation.
Critical strengths
- The local training problem is real - edge-deployable models are hardly trainable on edge devices, because full BP requires too much memory. So the motivation of paper is realistic and strong.
- Overall the proposed method has shown promising results in the evaluation, with competitive accuracy, and high memory efficiency for training. Obtaining competitive results on ImageNet-1K with CFF-based learning is a commendable achievement.
- Sequential execution of the contrastive and reconstruction branches further contributes to the enabling of memory-efficient progressive training.
Critical weaknesses
- The reported peak memory for stage 2 needs further clarification - the paper states "gradients must pass through the frozen suffix blocks to reach the active window", but the calculation of M_{stage2} seems to be a theoretical formulation rather than empirically recorded peak memory. Please report the actual peak memory for stage 2, and compare it against full BP.
- Although the authors claim that the paper specifically targets CFF training of ViTs, the lack of broader comparison with other baselines such as BIM, local contrastive reconstruction, LocalMIM, makes it difficult to assess the impact of the proposed method in the local training field of research.
- Some external baselines are taken from paper rather than reproduced (e.g. Table 2). This is not ideal.
Questions for authors
What ImageNet performance is obtained with head-only evaluation after Stage 1?