- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Feedback-Controlled Data Augmentation (FCDA). While keeping the I-JEPA network architecture and prediction targets unchanged, FCDA dynamically adjusts the number of target blocks for prediction based on feedback derived from internal representations. The method compares the features of the same visible patches—extracted from both full and occluded contexts—and employs cross-fitting along with covariance-based spectral filtering to compute normalized feature shifts along statistically reliable, low-variance directions. A controller then compares this metric against a time-varying reference curve (pre-calibrated using a baseline checkpoint) to incrementally increase or decrease the number of target blocks. Experiments conducted on ImageNet-1K using a ViT-Tiny/16 architecture demonstrate that FCDA outperforms both a fixed-mask baseline and an open-loop scheduling strategy (which maintains a similar total number of target blocks), while also yielding performance gains on certain classification transfer tasks.
Critical strengths
- The problem is clearly defined, and the design rationale for the feedback metric is sound. Comparing features at corresponding spatial locations helps mitigate interference from content-related variations; meanwhile, variance normalization, noise filtering, and cross-fitting prevent high-variance dimensions from dominating the measurements and ensure that low-variance noise is not disproportionately amplified. The proposed method requires neither modifications to the I-JEPA prediction objective nor the addition of a trainable mask-generation network.
- The paper provides valuable control experiments. Table 1 includes the original baseline, a setup with a fixed number of targets (using the same sampler), and an open-loop schedule with a roughly equivalent total number of targets. These comparisons demonstrate that the performance gains cannot be attributed solely to the change in the sampler. Furthermore, in this setting, FCDA outperforms the tested open-loop strategies, yielding improvements of 1.266 and 1.134 percentage points on linear probing and kNN Top-1 accuracy, respectively.
- The authors are transparent regarding the limitations of their experiments. They clearly distinguish between the feedback metric used during training and the quality of the resulting downstream representations, report unsuccessful experimental configurations, and acknowledge that the current study does not decouple the contributions of real-time feedback from those of the final mask schedule. Additionally, the transfer experiments honestly report that while classification tasks show improvement, tasks such as counting and depth estimation do not exhibit consistent gains.
Critical weaknesses
- The unique value of the core feedback metric has not been sufficiently validated. While the primary contribution in Section 3.3 centers on "normalized displacement along reliable weak directions," the paper fails to verify the necessity of this specific design by comparing it against alternative feedback signals—such as overall feature displacement, displacement along strong directions, or prediction loss. Furthermore, the components for noise filtering and cross-fitting lack independent ablation studies. Consequently, it remains unclear whether the observed gains stem from the proposed representation-geometric metric or from the more general dynamic masking mechanism.
- The roles of real-time feedback and the masking schedule have not been disentangled. In the open-loop control experiment (Table 1), the number of target blocks is increased abruptly from 4 to 7 at the 20th epoch, whereas FCDA employs a schedule that gradually increases this number to 8. Although the total number of targets is similar, the timing of adjustments, the amount of visible context, and the computational load differ between the two approaches. Thus, while the results support the overall strategy as tested, they are insufficient to conclude that real-time feedback is superior to a pre-defined, progressive masking schedule.
- The theoretical analysis fails to establish a robust link between the feedback metric and learning gains. Sections 3.2–3.4 explain how the metric measures and controls the process; however, the magnitude of feature perturbation does not necessarily reflect the "effective" learning difficulty, nor does a statistically reliable "weak" direction of variance guarantee the inclusion of meaningful semantic information. The use of reference curves derived from baseline checkpoints (trained with the same seed) is largely heuristic. Moreover, Table 3 shows that models with a fixed number of targets (using the same sampler) exhibit a broader feature spectrum yet perform worse—indicating that spectral expansion alone cannot account for performance improvements. This issue requires more direct mechanistic evidence.
- The empirical comparison with related methods and coverage of recent literature are inadequate. While I-JEPA serves as a reasonable direct baseline, the primary experiments focus solely on internal variants developed by the authors. Although the manuscript mentions HPM (CVPR 2023) and CL-MAE (WACV 2024), it provides no direct empirical comparison with these approaches; furthermore, the positioning of the work relative to more recent curriculum-based masking methods, such as CurMIM (ICASSP 2025), is insufficient. It is recommended to compare representative adaptation strategies within a consistent framework and budget, while clearly defining the limits of comparability arising from differing training objectives.
- The scope of the experiments limits the generalizability of the conclusions. The primary results are derived from a single ViT-Tiny model and a single pre-training seed; the reported confidence intervals reflect uncertainty related to the evaluation samples rather than the randomness inherent in the pre-training process. Experiments on CLEVR and Places205 lack comprehensive controls for the sampling process, and results on CLEVR do not show consistent improvements. Furthermore, as the full costs—including those for reference curve calibration and periodic measurements—have not been reported, one cannot assume that an equal number of training epochs implies an equivalent computational budget.
Questions for authors
- Is there existing evidence that distinguishes the benefits of "real-time representation feedback" from those of "incrementally increasing the number of target blocks"? If not, would you agree to limit the conclusions to the effectiveness of the complete strategy as tested, rather than claiming an independent advantage for real-time feedback?
- What is the rationale for selecting the weakest quartile of "reliable directions"? Have any experiments been conducted using metrics such as global displacement, strong directions, or prediction loss as feedback signals to justify the necessity of this specific design choice?
- Can the reference curves be directly transferred to new pre-training seeds or different model scales, or is it necessary to retrain the baselines and recalibrate the curves? What was the selection process for the current reference curves and their associated control parameters?
- In the fixed-reference experiments presented in the appendix, the warm-up period, the lower bound on the number of target blocks, and the stopping criteria were all altered simultaneously. Are there any results where these factors are held constant, which could specifically isolate and support the efficacy of a time-varying reference?
- Compared to HPM, CL-MAE, and more recent masking methods, what do the authors consider to be the most critical—and experimentally validated—distinguishing feature of FCDA? Are there any direct comparisons under identical experimental settings, or analyses that illustrate potential "adaptation barriers" (i.e., difficulties in applying these methods to other contexts)?
- Based on existing experimental logs, what is the additional computational overhead incurred by reference calibration, online measurement, and supplementary checks? Furthermore, are there results using independent pre-training seeds that could help assess the stability of the observed performance gains?
Dear Reviewers,
The authors have submitted their responses to your comments. Please review their rebuttals and engage in further discussion, particularly regarding any remaining concerns, unresolved issues, or points requiring additional clarification.
Thank you for your time and continued contributions to the review process.
Regards,
AC