acceptodds
Under review as a conference paper at ICLR 2027

Closing the Loop on Masking Difficulty for Joint-Embedding Predictive Learning

Abstract

Joint-embedding predictive architectures such as I-JEPA learn visual representations by predicting the embeddings of target blocks from a visible context. The masking policy that defines this task is typically fixed (I-JEPA samples M=4 target blocks throughout training), even as the representation it shapes evolves. However, a fixed mask does not imply fixed difficulty: the same masking operation can displace the representation differently at different stages of learning. We introduce Feedback-Controlled Data Augmentation (FCDA), a closed-loop framework that adapts masking using online feedback from representation geometry. A gradient-free sensor measures masking-induced displacement along weak but reliably estimated directions of the feature covariance, and a controller adapts the number of target blocks M so that this displacement tracks a reference calibrated on the original I-JEPA baseline, leaving the training objective untouched. In a matched- seed ImageNet-1K study with ViT-Tiny/16, FCDA improves linear-probe top-1 accuracy by 3.13 percentage points over the original baseline and also outperforms fixed and approximately target-count-matched open-loop controls. Transfer gains on natural-image recognition and a broader final covariance spectrum support the approach, while best-of-four CLEVR performance remains close to baseline. These results demonstrate that FCDA learns stronger representations by treating masking as a feedback-controlled component of predictive learning, one that adapts to the representations it helps shape.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Computer vision, self-supervised learning, representation learning, data augmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Yg8oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Feedback-Controlled Data Augmentation (FCDA). While keeping the I-JEPA network architecture and prediction targets unchanged, FCDA dynamically adjusts the number of target blocks for prediction based on feedback derived from internal representations. The method compares the features of the same visible patches—extracted from both full and occluded contexts—and employs cross-fitting along with covariance-based spectral filtering to compute normalized feature shifts along statistically reliable, low-variance directions. A controller then compares this metric against a time-varying reference curve (pre-calibrated using a baseline checkpoint) to incrementally increase or decrease the number of target blocks. Experiments conducted on ImageNet-1K using a ViT-Tiny/16 architecture demonstrate that FCDA outperforms both a fixed-mask baseline and an open-loop scheduling strategy (which maintains a similar total number of target blocks), while also yielding performance gains on certain classification transfer tasks.

Critical strengths
  1. The problem is clearly defined, and the design rationale for the feedback metric is sound. Comparing features at corresponding spatial locations helps mitigate interference from content-related variations; meanwhile, variance normalization, noise filtering, and cross-fitting prevent high-variance dimensions from dominating the measurements and ensure that low-variance noise is not disproportionately amplified. The proposed method requires neither modifications to the I-JEPA prediction objective nor the addition of a trainable mask-generation network.
  2. The paper provides valuable control experiments. Table 1 includes the original baseline, a setup with a fixed number of targets (using the same sampler), and an open-loop schedule with a roughly equivalent total number of targets. These comparisons demonstrate that the performance gains cannot be attributed solely to the change in the sampler. Furthermore, in this setting, FCDA outperforms the tested open-loop strategies, yielding improvements of 1.266 and 1.134 percentage points on linear probing and kNN Top-1 accuracy, respectively.
  3. The authors are transparent regarding the limitations of their experiments. They clearly distinguish between the feedback metric used during training and the quality of the resulting downstream representations, report unsuccessful experimental configurations, and acknowledge that the current study does not decouple the contributions of real-time feedback from those of the final mask schedule. Additionally, the transfer experiments honestly report that while classification tasks show improvement, tasks such as counting and depth estimation do not exhibit consistent gains.
Critical weaknesses
  1. The unique value of the core feedback metric has not been sufficiently validated. While the primary contribution in Section 3.3 centers on "normalized displacement along reliable weak directions," the paper fails to verify the necessity of this specific design by comparing it against alternative feedback signals—such as overall feature displacement, displacement along strong directions, or prediction loss. Furthermore, the components for noise filtering and cross-fitting lack independent ablation studies. Consequently, it remains unclear whether the observed gains stem from the proposed representation-geometric metric or from the more general dynamic masking mechanism.
  2. The roles of real-time feedback and the masking schedule have not been disentangled. In the open-loop control experiment (Table 1), the number of target blocks is increased abruptly from 4 to 7 at the 20th epoch, whereas FCDA employs a schedule that gradually increases this number to 8. Although the total number of targets is similar, the timing of adjustments, the amount of visible context, and the computational load differ between the two approaches. Thus, while the results support the overall strategy as tested, they are insufficient to conclude that real-time feedback is superior to a pre-defined, progressive masking schedule.
  3. The theoretical analysis fails to establish a robust link between the feedback metric and learning gains. Sections 3.2–3.4 explain how the metric measures and controls the process; however, the magnitude of feature perturbation does not necessarily reflect the "effective" learning difficulty, nor does a statistically reliable "weak" direction of variance guarantee the inclusion of meaningful semantic information. The use of reference curves derived from baseline checkpoints (trained with the same seed) is largely heuristic. Moreover, Table 3 shows that models with a fixed number of targets (using the same sampler) exhibit a broader feature spectrum yet perform worse—indicating that spectral expansion alone cannot account for performance improvements. This issue requires more direct mechanistic evidence.
  4. The empirical comparison with related methods and coverage of recent literature are inadequate. While I-JEPA serves as a reasonable direct baseline, the primary experiments focus solely on internal variants developed by the authors. Although the manuscript mentions HPM (CVPR 2023) and CL-MAE (WACV 2024), it provides no direct empirical comparison with these approaches; furthermore, the positioning of the work relative to more recent curriculum-based masking methods, such as CurMIM (ICASSP 2025), is insufficient. It is recommended to compare representative adaptation strategies within a consistent framework and budget, while clearly defining the limits of comparability arising from differing training objectives.
  5. The scope of the experiments limits the generalizability of the conclusions. The primary results are derived from a single ViT-Tiny model and a single pre-training seed; the reported confidence intervals reflect uncertainty related to the evaluation samples rather than the randomness inherent in the pre-training process. Experiments on CLEVR and Places205 lack comprehensive controls for the sampling process, and results on CLEVR do not show consistent improvements. Furthermore, as the full costs—including those for reference curve calibration and periodic measurements—have not been reported, one cannot assume that an equal number of training epochs implies an equivalent computational budget.
Questions for authors
  1. Is there existing evidence that distinguishes the benefits of "real-time representation feedback" from those of "incrementally increasing the number of target blocks"? If not, would you agree to limit the conclusions to the effectiveness of the complete strategy as tested, rather than claiming an independent advantage for real-time feedback?
  2. What is the rationale for selecting the weakest quartile of "reliable directions"? Have any experiments been conducted using metrics such as global displacement, strong directions, or prediction loss as feedback signals to justify the necessity of this specific design choice?
  3. Can the reference curves be directly transferred to new pre-training seeds or different model scales, or is it necessary to retrain the baselines and recalibrate the curves? What was the selection process for the current reference curves and their associated control parameters?
  4. In the fixed-reference experiments presented in the appendix, the warm-up period, the lower bound on the number of target blocks, and the stopping criteria were all altered simultaneously. Are there any results where these factors are held constant, which could specifically isolate and support the efficacy of a time-varying reference?
  5. Compared to HPM, CL-MAE, and more recent masking methods, what do the authors consider to be the most critical—and experimentally validated—distinguishing feature of FCDA? Are there any direct comparisons under identical experimental settings, or analyses that illustrate potential "adaptation barriers" (i.e., difficulties in applying these methods to other contexts)?
  6. Based on existing experimental logs, what is the additional computational overhead incurred by reference calibration, online measurement, and supplementary checks? Furthermore, are there results using independent pre-training seeds that could help assess the stability of the observed performance gains?
Reviewer_rZ7wOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Feedback-Controlled Data Augmentation (FCDA) for I-JEPA. It measures masking-induced changes in a common visible patch’s representation, using variance-normalized squared displacement in the weakest quarter of retained covariance directions. A controller adjusts the number of target blocks to follow an epoch-dependent response curve calibrated from baseline checkpoints. Experiments use ViT-Tiny/16 on ImageNet-1K, followed by frozen representation evaluations.

Critical strengths
  1. Adapting masking to the evolving representation is an interesting idea. The proposed sensor provides a concrete, label-free feedback signal without changing the predictive objective, with cross-fitting and reliability checks to stabilize measurement.

  2. Recognition gains are supported by useful controls. FCDA improves ImageNet linear-probe accuracy by 3.13 percentage points over the original baseline and 1.27 points over the approximately target-count-matched open-loop control. kNN and CIFAR-100 results also improve. The paper acknowledges both the limited CLEVR gains and the fact that broader spectra do not necessarily imply better representations.

Critical weaknesses
  1. The reference curve is central but insufficiently justified.

The sensor measures perturbation, while the baseline-derived reference specifies how much perturbation is desirable. It remains unclear why this baseline trajectory provides a useful target or how sensitive the gains are to its choice.

The unsuccessful constant-reference experiment also changes warm-up, controller bounds, and stopping rules, so it does not isolate the benefit of the proposed curve in App. A.

Requiring baseline checkpoints through epoch 100 additionally introduces a calibration dependency whose cost and reusability are not established. Also that poses a scalability concern.

  1. The specific sensor design remains insufficiently validated.

No ablation establishes why the weakest retained quarter is preferable to a simpler full-spectrum displacement measurement, or how sensitive the measurement is to the surviving probe location. Successful regulation and broader representation spectra do not establish the value of these particular choices; notably, the fixed-count control broadens the spectrum while reducing accuracy (Table 3).

  1. Robustness and generality remain uncertain.

Calibration and evaluation use one development seed, ViT-Tiny backbone, and a single training recipe. Frozen transfer deserves credit, but demonstrated benefits are concentrated in category recognition, with no clear CLEVR improvement or dense prediction evaluation. Incomplete total-compute profiling further limits assessment of the practical benefit.

Questions for authors
  1. Can you isolate reference choice while holding the other settings fixed—for example, comparing the proposed curve with a constant target and a simple increasing target estimated during warm-up? How reusable is a frozen reference across runs?

  2. On new initializations, does FCDA outperform a gradual or replayed schedule fixed using the development run? This would help distinguish adaptive feedback from the curriculum it produces.

  3. Does weakest-quarter sensing outperform a simpler full-spectrum displacement sensor under comparable calibration? What evidence supports the representativeness of the single surviving probe location?

  4. It would be interesting to see how FCDA pretraining perform on a different distribution, such as iNaturalist 18 [1], to see whether the sensor, reference, or controller requires recalibration? This would directly test generalization of the mechanism beyond the ImageNet setup.

  5. What is the complete computational cost, including baseline calibration, monitoring, and sweeps? How do the gains compare with allocating that additional compute to baseline training?

Reviewer_PJVvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that a fixed number of target blocks in I-JEPA does not give a fixed task difficulty, because the same mask changes the representation more or less at different stages of training. It proposes FCDA. FCDA measures how much a visible patch’s feature moves when context is removed, using weak but reliable covariance directions. It then adjusts the number of target blocks M so that this “dose” follows a reference curve measured on baseline I-JEPA checkpoints. Experiments use ViT-Tiny/16 on ImageNet-1K for 100 epochs with one seed. FCDA improves linear-probe top-1 by 3.1 pp over the baseline and by 1.3 pp over an open-loop schedule (M=4 for 20 epochs, then M=7). It also gives small gains on CIFAR-100 and Places205, no gain on CLEVR, and a broader feature spectrum.

Critical strengths
  1. The observation that the same mask perturbs the representation much more later in training (about 3.7x from epoch 1 to 100 at M=4) is interesting. It suggests the default M=4 may be too easy in later training, which is useful for people using I-JEPA.

  2. The authors are careful and honest about the setup. They use same-sampler controls and matched initialization, and they report failed runs (e.g., the constant-reference version) and many limitations.

Critical weaknesses
  1. The main claim is about closed-loop feedback, but the experiments do not isolate it. The realized FCDA schedule is a simple monotone ramp from M=4 to M=8. The open-loop 4→7 schedule already gets most of the gain (18.9 → 22.5 linear probe, vs. 23.8 for FCDA). The remaining 1.3 pp could come from a different schedule shape rather than from feedback. A replay of FCDA’s realized schedule, or a plain linear ramp from 4 to 8, is the key missing control. The paper mentions this but does not run it. Without it, the result reads as “slowly increasing M helps,” and the complex sensor may not be needed.

  2. The evidence is limited. There is only one pretraining seed, only ViT-Tiny, only 100 epochs, and absolute accuracy is around 20%. The confidence intervals only cover resampling of evaluation images, not training variance. The open-loop control was chosen after seeing FCDA, the reference was calibrated on the same seed, and probe heads were selected on the evaluation split. The sensor also has many design choices (MP edge, weakest quarter of directions, cross-fitting, ρ=1.2, anchor epochs) with no ablation, and the constant-reference version did worse than the baseline. So it is hard to tell whether the gain is robust or would hold at ViT-B/L scale, where I-JEPA is normally used.

Questions for authors

How does FCDA compare with (a) replaying its own realized M schedule and (b) a simple linear ramp from M=4 to 8? If these match FCDA, what does the feedback add?

Why is the custom sampler at fixed M=4 about 1.8 pp worse than the original sampler? Would an increasing-M schedule with the original sampler give similar gains?

How sensitive is FCDA to the sensor and reference choices? Can it work without a reference curve measured from a baseline run of the same setup?

Do the results hold for a larger model (e.g., ViT-B, ViT-L) or longer training? Saying that, a large M should work well for a large model.

Comments on the paper

1 comment

Dear Reviewers,

The authors have submitted their responses to your comments. Please review their rebuttals and engage in further discussion, particularly regarding any remaining concerns, unresolved issues, or points requiring additional clarification.

Thank you for your time and continued contributions to the review process.

Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.