- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how mask distributions shape the relationships learned between image regions during masked pre-training. It represents how often one region provides context for predicting another as a supervision graph that can be edited while keeping the data, model, and objective fixed. The authors characterize realizable graphs and construct corresponding mask distributions through constrained maximum-entropy projection. Experiments on glyphs, natural photographs, and face regions assess how these interventions affect predictive influence and the linear readability of relations between regions.
Critical strengths
- The authors explicitly acknowledge the restricted experimental setting
- The appendices provide detailed descriptions of the dataset construction, masking schedules, and evaluation procedures.
Critical weaknesses
- The paper’s broad framing overstates what is new about using masking to shape learned relationships. Prior work already formulates masked reconstruction through a mask graph [1], directly measures how masking changes spatial predictive influence [2], and uses masking to encourage relations between image parts [3]. Although Section 2 acknowledges some of these connections, the introduction’s claim that masking’s effects beyond accuracy have not been measured directly is a overstatement.
- The introduction states that the objective makes every context patch predict every target patch, but Equation (1) only counts context–target patch pairs. No decomposition of the training objective into these pairwise terms is provided.
- Section 2 argues that retraining after an intervention keeps the compared representations within their own training distributions, but this is not consistent with the evaluation protocol. The predictive-influence test replaces a region’s content and supplies 135 context patches, compared with 75 during the relevant training experiments.
- Section 3.1 introduces I-JEPA with VICReg variance and covariance penalties without giving the combined loss. Appendix A supplies regularization weights and effective-rank measurements but still does not specify which representations receive the penalties or how the statistics are aggregated over samples and patches. The architecture description gives the encoder’s width and depth without specifying the predictor architecture, even though predictive influence is a central measurement of that predictor. In addition, it is unclear why I-JEPA and VICReg are used after all.
- The description of the RELCIFAR relation is ambiguous. Section 3.1 introduces a class shift of for RELFOREST and then says that RELCIFAR carries the same relations using natural photographs. Appendix E instead describes the RELCIFAR relation as semantic identity between images.
- Section 5.4 presents an alignment–uniformity explanation for the relationship between co-exposure, effective rank, and relational readout, and states that Appendix C provides the derivation. Appendix C instead reports additional correlations and repeats the interpretation, with no derivation at all.
- The paper does not experimentally compare its graph-based mask design with established structured-masking methods such as SemMAE, AttMask, or AdaMAE, despite discussing them in Section 2. Default block and uniform random masking provide reference conditions, but the comparisons largely remain between the authors’ own masking schedules. The MAE experiment only replaces the prediction objective within the same setup and is restricted to RELFOREST at s = 0. These experiments do not establish whether the proposed design offers more effective or more selective control over learned relationships than existing masking methods.
- The reporting conventions are not consistent across the manuscript. Appendix A states that routing and readout results use eight seeds, or sixteen where indicated, whereas Appendix C describes readout correlations from four operating points with twelve seeds each. Table 6 also reports several values with “±” quantities without specifying whether these denote standard deviations, standard errors, or confidence intervals.
[1] How Mask Matters: Towards Theoretical Understandings of Masked Autoencoders. NeurIPS 2022.
[2] From Linearity to Non-Linearity: How Masked Autoencoders Capture Spatial Correlations. ICCV 2025.
[3] SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders. NeurIPS 2022.
Questions for authors
Refer to the weakness section.
2 comments
Response to Reviewer A3At (2/2)
Response to Reviewer A3At (2/2)
4. Loss, predictor, and why I-JEPA with VICReg. Now in Section 3.1 and App. A: averaged over the batch and the dimensions. Here are the context encoder's tokens for the patches in , is the predictor's output at target position , and are the tokens of the target encoder, an EMA of the context encoder (momentum 0.996→1) applied to the full image. LN is a layer norm without learned parameters. and are VICReg's variance term (hinge at unit standard deviation) and covariance term (squared off-diagonal entries divided by the dimension), computed on the context tokens pooled over all images of the batch. The predictor is a 2-block transformer of width 96 with 3 heads, sinusoidal positions and a shared mask token, projecting back to width 192. Masks are shared within a batch. We use I-JEPA because its predictor makes the context-to-target structure explicit, which is the object the graph describes, and lets us measure influence in representation space. The VICReg terms prevent the collapse we measured without them (effective rank 4.7 of 192), as in C-JEPA (Mo & Tong, NeurIPS 2024). Pixel reconstruction reproduces the removal result, at s=0 and at s=5 (App. B, Table 18).
5. RelCIFAR. Thank you. App. E is corrected. RelCIFAR uses the same forest over CIFAR-10 classes with the same shift: with probability κ the child's class is the parent's class shifted by s=5 modulo ten, and each node shows a different photograph of its class, so the relation is a class-level mapping between two independent natural images. The removal experiment on photographs added in the revision uses s=0, i.e. the same class in different photographs. Neither admits a pixel shortcut.
6. Derivation in App. C. Added. If the two node representations have covariance Σ and cross-covariance γΣ, the pair covariance has eigenvalues , so its effective rank is , with h the binary entropy in nats. It falls monotonically from 2·erank(Σ) at γ=0 to erank(Σ) at |γ|=1: aligning the two representations, which is what co-exposure trains, can lower the pair's rank by up to half. The drop observed in the larger encoder (130.8 → 74.8, a factor of 0.57) lies within that bound.
7. Comparison with SemMAE, AttMask and AdaMAE. These methods choose what to mask in order to raise downstream accuracy. None takes a designated relation or a specified graph as input, so we compare at the level the methods share with ours, the graph. Whole-part masking (SemMAE's second stage) sets within-part co-exposure to zero, the operation whose effect our removal experiments measure. Attention-guided and adaptive masking (AttMask, AdaMAE) change where targets fall, i.e. the target marginals, which is the quantity our design holds fixed so that effects on relations can be separated from visibility. We also computed the graph that standard schemes induce in our geometry (Table 19): random masking (MAE), block-wise masking (BEiT) and I-JEPA's multi-block sampler spread co-exposure evenly over the pairs (15.3–15.4, 19.9–24.6 and 22.1–22.3), whereas our edits single out one pair. The revision adds this discussion. Our aim is the mechanism and a tool for designing such schedules, not a competing schedule.
8. Reporting. App. A now states the seeds of each analysis in App. C (5 per schedule for the calibration, 12 per operating point for the pair-local analysis) and that every ± is a standard deviation over seeds. The ± in Table 6 is the standard deviation over 16 seeds, now stated in the caption. All headline results carry seed-paired 95% CIs (App. H, Table 13). For readability we also added an experiment map (App. G, Table 12) that lists, for every comparison, the data, the schedules, the quantities held fixed, the measurement and where the result appears.
We hope these changes address the concerns, and would welcome any further questions.
Response to Reviewer A3At (1/2)
Thank you for the detailed review. Every point is addressed in the revision. We take them in order.
1. Relation to [1]–[3]. The sentence "has not been measured directly" is replaced by the precise claim, and all three works are now discussed.
What is new is therefore not that masking affects relations, but (i) pair-level interventions at fixed marginals, with specificity, exchange and dose; (ii) the dissociation of co-exposure from co-occurrence; (iii) an exact characterization of the realizable graphs, and a compiler.
2. Eq. (1) and the objective. W is not a decomposition of the loss, and the paper does not rely on one. The loss of a step depends on the content of the input only through the predictor, which sees all of C: the content of patch c can enter the prediction of target t only if c∈C and t∈T. W counts these context–target pairs by region, so W(i→j)=0 means that no step rewards using region i to predict region j. How much a positive entry is used is what the experiments measure. The introduction now reads "every patch in C is available to predict every patch in T", and Section 3.2 states the above.
3. Training distribution and the influence query. The sentence in Section 2 concerns the compared models: each is trained under its own schedule, and nothing inside it is edited afterwards, unlike interventions on activations. The influence query is an evaluation choice, identical for all models. Its inputs are ones the data generator itself produces (node i's class is redrawn), while its context of 135 patches exceeds the training budget (75 for complement masks, up to 132 for block schedules). The revision states this (Sections 2 and 3.4, App. E). Measured with 75-patch contexts drawn as in training, removal is as specific in both directions, AUC 1.000 (Table 16). The readouts, which carry the face and RelCIFAR results, use the target encoder on full images, which is what it processes at every training step.