acceptodds
Under review as a conference paper at ICLR 2027

Mask Schedules Are a Programmable Supervision Graph

Abstract

Masked pretraining treats the mask distribution as a hyperparameter. But a mask draw also assigns regions to the context and to the target, and so decides which pairs of regions the objective makes predict one another. Averaged over draws this gives a co-exposure matrix over pairs of regions, a supervision graph that the mask distribution, or schedule, determines completely and that can be rewritten with the data, the network and the objective held fixed. Removing one edge takes predictive influence away from that pair alone (AUC 0.996) and drives its relational readout to chance, while each region's own content remains readable. It does so even when both regions are predicted more often than before and still appear together in every input. The effect reverses when two schedules are exchanged and is graded in co-exposure. It holds on real images: with the regions of CelebA faces tiled onto noise, raising the co-exposure between one eye and the mouth lifts that eye's readout of mouth opening from 65.4% to 71.1%, the level of the mouth region itself (70.9%). The graphs a fixed geometry admits form a polytope with exact linear-programming certificates, and a specified graph can be compiled by constrained maximum-entropy projection, which gets the direction of the effect right but, at the same co-exposure of the edited pair, achieves less of it than placing target blocks by hand. The effect shrinks with the share of the objective the relation carries and vanishes when the image's own coherence already delivers the information, as on whole faces.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Masked Image Modeling, Self-Supervised Learning, Representation Learning, Mask Schedules, Predictive Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_A3AtOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how mask distributions shape the relationships learned between image regions during masked pre-training. It represents how often one region provides context for predicting another as a supervision graph that can be edited while keeping the data, model, and objective fixed. The authors characterize realizable graphs and construct corresponding mask distributions through constrained maximum-entropy projection. Experiments on glyphs, natural photographs, and face regions assess how these interventions affect predictive influence and the linear readability of relations between regions.

Critical strengths
  1. The authors explicitly acknowledge the restricted experimental setting
  2. The appendices provide detailed descriptions of the dataset construction, masking schedules, and evaluation procedures.
Critical weaknesses
  1. The paper’s broad framing overstates what is new about using masking to shape learned relationships. Prior work already formulates masked reconstruction through a mask graph [1], directly measures how masking changes spatial predictive influence [2], and uses masking to encourage relations between image parts [3]. Although Section 2 acknowledges some of these connections, the introduction’s claim that masking’s effects beyond accuracy have not been measured directly is a overstatement.
  2. The introduction states that the objective makes every context patch predict every target patch, but Equation (1) only counts context–target patch pairs. No decomposition of the training objective into these pairwise terms is provided.
  3. Section 2 argues that retraining after an intervention keeps the compared representations within their own training distributions, but this is not consistent with the evaluation protocol. The predictive-influence test replaces a region’s content and supplies 135 context patches, compared with 75 during the relevant training experiments.
  4. Section 3.1 introduces I-JEPA with VICReg variance and covariance penalties without giving the combined loss. Appendix A supplies regularization weights and effective-rank measurements but still does not specify which representations receive the penalties or how the statistics are aggregated over samples and patches. The architecture description gives the encoder’s width and depth without specifying the predictor architecture, even though predictive influence is a central measurement of that predictor. In addition, it is unclear why I-JEPA and VICReg are used after all.
  5. The description of the RELCIFAR relation is ambiguous. Section 3.1 introduces a class shift of s=5s=5 for RELFOREST and then says that RELCIFAR carries the same relations using natural photographs. Appendix E instead describes the RELCIFAR relation as semantic identity between images.
  6. Section 5.4 presents an alignment–uniformity explanation for the relationship between co-exposure, effective rank, and relational readout, and states that Appendix C provides the derivation. Appendix C instead reports additional correlations and repeats the interpretation, with no derivation at all.
  7. The paper does not experimentally compare its graph-based mask design with established structured-masking methods such as SemMAE, AttMask, or AdaMAE, despite discussing them in Section 2. Default block and uniform random masking provide reference conditions, but the comparisons largely remain between the authors’ own masking schedules. The MAE experiment only replaces the prediction objective within the same setup and is restricted to RELFOREST at s = 0. These experiments do not establish whether the proposed design offers more effective or more selective control over learned relationships than existing masking methods.
  8. The reporting conventions are not consistent across the manuscript. Appendix A states that routing and readout results use eight seeds, or sixteen where indicated, whereas Appendix C describes readout correlations from four operating points with twelve seeds each. Table 6 also reports several values with “±” quantities without specifying whether these denote standard deviations, standard errors, or confidence intervals.

[1] How Mask Matters: Towards Theoretical Understandings of Masked Autoencoders. NeurIPS 2022.

[2] From Linearity to Non-Linearity: How Masked Autoencoders Capture Spatial Correlations. ICCV 2025.

[3] SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders. NeurIPS 2022.

Questions for authors

Refer to the weakness section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer A3At (1/2)

Thank you for the detailed review. Every point is addressed in the revision. We take them in order.

1. Relation to [1]–[3]. The sentence "has not been measured directly" is replaced by the precise claim, and all three works are now discussed.

  • [1] Zhang et al. define a graph whose vertices are views (the visible and the masked parts of images) and whose edges join complementary views. It yields downstream guarantees as a function of the mask ratio. Our graph has regions within one image as vertices and is a property of the schedule that we edit at fixed marginals.
  • [2] Bisulco et al. measure input–output Jacobians as a function of distance, averaged over pixel pairs, while varying global hyperparameters (mask ratio, patch size) that change how often every patch is masked. They neither hold per-region masking fixed nor intervene on specific pairs. Their Jacobian is the closest relative of our influence measure, and we now cite it in Section 3.4.
  • [3] SemMAE motivates its curriculum with the intuition we test (masking within parts teaches intra-part patterns, masking whole parts teaches inter-part relations) and evaluates it by downstream accuracy. In our terms its two stages move co-exposure from within parts to between parts, and whole-part masking is exactly the operation of our removal schedules. To our knowledge the paper is the first controlled test of that premise.

What is new is therefore not that masking affects relations, but (i) pair-level interventions at fixed marginals, with specificity, exchange and dose; (ii) the dissociation of co-exposure from co-occurrence; (iii) an exact characterization of the realizable graphs, and a compiler.

2. Eq. (1) and the objective. W is not a decomposition of the loss, and the paper does not rely on one. The loss of a step depends on the content of the input only through the predictor, which sees all of C: the content of patch c can enter the prediction of target t only if c∈C and t∈T. W counts these context–target pairs by region, so W(i→j)=0W(i\to j)=0 means that no step rewards using region i to predict region j. How much a positive entry is used is what the experiments measure. The introduction now reads "every patch in C is available to predict every patch in T", and Section 3.2 states the above.

3. Training distribution and the influence query. The sentence in Section 2 concerns the compared models: each is trained under its own schedule, and nothing inside it is edited afterwards, unlike interventions on activations. The influence query is an evaluation choice, identical for all models. Its inputs are ones the data generator itself produces (node i's class is redrawn), while its context of 135 patches exceeds the training budget (75 for complement masks, up to 132 for block schedules). The revision states this (Sections 2 and 3.4, App. E). Measured with 75-patch contexts drawn as in training, removal is as specific in both directions, AUC 1.000 (Table 16). The readouts, which carry the face and RelCIFAR results, use the target encoder on full images, which is what it processes at every training step.

AuthorsOpenReview ↗

Response to Reviewer A3At (2/2)

Response to Reviewer A3At (2/2)

4. Loss, predictor, and why I-JEPA with VICReg. Now in Section 3.1 and App. A: L=1∣T∣∑u∈TSmoothL1(z^u, LN(zˉu))+0.1,Lvar(zC)+0.004,Lcov(zC),L=\frac{1}{|T|}\sum_{u\in T}\mathrm{SmoothL1}\big(\hat z_u,\ \mathrm{LN}(\bar z_u)\big)+0.1\\,\mathcal{L}_{\rm var}(z_C)+0.004\\,\mathcal{L}_{\rm cov}(z_C), averaged over the batch and the dimensions. Here zCz_C are the context encoder's tokens for the patches in CC, z^u\hat z_u is the predictor's output at target position uu, and zˉ\bar z are the tokens of the target encoder, an EMA of the context encoder (momentum 0.996→1) applied to the full image. LN is a layer norm without learned parameters. Lvar\mathcal{L}_{\rm var} and Lcov\mathcal{L}_{\rm cov} are VICReg's variance term (hinge at unit standard deviation) and covariance term (squared off-diagonal entries divided by the dimension), computed on the context tokens pooled over all images of the batch. The predictor is a 2-block transformer of width 96 with 3 heads, sinusoidal positions and a shared mask token, projecting back to width 192. Masks are shared within a batch. We use I-JEPA because its predictor makes the context-to-target structure explicit, which is the object the graph describes, and lets us measure influence in representation space. The VICReg terms prevent the collapse we measured without them (effective rank 4.7 of 192), as in C-JEPA (Mo & Tong, NeurIPS 2024). Pixel reconstruction reproduces the removal result, at s=0 and at s=5 (App. B, Table 18).

5. RelCIFAR. Thank you. App. E is corrected. RelCIFAR uses the same forest over CIFAR-10 classes with the same shift: with probability κ the child's class is the parent's class shifted by s=5 modulo ten, and each node shows a different photograph of its class, so the relation is a class-level mapping between two independent natural images. The removal experiment on photographs added in the revision uses s=0, i.e. the same class in different photographs. Neither admits a pixel shortcut.

6. Derivation in App. C. Added. If the two node representations have covariance Σ and cross-covariance γΣ, the pair covariance has eigenvalues (1±γ)λk(1\pm\gamma)\lambda_k, so its effective rank is erank(Σ)exp⁡h((1+γ)/2)\mathrm{erank}(\Sigma)\exp\\{h((1+\gamma)/2)\\}, with h the binary entropy in nats. It falls monotonically from 2·erank(Σ) at γ=0 to erank(Σ) at |γ|=1: aligning the two representations, which is what co-exposure trains, can lower the pair's rank by up to half. The drop observed in the larger encoder (130.8 → 74.8, a factor of 0.57) lies within that bound.

7. Comparison with SemMAE, AttMask and AdaMAE. These methods choose what to mask in order to raise downstream accuracy. None takes a designated relation or a specified graph as input, so we compare at the level the methods share with ours, the graph. Whole-part masking (SemMAE's second stage) sets within-part co-exposure to zero, the operation whose effect our removal experiments measure. Attention-guided and adaptive masking (AttMask, AdaMAE) change where targets fall, i.e. the target marginals, which is the quantity our design holds fixed so that effects on relations can be separated from visibility. We also computed the graph that standard schemes induce in our geometry (Table 19): random masking (MAE), block-wise masking (BEiT) and I-JEPA's multi-block sampler spread co-exposure evenly over the pairs (15.3–15.4, 19.9–24.6 and 22.1–22.3), whereas our edits single out one pair. The revision adds this discussion. Our aim is the mechanism and a tool for designing such schedules, not a competing schedule.

8. Reporting. App. A now states the seeds of each analysis in App. C (5 per schedule for the calibration, 12 per operating point for the pair-local analysis) and that every ± is a standard deviation over seeds. The ± in Table 6 is the standard deviation over 16 seeds, now stated in the caption. All headline results carry seed-paired 95% CIs (App. H, Table 13). For readability we also added an experiment map (App. G, Table 12) that lists, for every comparison, the data, the schedules, the quantities held fixed, the measurement and where the result appears.

We hope these changes address the concerns, and would welcome any further questions.

Reviewer_woXrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper models mask schedules as “supervision graphs” defined by context–target co-exposure between image regions. Controlled experiments show that changing the schedule can alter predictive influence and linear relational readouts while holding data, architecture, and objective fixed. The authors also characterize feasible graphs and construct schedules through KL projection. The demonstrated effects are strongest in isolated-region layouts and weaken with greater image diversity or disappear on whole faces.

Critical strengths
  • The core idea is clear and testable. The co-exposure graph provides a concrete perspective on how masking shapes relationships between learned region representations.
  • The experiments provide complementary evidence. Matched removal controls, schedule exchanges, and graded interventions support the proposed effect. Content readouts help distinguish reduced relational accessibility from general representation collapse.
  • The theory connects characterization to construction. The covariance formulation supports feasibility analysis and schedule compilation. The paper also openly reports limitations and unsuccessful settings.
Critical weaknesses
  • Generality remains limited. Positive results are concentrated in constructed four-region layouts. Effects diminish as RelCIFAR image diversity increases, and routing on whole faces is negligible. Practical usefulness in standard natural-image pretraining and scalability beyond four regions remain unestablished.
  • The strict specificity claim exceeds the evidence. In the zero-co-exposure experiment, the edited relation’s readout falls from 59.83% to 49.53%, but the untouched relation also falls from 62.5% to 57.1%. This supports differential effects, rather than changes occurring “only where it was edited.” Changes to other masking statistics and limited uncertainty reporting further complicate attribution.
  • The graph’s explanatory role is not fully isolated. Schedule changes also alter higher-order masking dependencies and spatial structure. Placement and compilation match only one co-exposure entry, so their comparison does not establish how much of the learning effect is explained by the full graph.
  • Some theoretical conditions need clarification. Feasibility requires restricting the count distribution to states consistent with the target budget. Boundary graphs also require care because finite exponential-tilt parameters need not exist. These appear to be fixable formulation issues.
Questions for authors
  • Specificity: Can additional controlled readout comparisons clarify how much of the effect is specific to the edited relation?
  • Role of co-exposure: Can a targeted comparison help distinguish the contribution of co-exposure from other properties of the mask schedule? This would strengthen the interpretation of the graph as an explanatory description of learning.
  • Readout robustness: Do the main conclusions persist across probe regularization choices and against an untrained-encoder baseline? Reporting seed-paired uncertainty for the edited and untouched relations would help assess whether the results reflect robust changes in the learned representations rather than probe or evaluation choices.
5 comments
AuthorsOpenReview ↗

Response to Reviewer woXr*

Thank you for the careful reading, and for noting that content readouts separate reduced relational accessibility from collapse.

W1: Generality. Positive results are not confined to glyphs (App. I, with seed-paired 95% CIs in App. H).

  • Removal is specific on natural photographs at full CIFAR-10 diversity (both specificities negative, AUC 0.996) and on the transposed geometry (16 seeds, AUC 1.000).
  • Routing appears on a second, diverse natural-image source: tiles drawn from 60,000 unlabeled STL-10 photographs, +2.5 percentage points, positive in every seed.
  • On whole faces the null has a measured cause, redundancy: the eye node reads the mouth at 73.8% before any edit, and the relation's share of the objective is a ninth of that on tiles. The revision states this as the scope of the practical claim.

Beyond four regions, editing by placement is a sampling rule that does not depend on the number of regions, and W can be estimated for any schedule by sampling. Only the exact certificates enumerate count states, whose number grows as (m+1)n(m+1)^n. Section 5.1 states this limit.

W2/Q1: Specificity. We agree the data support a concentrated, differential effect. The revision says so and adds seed-paired CIs (Section 4.1, App. H).

  • Zero co-exposure. The edited relation falls from 59.8% to 49.5% at k=4096 and sits at chance at every k ≥ 256. The untouched one falls from 62.5% to 57.1% (95% CI [54.0, 60.2]) and stays above chance at every k ≥ 256. The edited relation loses 4.9 points more than the untouched one (95% CI [0.8, 9.0]). The untouched relation's co-exposure is unchanged (9.5), while the control deliberately triples the target marginals of the edited nodes.
  • Removal. The removed relation keeps 0.20 and 0.23 of its influence, and the other relation 1.30 and 1.10. Among the pairs that carry no information, only those whose source is a node of the removed pair lose influence (0.26 and 0.31, against 1.08 and 1.04 in the other direction). Removing an edge silences its two nodes as sources, whose content no longer informs any target, and leaves every other informative relation in place (Table 14).

W3/Q2: Co-exposure versus other properties of the schedule. W and the marginals are linear in the count distribution, so they can be matched exactly while higher-order structure changes. For the RelForest swap, the I-projection (3,328 count states) and two LP realizations (8 states with similar node variances, and 10 states with every node masked whole) share all 12 entries of W and all node means to within 1e-13. All three route the raised relation in every seed, by 24.4, 25.8 and 17.2 points (Table 17). The graph fixes the direction, and entropy does not change the size. Whole-node masking, which roughly doubles the node-count variance, reduces it. With the existing evidence (zero co-exposure removes the relation in two unrelated families, and mirror schedules exchange the learned structure), this isolates co-exposure as what sets the direction.

W4: Theoretical conditions. Both points are correct and now stated precisely (Section 5.1, App. D). Neither changes a reported number.

  • Budget. The LP and the projection range over states with q0(t)>0q_0(t)>0, i.e. those consistent with the target budget. At τ=0.48 (Fig. 4a) all 10410^4 states qualify. At τ=0.24 exactly one, t=(9,9,9,9)t=(9,9,9,9), does not, and the tilt inherits the support of q0q_0.
  • Boundary. For a target on the boundary of the feasible polytope, the I-projection exists and is unique but is a limit of tilts supported on a face (Csiszár, 1975). The one boundary graph we train on (Section 4.2) is solved to a residual below 1e-5 and concentrates on 5 effective states (Table 7), as this predicts.

Q3: Readout robustness. The effect grows with probe size (Table 9), and the same conclusions follow from predictive influence, which uses no probe. We also ran both controls you suggest (App. H). Across AdamW weight decay 0, 1e-4 (the paper's), 1e-2, 1 and 10, no relational readout moves by more than 0.6 points: under the zero-co-exposure control the edited relation stays at chance (49.5–49.8% at k=4096) and the untouched one above it (57.1–57.7%), and the edited relation falls 4.9–5.4 points further, with intervals excluding zero at every setting. An untrained encoder of the same architecture reads both relations at chance at every k (49.7–50.4%) while reading each node's glyph at 93–94%: relational readability comes from training, not from the probe.

Thank you for the detailed response and the additional experiments. They were helpful and clarified several of my concerns. However, a few questions remain unresolved, and one of them is central to my assessment.

In the current datasets, each node has one informative partner. Could you run a removal experiment in which one node takes part in two independent relations, and only one of them is removed with marginals held fixed? I would be interested in the predictive influence and relational readout of both relations.

Thank you. We ran this experiment.

Setup. RelForest with a fork: R0 is the parent of R1 and of R2, each copying R0's glyph (shifted by 5) independently with probability 0.8, and R3 is independent. R0 therefore takes part in two relations, R0–R1 and R0–R2. Group cuts at target probability 0.48 hold every per-patch marginal fixed across three schedules: neither relation removed, R0–R1 removed, and R0–R2 removed. Model, objective, influence query and probes are those of Section 4.1, with eight seeds (App. J, Table 20).

R0–R1 removedR0–R2 removed
Influence retained, removed relation0.380.37
Influence retained, other relation (shares R0)0.970.93
Relational readout change, removed relation (percentage points)−11.3−11.9
Relational readout change, other relation (percentage points)−2.5−2.5
  • Specificity (retention of the removed relation minus that of the other) is negative in all eight seeds under both removals (−0.59 and −0.56). The removed relation's readout falls from 64.7% and 64.1% to 53.4% and 52.2% (chance is 50%) in every seed. The other relation's readout change is within seed variation (both 95% CIs contain 0).
  • Removal therefore acts on the edge, not on the node. R0 stays a source for the partner it is still co-exposed with: its influence on that partner keeps 0.80 and 0.75 of its baseline, against 0.38 and 0.37 for the removed pair, and that partner's influence on R0 rises (1.24 and 1.21). R0 pays a small cost of its own: its content readout falls by about 2.5 points, from 99.5%.
  • Both children copy R0, so after a removal the cut child is predicted from the other child instead. The influence of R2 on R1 rises 1.9-fold when R0–R1 is removed, and the mirror holds. This is why the removed relation's readout stays slightly above chance.

This refines the locality account of Section 4.1. A node that is never seen while one partner is predicted stops informing that partner, and keeps informing any other partner it is co-exposed with.

If other questions remain open, we would be glad to address them before the discussion closes.

Reviewer_yjWtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper views masked pretraining schedules as supervision graphs, defining each edge by the expected co-exposure of a context region and a target region. It tests whether editing these edges changes predictive influence and linear relational readouts while keeping the data, architecture, and objective fixed. Experiments cover synthetic glyphs, CIFAR image tiles, and CelebA regions, with removal, schedule exchange, and graded interventions. The paper also characterizes feasible count-based graphs using linear programming and compiles specified graphs through a KL projection. Routing is strongest on tiled inputs and largely disappears on whole faces.

Critical strengths

S1. The intervention design is a clear strength. Matched marginals, schedule exchanges, and the zero-co-exposure control help distinguish the effect of predictive supervision from simple co-occurrence or prediction frequency. S2. The count-based formulation makes the proposed graph actionable. The covariance identities, feasibility tests, and KL-projection compiler provide a concrete way to construct mask distributions for specified co-exposure constraints. S3. The experiments go beyond a single positive example. Multiple datasets, repeated seeds, graded interventions, and explicit negative results on whole faces provide an informative account of when routing appears and when it weakens.

Critical weaknesses

W1. The experiments establish a causal effect of the schedule, but do not isolate the graph as the mediator. Figure 5 matches only one graph entry while other entries, marginals, and mask geometry differ. Schedules matched on the full graph and marginals are needed to test what the graph itself explains. W2. The claim that only the edited pair changes is too strong. In the zero-co-exposure control, the untouched relation falls from 62.5% to 57.1%; Table 2 also shows retained-pair influence increasing to 1.296. These changes warrant uncertainty estimates and a more qualified account of locality. W3. Practical relevance remains limited by the tested setting. Routing vanishes on whole CelebA faces, and the RELCIFAR gain drops below one point at E=32 or 64. The paper does not yet demonstrate useful graph-based control on coherent natural images with diverse content. W4. Generalization across objectives is not established. The main encoder requires VICReg regularization to avoid collapse, while pixel reconstruction is tested only at s=0, where pixel similarity provides a shortcut, and is unstable across seeds. The central result needs a stable, shortcut-resistant objective comparison. W5. Influence uses 135 context patches rather than 75 during training, and full-image probes allow inference-time mixing. Matched-context and target-hidden readouts are needed to clarify how these evaluation choices affect routing.

Questions for authors

Q1. Can schedules with identical full co-exposure matrices and marginals but different higher-order mask structure produce different routing? What predictive claim is intended for the graph beyond the causal effect of changing schedules? Q2. Can graph edits produce a measurable benefit on coherent natural images or at higher content diversity? Otherwise, how should the practical claim be restricted? Q3. Does the removal or exchange effect persist at s=5 under stable pixel reconstruction and across VICReg weights, after controlling for representation collapse? Q4. Do the conclusions persist when influence uses the training context budget and when the target region is hidden during the eye-node readout? These controls would clarify the role of inference-time information mixing.

2 comments
AuthorsOpenReview ↗

Response to Reviewer yjWt (1/2)

Thank you for a precise summary and for questions that each name a concrete control. We ran all of them. The results are below and in the revision.

W1/Q1: What does the graph explain beyond "changing the schedule changes learning"? We make two predictive claims. Both can be read off the schedule before training, and both hold across unrelated schedule families.

  • Zero co-exposure removes the relation. With W(i→j)=W(j→i)=0W(i\to j)=W(j\to i)=0, the pair loses most of its influence in the group-cut family (complement masks, retention 0.20 and 0.23) and its linear readability in the block-mask control (chance at every k), although in the latter both regions appear in every input and are predicted more often than under the default.
  • The sign of a change in a pair's entry predicts the sign of the change in that pair's influence or readout. This held in every family tested: placement (graded, slope 5.0 percentage points per unit λ), compiled swaps (+23.6 points), the compiled exchange on RelGraph (routing index 0.53), and removal.

We do not claim that W sets the magnitude. Section 5.5 shows it does not, and Table 8 lists the full graphs behind Fig. 5.

Your Q1 is the right test, and our framework makes it exact. W and the node marginals are linear in the count distribution, so both can be held fixed while everything else changes. We realized the RelForest swap three ways with identical 12-entry W and node means (difference < 1e-13): the paper's I-projection (5.70 nats, 3,328 states), an LP solution with similar node variances (1.74 nats, 8 states), and an LP solution in which every node is entirely target or context (1.90 nats, 10 states, roughly doubled node variances). Readout of the raised relation, raised minus lowered, k=4096, 8 seeds (Table 17):

  • I-projection: +24.4 points
  • LP, similar variances: +25.8 points (vs projection +1.4, not significant)
  • LP, whole nodes: +17.2 points (vs projection −7.2, 95% CI [−11.2, −3.3])

The mirror relation behaves the same way (21.7, 24.2, 15.2 points). With the full graph fixed, the direction is fixed in every seed. A 400-fold reduction in support leaves the size unchanged, while whole-node masking reduces it by about 30%. This is the claim of Section 5.5, that the graph sets the sign and not the magnitude, now tested with the full graph matched.

W2: Locality. Agreed: "only where it was edited" was broader than the evidence. The revision states that the effect concentrates on the edited pair and gives the full account with seed-paired 95% CIs (Section 4.1, App. H).

  • Removal (s=5, 8 seeds). The removed relation keeps 0.20 and 0.23 of its influence. The other relation keeps 1.30 and 1.10, and both 95% CIs contain 1, so it loses nothing on average and its rise is not significant at s=5. With 16 seeds at s=0 it is (1.94 and 1.40, every seed above 1), which we read as reallocation to the remaining informative source.
  • The pairs without information split cleanly by direction. Those whose source is a node of the removed pair fall nearly as far as the pair itself (0.26 and 0.31), and those whose target is one keep theirs (1.08 and 1.04). After the cut a node's content informs no target it is co-exposed with, so removing an edge silences both its nodes as sources, and every other informative relation stays in place (Table 14). The same pattern holds at full CIFAR-10 diversity.
  • Zero co-exposure (readout). The edited relation drops 10.3 points, to chance (49.5%). The untouched relation drops 5.4 points and stays above chance at every k ≥ 256 (57.1% at k=4096, 95% CI [54.0, 60.2]). The edited relation loses 4.9 points more than the untouched one (95% CI [0.8, 9.0]). The untouched relation's own co-exposure is unchanged (9.5), while the control deliberately triples the target marginals of the edited nodes and raises the co-exposure of the uninformative cross-row pairs from 9.5 to 25.0.
AuthorsOpenReview ↗

Response to Reviewer yjWt (2/2)

W3/Q2: Coherent natural images and content diversity. Diversity alone does not remove the effect. Removal stays specific on natural photographs at full CIFAR-10 diversity (5,000 images per class, AUC 0.996), and raising co-exposure routes class information on tiles drawn from 60,000 unlabeled STL-10 photographs, +2.5 percentage points, positive in every seed, with probe images disjoint from pretraining (App. I). What removes the effect is redundancy. On whole faces the eye node reads mouth opening at 73.8% before any edit, above what routing reaches on tiles, and the relation's share of the objective is a ninth of its share on tiles. On RelCIFAR the share falls fiftyfold from E=4 to E≥16, and the gain falls with it. The revision restricts the practical claim accordingly (Section 4.4): graph edits control the relations that the input does not already make redundant, and a baseline model shows before any edit where that is the case, as the 73.8% does on faces. We see a mechanism with a measured scope as the contribution, rather than a new schedule for standard pretraining.

W4/Q3: Objective and regularization. The variance and covariance terms prevent the collapse we measured without them (effective rank 4.7 of 192), the same reason C-JEPA (Mo & Tong, NeurIPS 2024) adds VICReg to I-JEPA. Every removal condition reports its effective rank (Tables 6 and 10), and no conclusion rests on a collapsed model. Pixel reconstruction already reproduces the removal result (16 seeds, both specificities negative, AUC 0.968). We agree s=5 is the cleaner test, since at s=0 the relation can be solved by copying, so we ran the 2×2 removal at s=5 (8 seeds each, Table 18). Under pixel reconstruction the removed pair keeps 0.6% and 0.7% of its influence and the other relation 1.09 and 1.07 of its own, both specificities negative, AUC 1.000. The encoder's pooled effective rank is low there (7.5–11.6 at baseline), and we report it for every run. Under I-JEPA at 0.2× and 3× the VICReg weights both specificities are negative (AUC 1.000 and 0.961, effective rank 29.8–40.4 and 41.6–50.6). Stronger regularization leaves the removed pair more of its influence (0.10, 0.19, 0.32 at 0.2×, 1×, 3×) but never removes the specificity.

W5/Q4: Evaluation choices. Two features of the design already limit what inference-time mixing can explain.

  • Influence is measured with node j hidden: the target never enters the query, so it cannot leak into its own prediction. All models are probed with the identical query, and specificity is a within-model contrast between two pairs, so an artefact of the query would have to act on one pair and not the other.
  • On faces, the held eye sees its own mouth corner exactly as the raised eye does, yet only the raised eye gains: routing 5.0 points, positive in 15 of 16 seeds, reversed by the mirror schedule (5.9 points). The readouts use the target encoder, which processes full, unmasked images at every training step.

We also ran both controls you ask for.

  • Training-budget influence. With 75-patch contexts drawn as in training (whole nodes in the target with probability 0.48, conditioned on i in the context and j in the target), the removed pair keeps 0.25 and 0.30 of its influence and the other relation 1.16 and 1.11, specificity negative in both directions, AUC 1.000, as with the 135-patch query on the same models (Table 16).
  • Mouth hidden. With both mouth tiles replaced by fresh noise in the probe images, routing stays positive in every seed, 3.6 points for mouth opening and 5.4 for smiling, against 5.3 and 6.8 points with the mouth visible in the same models. The paired differences, −1.8 and −1.4 points, are not significant (both 95% CIs contain 0). The schedule has changed what the eye's own representation encodes about the mouth, not only how the encoder combines the two at readout (App. J).

Comments on the paper

1 comment

We thank the reviewers for careful and constructive reviews. They found the intervention design a clear strength (matched marginals, schedule exchange and the zero-co-exposure control; yjWt, woXr), the count formulation actionable, connecting characterization to construction (yjWt, woXr), and the reporting of where the effect does not hold informative (all three). All three rate the soundness concerns as minor or fixable. We have uploaded a revision and summarize it here. Each review receives a point-by-point reply.

The question, and what is new. Prior work relates global masking hyperparameters to what is learned (mask ratio and patch size select short- or long-range correlations; Bisulco et al., ICCV 2025), or describes masking through a graph whose vertices are views (Zhang et al., NeurIPS 2022). Semantic masking (SemMAE) assumes that where one masks shapes which relations are learned, and evaluates that assumption by downstream accuracy. We ask whether, with every region's masking frequency held fixed, the schedule can select which pairs of regions become predictive of one another, and answer with interventions:

  1. Removing one edge removes about 80% of that pair's predictive influence, while the other relation keeps its own (AUC 0.996).
  2. Co-exposure, not co-occurrence, makes a relation readable: two regions that appear in every input, each predicted more often than before but never from the other, end at chance.
  3. Exchanging two schedules exchanges the learned structure (in every seed), and the effect is graded in co-exposure.
  4. The realizable graphs have an exact characterization (a covariance at fixed marginals, with LP certificates) and can be compiled.

New analyses from existing runs (seed-paired 95% CIs in App. H):

  • Locality, fully accounted (App. H). Under removal, the removed relation keeps 0.20 and 0.23 of its influence, and the other relation 1.30 and 1.10. The pairs that carry no information split by direction: those whose source is a node of the removed pair fall nearly as far (0.26, 0.31), those whose target is one keep theirs (1.08, 1.04). Removing an edge thus silences its two nodes as sources and leaves every other informative relation in place. Under zero co-exposure, at every probe size from k=256 up, the edited relation falls to chance (k=4096: 49.5%) and the untouched one stays above it (57.1%).
  • Generality (App. I). Removal stays specific on natural photographs at full CIFAR-10 diversity (5,000 images per class, AUC 0.996) and on the transposed geometry (16 seeds, AUC 1.000). Raising co-exposure routes information on tiles drawn from 60,000 unlabeled STL-10 photographs: +2.5 percentage points, positive in every seed.

New controls requested by the reviewers (8 seeds each, App. H and J):

  • Full-graph match (yjWt Q1, woXr). W and the marginals are linear in the count distribution, so they can be matched exactly. Three realizations of the RelForest swap with identical 12-entry W and node marginals all route the raised relation in every seed: the paper's projection (3,328 count states) by 24.4 points, an LP solution on 8 states by 25.8, and an LP solution that masks every node whole by 17.2 (Table 17). The graph fixes the direction, entropy does not change the size, and whole-node masking reduces it.
  • Influence at the training budget (yjWt Q4, A3At). With 75-patch contexts drawn as in training, removal stays specific in both directions, AUC 1.000 (Table 16).
  • Mouth hidden at readout (yjWt Q4). With both mouth tiles replaced by noise, face routing stays positive in every seed (3.6 and 5.4 points, against 5.3 and 6.8 with the mouth visible). Neither paired difference is significant.
  • Objective and regularization (yjWt Q3). At s=5, removal is specific under pixel reconstruction (AUC 1.000) and at 0.2× and 3× the VICReg weights (AUC 1.000 and 0.961, Table 18).
  • Probe and untrained encoder (woXr). Relational readouts move by at most 0.6 points over AdamW weight decay from 0 to 10, and an untrained encoder reads every relation at chance.

Revisions in the PDF (changes in blue). A precise novelty statement with discussion of Bisulco et al., Zhang et al. and SemMAE (Sections 1 and 2). Locality restated as a concentrated effect, with CIs (Section 4.1, App. H). The full loss, predictor and regularizer (Section 3.1, App. A). The corrected RelCIFAR description (Section 3.1, App. E). A derivation of the rank–alignment relation (App. C). Budget-consistent support and the boundary I-projection (Section 5.1, App. D). Consistent seed and ± reporting (App. A, Table 6). The scope of the practical claim (Section 4.4). Full-graph matching and the other new controls (Section 5.5, App. H, App. J). An experiment map listing every comparison and what it holds fixed (App. G).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.