acceptodds
Under review as a conference paper at ICLR 2027

An Identifiability Theory of Masked Prediction: Mode Blindness and Mask Schedules

Abstract

Masked prediction learns to infer missing variables from visible context selected by a mask schedule. When does small masked-prediction excess risk guarantee recovery of the true joint distribution? We study this question on finite product spaces under the joint masked-block log loss, with conditionals induced by a single joint distribution. To quantify recovery, we introduce an -identifiability modulus that measures the worst-case error among joint distributions with population excess risk at most . For data with separated modes pinned down by sufficient context, mask schedules supported only on large contexts can permit nonvanishing mode-weight errors at an exponentially small excess risk. An information decomposition explains why: the loss captures mode-weight mismatch only where context leaves the mode uncertain. We prove two-sided bounds showing that, over a fixed range of mode weights, masked-prediction sensitivity to mode reweighting is governed by schedule-averaged residual mode uncertainty. Assigning schedule mass to low-visibility masks that retain this uncertainty yields recovery bounds within the reweighting family. Moreover, positive full-mask probability characterizes uniform control of joint KL divergence by excess risk. Exact calculations and controlled gradient optimization validate these predictions.

Primary area:
learning theory
Keywords:
Masked prediction, identifiability, self-supervised learning, mask schedules, joint-distribution recovery

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hjMQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the gap between the masked-prediction objectives and the true joint distribution. The authors found that, under the mode-reweighting assumption, the masked prediction introduces an issue called mode blindness. This issue indicates that a model can achieve near-optimal loss given enough context while incurring substantial errors in mode weights. This paper also provides recovery guarantees from low-visibility masks, which ensure that the model stays uncertain about the mode to learn the global mode weights.

Critical strengths
  1. This paper studies the important mode blindness problem of masked prediction.

  2. The findings are reasonable, and the analysis provides a complete set of theoretical results in the mode-reweighting distribution family.

  3. The authors also provide empirical experiments to show that low-visibility interventions restore sensitivity to mode weights.

Critical weaknesses
  1. The theoretical analysis is restricted to the mode-reweighting family. However, the real data distribution may not lie in this family.

  2. Although low-visibility masking helps avoid the mode blindness problem, it may reduce the performance of masked prediction models.

  3. Although this paper is well-structured, its wording is hard to follow. There are many theorems and equations in the main paper, and more examples and illustrated figures are needed for better understanding.

Questions for authors
  1. Could the authors test whether mode blindness also exists in other data distributions?

  2. Could the authors show how different masking rates influence the mode blindness problem and model prediction performance? Is there any trade-off?

  3. Would the proposed low-visibility masking work for real-world datasets?

5 comments

We thank you for recognizing the importance of mode blindness, the theoretical results within the reweighting family, and the empirical evidence that low-visibility masking restores sensitivity to mode weights. We address your concerns and questions below point by point, along with the corresponding revisions in the manuscript (highlighted in blue).


W1 & Q1. Scope of the reweighting analysis and evidence across distributions

"The theoretical analysis is restricted to the mode-reweighting family. However, the real data distribution may not lie in this family."

The reweighting family is constructed directly from the target distribution and imposes no prespecified parametric form on its components. Given a nontrivial partition into two mode regions, defining p±=p(⋅∣X±)p^\pm=p(\cdot\mid\mathcal X_\pm) and w=p(X+)w=p(\mathcal X_+) gives p=wp++(1−w)p−p=wp^++(1-w)p^-. The restriction is on the alternative models: they change the mode weights while preserving the within-mode distributions, which may contain arbitrary dependencies.

The exponential blindness result additionally requires the mode-pinning assumption and high-visibility masking. Under these conditions, even alternatives with exactly correct within-mode distributions can have constant joint TV error at exponentially small excess risk. Because these alternatives are admissible joint distributions, they establish a lower bound on the full identifiability modulus.

Theorem 4.2 characterizes sensitivity within the bimodal reweighting family for arbitrary schedules and weights in a fixed compact interval containing the true weight. The low-visibility recovery guarantee in Corollary 4.3 additionally requires the stated residual-uncertainty assumption. Separately, Proposition 4.1 characterizes joint-KL control uniformly over all data distributions and admissible joint models: under joint masked-block loss, positive full-mask probability is necessary and sufficient.


Q1: "Could the authors test whether mode blindness also exists in other data distributions?"

Yes. Our existing experiments include an interacting Curie–Weiss distribution at β=1.5\beta=1.5, beyond the conditioned-product construction (Appendices H.1–H.2, Figure 5a). This provides an example with dependent coordinates that also exhibits strong suppression of the mode-weight signal. The unimodal Curie–Weiss control at β=0.5\beta=0.5 shows much weaker suppression. These comparisons test the role of data geometry while retaining reweighted alternatives.

Complementing this comparison across data distributions, our neural experiment trains all network parameters on the conditioned-product target, allowing both mode weights and within-mode distributions to change. The low-visibility interventions reduce mode-weight error and joint KL in all five paired seeds. Its protocol and results appear in Appendix H.7 (lines 3011–3099).

W2 & Q2. Effects of masking on prediction performance and mode-weight recovery

"Although low-visibility masking helps avoid the mode blindness problem, it may reduce the performance of masked prediction models."

Q2: "Could the authors show how different masking rates influence the mode blindness problem and model prediction performance? Is there any trade-off?"

We agree that stronger sensitivity to mode weights need not preserve prediction performance under a fixed training budget. Our theory relates recovery to the achieved excess risk, so prediction quality and optimization must also be evaluated empirically.

We thus compare fully trainable neural joint models on sequences of length 13. The baseline reveals 12 coordinates; the interventions reveal one coordinate with probability 1% or 10% and otherwise follow the baseline. These percentages describe how often the low-visibility masks are selected. Each schedule uses 10,000 updates with batches of 512 examples across five paired seeds. Training examples and optimizer updates are matched; computational work differs across schedules.

The table reports mean ±\pm sample standard deviation. Reference excess risk measures prediction loss above the Bayes optimum on the same task with 12 visible coordinates for every model. Joint KL measures overall joint-distribution error. All metrics are computed exactly by enumeration; reference excess risk is in nats per masked block, and joint KL is in nats.

Low-visibility probabilityMode-weight errorReference excess riskJoint KL
0%0.298±0.0004870.298 \pm 0.0004870.0542±0.05170.0542 \pm 0.05171.99±0.5591.99 \pm 0.559
1%0.243±0.01100.243 \pm 0.01100.0557±0.05180.0557 \pm 0.05180.991±0.5540.991 \pm 0.554
10%0.0698±0.05530.0698 \pm 0.05530.0123±0.01260.0123 \pm 0.01260.143±0.1620.143 \pm 0.162

At 1%, mode-weight error and joint KL decrease in every paired run, while reference excess risk increases slightly in every run. At 10%, both recovery metrics improve in every run. Reference excess risk decreases in three of the five runs and also decreases on average. Component diagnostics show better minority-component fit and worse majority-component fit under both interventions (Appendix H.7, Table 7).

Thus, improved joint recovery can come with a prediction cost, but this cost is not inevitable in the tested setting. This comparison evaluates conditional log loss on a common reference task; downstream representation performance remains untested. The baseline still underfits the minority component, so the observed errors involve both mode frequencies and within-mode fit. Appendix H.7 provides the full protocol and diagnostics.

W3. Clarity of the exposition

"... its wording is hard to follow. There are many theorems and equations in the main paper, and more examples and illustrated figures are needed for better understanding."

We have added a numerical illustration after Lemma 3.1 to make the mechanism concrete (lines 242--246). Changing the global mode weight from 0.50.5 to 0.750.75 changes a nearly certain mode posterior from 0.990.99 to approximately 0.99660.9966. This shows how a substantial global weight error can produce only a small change in the conditional mode posterior.

In addition, we have clarified the interpretation of the KL decomposition in Remark 3.1 (lines 252--255) and explained how residual mode uncertainty controls the prediction loss's sensitivity to mode weights following Theorem 4.2 (lines 378--382). We have explicitly presented the AT bound in Section 3.1 as preliminary background to distinguish it from our results (lines 169--172 & lines 189--190). We have also added Figure 3 to visualize the neural model’s mode-weight recovery, conditional prediction, and within-mode learning across mask schedules (lines 486--501).


Q3. Applicability of low-visibility masking to real-world data

Q3: "Would the proposed low-visibility masking work for real-world datasets?"

Yes, our theory suggests that low-visibility masks may help recover mode frequencies when visible context would otherwise reveal the mode. Theorem 4.2 characterizes this sensitivity within the reweighting family under joint masked-block loss, and Corollary 4.3 gives recovery guarantees under the stated conditions. Appendix H.6 and Figure 9 provide a related observation in natural text: uncertainty about corpus membership, estimated using fitted predictors, decreases as more context becomes visible in code–prose and German–English data. These observations motivate testing whether retaining such uncertainty during training improves mode-frequency recovery.

Low-visibility training also has practical precedents in generative modeling. Gupta et al. (2023, Table 2c) show that MaskViT's variable masking of 50% to nearly 100% of future-frame tokens improves video prediction over the tested fixed masking rates on BAIR. Nie et al. (2025) train LLaDA with masking probabilities sampled across [0,1][0,1], including nearly fully masked text, and obtain competitive language-model performance. These studies show that training with heavily masked targets is feasible, and our analysis identifies a potential role for low visibility in constraining mode frequencies.

An end-to-end intervention on real-world neural models remains beyond the present study. Such an evaluation must account for the prediction objective, finite-data estimation, model parameterization, and optimization. It must also measure effects on both mode frequencies and downstream performance.


References

Gupta, Agrim, et al. "Maskvit: Masked visual pre-training for video prediction." ICLR 2023.

Nie, Shen, et al. "Large language diffusion models." NeurIPS 2025.


Thank you again for your careful review. We particularly appreciate your emphasis on connecting the theoretical results to prediction performance, as well as your suggestions for clearer exposition.

Reviewer_D8bqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies whether minimizing masked prediction risk provably leads to the identification of the joint data distribution. The main result identifies a failure mode due to "mode blindness": when the data distribution has well-separated modes and the modes themselves are easily identified from the context, masked prediction cannot recover the mode weights, resulting in an uncontrollable TV distance between the learned joint distribution and the true joint distribution.

Critical strengths
  • The main idea of relating mode blindness to the potential failure modes of masked prediction is of interest.
Critical weaknesses
  • The motivation of the paper is actually a bit weak to me. In my opinion, the central question considered (when does small masked-prediction excess risk guarantee recovery of the true joint distribution?) is in fact not very relevant to many masked prediction methods used in practice. For example, the main goal of BERT, MAE, and JEPA (all mentioned in the first paragraph of the introduction) is to learn good representations rather than recover the full joint data distribution for generative modeling; this supports identification studies from the perspective of latent variable modeling/representation learning, but not the question of joint-distribution identification considered by the paper.
  • We all know that in general, learning conditional probabilities by itself (as done by masked prediction) does not imply identifying the joint distribution. The paper mainly identifies and analyzes a specific failure mode of mode blindness, where the TV distance between learned and true joint distributions could remain constant even if the conditional excess risk is exponentially small. However, it is not clear to me how this failure mode connects to masked prediction practice. For example, the motivating example of code and prose mainly says that if the unmasked context is informative enough to distinguish code and prose, then a model trained by masked prediction cannot recover the true weight of code and prose in the training distribution, which seems to be useless for any practical LMs. The notion of "mode blindness" itself is also not new: prior work has identified similar failure modes for score-based models such as diffusion [1].
  • The main analysis framework of approximate tensorization has already been introduced by Li et al. In general, the technical contribution of the paper is a bit limited to me.

[1] Blindness of score-based methods to isolated components and mixing proportions

Questions for authors
  • Why is the code-prose example a problem for LMs?
  • How does mode blindness connect to practical masked prediction methods such as masked language modeling or diffusion models?
4 comments

We thank you for your interest in the connection between mode blindness and masked prediction. We address your concerns and questions below point by point, along with the corresponding revisions in the manuscript (highlighted in blue).


W1. Motivation and scope

"... the main goal of BERT, MAE, and JEPA ... is to learn good representations rather than recover the full joint data distribution ...; this supports identification studies from the perspective of latent variable modeling/representation learning, but not the question of joint-distribution identification considered by the paper."

We agree that BERT, MAE, and JEPA primarily aim to learn useful representations, but this does not make joint-distribution analysis irrelevant to masked prediction. We cite these methods to situate masked prediction as a general learning paradigm. In generative masked modeling, the probabilities assigned to complete observations are themselves part of the learning target, making joint-distribution recovery directly consequential.

This setting has both theoretical and practical precedents. Li et al. (2024, Section 2.3) establish finite-sample joint-recovery guarantees under their stated assumptions. Sahoo et al. (2024, Section 5.1 and Table 4) show that a pretrained BERT-style model can be adapted for generation through a masked-diffusion objective while maintaining comparable downstream GLUE performance.

Latent-variable analysis and joint-distribution recovery address complementary aspects of what a masked-prediction objective determines. For example, Kong et al. (2023) study when MAE identifies shared latent factors under assumptions on the data-generating process. Our analysis distinguishes identifying the active mode from constraining its population frequency. Visible context can nearly determine the mode while conditional prediction remains weakly sensitive to its global weight. By working directly with the observable joint distribution and a mode partition, we can study this sensitivity without prescribing a particular latent-to-observation map.

In Appendix B, we now distinguish these recovery targets and their assumptions more explicitly (lines 988--1003). We have also made a revision to the introduction's first paragraph to clarify generative modeling as a motivation for the recovery question (lines 30--35).

W2(a). Exact identification and stable recovery

"... learning conditional probabilities by itself ... does not imply identifying the joint distribution. ... TV distance ... could remain constant even if the conditional excess risk is exponentially small."

Our negative result applies even when the exact optimum uniquely determines the joint distribution. For example, the construction in Appendix G.3 has full support. Under uniform single-site masking, attaining Bayes-optimal risk determines every full conditional. These conditionals fix probability ratios between configurations that differ in one coordinate and therefore determine the normalized joint distribution.

Nevertheless, Theorem 4.1 shows that, as dimension grows, constant-order TV error can coexist with exponentially small positive excess risk. Exact identification therefore does not ensure that recovery remains stable as the dimension grows. In our construction, this gap has a concrete interpretation: the model represents both within-mode distributions exactly, but assigns them incorrect probabilities; its joint TV error comes entirely from this mode-weight error.

Beyond this negative result, Theorem 4.2 identifies residual mode uncertainty as the quantity governing sensitivity to mode weights. Corollary 4.3 then gives recovery guarantees within the reweighting family under the stated low-visibility assumption. These results further explain how the mask schedule changes the prediction accuracy required to constrain global frequencies.


W2(b) & Q1. Why mode frequencies matter for generation

"... not clear to me how this failure mode connects to masked prediction practice. ... cannot recover the true weight of code and prose ... useless for any practical LMs."

Q1: "Why is the code-prose example a problem for LMs?"

We respectfully disagree that mode-weight errors lack practical relevance. They matter when generation is expected to match a specified target distribution. The code–prose example concerns a prefix that remains compatible with both types of continuation.

In our illustrative two-mode model, consider a prefix satisfying pV+(xV)=pV−(xV)>0p_V^+(x_V)=p_V^-(x_V)>0. The probability of a code continuation is then ww under the target and λ\lambda under the reweighted model. These probabilities differ by ∣w−λ∣|w-\lambda|, even though the within-domain distributions are identical. Accurate prediction within recognizable code or prose therefore leaves open whether the model assigns the intended probabilities when the domain remains ambiguous.

The example illustrates why matching a specified target distribution requires learning the relative frequencies of plausible responses. Distributional alignment also provides a concrete application: Sorensen et al. (2024, Section 3.3) formalize the task of matching response distributions to those of a specified population, while Santurkar et al. (2023) evaluate such alignment using OpinionQA.


Q2. Connection to language/diffusion modeling

Q2: "How does mode blindness connect to practical masked prediction methods such as masked language modeling or diffusion models?"

For masked language modeling, Appendix C.7 extends the negative result to factorized per-token log loss. Under Theorem 4.1's assumptions, Corollary C.1 introduces a factor of at most NN, preserving exponential decay when the minimum number of visible coordinates grows linearly with sequence length. The counterexamples use conditionals from a single joint distribution, so the obstruction does not depend on conditional incompatibility. Full-mask per-token loss scores only coordinate marginals, however, so Proposition 4.1's joint-KL guarantee does not apply.

For masking or absorbing-state diffusion, denoising objectives can be expressed as weighted masked-prediction losses (Sahoo et al., 2024; Shi et al., 2024). Applying our analysis requires accounting for the prediction loss and the effective mask weights induced by time sampling and loss weighting. Appendix C.7 supplies the negative extension for factorized losses under its stated assumptions; the two-sided sensitivity and recovery guarantees concern joint masked-block loss and do not automatically transfer to factorized losses.

Our population-level analysis clarifies what the training objective itself constrains, separately from finite-data and optimization effects. We believe this provides a basis for assessing what conclusions about distributional recovery are justified by prediction accuracy, informing both objective design and model evaluation. To complement this with evidence from model training, we added a neural experiment in which all model parameters are learned. The results show that the mask schedule affects joint recovery under the tested training budget (Section 5; Appendix H.7).

W2(c). Relationship to prior score-based blindness

"The notion of 'mode blindness' itself is also not new: prior work has identified similar failure modes for score-based models such as diffusion..."

We agree that similar insensitivity to mixture weights is known in score-based learning. Wenliang and Kanagawa (2021) show that Fisher and Stein discrepancies can become small despite incorrect mixing proportions. Koehler et al. (2023) also study mixture-weight failures and connect statistical efficiency to functional inequalities. We cite these works and discuss their relationship to our results in the revised Appendix B.

Our work instead characterizes how the mask schedule controls sensitivity to mode weights in masked prediction. When the visible coordinates nearly identify the mode, changing its global weight has little effect on the conditional distribution of the masked coordinates. Contexts that leave the mode uncertain retain a signal for that weight. The mask schedule determines how often the objective evaluates each kind of context, and therefore how strongly it penalizes incorrect mode frequencies. Theorem 4.2 quantifies this dependence within the reweighting family through two-sided bounds involving mode-weight KL and schedule-averaged residual mode MMSE. These bounds hold for arbitrary mask schedules and finite reweightings in a fixed compact interval containing the true weight.


W3. Technical contributions beyond the AT framework

"The main analysis framework of approximate tensorization has already been introduced by Li et al ... the technical contribution of the paper is a bit limited ..."

Section 3.1 recalls the established block approximate-tensorization (AT) framework (Caputo and Parisi, 2021; Li et al., 2024) and expresses the AT–Pinsker recovery argument in terms of our ε\varepsilon-identifiability modulus. We do not claim Eq. (9) as a new result. To make Section 3.1's preliminary role clearer, we have retitled it "Preliminaries: Block Approximate-Tensorization Bounds" and explicitly attributed Eq. (9) to the established framework (lines 169--190).

Our main blindness and sensitivity proofs instead analyze how mode reweighting affects conditional predictions, without relying on the AT bound (Lemma 3.1, Proposition 3.1, and Lemma 4.1). The table below summarizes our principal results and formulation.

Result/FormulationWhat it establishes
Theorem 4.1Shows that nearly optimal prediction can leave substantial joint error: under mode pinning and high-visibility masking, constant TV error persists at exponentially small positive excess risk.
Theorem 4.2 and Corollary 4.3Characterize how residual mode uncertainty controls sensitivity to mode weights within the reweighting family, through two-sided bounds for arbitrary schedules and finite reweightings in a fixed compact interval. Under the stated low-visibility assumption, these further yield recovery bounds within that family.
Corollary 4.1Explains when existing AT guarantees deteriorate: the blindness construction forces AT constants to grow exponentially at both the data distribution and a reweighted alternative.
Proposition 4.1Characterizes distribution-free joint-KL control by excess risk: under joint masked-block log loss, positive full-mask probability is both necessary and sufficient.
Definition 2.2Makes the near-optimal recovery question quantitative: how large can joint-distribution error remain at a given excess-risk tolerance?

References

Li, Yuchen, et al. "Promises and pitfalls of generative masked language modeling: Theoretical framework and practical guidelines." ICML 2024.

Sahoo, Subham S., et al. "Simple and effective masked diffusion language models." NeurIPS 2024.

Kong, Lingjing, et al. "Understanding masked autoencoders via hierarchical latent variable models." CVPR 2023.

Sorensen, Taylor, et al. "Position: A roadmap to pluralistic alignment." ICML 2024 Position Track.

Santurkar, Shibani, et al. "Whose opinions do language models reflect?" ICML 2023.

Shi, Jiaxin, et al. "Simplified and generalized masked diffusion for discrete data." NeurIPS 2024.

Wenliang, Li K., and Heishiro Kanagawa. "Blindness of score-based methods to isolated components and mixing proportions." NeurIPS Workshop 2021.

Koehler, Frederic, Alexander Heckett, and Andrej Risteski. “Statistical Efficiency of Score Matching: The View from Isoperimetry.” ICLR 2023.

Caputo, Pietro, and Daniel Parisi. "Block factorization of the relative entropy via spatial mixing." Communications in Mathematical Physics 388.2 (2021): 793-818.


Thank you again for your time in reviewing our work. We hope our responses clarify the paper's motivation and contributions.

Dear reviewer D8bq, we thank you very much for your review. Before the discussion period ends, we would like to check if our response has addressed your concerns or if there are remaining concerns we may clarify? Kind regards, Authors

Reviewer_K7zZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a practical problem in masked prediction from theoretical perspective: when a small masked-prediction excess risk guarantees that the model’s joint distribution is close to the data distribution ? It first defines ϵ\epsilon-identifiability modulus to quantify the largest possible error in the joint distribution among all models with ϵ\epsilon bounded masked-prediction excess risk. Then, mode-blindness theorem shows the mode-weight errors can remain constant with exponentially small masked excess risk. The work further derives two-sided sensitivity bounds of mode prediction error with schedule-averaged residual mode MMSE. Empirical results are provided to validate the above theorems.

Critical strengths
  1. This paper is a solid theoretical work with rigorous analysis and well-motivated research question.

  2. This work proposes some elegant and informative tool, e.g. ϵ\epsilon-identifiability and MMSE, to build the bounds, which makes it clean and straightforward.

Critical weaknesses
  1. The empirical results are limited to constructed settings that only global mode-weight parameter is optimized. There is no evidence to show the mode blindness when the component and mixture weights are learned.

  2. The discussion of relative known phenomena should be argued more sharply. It's still not so clear to me the relationship between the underlying phenomenon of mode blindness and previous studies. This work should build more connections with previous studies and clarify which parts are new and which are a careful formalization of existing results.

Questions for authors

Is it possible to add an experiment using a small neural model trained on a synthetic multimodal dataset ? This would test whether the proposed mechanism remains observable outside the analytically controlled model family.

3 comments

We thank you for recognizing the paper's motivation and the value of the ε\varepsilon-identifiability modulus and MMSE perspective. We address your concerns and questions below point by point, along with the corresponding revisions in the manuscript (highlighted in blue).


W1 & Q1. Jointly learning mode weights and within-mode distributions

"... only global mode-weight parameter is optimized. There is no evidence to show the mode blindness when the component and mixture weights are learned."

Q1. "Is it possible to add an experiment using a small neural model trained on a synthetic multimodal dataset?"

We have added a small neural model experiment in which all parameters are trained, allowing both mode probabilities and within-mode distributions to change. The model receives neither mode labels nor component probabilities.

The model is an MLP with two hidden layers of width 64. It assigns each sequence a scalar score fϕ(x)f_\phi(x) and defines a normalized joint distribution qϕ(x)∝exp⁡fϕ(x)q_\phi(x)\propto\exp f_\phi(x). We train it on the conditioned-product target with N=13N=13, θ=0.8\theta=0.8, and w=0.7w=0.7. The baseline reveals 12 of the 13 coordinates. The interventions reveal one coordinate with probability 1% or 10% and otherwise follow the baseline. For each schedule, we use 10,000 Adam updates with batches of 512 examples. We use five seeds paired across schedules by initialization and data stream. Training examples and optimizer updates are matched across schedules, although computational work differs. We compute conditional probabilities and distributional metrics by exact enumeration over the 8,192 states.

In the table below, DrefD_{\mathrm{ref}} denotes excess risk on the same task with 12 visible coordinates for every model. Within-mode KL is weighted by the target mode probabilities. Entries report the mean ±\pm sample standard deviation across five seeds. KL is measured in nats and excess risk in nats per masked block.

Training scheduleMode-weight errorDrefD_{\mathrm{ref}}Within-mode KLJoint KL
High visibility0.298±0.0004870.298 \pm 0.0004870.0542±0.05170.0542 \pm 0.05170.708±0.4850.708 \pm 0.4851.99±0.5591.99 \pm 0.559
1% low visibility0.243±0.01100.243 \pm 0.01100.0557±0.05180.0557 \pm 0.05180.698±0.5030.698 \pm 0.5030.991±0.5540.991 \pm 0.554
10% low visibility0.0698±0.05530.0698 \pm 0.05530.0123±0.01260.0123 \pm 0.01260.121±0.1360.121 \pm 0.1360.143±0.1620.143 \pm 0.162
Full-mask control0.0125±0.006550.0125 \pm 0.006550.000323±0.00003400.000323 \pm 0.00003400.00147±0.0002490.00147 \pm 0.0002490.00193±0.0004180.00193 \pm 0.000418

Both low-visibility interventions reduce mode-weight error and joint KL in all five paired seeds. At 1%, mean reference excess risk increases slightly from 0.0542 to 0.0557. At 10%, recovery improves further; mean reference excess risk also decreases, with reference prediction improving in three of the five paired runs. These results show that the mask schedule affects joint recovery when mode probabilities and within-mode distributions are learned together, under the tested training budget. The baseline still underfits the minority component, so the observed errors involve both mode frequencies and within-mode fit. The full-mask control shows that the same architecture can closely fit the target within the update budget when trained on full-mask loss.

Our original controlled experiments complement this newly added experiment by fixing both components at their true distributions. This isolates mode-weight sensitivity, with DTV(p,qλ)=∣w−λ∣D_{\mathrm{TV}}(p,q_\lambda)=|w-\lambda|. We have clarified this purpose at the beginning of Section 5 (lines 425--430). Figure 3 shows the neural learning curves for this experiment (lines 481--509). Appendix H.7 provides the full protocol, endpoint results, and component diagnostics (lines 3010--3099).

W2. Prior findings and the contribution of our analysis

"The discussion of relative known phenomena should be argued more sharply. ... clarify which parts are new and which are a careful formalization of existing results."

Mixture-weight blindness has precedents in score-based learning. Wenliang and Kanagawa (2021) and Koehler et al. (2023) study failures to recover mixture weights and the limitations of local fitting objectives. Our contribution is to quantify how coordinate masking governs mode-weight sensitivity under masked prediction. Specifically, Theorem 4.2 gives two-sided bounds on masked excess risk in terms of the product of mode-weight KL and schedule-averaged residual mode uncertainty. These bounds hold within the bimodal reweighting family for arbitrary mask schedules and finite reweightings in a fixed compact interval containing the true weight. Under mode pinning and high visibility, Theorem 4.1 establishes constant worst-case TV error at an exponentially small positive excess-risk tolerance. Under the stated low-visibility assumption, Corollary 4.3 gives recovery guarantees within the reweighting family.

The block AT framework and the AT–Pinsker argument in Eq. (9) are established results (Caputo and Parisi, 2021; Li et al., 2024). We express the established AT–Pinsker bound in terms of our ε\varepsilon-identifiability modulus as background for comparison. Corollary 4.1 uses the blindness result to establish exponential lower bounds on AT constants. The proofs of our main blindness and sensitivity results analyze conditional mode posteriors directly and do not rely on this framework.

To clarify this distinction, we have retitled Section 3.1 "Preliminaries: Block Approximate-Tensorization Bounds" and clarified the attribution of Eq. (9) (lines 169--172 & lines 189--190). The revised related-work discussion in Appendix B explicitly distinguishes prior mixture-weight failures, the established recovery framework, and our schedule-dependent results (lines 975--985).


References

Wenliang, Li K., and Heishiro Kanagawa. "Blindness of score-based methods to isolated components and mixing proportions." NeurIPS Workshop Your Model is Wrong: Robustness and misspecification in probabilistic modeling 2021.

Koehler, Frederic, Alexander Heckett, and Andrej Risteski. “Statistical Efficiency of Score Matching: The View from Isoperimetry.” ICLR 2023.

Caputo, Pietro, and Daniel Parisi. "Block factorization of the relative entropy via spatial mixing." Communications in Mathematical Physics 388.2 (2021): 793-818.

Li, Yuchen, et al. "Promises and pitfalls of generative masked language modeling: Theoretical framework and practical guidelines." ICML 2024.


Thank you again for your time in reviewing our work. We appreciate your constructive suggestions, which we believe have strengthened the paper.

Dear reviewer K7zZ, we thank you very much for your review. Before the discussion period ends, we would like to check if our response has addressed your concerns or if there are remaining concerns we may clarify? Kind regards, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank all three reviewers for their thoughtful evaluations and constructive feedback. We appreciate K7zZ’s recognition of the motivation, rigorous analysis, and ε\varepsilon-identifiability and MMSE tools. We also thank hjMQ for highlighting the importance of mode blindness and the theoretical and empirical results, and D8bq for their interest in its connection to masked prediction.


We have provided point-by-point responses and revised the manuscript, with changes highlighted in blue. The main additions and clarifications are:

  • New neural experiment (K7zZ, hjMQ, D8bq). We added a fully trainable neural model that learns both mode probabilities and within-mode distributions (Section 5; Appendix H.7). Across five paired seeds, assigning 10% probability to low-visibility masks reduces mean mode-weight error from 0.2980.298 to 0.06980.0698 and mean joint KL from 1.991.99 to 0.1430.143. We also evaluate prediction on a common reference task, report component-level diagnostics, and explain the experiment's limitations.

  • Clearer motivation and practical connections (D8bq, hjMQ). We clarify the generative motivation for joint-distribution recovery, explain why mode-frequency errors matter, and discuss connections to masked language modeling and diffusion. We distinguish joint-block from factorized objectives and clarify the scope of the practical implications (Introduction; Appendices B and C.7).

  • Sharper positioning of theoretical contributions (K7zZ, D8bq). We distinguish exact identification from stable recovery near the optimum, highlight our two-sided sensitivity bounds relating masked excess risk to mode-weight KL and schedule-averaged residual mode MMSE, and clarify how these results differ from prior score-based blindness and the established approximate-tensorization framework (Section 3.1; Appendix B).

  • Improved exposition and clearer empirical scope (hjMQ). We added a numerical posterior-shift illustration, clarified the interpretation of the KL decomposition and sensitivity bounds, and introduced neural learning curves (Figure 3). Our responses also clarify the scope of the existing interacting Curie–Weiss experiments and distinguish restrictions on the reweighting family from the assumptions underlying our theoretical guarantees.


We hope these revisions address the central concerns and make the paper's contributions and scope clearer. Our individual responses to K7zZ, D8bq, and hjMQ provide the detailed explanations, supporting evidence, and qualifications. We welcome any further questions or discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.