acceptodds
Under review as a conference paper at ICLR 2027

Beyond Local Prediction Frequencies: Hidden Path Coupling in Masked Diffusion

Abstract

Masked diffusion language models (MDLMs) are trained through local masked predictions, but generation composes these predictions along complete reveal schedules. A broad class of training objectives and train–inference alignment methods depends on a schedule only through how often each local prediction problem is used, so frequency-matched schedules induce the same training objective. We prove that this training equivalence does not imply generation equivalence: schedules with the same objective can still produce different output distributions. We characterize exactly what information this frequency-based view loses and how the missing structure changes generation, and further prove that training generically becomes sensitive to it in our shared-representation population model; a controlled Transformer shows the same effect under symmetric masking. Near mutually consistent local predictions, we also identify the minimal additional schedule information whose matching removes every second-order discrepancy and derive a likelihood-based test requiring no retraining. On MDLM-OWT, output distributions differ by a median in total variation across randomly sampled schedule pairs with identical local prediction frequencies.

Primary area:
generative models
Keywords:
masked diffusion language models, random-order generation, train-inference alignment, decoding schedules, path coupling, conditional consistency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ir3ZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper shows that two unmasking schedules for masked diffusion models (MDMs) that share the same "local frequencies" (defined as the frequency with which a schedule sees a certain unmasking problem of the form "unmask position z conditioned on positions x and y already being unmasked") and the same predictor (i.e., the trained model) can still yield distinct marginal distributions over generations. In experiments, they approximate this scenario with finite samples, and show the proven effect occurs in practice, under their specific setup.

Critical strengths

The theory seems to be sound, and the results seem correct (as far as I can tell), and could be interesting; however, I don't think the problem is motivated (see below), so the relevance of the results remains questionable.

Critical weaknesses
  • Main concern: I think the problem statement of the paper has no practical motivation. The main result says two schedules with the same local problem frequencies can end up generating different marginal distributions. However, the setup under which this occurs assumes a schedule where the next token position to be revealed depends not only on the set of revealed tokens up to that point, but also on the order in which they've been revealed; at least one of the two schedules needs to have this property. Indeed, if both schedules with identical "local frequencies" do not depend on the order in which previously unmasked positions have been unmasked, then they (trivially) lead to the same marginal distribution. I believe this never shows up with MDMs in practice; even the two references the authors give themselves, uniform random sampling in LlaDA [1] and ancestral sampling in MDLM [2], do not use such schedules (these schedules don't depend on the exact order in which previously unmasked tokens have been unmasked). I am not aware of any schedule where this scenario comes up in practice, and the authors don't provide one, either; they only show this happens in toy examples. Furthermore, in practice, one typically uses more sophisticated schedules that depend on model predictions in some form (e.g. via model confidence scores) and that reveal more than one token at a time, which is also not covered by the author's setup. If the authors are able to provide convincing examples/references where this problem setup is actually relevant, I'm happy to reconsider my rating, but I currently believe the problem has no motivation in practice.
  • The paper is very difficult to parse; sentences are verbose and syntactically overly complex. A lot of terminology is confusing and does not follow what's commonly used in the literature; example: "Revealed coordinates are never revised, so the terminal assignment retains every intermediate value."; much more natural and in line with common terminology would be something like "The MDM sampler does not remask/resample already unmasked positions." The whole paper reads like this. While I haven't checked with any tools, I'm pretty sure the paper is (mostly) AI generated.
  • Some statements in the paper seem inaccurate; e.g., in the first two paragraphs of the introduction alone, it says that at training, one typically "predict[s] one of the remaining masked positions"; I'm not aware of any paper that does that, usually all remaining masked positions are predicted at a training step; and the second paragraph makes it sound like the inference schedule always determines the "local prediction problems" at training; however, usually training uses random masking independent of the inference schedule, and only more recently it has been explored to align the training schedules (approximately) with the inference schedule (e.g. [3]).

[1] Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large Language Diffusion Models. arXiv:2502.09992, 2025.

[2] Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and Effective Masked Diffusion Language Models. arXiv:2406.07524, 2024.

[3] Jaeyeon Kim, Jonathan Geuter, David Alvarez-Melis, Sham M. Kakade, and Sitan Chen. Stop Training for the Worst: Progressive Unmasking Accelerates Masked Diffusion Training. Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.

Questions for authors
  • can you provide examples of schedules actually employed in practice that fit your setup, i.e. two schedules that have the exact same local problem frequencies, that do not depend on model predictions, and where at least one of the two schedules depends on the order in which past tokens have been umasked?
3 comments

Thank you for your careful review and helpful comments.

W1 & Q1: “the problem statement of the paper has no practical motivation” and “examples of schedules actually employed in practice”

The practical motivation is to determine whether matching the local prediction tasks used during generation is enough to preserve the output distribution. A relevant example in existing MDM practice is finite-step ancestral sampling with parallel reveals, covered by our block extension in Proposition 11. Its diffusion clock carries information beyond the filled set. This connects the question to the MDLM sampler itself and to what local training–inference alignment can guarantee.

In the ancestral sampler of Sahoo et al. (2024, Section 3.2.3), positions are revealed using a value-independent clock, and already revealed tokens are retained. For the time-independent MDLM-OWT predictor, we can omit empty updates and represent the remaining steps as an ordered sequence of blocks. Earlier blocks carry information about how much of the clock remains. To see this directly, take four positions and four equal time intervals, conditional on all positions being revealed before the final cleanup. Each position then independently chooses one of four reveal intervals. After the prefix ({1,2})(\{1,2\}), the probability that the next block is {3,4}\{3,4\} is 3/73/7; after ({1},{2})(\{1\},\{2\}), it is 2/32/3. The filled set is identical. More generally, under the same conditioning, with two positions remaining the probability of revealing them together is (m+2)/(2K−m)(m+2)/(2K-m) after mm nonempty blocks in KK intervals. The full state (S,t)(S,t) is Markovian, but the set SS alone omits clock information that affects the continuation.

We can preserve every local frequency while removing this information. On the branch that finishes before cleanup, let f(S,B)f(S,B) be the ancestral sampler's frequency of revealing block BB from set SS. Our matched control chooses

pˉ(B∣S)=f(S,B)∑C≠∅f(S,C).\bar p(B\mid S)=\frac{f(S,B)}{\sum_{C\ne\varnothing}f(S,C)}.

Flow conservation ensures that it visits every block edge with exactly the same frequency. Consequently, it also preserves every token-level frequency ftoken(S,i)=∑B∋if(S,B)f^{\mathrm{token}}(S,i)=\sum_{B\ni i}f(S,B). Both laws ignore predictions and generated values and use the same parallel block predictor. The baseline is the existing ancestral sampler; the set-only replacement is our control for testing whether its local statistics suffice. Retaining this branch's probability and leaving the complementary cleanup branch unchanged also preserves local frequencies for the full sampler.

Finite-order NADE ensembles provide the single-position counterpart (Uria et al., 2014, Section 4). They select a complete order before generation and retain it. Our libraries {1234,2143}\{1234,2143\} and {1243,2134}\{1243,2134\} illustrate how this operation permits equal edge frequencies with different continuations after the same set. We measured this effect in eight controlled MDLM-OWT comparisons between random 32-order libraries and their set-only replacements. With identical edge frequencies, predictor, and continuation, output TV was 4.36% to 5.69%, with positive simultaneous 95% lower bounds in all eight FP32 comparisons. The order-library measurements quantify the response under learned conditionals, while the ancestral-clock calculation identifies a source of history dependence in an existing MDM sampler.

The relevance to alignment is concrete. Shih et al. (2022) weight conditional prediction losses by inference-time visitation frequencies. Equal local frequencies make this additive objective identical for every predictor. Our results identify when the generation distributions can nevertheless differ, what path information is missing, and how to detect the difference without retraining. Set-only rules, including idealized single-position random LLaDA, form the sufficient case because their edge frequencies recover every transition. Finite-order ensembles and the clock-dependent block formulation explain why the broader question arises. Confidence-based policies require including the prediction-dependent action probabilities. We will bring these examples and distinctions into the Introduction and the discussion beside Definition 1, so the practical question is clear before the formal results.

W2 & W3: “paper is very difficult to parse” and “Some statements in the paper seem inaccurate”

A standard MDM training forward pass predicts all masked positions and aggregates their token losses; training masks are generally sampled independently of the inference schedule. We will make both points explicit in the Introduction with the following replacement. “A training forward pass predicts all masked positions and aggregates their token-level losses. Each loss term is indexed by the visible context and its target position. Training masks are generally sampled independently of the inference schedule.” Our local prediction term indexes one contribution to that sum. We will present training–inference alignment, including the work you cite by Kim et al. (2026), as an explicit training-design choice.

We will also use the simpler terminology you suggest. The sentence you highlighted will become “The sampler never remasks or resamples a revealed position.” We will explain the four-order example before the formal results, separate assumptions from conclusions in long sentences, and describe the largest TV difference between frequency-matched schedules before introducing its name. These changes will make the setting and the main question easier to follow from the start.

Reviewer_ir3ZOpenReview ↗

Thank you for your response

I thank the authors for the rebuttal. However, I don't believe my main concerns have been adequately addressed. My concern that the paper lacks motivation, and that their main result is not applicable to MDM samplers in practice, remains. The authors references two samplers: (1) the MDLM sampler; however, this sampler does not make position choices depend on previous reveal order in practice. (2) The paper by Uria et al. (2014); it seems like the authors just retrofitted this paper to my review, but it is not actually about MDM sampling at all, and predates MDMs by many years (while their sampler does seem to have the required order-dependence as far as I can see). Together, this means my main concern has not been sufficiently addressed. I also still believe the paper needs to be reworked in terms of its writing, and encourage the authors to do that. I will maintain my initial rating, and recommend rejection of the paper. I hope the authors understand my decision.

Reviewer_LsCVOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper focus on the orders of filling masked blanks. Every blank is filled once while the orders are chosen without using generated values results. Different rules may use each local prediction task equally often while still generate distinct distributions. The authors show that, this can first happen even with four positions. It explains that when changing the pairing of early stages and latter ones will change the output distribution. The authors also show that how training can perform this kind of effect, in a very simple model. Near a model whose predictions all align with one joint distribution, matching the frequencies of pairs of steps occur together eliminate the second-order difference. They also present a test based on four path probabilities and support the results with small models, training tests and several tasks on pre-trained language models.

Critical strengths

This paper explains the general effect. There are two cases in Theorem 4: changing the early order must change the probability of some early output; given that same output, changing the later order must also change the distribution of the remaining output. Only then does changing the pairing affect the final distributionThis helps us understand when the effect appears.

This paper connects its theory and the way of measuring the effect. The authors study how much extra information is needed to eliminate the difference. They give a test which utilize four path probabilities and doesn’t require retraining.

Critical weaknesses

The paper should be more clarify to show the link to commonly used generation rules. If one rule chooses the next position by only the set of filled blanks, and not the generated values, the local frequencies already determine the full order distribution.This paper must explain to which extent the existing rules have extra dependence on earlier orders that allows the hidden pairing to occur. It will affect how broadly the result of this paper can be applied in practice.

Questions for authors
  1. Which existing generation rules have the dependence on earlier orders needed for the hidden pairing? Please also identify the situations where the next position depends only on the filled set.

  2. Considering Appendix I.4, what does the MDLM pair-matching test establish beyond one example? Can you provide some results from other independently chosen sets of orders?

1 comment

Thank you for your careful review and helpful comments.

W1 & Q1: “Which existing generation rules have the dependence on earlier orders needed for the hidden pairing? Please also identify the situations where the next position depends only on the filled set.”

Finite-order ensembles can retain the required dependence on earlier orders. Finite-step MDLM ancestral sampling has a related dependence on the reveal history in the block setting of Proposition 11. Uniform random single-position reveal is the set-only case, where local frequencies determine the full order law.

In finite-order NADE ensembles, an order is selected before generation and followed with shared predictor parameters (Uria et al., 2014, Section 4). Our uniform libraries {1234,2143}\{1234,2143\} and {1243,2134}\{1243,2134\} give a matched example within this family. At S={1,2}S=\{1,2\}, the first continues prefix 1212 with 33 and prefix 2121 with 44, while the second reverses these choices. All local frequencies agree, but the earlier order determines the continuation. The relevant property is this conditional dependence. Uniform selection from all permutations, for example, is equivalent to choosing uniformly among the remaining positions at each step.

The MDLM ancestral sampler (Sahoo et al., 2024) provides a second source of history information through its diffusion clock. With a time-independent predictor, omitting empty updates gives a value-independent block chain. Two histories can reach the same filled set with different information about the remaining clock. With four positions and four equal time intervals, conditional on completion before cleanup, the probability of revealing {3,4}\{3,4\} next is 3/73/7 after ({1,2})(\{1,2\}) but 2/32/3 after ({1},{2})(\{1\},\{2\}). The sampler is Markovian in the set and clock together; the local prediction context retains only the set and token values. Proposition 11 covers this block formulation.

For the set-only case, let f(S,i)f(S,i) be the edge frequency and v(S)v(S) the probability of visiting SS. Every reachable transition is recovered by

p(i∣S)=f(S,i)v(S),v(S)=∑jf(S,j).p(i\mid S)=\frac{f(S,i)}{v(S)},\qquad v(S)=\sum_j f(S,j).

These transitions determine the complete order law and hence generation with a fixed predictor. This includes uniform random reveal and the idealized single-position random LLaDA rule. For parallel reveals the corresponding statement uses block edges (S,B)(S,B); confidence-based choices additionally depend on predictions. We will distinguish these cases beside Definition 1.

To measure what order memory can change in learned conditionals, we compared eight random 32-order libraries over six positions with their set-only replacements on MDLM-OWT. Every edge frequency, the predictor, and the continuation are preserved. Output TV is 4.36% to 5.69%, with positive simultaneous 95% lower bounds in all eight comparisons. These controlled order-library results show that preserving local exposures can still change generation, which is the distinction explained by Theorem 4.

Q2: “Considering Appendix I.4, what does the MDLM pair-matching test establish beyond one example? Can you provide some results from other independently chosen sets of orders?”

The MDLM pair-matching test now establishes a repeated reduction in total TV beyond one example. We added 16 independently chosen comparisons over all 720 six-position orders, and every T+JT+J-matched contrast has lower TV than its paired TT-matched contrast, with positive simultaneous 95% lower bounds. This conclusion uses direct likelihood estimates, so it remains applicable in the MDLM regime discussed in Appendix I.4.

Each comparison starts from an independent random direction, projected onto the TT and [T;J][T;J] kernels; its positive and negative parts are normalized into schedule laws. The designs were fixed before collecting outcomes. With the same checkpoint and original eight-prompt panel, 8,192 fresh FP32 samples score all 720 paths, and the simultaneous intervals account for shared samples.

Quantity across 16 comparisonsResult
Median TV with TT matching4.17%
Median TV with T+JT+J matching1.29%
Median paired TV reduction2.93 percentage points
Range of paired reductions2.59 to 3.57 percentage points

The reduction therefore persists across independent order distributions and extends beyond the original cube construction. In the cube, a three-pair product imposes a nonnegative TV ordering; the full-permutation comparisons allow sample-level differences of either sign. Their repeated reduction measures the total effect of matching, including changes in higher moments. The near-compatible experiments separately isolate second-order cancellation, addressing the mechanism at the scale where the expansion applies.

Reviewer_9NGBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies whether matching local prediction statistics guarantees identical generation distributions in masked diffusion language models. For value-independent schedules that reveal one position at a time without revision, identical edge-exposure statistics imply identical edge-additive training objectives but can yield different output distributions. The authors characterize this ambiguity, which first arises at four positions and derive a prefix–suffix factorization of path-swap responses. They further show that matching edge-pair co-occurrence statistics eliminates second-order responses near compatible conditionals and propose a four-path likelihood diagnostic requiring no retraining. Experiments on synthetic models, Transformers trained with symmetric masking, and pretrained masked diffusion models examine these mechanisms through controlled sampling.

Critical strengths
  1. The paper demonstrates differences in generation distributions while holding the predictor fixed and exactly matching local prediction frequencies, going beyond the general observation that generation order matters. The prefix–suffix decomposition in Theorem 4 explains the underlying mechanism, while Theorem 9 characterizes the statistics needed to control responses at different orders, extending the contribution beyond counterexamples.
  2. The four-path likelihood diagnostic directly estimates the target TV, closely aligning the experiments with the theoretical question. Evaluation spans exact enumeration and several pretrained models, with multiple-comparison correction and independent confirmation of selected findings. Reporting smaller responses and limitations of the theoretical bounds further supports a measured interpretation of the evidence.
Critical weaknesses
  1. For value-independent policies conditioned only on the revealed set, the edge frequencies in Eq. (1) uniquely determine the schedule distribution. Nontrivial hidden coupling in Eq. (5) therefore requires additional history dependence or latent path correlations. Although Appendix B.4 discusses the Plackett–Luce special case, the paper does not sufficiently establish which practical samplers exhibit these correlations or why this broader schedule class is relevant to train–inference alignment.
  2. Theorem 7 yields no order dependence under symmetric precedence probabilities, so its genericity result does not directly explain fixed symmetric masking. Likewise, Eq. (27) does not guarantee the nonzero contraction in Eq. (32) required for the positive response in Eq. (33). This condition is not verified for the experimental Transformer, leaving the connection between the observed effect and the proposed learning mechanism incomplete.
  3. Appendix I.4 shows that the main MDLM checkpoint lies outside the regime where Eq. (14) provides a nontrivial bound for the evaluated contrasts. The comparison in Figure 4b also changes higher-order path moments, so the observed TV reduction cannot be attributed specifically to second-order cancellation. The practical effectiveness of pairwise moment matching therefore remains insufficiently supported.
  4. Theorem 9 establishes minimality over the full perturbation class in Eqs. (11)–(13), rather than the restricted directions available to a fixed shared-parameter model. Proposition 23 controls coordinatewise estimation error, whereas Appendix E.4 requires aggregate moment-error control, without establishing a useful finite-sample connection. Consequently, the analysis does not yet show how estimating and matching these statistics delivers the distributional control described in Eq. (14).
  5. The headline 43% median TV is estimated from 500 random switches on eight fixed prompts, using 96 reward samples per switch. Eq. (16) and its multiple-comparison correction support response lower bounds, but do not directly quantify uncertainty in the median or its stability across prompts. Since the 512-prompt extension evaluates a different set of fixed comparisons, the headline effect size still lacks direct replication under the protocol.
Questions for authors
  1. The authors should clarify which practically relevant scheduling families admit distinct schedules with identical edge frequencies and what additional history dependence enables this.
  2. The authors should clarify how the symmetric Transformer experiment supports the proposed local learning mechanism, particularly whether the nonzero contraction required by Eq. (32) is verified for the model and supports the response in Eq. (33).
  3. Given the limitation of Eq. (14) established in Appendix I.4, the authors should specify the conclusion supported by Figure 4b and explain whether the effect of pairwise moment matching can be distinguished from changes in higher-order moments.
  4. The authors should explain under what model-specific assumptions Theorem 9 and Proposition 23 provide meaningful finite-sample control of generation distributions, and whether these assumptions hold for the evaluated models.
  5. The authors should provide evidence for the stability of the 43% median TV under independent repetitions and broader prompt sampling, with uncertainty estimates accounting for Monte Carlo error and variation across prompts and sampled switches.
4 comments

Thank you for your careful review and helpful comments.

W1 & Q1: “which practically relevant scheduling families admit distinct schedules with identical edge frequencies and what additional history dependence enables this”

Finite-order ensembles provide such a family for single-position generation; finite-step MDLM ancestral sampling provides a related example for the block extension in Proposition 11. The additional information is the retained order in the first case and the reveal clock in the second. Both can affect the continuation after the revealed set has been fixed.

Uria et al. (2014, Section 4) sample from a finite NADE ensemble by selecting an order and following it with shared predictor parameters. Within this family, our uniform libraries L+={1234,2143}L_+=\{1234,2143\} and L−={1243,2134}L_-=\{1243,2134\} have identical edge frequencies. At S={1,2}S=\{1,2\}, however, L+L_+ continues prefix 1212 with 33 and prefix 2121 with 44, while L−L_- reverses these choices. Neither schedule uses predictions or generated values. The earlier order determines which continuation remains available.

For MDLM, the finite-step ancestral sampler of Sahoo et al. (2024) chooses parallel reveals through a value-independent diffusion clock. With its time-independent predictor, omitting steps that reveal nothing gives a block chain of the kind covered by Proposition 11. Earlier blocks carry information about the remaining clock, so the filled set alone need not determine the next block. For example, with four positions and four equal time intervals, conditional on completion before the final cleanup, the probability of revealing {3,4}\{3,4\} next is 3/73/7 after the prefix ({1,2})(\{1,2\}) and 2/32/3 after ({1},{2})(\{1\},\{2\}). Both prefixes reach the same set. This is a property of the ancestral reveal law, derived from its update probabilities.

The set-only case has a precise sufficient condition. For a value-independent single-position policy, p(i∣S)=f(S,i)/∑jf(S,j)p(i\mid S)=f(S,i)/\sum_j f(S,j) recovers every reachable transition and hence the full order law. Uniform random reveal, including the idealized single-position random LLaDA rule, belongs to this class. Thus local matching guarantees equal generation within this class, while the examples above identify information beyond the set. We will make this distinction explicit beside Definition 1.

This matters for train–inference alignment because matching local training weights preserves the additive objective, but need not preserve generation. Shih et al. (2022) weight prediction losses by inference-time visitation frequencies. Our result identifies what such frequency alignment can guarantee. In eight controlled MDLM-OWT comparisons, we replaced a retained 32-order library with its set-only policy while preserving every edge frequency, the predictor, and the continuation. Output TV was 4.36% to 5.69%, with positive simultaneous 95% lower bounds in all eight cases. These order-library experiments show that equal local exposures can leave a measurable difference in generation.

W2 & Q2: “how the symmetric Transformer experiment supports the proposed local learning mechanism, particularly whether the nonzero contraction required by Eq. (32) is verified”

We added three training runs of the original 13.3M Transformer architecture and two symmetry controls per seed, verifying a nonzero contraction along the actual initial AdamW direction in all 18 seed-by-switch cases. We also tested the population projection in Eq. (32) in shared-parameter models where that projection can be computed directly. These experiments connect symmetric training to the required contraction, beyond observing a nonzero curl.

In the original architecture, the contraction remains nonzero with dropout disabled and with the mask expectation computed exactly on the same fixed data batch. Dropout and mask-sampling noise are therefore unnecessary for this initial response. Along the saved AdamW direction, the measured response also follows the predicted quadratic scaling. At scale 1/81/8, the relative discrepancy is at most 1.47% on paired sampled assignments. This comparison tests the local scaling along the optimizer's actual direction.

For the population mechanism, we used a regular shared logistic head on frozen Transformer features, satisfying the local projection assumptions of Proposition 18. All 270 combinations of three feature seeds, 15 joint-score directions, and six switches have positive coefficients. Near the compatible anchor, enumerated TV⁡/ε2\operatorname{TV}/\varepsilon^2 agrees with the prediction to within 0.49% relative error, checking the response in Eq. (33). A small Transformer trained in all parameters by population gradient flow gives 270 positive coefficients and agreement within 0.97%. The shared-head experiment verifies the population-minimizer formula; the original 13.3M model is checked along its AdamW update. We also included a nonzero-curl, zero-contraction control, which confirms why Eq. (27) alone is insufficient. The symmetric mechanism is therefore the projection mechanism in Section 5.2, whereas Theorem 7 describes the precedence-biased case and gives θ⋆=0\theta^\star=0 at b=0b=0.

Finally, we trained each original-architecture model for 4,000 steps and evaluated the same four fixed switches.

Training seedMean TV after trainingSimultaneous lower bound
10119.41%17.83%
20220.47%18.89%
30321.38%19.80%

All three runs learn a substantial response under symmetric masking. The initial-direction and population calculations test how a response arises locally; the training results show that the learned effect persists across seeds.

W3 & Q3: “the conclusion supported by Figure 4b” and “whether the effect of pairwise moment matching can be distinguished from changes in higher-order moments”

Figure 4b shows that pairwise moment matching reduces total TV in its MDLM comparison. To distinguish second-order cancellation from higher-order changes, we test the quadratic term near compatibility, using the scaling in Appendix I.2 and the coefficient checks above. At the pretrained MDLM checkpoint, the measured reduction includes the higher-order changes as well. Its evidence comes from direct likelihood estimates and simultaneous intervals, independently of the small-perturbation bound in Eq. (14).

We added 16 independently chosen comparisons over all 720 orders of six positions to test whether the reduction extends beyond the original construction. Each independent random direction gives a TT-matched and a T+JT+J-matched contrast, fixed before collecting outcomes. Unlike the original three-pair cube, whose product structure imposes a nonnegative TV ordering, these comparisons allow sample-level differences of either sign. Using 8,192 fresh FP32 samples on the original eight-prompt panel, we obtain the following results.

Quantity across 16 comparisonsResult
Median TV with TT matching4.17%
Median TV with T+JT+J matching1.29%
Median paired TV reduction2.93 percentage points

All 16 paired reductions have positive simultaneous 95% lower bounds. The improvement thus extends to independently chosen distributions over the full permutation space. These experiments support the practical effect of pairwise matching on total TV; the near-compatible experiments identify the second-order contribution specifically.

W4 & Q4: “restricted directions available to a fixed shared-parameter model” and “meaningful finite-sample control of generation distributions”

For a fixed shared-parameter model, the relevant statistics depend on its reachable perturbations. Finite-sample control combines aggregate moment errors with a uniform perturbation bound. We added a model-specific minimality calculation and a trained-model TV certificate.

(1) Minimality under parameter sharing. Theorem 9 establishes minimality over the full perturbation class. Appendix E.7 gives the corresponding model-specific quotient, retaining differences that are detectable along reachable parameter directions at a compatible anchor. We computed it in three shared-head Transformer feature models. Each second-order response map has certified rank six on the six-dimensional space ker⁡T\ker T, so every nonzero edge-matched contrast is detectable by some parameter direction. Unprojected compatible joint-score directions give a zero-response control. In these models, sharing parameters therefore preserves all six hidden directions.

(2) From coordinatewise to aggregate error. Fix two laws ρ,σ\rho,\sigma and draw NN independent orders from each. Let MjM_j count the known co-visitable jj-edge subsets and ε^j\widehat\varepsilon_j be their empirical L1L_1 moment mismatch, for j=1,2j=1,2. A simultaneous Hoeffding bound gives true aggregate mismatch at most Δj\Delta_j, where

\Delta_j=\widehat\varepsilon_j+2M_jr_N.$$ This supplies the connection from Proposition 23 to Appendix E.4, including the dependence on the number of moment coordinates. (3) From aggregate error to generation distributions. For a known full-support compatible anchor $P$ satisfying $|q_e/P_e-1|\le\alpha$ on all relevant contexts, with probability at least $1-\delta$, $$\mathrm{TV}(Q_{\rho}^{q},Q_{\sigma}^{q})\le\min\left(1,\frac{\alpha\Delta_1+\alpha^2\Delta_2}{2}+\mathrm{TV}(\rho,\sigma)\sum_{j=3}^{d}\binom{d}{j}\alpha^{j}\right).$$ We verified these assumptions in a trained shared-head model with $\alpha=0.01879$, $M_1=8$, and $M_2=20$. With 10,000 fresh orders per law, the 95% upper bound is **0.0046594**, compared with exact TV $5.44\times10^{-6}$. The bound includes the pair's nonzero pair-moment mismatch. Unknown support requires a bound on omitted coordinates, and selecting contrasts from data requires fresh confirmation samples or simultaneous selection control. For the evaluated checkpoints, the certificate applies to this trained shared-head model. We quantify the 13.3M Transformer and pretrained MDLM responses through direct likelihood estimates, which remain applicable in the MDLM regime described in Appendix I.4. ### **W5 & Q5: “the stability of the 43% median TV under independent repetitions and broader prompt sampling”** The 43% median TV is stable in magnitude under independent repetitions and broader prompt sampling. The three direct repetitions give medians between **43.92% and 45.08%**, and the broader prompt scan gives **43.81%**. The intervals below account for completion and switch sampling, plus prompt sampling in the population scan. | Evaluation | Median TV | Conservative 95% interval for the median | | --- | --- | --- | | Independent 500-switch repeat 1 | 44.59% | [26.31%, 63.79%] | | Independent 500-switch repeat 2 | 45.08% | [26.07%, 63.80%] | | Independent 500-switch repeat 3 | 43.92% | [24.93%, 63.81%] | | Full FP32 check of the original 500 switches | 44.58% | [25.89%, 63.10%] | | New 500-switch scan of a predefined prompt population | 43.81% | [31.92%, 56.49%] | The first three repeats independently resample switches under the original checkpoint, eight prompts, and visible span 43. The FP32 check retains the original 500 switches and draws fresh completions. These evaluations use 192 reward samples per switch; we will correct the count of 96 in the submission. The population scan samples from 8,372 prompts in eight equally weighted strata, using span 128 and 512 rewards per switch. It estimates the median across random switches of prompt-averaged conditional TV, with prompt, completion, and switch sampling all included in the interval. The repeats test the original protocol; the broader scan tests dependence on the fixed prompt panel. Both support a substantial effect of similar magnitude. We will also improve the presentation by explaining the set-only case before the general schedule family, placing the symmetric-learning argument beside its experiment, and distinguishing a token-level loss from a complete training forward pass.
Reviewer_9NGBOpenReview ↗

Thank you for the rebuttal

Thank you for the thoughtful rebuttal and the additional experiments. The checks under symmetric masking and the broader prompt evaluation addressed several of my original concerns. I appreciated how clearly you explained what each experiment was designed to test.

My main concern is that the number of theoretical results sometimes makes it difficult to follow the central argument and see which questions the experiments resolve. In Section 5, for example, Theorems 6 and 7 and Corollary 8 appear in quick succession. Figure 3d tests Theorem 7’s prediction about how reveal-order bias interacts with residual prediction error. Figure 4a shows that sensitivity to reveal order also emerges under symmetric masking, where that bias is absent. The rebuttal helped me connect this finding to the separate argument in Section 5.2 by showing how sensitivity can arise at the start of training. In the paper itself, that connection was easy to lose. Besides, another question remains for me as how far does that local explanation carry through training? For example, does the same mechanism help explain why some changes in reveal order produce larger responses than others after 4,000 steps? Making that connection more explicit in the paper would help readers appreciate both what the theory explains and what remains open.

I would encourage the authors to give the most space to the results that help readers answer the motivating question, while letting the more technical characterizations play a supporting role. Keeping the key experiments close to the results they illuminate would make the paper easier to follow. It would also help readers appreciate where a theorem’s assumptions hold, what the theorem explains in that setting, and which empirical questions remain open. I hope the authors can build on the clarity of the rebuttal to bring the paper’s motivation and central argument into sharper focus. I believe this would help readers see how the strongest theorems and experiments fit together and would make a stronger case for acceptance. I appreciate the authors’ thoughtful response, but given my remaining concerns about the connection between theory and evidence, I will maintain my original rating of Weak Reject.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.