acceptodds
Withdrawn from ICLR 2027

The Truncation Trap: Silent Failure of Fixed-Window Diffusion Likelihood and Information Estimators under Representation Drift

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion-based estimators of likelihood and mutual information evaluate an integral of denoising error over signal-to-noise ratio; in practice the integral runs over a fixed, truncated log-SNR window whose placement and data-scale assumptions are frozen at configuration time. We document a latent failure mode of this family: when the representation's scale drifts, most commonly because a jointly learned embedding grows under cross-entropy training, integrand mass silently migrates out of the window and the estimator retains an arbitrarily small fraction of the full-line integral, with no warning. The mechanism is predictive: the integrand peak sits at for representation scale . We demonstrate it end to end on a 306M-parameter diffusion language model whose truncated readout reports 0.78 BPC against a converged dense-grid value of 4.27. Auditing the released code of four published systems, we find that each carries an undocumented guard: per-forward embedding normalization (Plaid), default input standardization (MINDE, MMG), and data-adaptive windows (ITD). Disabling MINDE's guard shifts a scale-invariant MI estimate by 4.2 nats; ablating Plaid's guard in a from-scratch retrain leaves the model intact but sends the fixed-window readout to 278 BPC, whereas a guarded positive control reproduces Plaid's reported 1.12 BPC in three independent ways. We distill a defense kit, consisting of an entropy-floor assertion, convergence-verified windows with measured tail bounds, and a rescale stress test, that detects or precludes this failure mode before a number is reported.

Primary area:
generative models
Keywords:
diffusion models, likelihood estimation, mutual information estimation, information-theoretic diffusion, log-SNR truncation, diffusion language models, evaluation methodology, reproducibility, representation drift

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_ZrepOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies errors in diffusion-based likelihood and mutual information estimates caused by fixed integration windows. The authors show that when learned representations change scale, integrand mass can fall outside the window, leading to underestimation, and examine high-SNR overestimation issues. They audit existing models and propose practical evaluation checks.

Critical strengths
  • Addresses an important evaluation problem where truncation artifacts can mimic real model improvements.
  • The analysis goes beyond a single example, including norm/window ablations, Gaussian tasks, and a Plaid positive control.
  • The practical recommendations (such as recomputing scale-dependent bounds) are useful for the community.
Critical weaknesses
  • The paper treats its wider dense-grid integral as a reliable reference, but lacks strict tail guarantees for high-SNR regimes and learned denoisers.

  • The Plaid experiment modifies multiple components simultaneously (removing normalization, fixed windows, and unnormalized models), failing to isolate whether normalization removal alone breaks the evaluation. Numerical precision and out-of-distribution model errors are also not cleanly separated.

  • Some claims outrun the evidence. For example, Section 5.1 claims analytic predictions match MINDE within 1 nat, but Table 4 shows a discrepancy of 2.22 nats (5.80 vs 3.58).

  • The proposed input-rescaling check needs more rigorous definitions, as standard learned model likelihoods do not obey simple variable changes without consistent transformations.

Questions for authors
  • Can you provide rigorous tail bounds and grid-refinement results for the headline dense-grid evaluation?

  • In the Plaid experiment, can you separate the effects of numerical precision, evaluation outside the training window, and embedding normalization?

  • How do you account for the 2.22-nat difference in Table 4, and what exact transformations are required for the rescaling check?

Reviewer_Q8jMOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper identifies a potential source of bugs in the calculation of diffusion-based information estimators on learned representations, such as the evidence lower bound for continuous diffusion language models. It identifies mechanisms in the literature that control the scale of learned representations (or adjust the scale of the estimator) that implicitly prevent miscalculations. It also proposes some checks for catching the issue if no mechanism is in place to control representation scale.

Critical strengths

There is an observation worth documenting here, about the conditions under which a truncated information estimator is valid. It seems like an account of representation scale is not well-documented in the four prior works discussed in the audit. E.g., I confirmed that the Plaid paper doesn't mention embedding normalization, which is indeed present in the code and I agree it is important to obtaining a valid numerical estimate of the likelihood bound.

The "defense kit" is a possibly-useful collection of sanity checks.

Critical weaknesses

I find this paper extremely frustrating to read. This is a big problem, because the primary contribution of this paper is supposed to be the communication of the conditions under which information estimators are valid. The paper invents all sorts of non-standard terminology without much fanfare and expects the reader to do a lot of detective work to figure out how it is using its own invented language. I comment on this briefly regarding the abstract below, but the problem is endemic throughout the paper.

The abstract is difficult to parse. What is the "anchor" in the approximation claim? In the following sentence "We demonstrate it end to end" what is "it"? BPC is never defined (even in the main text); I infer it means bits per character for tokenization-agnostic comparison? What is a "readout"? You are referring to the ELBO? What is the difference between a "truncated readout" and a "fixed-window readout"? I don't understand what nan entropy-floor assertion is supposed to be and have to guess at what is meant by a "convergence-verified window" or "rescale stress test." And stylistically, I could also do without vacuous flourishes like "before a number is reported."

The rest of the paper is written in a similarly difficult style.

Questions for authors

Without the proposed defense kit, wouldn't an error in the calculation of an information estimator be discovered pretty quickly anyway? Indeed, in your own motivating example it seems like you caught the bug based on natural sanity checks and none of the published systems you discuss were caught by the issue. Presumably this would also be caught be other metrics or inconsistently bad downstream task performance?

What is the "chain-rule omission in a diffusion-LM likelihood pipeline" wrt Yang et al., 2026? I'm confused about what chain-rule are we talking about? Regardless of that, doesn't Yang explicitly normalize embeddings, i.e., explicitly documenting the operation in the Plaid codebase that controls representation scale. So while some papers aren't clear on this (including the 4 studied in the audit) it seems like others such as RePlaid are more clear?

Reviewer_r9NUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper discusses a potential numerical failure in diffusion-based estimators of likelihood and mutual information that could take place if one fixes an integration window while the representation's scale shifts under training dynamics. The authors also state that their findings do not invalidate any existing published results.

Critical strengths

None.

Critical weaknesses

I find this paper's writing to be of extremely poor quality, making it very hard to parse and follow. The main mathematical effect that shifting an integrand outside of a fixed integration window will affect the measured integral is trivial, and the paper does not demonstrate that the failure has affected any existing published result. Further, Kong et al., 2023, Section 4 already discuss in depth a similar issue about numerical integration and truncation, and explain how they define the window from data statistics (https://arxiv.org/pdf/2302.03792).

Questions for authors

No specific questions at this stage that could affect my assessment. I do not think the paper meets ICLR standards, nor do I see a clear path to remedy that without very substantial changes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.