acceptodds
Under review as a conference paper at ICLR 2027

Pairwise-Safe, Jointly-Unsafe: Higher-Order Residual Risk in Parallel Decoding of Diffusion Language Models

Abstract

Diffusion language models (dLLMs) obtain much of their inference advantage by committing multiple masked positions in parallel. Existing fast decoders therefore rely on token confidence or pairwise dependency estimates to decide which positions can be committed together. We identify a blind spot in this design: a token bundle can appear pairwise safe while remaining jointly unsafe. We formalize the portion of joint commitment risk that is unexplained by pairwise dependency, confidence, bundle size, and diffusion stage as Higher-Order Residual Risk (HOR). We first perform a Phase-0 diagnosis on Dream-7B and LLaDA. Pairwise-safe/jointly-unsafe bundles constitute only 13.9–18.2% of observed commit bundles, yet account for 49.2–58.6% of decoding errors, indicating a high-leverage failure mode. We then introduce a lightweight HOR predictor that combines pooled hidden-state representations with pairwise and meta features, and use its prediction as a pre-commit signal to commit, split, or defer candidate bundles. The full predictor reaches 0.896 AUROC and 0.792 R² for joint-risk prediction. Across GSM8K, MATH500, HumanEval+, and MBPP+, HOR-aware decoding improves accuracy/pass@1 over DEMASK by 4.1 points on average while reducing NFE by 8.0% and latency by 10.3%. HOR selection is also complementary to CoCommit-style within-set coordination, yielding further gains. These results suggest that pairwise safety is insufficient for reliable parallel dLLM decoding and that residual set-level risk can be both measurable and actionable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language models, parallel decoding, higher-order dependency, residual risk, efficient inference, dependency-aware decoding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bCTiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses a failure mode in parallel decoding of diffusion LLMs (dLLMs): a bundle of tokens committed together can be jointly unsafe even when every pairwise dependency test judges it safe (motivated by a parity construction where variables are pairwise-independent but jointly constrained). The authors define Higher-Order Residual Risk (HOR) as the residual of an offline joint-risk proxy R_J(S) after regressing out pairwise risk, confidence, bundle size, and diffusion stage (Eq. 1, 7). R_J(S) itself is a tractable "sequential-reveal vs. parallel" log-score discrepancy averaged over up to 8 permutations (Eq. 5). Experiments are done on Dream-7B and LLaDA across GSM8K/MATH500/HumanEval+/MBPP.

Critical strengths

The paper is unusually explicit that HOR is an operational residual, that the diagnostic prevalence is sampling-conditional, and that latency is hardware-specific.

Critical weaknesses
  1. The empirical evidence is not credible as presented. The combination of (a) highly regular, round-number tables across many independent cells, (b) an AI-use statement describing reconstruction of the manuscript from a PDF with figures "prepared from tabulated results," and (c) the absence of variance/CIs, raw logs, or released code for the main decoding experiments makes it impossible to verify that the reported gains (4.1 points, 8.0% NFE, 10.3% latency) come from executed experiments.

  2. The core construct (HOR) is not shown to measure what its name claims, and its marginal value is small. By the authors' own framing HOR is a catch-all residual; the hidden-state predictor could be exploiting generic error correlates rather than higher-order dependence, and the residual formulation beats direct R_J prediction by only ~0.03 AUROC. Thus even taking the numbers at face value, the paper does not establish that modeling higher-order structure is responsible for the gains.

Questions for authors
  1. Will you release the code, raw per-run logs, and per-seed numbers (with variance/CIs) for Tables 1, 3, 7, and 9? Given the AI-reconstruction note, can you confirm every reported number is the output of an executed run, and provide the run artifacts?
  2. Several tables show strikingly regular values (e.g., Table 9 scaling, Table 6/10 HOR means, Table 7 NFE). Can you provide the underlying distributions / histograms and seed-level dispersion that produced these aggregates?
  3. Can you show that HOR captures higher-order dependence specifically, e.g., on a semi-synthetic benchmark with controllable k-wise-but-not-pairwise structure, rather than generic bundle-error signal? What does the hidden-state predictor's advantage reduce to when the true joint structure is purely pairwise?
  4. Table 5 shows residual vs. direct R_J prediction differs by only ~0.03 AUROC. Why is the residual formulation worth the added two-stage complexity?
Reviewer_RERHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In dLLM, some tokens are not safe to decode together at once as they are jointly dependent. This paper raises and investigates further: a pairwise dependency check cannot ensure that committing multiple tokens in parallel is safe. One can have a concrete counterexample that every pair of variables can be independent while the whole set is still jointly dependent. The paper then proposes HOR to capture such sets: it measures how much the tokens' probabilities change when they are predicted one by one instead of all at once, and removes the part that inputing {pairwise score, mean and minimum confidence, set size, and decoding step} could predict with a small network. During decoding, another small network predicts HOR from the model's hidden states and splits or delays a token set when HOR is high. On Dream-7B, this improves accuracy over the pairwise baseline DEMASK.

Critical strengths

S1: The paper reports consistent improvements over the DEMASK baseline. The main table shows that on Dream-7B, HOR improves accuracy by 3.7–4.5 points on all four benchmarks while reducing forward evaluations by 8.0% and latency by 10.3%.

Critical weaknesses

W1: The definition of HOR is ad hoc. Although it may convince readers that the difference between parallel and one-by-one decoding can reveal token dependency, the plausibility of predicting this dependency using a small network is not justified, and in fact this approach feels brute-force. Meanwhile, the factors (pairwise score, confidence, set size, and decoding step) that the authors deliberately remove from this difference are also not justified, as this component is also predicted by a network. From the paper, this network fits poorly: for Dream on GSM8K, it predicts only 0.08 to 0.30 while the actual difference grows from 0.16 to 0.92 with set size (Table 9).

W2: The blind-spot diagnosis (Table 1) is largely circular. A bundle enters the blind spot only if its offline joint risk is high, and it is marked erroneous if its parallel tokens disagree with the tokens obtained by sequential reveal; both quantities come from the same sequential-reveal passes. The much higher error rate inside the blind spot (64.5–74.2% vs. 11.8–18.5%) therefore follows mostly from the definitions, rather than showing errors missed by the pairwise check.

Questions for authors

Q1: HOR only rejects, splits, or defers token sets proposed by DEMASK. Under the same setting, it should therefore need more forward evaluations. However, why does HOR commit more tokens per step than DEMASK on average (3.38 vs. 3.10, Table 7)? Why does it use fewer forward evaluations?

Reviewer_CEPKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets on parallel decoding in diffusion language models (DLMs) where many tokens appear safe under pairwise dependency estimates while remaining jointly inconsistent. Thus, the authors propose higher-order residual risk (HOR) that subtracts a prediction based on pairwise risk, confidence, bundle size, and diffusion stage from an offline sequential-reveal risk proxy. A lightweight predictor estimates this residual from pooled hidden states and scheduler features. Its predictions guide a gate that accepts, splits, or defers candidate bundles.

Critical strengths
  • The motivation is clear. The pairwise compatibility and joint consistency are different, leading to a meaningful limitation of dependency-aware decoding.

  • The predictor is based on existing hidden state and can be applied to different backbones.

  • The experiemental results show the effectiveness of the proposed method.

Critical weaknesses
  • The joint-risk proxy and diagnostic error labels depend on sequential-reveal predictions. However, their association does not independently establish, leading to flagged bundles that cause incorrect final answers.

  • HOR has many properties, including capturing baseline underfitting and distribution shift. However, higher-order dependence is not isolated.

  • For Algorithm 1, it commits the remaining subset (L) without re-evaluating its joint risk. Low individual token scores do not establish that this subset is jointly safe.

  • The experimental settings only involves 7-8B DLMs. It may be necessary to test the method on a larger-scale DLMs.

Questions for authors

Please see the weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.