acceptodds
Under review as a conference paper at ICLR 2027

ACE: Attention Coalition Envelopes for Parallel Decoding in Diffusion Language Models

Abstract

Masked diffusion large language models (dLLMs) enable parallel generation by predicting multiple unresolved positions in a single denoiser pass, but individually confident predictions need not be jointly compatible. We study this tension through a cooperative-game view of parallel commitment. Under a coherent reference distribution, the KL risk of a parallel commit decomposes into individual prediction errors and a supermodular dependence cost. This structure leads to an ordered modular envelope that assigns each candidate a fixed charge before selection, reducing coalition formation to sorting and a cumulative sum. We instantiate this principle as the Attention Coalition Envelope (ACE), which uses native attention to form compatibility-aware token coalitions without training or additional denoiser evaluations; its formal guarantee applies to the attention-induced surrogate game. Across six benchmarks on LLaDA and Dream, ACE reaches the empirical quality–efficiency Pareto frontier in all 12 benchmark–backbone settings, with 30/36 evaluated operating points being Pareto-optimal. It also attains the highest measured accuracy in 10/12 settings, including 5 statistically supported leads. On two matched LLaDA reasoning controls, ACE reduces denoiser evaluations by 32–36% at comparable accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion language models, cooperative game

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_PJxXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(−1)
Read review
Paper summary

This paper studies training-free decoding in masked diffusion language models, where individually confident token predictions may be incompatible when committed together. At each decoding step, ACE tests candidate rankings based on confidence, attention, and left-to-right text position. For each ranking, it uses native attention to score a candidate's interactions with candidates ranked ahead of it, even if they are not selected. Depending on the configuration, the selection score also includes a readiness penalty and an order prior. These scores stay fixed during selection, and ACE sorts them and selects as the number of tokens that fit the budget. Experiments on LLaDA and Dream show improved accuracy-efficiency tradeoffs.

Critical strengths

The fixed-cost selector is a useful alternative to sequential compatibility checks. Its scores include interactions with predecessors that may never be selected, so it could exclude too many candidates and require more denoiser calls. I find the matched Dream comparison convincing on this point. At the default and quality settings, ACE retains accuracy and denoiser-call counts close to greedy selection despite using these conservative scores (Figure 3 and Table 17).

Critical weaknesses

I found it difficult to follow how the theory in Sections 4–5 leads to ACE. Section 5 introduces attention-based scores and states a guarantee for the attention-induced game, but the set cost covered by that guarantee is defined only in Appendix A.8, Equation (35). The main text leaves the reader to infer what the practical scores bound or consult the appendix. That definition and its connection to the scores belong in the main method description, where the reader is trying to understand what the theory establishes for the decoder.

Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer PJxX

We thank the reviewer for the positive assessment and for finding the matched Dream comparison convincing. Table and figure numbers below refer to the revised PDF.


Weakness: how the theory in Sections 4–5 leads to ACE

Section 5 now defines the set cost covered by the guarantee and states the bound itself, so the main method description says exactly what the practical scores bound.

The definition now introduces the ACE scores (Equation 7), as their ordered marginal costs:

For an interaction threshold τ≥0\tau\geq0, the surrogate scores are the ordered marginal costs of the attention game czatt(S;τ):=∑{i,j}⊆Swz(i,j) 1[wz(i,j)>τ]c_z^{\mathrm{att}}(S;\tau):=\sum_{\{i,j\}\subseteq S}w_z(i,j)\,\mathbf 1[w_z(i,j)>\tau], as uku_k are for czc_z: skACE(τ):=∑ℓ<kwz(ek,eℓ) 1[wz(ek,eℓ)>τ],s1ACE(τ)=0.s_k^{\mathrm{ACE}}(\tau):=\sum_{\ell<k}w_z(e_k,e_\ell)\,\mathbf 1[w_z(e_k,e_\ell)>\tau],\qquad s_1^{\mathrm{ACE}}(\tau)=0.

The section's closing paragraph now states the bound, followed, as before, by the condition under which it also controls the true dependence cost:

The guarantee covers the thresholded attention game, ∑ek∈SskACE(τ)≥czatt(S;τ)\sum_{e_k\in S}s_k^{\mathrm{ACE}}(\tau)\geq c_z^{\mathrm{att}}(S;\tau) for every S⊆EzS\subseteq E_z (Theorem 2); it also controls true dependence if practical scores upper-bound ideal scores on the selected set (Appendices A.7 and A.8).

The reader therefore sees, where the scores are defined, that the one-shot prefix certifies the attention game's cost for every coalition it can select. Appendix A.8 keeps the formal statement and proof, and the additions are marked in blue in the revised PDF. Thank you for this suggestion.


We are happy to answer any further questions.

After checking the response and checking the additional experiments added in responses to other reviewers, my outstanding questions have been addressed. That said, I am maintaining my score.

Dear Reviewer PJxX,

 

Thank you very much for your recognition and positive assessment of our manuscript. Your positive score is the greatest support and encouragement for our work. We sincerely appreciate the time and effort you dedicated to reviewing our paper.

 

Best regards,

The Authors

Reviewer_f68yOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposed Attention Coalition Envelopes (ACE), a training-free parallel decoding algorithm for DLMs. The method is motivated by the idea that a good set of tokens to commit to should incur a low dependency-induced KL penalty (i.e., should be weakly correlated with the other tokens. It uses attention weights as a proxy for dependency costs and introduces an ordered modular envelope that uses these weights to efficiently select the best set of tokens to commit. Experiments on open-source DLMs demonstrate promising accuracy-efficiency tradeoffs across several benchmarks

Critical strengths
  1. The motivation is interesting and intuitive. Using dependency-induced KL risk to guide parallel token commitment provides a more principled perspective than the common confidence-based decoding. The idea of approximating token dependencies with native attention weights is also intuitive and leads to a simple, training-free approach that requires no additional model evaluations.

  2. The empirical results are promising. ACE achieves substantial latency reductions compared with several existing decoding baselines across several benchmarks and two backbones.

Critical weaknesses
  1. The method relies on several heuristics, making it difficult to isolate the contribution of its central idea.

While the theoretical motivation is appealing, the practical implementation introduces several additional heuristics, including the readiness penalty, left-to-right order prior... These components make it difficult to determine how much of the reported improvement actually comes from the proposed attention-based dependency modeling.

The authors do show that on constructed examples, attention weights distinguish dependent token pairs more reliably than metrics. However, this does not necessarily establish that attention-based dependency contributes the most to the downstream decoding improvements. So, it would be better to show whether directly using attention weights to guide commitment, without the additional heuristics, could already achieve most of the gains.

  1. The sources of the reported acceleration and the fairness of comparisons need further clarification.

First, according to the appendix, the implementation of ACE uses early EOS termination (and unclear if its used for the other baseline). If ACE terminates generation earlier than competing methods, some of its wall-clock acceleration could come from generating shorter sequences rather than more efficient parallel token commitment. Thus, it would be better to report additional metrics, such as the average generated sequence length and NFE, and place the comparison under matched termination policies.

Second, I am concerned about the use of prefix KV caching in the efficiency comparison. If ACE benefits from prefix caching while some baselines are evaluated without it, the wall-clock comparison may conflate improvements from the decoding policy with improvements from caching. According to Fast-DLLM[1], applying caching alone can substantially reduce latency with very minor quality degradation. Thus, I think the authors should clarify whether the competing baselines are also compatible with this optimization and, if so, provide comparisons with caching enabled for all applicable methods.

[1] Wu et al. "Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding." ICLR 2026

Questions for authors

See weaknesses.

9 comments
AuthorsOpenReview ↗

Response to Reviewer f68y (1/3): Weakness 1

We thank the reviewer for the thoughtful review and for finding the motivation intuitive and the empirical results promising. Table and figure numbers below refer to the revised PDF.


Weakness 1: isolating the contribution of the central idea

We ran the variant the reviewer asks for, and it shows the two terms work as one mechanism: attention on a plain confidence gate changes little, while inside the shared budget it adds accuracy on every task and the budget cuts evaluations by 27–45%.

The central idea is a framework with two parts. The first is a coalition-level cost that prices a set of tokens jointly, R(S) = Σ e(i) + c(S) (Proposition 1). The second is ordered modular selection, which turns that set cost into fixed per-token scores so that the coalition is chosen in one shot (Theorem 1). The components the reviewer lists instantiate this framework: the readiness penalty is the readiness term Σ e(i), the attention score is the dependence term c(S), and the budget constrains their sum. The left-to-right order prior is the one optional term: only the LLaDA quality tier uses it, and no cell in the table below does.

The variant uses attention weights to guide commitment on top of a standard confidence gate (0.9), with no readiness penalty and no order prior. With our existing cells it completes a 2×2 on LLaDA, where block structure, prompts and termination are fixed (Table 3; accuracy % / NFE):

TaskConfidence gateConfidence gate + attentionShared budget, no attentionShared budget + attention (ACE)
HumanEval34.8 / 38.035.4 / 38.934.8 / 23.738.4 / 25.9
MBPP19.8 / 34.919.8 / 38.925.8 / 19.229.0 / 21.8
GSM8K76.6 / 74.076.7 / 76.576.6 / 41.578.3 / 49.6
MATH-50027.8 / 96.327.6 / 98.028.2 / 57.228.4 / 65.4
IFEval57.7 / 126.157.4 / 126.556.5 / 91.958.6 / 96.9

The last column is the deployed selector, including its four-order arbitration, which is defined only when attention is present.

On a plain confidence gate, attention leaves accuracy within 0.6 points and adds evaluations: at a 0.9 gate nearly every admitted token is already safe, so compatibility has little to decide. The shared budget is what cuts evaluations, because it admits less-ready tokens as long as the coalition's total risk stays within budget. Compatibility matters exactly there: inside the budget, attention adds 0.2–3.6 points on every task.

Both terms of the coalition cost therefore contribute: readiness drives efficiency, and attention-based dependence adds accuracy where less-ready tokens are admitted.

AuthorsOpenReview ↗

Response to Reviewer f68y (2/3): Weakness 2, first point

Weakness 2, first point: early EOS termination

With termination policies made identical, ACE still uses 23–61% fewer evaluations than the threshold, EB-Sampler and KLASS rules, with higher accuracy on every task.

The baselines run their published runners, which do not use ACE's termination policies, so we ran each baseline's selection rule under ACE's exact policies on five LLaDA tasks (Table 4; accuracy % / NFE):

TaskThreshold ruleEB-Sampler ruleKLASS ruleACE (default)
HumanEval34.8 / 38.036.0 / 47.336.0 / 61.838.4 / 25.9
MBPP19.8 / 34.919.4 / 43.618.8 / 56.529.0 / 21.8
GSM8K76.6 / 74.077.2 / 88.677.0 / 126.978.3 / 49.6
MATH-50027.8 / 96.327.2 / 113.026.4 / 146.328.4 / 65.4
IFEval57.7 / 126.156.8 / 133.257.6 / 153.458.6 / 96.9

Termination itself matters on the two tasks with stop strings: it cuts threshold decoding's evaluations from 92.4 to 38.0 on HumanEval and from 42.0 to 34.9 on MBPP. On GSM8K, MATH-500 and IFEval it changes them by less than 1.5%, so ACE's savings there (23–33%) owe nothing to termination. For DAPD, applying ACE's stop-sequence policy saves at most 3% of its calls (Table 13), because its scattered commits rarely complete the prefix that contains the stop string; under the identical policy, ACE's throughput tier still beats DAPD-Direct on both axes on LLaDA HumanEval (36.0 / 22.6 against 31.7 / 22.8).

Termination also does not shorten what is scored. Stop-sequence termination halts decoding at the evaluation harness's own stop string, so the scored output is bit-identical by construction. Early EOS stopping only skips positions after a committed end-of-sequence token, which the harness discards.

As requested, we report the average scored response length alongside NFE (tokens / NFE; ACE at its default tier; the full table for all decoders is Table 16 in Appendix E.3):

BackboneDecoderHumanEvalMBPPGSM8KMATH-500IFEvalBBH
LLaDAACE80 / 25.948 / 21.8212 / 49.6206 / 65.4150 / 96.9173 / 29.5
LLaDAFast-dLLM80 / 92.442 / 42.0211 / 74.5206 / 96.9148 / 127.6173 / 41.4
LLaDADAPD-Direct52 / 22.939 / 22.4136 / 35.4131 / 46.272 / 43.7167 / 16.1
DreamACE177 / 103.447 / 20.7163 / 61.7218 / 96.0191 / 20.0173 / 36.1
DreamFast-dLLM55 / 111.517 / 66.881 / 98.8101 / 158.28 / 53.3153 / 66.5
DreamDAPD-Direct136 / 113.441 / 26.6126 / 55.4180 / 63.358 / 18.6158 / 49.8

On LLaDA, ACE's answers are as long as those of the unary baselines and longer than DAPD's on every task. On Dream, ACE writes the longest answers of any decoder on all six tasks.

AuthorsOpenReview ↗

Response to Reviewer f68y (3/3): Weakness 2, second point

Weakness 2, second point: prefix KV caching

ACE's speed advantage does not come from caching: with caching on for every applicable baseline, ACE is faster than each of them on all six LLaDA tasks, and with caching off it stays on the accuracy–latency frontier in all twelve settings.

In Table 1, Fast-dLLM runs with its own prefix KV cache on both backbones. EB-Sampler, KLASS and DAPD release no cached implementation, so we run them as published.

The cleanest way to separate the decoding policy from caching is to remove caching. With its cache off, ACE lies on the accuracy–NFE Pareto frontier in all twelve settings (Table 12). We also timed every decoder with caching off, all run sequentially on the same GPU. The one exception is Dream Fast-dLLM, whose released threshold decoder is implemented with its prefix cache; it keeps that cache, which favors the baseline. ACE again stays on the accuracy–latency frontier in all twelve settings and is the fastest decoder in six.

We also enabled caching for every applicable baseline, as the reviewer asks. EB-Sampler and KLASS are compatible with a prefix cache, so we ran their selection rules inside our decoding loop with exactly ACE's prefix cache. Uncached, these rules reproduce the released baselines' NFE exactly on all five tasks checked, so the cached runs are the baselines plus a cache. On LLaDA, with every method timed in one job (accuracy % / seconds per sample):

TaskEB-Sampler, cachedKLASS, cachedFast-dLLM, shipped cacheACE (default), cached
HumanEval36.0 / 1.9140.9 / 2.7436.6 / 1.7737.8 / 0.54
MBPP19.6 / 1.2918.4 / 1.7420.2 / 0.9228.2 / 0.56
GSM8K76.5 / 2.4076.1 / 3.5879.4 / 1.7276.7 / 1.45
MATH-50028.2 / 2.5826.0 / 3.3927.0 / 2.0126.8 / 1.54
IFEval54.7 / 2.3155.2 / 2.7753.8 / 2.3958.6 / 1.71
BBH (675-problem subset)58.1 / 1.7257.2 / 2.0653.6 / 1.0551.9 / 0.86

Caching speeds the two rules up by 1.2–2.0 times, and ACE remains faster than every cached baseline on all six tasks: the fastest of them takes 1.2–3.3 times as long per sample. In accuracy it leads on MBPP and IFEval, is within the baselines' range on HumanEval and MATH-500, and trails cached Fast-dLLM on GSM8K and every cached baseline on BBH, as in Table 1. DAPD is covered by the uncached comparison above. The revision adds the uncached timing table to Appendix E.3 (Table 15) and this cached comparison to Section 6 (Table 5).


We hope these controlled comparisons address the reviewer's concerns. The revision adds them to Section 6 (Tables 3–5): readiness and attention under one budget, gains under identical termination, and caching for every applicable baseline. We would be grateful if the reviewer would reconsider the assessment in light of them, and we are happy to answer any further questions.

Reviewer_6WZLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ACE for parallel decoding in diffusion language models, where individually confident predictions need not be jointly compatible. ACE uses native attention to form compatibility-aware token coalitions, reducing selection to sorting and a cumulative sum without training or additional denoiser evaluations. Across six benchmarks on LLaDA and Dream, ACE achieves favorable quality–efficiency trade-offs.

Critical strengths
  • The ordered modular envelope reduces coalition selection to sorting and a cumulative sum, without training or additional denoiser evaluations. This provides a simple, practical way to account for token dependencies during parallel decoding.
  • The experiments cover two backbones and six benchmarks, with useful controls in the appendix. Table 13 is particularly convincing: with decoding region and ordering matched, ACE reduces denoiser evaluations by 32–36% at comparable reasoning accuracy.
Critical weaknesses
  • The main results rely on configurations selected post hoc, with additional task-specific tuning on Dream. Since the baseline tuning effort is unclear, the comparison does not establish how much of ACE’s advantage would remain under equal tuning budgets.
  • The contribution of the core method is not clearly isolated. Removing attention interactions makes little difference on some tasks, while the comparison with greedy selection also changes candidate ordering. The experiments therefore support the overall recipe more strongly than the individual benefits of attention scoring and envelope selection.
Questions for authors
  • How were the baseline hyperparameters chosen? Were the baselines given a tuning budget comparable to ACE?
  • In Table 14, removing attention interactions barely changes the results on some tasks. Does the same hold on LLaDA when the other selector components are kept unchanged?
  • With the same attention inputs, candidate orders, and caching settings, how does ACE compare with greedy selection in selection time and total decoding time?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 6WZL (1/3): Weakness 1

We thank the reviewer for the positive assessment, in particular of the ordered modular envelope and the matched controls. Table and figure numbers below refer to the revised PDF.


Weakness 1: post-hoc configurations and baseline tuning effort

ACE's advantage does not come from extra tuning: it holds under one fixed configuration on each backbone, and tuning Fast-dLLM's own threshold never reaches ACE.

Fixed configurations. On LLaDA, each of the three tiers is one recipe used unchanged on all six tasks, and each tier on its own lies on the accuracy–NFE Pareto frontier in five of six tasks (Table 12). On 100 held-out GSM8K problems, the validation results reproduce every deployed recipe choice (Table 28). On Dream, we evaluated one fixed configuration (γ = 0.8, τ = 0.1) on all six tasks (accuracy % / NFE):

TaskACE, fixed configurationAgainst the five baselines
HumanEval58.5 / 96.3better on both axes than all five
MBPP57.2 / 25.2better on both axes than all five
GSM8K57.9 / 68.1most accurate, on the frontier
MATH-50031.0 / 96.0below DAPD-Direct (31.6 / 63.3)
IFEval50.0 / 131.2most accurate by 12.2 points, on the frontier
BBH56.4 / 33.8better on both axes than all five

This single configuration stays on the frontier in five of six tasks, beats every baseline on both axes in three, and is the most accurate decoder in five. The revision adds these results to Appendix E.6 (Table 27).

Tuning budget. Every baseline runs at its published settings, which its authors tuned. ACE's LLaDA side is three recipes from a single shared grid, used unchanged on every task. To test equal tuning directly, we swept Fast-dLLM's own confidence threshold (Table 29; accuracy % / NFE):

BackboneTaskFast-dLLM 0.9Fast-dLLM 0.8Fast-dLLM 0.7ACE (throughput)ACE (default)
LLaDAHumanEval35.4 / 92.436.6 / 65.337.2 / 50.936.0 / 22.638.4 / 25.9
LLaDAGSM8K77.0 / 74.577.1 / 56.275.3 / 43.876.9 / 42.778.3 / 49.6
DreamHumanEval43.3 / 111.541.5 / 81.437.2 / 56.650.0 / 90.459.8 / 103.4
DreamGSM8K43.1 / 98.841.6 / 82.239.7 / 65.654.2 / 48.955.8 / 61.7

No threshold reaches ACE on either task. On LLaDA, an ACE tier beats every Fast-dLLM setting on both axes, on both HumanEval and GSM8K. On Dream, every setting is at least 12.7 points less accurate than ACE's default tier, which on GSM8K also uses fewer evaluations than all of them. Tuning Fast-dLLM along its own knob moves it along a curve that stays below ACE's.

Flexibility. ACE's coalition budget on LLaDA, and its readiness gate on Dream, form a single dial that trades accuracy for efficiency along a Pareto frontier. The three tiers in Table 12 are points on that dial, so ACE offers both strong operating points and an explicit trade-off, and its frontier lies above the settings Fast-dLLM's own knob reaches.

AuthorsOpenReview ↗

Response to Reviewer 6WZL (2/3): Weakness 2 and Questions 1–2

Weakness 2: individual benefits of attention scoring and envelope selection

Changing one component at a time on LLaDA isolates both: attention scoring raises accuracy on every task, and envelope selection stays within 1.3 evaluations per sample of sequential selection with the candidate order held fixed.

Both ablations hold the block structure, prompts and termination policies fixed.

Attention scoring. Starting from confidence decoding, we add ACE's components one at a time (Table 18; accuracy % / NFE):

TaskConfidence+ coalition budget+ attention+ multi-order (ACE)
HumanEval34.8 / 38.034.8 / 23.737.8 / 27.638.4 / 25.9
MBPP19.8 / 34.925.8 / 19.227.8 / 22.629.0 / 21.8
GSM8K76.6 / 74.076.6 / 41.577.4 / 50.478.3 / 49.6
MATH-50027.8 / 96.328.2 / 57.230.2 / 65.728.4 / 65.4
IFEval57.7 / 126.156.5 / 91.957.2 / 98.058.6 / 96.9

The coalition budget alone cuts denoiser evaluations by 27–45%. Adding attention then raises accuracy on every task by 0.7–3.0 points, and multi-order arbitration recovers a few percent of the evaluations. Against the attention-free coalition, full ACE gains 3.2 points on MBPP. A small effect on some tasks is therefore consistent with attention's role as the compatibility term of the coalition cost: it refines the coalition, while the budget carries the efficiency.

Envelope selection, with ordering held fixed. To remove the ordering confound, we keep ACE's scores and candidate orders fixed and change only how each candidate is charged. The envelope charges it one-shot, against all higher-ranked candidates; sequential charging uses only those actually selected (Table 18; accuracy % / NFE):

TaskOne-shot envelope (ACE)Sequential charging
HumanEval38.4 / 25.937.8 / 25.6
MBPP29.0 / 21.829.6 / 20.7
GSM8K78.3 / 49.678.2 / 48.3
MATH-50028.4 / 65.427.2 / 64.1
IFEval58.6 / 96.958.0 / 95.7

The one-shot envelope uses only 0.3–1.3 more evaluations per sample than the sequential rule, at comparable accuracy. The envelope's isolated cost is therefore small, and its benefit is that coalitions come from one sort and one cumulative sum with a proved cost bound (see Question 3).

Question 1: how the baseline hyperparameters were chosen

Every baseline runs at its authors' published settings, and no setting of Fast-dLLM's own threshold reaches ACE (see Weakness 1, Tuning budget).

Question 2: removing attention on LLaDA

On LLaDA, removing attention, and with it the multi-order arbitration it enables, lowers accuracy on every task, by 0.2 to 3.6 points (see the first table under Weakness 2).

AuthorsOpenReview ↗

Response to Reviewer 6WZL (3/3): Question 3

Question 3: selection time and total decoding time against greedy selection

Under the matched conditions the reviewer describes, the two selectors cost the same in both selection time and total decoding time; the envelope's saving is against greedy selection that re-evaluates costs, which needs one denoiser pass per admitted token.

With the candidate order, attention weights, thresholds, gates and cache setting matched on Dream, and both selectors run in one job (Table 19):

TaskSelectorAccuracy / NFESelection per callSeconds per sample
HumanEvalGreedy63.4 / 112.31.56 ms2.84
HumanEvalACE envelope63.4 / 114.81.53 ms2.89
MBPPGreedy57.2 / 25.21.81 ms0.90
MBPPACE envelope57.2 / 25.21.68 ms0.89

ACE's entire non-denoiser cost, covering attention aggregation, selection and bookkeeping, is 2–4 ms per denoiser call, against a 16–51 ms denoiser forward pass (Table 14). Total decoding time is therefore governed by the number of denoiser evaluations, and there the one-shot envelope uses at most 2.6 more evaluations per sample than greedy at comparable accuracy (Table 25 and the second table under Weakness 2).

The two selectors tie here because both read one pass's attention weights. Greedy selection in general adds the candidate with the smallest marginal cost given the tokens already admitted, and in the attention game that cost is a sum of fixed pairwise weights, so re-evaluating it is a lookup. On the model's own dependence cost, re-evaluation needs a denoiser pass per admitted token. We ran and timed this greedy procedure on 200 recorded LLaDA decoding rounds per task (Appendix E.1; timing in Table 20):

TaskGreedy commit size per round (median)Re-evaluation stepRe-evaluation per roundACE selection per round
HumanEval620.0 ms100 ms1.8 ms
GSM8K549.3 ms195 ms1.9 ms
MATH-500436.5 ms109 ms1.8 ms
IFEval519.5 ms78 ms1.7 ms

Greedy selection with re-evaluation therefore commits about one token per denoiser pass, while ACE commits 2.1–3.5 tokens per pass at the same decoding states with no extra pass (Table 8).

The envelope's advantage is therefore not selector speed against a lookup-based greedy. It delivers greedy-level coalitions from one sort and one cumulative sum with a proved cost bound (Theorem 2), which also lets four candidate orders be scored together for 0.3–0.4 ms per call.


We hope these results answer the reviewer's questions on tuning, attribution and selection time. The revision adds the fixed-configuration results, the component ablation and the selector timing discussed above, and we are happy to clarify anything further.

Reviewer_Zat5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Attention Coalition Envelopes (ACE), a training-free decoding strategy designed to improve the quality–efficiency trade-off of parallel generation in diffusion language models (dLLMs). The authors argue that independently confident token predictions may be mutually incompatible when committed simultaneously, motivating the need to consider token dependencies during parallel decoding.

Critical strengths
  1. Interesting theoretical formulation: The paper introduces a cooperative-game perspective on parallel token commitment, distinguishing individual token readiness from compatibility between simultaneously generated tokens. The decomposition of prediction risk into individual errors and dependence costs provides a useful theoretical motivation for dependency-aware decoding.

  2. Efficient coalition selection: The proposed ordered modular envelope transforms coalition selection into a sorting and cumulative-sum operation, avoiding sequential dependency evaluations. This is particularly appealing for diffusion language models, where preserving parallelism is important for inference efficiency.

  3. Training-free and computationally efficient: ACE uses attention weights already available from the denoiser's forward pass, without requiring additional model training or extra denoiser evaluations. This makes the approach relatively straightforward to integrate into existing diffusion language models.

  4. Comprehensive empirical evaluation: The authors evaluate ACE on two diffusion language model architectures across six benchmarks covering coding, mathematical reasoning, instruction following, and multi-step reasoning. The results demonstrate competitive accuracy–efficiency trade-offs against multiple established decoding baselines.

  5. Useful ablation studies: The paper investigates several components of the proposed approach, including token ordering, compatibility constraints, adaptive commit sizes, and attention-based dependency estimation. The matched experiments provide useful evidence that ordering and compatibility-aware selection play complementary roles in decoding performance.

Critical weaknesses
  1. Disconnect between theoretical guarantees and the practical method: The theoretical results establish an upper bound on dependence costs under a coherent reference distribution. However, ACE replaces the theoretical mutual-information costs with attention-based scores, which are not guaranteed to upper-bound the true dependence costs. Consequently, the theoretical guarantees apply only to the constructed attention-based surrogate game rather than the actual decoding distribution. This limits the theoretical justification for the practical method.

  2. Attention may not reliably capture token dependencies: The method assumes attention weights provide a useful approximation of dependencies between tokens. However, the experiments show substantial variation in dependency-detection performance between LLaDA and Dream. In particular, attention performs poorly on some same-list independence controls, and the authors acknowledge that attention magnitudes are not calibrated dependence estimates. A more extensive investigation of when attention accurately reflects token dependencies would strengthen the proposed approach.

  3. Limited evidence for the advantage of the modular envelope over simpler selection strategies: Although the modular envelope is a central contribution, the matched comparisons against sequential greedy independent-set selection show nearly identical accuracy and denoiser evaluations in many settings (Table 17). ACE sometimes requires slightly more evaluations. Therefore, the practical advantage of the proposed envelope over simpler attention-based coalition selection remains unclear.

  4. Performance improvements are influenced by additional decoding heuristics: ACE combines attention-based compatibility with confidence thresholds, positional ordering, multiple candidate orderings, adaptive budgets, and early termination. Some of these components substantially affect performance independently of the proposed coalition mechanism. For example, the authors show that ordering accounts for much of the reasoning accuracy improvement, while stop-sequence termination can reduce denoiser evaluations by more than 50% in certain configurations. Stronger component-wise comparisons would help isolate the improvements attributable specifically to attention-based coalition selection.

  5. Fairness and generalizability of efficiency comparisons: The main latency comparisons use different caching configurations across methods, with ACE and Fast-dLLM using KV caching while several other baselines remain uncached. Although the authors provide additional uncached evaluations, a fully matched deployment comparison would make the efficiency claims more convincing. Furthermore, experiments are conducted on a single hardware configuration with a fixed 256-token generation canvas, leaving scalability to longer sequences and different hardware settings insufficiently explored.

  6. Task-specific hyperparameter selection: The reported balanced configurations are selected separately for each benchmark, and the authors explicitly describe them as post-hoc choices. Several parameters, including confidence thresholds, interaction thresholds, and decoding regions, vary across tasks. This raises questions about the robustness of ACE under a single configuration and the amount of tuning required for deployment. Evaluating a fixed configuration across benchmarks or performing held-out hyperparameter selection would strengthen the practical claims.

  7. Limited evaluation of stochastic decoding: Most experiments use greedy token commitment, whereas the theoretical formulation considers distributions over jointly sampled tokens. The authors report that dependence-score coverage decreases when conditioning on sampled rather than predicted tokens. More extensive stochastic decoding experiments would help establish whether the method's improvements and theoretical motivation generalize beyond greedy generation.

Questions for authors

Check the weakness for answering this,

6 comments
AuthorsOpenReview ↗

Response to Reviewer Zat5 (1/4): Weaknesses 1–2

We thank the reviewer for the detailed review and for recognizing the cooperative-game formulation, the one-shot envelope selection, and the breadth of the evaluation. Table and figure numbers below refer to the revised PDF.


Weakness 1: theoretical guarantees and the practical method

The theory fixes how ACE forms coalitions, and the one step where practice departs from it, estimating dependence with attention, is validated by measurement: the deployed score covers the measured commit cost in 94–97% of nontrivial rounds on every task (Table 8).

The decomposition, supermodularity and the envelope theorem hold for the true dependence cost, and Theorem 2 makes the envelope exact for the attention game. Each result fixes part of how ACE forms coalitions:

  • The decomposition into readiness and dependence (Proposition 1) is why ACE prices both under one shared budget. That budget alone cuts denoiser evaluations by 27–45% against threshold decoding, and the compatibility term adds accuracy (see Weakness 4).
  • Supermodularity (Propositions 2–3) implies that the feasible coalition size varies with the decoding state. Measured greedy chains range from 1 to 24 tokens per round, and adaptive ACE outperforms the best fixed commit size on three of four tasks (Figure 5).
  • The envelope theorem (Theorem 1) removes the sequential dependency of exact greedy selection. On Dream, one-shot selection matches the greedy selector's accuracy with at most 2.6 more evaluations per sample (Table 25; Table 17 in the submission).

The attention estimate itself is tested directly: it detects dependent pairs at AUROC 0.96 on LLaDA and 0.77 on Dream (Table 2). Coverage under sampled conditioning is discussed under Weakness 7.

Weakness 2: attention and token dependencies

Attention detects dependent token pairs on both backbones, at AUROC 0.96 on LLaDA and 0.77 on Dream against 0.50 for token proximity (Table 2), and its one failure mode only makes ACE more conservative.

The separation holds on interleaved and nested layouts, where token proximity fails. The failure mode is shared context: two independent draws from the same list attract attention to each other. This error is conservative, since a false alarm makes ACE commit fewer tokens in that round. On real decoding rounds, attention also ranks pairwise influence better than readiness alone (Table 8).

Part of the gap between backbones comes from position mapping. Dream predicts position p from the hidden state at p−1, and shifting the attention indices accordingly raises its AUROC to 0.90 in a post-hoc diagnostic (Appendix E.1). The deployed selector uses the original mapping, so every Dream result in the paper is obtained with the weaker signal. On Dream the selector only asks whether a candidate has an above-threshold interaction with its predecessors, which requires detecting dependence, not measuring its strength.

Calibration comes from the combined score, not from attention alone: with the readiness term, the deployed risk covers the measured commit cost in 94–97% of nontrivial rounds on every task (Table 8). Finally, ACE does not rest on attention alone: with attention removed on LLaDA, the shared budget keeps a 27–45% reduction in evaluations over threshold decoding, and attention adds accuracy on top of it.

AuthorsOpenReview ↗

Response to Reviewer Zat5 (2/4): Weaknesses 3–4

Weakness 3: modular envelope and simpler selection strategies

Matching greedy selection is the envelope's purpose: it forms greedy-level coalitions in one shot with a certified cost bound (Theorem 2), where exact greedy selection needs one denoiser pass per admitted token.

The greedy selector in Table 25 (Table 17 in the submission) is not a simpler alternative to the envelope; it follows the sequential procedure our theory implies, applied to precomputed attention weights. Because the dependence cost is supermodular (Proposition 2), a token's marginal cost depends on the tokens already admitted, so exact coalition formation recomputes costs after every addition (Proposition 3). For the true cost, each such step needs a fresh denoiser pass: our greedy-chain diagnostic spends 23 passes to build a 24-token chain (Appendix E.1). The ordered modular envelope removes this dependency. Every candidate receives a fixed charge before selection, so the coalition is formed with one sort and one cumulative sum.

Table 25 is therefore a parity test, and matching the greedy selector is the intended outcome. Accuracy is comparable in every Dream cell, and the envelope adds at most 2.6 evaluations per sample. The small surplus is expected: the envelope is an upper bound, so it is slightly conservative by construction. A new LLaDA control isolates this price by keeping ACE's scores and candidate orders fixed and restoring only the sequential charging (Table 18): the one-shot rule uses 0.3–1.3 more evaluations per sample at comparable accuracy (accuracy % / NFE):

TaskOne-shot envelope (ACE)Sequential charging
HumanEval38.4 / 25.937.8 / 25.6
MBPP29.0 / 21.829.6 / 20.7
GSM8K78.3 / 49.678.2 / 48.3
MATH-50028.4 / 65.427.2 / 64.1
IFEval58.6 / 96.958.0 / 95.7

This conservative approximation therefore costs neither accuracy nor a meaningful number of evaluations.

Weakness 4: additional decoding components

With ordering and termination held fixed, the coalition mechanism accounts for the gains: its shared budget cuts evaluations by 27–45% against confidence decoding, and its attention term then adds accuracy on every task.

The component-wise comparison the reviewer asks for adds ACE's components one at a time on LLaDA, starting from confidence decoding (Table 18). Every rung holds the block structure, prompts and termination policies fixed, so each change is attributable to the added component alone. Cells are accuracy (%) / NFE.

TaskConfidence+ coalition budget+ attention+ multi-order (ACE)
HumanEval34.8 / 38.034.8 / 23.737.8 / 27.638.4 / 25.9
MBPP19.8 / 34.925.8 / 19.227.8 / 22.629.0 / 21.8
GSM8K76.6 / 74.076.6 / 41.577.4 / 50.478.3 / 49.6
MATH-50027.8 / 96.328.2 / 57.230.2 / 65.728.4 / 65.4
IFEval57.7 / 126.156.5 / 91.957.2 / 98.058.6 / 96.9

The coalition budget is the efficiency step, with accuracy maintained or improved on four of five tasks. Attention compatibility is the accuracy step: it adds 0.7–3.0 points on every task and spends part of that saving, leaving ACE with 23–38% fewer evaluations than confidence decoding. Multi-order arbitration then recovers a few percent of the evaluations.

The two components the reviewer names do not enter these gains. The confidence rung runs the same termination policies as ACE, so the savings are net of early termination. The ladder's tier uses no order prior. Where the prior is used, Table 21 applies it to both selectors, and ACE keeps the accuracy with 32–36% fewer evaluations. Table 21 compares selectors as a whole; the ladder above adds the isolated attention-term comparison.

AuthorsOpenReview ↗

Response to Reviewer Zat5 (3/4): Weaknesses 5–6

Weakness 5: fairness and generalizability of efficiency comparisons

ACE's efficiency advantage does not depend on caching: it holds with caching off, with caching on for every applicable baseline, and at each method's shipped setting, and it persists on canvases four times longer.

On NFE, which caching does not change, ACE with its cache off lies on the accuracy–NFE Pareto frontier in all twelve settings (Table 12). We also timed every Table 1 row with caching off, one job per setting, with all methods run sequentially on the same GPU on the same prompts and canvas. The one exception is Dream Fast-dLLM, whose released threshold decoder is implemented with its prefix cache; it keeps that cache, which favors the baseline. In this round ACE stays on the accuracy–latency Pareto frontier in all twelve settings and is the fastest decoder in six. On Dream HumanEval, GSM8K and BBH, it is outpaced only by the cached Fast-dLLM, while dominating every uncached baseline. With caching switched on for every applicable baseline instead, ACE is still faster than each of them on all six LLaDA tasks: the fastest cached baseline takes 1.2–3.3 times as long per sample. With each method at its shipped cache state (Table 1), cached ACE is again on the frontier in all twelve settings and fastest in ten. The revision adds the uncached timing table to Appendix E.3 (Table 15) and the cached comparison to Section 6 (Table 5).

ACE also scales to longer outputs (Table 17). With the deployed 64-token blocks on LLaDA HumanEval, MBPP and GSM8K, moving from 256 to 1024 tokens changes ACE's accuracy by at most 1.6 points and its NFE by at most 32%; on GSM8K its NFE even falls. Over the same range, DAPD-Direct loses 6.1 points on HumanEval and 3.2 on GSM8K. On GSM8K, ACE also leads Fast-dLLM at both longer lengths with 37% and 44% fewer evaluations:

TaskMethod2565121024
HumanEvalACE38.4 / 25.938.4 / 31.937.2 / 34.2
HumanEvalDAPD-Direct31.7 / 22.934.8 / 24.325.6 / 26.4
MBPPACE29.0 / 21.828.4 / 20.729.2 / 20.5
MBPPDAPD-Direct36.6 / 22.435.8 / 22.636.2 / 22.8
GSM8KACE78.3 / 49.678.8 / 46.676.7 / 40.9
GSM8KDAPD-Direct70.5 / 35.469.5 / 28.767.3 / 26.3
GSM8KFast-dLLM77.0 / 74.575.7 / 73.774.5 / 73.2

Finally, every comparison is also reported in NFE, which does not depend on hardware. ACE adds only 2–4 ms of attention, selection and bookkeeping per denoiser call (Table 14), so its decoding time is governed by the number of evaluations.

Weakness 6: task-specific hyperparameter selection

ACE holds up under one fixed configuration on each backbone: a single recipe lies on the accuracy–NFE frontier in five of six tasks on both LLaDA and Dream.

On LLaDA, each of the three ACE tiers (throughput, default, quality) is already one recipe used unchanged on all six tasks. The only per-task switch is stop-sequence termination, which leaves scored outputs unchanged. The balanced row of Table 1 only chooses which fixed tier to display. Table 12 reports each tier on its own. The recipe choices are also recoverable from held-out data: on 100 held-out GSM8K training problems, the validation results reproduce every deployed choice of decoding region, order prior and budget (Table 28). Dream is where the readiness gate is set per task. We therefore evaluated one fixed configuration on all six Dream tasks (γ = 0.8, τ = 0.1, every other setting unchanged; accuracy % / NFE):

TaskACE, fixed configurationAgainst the five baselines
HumanEval58.5 / 96.3better on both axes than all five
MBPP57.2 / 25.2better on both axes than all five
GSM8K57.9 / 68.1most accurate, on the frontier
MATH-50031.0 / 96.0below DAPD-Direct (31.6 / 63.3)
IFEval50.0 / 131.2most accurate by 12.2 points, on the frontier
BBH56.4 / 33.8better on both axes than all five

With this single configuration, ACE stays on the frontier in five of six tasks, beats every baseline on both axes in three, and is the most accurate decoder in five. The per-task gate refines efficiency, most visibly on IFEval, rather than producing the result. The revision adds the fixed-configuration results to Appendix E.6 (Table 27).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised PDF: theory-to-method link, matched controls, and robustness checks

We thank all four reviewers for their careful reading. We have uploaded a revised PDF in which every addition is marked in blue, and the main text still fits within the 10-page limit. Our replies use the revised numbering. Three tables are new in the main text, so later numbers shift: the tables cited in the reviews are now Table 21 (formerly 13), Table 22 (formerly 14) and Table 25 (formerly 17), and the former Appendices E.3, E.4 and E.5 are now E.4, E.5 and E.7.

  1. Theory to method (PJxX; f68y W1). The set cost covered by ACE's guarantee was defined only in Appendix A.8. Section 5 now defines it where the ACE scores are introduced (Equation 7), states the bound the scores certify (Theorem 2), and notes that readiness is priced under the same budget as the attention cost.
  2. What drives the gains (f68y W1–W2; Zat5 W5). To separate attention, termination and caching, Section 6.3 adds three controls. Inside the shared budget, attention adds accuracy on every task while the budget cuts evaluations by 27–45% (Table 3); under identical termination, ACE uses 23–61% fewer evaluations than the threshold, EB-Sampler and KLASS rules (Table 4); and with caching on for every applicable baseline, ACE is fastest on all six LLaDA tasks (Table 5).
  3. Caching and output length (Zat5 W5; f68y W2). To test whether the latency advantage depends on caching or on shorter answers, the new Appendix E.3 shows that with caching off ACE stays on the accuracy–latency frontier in all twelve settings (Table 15), its answers are not shorter (Table 16), and its accuracy holds within 1.6 points up to 1,024 tokens (Table 17).
  4. Components and selection time (Zat5 W3–W4; 6WZL W2, Q2–Q3). To isolate each component, Table 18 adds them one at a time; the one-shot envelope costs at most 1.3 evaluations per sample over sequential charging. Greedy selection ties with the envelope on matched inputs (Table 19), but exact greedy selection needs one denoiser pass per admitted token: 78–195 ms per round against ACE's 1.7–1.9 ms (Table 20).
  5. Tuning and sampling (Zat5 W6–W7; 6WZL W1, Q1). To test dependence on per-task tuning and greedy decoding, the new Appendix E.6 shows that one fixed Dream configuration stays on the frontier in five of six tasks (Table 27), held-out data recovers every LLaDA recipe choice (Table 28), no Fast-dLLM threshold reaches ACE (Table 29), and ACE keeps its greedy accuracy under sampling (Table 30).

We respond to each reviewer in their own thread.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.