acceptodds
Under review as a conference paper at ICLR 2027

Self-Repulsive Sampling for Diffusion Language Models

Abstract

Sampling several responses and voting over their answers can improve a language model's accuracy, but repeated answers limit the benefit of additional samples. Raising temperature increases diversity at a potential cost to per-sample accuracy. We introduce Self-Repulsion (SR), a sampler for masked diffusion language models that uses peer commitments to diversify the pool. At each penalized denoising step, each path lowers a token's logit according to how many peers have committed that token at the same position. Paths share a batched forward pass and then commit in sequence, so later paths observe choices made earlier in the same step. This coupling requires no training or additional forward or backward pass and can produce distinct paths even at temperature zero. When all paths commit a position together from identical logits, the update exactly maximizes total logit minus a convex duplication cost. On LLaDA-8B-Instruct with ten paths and 128 denoising steps, deterministic SR reaches 80.38% plurality accuracy on GSM8K, compared with 70.17% for the unpenalized greedy decoder. At temperature 0.6 and matched model-evaluation budgets, the count penalty improves over self-consistency by 2.06 percentage points in blocks of 32 and 14.50 under pure diffusion. Experiments on GSM8K, MATH and TruthfulQA show that voting gains arise mainly from higher coverage of correct answers, with gains that vary by benchmark and decoding regime.

Primary area:
generative models
Keywords:
discrete diffusion, masked diffusion language models, self-consistency, repulsive sampling, sample diversity, test-time compute

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SdNPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the problem of how to increase the diversity among a batch of samples/generations in masked dLLMs like LLaDA. They propose a sampler that promotes diversity within a batch by down weighting tokens that other samples in a batch sampled at a given position (Algorithm 1). While the proposed idea is interesting, some theoretical claims are unfounded (Proposition 1, see weakness section below) and experimental evidence is inconclusive about how the proposed approach compares with highly related work on improving diversity in dLLMs [1, 2].

Also, ofc I might be wrong, but the submitted manuscript feels AI generated in a lot of parts (introduction, entire Section 2). Why submit such a manuscript to a conference when the peer-review system is already so overloaded? :(

Critical strengths
  • The setting of studying diversity in dLLMs is interesting and important in the light of recently discovered Flexibility Trap [3].
Critical weaknesses
  • The writing is quite bad. For example introduction really doesn't read smoothly and it in many parts it provides too much information for introduction without first providing the right context. Some sentences/parts read like AI slop imo, for example: Ensembles without randomness. At temperature zero a symmetric start-of-step sampler is degenerate: its K samples coincide, so the vote cannot beat a single decode. Reading the peers within the step can make the paths differ.
  • There is no background section on dLLMs or sampling in dLLMs. Some background info is scattered around the introduction and then Section 2 jumps right into the proposed method...
  • The paper seems to claim that if we generate K samples/generations in dLLMs at τ=0\tau=0 then we get no diversity between K samples. But that's not true or I am misunderstanding the claims (Proposition 1)? Because even for greedy decoding (T=0) we can vary the generation order in dLLMs which will results in different generations for the same prompt? just take LLaDA ckpt from HuggingFace and generate with random unmasking strategy. see also see also TposT_{\text{pos}} in [1]
  • Section 2.2 also feels AI generated, it's hard to read through it in details
  • Experiments are weak. Main experimental figure is Figure 2 where they do not compare with highly related works like ODD [2] or TLC/TCT [1]. Also showing results only up to k=10 is not enough, at least k=16 or k=32 would be nice. And at least one more model besides LLaDA/LLaDA-1.5, it could be Dream/iLLaDA/SDAR/LLaDA-2 etc. Because of this limited experiments scope, it's impossible to judge the proposed method effectiveness at this point
Questions for authors

See weaknesses above.

  • Also do I understand correctly that you assume that each sample in a batch unmasks same number of tokens in each sampling steps? Could you comment on how your proposed sampler is compatible with adaptive samplers like Fast-dLLM [4] for which the number of unmasked tokens can vary between sampling steps?

[1] Olausson et al. A Tale of Two Temperatures: Simple, Efficient, and Diverse Sampling from Diffusion Language Models

[2] Lamont et al. Free Lunch for Pass@k? Low Cost Diverse Sampling for Diffusion Language Models

[3] Ni et al. The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models

[4] Wu et al. Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding

1 comment

We address the reviewers requests below. Several listed weaknesses concern material already in the submitted paper; we give the location in each case, and report the requested experiments.


W1 / W2. Writing; background

Response: We revise the introduction so that the setting (masked diffusion decoding, self-consistency, why K samples duplicate) precedes the method, and we add a background section on masked diffusion decoding, the confidence commit rule, pure diffusion versus blocks, and self-consistency. The quoted paragraph is Proposition 1 in two sentences; we unpack it. In Section 2.2 one sentence will introduce each result.


W3. Proposition 1 and random unmasking order

At T=0 a random unmasking order gives diverse samples, so the claim seems wrong.

Response: Proposition 1 does not make the claim attributed to it. Its hypothesis is that each path's commit is one deterministic function of the canvases read at the start of the step; it then states that such a pool holds one distinct sample. A random unmasking order is not such a function. It is randomness applied to each path independently of its peers, the same class as token temperature (Section 2.3), and every temperature-zero statement in the paper concerns deterministic commit rules. The distinction is measurable: at matched disagreement, peer-blind randomness costs 7 to 13 per-sample points relative to the peer statistic (Table 3). We move the hypothesis into the statement and add the random-order case as a remark; the proposition and its proof are unchanged.


W4a. ODD and TLC/TCT

Figure 2 does not compare with ODD [2] or TLC/TCT [1].

Response: ODD is compared throughout: at both step sizes in the main comparison (Table 2), at its published temperature with paired intervals (Table 12), and in pass@k (Table 16). Figure 2 isolates the coverage gap against self-consistency; we add ODD to it. For the rebuttal we also ran ODD under pure diffusion with the same EOS correction as SR (LLaDA-8B-Instruct, GSM8K, 1280 evaluations). SR leads ODD at both strengths, and its overhead over SC is a fraction of ODD's (cost: H200, K=10, blocks of 32):

Plurality, pure diffusionTime vs SCPeak memory vs SC
SC57.02
ODD alpha=6458.97
ODD alpha=25666.09+19.5%+37.1%
SR71.53+1.3%+0.7%

TLC/TCT is cited in the introduction and related work. It samples which positions each path commits, independently of its peers, so it is a peer-blind sampler, and its authors explicitly report that this diversity does not reliably improve the self-consistency vote. SR couples the paths, which is why it yields distinct samples at temperature zero.


W4b. K beyond 10

Only up to K=10; K=16 or K=32 needed.

Response: Table 6 (Appendix C.3) reports K up to 32. Coverage rises with K and the vote is flat; the paper states this as a limit.

GSM8K, blocks 32K=10K=20K=32
Plurality78.9379.4079.38
Coverage93.3395.5096.33

W4c. Models beyond LLaDA

Response: We add Dream-v0-Instruct-7B (Qwen2.5-initialised, independent of LLaDA); LLaDA-1.5 is Table 11.

GSM8K, blocks 32, K=10, 2 seedsPluralityCoverage
Dream, SR87.9996.00
Dream, SC t=0.684.7693.75
Dream, SC t=1.087.1095.50

SR beats SC at matched temperature on both metrics (significant over 1200 pairs) and stays ahead of SC at its best temperature: the LLaDA block pattern. lambda=32 was picked on 200 tuning problems disjoint from evaluation.


Q1. Equal counts; Fast-dLLM

Response: Equal counts are not assumed. Within a step the paths commit in sequence, and the peer count at position j updates whenever a peer commits j, so the number of positions a path commits does not enter. Under a confidence threshold as in Fast-dLLM, each path commits its own set and the penalty applies unchanged. We ran only the fixed-count schedule and state adaptive schedules as untested.

We hope these corrections and experiments allow a reassessment of the soundness and informativeness scores.

Reviewer_p977OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces self-repulsion, a training-free sampler for masked diffusion language models. Multiple generation paths share a batched model evaluation, then commit tokens sequentially and each path penalizes tokens already committed by peers at the same position. This ordering allows deterministic paths to diverge even at zero temperature. Plurality accuracy, coverage, and diversity are evaluated on GSM8K, MATH, and TruthfulQA, primarily using LLaDA-8B-Instruct.

Critical strengths
  1. The method is simple and produces a useful deterministic ensemble. The count penalty requires no training or additional model evaluations, and the within-step commitment order provides a clear mechanism for breaking identical greedy trajectories. The deterministic GSM8K result of 80.38% versus 70.17% for greedy decoding is compelling evidence that the intervention can improve inference-time performance.
  2. The paper investigates more than headline accuracy. It includes temperature controls, alternative peer statistics, ensemble-size sweeps, deterministic ablations, and a second checkpoint on GSM8K. The distinction between coverage and plurality is valuable, and the local optimization results appear correct under their explicitly restricted assumptions. The five-seed GSM8K comparison also reports positive improvements over self-consistency for every seed.
Critical weaknesses
  1. The central mechanism attribution is not established by the reported diagnostics. Appendix D.5 concludes that coverage gains arise from changed marginals rather than dependence, but its reported 1.94-point dependence ceiling still permits roughly one third of the 5.94-point gain. More fundamentally, a three-path within-/cross-pool diagnostic cannot determine the dependence contribution to ten-path coverage. Appendix D.6 then applies a nonlinear IID coverage formula to stratum-level accuracies and interprets the residual as dependence, although within-stratum heterogeneity, unequal path marginals, and reuse of baseline samples for stratification can also produce that residual. These issues weaken the explanatory claims without invalidating the observed accuracy gains.
  2. The comparative evidence is less decisive than the strongest headline results suggest. Pure-diffusion TAPS uses an uncorrected EOS protocol while SR uses the correction, and the closest baseline, ODD, is absent from that regime. In block decoding, the deployed count variant is approximately tied with self-consistency on MATH, its GSM8K advantage over temperature-adjusted ODD is 0.50 points, and tuned ODD leads on TruthfulQA. Paired uncertainty is needed to assess the smaller differences. Furthermore, matched model-evaluation counts do not establish negligible runtime overhead for the sequential commitment loop.
Questions for authors
  1. Can you estimate ten-path dependence directly by comparing intact SR pools with independent pools assembled from index-matched samples across independent runs? How does the marginal/dependence decomposition change when difficulty strata and marginal estimates use held-out runs?
  2. What are the pure-diffusion results for TAPS with the same EOS correction and for ODD under the matched schedule? If either comparison is unavailable, what specific implementation or methodological limitation prevents it?
  3. Were the 200 problems used to select penalty strength disjoint from evaluation? Please provide paired problem-level confidence intervals for the principal comparisons against temperature-adjusted baselines and clarify how sampler variants were selected.
  4. Please give explicit formulas and probability conventions for the expected-count and collision-weighted variants. What additional latency and memory does the cascade incur relative to self-consistency at the same batch size, model-evaluation budget, and hardware?
1 comment

We thank the reviewer for proposing the direct test.


W1 / Q1. Ten-path dependence; D.6 with held-out runs

Response: We ran both on stored outputs of existing runs.

Direct estimate. We compare the coverage of an intact pool (ten paths from one run) with a pool assembled from paths of different runs, which has the same per-path accuracies and no dependence. Three to five runs do not give ten independent paths, so we assemble pools from two to five runs and extrapolate to full independence. SC paths are independent by construction, so what the procedure reports for SC is its error, which we subtract. Path order is drawn uniformly, which can only overstate SR's dependence. GSM8K coverage, blocks of 32:

3 runs, 2560 eval5 runs, 1280 eval
SR, intact pool93.8993.53
SR, paths from different runs93.4993.48
SC, intact pool87.9489.70
SC, paths from different runs87.5989.90
Dependence in SR (SR minus SC difference)+0.04+0.25
SR coverage gain over SC+5.94+3.83

Dependence is a few percent of the gain at most, indistinguishable from zero.

Held-out check of D.6. D.6 predicts the coverage gain from per-path accuracies alone, assuming independence, and compares it with the measured gain per difficulty bin. We repeated it with bins, accuracies and coverage each from a different run. The measured gain is 1.05 and 1.04 times the predicted one (paper: 1.07), so per-path accuracies explain the gain as before. The formula is slightly biased with ten samples per problem, so we treat this as a consistency check; the direct estimate is the measurement and uses no difficulty bins, so within-bin heterogeneity does not enter it.

We replace D.5's diagnostic and ceiling argument with the direct estimate. The conclusion is unchanged: the coupling raises coverage by changing which answers each path produces, not by making the answers dependent. A peer-blind control with the same commit allocation (1280 eval) stays at the self-consistency level, so the peer read produces the gain. The accuracy results do not depend on this attribution.

SC, t=0.6Control, T=0SR, T=0
GSM8K plurality77.2377.9280.38

W2 / Q2. Pure diffusion: TAPS and ODD

Response: ODD under the Table 2 pure-diffusion protocol (K=10, 128 steps, t=0.6, EOS correction, two seeds). Plurality, best in bold:

SCODD 256ODD 64SR (count)SR (collision)
GSM8K57.0266.0958.9771.5376.27
MATH24.0032.6727.7329.7935.37
TQA56.2758.9458.6358.5456.99
MATH, unparsed %18.274.12

Count leads ODD 256 on GSM8K and trails it on MATH, where many ODD answers do not parse; collision leads both on GSM8K and MATH; TQA is within one point. ODD is a separate run: means only.

TAPS's released loop has no hook for the EOS correction and its paper reports LLaDA only at block length 128, so TAPS stays in the block regime.


W2 / Q3. Block margins, intervals, tuning, variants

Response: Blocks of 32, plurality (paired intervals in Table 12):

SRSC 0.6SC 1.0TAPSODD 256ODD 64
GSM8K79.1177.0578.2877.8778.6178.23
MATH40.0440.1439.4739.3137.5236.66
TQA59.7759.0258.6557.7460.1260.86

SR is best on GSM8K, tied with SC on MATH, and about a point behind tuned ODD on TQA, at 15% less time and 26% less memory than ODD (Q4). SR also leads every baseline on coverage in both regimes (Table 2). Significant: GSM8K vs SC 0.6, MATH vs both ODD.

Tuning. Strengths (count 64, expected count 64, collision 128) were tuned at one seed on the first 100-200 problems of an earlier seeded GSM8K draw, with 0 of 600 evaluation problems in common. MATH and TQA reuse them. The T=0 strength follows a fixed rule.

Variants. Count is the deployed method, fixed before evaluation; the best per column is not presented as one method.


W2 / Q4. Formulas; cost

Response: l_k[j,v] <- l_k[j,v] - lambda/(K-1) * kappa_k[j,v], with p_k'(j,.) = softmax(l_k'[j,.]/tau) at step start and pbar_k(j,v) = mean over peers k' != k of p_k'(j,v).

  • Count: peers that have committed v at j (updated within the step).
  • Expected count: pbar_k(j,v).
  • Collision-weighted: (w_j / mean_masked w) * pbar_k(j,v), w_j = sum_v p_k(j,v) pbar_k(j,v), the collision probability with peers; mean weight one.

Both distribution-aware statistics use step-start state and are degenerate at T=0 (Prop. 1). Formulas go to App. A.

H200, K=10, per problemTimePeak memory
SC33.95 s18.89 GiB
SR34.41 s (+1.3%)19.03 GiB (+0.7%)
ODD40.58 s (+19.5%)25.89 GiB (+37.1%)

All analyses above enter the revision; we hope this resolves the soundness concerns.

Reviewer_iyw7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a training-free sampling method for diffusion language models that penalizes tokens already committed by other generation paths. It enables deterministic diversity and improves answer coverage and voting accuracy on GSM8K, MATH, and TruthfulQA.

Critical strengths

The method introduces position-wise, within-step coupling without additional model evaluations. Unlike existing diversity methods, it exploits intermediate token commitments during diffusion decoding. This is a meaningful algorithmic distinction despite conceptual similarities to previous work.

The paper includes temperature sweeps, matched-computation comparisons, peer-blind controls, and multi-seed experiments. On GSM8K, the default method improves plurality accuracy by 2.06 percentage points over self-consistency across five paired seeds. The coverage-versus-selectivity analysis also helps explain when increased diversity translates into better answers.

Critical weaknesses

The underlying repulsion principle is closely related to Diverse Beam Search and ODD. The main innovation is its adaptation to position-wise commitments in diffusion decoding. The theoretical allocation result applies only under restrictive shared-logit conditions, limiting its implications for complete generation trajectories.

The largest gains occur under pure diffusion, where the baseline is substantially weaker. Under stronger block decoding, improvements over temperature-tuned self-consistency and ODD are relatively modest and benchmark-dependent.

Experiments are largely restricted to the LLaDA family. Moreover, matched model-evaluation counts do not establish matched wall-clock latency, and D5P4 is discussed without a direct experimental comparison.

Questions for authors

Can the authors report wall-clock latency and throughput against ODD and D5P4 under matched hardware and decoding budgets?

Do the gains transfer to independently developed diffusion language model architectures?

Can the authors better isolate the advantages of within-step commitment feedback from conventional sequential diversity penalties?

3 comments

We thank the reviewer for the detailed analysis, we address the poitns below.


W1. Relation to DBS and ODD; shared-logit theory

Response: We agree that the within-step position-wise coupling is the algorithmic distinction, and the evidence shows it is also the component responsible for the gain. In fixed order, our count read is DBS's Hamming penalty at one path per group, and at a position a peer has taken, a fixed-order decoder has two options: the peer's token or a lower-confidence substitute. An any-order decoder has a third: leave j masked and commit elsewhere. The paper measures which SR takes: at T=0 only 3.00% of commits change token under the penalty (Sec. 2), so the paths separate by where they commit. ODD repels each sample from its predecessors only, pools over positions and needs an autograd step per denoising step. The shared-logit condition enters only the allocation result (Prop. 2). The tilt result holds at every position and step at positive temperature: the penalty is the I-projection of the model's distribution onto laws with bounded expected peer agreement.


W2. Largest gains where the baseline is weak

Response: The baseline is weak under pure diffusion for a specific reason, and that reason is the object of study: in this fully parallel regime, self-consistency's ten samples duplicate, so coverage sits barely above per-sample accuracy. The baseline is not weak for lack of tuning: Table 1 sweeps temperature to its usable limit and the vote still peaks well below SR, and ODD, the closest competing sampler, also falls below SR under the Table 2 protocol. On GSM8K the margin over the best alternative is largest where duplication is most severe.

GSM8K, pure diffusionSCODD 256SR
Plurality, Table 1 sweep (SC at best tau)62.0073.09
Plurality, Table 2 protocol57.0266.0971.53
Coverage, Table 2 protocol67.0087.75

Temperature cannot remove the duplication without a large loss in per-sample accuracy: at matched disagreement it costs 7 to 13 per-sample points relative to the peer statistic (Table 3). SR removes it at unchanged per-sample accuracy and brings pure-diffusion coverage to within two points of block self-consistency.

Block decoding is a semi-autoregressive remedy for the same duplication, so the two overlap and SR's margin shrinks in blocks, as the mechanism predicts. Two claims differ there. Coverage: SR leads every baseline on all three benchmarks in both regimes (Table 2). Vote: only the GSM8K gain over SC at matched temperature is significant over five paired seeds (Table 12); the revision states both. Pass@k puts SR above self-consistency at every k >= 3 in blocks, including MATH (Figure 2).


W3 / Q1. Latency; D5P4

Response: One H200, one method at a time, 50 GSM8K problems, three repetitions, K=10, 2560 evaluations. SR's overhead is an integer count per position and does not depend on the model; ODD's is an autograd pass through it.

Per problemTimePeak memory
SC33.95 s18.89 GiB
SR34.41 s (+1.3%)19.03 GiB (+0.7%)
ODD 25640.58 s (+19.5%)25.89 GiB (+37.1%)

D5P4 selects from a larger candidate pool and is not budget-comparable at matched evaluations; the revision states this.


W3 / Q2. Independently developed architectures

Response: We add Dream-v0-Instruct-7B (Qwen2.5-initialised, independent of LLaDA); LLaDA-1.5 is Table 11:

GSM8K, blocks 32, K=10, 2 seedsPluralityCoverage
Dream, SR87.9996.00
Dream, SC t=0.684.7693.75
Dream, SC t=1.087.1095.50

SR exceeds SC at matched temperature on both metrics (significant over 1200 pairs) and stays ahead of SC at its best temperature: the LLaDA block pattern. lambda=32 was selected on 200 tuning problems disjoint from evaluation.


Q3. Within-step feedback versus sequential penalties

Response: Appendix D.3 runs both with the count penalty: ten parallel paths reading peers within the step, and a chain of five drafts, each repelled from completed drafts.

GSM8K, T=0, 2560 evalPluralityCoverage
Parallel SR, K=1078.6092.33
Sequential chain, 5 drafts78.0390.33

At temperature 0.6 the chain at half the evaluations matches SC (not significant) and falls below parallel SR (significant), at K times the latency. A matched control that keeps the allocation law and removes the peer read stays at the self-consistency level (1280 evaluations):

SC, t=0.6Control, T=0SR, T=0
Plurality77.2377.9280.38

Proposition 1 gives the reason: a penalty read only at the start of a step cannot separate deterministic paths.

Q1 to Q3 are answered by runs, and the single-family and wall-clock items of W3 are addressed. We hope this supports a reassessment of soundness and rating.

Thank you for the response. The new results address my concerns regarding wall-clock overhead and single-family evaluation, and the peer-blind control for Q3 is convincing.

That said, my core concerns remain. In block size of 32, the voting gain over tuned baselines is under 1 point on GSM8K and largely mixed on MATH and TruthfulQA. Moreover, the 3% statistic does not establish that paths actually diverge due to deferral, leaving the distinction from DBS indirect. A direct comparison against D5P4 at matched forward passes is also still missing, even though its performance looks comparable.

Also, the coverage claims are restricted to GSM8K and MATH, as Table 2 neither reports TruthfulQA coverage nor includes a pure-diffusion OOD baseline.

We thank the reviewer for the follow-up. We ran the D5P4 comparison and a count of deferral (GSM8K, 600 held-out problems, K=10, 1280 evaluations) and answer the four points in order.

Block-decoding margin

For the sampled default row (t=0.6) we agree, and Table 12 reports it. That comparison gives each baseline its best temperature and holds SR at 0.6. The paper's headline configuration is SR at temperature zero (abstract; same count statistic, fixed-rule strength), so we computed its paired differences to the same tuned baselines from the Table 12 decodes (600 problems, 95% intervals):

GSM8K, blocks of 32, 1280 evaluationsPluralitySR (T=0) minus rowCoverage
SC, t=1.0 (its best)78.28+2.09 [+0.06, +4.21]91.42
TAPS, t=1.077.87+2.51 [+0.54, +4.56]93.00
ODD 64, t=1.078.23+2.14 [+0.11, +4.21]92.17
ODD 256, t=1.078.61+1.77 [-0.49, +4.06]91.83
SR, t=0.679.1193.53
SR, T=080.3894.33

The vote of SR at temperature zero is above that of every baseline at its best temperature, and three of the four intervals exclude zero; the exception is ODD 256. Against self-consistency at t=1.0, coverage is also significantly higher (+2.92 [+1.00, +4.92]). The other diversity methods do not show such a gain in blocks: at their best temperature, TAPS and ODD are within half a point of self-consistency at t=1.0, and D5P4 does not improve the vote over independent sampling (next section). SR at temperature zero also has no temperature to tune, and it is a small change to the decoder: each path lowers a token's logit in proportion to the number of peers that have committed that token at the same position. In comparison, ODD takes a gradient step on the logits at every denoising step, TAPS adds annealed noise to the prompt embeddings, and D5P4 forwards extra candidates and selects among them by their hidden states.

On MATH and TruthfulQA, the votes of SR and self-consistency do not differ significantly, and we agree with the reviewer on these two benchmarks. On MATH, SR is significantly above both ODD settings. The revision adds the temperature-zero comparison to Table 12.

Do the paths diverge by deferral?

We now count it directly. At temperature zero we log, for every position where a path's preferred token is already held by a peer, what the path does.

SR, T=0, positions inside the penalty windowShare
Preferred token already held by a peer31.39%
... of these, left masked while the path commits elsewhere99.65%
... of these, a different token written at once0.35%
Deferred positions that later commit the token first refused88.71%

The paths separate by commit order: a path that meets a peer's token leaves the position masked and commits elsewhere, and most deferred positions later take the token first refused. A fixed-order decoder such as Diverse Beam Search has no such option; at a contested position it must write a different token at once. The distinction from DBS is now measured directly; the revision adds this table.

D5P4 at matched forward passes

We ran the authors' released selector (pinned commit, greedy MAP) inside their LLaDA loop at ten forward passes per step: five groups of two candidates, one kept per group. The diversity weight was tuned on 200 disjoint problems. The first row is the same loop with ten independent samples and no selection.

GSM8K, t=0.6, 2 seedsPure diffusion: pluralitycoverageBlocks of 32: pluralitycoverage
Independent samples55.4766.5076.3291.50
D5P456.0863.7576.5087.50
SR (Table 2)71.5387.7579.1193.53

At matched forward passes D5P4 does not change the vote (not significant in either regime), raises per-sample accuracy by about one point (significant), and lowers coverage, since half of the forward passes go to discarded candidates. This agrees with the D5P4 paper's own LLaDA results, where pass@1 equals independent sampling. The revision adds this table and drops the statement that D5P4 is not budget-comparable.

Table 2: TruthfulQA coverage and pure-diffusion ODD

TruthfulQA is four-way multiple choice. Ten uniform random answers contain the correct option with probability 94.37%, so coverage carries no information there; the caption now says so. ODD under pure diffusion, with the same end-of-sequence correction as the other rows, enters Table 2:

Pure diffusionSRODD 256ODD 64SC
GSM8K plurality71.5366.0957.02
GSM8K coverage87.7581.9278.4267.00
MATH coverage50.5151.42

On MATH coverage, ODD at its larger strength is ahead of the count variant by under one point, and the revision reports it. On GSM8K, SR leads ODD on both metrics.

Each remaining point is answered by a direct measurement; we hope this supports a reassessment of the rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.