-
The experimental scope is insufficient to support broad conclusions about diffusion language models.
Although the paper describes three testbeds spanning several orders of magnitude in model size, two are controlled Sudoku models with only 0.54M and 6.6M parameters. The only large language-model evaluation uses LLaDA-8B-Instruct on GSM8K at temperature 0.9.
This leaves several important questions unresolved. The paper does not establish whether the same conclusions hold for other large diffusion language models, other reasoning benchmarks, or different sampling configurations. In particular, the temperature sweep is conducted only on the smaller Sudoku model, even though the paper's central finding is that the amount of exploitable within-prompt variation depends strongly on the sampler and model.
The results themselves illustrate this dependence: at r=0.5, the estimated within-prompt variance for Sudoku-M is below 10−3, compared with approximately 0.05 for Sudoku-S at temperature 0.9. Such differences suggest that the availability of selection opportunities is highly model-dependent.
The paper therefore provides evidence for the proposed framework in the tested settings, but not yet for a general characterization of inference-time search in diffusion language models. Additional experiments on multiple large models, tasks, and sampling temperatures are needed to establish the external validity of its conclusions.
-
Limited sample sizes and run-to-run variability weaken some quantitative conclusions.
The smaller Sudoku testbed uses only 50 held-out puzzles. The paper itself reports substantial uncertainty in the perfect-selector ceiling: at r=0.95, the estimated accuracy is 0.879 with a 95% confidence interval of [0.780, 0.960]. It also reports that two nominally identical runs on the MPS backend produce selector ceilings of 0.958 and 0.879, respectively.
Importantly, the reported confidence intervals are bootstrapped over prompts and do not include this run-to-run variability. Thus, they may understate the uncertainty in quantitative estimates when execution-level randomness is material. The large-model branching experiment uses 120 prompts, while the linear-probe analysis is based on a 30-prompt run. These sample sizes limit the precision with which small improvements or null effects can be distinguished.
These limitations do not invalidate the paper's central observations, but they affect the strength of its quantitative claims. More independent runs and uncertainty estimates that account for both prompt-level variation and run-to-run variability would make the conclusions more robust.
-
The reported selection headroom should be distinguished from the gain achievable by a partial-state selector.
On LLaDA-8B, the paper reports a baseline accuracy of 0.768 and an accuracy of 0.919 for its perfect selector, corresponding to an upper-bound improvement of approximately 15 percentage points. Model confidence at r=0.5, by contrast, yields an accuracy of 0.785, capturing only about 1.6 percentage points of improvement.
However, the paper explicitly defines the LLaDA perfect selector by choosing a correct completed candidate whenever one exists. This is an upper bound on selection among the completed trajectories, not the attainable ceiling for a selector operating on partial states at r=0.5. The existence of substantial outcome variation among branches does not by itself establish that the same amount of improvement can be achieved using information available at that intermediate step.
The branching estimator provides evidence that within-prompt variation exists, but the relationship between this variation and the accuracy gain achievable by a realistic partial-state verifier remains incompletely characterized. The paper should more carefully distinguish the measured variance available in principle, the upper bound obtained from completed outcomes, and the gain achievable by a practical selection rule at a specified decoding stage.
-
The evaluated internal-state signals do not exhaust plausible selection mechanisms.
The paper evaluates model confidence, entropy, and linear probes over residual-stream representations. These are reasonable baselines, but the negative results do not establish that internal representations generally lack useful information about partial-state correctness.
In particular, the large-model linear-probe experiments use only 30 prompts. The paper also acknowledges that a trained nonlinear verifier could behave differently. Moreover, the experiments do not establish whether task-aware features, more expressive learned selectors, or alternative readouts could extract information that the tested signals fail to capture.
The exact symbolic oracle performs well on Sudoku, demonstrating that useful selection information can be extracted from task structure. However, this oracle is not available for general reasoning tasks such as GSM8K. The gap between a strong task-specific oracle and the tested internal-state signals therefore remains unresolved.
The appropriate conclusion is that the tested signals are ineffective or insufficient at the evaluated operating points, rather than that internal-state information is generally unsuitable for inference-time selection. A broader set of selection mechanisms, or a more explicit characterization of the information available in the model representations, would strengthen the paper's conclusions.
-
The practical benefits of the proposed framework remain limited, and the branching estimator is primarily diagnostic.
The downstream experiments provide only modest evidence of practical gains. Difficulty-aware allocation improves Sudoku-M accuracy by approximately 1.5–1.9 percentage points under the reported budgets, but the gains on LLaDA are only 0.7–0.8 percentage points and are not statistically distinguishable from zero. Sharing early generation prefixes reduces LLaDA accuracy by 4.0 percentage points at r⋆=0.5 and 4.7 percentage points at r⋆=0.75, with the latter result statistically distinguishable from zero under the reported interval.
Furthermore, the compute frontier in Figure 5 assumes a perfect selector. It therefore illustrates the efficiency of an idealized search strategy rather than the performance of a practical method using the internal-state signals evaluated in the paper. The actual confidence-based selector fails to approach this idealized frontier on the large-model benchmark.
The branching estimator is useful for diagnosing how much selection value may exist, but it does not itself provide a cheap online signal capable of exploiting that value. Although the paper discusses the cost of branching, the sample efficiency and total overhead required to estimate within-prompt variance reliably across models and sampling configurations remain important practical considerations.