- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper addresses the important problem of variable-length generation in diffusion large language models (dLLMs). The authors propose NaVi, a method that uses a Navigator Token and a lightweight regression probe to predict the remaining generation length. The framework combines initial length prediction, dynamic length recalibration, semantic momentum injection, and schedule-aware threshold decoding.
The paper presents positive accuracy and efficiency results across several models and benchmarks. The research problem is interesting and well motivated. However, my primary concern is that the experimental evidence is not sufficiently decisive to support the central claims, particularly the existence of transferable latent length awareness and the proposed distribution-preservation mechanism. I also identified issues involving validation methodology, statistical interpretation, algorithmic correctness, and reproducibility.
Critical strengths
-
The paper studies a relevant and well-motivated research problem. Fixed-length generation is a practical limitation of diffusion language models, and developing efficient variable-length inference strategies is valuable.
-
The Navigator Token idea is interesting. Predicting the remaining generation length from a single hidden-state representation could be useful if its generalization properties are established convincingly.
-
The authors evaluate the approach across multiple diffusion language models and several benchmarks involving mathematical reasoning, code generation, and instruction following. The reported improvements in accuracy and efficiency are encouraging.
-
The paper includes useful ablation studies, including experiments on semantic momentum injection, repetition behavior, threshold sensitivity, and probe configuration.
-
The method does not require retraining the underlying diffusion language model, although it does require offline supervised fitting of a regression probe. This makes the proposed approach potentially practical for existing models.
Critical weaknesses
1. Insufficiently decisive evidence for latent length awareness (Major).
My main concern is the evaluation of the paper's central contribution.
Appendix E.2 explains that 700 successful MATH trajectories are converted into 3,500 prefix/target examples using five truncation ratios. These examples are then randomly shuffled and divided into training and validation sets.
Under this stated protocol, different prefixes of the same original response can appear in both sets. These observations are not independent. Consequently, the reported validation R-squared may partly reflect trajectory-specific information rather than reliable length prediction on unseen problems.
Moreover, the same validation set is used to choose the transformer layer and ridge regularization coefficient, and the resulting best validation R-squared is reported. Hyperparameter selection using validation data is standard, but that selected score is not independent test evidence.
This is a major problem because transferable latent length awareness is one of the paper's central claims.
The authors should perform trajectory- and problem-disjoint splitting, select hyperparameters using training/validation data, and evaluate the final probe on an untouched test set. Initial-prompt prediction should also be reported separately from intermediate-prefix prediction.
The fitting procedure additionally selects correctly answered generations, which may not represent the general online inference distribution. The authors should examine how the probe behaves on unsuccessful and unusually long trajectories.
To be clear, this concern is not an allegation of benchmark-test contamination. The paper states that the fitting data originate from the MATH training split.
2. Statistically problematic interpretation of R-squared and inverse-log predictions (Major).
Table 5 compares R-squared values before and after applying a logarithmic transformation to the target remaining length.
However, R-squared calculated on the original length scale and R-squared calculated on the logarithmic scale are not directly comparable. A higher value after transformation does not necessarily demonstrate better prediction of the actual number of remaining tokens.
Both approaches should be evaluated on the same original token-length scale, with MAE, RMSE, and calibration statistics.
There is another important statistical issue. Under ideal squared-error regression in logarithmic space, exponentiating the conditional log expectation produces a geometric-mean-type estimate rather than the conditional arithmetic mean.
For example, consider two equally probable remaining lengths of 9 and 99. The inverse-log estimator predicts approximately 30.6 tokens, while the conditional arithmetic mean is 54 tokens.
This does not imply that logarithmic regression is inherently invalid. However, it can lead to budgets below the conditional mean. Since under-allocation can trigger additional recalibration and expansion, the paper should justify its prediction target and evaluate under-allocation explicitly.
3. The distribution-preservation claim is insufficiently supported (Major).
The paper claims that semantic momentum injection mitigates the disturbance caused by dynamic length expansion.
However, the paper does not establish mathematically that adding the scaled Navigator hidden-state vector to mask embeddings necessarily brings the resulting output distribution closer to the chosen reference distribution.
The KL experiment in Appendix D.13 is particularly concerning.
The authors state that the expanded and upfront-allocation reference runs share the same verified prefix and total generation length. Under a deterministic, stateless forward-pass formulation, a no-momentum expanded run and an upfront-allocation run with identical token IDs, embeddings, positions, attention masks, and execution state should produce identical logits, apart from numerical differences.
Nevertheless, Table 15 reports substantial nonzero KL divergence, including 7.59 for LLaDA-8B-Instruct at k=128.
This does not necessarily mean that the experiment is incorrect. It could indicate differences in model cache, decoding state, attention structure, positional treatment, or another input variable. However, these differences must be explicitly described and controlled.
Without this clarification, the experiment does not convincingly isolate the effect of expansion history or demonstrate distribution preservation.
I acknowledge that the repetition analysis and random-bias ablation provide useful empirical evidence. However, they are insufficient to establish the stronger claim that the original generation distribution is preserved.
4. Insufficient attribution of the reported improvements (Major).
NaVi simultaneously changes the initial generation budget, length recalibration, mask embeddings, and token acceptance thresholds.
The experiments do not sufficiently isolate the effect of the Navigator-based length predictor from the effects of these other decoding changes.
In particular, a fixed-length baseline using exactly NaVi's token acceptance and threshold policy is needed. An initial-prediction-only variant without recalibration or injection would also be highly informative.
A simple heuristic length controller would help distinguish the contribution of learned representations from the more general benefit of dynamically changing the generation canvas.
Therefore, although the benchmark results are positive, it remains unclear which proposed component is responsible for the headline improvements.
5. Algorithmic inconsistencies and missing termination guarantees (Major).
Algorithm 1 appears to initialize the fitting dataset using the same symbol as the input trajectory collection. Taken literally, this overwrites the input collection with an empty set before the subsequent iteration. The two datasets should be represented separately.
To the best of my understanding, Algorithm 2 also contains an inconsistent SPI calculation. The algorithm computes gamma_k but subsequently uses S_k in the threshold expression.
Moreover, the threshold is computed before possible length recalibration, even though recalibration changes the allocated budget and therefore the SPI denominator.
More importantly, the inner decoding loop does not provide an explicit progress or termination guarantee. Tokens are accepted only when their confidence exceeds the threshold. If no remaining masked position satisfies this requirement and the input state remains unchanged, the loop can continue indefinitely.
A forced-acceptance fallback, iteration limit, or other guaranteed-progress mechanism should be specified.
The outer stopping condition also needs clarification under repeated expansion.
6. Additional mathematical and reproducibility issues.
Equation (7) produces real-valued length predictions, whereas the number of allocated masks must be a nonnegative integer. The paper should specify rounding, clipping, maximum lengths, block alignment, and boundary handling.
The paper should also distinguish distribution shift from violation of the mathematical Markov property. A variable-length state process can still be Markovian.
There are numerical inconsistencies affecting reproducibility:
- Table 1 reports 101,692 seconds for the LLaDA-8B-Instruct/GSM8K fixed-length configuration achieving 77.9% accuracy, whereas Table 8 reports 10,169 seconds.
- Table 17 identifies layer 23 as optimal for LLaDA-8B-Instruct, whereas Appendix D.11 describes layer 22 as the chosen optimum.
- Table 4 reports Avg@30 with nonzero standard deviations, while Appendix E.4 describes greedy decoding without random sampling.
Finally, AIME2024 and AIME2025 each contain only 30 problems. The accuracy improvements in Table 3 correspond to approximately two and one additional correct problems, respectively. These results require careful problem-level uncertainty analysis.
Overall assessment.
I find the research problem interesting and the empirical improvements encouraging. However, the central claims are stronger than the evidence currently supports.
The non-independent probe evaluation, the insufficiently controlled distribution-shift experiment, and the unresolved algorithmic details substantially weaken my confidence in the claimed contribution.
These are central scientific issues rather than minor presentation problems. Therefore, I lean toward rejection in the current form.
Questions for authors
1. Independent evaluation of latent length awareness
Can the authors rerun the probe study with all five prefixes of each generated response, and all responses associated with the same underlying problem, assigned to a single partition?
Please select the transformer layer and ridge regularization coefficient using training/validation data, then report performance on an untouched, problem-disjoint test set.
Please report initial-prompt prediction separately from intermediate-prefix prediction, including length error in tokens, and compare against a simple predictor using prompt length, prefix length, and decoding stage.
If held-out performance drops materially, which claims about latent length awareness would the authors revise?
2. Interpretation and controls of the KL experiment
For Table 15, what are the exact token IDs, position IDs, attention masks, embedding modifications, and model cache or execution states used for the no-momentum expansion and upfront-allocation reference?
How is an identical verified prefix enforced when the reference trajectory is rerun?
Under otherwise identical inputs and a fresh deterministic forward pass, why is the KL divergence nonzero?
Please explain which variables differ between conditions or provide a revised controlled diagnostic that genuinely isolates the effect of expansion history.
3. Matched baselines and component attribution
Can the authors report a fixed-length baseline using precisely NaVi's token acceptance and threshold policy?
Please also evaluate a variant in which the Navigator predicts only the initial budget, without dynamic recalibration or semantic momentum injection.
A simple heuristic length controller and a norm-matched randomized or shuffled Navigator-vector control would further clarify which improvements arise specifically from the learned representation.
Please include all Navigator forward-pass costs in online runtime and report offline probe fitting and trajectory collection costs separately.
4. Algorithmic correctness and termination
Can the authors correct Algorithm 1's dataset notation and reconcile gamma_k and S_k in Algorithm 2?
Is SPI and the decoding threshold recalculated after a length adjustment?
What guarantees progress when no remaining masked position exceeds the confidence threshold?
Please specify the forced-acceptance fallback, iteration limit, outer-loop stopping condition, rounding and clipping of predicted lengths, and handling of incomplete decoding blocks.
5. Statistical evaluation of the length predictor
Can the authors compare the original and log-transformed regression approaches using evaluation metrics calculated on the same original token-length scale?
Please report MAE, RMSE, under-allocation frequency, and prediction calibration across different length ranges.
How does the inverse-log transformation affect length underestimation, especially for long responses?
6. Reproducibility and uncertainty
Please reconcile the runtime discrepancy between Tables 1 and 8 and the optimal-layer discrepancy between Table 17 and Appendix D.11.
Table 4 reports Avg@30 with nonzero standard deviations, whereas Appendix E.4 describes greedy decoding without random sampling. What exactly varies across the 30 evaluations, and what statistical unit is used?
For the AIME benchmarks, can the authors report paired problem-level uncertainty estimates and clarify the robustness of the reported one- or two-problem improvements?