- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper jointly trains an autoregressive encoder and a parallel decoder, using random auxiliary variables to coordinate parallel tokens and reducing a variational objective to AR cross-entropy plus parallel reconstruction cross-entropy. Relative to PTP, the main contributions are joint training with parallel construction of auxiliary variables, geometric encoding, and validation at larger scales through language pretraining and post-training.
Critical strengths
-
Joint training addresses the sequential bottleneck in constructing auxiliary variables when training the prior approach directly from data, giving it a concrete methodological value. Geometric encoding and representation alignment also have direct ablations supporting improved reconstruction capability in the tested configurations.
-
The authors complete pretraining and post-training experiments up to 3B parameters and measure actual inference benefits. At 1.3B, an independently trained AR baseline uses the same data and post-training recipe, supporting a useful quality–speed trade-off under this low-budget training setup. Differences in absolute scores from mature models alone do not invalidate this result.
Critical weaknesses
-
Key controls are missing for determining whether the extra joint-training investment is worthwhile. Training the 1.3B PLM costs approximately 1.89–3.55 times as much as AR training, so a comparison at the same token budget does not establish superiority at the same total compute. The largest 3B model uses its own AR decoding mode as the reference and lacks an independently trained pure AR baseline. Faster decoding within the same model does not directly establish an advantage of the training method over pure AR training.
-
Fair comparisons with the closest alternatives are insufficient. PTP already includes random auxiliary variables, training directly from data, and self-verification; the quantile ablation does not replace a comparison with the original PTP training procedure. A unified quality–throughput comparison with Fast-dLLM v2 is also missing. Although AR-to-PLM conversion is tested, it reports only NLL and reconstruction-prefix length, rather than comparing the practical benefits of the two conversion routes from a common backbone. Inconsistent descriptions of timing hardware and time aggregation also require clarification for each table.
Overall assessment: The joint-training approach and validation at larger scales are valuable; low absolute task scores under a limited training budget are not my main concern. The key gaps are the benefit at matched compute, an independent 3B AR baseline, and a fair Fast-dLLM v2 comparison. I lean toward weak rejection, and controlled evidence addressing these questions could materially change my assessment.
Questions for authors
-
At the same total training FLOPs or cumulative GPU-hours on the same hardware, does PLM still outperform a sufficiently trained AR model? Please account consistently for pretraining and post-training costs and report quality versus training cost and quality versus throughput.
-
Could you provide an independently trained 3B pure AR baseline with 300B training tokens and matched SFT data, preference data, and model-selection rules? Otherwise, please limit the corresponding claim to decoding trade-offs within the same PLM.
-
Starting from the same public AR checkpoint, could you fairly compare PLM and Fast-dLLM v2 with matched conversion data and additional compute? Please match post-training, hardware, tasks, and timing; report quality for the same configuration at each speed point; and state whether the acceptance rule changes the sampling distribution.
-
Can the benefit of the new training mechanism over the original PTP be quantified directly? Please also clarify the timing protocol for each table and provide a verifiable anonymous implementation, configurations, and checkpoints for checking the key results.