acceptodds
Under review as a conference paper at ICLR 2027

Parallel Language Models via Variational Decoding

Abstract

Autoregressive language models emit one token per forward pass. This causes high inference latency, in particular as models are scaled up for quality. In this paper, we introduce Parallel Language Models (PLMs), which generate mutually consistent tokens in one forward pass by routing dependencies through latent auxiliary variables. PLMs are trained using a variational bound that reduces to an autoregressive loss plus a parallel decoding loss. Empirically, PLMs are faster and better than autoregressive and discrete diffusion baselines. Pretrained PLMs achieve near-autoregressive pretraining loss on model scales of 80M to 3B parameters, and speedup follows a power law in model size. When post-trained, our 1.3B model matches or exceeds an identically trained autoregressive model on math, coding, and instruction-following benchmarks, speeding up inference 2.3-5.7x. Together, we show that parallel generation can be trained from a single training objective at autoregressive quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, fast inference, multi token prediction, parallel token prediction, generative model, variational methods

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RTiPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper jointly trains an autoregressive encoder and a parallel decoder, using random auxiliary variables to coordinate parallel tokens and reducing a variational objective to AR cross-entropy plus parallel reconstruction cross-entropy. Relative to PTP, the main contributions are joint training with parallel construction of auxiliary variables, geometric encoding, and validation at larger scales through language pretraining and post-training.

Critical strengths
  1. Joint training addresses the sequential bottleneck in constructing auxiliary variables when training the prior approach directly from data, giving it a concrete methodological value. Geometric encoding and representation alignment also have direct ablations supporting improved reconstruction capability in the tested configurations.

  2. The authors complete pretraining and post-training experiments up to 3B parameters and measure actual inference benefits. At 1.3B, an independently trained AR baseline uses the same data and post-training recipe, supporting a useful quality–speed trade-off under this low-budget training setup. Differences in absolute scores from mature models alone do not invalidate this result.

Critical weaknesses
  1. Key controls are missing for determining whether the extra joint-training investment is worthwhile. Training the 1.3B PLM costs approximately 1.89–3.55 times as much as AR training, so a comparison at the same token budget does not establish superiority at the same total compute. The largest 3B model uses its own AR decoding mode as the reference and lacks an independently trained pure AR baseline. Faster decoding within the same model does not directly establish an advantage of the training method over pure AR training.

  2. Fair comparisons with the closest alternatives are insufficient. PTP already includes random auxiliary variables, training directly from data, and self-verification; the quantile ablation does not replace a comparison with the original PTP training procedure. A unified quality–throughput comparison with Fast-dLLM v2 is also missing. Although AR-to-PLM conversion is tested, it reports only NLL and reconstruction-prefix length, rather than comparing the practical benefits of the two conversion routes from a common backbone. Inconsistent descriptions of timing hardware and time aggregation also require clarification for each table.

Overall assessment: The joint-training approach and validation at larger scales are valuable; low absolute task scores under a limited training budget are not my main concern. The key gaps are the benefit at matched compute, an independent 3B AR baseline, and a fair Fast-dLLM v2 comparison. I lean toward weak rejection, and controlled evidence addressing these questions could materially change my assessment.

Questions for authors
  1. At the same total training FLOPs or cumulative GPU-hours on the same hardware, does PLM still outperform a sufficiently trained AR model? Please account consistently for pretraining and post-training costs and report quality versus training cost and quality versus throughput.

  2. Could you provide an independently trained 3B pure AR baseline with 300B training tokens and matched SFT data, preference data, and model-selection rules? Otherwise, please limit the corresponding claim to decoding trade-offs within the same PLM.

  3. Starting from the same public AR checkpoint, could you fairly compare PLM and Fast-dLLM v2 with matched conversion data and additional compute? Please match post-training, hardware, tasks, and timing; report quality for the same configuration at each speed point; and state whether the acceptance rule changes the sampling distribution.

  4. Can the benefit of the new training mechanism over the original PTP be quantified directly? Please also clarify the timing protocol for each table and provide a verifiable anonymous implementation, configurations, and checkpoints for checking the key results.

Reviewer_BePrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Parallel Language Models, which train an autoregressive encoder that maps each token to an auxiliary variable and a decoder that reconstructs a block of tokens in parallel from those auxiliaries, using one variational bound that reduces to the usual autoregressive loss plus a decoder reconstruction loss. Auxiliaries are assigned to tokens by a weighted Voronoi partition in embedding space, a representation-alignment loss speeds up training, and inference uses Quadratic Decoding with exact or relaxed acceptance. Models from 80M to 3B parameters reach near-autoregressive encoder loss, and a fitted scaling law says the number of reconstructed tokens grows with size. After SFT and DPO, the 1.3B model is reported to match an autoregressive baseline at 2.3 to 5.7 times the speed.

Critical strengths
  • The recipe is ablated step by step at 150M: no auxiliaries give 1.443 reconstructed tokens (R), quantile auxiliaries 1.705, Voronoi 1.785, and Voronoi with alignment 1.850, at AR NLL 2.829 versus 2.788 for the AR model.
  • Representation alignment reaches the same R after 37% to 38% of the steps.
  • The loss gap to autoregressive models is small and shrinks slightly with size at stride 32: 0.042 (80M), 0.041 (150M), 0.031 (350M), 0.029 (1.3B).
  • The pretraining comparison is broad: 14 public autoregressive models and five diffusion or flow models, with 87,958 judged matches, bootstrap intervals and judging in both orders (Appendix E, Figure 1).
Critical weaknesses
  1. The post-training claim that PLMs match or exceed autoregressive quality is not statistically supported and is partly contradicted by the tables. The text says both the 1.3B and the 3B model match or exceed autoregressive quality, but no AR model was post-trained at 3B, and the 3B baseline in Table 1 is the PLM itself in AR mode (Table 1 footnote). At 1.3B the differences are small against the noise: the GSM8K 95% interval is about 2.5 points, HumanEval has 164 problems (Table 10), all post-training comparisons use one seed, and the checkpoint is chosen by GSM8K accuracy, which the authors call a development metric. Decoding modes of the same PLM that should give the same output distribution differ a lot: on HumanEval the beta=1 model scores 8.5 in AR mode and 10.4 under exact decoding, the beta=0.5 model scores 19.5 in AR mode and 16.5 under exact decoding, and on GSM8K the beta=0.5 model moves from 27.0 (AR mode) to 31.5 (exact) (Table 11); this is sampling noise at the size of the claimed gains. Several cells are below the baseline: IFEval is 45.1 and 45.7 against 48.2, MATH-500 is 7.2 against 7.4 and HumanEval is 10.4 against 12.8 for beta=1 (Table 1).

  2. The baselines do not test the claimed advantage, and the training cost is kept out of the main text. Nothing in Section 4 compares against speculative decoding, Medusa, Hydra or EAGLE on an AR model of the same quality, against multi-token-prediction models, against a distilled PTP model, or against block diffusion, although Related Work describes them. Figure 1 emphasizes that PLMs saw 25B to 300B tokens against trillions, but PLM training costs 1.89x, 2.48x and 3.55x the GPU-hours of the AR model at 1.3B for beta=2, 1 and 0.5 (Table 3), and this appears only in Appendix C. The post-trained comparison uses a single AR baseline at 1.3B.

Questions for authors

N/A

Reviewer_mkgJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, authors have proposed a novel parallel language model (PLM), for parallel token prediction via leveraging a variational formulation. It generate mutually consistent tokens in one forward pass by routing dependencies through latent auxiliary variables. During the training, PLM uses a variational bound to minimize the autoregressive (AR) loss and parallel decoding loss. Ultimately, the pretrained PLM achieves faster and better Pareto frontier compared to the AR and discrete diffusion baselines.

Critical strengths
  1. proposed an effective method for parallel token prediction. The PLM contains a jointly trained encoder and decoder, where encoders maps the inputs to auxiliaries and decoders converts auxiliaries to output sequence in parallel.

  2. Experiments scale up to 3B to demonstrate the effectiveness of PLM. The model achieves 2.3 to 5.7 times faster generation while maintaining the generation quality.

Critical weaknesses
  1. in figure 5, during the pretraining stage, we can still see a clear loss gap between PLM and AR baseline. Based on the current pretraining scale, we cannot tell the gap will be enlarged or reduced with growing model size.

  2. The speedup evaluation method is confusing. In appendix F.3, authors claim the speed is tested on GH200; in appendix H3, it says using RTX 6000 and E.3 says using vLLM. Authors should using a fixed system setup for evaluation consistency. Moreover, batching is important for model serving. However, current batch size is 1, the advantage of PLM on large scale, long context window is not proved.

  3. Lacking the comparison with competing inference acceleration method. In current configuration, the PLM just compare with vanilla AR baseline. However, as PLM still adopts parallel proposal, AR verification, and together with quadratic decoding, more in-depth comparison with SOTA works should be included.

  4. For the post-training experiments, current evaluations are performed on GSM8K, math-500, human-eval and IFeval. It lacks evaluations on a broader range of benchmarks, rather than just math, coding and instruction following. Moreover, the ablation study setup is not consistent. For 1.3B, AR baseline is independently trained, but for 3B, there lacks a independently trained AR.

Questions for authors

please check the weakness above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.