acceptodds
Under review as a conference paper at ICLR 2027

π-XLNet: Revisiting XLNet for Text Generation

Abstract

In pursuit of efficient inference and flexible generation, text diffusion architectures have gradually moved closer to XLNet-style arbitrary-order models, motivating us to revisit XLNet's potential for training and generation. We find that, under certain conditions, set diffusion and XLNet are nearly equivalent in architecture. To systematically study this architecture, we propose the π-XLNet framework, which treats the training order distribution as a design variable, and introduce the Mallows distribution to continuously control the degree of training disorder. Experiments show that the additional loss introduced by switching evaluation order decays approximately as a power law as training disorder increases. The training cost for π-XLNet to reach the same natural-order loss as the autoregressive baseline is estimated at approximately 3.1–3.3 times that of the baseline, demonstrating its potential to reduce the training cost of masked text diffusion. Furthermore, we propose the HBD-n decoding algorithm, which uses local future information for stochastic branching and backtracking, translating arbitrary-order conditional prediction ability into generation gains through increased inference computation. The method improves the quality–diversity trade-off over natural-order decoding on a self-trained π-XLNet and raises GSM8K accuracy on a public SetDLM model from 64.67% with official SetDiff decoding to 69.14%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
XLNet, text diffusion, arbitrary-order language modeling, Mallows distribution, scaling laws, tree-search decoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rkYsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper trains an XLNet with a systematically controlled training-order distribution. They provide empirical evidence and analysis regarding the efficiency of this method over regular dLLM training. They propose a new decoding algorithm for this kind of model as well. They also provide scaling laws.

Critical strengths
  • The paper uncovers a clear link in the trajectory of language modeling research over the past 7 years, spanning XLNet to Set Diffusion models, and makes an actionable training algorithm based on these observations.
  • The paper provides good empirical insight into the efficiency of different multi-order training algorithms, showing that theirs is superior.
  • The paper shows some nice scaling laws.
Critical weaknesses
  • What is your justification for specifically choosing the Mallows distribution of orders in Section 3.1? It seems like there's a lot of distributions that could satisfy this multi-order training with a tunable knob for disorder.
  • Can you provide a diagram of what kinds of orders the Mallows distribution would select? And a diagram explaining how it selects distributions? Right now, the textual description is not so clear to me.
  • Also add an algorithm box in the main text showing the train-time order selection algorithm with Mallows distribution. You should clarify that m is decremented with each pass, and that we do this until we've selected all the tokens. You should also clarify that "select and remove" means use them to build a new sequence.
  • The abstract should frame the training convergence as an improvement over dLLMs, rather than still being >3x slower than AR.
  • Am I correct in saying that the HBD-n algorithm doesn't give the same exact distribution as regular decoding in typical usage?
Questions for authors

See weaknesses.

Reviewer_w8S6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper revisits the XLNet architecture, which was originally proposed as an arbitrary order autoregressive pretraining method for language modeling and successor to BERT. This paper revisits it for text generation, making connections with set diffusion language modeling. The paper further makes two modifications to XLNet to utilize it for text generation:

(1) Instead of using the extreme strategy of uniformly random permutations of sequences, the paper uses a Mallows Distribution which has a controllable disorder parameter. Lower values make the training closer to autoregressive training. The paper studies scaling laws and the relation between different training and eval disorder parameters.

(2) The paper proposes a hindsight branching decoding strategy, where intermediate tokens are rescored using the arbitrary-order LLM and potentially reject weak partial sequences. The paper shows that this decoding strategy improves the quality of generated text over SetDiff decoding.

Critical strengths

This paper is an interesting throwback to a popular arbitrary-order LLM architecture from 2019, and has good controlled experiments on (1) making its training more compatible for text generation by making permutation ordering less random (2) using it to derive a new decoding strategy where intermediate tokens are rescored. While the practical value is likely limited in the current form, it's refreshing to see research with very different architectural approaches compared to industry-standard autoregressive LLMs.

Critical weaknesses
  1. I'm not able to see the practical value of the work compared to standard autoregressive LLM training. From what I could infer, the method takes 2x longer to train and 4x longer to decode, and the perplexity loss is worse. Given these issues, I think compelling reasons are needed for people to consider non-linear generation methods (see the bottom of this section for a couple potential application in modern LLM engineering).

  2. The paper is also a bit far-removed from modern scale LLMs: The LMs in this work seem to be trained from scratch and fairly small scale. I would have much rather appreciated some form of "mid-training" / tailpatching of more powerful LLMs from 2025/26 with this arbitrary order loss function and some architectural modifications. This would have made the work more relevant to modern LLMs and also potentially more practical.

  3. The evaluation methodology is weak: a few conclusions are drawn solely using the MAUVE metric, and the test/baseline here utilized different decoding hyperparameters. The MAUVE metric is using GPT2-large for scoring generated text, which is a very old language model from 2019, and far weaker than LLM backbones being used to generate the text (Qwen 2.5). Why not instead use modern LLM judges to test the quality of the generated text in terms of fluency, coherence, and factuality? A human evaluation would also be helpful.

  4. The paper is hard to read because it assumes a lot of background knowledge from the readers. The paper expects readers are very familiar with set diffusion, XLNet, and self-speculative decoding (set diffusion seems to be a new paper < 3 months old and has not caught up much in the literature). A lot of crucial background text has been moved to the Appendix.

  5. Minor, but this paper lacked a bit in focus and felt a bit like 2 separate short papers stitched together. I think the paper would be more compelling by removing emphasis on diffusion LLMs and focusing solely on the XLNet permutation ordering and more downstream applications of arbitrary-order LLM scorers (as a concrete example, can this effectively backfill intermediate agentic turns or missing thoughts while creating distillation/SFT data?).

Questions for authors

N/A, see weaknesses section

Reviewer_niaxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits XLNet-style arbitrary-order language modeling and investigates its relationship with recent diffusion language models. The authors identify a structural correspondence between XLNet's two-stream attention architecture and a specific configuration of Set Diffusion, and propose π-XLNet, which treats the training-order distribution as an explicit design variable. The Mallows distribution is introduced to control the degree of training-order disorder.

The paper further studies how training disorder affects prediction loss, scaling behavior, and training efficiency. In addition, the authors propose Hindsight Branching Decoding (HBD-n), a decoding algorithm that leverages local future context through stochastic branching and backtracking. Experimental results show improvements in generation quality and mathematical reasoning performance, although at the cost of additional inference computation.

Critical strengths

Interesting perspective. The paper provides an interesting connection between arbitrary-order language modeling and diffusion language models, particularly the structural relationship between XLNet and Set Diffusion.

Systematic empirical analysis. The authors investigate the effects of different training-order distributions on prediction loss, cross-order generalization, and scaling behavior. The observed relationship between training disorder and evaluation-order loss increments is an interesting empirical finding.

Promising decoding results. The proposed HBD-n algorithm demonstrates improvements in generation quality and reasoning accuracy. These results suggest that leveraging arbitrary-order conditional prediction and local future information may provide additional benefits during inference.

Critical weaknesses

W1. The proposed framework mainly extends existing XLNet and Set Diffusion architectures, with limited methodological novelty.

The core architecture of π-XLNet largely follows the two-stream design of XLNet and a specific configuration of Set Diffusion. Arbitrary-order language modeling and the use of different token prediction orders have already been extensively investigated in previous studies.

The main methodological difference appears to be the introduction of the Mallows distribution to control training-order disorder, together with a reformulation of the relationship between XLNet and Set Diffusion.

However, Set Diffusion has already explored flexible order distributions and the interpolation between autoregressive and diffusion-based generation. The paper does not sufficiently establish why the proposed Mallows distribution offers fundamental advantages over existing approaches.

Therefore, the incremental technical contribution remains somewhat limited. More direct comparisons with existing arbitrary-order modeling approaches, particularly Set Diffusion, would help clarify the novelty and necessity of the proposed framework.

W2. HBD-n is not sufficiently compared with existing test-time scaling and iterative refinement approaches.

The proposed HBD-n algorithm aims to improve generation quality by exploiting local future information through stochastic branching and backtracking. However, this objective is closely related to existing test-time scaling, search-based decoding, and iterative refinement approaches, which also improve generation quality by spending additional inference computation.

The paper mainly compares HBD-n with natural-order autoregressive decoding and official SetDiff decoding, without sufficiently evaluating representative alternatives such as Best-of-N sampling, beam search, self-consistency, or remasking-based iterative refinement.

This limitation is particularly important because HBD-n introduces substantial additional inference overhead. Without comparisons under matched computational budgets, it remains unclear whether the proposed algorithm provides a better quality–efficiency trade-off than existing inference-time scaling techniques.

W3. The claimed training-efficiency advantage is not sufficiently supported by controlled experiments.

The paper estimates that π-XLNet requires approximately 3.1–3.3 times the training cost of an autoregressive baseline to achieve comparable natural-order prediction loss. This is contrasted with the substantially higher training costs reported for masked diffusion models in previous studies.

However, these comparisons are based on different model architectures, training configurations, evaluation objectives, and computational cost measurements. Furthermore, the reported 3.1–3.3× training-cost multiplier is estimated from empirical scaling fits rather than directly measured under matched target-quality conditions.

Consequently, the current results do not convincingly establish that π-XLNet provides a substantial training-efficiency advantage over existing diffusion-based approaches. Direct experiments with representative SetDLM or masked diffusion baselines under comparable training budgets would strengthen this claim.

W4. The evaluation of HBD-n does not fully isolate the contribution of the proposed decoding algorithm.

The reported generation improvements are encouraging, but some experimental comparisons employ different sampling configurations.

For example, in the GSM8K experiment, the official SetDiff baseline uses deterministic decoding, whereas HBD-n employs stochastic sampling with nonzero temperature and top-p truncation. Similarly, the MAUVE comparison uses different top-p values for the autoregressive baseline and HBD-n.

Therefore, the observed improvements cannot be attributed solely to the proposed branching and backtracking mechanism.

A more controlled evaluation using matched sampling parameters, together with quality–computation trade-off curves, would be necessary to establish the effectiveness of HBD-n.

W5. The relationship between the training-order framework and the proposed decoding algorithm is not sufficiently established.

The paper presents Mallows-based training and HBD-n decoding as two major contributions. However, HBD-n does not inherently depend on the proposed Mallows distribution and is also evaluated using publicly available SetDLM models.

The experiments do not sufficiently investigate how different training-order distributions affect HBD-n performance or whether Mallows-based training specifically benefits the proposed decoding strategy.

As a result, the training and inference contributions appear relatively independent rather than forming a tightly integrated framework. Additional experiments examining their interaction would help establish a more coherent methodological contribution.

Questions for authors

Novelty and related work: What are the key methodological differences between π-XLNet and existing XLNet-style arbitrary-order models or Set Diffusion? Can the authors demonstrate the advantages of Mallows order sampling over existing order-distribution strategies under matched training settings?

Comparison with test-time scaling: How does HBD-n compare with representative inference-time scaling approaches, such as Best-of-N sampling, self-consistency, beam search, and iterative refinement, under comparable computational budgets?

Training efficiency: Can the authors provide direct training-efficiency comparisons with SetDLM or other masked diffusion models, rather than relying primarily on cross-study cost estimates?

Fairness of decoding experiments: Why are different sampling parameters used for HBD-n and the baselines in the GSM8K and MAUVE experiments? Can the authors provide controlled experiments with identical temperature, top-p, and stopping configurations?

Training–decoding interaction: How does the training disorder parameter affect HBD-n's performance? Does Mallows-based training provide additional advantages for HBD-n compared with other arbitrary-order training distributions?

Generalization of the empirical findings: Does the observed power-law relationship between training disorder and evaluation-order loss increments hold across different model sizes, training budgets, and random seeds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.