acceptodds
Under review as a conference paper at ICLR 2027

Rethinking the Roles of Diffusion and Decoding: Predictive-State Flows for Parallel Language Generation

Abstract

In continuous diffusion language models (DLMs), latents generated by the diffusion prior are mapped almost deterministically to tokens, either by a decoder or through a token embedding matrix. Unlike in image generation, where visual details can vary continuously, the diffusion prior must capture fine-grained discrete token choices in continuous space, which can make the joint latent distribution difficult to model. We argue that generative roles should instead be divided: the diffusion prior should generate high-level predictive information, while the decoder should resolve the remaining token-level choices. Based on this view, we introduce Predictive-State Flows (PSF), which partitions a sequence into blocks and associates each block with a predictive state computed solely from the preceding context. A rectified-flow prior generates the sequence of predictive states, after which a block-causal decoder generates all blocks in parallel, autoregressively only within each block. This design reduces diffusion from L token-level states to K ≪ L predictive states and reduces token-level sequential depth to L/K. On unconditional OpenWebText generation, a 197M-parameter model with K=8 achieves a generative perplexity of 22.8 at entropy 5.16 using 0.31 TFLOPs per 1024-token document, compared with 24.0 and 7.2 TFLOPs for ELF-B with 32 sampling steps. The corresponding measured operating points yield MAUVE@512 of 0.93 and 0.73, respectively. Controlled comparisons further show that content states, which additionally encode their own blocks, yield 8–16% higher generative perplexity than predictive states, and that replacing sampled predictive-state trajectories with a shared mean lowers MAUVE@512 by 0.11. These results suggest that latent design for continuous DLMs should consider not only how compactly text is represented, but also how generative roles are divided between the prior and the decoder.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language models, flow matching, latent diffusion, parallel decoding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Mtv7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces the Predictive State Flows (PSF) which decouples predictive information from the token generation. It uses rectified flow to generate the predictive states representing the context before each block, then generates the blocks in parallel with the autoregressive decoding within each. As a result, it reduces the number of FLOPs while achieving competitive perplexity.

Critical strengths

The paper proposes an interesting combination of diffusion and autoregressive decoding that allows for multiple blocks to be generated in parallel. The experiments include useful ablations on block length, landing targets, and realization codes to inform how the different components influence generation.

Critical weaknesses

1- While the paper claims that predictive states are easier for the diffusion prior to model than content states, the experiments do not show this hypothesis properly. The content state baseline reuses a model trained for predictive states, and the performance advantage remains even when the prior is bypassed (Table 3). This suggests that the improvement could be due to either the prior or the decoder. The comparison with ELF-B is also not balanced: PSF uses more parameters and training compute, its perplexity advantage is relatively small, and its throughput advantage depends on the batch size (Table 1). More controlled experiments are needed to support the main claims.

2- Also, novelty and positioning of the paper is unclear (described in Q3)

Questions for authors

1- The role of "realization codes" is unclear and contradictory, as removing them improves perplexity but hurts MAUVE, diversity, and topic consistency (Table 1, Figure 11). Could the authors please elaborate on this trade-off and provide a more balanced evaluation about whether predictive states cause better generation on average?

2- PSF demonstrates weaker topic consistency and coherence than several baselines despite its perplexity improvements (Figure 11, Table 18). Could the authors test stronger cross-block conditioning and demonstrate whether it improves coherence without sacrificing the efficiency gains of a block decoder? It is more of a scope question.

3- The novelty of the method compared to earlier work is unclear. Zhu et al. (2025) already use diffusion-generated segment representations with autoregressive decoding. Diffusion guided language modeling has also been explored by Lovelace et al. (2024) and latent planning approaches have been proposed for text generation by Wang et al. (2022). Could the authors clarify the differences between their approach and these earlier ones or offer a comparison?

Zhu et al. "Segment-Level Diffusion: A Framework for Controllable Long-Form Generation with Diffusion Language Models." ACL, 2025. Lovelace et al. "Diffusion Guided Language Modeling." Findings of ACL, 2024. Wang et al. "Language Modeling via Stochastic Processes." ICLR, 2022.

5 comments

Thank you for your careful review.

W1: The content-state control and prior versus decoder. To address the concern about the asymmetric content-state control in Table 3, we trained both models from scratch at K=8 with their own encoders and decoders, using identical architectures and training budgets (General Response, R1; new Appendix F.1). At matched entropy, the predictive model achieves substantially lower perplexity (26.0 versus 41.7), and the 32B judge prefers its whole documents by 32:10. The gap also persists under true-plan decoding, confirming a decoder-side contribution. To evaluate the prior separately, we measured its performance without the decoder on held-out text. The predictive prior achieves much lower normalized denoising error at a 50% noise level (14% versus 41%) and a smaller Fréchet distance between generated and real states (11.6 versus 40.3). These results support both advantages: predictive states are easier for the prior to model, and the decoder generates better text from them. The revision now reports these effects separately.

W1 (continued): Balanced comparison with ELF-B. We trained PSF-S at ELF-B's model size (104M versus 105M generation parameters) using fewer training tokens (31B versus 46B; General Response, R3; new Appendix F.3). At entropy 5.16, PSF-S achieves 22.8 perplexity versus 24.0 for 32-step ELF-B, and 0.86 versus 0.73 MAUVE@512, with 46× fewer inference FLOPs. Thus, the advantage holds at matched model size with a smaller training budget. PSF-S is now included in Table 1. For our 197M model, the perplexity advantage reaches 21% on length-controlled 512-token windows (Table 15), with MAUVE@512 of 0.93 versus 0.73. The 23× reduction in FLOPs per document holds across batch sizes, while the throughput advantage becomes apparent at larger batches (62.9 versus 49.8 documents/s at batch 256). The revision distinguishes these two efficiency measures.

W2 / Q3: Novelty relative to Zhu et al. (2025), Lovelace et al. (2024), and Wang et al. (2022). We now discuss these works in Related Work and provide a detailed comparison in Appendix F.4 (Table 22). Our core contribution is not the combination of diffusion latents and autoregressive decoding, but the choice of what the diffusion prior should generate. Unlike content latents that encode the text they represent, PSF uses predictive states computed strictly from preceding context, leaving token realization to the decoder. Our symmetric control shows why this distinction matters: content states have nearly three times the prior denoising error and 60% higher perplexity under identical training. PSF realizes this principle through a rectified-flow prior over predictive states and parallel block decoding with one-token landing targets. The three cited works use different latent targets and decoding schemes, as detailed in Table 22.

Q1: Realization codes: contradiction or trade-off? This reflects a trade-off between two operating points of the same model. Realization codes are 64-dimensional, KL-limited side inputs that carry a small amount of information about each block. Generating these codes helps coordinate wording across blocks (6.9% topic-switch rate, 0.93 MAUVE), whereas setting them to zero lowers perplexity by 6–9% but reduces MAUVE by 0.05 and raises the topic-switch rate to 12–13%. Regarding whether predictive states improve generation quality overall, our symmetric control shows that they outperform content states in perplexity, MAUVE, and judged coherence simultaneously.

Q2: Stronger cross-block conditioning. We evaluated two variants that condition blocks on realized text (new Appendix F.2): a refinement round that re-encodes the generated document (+0.22 TFLOPs), and sequential block decoding, where each block reads the realized text of preceding blocks (8× serial depth). Neither improves judged whole-document coherence over the single-pass model (12:14 and 16:23 order-consistent pairs). In these tests, stronger cross-block conditioning did not improve coherence, despite its additional cost. For topic consistency, the lexical coupling in Appendix D.5 improves lexical topic persistence to the level of real text without additional FLOPs, at the perplexity cost reported there.

Dear Reviewer,

As the discussion period approaches its end, we hope our responses and additional experimental results have addressed your concerns. We welcome any further discussion, as time allows.

Reviewer_Mtv7OpenReview ↗

Thanks

Thank you for the response. The added controls and clarifications help, but I still find the novelty over prior latent diffusion and planning methods limited. Inconsistencies in the reported results and training descriptions remain, and the evidence for comparable coherence is still not strong enough. In my assessment, these concerns leave the contribution below the standard I expect at ICLR. I therefore maintain 2 (weak rejection).

Reviewer_sBC2OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PSF, which first generates a short sequence of predictive states and then decodes text blocks in parallel, autoregressively within each block. Each state is learned from preceding context, and a one-token target connects neighboring blocks. OpenWebText experiments show substantial FLOP savings over ELF and improvements in generative perplexity and MAUVE@512 at selected settings. The ablations support the design, although the content-state control inherits predictive training and document coherence remains a limitation.

Critical strengths
  1. Generating a short sequence of states and leaving token choices to the decoder is an interesting approach. The plain PSF model at K=8K=8 uses about 23× fewer FLOPs than 32-step ELF-B, with slightly lower generative perplexity and substantially higher MAUVE@512.

  2. The same-data AR baseline, throughput measurements and coherence evaluations help distinguish computational savings from improvements in generation quality. The authors clearly report the limitations of their strongest perplexity results.

Critical weaknesses
  1. What causes the predictive-state advantage? The content-state control retains an encoder learned under predictive training and starts from a predictive-state decoder. Matching the final adaptation budget does not remove this asymmetry. Although the authors acknowledge this limitation, it restricts how strongly the comparison supports the broader representation principle. The improvement is valid for this setup, but it remains unclear whether it comes from excluding the current block or from training that favors this choice. The gap also persists when the prior is bypassed, so it does not establish that predictive states are easier for the flow model to generate.
  2. How much computation is saved at comparable document quality? The headline comparisons use entropy 5.16, inherited from ELF’s 32-step samples, versus 5.41 for held-out text. This is a valid comparison point, but the relative advantages depend on the operating range. Larger ELF models can overtake the plain PSF samplers in generative perplexity at higher entropy, although at substantially greater cost. For example, 64-step ELF-L uses about 88× the FLOPs of PSF-L at K=8K=8. Within PSF at K=8K=8, removing realization codes improves perplexity but reduces MAUVE and lexical topic persistence. The small states-only configuration with the 4-gram ban also loses the tested coherence comparisons, and the plain sampler loses to 32-step ELF-B on the first 300 words under the 32B judge. The judges are often order-sensitive, but these results do not yet establish the magnitude of the computational savings at comparable document coherence.
Questions for authors
  1. How does the predictive-versus-content gap change when the content encoder and decoder are trained for their own objective at comparable training cost?
  2. Do the small changes from increasing prior sampling steps persist at entropy 5.41, where the perplexity gap to held-out text is larger?
  3. Which PSF settings give the best document coherence, and how do their quality and cost compare with ELF and the same-data AR?
4 comments

Thank you for your close reading of our controls and evaluation.

W1 / Q1: What causes the predictive-state advantage? A symmetric from-scratch comparison. As you point out, the content-state control in Table 3 inherits an encoder trained for predictive states and starts from a predictive-state decoder. To address this, we trained both models from scratch at K=8, each with its own encoder, decoder, and objective, while keeping the architecture, data, training steps, and prior training identical (new Appendix F.1). The gap widens rather than closes.

Trained from scratch, K=8Perplexity (entropy 5.16) ↓MAUVE@512 ↑Judged coherence (pairs won) ↑Prior denoising error ↓Prior Fréchet distance ↓
Predictive states26.00.913214%11.6
Content states41.70.851041%40.3

(1) Predictive states generate better text. The content model's perplexity is 60–85% higher across matched entropy levels from 5.16 to 5.37, compared to 8–16% in Table 3. The 32B judge also prefers the predictive model in all evaluation windows (54:25 for words 1–300, 61:19 for words 300–600, and 32:10 for whole documents). The advantage thus persists when each representation is trained independently, without shared adaptation that favors predictive states.

(2) Predictive states are easier for the flow prior to model. We agree that Table 3 alone cannot establish this, since the quality gap persists even when the prior is bypassed. We therefore evaluated both priors independently of their decoders, using the same architecture, training steps, and state scale. The predictive prior achieves substantially lower normalized denoising error at a 50% noise level (14% versus 41%) and a smaller Fréchet distance between generated and real states (11.6 versus 40.3; real versus real: 3.5). One key difference is that content states must also encode the wording of their own blocks, which PSF leaves to the decoder. Accordingly, content states have a lower teacher-forced loss (1.37 versus 3.25 nats per token), a higher effective dimensionality (299 versus 90), and weaker linear predictability between consecutive states (5% versus 37%).

The revision reports the prior-side and decoder-side effects separately, makes this experiment our primary evidence, and retains Table 3 as the shared-adaptation variant.

W2: Compute savings at comparable document quality, operating range, and realization codes.

Savings at comparable coherence. We re-evaluated our default K=8 setting against 32-step ELF-B using the same 32B judge on 200 document pairs in both presentation orders (new Appendix F.2). The judge shows no clear preference in whole-document coherence (12:16 order-consistent pairs; 10:13 in Table 18) and favors PSF on repetition (20:5). ELF-B's advantage is mainly in the first 300 words. At comparable whole-document coherence, PSF uses 0.31 versus 7.2 TFLOPs per document (23× fewer) and achieves higher MAUVE@512 (0.93 versus 0.73).

Operating range. PSF maintains its perplexity advantage over 32-step ELF-B at higher entropy: about 45 versus 67 at the held-out entropy of 5.41. The larger 64-step ELF-M and ELF-L are ahead at their own higher-entropy operating points, but at substantially higher cost: ELF-M uses 165× the FLOPs of PSF and ELF-L 88× those of PSF-L, while PSF-L remains within 6% of ELF-L in perplexity at ELF-L's operating point. Section 3.1 now states this range.

Realization codes. Setting the codes to zero at K=8 reduces perplexity by 6–9%, but lowers MAUVE by 0.05 and doubles the topic-switch rate. Table 1 now distinguishes the distribution-matching and lowest-perplexity operating points, and Section 3.1 explains our selection criteria (General Response, R4).

Q2: Prior sampling steps at entropy 5.41. The changes remain small. With 32, 50, and 64 prior steps, PSF's perplexity is 46.4, 45.9, and 45.8 at entropy 5.41 (a spread of 1.3%). The largest spread across our four K=8 configurations is 2.5% (PSF-L). Appendix B.4 now reports results at both entropy levels.

Q3: Which PSF settings give the best document coherence, and at what quality and cost? Our default K=8 setting (22.8 perplexity, 0.93 MAUVE, 0.31 TFLOPs) gives the best document coherence, as detailed in W2. The states-only setting achieves the lowest perplexity (19.0) and is preferred over ELF-B on repetition (27:4), but sacrifices whole-document coherence. Additional inference compute does not improve judged coherence: neither a refinement round nor sequential block decoding outperforms the single-pass model. Compared with the same-data AR baseline, PSF's main advantage is inference efficiency (178 versus 1024 serial network evaluations; 36.7 versus 4.6 documents/s at batch 32, Table 14), while AR remains the quality reference (Table 1).

Presentation. Section 2.4 now introduces the three-stage training recipe first and clarifies what realization codes are and when they are used.

Dear Reviewer,

As the discussion period approaches its end, we hope our responses and additional experimental results have addressed your concerns. We welcome any further discussion, as time allows.

Thank you for the additional experiments. The from-scratch comparison substantially addresses my concern about the content-state control. The whole-document coherence result remains inconclusive, so I am not yet convinced that the computational savings come at comparable whole-document coherence. Overall, the stronger representation evidence and substantial computational savings shift my assessment to 3 (weak accept).

Thank you for your careful consideration and for recognizing our additional experimental results. We appreciate your constructive feedback and the time you spent reviewing our work.

Reviewer_rCRyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies continuous diffusion language models. In particular, it focuses on the relation between the diffusion prior and the decoder: the authors propose a framework in which the former generates high-level predictive information, while the latter focuses on the token-level choices. To this end, they introduce Predictive-State Flows (PSF): the sequence is partitioned into K blocks, and each block is associated with a predictive state computed only from the preceding blocks. A rectified-flow prior generates the sequence of predictive states; a block-causal decoder then generates all blocks in parallel, autoregressively within each block, with a one-token landing target coupling adjacent blocks. On unconditional OpenWebText generation, a 197M-parameter model with K = 8 reaches a generative perplexity of 22.8 at entropy 5.16. Controlled ablations show that predictive states outperform content states by 8–16%.

Critical strengths

The paper addresses a clear question (which information the prior should generate and which should be left to the decoder) and supports its answer. The reduction in FLOPs and in token-level sequential depth is a concrete finding, although the throughput advantage appears only at large batch sizes. Finally, the evaluation is honest.

Critical weaknesses

The training procedure is very complex: three stages, realization codes, source labels, a chain over K=64→8K=64\to 8 and a separate prior for each KK. This makes it hard to attribute the results to a specific component and to reproduce them.

Moreover, the headline metric is generative perplexity interpolated at matched entropy with a sharpened sampler. However, coherence and topic persistence go in the opposite direction: the states-only configuration, which gives the lowest perplexity, loses to ELF-B, BD3-LM and the same-data AR in the pairwise LLM-judged comparisons, and is also less diverse (Self-BLEU) with weaker topic persistence.

Questions for authors

Why is K=8K=8 the best configuration? Fewer seams, more training along the chain, or something else? Since the chain does not isolate block length at equal training budget, would a budget-controlled ablation for each KK be feasible?

Moreover, since LLM-judged coherence gets worse at the lowest perplexity, which metric should be used to select the configuration? Would it be better to highlight MAUVE and coherence as headline results together with perplexity?

2 comments

Thank you for your constructive review and clear questions.

W1: Training complexity, attribution, and reproducibility. The additional training complexity is optional. During the rebuttal, we trained PSF at K=8 from scratch using a simple three-stage recipe (training an encoder and block decoder, a single self-realized-prefix stage, and training a prior over the states), without the K=64→8 chain or realization codes. With only a quarter of the full model's training tokens, it achieves a perplexity of 23.3 at entropy 5.16 under nucleus sampling with a 4-gram ban, outperforming ELF-B at 24.0 in its best 32-step setting (General Response, R4). The revision now presents this simpler recipe as the core method (Section 2.4), with the progressive chain and realization codes as optional extensions.

For attribution, Table 11 incrementally ablates each component at K=64, Table 12 retrains the K=8 model without the landing target or realization codes, and Appendix B.3 shows that the progressive chain only reduces wall-clock training time. For reproducibility, the supplementary zip includes the training scripts, sampling settings for every row in Table 1, and evaluation code. Checkpoints will be released with the camera-ready paper.

W2: The lowest-perplexity configuration loses on coherence, diversity, and topic persistence. We agree with the reviewer's concern that lower generative perplexity does not necessarily imply better overall generation quality. The revision therefore presents the two K=8 settings as distinct operating points of the same model rather than ranking them by perplexity alone. Our default setting (22.8 perplexity) prioritizes distribution matching, achieving a MAUVE@512 of 0.93 compared to 0.73 for 32-step ELF-B. It also maintains comparable cross-sample diversity (Self-BLEU of 0.30 versus 0.29) and comparable judged whole-document coherence, while using 23× fewer FLOPs (General Response, R2). The states-only setting further reduces perplexity to 19.0, but at the cost of lower MAUVE (0.85, still above ELF-B's 0.73), reduced diversity, and weaker judged coherence. Table 1 now distinguishes these as the distribution-matching and lowest-perplexity operating points, while the new Table 20 compares both settings with ELF-B in terms of cost, perplexity, MAUVE, Self-BLEU, and judged coherence.

Q1: Why does K=8 perform best, and is the advantage due to additional training? The main advantage comes from block length rather than additional training along the progressive chain. At K=8, each document has only 7 block boundaries compared to 63 at K=64, and the first token of each block is the hardest to predict (Appendix B.1). Longer blocks also provide richer context for the decoder and reduce the sequence length modeled by the prior from 64 to 8 states. K=8 achieves the lowest perplexity across all measured entropy levels from 5.00 to 5.25 (Table 15). This comes with a trade-off: prior compute decreases from 631 to 138 GFLOPs, while the serial decoding path becomes longer.

To test whether the advantage depends on additional training, we trained a K=8 model from scratch without the progressive chain or realization codes (W1):

Plain sampling, entropy 5.16Training tokensPerplexity ↓
K=64 (paper)44B39.2
K=32 (paper)48B32.2
K=8, from scratch (new)13.7B26.0

Despite using less than a third of the training tokens, the new K=8 model achieves substantially lower perplexity. Moreover, at K=8, direct fine-tuning with the same total token budget as the chain yields nearly identical results (22.9 versus 22.8; Appendix B.3). These results support our explanation that K=8 benefits primarily from its block length rather than additional training. We clarify the limitations of the cross-K comparison in Section 3.3.

Q2: Which metrics should guide configuration selection and be highlighted in the results? We agree that perplexity alone is insufficient to assess generation quality. We select our default based on perplexity at matched entropy (5.16), MAUVE@512, and inference cost, with judged coherence as an additional check (General Response, R4). This favors K=8 with generated realization codes (22.8 perplexity, 0.93 MAUVE, 0.31 TFLOPs) over the states-only setting (19.0 perplexity, 0.85 MAUVE), which sacrifices coherence. The judge shows no clear preference between our default and 32-step ELF-B in whole-document coherence (12:16 order-consistent pairs). Following the reviewer's suggestion, the revision now highlights perplexity and MAUVE together and reports judged coherence in the main results.

Dear Reviewer,

As the discussion period approaches its end, we hope our responses and additional experimental results have addressed your concerns. We welcome any further discussion, as time allows.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all three reviewers for their detailed feedback. Across the reviews, three main concerns were raised. To address them, we conducted one additional experiment for each concern following the same evaluation setup. The new results are summarized in the tables below and reported in full in Appendix F of the revised PDF, with changes in red and all existing section, table, and figure numbers unchanged. The code is in the supplementary zip.

R1: Predictive versus content states: a from-scratch controlled comparison (Reviewers sBC2 and Mtv7; new Appendix F.1). We trained predictive-state and content-state models from scratch at K=8, each with its own encoder and decoder, so the content-state model reuses nothing from predictive training. The architecture, data, training steps, and prior training are identical.

Trained from scratch, K=8Perplexity (entropy 5.16) ↓MAUVE@512 ↑Judged coherence (pairs won) ↑Prior: denoising error ↓Prior: Fréchet distance ↓
Predictive states26.00.913214%11.6
Content states41.70.851041%40.3
  • Predictive states generate better text (first three columns). At matched entropy, the content-state model has 60% higher perplexity (8–16% in Table 3) and lower MAUVE, and a 32B LLM judge prefers whole documents from the predictive-state model.
  • Predictive states are easier for the prior to learn (last two columns). We evaluated each prior directly on held-out text, without the decoder. The two priors use the same architecture and training steps, and the states are standardized to the same scale. Denoising error measures the prior's reconstruction error at a 50% noise level, normalized by the state variance. Fréchet distance is computed between prior-generated and real states (3.5 between two sets of real states). The predictive prior performs substantially better on both metrics. One key difference is that content states must also encode the wording of their own blocks, requiring at least 1.9 nats per token, whereas PSF leaves this to the decoder.

Thus, predictive states are both easier for the prior to model and better for text generation. The predictive model in this experiment is also trained using a simple three-stage procedure, without the K-chain or realization codes mentioned by Reviewer rCRy.

R2: Comparable whole-document coherence to 32-step ELF-B, with higher MAUVE at 23× fewer FLOPs (all reviewers; new Appendix F.2).

Inference TFLOPs / doc ↓Perplexity (entropy 5.16) ↓MAUVE@512 ↑Judged coherence (pairs won) ↑
PSF, K=8, default setting0.3122.80.9312
ELF-B, 32 steps7.224.00.7316

Using the same judge, we evaluated 200 document pairs in both presentation orders. We find no clear preference in whole-document coherence between PSF at K=8 in its default setting and 32-step ELF-B (12:16 among order-consistent pairs; 10:13 in Table 18). PSF is preferred for lower repetition (20:5) and achieves higher MAUVE@512 while using 23× fewer FLOPs per document. We also evaluated variants that condition each block on realized text, including one refinement round and sequential block decoding, but neither improves judged coherence over the single-pass model.

R3: The advantage remains at matched model size and with a smaller training budget (Reviewer Mtv7; new Appendix F.3). We trained an additional model matched to the size of ELF-B, PSF-S.

ParametersTraining FLOPsInference TFLOPs / doc ↓Perplexity (entropy 5.16) ↓MAUVE@512 ↑
PSF-S (new)104M2.5×10¹⁹0.15722.80.86
ELF-B, 32 steps105M3.2×10¹⁹7.224.00.73

With matched parameters and less training (31B versus 46B tokens), PSF-S outperforms 32-step ELF-B, the baseline used in Section 3.1, on both perplexity and MAUVE@512, with 46× fewer inference FLOPs.

R4: Choosing the default setting (Reviewers rCRy and sBC2; Section 3.1). We select the default setting based on three criteria. First, we compare perplexity at the reference entropy of 5.16 to evaluate fluency at the same level of lexical diversity. Second, we compare MAUVE@512 to ensure that lower perplexity does not come at the cost of distributional similarity. Finally, we consider inference cost, with judged coherence as an additional check. Based on these criteria, we choose K=8 with generated realization codes as our default setting (the first row of the R2 table). We separately report the states-only setting (19.0 perplexity and 0.85 MAUVE) as the lowest-perplexity operating point of the same model. Both settings outperform 32-step ELF-B in perplexity and MAUVE. For ELF-B, we use its best 32-step configuration, with SDE sampling and guidance 3. None of the other configurations we tested improves on this result beyond sampling variation (Appendix A.5). Thus, each system is evaluated in its own best setting at ELF-B's reference entropy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.