acceptodds
Under review as a conference paper at ICLR 2027

Improving Sample Diversity in Autoregressive Text-to-Image Generation via Cluster Truncation

Abstract

While diffusion models achieve state-of-the-art image quality for text-to-image (T2I) generation, recent work has demonstrated that they suffer from sample diversity collapse. In this work, we investigate whether autoregressive (AR) image generation models can deliver greater sample diversity at an acceptable level of image quality. With recent advances in quality and efficiency, AR models have emerged as a viable alternative to diffusion-based image generation. Beyond enabling new use cases such as interleaved image-text generation, their sequential generation process makes them compatible with a wide range of token-based decoding strategies originally developed to improve diversity in text generation. Motivated by the potential of a better diversity-quality tradeoff in the AR paradigm, we present, to our knowledge, the first systematic study of sample diversity in next-token AR image generation models. We show that two key properties of AR image generation, persistently high token-level entropy and substantial redundancy in visual token spaces, limit the effectiveness of existing token-level decoding methods for diversity enhancement. We therefore propose p-less cluster, a new decoding strategy that performs entropy-based truncation sampling at cluster level rather than at token level. We evaluate our approach and baseline decoding methods across four autoregressive T2I models and two datasets using a comprehensive suite of metrics spanning image quality, prompt alignment, and diversity. Our results show that p-less cluster unlocks the greatest diversity across most evaluated autoregressive T2I models and datasets while keeping image quality and prompt alignment at an acceptable level.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
autoregressive text-to-image, sampling diversity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3TK1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates limited sample diversity in autoregressive text-to-image generation and proposes a training-free cluster-level truncation strategy that samples over groups of visually similar tokens rather than individual tokens. The method is simple, broadly applicable, and improves the diversity–quality trade-off across several AR image generators. However, the overall novelty and technical depth are relatively limited, as the approach mainly combines visual-token clustering with existing truncation-based sampling, and stronger evidence is needed to verify that the gains reflect meaningful high-level semantic diversity.

Critical strengths
  1. This paper Identifies an interesting mismatch between token-level randomness and semantic-level diversity in autoregressive image generation.

  2. The authors propose a simple, training-free cluster-based sampling strategy that can be easily integrated into existing AR models. They also provide reasonably broad experiments across multiple models, showing consistent improvements in the diversity–quality trade-off.

Critical weaknesses
  1. The core idea mainly extends existing truncation strategies from token-level to cluster-level sampling. Similar visual-token clustering has already been explored in prior work such as ClusterMark [1], although for a different purpose, making the methodological novelty somewhat limited. The proposed method is essentially a training-free decoding heuristic based on offline K-means clustering, without substantial modifications to the model architecture, training objective, or representation learning.

  2. The experiments mainly rely on existing diversity metrics, while it remains unclear whether the method truly improves high-level diversity in composition, object relations, and semantic modes.

[1] D. Lukovnikov, A. Müller, E. Quiring, and A. Fischer, “ClusterMark: Towards Robust Watermarking for Autoregressive Image Generators with Visual Token Clustering,” CVPR 2026.

Questions for authors
  1. How does the proposed cluster truncation fundamentally differ from prior visual-token clustering methods such as ClusterMark, beyond applying clustering to a different downstream objective?

  2. How sensitive is the method to the choice of clustering algorithm, number of clusters, and feature space used for clustering? Would alternative clustering strategies lead to significantly different results?

  3. Can the authors provide stronger evidence that the diversity improvement comes from high-level semantic variations, such as composition, object relations, or pose, rather than mainly low-level appearance or texture changes?

  4. Given that the method is training-free and technically lightweight, what is the main methodological contribution beyond combining K-means clustering with existing truncation-based sampling?

  5. Does cluster-level sampling introduce any additional inference overhead, especially for large visual vocabularies, and how does this overhead compare with standard top-k/top-p sampling?

3 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for appreciating our analysis of the mismatch between token-level randomness and semantic diversity in autoregressive (AR) image generation. Beyond the proposed method itself, we believe these insights offer valuable guidance for future research, particularly given the growing interest in unified autoregressive image–text generation. We would like to address your feedback and questions below.

W1 + Q1 + Q4. Discussion on our methodological contribution and positioning against other cluster-based method.

We thank the reviewer for pointing us to ClusterMark. We will revise the manuscript to cite this work in Related Works and clarify our methodological distinction as below.

While both methods leverage codebook-based token clustering, they address different problems through fundamentally different decoding operations. ClusterMark uses clusters to assign visually similar tokens to the same watermark partition, improving watermark robustness under image perturbations. In contrast, our method uses clusters as the unit of probability aggregation and entropy-adaptive truncation to address the mismatch between token-level randomness and image-level diversity. We further provide a theoretical justification for why clustering is needed for this diversity-tradeoff problem in Appendix B.

We would also like to highlight the timely and potentially broad impact of a training-free decoding algorithm such as p-less cluster. With growing interest in developing new autoregressive models for image generation, there is an increasing need for decoding strategies that can improve generation quality and diversity without requiring model retraining. As a plug-and-play method that makes minimal assumptions about model architecture and training procedures, p-less cluster can be readily integrated into different AR image generators. Its effectiveness across four distinct AR models demonstrates its potential for broad applicability, while its low computational overhead makes it practical for deployment.

W2 + Q3. The experiments mainly rely on existing diversity metrics, while it remains unclear whether the method truly improves high-level diversity in composition, object relations, and semantic modes.

We agree that our paper does not explicitly evaluate high-level diversity in composition, object relationships, and semantic modes, and we will acknowledge this limitation in the revised manuscript. To ensure a rigorous and well-grounded evaluation, we adopt the diversity metrics established in [1].

[1] Parmar, Gaurav, et al. "Scaling group inference for diverse and high-quality generation." International Conference on Learning Representations. Vol. 2026. 2026.

AuthorsOpenReview ↗

Response 2/2

Q2. How sensitive is the method to the choice of clustering algorithm, number of clusters, and feature space used for clustering?

We included a study on the effect of the number of clusters on both image quality and diversity in Appendix E.3. While diversity gains tend to increase with a larger number of clusters, using fewer clusters provides greater robustness to quality degradation across temperatures.

To study the effect of the clustering algorithm, we use a small set of 120 prompts from GenEval and compare the performance of two clustering algorithms: K-Means (reported in the paper) and balanced K-Means (using the same number of clusters, CFG, temperature, and k as defined for each base model in Table 3, Appendix D.2). On Janus Pro the two variants are indistinguishable. On Anole, balanced K-means raises CLIP alignment slightly and costs a small, significant amount of diversity.

ModelSettingVariantCLIPDINOv2Vendi
Janus ProCFG 5, T 4.0, k 1.5, n 500K-means0.3190.3322.500
Balanced K-means0.3180.3362.519
Δ−0.001 [−0.003, +0.002]+0.004 [−0.011, +0.018]+0.019 [−0.041, +0.079]
AnoleCFG 3, T 2.5, k 1.5, n 1000K-means0.2980.5683.577
Balanced K-means0.3100.5463.461
Δ+0.012 [+0.008, +0.015]−0.022 [−0.040, −0.004]−0.115 [−0.192, −0.039]

GenEval tuning subset (120 prompts, 5 images each), same prompts for both variants. Δ = balanced − K-means, 95% paired bootstrap CI; bold = CI excludes 0.

Q5. Does cluster-level sampling introduce any additional inference overhead, especially for large visual vocabularies, and how does this overhead compare with standard top-k/top-p sampling?

We measure per-image generation time on an A100-80GB GPU using 20 PartiPrompts with five images generated per batch, including the generation loop and VQ decoding. Measurements are CUDA-synchronized, with one warm-up prompt excluded. Percentages in parentheses indicate runtime increases relative to Default.

ModelDefault (s/image)Dyn. Temp. (s/image)p-less cluster (s/image)
Janus Pro2.082.17 (+4.3%)2.17 (+4.4%)
Emu374.8775.19 (+0.4%)75.71 (+1.1%)
LlamaGen1.071.10 (+2.8%)1.10 (+2.3%)
Anole2.992.98 (−0.3%)3.03 (+1.2%)

Our method incurs minimal runtime overhead, increasing per-image generation time by only 1.1–4.4% relative to Default across all four models. For Janus Pro and LlamaGen, its runtime is nearly identical to Dynamic Temperature, suggesting that cluster-based truncation introduces negligible additional overhead.

Summary. We believe that your feedback and suggested additional experiments have improved our paper. We hope that you can re-evaluate our paper in light of these new results and response.

Thanks for the rebuttal. After reading the rebuttal, I am still concerned that the method shares many similarities with cluster-based method, especially in the comparison with the cluster-based by experiments and analysis experiments are important. Besides, the diversity measurement is important and needs to be analyzed from many perspectives. Therefore, I would like to keep the rating unchanged.

Reviewer_Fa2tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present what they describe as the first study of per-prompt sample diversity in next-token AR T2I models, arguing that persistently high token-level entropy and redundancy in the visual token space limit existing token-level decoding methods. They propose p-less cluster, which groups codebook tokens via offline K-means, applies p-less truncation to the summed cluster probabilities, and samples a cluster and then a token within it. They show that, on four AR T2I models and two prompt benchmarks, p-less cluster reaches higher sample diversity than the models’ default samplers and entropy-based dynamic temperature in most settings while keeping image quality and prompt alignment acceptable. The gains are largest for open-ended prompts, and AR models in general are more diverse than the FLUX diffusion baselines.

Critical strengths
  • The paper tackles a timely and practically relevant question, whether AR decoding can avoid the well-documented diversity collapse of diffusion T2I models.
  • The method is simple, training-free, and lightweight by design, since clustering is done offline and two-stage sampling adds little per-step work, though runtime overhead is not reported.
Critical weaknesses
  • p-less cluster’s diversity gains over the best baseline are mostly within one to two bootstrap standard errors (0.005–0.008); only Emu3 and Janus Pro on PartiPrompts clearly exceed them. The method is beaten on Janus Pro/GenEval and tied on Emu3/GenEval, and per Appendix D, the LlamaGen/GenEval setting falls below the paper’s own alignment threshold. The abstract’s claim of “greatest diversity across most” models therefore overstates the evidence.
  • The baselines are incomplete and unevenly tuned: min-p, tuned top-p/top-K, and clustering without truncation are missing from the main comparison. Token-level p-less appears only in an appendix ablation (Table 8, Vendi only), and low CFG is tested only for Janus Pro. Dynamic temperature uses α and θ borrowed from another model for three of the four models, while p-less cluster gets a 2×2 grid over k and n. No random-partition control shows that semantic clustering, rather than any vocabulary partition, drives the gains.
Questions for authors

Please address the weaknesses.

3 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for your recognition of the simplicity and light-weight nature of our proposed method as well as the timely importance of the diversity-quality tradeoff in AR image generation. We would like to address your feedback as below.

W1. p-less cluster’s diversity gain compared to baseline methods

We agree that p-less cluster's diversity gains are not uniform across all eight model–benchmark pairs. As stated in the abstract, pp-less cluster achieves the highest diversity in most settings (6 of 8 model–benchmark pairs at default CFG), rather than universally. For the two models with mixed results (Emu3 and Janus Pro), we analyze the method's strengths and limitations in Section 5.1. For example, on Janus Pro/GenEval, p-less cluster sacrifices 0.046 in DINOv2 diversity but achieves substantially higher alignment than Default (4.06 vs. 3.48). Our results also show that p-less cluster is particularly effective for open-ended, complex prompts (Figure 3).

Though pp-less cluster does not win on all 8 model-dataset pairs, we believe our paper makes three valuable contributions to the field:

  1. A practical and broadly applicable decoding method. We introduce a training-free, plug-and-play decoding algorithm that requires minimal assumptions about model architecture or training, incurs negligible runtime overhead, and improves diversity particularly for open-ended, imaginative prompts. This contribution is especially timely given the growing interest in unified autoregressive image–text generation models.

  2. New empirical and theoretical insights into failure of token-level decoding. We provide an empirical analysis of the mismatch between token-level randomness and perceptual diversity, complemented by a theoretical analysis in Appendix B. These findings help explain the limitations of conventional token-level decoding and motivate future research into more effective decoding strategies for AR image generation.

  3. A systematic study of the diversity–quality tradeoff in AR image generation. To our knowledge, we present the first systematic comparison of diversity–quality tradeoffs across four AR image generators, examining how different decoding strategies behave under varying CFG scales, sampling temperatures, and prompt complexities. Beyond evaluating our proposed method, this analysis provides insights into the strengths and limitations of existing decoding strategies and offers guidance for future research.

W1. Clarification on LlamaGen/GenEval quality and alignment results in Appendix D.

The acceptability criterion (VLM quality and alignment scores > 3.0) is applied only to the tuning subset (120 GenEval and 90 PartiPrompts prompts) to select one temperature per method. It is neither re-applied nor guaranteed on the full benchmark, where scores may fall slightly below 3.0. This is not unique to p-less cluster: for example, Default on Emu3/GenEval also achieves an alignment score of 2.96 on the full benchmark. Thus, a full-benchmark score below 3.0 does not indicate a violation of the temperature-selection protocol.

W2. the baselines are incomplete.

We thank the reviewer for suggesting additional baselines. We conduct additional experiments and present a comparison of pp-less cluster with min-p, top-p, and token-level pp-less. Each method’s hyperparameters are tuned following our protocol (Appendix D). On both benchmarks, pp-less cluster is more diverse than all three additional baselines. For a full comparison of token-level pp-less and pp-less cluster across all 8 model-dataset pairs, we refer the reviewer to Table 8. Both Vendi and DINOv2 metrics rank methods similarly.

MethodJanus Pro / GenEval (CFG 5)Janus Pro / PartiPrompts (CFG 5)
Default0.374 (3.76 / 3.48)0.426 (3.96 / 3.40)
Dynamic Temperature0.222 (4.76 / 4.76)0.375 (4.50 / 4.01)
top-p (p = 0.9)0.368 (3.75 / 3.55)0.399 (4.12 / 3.62)
min-p (p = 0.1)0.316 (4.16 / 4.12)0.426 (4.06 / 3.44)
token-level p-less0.253 (4.57 / 4.58)0.393 (4.29 / 3.82)
p-less cluster0.328 (4.10 / 4.06)0.454 (3.72 / 3.18)

Entries are DINOv2 (quality / alignment); bold marks the most diverse method per column.

AuthorsOpenReview ↗

Response 2/2

W2. low CFG is tested only for Janus Pro.

We report low-CFG results only for Janus Pro because the other models produce severely degraded images at CFG = 1.0, making meaningful comparisons of diversity and quality unreliable.

W2. no random-partition control

We add the comparison between pp-less cluster and a random-partition variant with matched cluster size using Janus Pro and Anole (both at default CFG) on PartiPrompt benchmark. The results show that pp-less cluster with random-partition not only yields less diverse sets of images but also yields lower quality and prompt alignment. A qualitative analysis shows that images generated by random-partition pp-less cluster tend to degenerate or collapse to white canvas.

ModelPartitionQualityAlignmentDINOv2Vendi
Janus Pro (CFG 5.0)K-means (Table 1)3.723.180.4543.08
random-partition3.232.060.4112.89
Anole (CFG 3.0)K-means (Table 1)3.823.290.5423.47
random-partition2.271.210.4252.98

Summary. We thank the reviewer for your suggested additional experiments. We believe that these additional results have strengthened our paper. Due to the lack of compute during the rebuttal period, we could not run additional baseline comparisons for all the models but we will add the results for the two new suggested experiments for all 4 models upon acceptance.

Thank you for the response and additional experiments. The random-partition control and runtime measurements address two of my concerns. However, the new baselines cover only one model and are mixed on GenEval, most gains remain within the standard error, and the LlamaGen/GenEval entry still falls below the threshold on the tuning subset. The uneven tuning of dynamic temperature and the missing comparison to clustering without truncation are not addressed. I therefore keep my score at 2.

Reviewer_z8gMOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents the first systematic study of sample diversity—the variation among images generated from the same prompt—in next-token autoregressive (AR) text-to-image models. The authors identify two properties of AR image generation that limit existing token-level decoding methods: AR models maintain persistently high token-level entropy across decoding steps, and visual token spaces are highly redundant. To address this, the authors propose p-less cluster, a cluster-aware decoding method. The method is evaluated on four AR models (Janus Pro-7B, LlamaGen, Emu3, Anole) against two diffusion baselines (Flux.1-Dev and Schnell) over the PartiPrompts and GenEval benchmarks. Results show that all AR models inherently outperform diffusion baselines on diversity, and p-less cluster achieves the highest acceptable diversity in most model-dataset settings.

Critical strengths
  • Novelty and breadth of the study: systematic investigation of per-prompt sample diversity in next-token autoregressive T2I models that I feel I learn something new

  • Well-motivated, low-cost method. The proposed p-less cluster decoder is grounded in two clearly articulated empirical observations (persistently high token-level entropy, and high redundancy of visual token space) that explain why token-level decoding fails for images.

Critical weaknesses
  • Inconsistent and sometimes modest gains. p-less cluster does not uniformly win: it matches or underperforms default sampling for Emu3 and Janus Pro on GenEval

  • Dependence on tuning and model/prompt characteristics. The method introduces hyperparameters that must be tuned per model

Questions for authors

The paper claims to be the first study of sample diversity in next-token AR models. How does this differ substantively from DiverseAR (Yang et al., 2025), which improves diversity in bitwise/next-scale AR? Is the token-level vs. cluster-level distinction really that fundamental?

The K-means partition is computed offline on codebook vectors. How sensitive are results to the clustering quality itself — i.e., is the benefit coming from semantic clusters, or would random partitions of the vocabulary of the same size work nearly as well? An ablation with random/fixed clusters would be very informative.

The VLM judge (InternVL3-14B) is used for both quality and alignment, and these scores also act as the acceptance constraint. Is there a risk of circularity — the same model that gates the comparison also determines which method is "acceptable"? Was human validation performed on at least a subset?

Does p-less cluster, by explicitly encouraging exploration of rare modes, increase the chance of generating undesirable or unsafe content (e.g., distorted anatomy, prompt-violating attributes)?

3 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for your recognition of the novelty and breadth of the study as well as the well-motivated and low-cost nature of our proposed method. We would like to address your feedback as below.

W1. Inconsistent and sometimes modest gains.

We agree that p-less cluster's diversity gains are not uniform across all eight model–benchmark pairs. For the two models with mixed results (Emu3 and Janus Pro), we analyze the method's strengths and limitations in Section 5.1. For example, on Janus Pro/GenEval, p-less cluster sacrifices 0.046 in DINOv2 diversity but achieves substantially higher alignment than Default (4.06 vs. 3.48). Our results also show that p-less cluster is particularly effective for open-ended, complex prompts (Figure 3).

Though pp-less cluster does not win on all 8 model-dataset pairs, we believe our paper makes three valuable contributions to the field:

  1. A practical and broadly applicable decoding method. We introduce a training-free, plug-and-play decoding algorithm that requires minimal assumptions about model architecture or training, incurs negligible runtime overhead, and improves diversity particularly for open-ended, imaginative prompts. This contribution is especially timely given the growing interest in unified autoregressive image–text generation models.

  2. New empirical and theoretical insights into failure of token-level decoding. We provide an empirical analysis of the mismatch between token-level randomness and perceptual diversity, complemented by a theoretical analysis in Appendix B. These findings help explain the limitations of conventional token-level decoding and motivate future research into more effective decoding strategies for AR image generation.

  3. A systematic study of the diversity–quality tradeoff in AR image generation. To our knowledge, we present the first systematic comparison of diversity–quality tradeoffs across four AR image generators, examining how different decoding strategies behave under varying CFG scales, sampling temperatures, and prompt complexities. Beyond evaluating our proposed method, this analysis provides insights into the strengths and limitations of existing decoding strategies and offers guidance for future research.

W2. Dependence on tuning and model/prompt characteristics.

We acknowledge that p-less cluster introduces two hyperparameters that require tuning. However, we deliberately restrict the search to a small 2×2 grid and perform tuning on a limited subset of prompts, rather than the full benchmark. The selected hyperparameters are then fixed for evaluation on the full benchmark, ensuring that the reported results do not rely on benchmark-specific tuning.

Q1. How does this differ substantively from DiverseAR (Yang et al., 2025)?

We discuss the distinction between DiverseAR and our work in Section 2.1. DiverseAR is only restricted to bitwise next-scale AR models while our paper provides a systematic evaluation of 4 different next-token AR models and our proposed method is a plug-in decoding method for any future next-token AR models.

Q2. An ablation with random/fixed clusters would be very informative.

This is a great suggestion. We add the comparison between pp-less cluster and a random-partition variant with matched cluster size using Janus Pro and Anole (both at default CFG) on PartiPrompt benchmark. The results show that pp-less cluster with random-partition not only yields less diverse sets of images but also yields lower quality and prompt alignment. A qualitative analysis shows that images generated by random-partition pp-less cluster tend to degenerate or collapse to white canvas.

ModelPartitionQualityAlignmentDINOv2Vendi
Janus Pro (CFG 5.0)K-means (Table 1)3.723.180.4543.08
random-partition3.232.060.4112.89
Anole (CFG 3.0)K-means (Table 1)3.823.290.5423.47
random-partition2.271.210.4252.98
AuthorsOpenReview ↗

Response 2/2

Q3. Risk of circularity of VLM judge

We use VLM judge scores only as the acceptance threshold in the hyperparameter tuning protocol, and they do not affect our primary quantity of interest, the diversity metrics. To validate the ranking quality of the VLM judge, we add another quality metric, HPSv2. On the GenEval tuning set for Janus Pro (120 prompts × 5 images, CFG 5), we score all 21 configurations of the tuning protocol (Default, Dynamic Temperature and p-less cluster, each at 7 temperatures; 12,600 images). VLM quality and alignment rank these configurations almost identically to HPSv2 (Spearman ρ = 0.995 and 0.979), and per image the correlation is ρ = 0.66 and 0.65.

Q4. Does p-less-cluster increase the chance of unsafe content?

To check for unsafe content, we ran a NSFW classifier (Falconsai/nsfw_image_detection) on 553 x 5 images (GenEval) Janus Pro generated using all 3 methods. The results shows that all three decoding methods have very small probability of unsafe generated contents.

MethodFlagged (score ≥ 0.5)Flag rateMean NSFW score
Default60.22%0.0028
Dynamic Temperature50.19%0.0029
p-less cluster90.34%0.0046

Summary. We thank the reviewer for your suggested additional experiments. We believe that these results are a good addition to our paper and we hope that you can re-evaluate our paper in light of our response and additional results.

Thanks for the response. I'll keep the score unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.