acceptodds
Under review as a conference paper at ICLR 2027

Hidden in Plain Sight: Recovering Latent Generative Diversity in Visual Autoregressive Models

Abstract

Visual Autoregressive (VAR) models have recently garnered significant attention for their innovative next-scale prediction paradigm, offering notable advantages in both inference efficiency and image quality compared to traditional multi-step autoregressive (AR) and diffusion models. However, despite their efficiency, VAR models often suffer from the diversity collapse i.e., a reduction in output variability, analogous to that observed in few-step distilled diffusion models. In this paper, we introduce DiverseVAR, a simple yet effective approach that restores the generative diversity of VAR models without requiring any additional training. Our analysis reveals the pivotal component of the feature map as a key factor governing diversity formation at early scales. By suppressing the pivotal component in the model input and amplifying it in the model output, DiverseVAR effectively unlocks the inherent generative potential of VAR models while preserving high-fidelity synthesis. Empirical results demonstrate that our approach substantially enhances generative diversity with only neglectable performance influences. Our code will be publicly released.

Primary area:
generative models
Keywords:
Visual Autoregressive Models, Text-to-Image Generation, Generative Diversity, Diversity Collapse, Training-Free Inference

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_EEaSOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This study investigates diversity collapse in VAR models and proposes DiverseVAR, a training-free approach to address this issue during inference. The core idea is to identify early-scale components that are most closely associated with structure formation and modulate them through soft suppression and soft amplification. The paper presents a series of analyses and visualizations to illustrate the method and evaluates it on multiple benchmarks. However, a key concern is that the paper does not provide sufficient analysis to establish whether VAR itself suffers from diversity problems, as its experiments are conducted only on Infinity, a variant of VAR. Therefore, there is a potential problem of analytical errors.

Critical strengths

1.The paper is well written and easy to understand.

2.The experiments are rich, and the paper reports results on multiple benchmarks.

Critical weaknesses

I am very familiar with VAR models and various VAR-based text-to-image models.

1.The analysis of why VAR models lack diversity is insufficient. The paper only presents the phenomenon, but does not explain why VAR models lack diversity. In fact, the original VAR [1] and HART models do not lack diversity. The authors can check Table 1 of VAR [1], where VAR achieves the highest IS score, which does not suggest a lack of diversity. In addition, the authors can show the Figure 8 of the VAR and HART, and compare the diversity of HART with diffusion models (e.g., SD3) or AR models (e.g., Emu3 or Show-o). These additional results would show that the diversity issue may not come from the VAR model itself, but rather from other reasons, such as the drawback of Infinity. Therefore, there may be a factual error in the paper.

2.I suspect that the diversity examples of HART shown in Fig. 1 were carefully cherry-picked. This is a completely unreasonable comparison, because according to my own experimental results, HART exhibits very rich diversity.

[1] Tian et al. Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction. NeurIPS 2024.

Questions for authors

None

9 comments
AuthorsOpenReview ↗

Response to Reviewer EEaS (1/2)

We appreciate your comments and would like to clarify the following points.

W1: VAR and HART do not lack diversity (highest IS in VAR, Fig. 8 of VAR/HART); the issue may come from Infinity rather than the VAR paradigm, implying a factual error.

  1. IS and the samples in the VAR paper do not reflect prompt-conditional diversity. VAR [1] is a class-conditional ImageNet model, and IS measures fidelity and class coverage rather than the variability among multiple samples under the same condition; as shown in CADS [2, App. F], insufficiently diverse samples can even obtain higher IS/Precision than real data.
  2. Our main claim is grounded on Infinity rather than HART. The evidence of diversity collapse in our Introduction refers to rows 1, 3 and 5 of Fig. 1 (Infinity-2B/8B), and our main experiments use Infinity-2B/8B. HART is included to examine the applicability of DiverseVAR beyond Infinity, not to claim that HART severely collapses. We agree that the original caption of Fig. 1 was overly broad.
  3. The phenomenon is independently confirmed, and we have narrowed the claim. Diversity is a matter of degree, and our claim concerns the per-prompt diversity of current mainstream text-to-image VAR models. Park et al. [3] report that text-conditioned VAR models (the image model Infinity and the video model InfinityStar) lack per-prompt diversity and propose a training-free remedy evaluated on Infinity and Switti; DiverseAR [4] analyzes the cause in Infinity from the perspective of bitwise modeling. Infinity is currently the strongest and most widely used text-to-image VAR model, so its diversity issue is representative and of practical importance.

Accordingly, we have revised the paper (highlighted): the Abstract and Introduction now restrict the claim to "mainstream text-to-image VAR models such as Infinity" and to per-prompt diversity and cite [3, 4] as independent evidence; Sec. 2 discusses [3, 4]; and the Fig. 1 caption now states that the vanilla Infinity models exhibit limited per-prompt diversity and that our method produces more diverse outputs for both Infinity and HART, no longer stating that HART suffers from diversity collapse.

[1] Tian et al., Visual Autoregressive Modeling, NeurIPS 2024.

[2] Sadat et al., CADS, ICLR 2024.

[3] Park et al., DiverseVAR, ECCV 2026.

[4] Yang et al., DiverseAR, arXiv 2025.

AuthorsOpenReview ↗

Response to Reviewer EEaS (2/2)

W2: The HART examples in Fig. 1 appear cherry-picked; in the reviewer's experiments HART shows rich diversity.

The samples in Fig. 1 are not cherry-picked: they are randomly drawn, all prompts are listed in the caption, and the vanilla model and DiverseVAR use identical random seeds, differing only in whether SSR/SAR is enabled. The HART rows are intended to illustrate the relative improvement brought by our method under the same seeds, not to claim that HART lacks diversity entirely. In the revision, the Fig. 1 caption states that both methods use identical seeds and no longer describes HART as collapsed. In addition, Fig. S6 provides sample grids with fixed seeds 0–7 for the vanilla model and DiverseVAR (Infinity-2B), without any selection within each prompt. We would also be glad to compare on the specific prompts and settings under which you observed rich diversity for HART.

Ethics flag.

We discuss potential risks (e.g., deepfakes, privacy and bias) in the Broader Impacts and Ethical Statement of App. B. All models and datasets are publicly available, and no sensitive personal data are used. We would appreciate it if the reviewer could point out any specific ethical concern so that we can address it.

We hope these responses address your concerns, and we are happy to provide further clarification.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer EEaS,

Thank you once again for your constructive comments. We would like to know if our responses have adequately addressed your concerns. We are also eager to hear any additional concerns or suggestions from you that could help us further strengthen our paper. Your feedback would be invaluable in making this work even more compelling and worthy of acceptance.

As the rebuttal window is closing soon, we genuinely appreciate your feedback. We hope our response has addressed your concerns.

Best Regards, Authors of submission 745

Reviewer_yutrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DiverseVAR, a training-free technique for improving the generative diversity of VAR models on image generation. The paper provides the key observation that diversity in early scales relies on the pivotal token of feature maps, and builds the main method that exploits this property to enhance the diversity. The proposed techniques, soft-suppression regularization and soft-amplification regularization jointly enhance the generation diversity of multiple VAR-based text-to-image models without further fine-tuning.

Critical strengths
  • Writing is clear and well-structured. The paper is easy to follow, with a clear motivation to resolve the diversity collapse issue in VAR generative models. The connection between the main observations on the generation procedure in VAR and the proposed technique is also reasonable and grounded.

  • Simple and efficient solution. The proposed method can be simply integrated into pre-trained VAR models, making it practical to apply to existing models and future ones.

Critical weaknesses
  • Evaluation on diversity is not convincing. Although recall, coverage, and FID evaluate the matching with reference distributions, they don't directly prove increased diversity across multiple generations of the same prompt. Please refer to CADS [1], which proposes a diversity enhancement technique for diffusion models and evlauation per-prompt diversity using Mean Similarity Score (MSS) and Vendi score. Reporting these metrics would help distinguish whether the proposed method reliably increases the generation diversity from each prompt.

  • Missing related work and baselines. DiverseVAR [2] with the same name addresses the same issue of diversity collapse in VAR, and also solves it through a training-free technique. As it has been public since 2025, it would have been clearer if the paper at least provided discussions in comparison to it or further provided experiments.

  • The qualitative evidence on diversity increase is unclear. In figure 8, 9, and S2, many samples from DiverseVAR still show similar overall composition and structure, while having some variations in color or small appearance details. These results don't sufficiently show improvements in diversity of scene layouts, viewpoints, or compositions. Is the main motivation of the paper to increase the diversity of the details of the generated images, or to generate diverse image compositions and structures as well? A clearer distinction would strengthen the argument.

  • VAR-specific contribution is unclear. While DiverseVAR shows clear improvements VAR diversity, I'm curious if the approach itself is grounded on the properties of VAR, or it is a general technique and the paper just applies it to VAR. Observation 1 has already been widely reported in diffusion model literature, and the proposed singular-value manipulation does not appear to inherently depend on VAR but rather potentially apply to any transformer-based generative model (e.g., text-to-image diffusion models). Prior work has also discussed attention-matrix decomposition to control diversity of diffusion models [3]. Given these points, the paper should clarify which insights or design choices specifically address the issues in VAR, and whether the proposed method is intended to generalize across different types of generative models.


[1] CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling, Sadat et al., ICLR 2024

[2] DiverseVAR: Balancing Diversity and Quality of Next-Scale Visual Autoregressive Models, Park et al., ECCV 2026

[3] Balancing Fidelity and Diversity in Diffusion Models via Symmetric Attention Decomposition: Hopfield Perspective, Cho et al., ICML 2026

Questions for authors
  • Related to the final point in the weaknesses section, do the authors view the proposed as VAR-specific, or can it readily be applied to other types of generative models (e.g., text-to-image diffusion models)?

  • While the paper starts from the empirical observations that VAR lacks generation diversity, I'm curious about the fundamental causes of this diversity collapse in VARs. Does it fundamentally stem from the architecture and generation mechanism of VARs, or is it because current available VAR models have not been scaled further yet compared to diffusion models?

6 comments
AuthorsOpenReview ↗

Response to Reviewer yutr (1/3)

We appreciate your constructive comments and will incorporate the discussions below into the revised paper.

W1: Report per-prompt diversity with MSS and Vendi as in CADS.

We agree and adopted the CADS protocol [1] (main paper Sec. 4.3, Tab. 5; details in Sec. C.1, Sec. E.8, Tab. S8): 1000 COCO 2017 prompts, 10 samples per prompt, identical seeds for the vanilla model and DiverseVAR, MSS and Vendi Score computed with SSCD features, together with pairwise LPIPS and the DINO structural distance to separate appearance and structural diversity.

Method (Infinity-2B)LPIPS↑MSS↓Vendi↑DINO↑CLIP↑
Vanilla0.4310.5355.440.1440.313
Ours0.5360.3917.260.1710.313

MSS decreases by 27% and Vendi increases by 33%, with the CLIP Score unchanged; all differences are far larger than the standard errors (< 0.03). To answer whether the method reliably increases diversity for each prompt: DiverseVAR is more diverse than the vanilla model on 99.9% (LPIPS), 100% (MSS), 100% (Vendi) and 98.3% (DINO) of the 1000 prompts (Wilcoxon signed-rank test, p < 10⁻¹⁶⁰), so the gain is not driven by a few prompts.

W2: Missing discussion and comparison with DiverseVAR of Park et al. (ECCV 2026).

Thank you for pointing out this work. We now cite and discuss it in Sec. 2. It also addresses the limited per-prompt diversity of text-to-image VAR models (Infinity and Switti) in a training-free manner, which independently supports our problem setting. The main differences are:

  1. Where the intervention happens. Park et al. inject noise into the text condition (CADS-style condition annealing) and introduce scale-travel, which re-encodes the accumulated features back to a coarser scale and regenerates them to correct the noise-induced artifacts. DiverseVAR leaves the condition unchanged; it softly suppresses the singular values of the block input features (SSR) and amplifies those of the block output features (SAR) at early scales, motivated by our analysis that the dominant singular components govern structure formation.
  2. Mechanism and cost. Their diversity comes from random noise whose artifacts must be repaired by extra encoding and regeneration, while ours is a structured spectral operation without external noise. Measured on the same A100 with their official code (VAE decoding included), Park et al. add 24.3% latency over their vanilla baseline, whereas DiverseVAR adds 5.4% with no extra memory (main paper Sec. 4.3; Sec. E.10, Tab. S10).
  3. Evaluation. Their per-prompt metrics (pairwise LPIPS, Vendi) are now also reported in our paper, together with MSS and the DINO structural distance (Tab. 5, Sec. E.8).
  4. Complementarity. The two approaches act on different signals (condition noise vs. feature spectrum) and can in principle be combined, which we leave for future work.

[1] Sadat et al., CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling, ICLR 2024.

AuthorsOpenReview ↗

Response to Reviewer yutr (2/3)

W3: In Figs. 8, 9 and S2, samples mainly vary in color and details; is the goal detail diversity or also composition and structure diversity?

Our goal covers both structural (composition) and appearance diversity. The prompts of Figs. 8, 9 and S2 (Fig. S3 in the revision, due to renumbering) are simple, e.g., "A hot air balloon floating above the clouds" or "The face of a cat"; the latter follows the category-level, face-centric setting of AFHQ to compare with its real images (Tab. 2), whose real images are aligned, centered faces. Such prompts strongly constrain the layout of a single subject, so the variation mainly appears in pose and appearance. To show composition-level diversity more clearly, we added:

  • Qualitative evidence (Fig. S6). On five scene-level prompts (e.g., "A mountain lake with a small wooden cabin and pine trees", "A dining room with a long table set for dinner", "A garden with a stone fountain and colorful flower beds"), the vanilla model and DiverseVAR use the same fixed seeds 0–7, without any selection within each prompt. The vanilla model repeatedly produces nearly the same layout and viewpoint (the cabin always on a small island at the center of the lake, the dining table always from the same frontal view, the fountain always at the center of the garden), whereas DiverseVAR varies the camera viewpoint, the scene layout, and the position, scale and orientation of objects (e.g., a cabin on stilts or on the shore, a diagonal view of the table, fountains of different positions and scales), while preserving all objects described in the prompt.
  • Quantitative evidence (Tab. 5). The DINO structural distance, which is insensitive to color and texture, increases from 0.144 to 0.171 and is higher than the vanilla model on 98.3% of the prompts; increasing the sampling temperature barely changes it (0.150 at τ = 3, Tab. 6).

Together, these results show that DiverseVAR increases diversity in composition and structure, not only in color and details.

AuthorsOpenReview ↗

Response to Reviewer yutr (3/3)

W4 / Q1: The VAR-specific contribution is unclear; is the method VAR-specific or a general technique? (Observation 1, Cho et al.)

We agree that the SVD-based suppression/amplification itself is architecture-agnostic and that structure forming early is not a new phenomenon. Our claim is that the form of the collapse, the intervention point and the design choices are tied to the next-scale structure of VAR:

  1. Source of randomness. Diffusion models inject noise at initialization and at every denoising step, whereas VAR has no such channel: its randomness comes only from token sampling at each scale. Fig. 8 shows that the logits of different samples of the vanilla VAR almost coincide, which is the collapse SAR is designed for.
  2. VAR-specific state. SSR acts on the input F̃_{k−1} obtained by downsampling the accumulated features of the previous scales (Eq. 3), i.e., the cross-scale state passing of next-scale prediction, which has no counterpart in diffusion models.
  3. Discrete coarse-to-fine stages. The structure is formed at the early scales (Figs. 2, 3), and the method acts only at scales {4, 6}; Tab. S1 shows that adding more scales quickly degrades quality.
  4. Ablations. Applying the operation to the logits or at the model level yields almost no diversity gain, and only the block level is effective (Tabs. S2, S3).

Empirically, diffusion-style randomness injection does not transfer directly to VAR: Park et al. need an extra re-encoding and regeneration step to make CADS-style condition noise usable on VAR, which costs +24.3% latency vs. +5.4% for Ours (Tab. S10); and isotropic feature noise at the same strength gives lower structural diversity and GenEval than suppressing the dominant components (Tab. S7).

Cho et al. (2026) decompose the attention matrices of diffusion transformers into symmetric and antisymmetric parts and amplify the antisymmetric part; DiverseVAR instead regularizes the singular value spectrum of block features at the early scales of VAR, with SAR compensation at the output. The operated object (attention matrix vs. feature spectrum), the mechanism and the stage differ, although both belong to the broad family of decomposition-based re-weighting; we now discuss this in Sec. 2. Whether the underlying spectral regularization benefits diffusion models is an interesting direction for future work.

Q2: Is the collapse caused by the VAR architecture and generation mechanism, or by insufficient scaling?

A complete causal answer is beyond this paper, but the evidence points to the generation mechanism rather than scale alone. (1) Fig. 8 shows that the logits of different samples of the vanilla model almost coincide; the independent analysis of DiverseAR [1] attributes the limited diversity of Infinity to its bitwise modeling, where overconfident bit-level predictions leave little randomness at early scales, consistent with our finding that structure (and its collapse) forms at early scales. (2) Sampling hyper-parameters are not the main cause: lowering CFG from 4 to 2 only increases LPIPS from 0.433 to 0.455 and DINO from 0.142 to 0.150 (Tab. S9). We also acknowledge that training data and post-training may play a role.

[1] Yang et al., DiverseAR: Boosting Diversity in Bitwise Autoregressive Image Generation, arXiv 2025.

We hope these responses address your concerns, and we are happy to provide further clarification.

Reviewer_rdCiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents an approach to improve the diversity of VAR generated images through the concept of modifying the pivotal tokens thorough a plug and play approach. Similar to Large language models where some tokens determine the quality of response, the authors find that in VAR tokens of some particular scale determines the overall diversity. Based on this the authors propose to modify the pivotal tokens to enhance diversity of VAR. Experiments performed on some benchmarks show that modifying pivotal tokens enhance diversity

Critical strengths
  1. Paper introduces the concept of pivotal tokens, these are tokens that determine the overall diversity of the generation process.
  2. Simialr to some previous works, authors utilize Dino structural similarity statistic to measure to find the overall diversity of generation.
  3. The authors propose a SVD based suppresion approach to modify the pivotal tokens to improve the diversity.
  4. Results over a few benchmarks show performance improvement.
Critical weaknesses
  1. All experiments are performed over small scale benchmarks. The results in ImageNet and large scale T2I benchmarks is missing which gives me the confusion whether the approach scales .
  2. Although methods has been presented to measure diversity and diversity improvement, these misses out theoritical foundations on why these are the best method for the approach.
  3. The authors fail to mention why DINO structural similarity benchmark presents a measure of diversity
  4. Why not do another approach like resampling or noise addition rather than a SVD based suppresion of pivotal component to improve the diversity
Questions for authors
  1. Why does DINO structural similarity benchmark presents a measure of diversity
  2. Why not do another approach like resampling or noise addition rather than a SVD based suppresion of pivotal component to improve the diversity?
10 comments
AuthorsOpenReview ↗

Response to Reviewer rdCi (1/3)

We appreciate your constructive comments and will incorporate the discussions below into the revised paper.

W1: Experiments are on small-scale benchmarks; results on ImageNet and large-scale text-to-image benchmarks are missing.

We would like to clarify the scale of our evaluation: we use COCO 2014 with 30K prompts, COCO 2017 with 5K prompts, GenEval and DPG, which are the standard text-to-image benchmarks used by Infinity and recent VAR works, on three text-to-image VAR models (HART, Infinity-2B and Infinity-8B, up to 8B parameters). In the revision, we further added a per-prompt diversity evaluation with 10K images per setting (1000 prompts × 10 seeds, Tab. 5, Sec. E.8) and a full-resolution evaluation of Infinity-8B (Sec. E.6, Tab. S6).

Regarding ImageNet: our study concerns the diversity of text-to-image VAR models under the same prompt. HART and Infinity are text-conditioned models without class-conditional ImageNet versions, so they cannot be evaluated on ImageNet directly. Class-conditional VAR models (e.g., VAR on ImageNet) address a different setting whose condition is only a class label, and ImageNet metrics such as IS do not measure the diversity among samples under the same condition (see also our response to Reviewer EEaS). Since our analysis and design target the per-prompt diversity of text-to-image VAR models, we extend the evaluation on large-scale text-to-image benchmarks instead.

AuthorsOpenReview ↗

Response to Reviewer rdCi (2/3)

W2: The paper lacks theoretical foundations explaining why the chosen diversity measures are the best choices.

Our goal is a practical training-free method rather than a new diversity measure, and all metrics are standard choices in the literature: FID [1], Recall [2] and Coverage [3] for distribution-level diversity following the COCO protocol of prior works [4]; pairwise LPIPS [5], MSS and Vendi Score [6] with SSCD features [7] following the CADS protocol [8] for per-prompt diversity; the DINO self-similarity distance [9, 10] for structural diversity; and CLIPScore [11] and GenEval [12] for alignment.

These metrics are complementary (now stated in Sec. C.1): distribution-level metrics measure the coverage of the real data, while per-prompt metrics measure the variation among samples of the same prompt; among the latter, LPIPS, MSS and Vendi reflect appearance-level variation and DINO reflects layout- and structure-level variation. Our conclusions are consistent across all of them (Tab. 1, Tab. 5), so they do not rely on any single metric.

Since large generative models remain largely black-box, we ground our design in mechanism-oriented empirical analysis rather than in a formal theory. We find that the structure of an image is formed at the early scales (Observation 1, Figs. 2–3) and that a few pivotal components govern this structure (Observation 2, Figs. 4–5). Their singular value spectrum is dominated by a few components shared across samples; softly suppressing these components reduces the shared structural information, so that the logits of different samples become less concentrated (Fig. 8). The controlled study in Sec. E.7 / Tab. S7 further verifies this mechanism: regularizing only the top-2 singular values reproduces the full diversity gain, whereas regularizing the tail has no effect.

W3 / Q1: Why does the DINO structural similarity statistic measure diversity?

The self-similarity map of DINO ViT patch features is a well-established descriptor of image structure [9, 10]: it is largely insensitive to color and texture but sensitive to layout, pose, object position and shape. For each prompt, we average the pairwise distances between the patch self-similarity matrices of its 10 samples; a higher value means higher structural diversity. We added this explanation in Sec. 3 and the definition in Sec. C.1.

We also verified that it behaves as a diversity measure: (i) it moves consistently with LPIPS and Vendi and inversely with MSS (Tab. S8); (ii) it captures structural variation that appearance metrics miss: raising the temperature τ from 1 to 3 increases LPIPS from 0.433 to 0.492, while the DINO distance only rises from 0.142 to 0.150, and suppressing only the top-2 singular components reaches 0.167 (Tabs. S7, S9). In Figs. 2–3, it is used to locate the scale where the structure forms (Observation 1), not as the only diversity measure.

[1] Heusel et al., GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium, NeurIPS 2017.

[2] Kynkäänniemi et al., Improved Precision and Recall Metric for Assessing Generative Models, NeurIPS 2019.

[3] Naeem et al., Reliable Fidelity and Diversity Metrics for Generative Models, ICML 2020.

[4] Kang et al., Distilling Diffusion Models into Conditional GANs, ECCV 2024.

[5] Zhang et al., The Unreasonable Effectiveness of Deep Features as a Perceptual Metric, CVPR 2018.

[6] Friedman and Dieng, The Vendi Score: A Diversity Evaluation Metric for Machine Learning, TMLR 2023.

[7] Pizzi et al., A Self-Supervised Descriptor for Image Copy Detection, CVPR 2022.

[8] Sadat et al., CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling, ICLR 2024.

[9] Caron et al., Emerging Properties in Self-Supervised Vision Transformers, ICCV 2021.

[10] Tumanyan et al., Splicing ViT Features for Semantic Appearance Transfer, CVPR 2022.

[11] Hessel et al., CLIPScore: A Reference-free Evaluation Metric for Image Captioning, EMNLP 2021.

[12] Ghosh et al., GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment, NeurIPS 2023.

AuthorsOpenReview ↗

Response to Reviewer rdCi (3/3)

W4 / Q2: Why not resampling or noise addition instead of SVD-based suppression?

We added both families of baselines on Infinity-2B (200 COCO 2017 prompts × 10 identical seeds, plus GenEval): resampling with a higher temperature τ, top-p or CFG (main paper Tab. 6; Sec. E.9, Tab. S9, Fig. S1), and noise addition with Gaussian noise on the same early-scale input features, whose strength Δ (relative feature change) covers Δ = 0.32 of Ours (Sec. E.7, Tab. S7).

Table A. Resampling and noise addition vs. DiverseVAR.

TypeSettingLPIPS↑DINO↑CLIP↑GenEval↑
–Infinity-2B0.4330.1420.3170.73
Resamplingτ = 1.5 / 2 / 30.459 / 0.474 / 0.4920.147 / 0.149 / 0.1500.3170.74 / 0.73 / 0.72
Resamplingtop-p = 0.9 / 1.00.423 / 0.4350.140 / 0.1430.3170.73 / 0.72
ResamplingCFG = 2 / 30.455 / 0.4410.150 / 0.1460.315 / 0.3160.71 / 0.73
Noise additionGaussian, Δ = 0.2 / 0.30.525 / 0.5640.159 / 0.1650.3160.71 / 0.67
Oursβ = 0.0070.5060.1600.3170.72
Oursdefault (Δ = 0.32)0.5370.1680.3170.70
  • Resampling barely improves structural diversity: at τ = 3 the DINO distance only rises from 0.142 to 0.150. Under the same GenEval (0.72), a milder DiverseVAR (β = 0.007) achieves higher DINO (0.160 vs. 0.150) and LPIPS (0.506 vs. 0.492) than the strongest resampling baseline.
  • Noise addition at a comparable strength (Δ = 0.3) mainly increases appearance diversity (LPIPS 0.564 vs. 0.537) but yields lower structural diversity (DINO 0.165 vs. 0.168) and lower GenEval (0.67 vs. 0.70).

This is consistent with our analysis: the diversity collapse stems from structural information shared by a few dominant components at early scales (Figs. 4, 5, 7), so suppressing only these components is more targeted than isotropic noise. Regularizing the top-2 singular values alone reproduces the full effect, whereas regularizing the tail is ineffective (Tab. S7).

We hope these responses address your concerns, and we are happy to provide further clarification.

Reviewer_6fzbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DiverseVAR, a training-free method for improving the generative diversity of visual autoregressive (VAR) models. The authors first observe that image structures are primarily formed at early scales and argue that pivotal components play an important role in diversity formation. Based on these observations, they propose Soft-Suppression Regularization (SSR) and Soft-Amplification Regularization (SAR), which modify the singular values of intermediate features to encourage diversity while preserving generation quality.

Experiments on HART, Infinity-2B, and Infinity-8B demonstrate improvements in FID, Recall, and Coverage. Additional ablation studies investigate the effects of different regularization strategies, scales, blocks, and hyperparameters.

Critical strengths

Strengths

  1. The paper is well-written and easy to follow, with clear illustrations of the proposed method.
  2. The generated images demonstrate noticeably improved diversity, and the appendix provides relatively comprehensive ablation studies.
Critical weaknesses

Weaknesses

  1. In Section 3.2, the authors identify pivotal tokens based on their deviations from the mean feature. However, around Lines 300–306, the authors assume that dominant singular values correspond to pivotal components and directly use them for regularization. I am not convinced that these two concepts are equivalent, especially since SVD is performed on the uncentered feature matrix. Figure 7 shows that removing dominant singular components affects generation, but this does not necessarily establish their relationship with pivotal tokens. More importantly, it remains unclear whether the improvement comes from suppressing the proposed pivotal components or simply perturbing the feature representations. I suggest comparing the effects of suppressing dominant and non-dominant singular components, as well as random feature perturbations under comparable perturbation strengths.
  2. In Table 1, FID, Recall, and Coverage are computed using one generated image per text prompt. These metrics characterize the overall generated distribution but do not directly measure the diversity of multiple samples under the same prompt, which is the main problem this paper aims to address. Although Table 2 includes multiple sampling experiments, they are limited to a few category-level prompts on AFHQ and CelebA-HQ, and several Recall scores remain close to zero. I suggest evaluating conditional diversity using multiple samples per prompt on a larger and more diverse prompt set, with metrics such as pairwise LPIPS.
  3. The paper lacks comparisons with simpler diversity-enhancing strategies, such as adjusting sampling temperature or top-kk/top-pp parameters. As shown in Table 3, the GenEval score decreases from 0.730.73 to 0.700.70 on Infinity-2B and from 0.790.79 to 0.760.76 on Infinity-8B. Table S1 also indicates that stronger regularization generally improves diversity metrics at the cost of semantic alignment. Therefore, I would like to know whether the proposed method achieves a better diversity-quality trade-off than these simpler sampling strategies under comparable generation quality.
  4. The proposed method requires SVD on both input and output features, and these operations are applied across all eight blocks at the selected early scales. However, the paper does not report the additional inference latency or memory overhead. Considering the potentially high computational cost of SVD on large feature matrices, I believe these measurements are necessary to assess the practical applicability of the method.

Errors

  1. Around Lines 366–367, the authors state that Infinity-2B and Infinity-8B are evaluated at a resolution of 1024×10241024 \times 1024. However, Appendix C.1 (around Lines 825–829) states that Infinity-8B is evaluated at scale 48 due to GPU memory limitations, which corresponds to 768×768768 \times 768 according to Figure 2. This inconsistency should be clarified, particularly whether the vanilla and modified models are evaluated under identical settings.
  2. In Table 1, the CLIP score of HART on COCO2014 increases from 0.3100.310 to 0.3950.395 after applying the proposed method, whereas the corresponding score on COCO2017 remains unchanged at 0.3090.309. The value 0.3950.395 seems inconsistent with the other results and should be verified.
Questions for authors

Question

Around Equations (4) and (5), the authors describe applying regularization to each singular value. However, Appendix E.5 states that DiverseVAR applies regularization to the top-KK singular values, and Table S5 reports the default setting separately from different KK values. Could the authors clarify the exact implementation of SSR and SAR in the default setting?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 6fzb (1/4)

We appreciate your constructive comments and will incorporate the discussions below into the revised paper.

W1: The relation between dominant singular values and pivotal tokens is assumed; is the gain from suppressing the pivotal component or from merely perturbing the features? Compare dominant vs. non-dominant components and random perturbation under comparable strength.

We agree that the pivotal score is a token-level quantity defined on centered features, whereas SSR acts on the singular values of the uncentered feature matrix, so the two are not strictly equivalent. As stated in Sec. 3.3 ("we assume ...", inspired by [1]), the correspondence between the dominant singular values and the pivotal component is an empirical assumption rather than a theorem, and the experiments below are designed to test it. We use the singular value spectrum as a practical proxy because self-attention entangles token information, making it hard to isolate the pivotal component token by token; the leading direction of the uncentered SVD captures the information shared across tokens.

Following your suggestion, we added a controlled study (Sec. E.7, Tab. S7; Infinity-2B, same scales {4, 6} and blocks; 200 COCO 2017 prompts × 10 identical seeds; GenEval for alignment). The perturbation strength is measured by the relative feature change Δ = ‖F′ − F‖_F / ‖F‖_F.

Table A. Dominant vs. non-dominant components and random perturbation (Tab. S7).

SettingΔLPIPS↑MSS↓Vendi↑DINO↑CLIP↑GenEval↑
Vanilla00.4330.5335.470.1420.3170.73
Top-2 singular values0.330.5340.3937.230.1670.3170.69
Tail singular values0.010.4360.5305.500.1440.3170.73
Tail singular values zeroed0.060.5070.3957.130.1780.3030.53
Gaussian noise0.200.5250.4296.800.1590.3160.71
Gaussian noise0.300.5640.3737.520.1650.3160.67
Ours0.320.5370.3907.280.1680.3170.70
  • The gain comes from the dominant components. Regularizing only the top-2 singular values reproduces the diversity of Ours (DINO 0.167 vs. 0.168), whereas the same regularization on the tail barely changes the features (Δ = 0.01) and yields no gain. Destroying the tail (zeroing) raises DINO but severely degrades alignment (GenEval 0.53).
  • It is not arbitrary perturbation. At a comparable strength (Δ = 0.3), Gaussian noise slightly increases appearance diversity (LPIPS 0.564 vs. 0.537) but yields lower structural diversity (DINO 0.165 vs. 0.168) and lower alignment (GenEval 0.67 vs. 0.70).

Therefore, the diversity gain stems from the targeted suppression of the dominant singular components rather than from feature perturbation per se.

[1] Gu et al., Weighted Nuclear Norm Minimization with Application to Image Denoising, CVPR 2014.

AuthorsOpenReview ↗

Response to Reviewer 6fzb (2/4)

W2: FID, Recall and Coverage use one image per prompt and do not measure the diversity of multiple samples under the same prompt; evaluate conditional diversity on a larger prompt set with metrics such as pairwise LPIPS.

Following StudioGAN [1], we use FID, Recall and Coverage to measure the diversity of the generated distribution. As suggested, to make the evaluation complete, we further follow CADS [2] and Park et al. [3] and add a per-prompt evaluation (main paper Sec. 4.3, Tab. 5; details in Sec. C.1, Sec. E.8, Tab. S8): the first caption of the first 1000 images in COCO 2017, 10 samples per prompt with fixed seeds 0–9, identical seeds for the vanilla model and DiverseVAR (10K images per setting). For each prompt we compute the mean pairwise LPIPS (as you suggested), the MSS and Vendi Score with SSCD features following CADS, and the DINO structural distance (mean pairwise distance between DINO self-similarity maps), and average over prompts.

Table B. Per-prompt diversity on Infinity-2B (1000 prompts × 10 seeds, Tab. 5 / Tab. S8).

MethodLPIPS↑MSS↓Vendi↑DINO↑CLIP↑
Vanilla0.4310.5355.440.1440.313
Ours0.5360.3917.260.1710.313

DiverseVAR improves all four per-prompt metrics (LPIPS +24%, MSS −27%, Vendi +33%, DINO +19%; standard errors < 0.03) while keeping the CLIP Score unchanged. The improvement is consistent across prompts: DiverseVAR is more diverse than the vanilla model on 99.9%, 100%, 100% and 98.3% of the 1000 prompts for LPIPS, MSS, Vendi and DINO, respectively (Wilcoxon signed-rank test, p < 10⁻¹⁶⁰). The higher DINO distance shows that the gain covers structure as well as appearance.

Regarding Tab. 2, the near-zero Recall comes from the category-level prompts of AFHQ and CelebA-HQ; the per-prompt protocol above addresses this limitation.

W3: Compare with simpler strategies (temperature, top-k/top-p) under comparable generation quality.

We added these baselines (main paper Tab. 6; full results in Sec. E.9, Tab. S9, Fig. S1; same protocol as Table A). The default sampling setting follows the official Infinity script (τ = 1, top-p = 0.97, CFG = 4).

Table C. Comparison with sampling-based baselines (Tab. S9).

SettingLPIPS↑MSS↓Vendi↑DINO↑CLIP↑GenEval↑
Vanilla (τ=1, p=0.97, CFG=4)0.4330.5335.470.1420.3170.73
τ = 1.5 / 2 / 30.459 / 0.474 / 0.4920.508 / 0.489 / 0.4715.78 / 6.02 / 6.240.147 / 0.149 / 0.1500.3170.74 / 0.73 / 0.72
top-p = 0.9 / 1.00.423 / 0.4350.544 / 0.5315.34 / 5.490.140 / 0.1430.3170.73 / 0.72
CFG = 2 / 30.455 / 0.4410.499 / 0.5235.90 / 5.590.150 / 0.1460.315 / 0.3160.71 / 0.73
Ours (β = 0.007)0.5060.4376.670.1600.3170.72
Ours0.5370.3907.280.1680.3170.70
  • Adjusting the sampling hyper-parameters keeps GenEval close to the vanilla model but barely improves diversity: even with τ = 3, DINO only rises from 0.142 to 0.150.
  • Under comparable quality, a milder setting of DiverseVAR (β = 0.007, from Tab. S4) achieves the same GenEval as the strongest sampling baseline (τ = 3, 0.72) with higher DINO (0.160 vs. 0.150) and LPIPS (0.506 vs. 0.492). In the trade-off plot (Fig. S1), both DiverseVAR points lie above all sampling baselines.
  • Top-k is not applicable to Infinity, whose bitwise prediction has only two outcomes per bit (also noted by [4]), so we evaluate temperature and top-p.

[1] Kang et al., StudioGAN: A Taxonomy and Benchmark of GANs for Image Synthesis, TPAMI 2023.

[2] Sadat et al., CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling, ICLR 2024.

[3] Park et al., DiverseVAR: Balancing Diversity and Quality of Next-Scale Visual Autoregressive Models, ECCV 2026.

[4] Yang et al., DiverseAR: Boosting Diversity in Bitwise Autoregressive Image Generation, arXiv 2025.

AuthorsOpenReview ↗

Response to Reviewer 6fzb (3/4)

W4: The additional inference latency and memory overhead of SVD are not reported.

Thanks for the reminder. We measured the overhead on Infinity-2B (main paper Sec. 4.3; details in Sec. E.10, Tab. S10; NVIDIA A100 40 GB, 1024×1024, batch size 1, average over 50 samples, VAE decoding included).

Table D. Inference overhead on Infinity-2B (Tab. S10).

MethodLatency (s)↓Overhead↓Peak memory (GB)↓
Infinity-2B1.831–15.75
+Ours1.931+5.4%15.75

DiverseVAR adds only 5.4% latency (1.831 s → 1.931 s) and no measurable memory (15.75 GB for both). The overhead is small because SSR/SAR are applied only at scales 4 and 6, where the token maps contain 16 and 36 tokens, so each SVD is computed on a small 16×2048 or 36×2048 matrix; the high-resolution later scales (e.g., 4096 tokens at scale 64), which dominate the inference cost, are left unchanged.

E1: Infinity-8B is said to be evaluated at 1024×1024 in Sec. 4.1 but at scale 48 (768×768) in App. C.1; are the vanilla and modified models evaluated under identical settings?

Thank you for pointing out this inconsistency. Infinity-2B is evaluated at 1024×1024 throughout. Infinity-8B runs out of memory on a 40 GB A100, so its results in the original submission were obtained at scale 48 (768×768); the vanilla model and DiverseVAR use exactly the same resolution, seeds, CFG and sampling settings, differing only in whether SSR/SAR is enabled. We corrected the description in Sec. 4.1.

To verify that this setting does not affect our conclusions, we re-evaluated Infinity-8B at the full resolution (1024×1024, all 13 scales) on a 96 GB GPU with identical seeds (Sec. E.6, Tab. S6).

Table E. Infinity-8B at scale 48 and at full resolution (Tab. S6).

ResolutionMethodTwo Obj.↑Position↑GenEval↑Recall↑Cov.↑FID↓CLIP↑
768×768 (scale 48)Vanilla0.900.610.790.5630.89229.470.319
+Ours0.890.590.760.5850.89225.010.316
1024×1024 (scale 64)Vanilla0.910.600.790.5640.88629.290.319
+Ours0.880.560.750.5910.89425.620.317

The diversity improvement is preserved and even more pronounced at full resolution (Recall +0.027 vs. +0.022; Coverage +0.008 vs. unchanged), the CLIP Score is nearly unchanged, and the GenEval decrease (0.79 → 0.75) is comparable to that at scale 48 (0.79 → 0.76). Therefore, the conclusions drawn at scale 48 also hold at full resolution.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all reviewers (R1=6fzb, R2=rdCi, R3=yutr, R4=EEaS) for their thoughtful and constructive feedback. We are encouraged that they find the paper well-written, clear and easy to follow (R1, R3, R4), with a clear motivation and a reasonable, grounded connection between the observations on VAR generation and the proposed technique (R3); that the pivotal-component perspective and the SVD-based suppression are interesting contributions (R2); that the method is simple, efficient and easily integrated into pre-trained VAR models (R3); that the generated images show noticeably improved diversity (R1) with performance improvements across benchmarks (R2); and that the experiments are rich, covering multiple benchmarks (R4) with comprehensive ablation studies in the appendix (R1).

The revised parts of the paper are highlighted in yellow. The main revisions are:

  • Per-prompt diversity under the CADS protocol (R1, R2, R3): 1000 COCO 2017 prompts × 10 identical seeds, reporting pairwise LPIPS, MSS, Vendi Score and DINO structural distance (main paper Tab. 5; details in Sec. C.1, Sec. E.8, Tab. S8). DiverseVAR improves all four metrics on 98.3–100% of the prompts.
  • Dominant vs. non-dominant singular components and random perturbation under matched perturbation strength (R1, R2) (Sec. E.7, Tab. S7).
  • Comparison with sampling-based baselines (temperature, top-p, CFG) and a diversity–quality trade-off plot (R1, R2) (main paper Tab. 6; details in Sec. E.9, Tab. S9, Fig. S1).
  • Inference overhead (R1, R3): +5.4% latency and no extra memory, vs. +24.3% for Park et al. (2026) on the same GPU (main paper Sec. 4.3; details in Sec. E.10, Tab. S10).
  • Infinity-8B at full resolution (1024×1024, all 13 scales) (R1) (Sec. E.6, Tab. S6).
  • Structural diversity under complex scene prompts with fixed seeds (R3, R4) (Fig. S6).
  • Related work and claim scope (R3, R4): discussion of Park et al. (2026), DiverseAR (Yang et al., 2025) and Cho et al. (2026); the claim is restricted to mainstream text-to-image VAR models such as Infinity.
  • Corrections (R1): the HART CLIP typo in Tab. 1, the resolution description of Infinity-8B, the description of the default SSR/SAR setting, and the rerun Color Attri. values in Tabs. S3–S5.

We use W to abbreviate Weaknesses, Q to represent Questions, and E for the reviewer's other comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.