acceptodds
Under review as a conference paper at ICLR 2027

Free Diversity for Efficient Test-Time Scaling in Visual Autoregressive Models

Abstract

Visual Autoregressive (VAR) models generate images by next-scale prediction, and their earliest scales strongly shape the final image. This sensitivity has been taken to imply that these scales must remain text-conditioned. We revisit this assumption and find that text conditioning has little effect at the coarsest scales: sampling them unconditionally leaves mean quality nearly unchanged while substantially diversifying the generated images. Existing diversity-boosting methods for VAR improve best-of- performance at the cost of either latency or mean quality; in contrast, unconditioning the earliest scales incurs no extra latency and a smaller drop in mean quality while yielding higher diversity and best-of- performance. Building on this finding, we propose early attention pruning (\eap), an efficient verification method that prunes failing candidates at early scales using cross-attention signals, without decoding intermediate images. Our training-free pipeline outperforms existing diversity-boosting and test-time scaling methods on T2I-CompBench++ and GenEval2 with much less computation, and generalizes across VAR architectures and model sizes.

Primary area:
generative models
Keywords:
Text-to-Image Generation, Visual Autoregressive Models, Test-Time Scaling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fdVfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether the earliest scales in text-to-image visual autoregressive models need text conditioning, and uses its findings to improve best-of-N generation. Attribute probes and conditioning interventions suggest weak object-level attribute binding at the coarsest scales. The proposed unconditioned anchors (UA) sample the first two scales without text conditioning to increase candidate diversity without additional generation steps. Early attention pruning (EAP) then uses cross-attention signals to discard candidates before intermediate-image decoding, followed by preview verification in a cascade. Experiments on Infinity and Switti report diversity, mean quality, oracle best-of-N scores, and actual selection performance on compositional benchmarks. The central claim is that early unconditioning incurs little mean-quality loss while the combined pipeline improves the quality–compute trade-off.

Critical strengths
  1. Some settings show useful efficiency gains. On Infinity-2B, T2I-CompBench with IVL, the cascade improves the score from approximately 0.524 to 0.538 over standard preview BoN while reducing computation by 27% and latency by 29%. The equal-compute curves provide further support.

  2. The attribute probes control for object identity. The noun-only baseline separates attribute information in visual features from predictable object–attribute associations. Scale-specific conditioning interventions further support the method’s motivation.

The efficiency gains support practical value, while the controlled probes provide an empirical motivation for how early scales are treated.

Critical weaknesses
  1. UA’s distinct contribution over simple alternatives remains unclear. Table 17 gives random-token mean/oracle scores of approximately 0.469/0.645 versus 0.468/0.650 for UA, without comparing actual final selection. Weak guidance also reaches 0.5351 versus UA’s 0.5380 in Table 3. Thus, higher oracle scores alone do not establish a meaningful practical advantage from the proposed mechanism.

  2. The evidence does not yet establish negligible quality loss or reliable useful diversity. UA lowers mean scores on the reported Infinity-8B and Switti tasks, including Switti T2I from 0.4608 to 0.4539. EAP separately reduces GenEval2 GM from 0.3651 to 0.3540 on the same UA pool, an acknowledged pruning trade-off. Equal-compute gains are valuable but do not establish that these capability losses are minor. Layout-Vendi can also reward incorrect layouts, and final-selection gains are mixed across models and selectors. Paired uncertainty, failure severity, and diversity among valid candidates are needed to substantiate the central claims.

Questions for authors
  1. How do random tokens and UA compare with the same final selector and matched candidate counts and compute, alongside standard sampling and weak guidance?

  2. Building on Tables 14–16, please report paired difference intervals, the fraction of prompts with substantial degradation under a justified criterion, and key failure categories. Additional generation seeds should test the stability of the small gains and losses.

  3. Does diversity increase among candidates that satisfy the prompt? Please assess candidate validity and diversity within the valid subset, with a justified criterion. A targeted independent or blinded assessment of selected-image quality could also reveal artifacts missed by compositional scores.

  4. How were the pruning scale and attention layers chosen, and were the preliminary GenEval2 prompts independent of the final test set?

10 comments
AuthorsOpenReview ↗

Rebuttal (1/6)

We thank the reviewer for the thorough and constructive review. All new results below are computed from the stored candidate pools of the paper (Infinity-2B, N = 16 unless stated). We address each point below.

W1 / Q1. Comparing ways to widen the coarsest anchors: random tokens, weak guidance, and UA.

Setup. We compare four candidate pools on all 2,400 T2I-CompBench prompts: standard sampling, random tokens, weak guidance (ww=1), and UA (ww=0). Values in parentheses are paired differences from UA, with 95% bootstrap confidence intervals over prompts:

Table R8. UA vs. simple alternatives under the same selector (T2I-CompBench, N = 16).

PoolMeanOracleSelected Image (IVL)Selected Image (IR)
Standard sampling0.47000.6183 (−0.032 [−0.039, −0.026])0.5285 (−0.009 [−0.017, −0.002])0.5159 (−0.015 [−0.023, −0.008])
Random tokens0.46900.6449 (−0.005 [−0.010, −0.001])0.5365 (−0.002 [−0.009, +0.006])0.5302 (−0.001 [−0.009, +0.007])
Weak guidance (w=1)0.47470.6355 (−0.015 [−0.020, −0.010])0.5351 (−0.003 [−0.010, +0.005])0.5270 (−0.004 [−0.012, +0.003])
UA (w=0)0.46770.65030.53800.5312

The gain comes from where the anchors are widened, not from a particular way of widening them. Our claim is not that UA is the only way to widen the anchors. Sec. 3 identifies the coarsest scales as ones that do not need the text condition; intervening there widens the candidate pool, and the wider pool leads to a better selected image. Random tokens, weak guidance, and UA are three such interventions. Among them, UA reaches the highest oracle, significantly above both alternatives, and the highest selected-image score with both verifiers, although after selection the three are statistically tied. Random tokens and UA both select a significantly better image than standard sampling with both verifiers. We use UA because it is the simplest: it changes only a guidance weight and injects no codes.

In the revised PDF, we have added the selected-image columns to Tab. 25.

AuthorsOpenReview ↗

Rebuttal (2/6)

W2 / Q2. Paired-difference intervals, the fraction of prompts with substantial degradation under a justified criterion, key failure categories, and additional generation seeds.

(1) Paired-difference intervals for Tables 14–16. In the revised PDF, we have added paired 95% bootstrap intervals over prompts to every row of Tables 14–16 (now 15–17 in the updated PDF). Below, we show a simplified version focusing on UA’s effect on the pool mean, EAP’s effect on the selected image, and the resulting test-time scaling gain:

Table R9. Paired differences by category: UA’s effect on the pool mean, EAP’s effect on the selected image, and the test-time scaling gain.

Subtask (prompts)UA: pool meanEAP: selected imageTTS gain
color (300)−0.027 [−0.041, −0.013]+0.001 [−0.007, +0.010]+0.015 [−0.002, +0.032]
shape (300)+0.013 [−0.001, +0.026]−0.008 [−0.023, +0.008]+0.039 [+0.017, +0.061]
texture (300)+0.001 [−0.013, +0.014]+0.005 [−0.002, +0.013]+0.026 [+0.008, +0.045]
spatial (300)+0.015 [−0.001, +0.031]+0.002 [−0.017, +0.020]+0.074 [+0.045, +0.106]
3D-spatial (300)−0.011 [−0.020, −0.003]+0.018 [+0.006, +0.029]+0.003 [−0.013, +0.020]
non-spatial (300)−0.001 [−0.002, −0.000]+0.000 [−0.001, +0.001]+0.001 [−0.001, +0.002]
numeracy (300)+0.000 [−0.011, +0.011]−0.011 [−0.022, +0.001]+0.040 [+0.017, +0.063]
complex (300)−0.008 [−0.013, −0.003]−0.001 [−0.006, +0.004]+0.001 [−0.007, +0.009]
T2I-CompBench overall (2,400)−0.002 [−0.006, +0.002]+0.001 [−0.003, +0.005]+0.025 [+0.018, +0.031]
object (800)−0.004 [−0.010, +0.002]+0.006 [+0.001, +0.012]+0.020 [+0.011, +0.030]
count (800)−0.010 [−0.017, −0.002]−0.002 [−0.011, +0.006]+0.048 [+0.033, +0.063]
attribute (721)+0.007 [−0.001, +0.014]−0.005 [−0.015, +0.005]+0.043 [+0.026, +0.060]
position (519)+0.010 [−0.004, +0.024]−0.011 [−0.025, +0.003]+0.088 [+0.060, +0.117]
verb (82)−0.006 [−0.039, +0.023]+0.016 [−0.017, +0.054]+0.061 [−0.001, +0.124]
GenEval2 AM (800)−0.001 [−0.006, +0.004]−0.001 [−0.005, +0.003]+0.042 [+0.033, +0.052]
GenEval2 GM (800)+0.001 [−0.006, +0.008]−0.011 [−0.020, −0.001]+0.065 [+0.046, +0.084]
  • UA: pool mean: (UA − standard), averaged over the 16 candidates. A statistically significant drop appears only in a few categories: color, 3D-spatial, non-spatial, complex, and GenEval2 count.
  • EAP: selected image (UA cascade − UA BoN@s9), i.e., the effect of pruning on the selected image. Pruning makes no T2I-CompBench category and no GenEval2 skill significantly worse, and it even yields significant gains on 3D-spatial and object atoms. Its only significant loss is the GM drop the reviewer cites (0.3651 → 0.3540), which vanishes under AM. This is because GM multiplies the atom scores, so a single wrong atom pulls a sample’s score toward zero. A 30% cut is aggressive for GenEval2, and a smaller cut loses less (App. C.2). Even so, at equal compute, the cascade still outperforms best-of-N (Fig. 4 in the main paper).
  • TTS gain: our method (UA cascade, N = 16) minus standard best-of-7, both with the IVL verifier. Best-of-7 is the cheapest standard best-of-N that uses at least as much compute as our cascade (cost computed as in Fig. 4). Despite significant mean losses in some categories, the practical test-time scaling gains are significant in most categories, including GenEval2 count and GM, where UA and EAP respectively incur significant costs. Gains are not significant on color, 3D-spatial, non-spatial, complex, and verb
AuthorsOpenReview ↗

Rebuttal (3/6)

(2) Criterion for substantial degradation. We define substantial degradation as losing the ability to solve a prompt: best-of-N fails on a prompt when none of its candidates satisfies it, regardless of how good the selector is. A candidate is valid when it satisfies the prompt under the benchmark's own criterion. On GenEval2, a candidate is valid when every atom is judged correct (Soft-TIFA atom score ≥ 0.5). On T2I-CompBench, we use the five categories whose scores can determine whether a candidate satisfies the prompt: spatial (detector score > 0), numeracy (every count correct), and color, shape, and texture (BLIP-VQA score ≥ 0.8; the conclusions hold for any threshold between 0.25 and 0.8). These cover 1,500 prompts in total. The other three categories have no such threshold: 3D-spatial is a continuous depth-based score, non-spatial is a CLIP similarity, and complex is an average of three different scorers. These are well-covered by the mean. A prompt is lost when standard sampling solves it at least k times out of N = 16 and UA never does, and gained in the reverse case. A larger k indicates a more severe loss.

As a reference, we compute the chance rate on the same 16 + 16 candidates: we pool each prompt's 32 candidates and randomly split them into two sets of 16 (2,000 permutations). This gives the loss rate expected if UA's candidates were interchangeable with standard ones. Each cell shows lost / gained rates, with the chance loss rate in parentheses. *: lost exceeds chance (permutation p < 0.05).

Table R10. Prompts lost / gained by UA, relative to chance.

Model, benchmark (prompts)k ≥ 1k ≥ 2k ≥ 3k ≥ 4
Infinity-2B, GenEval2 (800)4.0 / 7.5% (4.7%)1.5 / 3.4% (1.4%)0.8 / 2.0% (0.5%)0.1 / 1.5% (0.2%)
Infinity-2B, T2I-CompBench (1,500)4.1 / 8.3% (4.7%)1.3 / 4.8% (1.5%)0.7 / 2.9% (0.6%)0.5* / 1.9% (0.2%)
Infinity-8B, GenEval2 (800)5.6 / 8.6% (5.1%)2.8* / 4.4% (1.5%)1.9* / 2.8% (0.6%)1.0* / 2.2% (0.3%)
Infinity-8B, T2I-CompBench (1,500)1.9 / 5.7% (2.7%)0.6 / 3.6% (1.0%)0.3 / 2.0% (0.3%)0.0 / 1.4% (0.1%)
Switti, GenEval2 (800)4.5 / 5.1% (4.7%)1.4 / 1.9% (1.5%)0.8 / 0.6% (0.5%)0.1 / 0.4% (0.1%)
Switti, T2I-CompBench (1,500)3.4 / 4.6% (4.0%)1.2 / 1.5% (1.3%)0.4 / 0.7% (0.5%)0.1 / 0.3% (0.2%)
  • UA loses prompts at the chance rate in 20 of the 24 cells. Simply redrawing the pool loses just as many.
  • On Infinity, gained exceeds lost in every cell, e.g., 4.8% gained vs. 1.3% lost of T2I-CompBench prompts at k ≥ 2 on Infinity-2B.
  • On Switti, both losses and gains remain at the chance level.

EAP. The same question applies to pruning: does EAP's 30% cut remove every valid candidate of a prompt? For the UA pool that the cascade prunes, we count the prompts with at least k valid candidates, none of which survives the cut. The chance rate can be computed exactly: cutting the same number of candidates at random removes all v valid ones with probability C(c, v) / C(16, v), where c is the number of candidates cut. Values in parentheses are the rates for a random cut of the same size. *: above that rate (exact one-sided test, p < 0.05).

Table R11. Prompts whose valid candidates are all removed by EAP, relative to a random cut.

Model, benchmark (prompts)k ≥ 1k ≥ 2k ≥ 3k ≥ 4
Infinity-2B, GenEval2 (800)1.5% (2.7%)0.0% (0.4%)0.0% (0.1%)0.0% (0.0%)
Infinity-2B, T2I-CompBench (1,500)1.9% (2.4%)0.1% (0.4%)0.0% (0.1%)0.0% (0.0%)
Infinity-8B, GenEval2 (800)2.9% (2.9%)0.6% (0.5%)0.1% (0.1%)0.0% (0.0%)
Infinity-8B, T2I-CompBench (1,500)1.3% (1.5%)0.3% (0.2%)0.1% (0.0%)0.1% (0.0%)
Switti, GenEval2 (800)2.1% (2.7%)0.0% (0.4%)0.0% (0.0%)0.0% (0.0%)
Switti, T2I-CompBench (1,500)2.0% (2.6%)0.3% (0.5%)0.2% (0.1%)0.1%* (0.0%)
  • In 23 of the 24 cells, EAP removes all valid candidates no more often than a random cut does; the single exception involves 2 of 1,500 prompts.
  • When a prompt has two or more valid candidates, EAP almost never removes them all (0.0–0.6%).
Reviewer_KdbPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free framework to improve diversity and test-time scaling efficiency in VAR models. It combines unconditional sampling at the earliest scales with cross-attention-based early candidate pruning and subsequent verification. The method is evaluated on Infinity-2B, Infinity-8B, and Switti, with the main experiments conducted on T2I-CompBench++ and GenEval2.

Critical strengths

1.The paper presents a clear and easy-to-understand idea.

2.The experiments are relatively comprehensive, validating the method across Infinity-2B, Infinity-8B, and Switti.

Critical weaknesses
  1. The paper’s methodological novelty remains limited. Its core ideas, including increasing diversity at early VAR scales and evaluating candidate generation paths at intermediate scales, have already been explored in prior work [3,4]. The authors’ response does not sufficiently demonstrate a substantive technical advance beyond these ideas.

  2. The paper’s motivation and supporting analysis remain insufficient. Although the paper provides probing and intervention experiments, it does not adequately justify the need for further diversity enhancement across the evaluated backbones or fully explain why their diversity differs. In particular, Switti exhibits substantially higher layout diversity than Infinity, but the reasons for this difference remain unclear.

  3. The scope of the reported diversity limitations needs further clarification. In my own experiments, I have not observed obvious diversity limitations in the original VAR [1] or its variant HART [2]. VAR also achieves the highest IS reported in Table 1 of [1]. I strongly encourage the authors to conduct additional analyses and clarify whether the reported limitations are specific to particular backbones and sampling settings, rather than a general property of VAR models.

References:

[1] Tian et al. Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction. NeurIPS 2024.

[2] Tang et al. HART: Efficient Visual Generation with Hybrid Autoregressive Transformer. ICLR 2025.

[3] Wang et al. Diversity Has Always Been There in Your Visual Autoregressive Models. 2025.

[4] Yang et al. DiverseAR: Boosting Diversity in Bitwise Autoregressive Image Generation. 2025.

Questions for authors

None

7 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for the constructive review. We address each point below.

W1. Explanation of why diversity differs across backbones.

First, we would like to clarify that our paper neither claims a general diversity limitation of VAR nor attempts to propose a mechanism why diversity differs across models. Our probing and intervention experiments serve to locate where intervening effectively boosts generation diversity, not to claim that these scales impose significant diversity limitations.

As for the diversity gap between backbones, Infinity and Switti differ in nearly every component: code format (32 independent bits vs. a 4096-entry codebook), text encoder, attention structure, training data, and training techniques such as Infinity's Bitwise Self-Correction. Isolating the exact cause of the diversity gap would require pretraining ablations, which are beyond the scope of this work. As a practical exploration, we compare the diversity of Infinity-2B and Infinity-8B, a controlled comparison in which the two models differ primarily in model size. Following Reviewer FUzx's suggestion, we additionally measure (1) semantic diversity using DINOv2 CLS embeddings, (2) color diversity using the total variation between RGB histograms, and (3) structural diversity using color-free luminance maps, in addition to the Layout-Vendi reported in the main paper:

Table R7. Diversity across backbones and model sizes, standard sampling → UA.

Model, benchmarkLayout-VendiSemanticsColorStructure
Infinity-2B, T2I-CompBench (2,400)7.07 → 9.630.205 → 0.3200.325 → 0.5200.139 → 0.242
Infinity-8B, T2I-CompBench (2,400)4.77 → 6.480.109 → 0.1660.186 → 0.2920.068 → 0.106
Switti, T2I-CompBench (2,400)10.47 → 11.490.314 → 0.3730.576 → 0.7050.254 → 0.312
Infinity-2B, GenEval2 (800)6.94 → 9.290.163 → 0.2380.349 → 0.5300.125 → 0.217
Infinity-8B, GenEval2 (800)4.80 → 6.250.081 → 0.1180.180 → 0.2760.060 → 0.088
Switti, GenEval2 (800)10.79 → 11.550.241 → 0.2780.587 → 0.6910.239 → 0.288

As shown in the table above, scaling up the model alone, with an identical architecture, leads to a substantial change in diversity. This suggests that diversity depends on many factors beyond architecture; understanding why it differs across VAR backbones remains an interesting direction for future work, and we do not propose a mechanism for it here.

AuthorsOpenReview ↗

Rebuttal (2/2)

W2. The scope of diversity limitations.

We agree that limited diversity is not an inherent property of VAR models, and our paper does not claim that it is. Rather, we propose a way to further boost diversity at no extra computational cost, which is useful regardless of the base model's diversity. As discussed in Related Work (Sec. 2 of the main paper, "Guidance and diversity" paragraph), our motivation is more closely related to the observations of Sadat et al. [1] and Kynkäänniemi et al. [2] that conditioning or guidance at high noise levels in diffusion models may harm diversity. DiverseVAR [3] brought these ideas to VAR but observed a sharp diversity–quality trade-off and required scale-travel to recover quality. This is consistent with the view that the earliest scales are highly sensitive to intervention [4, 5] (Sec. 1 of the main paper, L034–039). We find that this trade-off largely disappears when conditioning is removed only at the scales before attributes localize (1×1 and 2×2; Sec. 3.2 of the main paper), whereas extending the removal to later scales reintroduces the trade-off (Tab. 3 of the main paper).

Furthermore, we note that IS is computed over samples pooled across all classes and thus reflects inter-class diversity. In contrast, we mainly address text-to-image generation, where diversity is measured per prompt. The more relevant metric is therefore recall, on which the original VAR [6] does not clearly outperform other architectures.

References

[1] Sadat, Seyedmorteza, et al. "CADS: Unleashing the diversity of diffusion models through condition-annealed sampling." International Conference on Learning Representations. Vol. 2024. 2024.

[2] Kynkäänniemi, Tuomas, et al. "Applying guidance in a limited interval improves sample and distribution quality in diffusion models." Advances in Neural Information Processing Systems 37 (2024): 122458-122483.

[3] Park, Mingue, et al. "DiverseVAR: Balancing Diversity and Quality of Next-Scale Visual Autoregressive Models." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026.

[4] Tang, Weidong, et al. "Inference-Time Scaling for Visual AutoRegressive Modeling by Searching Representative Samples." Chinese Conference on Pattern Recognition and Computer Vision (PRCV). Singapore: Springer Nature Singapore, 2025.

[5] Liu, Keli, et al. "StepVAR: Structure-Texture Guided Pruning for Visual Autoregressive Models." Proceedings of the 2026 International Conference on Multimedia Retrieval. 2026.

[6] Tian, Keyu, et al. "Visual autoregressive modeling: Scalable image generation via next-scale prediction." Advances in neural information processing systems 37 (2024): 84839-84865.

Thank you for the clarifications. However, my concerns regarding the paper’s motivation and methodological novelty remain.

  1. If a substantial diversity limitation has not been clearly demonstrated in the evaluated VAR models, the practical value of further increasing diversity remains insufficiently justified, particularly when the full multi-candidate pipeline introduces additional latency relative to single-sample generation or when average sample quality decreases.

  2. Closely related ideas have already been explored. Early-scale interventions to enhance diversity are studied in [1], while DiverseAR [2] combines early-stage distribution smoothing with intermediate-stage candidate-path evaluation and selection. The paper does not sufficiently discuss or compare against these closely related approaches, leaving its technical advance beyond prior work unclear.

Based on these two concerns, I maintain my original rating.

[1] Wang et al. Diversity Has Always Been There in Your Visual Autoregressive Models.

[2] Yang et al. DiverseAR: Boosting Diversity in Bitwise Autoregressive Image Generation.

Reviewer_FUzxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a simple test-time intervention technique for VAR models to increase the generation diversity without significant computational overhead. The authors observe that text conditioning has minimal effect at the early scales in VAR, and removing the text conditions in those steps remains the outputs almost unchanged. The proposed early attention pruning (EAP) leverages such property by using unconditional anchors in early stages and then pruning out candidates in the later scales based on the cross-attention verification. The evaluation suggests that this method achieves enhanced diversity in VARs while using notably less compute than alternative approaches.

Critical strengths
  • Clear and coherent writing. The paper is well written and easy to follow. The connection between the empirical observations to the proposed method is also clearly established.

  • Practical advantage in efficiency. The paper shows practical gains on the computational cost while matching or outperforming previous methods in terms of generation diversity.

  • Intuitive and effective method. The early attention pruning (EAP) technique is straightforward and grounded in the properties of scale-wise generation in VAR.

Critical weaknesses
  • Unclear connection between attribute localization and generative diversity. In the paper, object-level attribute localization plays a critical role in forming the method, but its relationship to diversity seems yet unclear and weak. Diversity would take into account multiple factors of the image such as composition, layout, style, color, and other details, but the current methods seems to be quite focused on object composition. Since the main evaluations are done on T2I-CompBench++ and GenEval2 which also largely deal with object compositions, the proposed method could be specifically targeting for achieving higher scores on such benchmarks, rather than fundamentally "improving the diversity" of images. Could the authors elaborate further on this?

  • Underlying cause of limited diversity in VAR remains unclear. The paper prosents an effective test-time technique, but I'm still curious about whether the observed diversity collapse primarily stems for the scale-wise generation scheme of VAR, or the lack of large-scale training as in diffusion models. In other words, would the diversity issue be resolved with larger scales of VAR models to come, or would it remain as an issue? I believe this discussion is critical in understanding whether the proposed technique is more of an instant plug-in for the current state of small VAR models, or it could consistently be used as a standard technique in the field of VAR-based generation in the future.

Questions for authors
  • Would the proposed technique be applcable to VAR-based video generation models such as InfinityStar [1]? Reducing computation cost would be more valuable in video generation as the default required time and TFLOPs is higher than generating a single image.

[1] InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generation, Liu et al., NeurIPS 2025

5 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

We thank the reviewer for valuable feedback. We address each point below.

W1. How attribute localization relates to diversity, and diversity beyond object composition.

(1) Relationship between attribute localization and diversity. To clarify, localization does not itself create diversity; removing the text condition does. Localization tells us where this intervention costs little and when to prune.

  • Where to drop the condition. Probing (Sec. 3.2.1 of the main paper) shows that the 1×1 and 2×2 scales carry little object-level attribute information beyond the noun; instead, they encode object-agnostic factors such as global color statistics and coarse layout. unconditioned anchors (UA) sample only these scales without the text conditioning, thereby increasing variation in these global factors. Text conditioning is restored from the next scale onward, where attributes begin to localize and removing the condition starts to degrade mean quality.
  • When to prune. EAP inspects cross-attention at scale 5, after attributes have localized, and discards candidates whose attention fails to separate the objects or to bind attributes to them.

(2) Diversity beyond object composition. We agree that diversity spans multiple aspects beyond object composition. In the main paper, we report Layout-Vendi (Fig. 2) and conduct experiments on Pick-a-Pic (Tab. 6), a human-preference benchmark whose prompts are generic and non-compositional (e.g. “insanely detailed portrait, wise man”). These results show that UA consistently increases Layout-Vendi on T2I-CompBench++ and GenEval-2, and raises the oracle gain on Pick-a-Pic. Condensed tables are provided below for reference:

Table R1. Layout-Vendi, standard sampling vs. UA (N = 16).

ModelT2I-CompBench++ Layout-VendiGenEval2 (AM) Layout-Vendi
Infinity-2B-vanilla7.076.94
Infinity-2B-UA, A={0, 1}9.639.29
Switti-vanilla10.4710.79
Switti-UA, A={0, 1}11.4911.55

Table R2. Pick-a-Pic (500 prompts): per-sample mean and best-of-16 oracle.

ModelHPSv2.1 meanHPSv2.1 oracleImageReward meanImageReward oracle
Infinity-2B-vanilla0.29730.31421.0451.423
Infinity-2B-UA, A={0, 1}0.29440.31820.9921.492

To broaden the scope of our diversity evaluation, we additionally measure (1) semantic diversity using DINOv2 CLS embeddings, (2) color diversity using the total variation between RGB histograms, and (3) structural diversity using color-free luminance maps. As shown in the table below, UA significantly improves all diversity metrics (paired t-test, t ≥ 26):

Table R3. Diversity on four axes, Infinity-2B, standard sampling → UA.

BenchmarkLayout-VendiSemanticsColorStructure
T2I-CompBench (2,400)7.07 → 9.63 (×1.36)0.205 → 0.320 (×1.56)0.325 → 0.520 (×1.60)0.139 → 0.242 (×1.74)
GenEval2 (800)6.94 → 9.29 (×1.34)0.163 → 0.238 (×1.46)0.349 → 0.530 (×1.52)0.125 → 0.217 (×1.73)
Pick-a-Pic (500)6.71 → 9.18 (×1.37)0.198 → 0.317 (×1.60)0.293 → 0.492 (×1.68)0.125 → 0.228 (×1.82)

In the revised PDF, we have added these diversity metrics (App. B.1) and clarified that understanding attribute localization allows us to determine where diversity can be effectively introduced during generation (Sec. 3).

W2. Underlying cause of limited diversity in VAR remains unclear.

First, we would like to clarify that our method does not presuppose a diversity deficiency in VAR models. Rather, it further boosts diversity at no extra computational cost, which is useful regardless of the base model's level of diversity.

Moreover, we additionally compare the diversity of Infinity-2B and Infinity-8B. As shown in the following table, scaling up the model does not increase diversity:

Table R4. Diversity and mean compositional generation scores, Infinity-2B vs. Infinity-8B, standard sampling → UA.

Model, benchmarkLayout-VendiSemanticsColorStructureMean Compositional Generation Score
Infinity-2B, T2I-CompBench (2,400)7.07 → 9.630.205 → 0.3200.325 → 0.5200.139 → 0.2420.4700 → 0.4677
Infinity-8B, T2I-CompBench (2,400)4.77 → 6.480.109 → 0.1660.186 → 0.2920.068 → 0.1060.5354 → 0.5314
Infinity-2B, GenEval2 (800)6.94 → 9.290.163 → 0.2380.349 → 0.5300.125 → 0.2170.6192 → 0.6185
Infinity-8B, GenEval2 (800)4.80 → 6.250.081 → 0.1180.180 → 0.2760.060 → 0.0880.7487 → 0.7437

Despite showing much stronger compositional generation capability than Infinity-2B, Infinity-8B exhibits substantially lower diversity in every aspect. This suggests that the level of diversity is model-specific, whereas our method is model-agnostic and increases generation diversity regardless of the base model.

AuthorsOpenReview ↗

Rebuttal (2/3)

Q1. Is the technique applicable to VAR-based video models such as InfinityStar?

We agree that this is a promising direction. To investigate this, we conducted preliminary experiments evaluating the applicability of our method to VAR-based video models.

Setup. We tested both components on InfinityStar-8B [1] at its default 480p settings (81-frame videos). As a preliminary study, we used 14 T2V-CompBench [2] prompts (two per category, listed below) with 8 candidates per prompt. Anchor diversity is the mean pairwise normalized bit-wise Hamming distance among a prompt's candidates at the coarsest (1×1) scale. Video diversity is the Vendi score of a prompt's 8 videos, computed on DINOv2 features of the middle frame (Layout-Vendi) and on I3D clip features (I3D-Vendi). Text alignment is measured by ViCLIP text–video similarity, and quality by the VBench aesthetic score.

Unconditioned Anchors. InfinityStar differs from Infinity in two ways that matter for UA. First, it uses APG [3] (guidance scale 34), which keeps only the guidance component orthogonal to the conditional prediction and clips its norm, whereas Infinity uses CFG with a constant scale. Second, it generates a video as a first-frame image pyramid followed by spatiotemporal scales, so the coarsest scales that UA acts on determine only the first frame's layout. The following table shows the results. Values in parentheses are the paired difference from the default, with its 95% bootstrap interval over prompts:

Table R5. UA on InfinityStar-8B (14 prompts, 8 candidates per prompt).

Sampler1×1 HammingLayout-VendiI3D-VendiViCLIP textAesthetic
Default (APG 34)0.1875.283.320.2610.566
UA0.141 (−0.046 [−0.093, +0.004])5.26 (−0.02 [−0.21, +0.18])3.54 (+0.22 [−0.10, +0.53])0.264 (+0.003 [−0.003, +0.010])0.567 (+0.001 [−0.009, +0.012])

UA does not transfer directly. Unlike in Infinity, where guidance narrows the first-scale anchors (Hamming 0.08, rising to 0.47 under UA; App. E), in InfinityStar, it does not. Removing guidance entirely does not significantly widen the anchors (0.212 vs. 0.187, +0.026 [−0.005, +0.065]), and UA neither widens them nor makes the videos more diverse.

Early Attention Pruning. A video candidate costs far more than an image candidate: about 53 s per InfinityStar video on an A100, compared with 2.3 s per Infinity-2B image. InfinityStar has no separate cross-attention layers, as text tokens are prepended to the visual tokens in joint self-attention. We therefore extracted EAP's attention maps from the visual-to-text part of this attention (conditional branch, first 25 of 36 blocks, averaged over heads). We computed CC and SS and applied the pruning rule exactly as in Sec. 4, without retuning, and pruned the 2 lowest-scoring of the 8 candidates at scale 11 (a first-frame scale with 18×32 tokens). Reaching scale 11 takes only 3.3% of the wall-clock time of a full generation. The candidates come from InfinityStar’s standard sampler (APG 34). Each final video was scored with Soft-TIFA: Qwen3-VL-8B-Instruct answers atomic questions derived from the prompt (objects, attributes, actions, motion) given 16 uniformly sampled frames, and the score is the geometric mean of the probabilities assigned to the correct answers. We report the fraction of pruned candidates that are among the 2 lowest-scoring videos (capture rate) and how often the best video survives pruning (survival rate):

Table R6. EAP on InfinityStar-8B: 2 of 8 candidates pruned at scale 11.

Pruner (2 of 8, scale 11)Bottom-2 captured ↑Best video survives ↑
Random (chance)0.250.75
EAP (visual-to-text attention)0.29 (8/28)0.86 (12/14)

EAP performs above chance on both measures, using a signal available after only 3.3% of the compute. With only 14 prompts, however, the confidence intervals still include the chance level (capture [0.14, 0.43], survival [0.64, 1.00]), so we treat this as preliminary evidence. Turning EAP into a test-time scaling method for video generation would require a larger prompt set, signals from the spatiotemporal scales, and a video verifier for the surviving candidates, which we leave to future work.

In the revised PDF, we have added both results to App. E and discussed extending EAP to video generation as future work.

AuthorsOpenReview ↗

Rebuttal (3/3)

Prompts.

  • Consistent attribute: "Silver-haired woman admiring a black grand piano"; "A bride in a blue gown holds a white bouquet, petals fluttering as she walks down the aisle"
  • Dynamic attribute: "Candle burns and melts as adjacent ice cube melts into water"; "The fabric is stretching from a wrinkled bundle into a taut, flat surface."
  • Spatial relationship: "A puppy dozing under a crowded cafe table during lunch hour"; "A soldier saluting behind a flagpole"
  • Motion binding: "An elderly man walking slowly to the left in a sunny park"; "An airplane lifts upward from the runway, engines roaring as it ascends into the clouds."
  • Action binding: "A kid climbs a tree, a horse grazes nearby"; "A girl is blowing bubbles in the yard, while a puppy is jumping."
  • Interaction: "Water sprite freezes lake with a touch"; "Grandpa teaches his grandson to fly a kite in the park."
  • Numeracy: "Two girls paint a large canvas together."; "Seven oranges roll across the table."

References

[1] Liu, Jinlai, et al. "InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generation." Advances in Neural Information Processing Systems 38 (2026): 170054-170072.

[2] Sun, Kaiyue, et al. "T2v-compbench: A comprehensive benchmark for compositional text-to-video generation." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

[3] Sadat, Seyedmorteza, Otmar Hilliges, and Romann M. Weber. "Eliminating oversaturation and artifacts of high guidance scales in diffusion models." The Thirteenth International Conference on Learning Representations. 2024.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Discussion and Revisions

We thank all reviewers for their time and constructive feedback. Below, we restate our contribution and summarize the strengths noted by the reviewers, our responses, and the revisions.

Contribution. Our paper shows that text conditioning is not needed at the coarsest scales of text-to-image VAR models (1×1 and 2×2), where object-level attributes are not yet localized. Building on this finding, we propose two components for efficient test-time scaling: unconditioned anchors (UA), which sample these scales without the text condition to diversify candidates at no extra compute, and early attention pruning (EAP), which discards failing candidates early using cross-attention signals.

Concerns are indexed as in our responses (W: weakness, Q: question, F: follow-up comment), and R1–R18 refer to the tables in our responses.

Strengths noted by reviewers

  • Clear writing and an intuitive method grounded in the scale-wise structure of VAR (FUzx, KdbP)
  • Practical efficiency gains over prior methods (FUzx, fdVf)
  • Controlled attribute probes that motivate the method (fdVf)
  • Comprehensive experiments across Infinity-2B, Infinity-8B, and Switti (KdbP)

Reviewer FUzx

  • W1. Relationship between attribute localization and diversity, and diversity beyond object composition: We clarified that localization determines where the condition can be removed and when to prune (Sec. 3.2). Beyond Layout-Vendi, we measured semantic, color, and structural diversity: UA significantly increases all four axes on T2I-CompBench, GenEval2, and the non-compositional Pick-a-Pic (Tables R1–R3).
  • W2. Cause of limited diversity in VAR, and whether scaling resolves it: We clarified that our method does not presuppose a diversity deficiency in VAR models. In an additional comparison, Infinity-8B is less diverse than Infinity-2B, so limited diversity is not explained by small model size; UA increases diversity on both models (Table R4).
  • Q1. Applicability to video generation: We conducted a preliminary study on InfinityStar, where EAP transfers above chance (Tables R5–R6).

Reviewer KdbP

  • W1. Why diversity differs across backbones (now part of Weakness 2 in the updated review): We clarified that our paper does not propose a mechanism for this difference, which would require pretraining ablations. We conducted an additional controlled comparison across backbones and model sizes (Table R7).
  • W2. Scope of diversity limitations (Weakness 3 in the updated review): We clarified that we do not claim a general diversity limitation of VAR; our method increases diversity regardless of the base model's level of diversity.
  • F1. Motivation and practical value (Weakness 2 in the updated review): We clarified that, like prior diffusion work improving diversity, our motivation does not rest on a diversity deficiency. UA alone significantly increases diversity at no extra compute, lowering only the per-candidate mean slightly, not the quality of the selected image (Tables R7, R8, R17, R18). Combined with EAP, it outperforms standard best-of-16 at 0.38× its compute and yields significant gains over standard best-of-N at equal compute (Table R9).
  • F2. Relation to Wang et al. and DiverseAR (Weakness 1 in the updated review): We added both arXiv preprints to Related Work and clarified how our work differs in analytical perspective, method, scope of applicability, and experimental conclusions.

Reviewer fdVf

  • W1/Q1/F1. Comparison of UA with simple alternatives: We conducted additional comparisons with random tokens and weak guidance under the same selector (Table R8). In the follow-up, we further showed that, at identical compute, UA is the only sampler whose selected image satisfies the prompt significantly more often than with standard sampling under both verifiers (Table R18).
  • W2/Q2. Paired difference intervals and failure analysis: We added paired intervals for all per-category results and a prompt-level failure analysis. On Infinity-2B, there is no significant overall mean degradation. Across models, UA and EAP rarely remove all valid candidates for a prompt, no more often than around chance rate (Tables R10–R11).
  • Q3. Diversity among valid candidates and independent quality check: We conducted additional analyses showing that diversity increases among prompt-satisfying candidates (Table R13) and that independent preference models detect no hidden artifacts (Table R14).
  • Q4. Choice of the pruning scale and attention layers: We clarified the selection protocol and showed that neither choice is tuned to the score on either benchmark (Tables R15–R16).

Revisions to the PDF. Updated Related Work (Sec. 2), clarified role of localization (Sec. 3.2), and the new analyses above (App. B.1, C, D.2, D.3, D.5, D.7, and E).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.