acceptodds
Withdrawn from ICLR 2027

More than a Single Guidance Strength: Rethinking Classifier-Free Guidance in Next-Token Autoregressive Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Classifier-free guidance (CFG) in next-token autoregressive image generation typically uses the difference between conditional and unconditional predictions as a fixed guidance direction, controlled by a single scalar strength. This raises a basic question: Is a single guidance strength sufficient to characterize how CFG should be applied? First, we view CFG as increasing conditional evidence while limiting its KL deviation from the model’s conditional prediction, and further study this trade-off under the fisher geometry. In this geometry, we decompose the CFG direction into two components with distinct effects on token prediction. One preferentially amplifies class-specific tokens, while the other preserves tokens favored by both conditional and unconditional predictions. We then investigate how the guidance direction should be combined for a given guidance strength and observe a systematic shift in the preferred balance: weak guidance favors more class-specific amplification, while stronger guidance requires greater preservation of jointly supported tokens. We further study how guidance should be applied for a given strength and composition, and find that the update strategy also matters: successive local updates become more useful with stronger guidance and greater emphasis on class-specific amplification. Together, these findings show that a single guidance strength is insufficient to characterize CFG, as its behavior also depends on how the guidance direction is composed and how the guidance is applied. Experiments across multiple autoregressive backbones validate these findings and show consistent improvements in both class-to-image and text-to-image generation.

Primary area:
generative models
Keywords:
Classifier-Free Guidance, Autoregressive Image Generation, Next-Token Prediction, Guidance Composition, Fisher Geometry, Local Guidance Updates

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RvfQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies autoregressive image generation from the perspective of Evidence-KL optimization and Fisher information geometry. It decomposes the CFG signal into orthogonal and parallel components with different roles. The paper finds that the optimal guidance composition varies with guidance strength. It further proposes successive local updates to improve generation quality. The mechanism is validated on multiple autoregressive models.

Critical strengths
  1. The analysis of the problem is thorough. The paper is well written and easy to follow. The topic is interesting.

  2. The theoretical analysis is solid. The mechanism design is self-consistent.

Critical weaknesses
  1. The paper states that weaker guidance emphasizes the orthogonal direction more. It also shows that successive local updates are useful for stronger guidance and for updates that emphasize the orthogonal direction. This seems contradictory.

  2. The proposed CFG decomposition is only applied to autoregressive generation. Why? Can it be applied to other settings, such as CFG in diffusion models?

Questions for authors
  1. In Figure 4(a) and Figure 5, the smallest FID is achieved by standard CFG with λ=1. Given this, what is the benefit of the proposed decomposition? Where does the improvement come from?

  2. How exactly does successive local updates perform each update? Do the repeated updates increase the computational cost? Moreover, the effectiveness of successive local updates is questionable based on the results. Does standard CFG show similar behavior at larger guidance scales?

  3. The overall experimental gains are small. The advantage over single guidance is not evident. The practical value seems limited.

  4. The visualization does not show clear advantages of the method. In Figure 7(b), the images generated by standard CFG do not collapse. However, λ=15 is already so large that this setting has little practical value.

Reviewer_zNHcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies an important question: how to apply CFG correctly in the AR models? The authors reveal that we should decompose the CFG according to Fisher information and apply it according to this decomposition. The experimental results validate the authors' algorithm is effective.

Critical strengths
  • The paper studies an important question: how to apply CFG properly?

  • The Fisher information view of the paper is novel to the community.

  • The proposed algorithm is effective across multiple models.

Critical weaknesses
  • After reading the paper, I do not understand why the choice of the reference mm is not sensitive. For example, rotating mm will obviously change the parallel and orthogonal components. This should greatly influence the analysis in the paper.

  • The decomposition according to the Fisher matrix is not motivated. It is confusing why we care about the Fisher matrix. For example, can we use the Hessian geometry? In addition, it is not clear why we define the Fisher matrix using pcp_c. Can we use pup_u?

  • The claims in the paper should also hold across text-conditioned models. However, all the experiments are conducted on class-conditioned models.

  • The experiments in Section 4 are only on AliTok-L, which greatly constrains the applicability of the proposed analysis.

  • The metrics adopted in the paper are not comprehensive. For image generation, please consider benchmarks like GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment.

  • More closely related works should be discussed:

  1. ScalingAR: Scaling Confidence for Autoregressive Image Generation. Chen et al

  2. FARMER: Flow AutoRegressive Transformer over Pixels. Zheng et al

  3. Demystifying Classifier-Free Guidance for Auto-Regressive Image Generation. Zhou et al.

  4. Reward Weighted Classifier-Free Guidance as Policy Improvement in Autoregressive Models. Peysakhovich and Berman

  5. Less is More: Improving LLM Reasoning with Minimal Test-Time Intervention. Yang et al.

  6. Autoregressive Styled Text Image Generation, but Make it Reliable. Zaccagnino et al.

  • Minor points: all the math notations should be defined. For example, Δpgt\Delta p_{gt} is not defined in the paper.
Questions for authors
  • It will be better to conduct the experiments in Section 4 on more models.
Reviewer_bEWiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies classifier-free guidance (CFG) in next-token autoregressive generation over discrete visual tokens. Starting from a trade-off between conditional evidence and KL deviation, the authors use local Fisher geometry to decompose the difference between conditional and unconditional logits into components parallel and orthogonal to the midpoint logits. They associate these components with preserving jointly supported tokens and amplifying class-specific tokens, respectively. The paper investigates how the preferred composition changes with guidance strength and how multiple updates that recompute the local decomposition affect generation, then uses these observations to design a configuration search. Experiments cover five class-conditional backbones and text-conditional generation with SimpleAR. FID improves across all backbones in Table 1, and the overall GenEval score increases from 0.585 to 0.611 in Table 2.

Critical strengths
  1. The mechanistic analysis connects to specific prediction behavior.

    Figure 3 on page 5 measures the increase in next-ground-truth-token probability and its KL-normalized counterpart for each component under ground-truth prefixes, reducing interference from accumulated sampling errors. Appendix B.1 compares midpoint, conditional, and unconditional references and checks different Top-k thresholds. In Table 3 on page 21, the midpoint reference achieves 98.7% consistency for both functional orderings.

    These controls make the functional distinction more convincing than final FID results alone.

  2. Practical gains without retraining are demonstrated across backbones.

    Table 1 on page 8 compares standard CFG, SoftCFG, and the proposed method using the same released checkpoints and evaluation pipeline, with improved FID on all five backbones. Appendix D.3 also reports the FID–IS trade-off, results across three random seeds, and search costs.

    The cheaper greedy search in Table 7 still reduces FID from 3.72 to 3.36 on LlamaGen-L and from 1.39 to 1.27 on AliTok-L.

    The method operates directly on existing checkpoints, keeping the model and tokenizer unchanged (pages 28–29 of the submission).

    This use without retraining gives the reported consistent gains independent practical value.

Critical weaknesses
  1. The contribution of the Fisher decomposition has not been separated from changes in effective temperature and CFG strength.

    From Equations (8) and (12) on pages 4–5, define:

    \qquad \delta=(b-a)\kappa.$$ The logits after a single update are exactly: $$z_{a,b} =\left(1+a+\frac{\delta}{2}\right)z_c +\left(-a+\frac{\delta}{2}\right)z_u.$$ When $1+\delta>0$, this can be written as CFG with temperature and effective strength: $$\tau=\frac{1}{1+\delta}, \qquad w_{\mathrm{eff}}=\frac{a-\frac{\delta}{2}}{1+\delta}.$$ If $w_{\mathrm{eff}}\ge 0$ also holds, the strength lies within the paper's standard CFG parameter range. Since $\kappa$ varies with the prediction state, this does not imply that a single fixed temperature can reproduce the method, nor does it negate the value of using a Fisher-based rule to choose coefficients. It does show that changing the component ratio jointly changes a state-dependent temperature and effective guidance strength. Table 6 on page 29 fixes temperature; joint CFG–temperature tuning and controls that retain only one of these state-dependent changes are missing. The existing results therefore support the effectiveness of the coefficient-selection rule, but do not establish how much the preservation of jointly supported tokens and amplification of class-specific tokens reflect a distinctive role of Fisher geometry or can be explained by this reparameterization. The reference-vector ablations and cross-backbone results do not directly answer this question. This affects the mechanistic interpretation and independent value of the additional sampling flexibility, rather than whether the reported gains exist.
  2. The reported numerical results do not fully satisfy the monotonicity assumed for search pruning, and the practical handling of this issue is unclear.

    Appendix C makes its pruning guarantee conditional on two orderings: the optimal composition is nondecreasing with strength (Equation 65), and the gain from local updates is nonincreasing with composition (Equation 67). However, Table 4 on page 27 reports FID values of 1.27 and 1.29 for K=1K=1 and K=2K=2 at (a,λ)=(4,0)(a,\lambda)=(4,0), giving G=−0.02G=-0.02 under Equation (66).

    At λ=1\lambda=1, the updates in Equations (13)–(14) reduce exactly to standard CFG, independently of KK, so the gain at the same strength should be 0. The reported point estimates thus give G(λ=0)<G(λ=1)G(\lambda=0)<G(\lambda=1), contrary to Equation (67).

    This may reflect finite-sample evaluation noise or a failure of the ordering at this configuration; it does not establish that the search misses the optimum. Nevertheless, if pruning directly depends on noisy FID rankings, the paper should explain whether it uses tolerances, repeated evaluations, or fallback checks, and whether these change the search cost.

    The checks in Figure 17 of Appendix D.2 and the observation in Appendix B.3 that update-count differences are small for λ>1\lambda>1 remain valuable, but do not directly explain this particular violation.

  3. Fewer configuration evaluations have not been sufficiently connected to preservation of the optimum or gains under equal budgets.

    The full candidate sets in Appendix D.3 on pages 30–31 contain 154 configurations for LlamaGen-L and 210 for AliTok-L, with λ\lambda extending to 2. Table 7 reports 93/125 evaluations for systematic search, but does not report the exhaustive-search optimum over the same candidates.

    The reduction in evaluation count is therefore demonstrated, while the possible loss in search quality remains unclear.

    Standard CFG and SoftCFG each use 11/15 evaluations, so the final gains from systematic search also involve a larger configuration-search investment. Greedy search reduces this cost to 17/21 evaluations while retaining clear gains, which is important positive evidence. However, it has not been compared with simple search strategies using the same candidates and budget, making it difficult to quantify the evaluation savings attributable to the mechanistic prior itself.

Questions for authors
  1. Mechanistic attribution and key controls:

    What are the distributions of the effective temperature τ\tau and CFG strength weffw_{\mathrm{eff}} in the configurations actually used? Do cases with 1+δ≤01+\delta\le 0 or weff<0w_{\mathrm{eff}}<0 occur?

    Using the same fixed checkpoint and comparable search budgets, could you compare joint CFG–temperature tuning and separately fix temperature or effective strength while retaining only the state dependence of the other?

    These controls would help distinguish the contributions of additional sampling flexibility and the Fisher-based coefficient-selection rule.

    Here τ\tau is a post-processing temperature on the output logits, distinct from the model input β\beta defined on page 4 of GFT; GFT's β\beta-tuning experiments do not replace this control.

  2. Monotonicity violations and implementation:

    Please explain the ordering violation between G=−0.02G=-0.02 in Table 4 and G=0G=0 at λ=1\lambda=1. Does it arise from evaluation noise, differences in sampling settings, or failure of the monotonicity assumption at that configuration?

    Does the algorithm use a single FID point estimate, or ranking tolerances, repeated evaluations, or fallback checks?

    If additional handling is used, please describe the rule and its evaluation cost; otherwise, please explain whether the violation affects the dominance argument in Equation (69).

  3. Search quality and budget validation:

    Using existing results or an affordable representative candidate grid that includes λ>1\lambda>1, could you compare systematic search with the exhaustive optimum and report the FID gap potentially attributable to unevaluated configurations?

    With the same candidates and number of evaluations, could you also compare greedy or systematic search with a simple reference such as random search, to establish whether the mechanistic prior reaches comparable quality sooner?

    The existing greedy results with 17/21 evaluations already indicate potential for inexpensive use. These checks would further distinguish gains from a larger search budget and gains from the search rule itself。

  4. Comparison with GFT and MG, and positioning of the contribution:

    Could you directly compare the method with Guidance-Free Training (GFT; Chen et al., 2025) and Model-guidance (MG; Tang et al., 2025) side by side, explaining their relationships and differences in target distributions, guidance mechanisms, and treatment of guidance during training versus inference?

    GFT is already cited on pages 3 and 10 of the submission, and gives distribution-tilting and logit formulations in Equation (5) and the discrete-CFG discussion on its page 3. MG incorporates the guidance-related posterior correction into the training objective. Please clarify what new, testable mechanistic insights the proposed Fisher decomposition, interpretation of component roles, and local updates provide beyond these existing explanations, and distinguish those insights from differences in sampling or training procedures.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.