acceptodds
Under review as a conference paper at ICLR 2027

Adversarial Error Correction for Visual Autoregressive Generation

Abstract

Visual Autoregressive (VAR) models have emerged as a powerful paradigm for image synthesis by performing hierarchical next-scale prediction. However, VAR models are inherently prone to cascading error propagation, where subtle coarse-scale mispredictions are amplified across the hierarchy, ultimately distorting the final synthesis. To mitigate this, we propose AID-VAR, a plug-and-play framework that enhances pre-trained VARs through Adversarially Injected Diagnosis. Instead of a standard passive generation, AID-VAR introduces a proactive error-correction mechanism inspired by the adversarial feedback in GANs. We deploy a discriminator to diagnose fidelity gaps at each scale transition, coupled with a lightweight guidance injector. This module operates as a non-invasive adapter that refines the feature manifold of a frozen VAR backbone, effectively steering the generation toward the distribution of real images without destabilizing the pre-trained latent space. Furthermore, to rigorously evaluate this cross-scale progression, we introduce the Inter-Scale Consistency Score (ISCS), a novel metric that quantifies the fidelity and structural alignment between consecutive resolution scales. Experimental results across various backbones demonstrate that AID-VAR delivers sharper textural details and fewer structural distortions with negligible overhead. For instance, AID-VAR-d20 achieves a 16% improvement in FID with only a 3% increase in parameters. These results establish AID-VAR as a highly efficient and scalable pathway for upgrading large-scale VAR generators—enhancing global coherence and local detail without altering training data, base architectures, or sampling schedules.

Primary area:
generative models
Keywords:
Visual Autoregressive Generation, Error Accumulation, Adversarial Guidance, Generative Models, Cross-Scale Consistency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_J9KiOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes AID-VAR, a plug-in module attached to a frozen VAR backbone to deal with cross-scale error accumulation. A light Transformer injector builds a spatial guidance map from the previous-scale feature and adds it to the hidden state of the next scale. The injector is trained adversarially against a DINO-based projected discriminator in RGB space. The paper also proposes ISCS, a metric for consistency between adjacent scales, and reports FID improvement on VAR-d16/d20/d24 and on FlexVAR in the appendix. Post-hoc enhancement of a frozen backbone is a useful direction and the idea is intuitive. However, the actual mechanism looks closer to learned guidance than to error correction. Also, the source of the baseline numbers and the validity of ISCS remain questionable.

Critical strengths
  1. Post-hoc enhancement of a frozen VAR backbone is a practical setting, and in principle it applies to the fast-growing VAR family, though this is not quite proven in this paper and the injector requires retraining per backbone.

  2. The motivation and explanation are clear, and Fig. 1 delivers the idea well.

Critical weaknesses
  1. The framing of the paper does not match the actual mechanism, which brings up a novelty concern. The paper calls it "Adversarially Injected Diagnosis", but what happens at inference is only adding a learned additive term to the hidden state (Eq. 4). The projected discriminator, hinge loss, and softmax relaxation are all existing techniques. So it is more natural to see this as another learned guidance on top of CFG. The U-shaped guidance weight curve in Tab. 2 also looks the same as a CFG scale sweep. The injector can fool the discriminator just by making the logits sharper, so the difference from lowering the sampling temperature is unclear as well. Yet there is no discussion or quantitative comparison with VAR guidance methods such as SSG [3] and IGG [4], or with the bitwise self-correction of Infinity [2], which directly handles next-scale exposure bias.

  2. The source of the baseline numbers is mixed, and backbone diversity is thin as only VAR is experimented with. The reported VAR baseline FID (3.55/2.95/2.33) matches the released checkpoint numbers in the official repository. But the IS/Precision/Recall in the same row match the original paper table, which is the checkpoint giving FID 3.30/2.57/2.09. So within one row, FID and the other metrics come from different sources. Moreover, Tab. 11 (App. E.1) says these baselines are 3-seed means, but the mean FID is the same as the repository value and the mean IS is the same as the original paper value to one decimal at all three depths (3.55±0.073.55 \pm 0.07 / 274.4±0.9274.4 \pm 0.9, 2.95±0.052.95 \pm 0.05 / 302.6±1.3302.6 \pm 1.3, 2.33±0.062.33 \pm 0.06 / 312.9±1.2312.9 \pm 1.2). A 3-seed mean cannot match an external single number three times. So I doubt whether the baseline was actually measured. With this concern in mind, the AID-VAR FID (3.24/2.54/2.08) sits at the level of the original paper (3.30/2.57/2.09), so a large part of the gain may simply be the gap between the released checkpoint and the paper checkpoint. Also, the CFG scale and top-kk are not stated in the paper, so it is impossible to check how the authors measured such statistics. In addition, "various backbones" in the abstract is just a depth change of the same VAR. FlexVAR in App. E.3 reports FID only and shares the same VQ codebook family, so applicability to backbones like HART [1] and Infinity [2] is not verified. Also, with 8×8\timesA800, a d30 experiment should have been possible.

  3. ISCS is the inverse of the Fréchet Distance (FD), and the weighting inflates the result. ISCS flips FD in DINO feature space, which is not essentially different from FID. In Fig. 7 the two models are almost the same at scales 0 to 7, and the difference appears only at scales 8 and 9. With αk∝2k\alpha_k \propto 2^k, scales 8 and 9 take roughly 75% of the total weight, so most of the weighted score comes from these two scales. Taking the inverse amplifies small FD differences at fine scales, so it is unclear whether the gap in Fig. 7 reflects a real quality difference.

Questions for authors
  1. Were the baseline FID, IS, Precision, and Recall each measured directly, or taken from the original paper or the official repository? Why do the 3-seed means in Tab. 11 match the repository FID and the original paper IS to one decimal? What exact checkpoint, CFG scale, and top-kk were used, and does the AID-VAR gain hold when the baseline is reproduced with the original paper setting?
  2. Are the other numbers in Tab. 1 under the same condition? MAR in Tab. 1 is 2.35/227.8/0.79/0.62, which is the number without CFG in the original MAR paper. With CFG, should it not be 1.55/303.7/0.81/0.62?
  3. Can you compare with VAR guidance methods such as SSG [3] and IGG [4] under the same CFG setting, or show that the gain remains when CFG is replaced with IGG? Also, could you report per-scale logit entropy with and without the injector, which would help to judge whether this is simple sharpening?
  4. The fake set is made by free-running inference, but Lrec\mathcal{L}_{\text{rec}} is CE against the GT tokens. Is the injector training teacher-forced or free-running? If teacher-forced, the injector never sees the model's own errors, so how does it learn to correct error accumulation?
  5. What is the wall-clock latency with and without the injector? When CFG is used, does the injector run in both the conditional and unconditional branches?

[1] HART: Efficient Visual Generation with Hybrid Autoregressive Transformer (ICLR 2025)

[2] Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (CVPR 2025)

[3] SSG: Scaled Spatial Guidance for Multi-Scale Visual Autoregressive Generation (ICLR 2026)

[4] Rethinking Visual Autoregressive Sampling with Information-Grounding Guidance (ICML 2026)

The initial rating below is driven mainly by Critical Weakness 2 (W2). I will reconsider it once the authors clarify W2.

10 comments
AuthorsOpenReview ↗

Response to Reviewer J9Ki [1/5]

We thank Reviewer J9Ki for the thorough, insightful review and for explicitly stating a willingness to reconsider the rating once Critical Weakness 2 (W2) is clarified. We address W2 (Q1 & Q2) first with full transparency and unified re-evaluations, followed by W1 (Q3), Q4, W3, and Q5.


1. Clarification & Unified Re-Evaluation of Baselines and New Backbones (W2, Q1 & Q2)

(a) Source of Original Baseline Numbers & Unified 3-Seed Re-Evaluation (Q1)

We sincerely apologize for the baseline reporting errors in Tables 1 and 11, and thank the reviewer for identifying them. We distinguish the two issues explicitly.

  1. Why Table 1 mixed repository FID and paper IS/Pre/Rec: When evaluating the officially released VAR checkpoints directly, we observed a known gap between the released weights (which yield FID 3.55) and the numbers printed in the original VAR paper (3.30). Because we could not reproduce the original paper's 3.30 FID using the public checkpoints (or our implementation), we initially aligned our Table 1 baseline reporting with recent follow-up works evaluated on the same official checkpoints (such as NFIG [1] and FlexVAR [2]), which reference the official repository's FID (3.55/2.95/2.33) alongside the paper's IS/Precision/Recall. We have corrected Table 1 in the revised manuscript to report consistently evaluated metrics
  2. Why Table 11 showed external means with 3-seed standard deviations: During our multi-seed evaluation, we computed the standard deviations across 3 sampling seeds, but inadvertently copied the reference mean values from Table 1 into Table 11 instead of updating them with our newly measured 3-seed sample means. We have corrected Table 11 in the revised manuscript to report the measured means and standard deviations across the three sampling seeds. Our correction separates directly evaluated results from literature references. We do not attribute the difference between repository and paper results to checkpoint weights without verifying the checkpoint and evaluation configurations.

To ensure complete rigor and apple-to-apple fairness, we have replaced all baseline numbers in Tables 11 with our directly measured 3-seed results using the exact same official checkpoints and sampling settings (cfg=1.5\text{cfg} = 1.5, top-k=900\text{top-k} = 900, top-p=0.96\text{top-p} = 0.96, 50,000 images ×\times 3 seeds: seed 0, seed 1, seed 2), and we also mark the original VAR paper's reported numbers separately with a dagger (†\dagger) for transparent reference (the scaling results on VAR-d30 (2.0B) are reported separately in Section 1(c) below):

Model / Seed Configuration (cfg=1.5\text{cfg}=1.5, top-k=900\text{top-k}=900, top-p=0.96\text{top-p}=0.96)ParamsFID ↓\downarrowsFID ↓\downarrowIS ↑\uparrowPrecision ↑\uparrowRecall ↑\uparrow
VAR-d16 (seed 0 / 1 / 2)310M3.61±0.073.61 \pm 0.078.28±0.038.28 \pm 0.03271.7±0.9271.7 \pm 0.90.84±0.0060.84 \pm 0.0060.51±0.0080.51 \pm 0.008
AID-VAR-d16 (Ours)321M3.24±0.063.24 \pm 0.067.85±0.047.85 \pm 0.04280.0±1.1280.0 \pm 1.10.85±0.0070.85 \pm 0.0070.51±0.0090.51 \pm 0.009
VAR-d20 (seed 0 / 1 / 2)600M2.96±0.052.96 \pm 0.057.69±0.057.69 \pm 0.05299.7±1.3299.7 \pm 1.30.83±0.0080.83 \pm 0.0080.56±0.0070.56 \pm 0.007
AID-VAR-d20 (Ours)619M2.54±0.082.54 \pm 0.087.42±0.057.42 \pm 0.05309.4±1.0309.4 \pm 1.00.83±0.0060.83 \pm 0.0060.56±0.0090.56 \pm 0.009
VAR-d24 (seed 0 / 1 / 2)1.0B2.23±0.062.23 \pm 0.067.71±0.047.71 \pm 0.04310.9±1.2310.9 \pm 1.20.82±0.0070.82 \pm 0.0070.59±0.0100.59 \pm 0.010
AID-VAR-d24 (Ours)1.02B2.08±0.052.08 \pm 0.057.51±0.047.51 \pm 0.04316.3±1.5316.3 \pm 1.50.82±0.0090.82 \pm 0.0090.60±0.0080.60 \pm 0.008
AuthorsOpenReview ↗

Response to Reviewer J9Ki [2/5]

(b) Correction of MAR Baseline in Table 1 (Q2)

Thank you for catching this detail. The values 2.35 / 227.8 / 0.79 / 0.62 correspond to MAR-L without CFG, whereas the reported CFG-enabled entry is 1.55 / 303.7 / 0.81 / 0.62 (FID / IS / Precision / Recall).

When compiling the unified comparison table, we initially followed the reporting convention of recent AR work (such as HMAR [3]), which list the default non-CFG entry of MAR alongside CFG-enabled baselines. However, we fully agree with your suggestion that mixing CFG settings without explicit labels can be misleading. To make the comparison completely transparent, we have updated Table 1 in the revised manuscript to explicitly include and label both MAR-L (w/o CFG) (2.35 / 227.8 / 0.79 / 0.62) and MAR-L (w/ CFG) (1.55 / 303.7 / 0.81 / 0.62), along with clear CFG annotations for all baselines.The corresponding clarifications have also been incorporated into the main text of the revised manuscript.


(c) Architectural Diversity: Full FlexVAR Metrics & Bitwise Text-to-Image Infinity-2B (W2)

To verify generalizability beyond standard VAR depths and VQ codebooks, we:

  1. Report all evaluation metrics for FlexVAR and VAR-d30:
  • FlexVAR: FID 3.05, sFID 7.91, IS 282.4, Precision 0.81, Recall 0.55
  • AID-FlexVAR-d16 (Ours): FID 2.81, sFID 7.75, IS 288.8, Precision 0.81, Recall 0.55
  • VAR-d30: FID 1.98, sFID 8.37, IS 315.9, Precision 0.81, Recall 0.59
  • AID-VAR-d30 (Ours): FID 1.90, sFID 8.37, IS 317.6 , Precision 0.82, Recall 0.60
  1. Transferred AID-VAR to Infinity-2B [6] (Bitwise Multi-Scale Quantization & Text-to-Image):

We trained our lightweight injector on top of the frozen Infinity-2B backbone (32 layers, bitwise residual quantization tokenizer, 256×256256\times 256 / 0.06M pixels) using the Fine-T2I (synthetic_enhanced_prompt_random_resolution) subset, and evaluated on the standard GenEval benchmark (cfg=4.0, tau=1.0, rewrite=0):

Model (Bitwise Text-to-Image on GenEval)Single Obj ↑\uparrowTwo Obj ↑\uparrowCounting ↑\uparrowColors ↑\uparrowPosition ↑\uparrowColor Attr ↑\uparrowOverall ↑\uparrow
Infinity-2B [6] (Baseline)80.31%18.19%7.82%53.44%3.46%5.74%28.16%
AID-Infinity-2B (Ours)84.06%20.96%9.69%57.45%7.25%7.00%31.07%
Gain+3.75%+2.77%+1.87%+4.01%+3.79%+1.26%+2.91% (+10.3% rel.)

AID-VAR yields consistent gains across all six GenEval compositional and structural subtasks on Infinity-2B (including more than doubling the Position score from 3.46% to 7.25%), proving that our framework generalizes effectively to bitwise tokenizers and text-to-image backbones. Notably, since Infinity-2B already incorporates bitwise self-correction during backbone training to address next-scale exposure bias, the fact that attaching our frozen-backbone AID-VAR injector on top of Infinity-2B yields an additional gain directly demonstrates that our post-hoc external guidance is orthogonal and complementary to internal self-correction mechanisms (W1).

AuthorsOpenReview ↗

Response to Reviewer J9Ki [3/5]

2. Comparison with SSG [4] and IGG [5], Logit Entropy, and Error Correction (W1 & Q3)

Framing Alignment and Manuscript Revision (W1)

Thank you for this insightful observation. We agree that while the discriminator diagnoses scale-level visual artifacts during training, the inference-time operation is a learned, trajectory-conditioned spatial guidance injection (Eq. 4) rather than explicit online error detection. In the revised manuscript, we have refined the Abstract and Sections 1 & 3 to align our terminology more accurately with the underlying mechanism—framing AID-VAR as discriminator-supervised spatial guidance for cross-scale error mitigation—while clarifying how it differs from training-free logit guidance (CFG/SSG/IGG) and internal bitwise self-correction (Infinity). The corresponding explanations and clarifications have also been incorporated into the revised main text.

(a) Quantitative Comparison & Complementarity with SSG [4] and IGG [5] (Q3)

We evaluated AID-VAR against and in combination with SSG [4] (cfg=1.5, ssg=1.0) and IGG [5] (replacing CFG with pure IGG, igg=1.0) on VAR-d16:

Guidance Method (VAR-d16, 50K samples)FID ↓\downarrowsFID ↓\downarrowIS ↑\uparrowPrecision ↑\uparrowRecall ↑\uparrow
VAR + CFG (1.5)3.618.28271.70.840.51
VAR + CFG (1.5) + SSG [4]3.297.99287.20.810.53
AID-VAR + CFG (1.5) (Ours)3.247.85280.00.850.51
AID-VAR + CFG (1.5) + SSG [4] (Combined)3.127.73292.40.840.51
VAR + Pure IGG [5] (Replacing CFG)4.408.50250.60.790.54
AID-VAR + Pure IGG [5] (Replacing CFG)4.018.30257.90.800.53
  1. Competitive Standalone Performance: Under the same cfg=1.5 setting, AID-VAR (3.24 FID) outperforms VAR + SSG (3.29 FID) without requiring extra forward passes through the heavy VAR backbone.
  2. Complementarity to SSG and IGG: Combining AID-VAR with SSG further reduces FID from 3.29 to 3.12 (and sFID from 7.99 to 7.73). Similarly, when CFG is replaced by IGG, adding AID-VAR reduces FID from 4.40 to 4.01 (+7.3 IS). Unlike training-free logit extrapolation (CFG/SSG/IGG), AID-VAR injects learned spatial feature corrections informed by an image-space discriminator, supporting compatibility with the tested sampling-time guidance settings. These FID improvements accompany Recall decreases from 0.533 to 0.506 with SSG and from 0.538 to 0.526 with IGG.

(b) Per-Scale Logit Entropy Analysis: Is AID-VAR Just Logit Sharpening? (W1 & Q3)

To test whether the injector merely sharpens logits (equivalent to lowering sampling temperature), we measured the per-scale token distribution entropy H(pk)\mathcal{H}(p_k) across 50,000 samples on VAR-d16 (cfg=1.5, top-k=900, top-p=0.96):

Scale kk (Patch Grid)0 (121^2)1 (222^2)2 (323^2)3 (424^2)4 (525^2)5 (626^2)6 (828^2)7 (10210^2)8 (13213^2)9 (16216^2)
VAR Entropy4.64375.93846.41996.83766.67366.84857.00796.59886.30725.6758
AID-VAR Entropy4.64215.95596.46156.88616.71876.88797.03486.63466.33875.6921
Relative Change-0.03%+0.29%+0.65%+0.71%+0.68%+0.58%+0.39%+0.54%+0.50%+0.29%

Lowering temperature reduces entropy across scales. In contrast, AID-VAR does not reduce logit entropy—in fact, entropy slightly increases (+0.29% to +0.71%) across Scales 1–9. Furthermore, in our matched-perturbation experiment (injecting σ=0.25\sigma=0.25 Gaussian noise at Scale 5; see Response to Reviewer #1, Section 1), AID-VAR dampens trajectory divergence (LPIPS to clean trajectory) to 0.0005 at Scale 6 (vs. 0.3208 for VAR) and 0.3160 at Scale 10 (vs. 0.5625 for VAR), providing separate evidence of reduced perturbation sensitivity. Neither scale-averaged entropy nor this single intervention substitutes for a temperature-matched quality comparison.

Reviewer_Vv9LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AID-VAR, a lightweight guidance injector attached to a frozen visual autoregressive generator. The injector receives adversarial supervision from an RGB-space discriminator through differentiable soft codebook decoding. The paper also introduces ISCS, which compares joint feature distributions of adjacent generation scales.

Critical strengths
  1. Relevant problem and practical design. Improving pretrained generators while preserving their weights is useful. The discriminator is needed only during training, making the inference design relatively simple.
  2. Consistent reported improvements across model sizes. FID results of different model scales show profits in Table 1. Precision and recall show no obvious overall deterioration.
Critical weaknesses
  1. The training procedure is insufficiently specified. Two central details remain unclear. First, Equation (5) constructs soft codebook embeddings from predicted logits, whereas real-image encoding supplies discrete tokens. The claimed matching soft-decoding pipeline for real samples is not mathematically defined. Second, the reconstruction loss requires ground-truth token targets, while the training description includes class-conditioned generated trajectories. The paper should distinguish teacher-forced and free-running branches and explain how reconstruction targets are paired with their contexts.

  2. ISCS is not sufficiently validated as a specific consistency metric. Fréchet distance between concatenated adjacent-scale features depends on both marginal image quality and cross-scale relationships. Consequently, an ISCS improvement does not by itself establish better conditional consistency or reduced error propagation.

The discriminator and ISCS also share DINO features, creating possible evaluation alignment with the training representation. This does not invalidate ISCS, but requires independent validation. The additional reference-based metrics lack the necessary pairing protocol, while CKA measures representation association rather than correction.

Useful validation would include shuffled adjacent-scale pairings with unchanged marginals, comparisons against marginal per-scale distances, independent feature extractors, and human structural-consistency judgments. Weight normalization, the first transition, and numerical handling near zero distance also require clarification.

Questions for authors
  1. Can the author provide a detailed explanation of the training process in Weaknesses 1 ?
  2. Can the author verify the accuracy of ISCS in measuring cross scale consistency?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Vv9L [1/3]

We thank Reviewer Vv9L for the constructive comments and for recognizing the practical design of our frozen-backbone framework and its consistent improvements across model scales. We clarify the four core aspects of our training pipeline (W1/Q1) and the ISCS validation (W2/Q2) below.

1. Exact Specification of the Training Pipeline (W1 & Q1)

We clarify four key mechanisms in Sections 3.1–3.2 and Figure 3.

(1) Is the Adversarial Branch Free-Running?

Yes. Fake samples are generated through free-running autoregressive rollout, conditioned on the class label cc of the input image II. At each scale k>1k>1, frozen VAR uses its own previously sampled tokens:

S^<k\hat{S}_{<k}

The previous-scale feature is then fed into the Guidance Injector to obtain spatial guidance:

Gk=Pϕ(x^prev).G_k=P_\phi(\hat{x}_{\mathrm{prev}}).

Thus, the injector observes exposure-bias errors from model-generated contexts and learns to correct them under adversarial supervision Ladv\mathcal{L}_{\mathrm{adv}}.

(2) How Are the Cross-Entropy Context and GT Tokens Paired?

Each real image II is encoded by the frozen VQ-VAE into ground-truth token maps s1,…,sKs_1,\ldots,s_K. We apply an auxiliary reconstruction loss:

Lrec=Ek[CE(zk′,sk)].\mathcal{L}_{rec}=\mathbb{E}_k[\mathrm{CE}(z'_k,s_k)].

with λrec=0.01\lambda_{rec}=0.01 (Eq. 7).

In the teacher-forced regularization pass, we use the ground-truth prefix s<ks_{<k} and feature xkgtx_k^{gt}. The guided forward pass is:

zk,TF′=fVAR(xkgt+wPϕ(xprevgt))z'_{k,\text{TF}} = f_{\text{VAR}}(x_k^{\text{gt}} + w P_\phi(x_{\text{prev}}^{\text{gt}}))

The resulting logits align with target tokens sks_k, preserving valid predictions on clean contexts while the free-running branch addresses off-manifold errors.

(3) How Is Real/Fake Soft-Label Decoding Implemented?

Fake samples: Discrete codebook lookup is non-differentiable. We therefore use softmax relaxation over the VQ codebook WW (Eq. 5):

Hk′=WTsoftmax(zk′/τ).H'_k=W^T\mathrm{softmax}(z'_k/\tau).

f^k′=f^k−1′+Φk(Hk′).\hat{f}^{\prime}_{k} = \hat{f}^{\prime}_{k-1} + \Phi_{k}(H^{\prime}_{k}).

I^k′=DecVQ(f^k′).\hat{I}^{\prime}_{k} = \mathrm{Dec}_{\mathrm{VQ}}(\hat{f}^{\prime}_{k}).

Real samples: We apply the same soft-decoding pipeline to avoid discrimination based on hard-versus-soft quantization artifacts. Given encoder latent ze(I)z_e(I) and reference logits zkrealz_k^{real}, obtained from codebook distances or teacher-forced VAR predictions, we compute:

Hkreal=WTsoftmax(zkreal).H_k^{real}=W^T\mathrm{softmax}(z_k^{real}).

Ik=DecVQ(∑j≤kΦj(Hjreal))I_{k} = \mathrm{Dec}_{\mathrm{VQ}}\left(\sum_{j \le k} \Phi_{j}(H_{j}^{\text{real}})\right)

Thus, real and fake samples pass through the same continuous codebook relaxation and frozen VQ-VAE decoder before entering discriminator DD.

(4) Exact Gradient Flow and detach() Placement

As reported in Appendix C.2 (Table 8), AID-VAR requires only 14.8 GB peak VRAM on VAR-d16, compared with 46.2 GB for LoRA. This is achieved by restricting backpropagation to the current-scale computation graph:

  1. Cross-Scale Context Detachment (stop-gradient between scales): During the free-running rollout across scales k=1…Kk = 1 \dots K, previously sampled tokens S^<k\hat{S}_{\lt k} and accumulated features f^k−1\hat{f}_{k-1} are detached using .detach() before initiating step kk. Therefore, gradients do not propagate through earlier autoregressive scales 1…k−11 \dots k-1.
  2. Injector Update (step_G): The VAR backbone, VQ-VAE encoder/decoder, DINO backbone EE, and discriminator heads hθh_\theta have frozen parameters (requires_grad=False), while their forward computation graphs remain active. Only the injector PϕP_\phi is updated using: LP=Ladv(I^k′)+λrecLrec\mathcal{L}_P = \mathcal{L}_{adv}(\hat{I}'_k) + \lambda_{rec}\mathcal{L}_{rec}

Gradients flow through the frozen discriminator, VQ-VAE decoder, soft codebook embeddings, and current VAR computation to PϕP_\phi. 3. Discriminator Update (step_D): Both real and fake decoded images are detached from the injector graph. Only classification heads hθh_\theta are updated using hinge loss LD\mathcal{L}_D (Eq. 3) and the R1R_1 penalty with γ=0.2\gamma = 0.2 (Table 10).

AuthorsOpenReview ↗

Response to Reviewer Vv9L [2/3]

2. Comprehensive Validation and Mathematical Specification of ISCS (W2 & Q2)

To address all aspects of W2 and Q2, we performed four targeted validations (over 50,000 samples, 1,000 classes, 10 scales):

(a) Independent Feature Extractor, Shuffled Pairings, and Marginal vs. Joint FD

To test whether the findings depend on the DINO representation used by the discriminator, we re-evaluated ISCS using an independent feature extractor (the native VAR feature representation, completely separate from DINO). We compared Marginal per-scale FD against Original Joint FD, Global Shuffled Joint FD (20 trials), and Intra-Class Shuffled Joint FD (20 trials, preserving class labels while shuffling instance pairings at scale kk), while keeping per-scale image sets (and thus Marginal FDs) unchanged:

Scale kkMarginal FD (VAR / AID)Orig. Joint FD (VAR / AID)Global Shuffled Joint FD (VAR / AID, 20 trials)Intra-Class Shuffled Joint FD (VAR / AID, 20 trials)Real Data Shuffled FD (Sanity Check)
10.021789 / 0.0213030.071651 / 0.0707000.072439±6e-50.072439_{\pm 6\text{e-}5} / 0.071633±8e-50.071633_{\pm 8\text{e-}5}0.070689±6e-50.070689_{\pm 6\text{e-}5} / 0.069904±6e-50.069904_{\pm 6\text{e-}5}0.006361±1e-40.006361_{\pm 1\text{e-}4}
20.008535 / 0.0084540.031282 / 0.0306870.035013±4e-50.035013_{\pm 4\text{e-}5} / 0.034470±4e-50.034470_{\pm 4\text{e-}5}0.032041±2e-50.032041_{\pm 2\text{e-}5} / 0.031475±2e-50.031475_{\pm 2\text{e-}5}0.005141±4e-50.005141_{\pm 4\text{e-}5}
30.002727 / 0.0026570.011595 / 0.0114470.012982±2e-50.012982_{\pm 2\text{e-}5} / 0.012832±1e-50.012832_{\pm 1\text{e-}5}0.012123±1e-50.012123_{\pm 1\text{e-}5} / 0.011985±9e-60.011985_{\pm 9\text{e-}6}0.001792±2e-50.001792_{\pm 2\text{e-}5}
40.001341 / 0.0013030.004207 / 0.0040960.004649±8e-60.004649_{\pm 8\text{e-}6} / 0.004542±6e-60.004542_{\pm 6\text{e-}6}0.004373±5e-60.004373_{\pm 5\text{e-}6} / 0.004267±4e-60.004267_{\pm 4\text{e-}6}0.000622±9e-60.000622_{\pm 9\text{e-}6}
50.000644 / 0.0006290.002038 / 0.0019880.002518±4e-60.002518_{\pm 4\text{e-}6} / 0.002463±4e-60.002463_{\pm 4\text{e-}6}0.002443±6e-60.002443_{\pm 6\text{e-}6} / 0.002389±6e-60.002389_{\pm 6\text{e-}6}0.000571±6e-60.000571_{\pm 6\text{e-}6}
60.000128 / 0.0001290.000796 / 0.0007830.000830±1e-60.000830_{\pm 1\text{e-}6} / 0.000817±1e-60.000817_{\pm 1\text{e-}6}0.000817±1e-60.000817_{\pm 1\text{e-}6} / 0.000804±1e-60.000804_{\pm 1\text{e-}6}0.000061±1e-60.000061_{\pm 1\text{e-}6}
70.000186 / 0.0001800.000330 / 0.0003270.000340±1e-60.000340_{\pm 1\text{e-}6} / 0.000335±1e-60.000335_{\pm 1\text{e-}6}0.000335±1e-60.000335_{\pm 1\text{e-}6} / 0.000331±0e-60.000331_{\pm 0\text{e-}6}0.000027±1e-60.000027_{\pm 1\text{e-}6}
80.000087 / 0.0000850.000285 / 0.0002770.000357±1e-60.000357_{\pm 1\text{e-}6} / 0.000349±1e-60.000349_{\pm 1\text{e-}6}0.000340±1e-60.000340_{\pm 1\text{e-}6} / 0.000333±1e-60.000333_{\pm 1\text{e-}6}0.000091±1e-60.000091_{\pm 1\text{e-}6}
90.000014 / 0.0000130.000105 / 0.0001030.000107±0e-60.000107_{\pm 0\text{e-}6} / 0.000105±0e-60.000105_{\pm 0\text{e-}6}0.000106±0e-60.000106_{\pm 0\text{e-}6} / 0.000103±0e-60.000103_{\pm 0\text{e-}6}0.000007±0e-60.000007_{\pm 0\text{e-}6}
  1. Independence from DINO: Using the independent feature extractor, AID-VAR still achieves lower raw Joint FD than VAR at every single scale transition (k=1…9k=1 \dots 9), and even at Scale 6 where Marginal FD is virtually identical (0.000128 vs. 0.000129), AID-VAR achieves a lower Joint FD (0.000783 vs. 0.000796).
  2. Sensitivity to Cross-Scale Pairing: At most transistions, destroying cross-scale correspondence via Global or Intra-Class Shuffling significantly degrades Joint FD (e.g., increasing Joint FD by +20.2% under intra-class shuffling at Scales 5 and 8, and jumping from 0 to 0.00636 on shuffled real data) while Marginal FD remains fixed.

(b) Human Structural-Consistency Judgments

We conducted a blinded human evaluation involving 15 raters. Each rater evaluated paired generations across a 7-point Likert scale spanning four key dimensions. Specifically, the raters assessed a set of 20 randomly sampled images, yielding the comparative results detailed below:

  • Overall Preference: 60.0% (9/15) preferred AID-VAR, 33.3% (5/15) preferred standard VAR, and 6.7% (1/15) rated a tie.
  • Multi-Dimensional Scores (Mean on 7-point scale, AID-VAR vs. VAR):
  • Structural Integrity: 6.33 vs. 6.00 (+0.33)
  • Detail & Texture: 6.27 vs. 5.87 (+0.40)
  • Visual Naturalness: 6.27 vs. 5.93 (+0.34)
  • Semantic Alignment: 6.27 vs. 6.13 (+0.14)
AuthorsOpenReview ↗

Response to Reviewer Vv9L [3/3]

(c) Mathematical Clarifications (Weight Normalization, First Transition, Near-Zero Handling)

  • Weight Normalization: In Eq. (10), weights are normalized as αk=2k∑j=1K−12j\alpha_k = \frac{2^k}{\sum_{j=1}^{K-1} 2^j} such that ∑kαk=1\sum_k \alpha_k = 1. The exponential weighting is a heuristic preference for later refinement stages, not a derivation from the token counts, which do not double at every scale. As shown in the raw Joint FD table above, AID-VAR consistently outperforms VAR even under uniform weights or unweighted per-scale comparisons. First-transition qualification. With zero-based scale indexing, transition 1 pairs the 1×1 and 2×2 representations. Intra-class shuffling decreases its joint FD from 0.071651 to 0.070689 for VAR and from 0.070700 to 0.069904 for AID-VAR. Thus, shuffling does not uniformly worsen the metric. A single spatial token provides a different representation of structure from later spatial grids, but the present experiment does not establish the cause of this reversal. The result limits a strictly monotonic interpretation of ISCS as consistency: it measures agreement with reference joint statistics, and disrupting a generated association can sometimes bring those statistics closer to the reference. At transitions 2–8, intra-class shuffling increases FD for both models; at transition 9, the AID difference is unresolved at the displayed precision.

Numerical Handling Near Zero: To prevent division-by-zero instability when the Fréchet distance between real and generated features at scale k approaches zero, we implement Eq. (9) by adding a small constant epsilon (1e-8) to the denominator before taking its reciprocal. We additionally apply covariance regularization by adding 1e-6 times the identity matrix to each covariance matrix, and perform the matrix square-root computation in double precision (float64) for numerical stability.

Reviewer_8iQZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AID-VAR, a lightweight guidance framework designed to mitigate cross-scale error accumulation in pretrained visual autoregressive models without modifying their backbone weights, training data, or sampling schedules. A Guidance Injector conditions on features from the preceding scale to produce spatial guidance for next-scale prediction. During training, a discriminator comprising a frozen DINO encoder and trainable heads provides adversarial supervision on decoded RGB images. Soft-label decoding establishes a differentiable path from image-space supervision to the injector, while an auxiliary token-level reconstruction loss regularizes the guided predictions.

Critical strengths
  1. Simple and intuitive design. The method uses adversarial supervision to train a lightweight guidance module that provides spatial guidance during coarse-to-fine generation while keeping the pretrained VAR backbone frozen. At inference, only the guidance module is required alongside the backbone, with no need to run the discriminator. This offers a parameter-efficient and readily integrable approach to enhancing existing VAR models, with clear practical value.

  2. Practical, parameter-efficient enhancement of pretrained generators. The framework trains only the lightweight guidance injector and discriminator heads while preserving the original VAR parameters. The discriminator is unnecessary at inference, and the number of generation scales remains unchanged. This design has a clear deployment rationale: improving the generative capabilities of existing models without retraining the entire backbone or altering the sampling schedule.

Critical weaknesses
  1. Insufficient direct evidence of cross-scale error correction. Higher ISCS indicates that the joint feature statistics of adjacent scales are closer to those of real data, but does not by itself establish that the model better corrects errors introduced at earlier scales. Such improvements may also reflect better image quality at individual scales. Similarly, CKA measures representational association rather than the correctness of intermediate predictions. Therefore, the current evidence does not sufficiently distinguish cross-scale error correction from general adversarial quality enhancement.

  2. Clarification of the reference-based evaluation. I appreciate the authors’ explicit acknowledgment of the reporting limitations in Appendix F.1. Could the authors clarify how generated samples are paired with GT references in Table 12, whether identical pairing and resizing procedures are used for VAR and AID-VAR, and the configurations used for LPIPS and CKA? These details would help assess the extent to which the supplementary results support the proposed cross-scale correction interpretation.

Questions for authors
  1. Direct validation of error correction. Could the authors extend the noise-injection analysis in Figures 8–9 to compare VAR and AID-VAR under matched perturbation locations, strengths, and sampling conditions? Measuring how perturbations propagate relative to each model’s unperturbed trajectory, alongside final-image quality, would help distinguish improved error correction from general quality enhancement.

  2. Validation of ISCS. How do the authors establish that ISCS captures cross-scale dependencies rather than primarily reflecting image quality at individual scales? For example, could the authors evaluate its sensitivity to shuffling the correspondence between adjacent-scale samples while preserving the image collection at each scale?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8iQZ [1/3]

We thank Reviewer 8iQZ for recognizing the simplicity, intuitiveness, and practical parameter efficiency of AID-VAR. Below we address your questions with new controlled experiments and protocol specifications.

1. Direct Validation of Error Correction via Matched Perturbations (W1 & Q1)

Following your suggestion, we extended the noise-injection analysis in Figures 8 to compare VAR-d16 and AID-VAR-d16 under strictly matched conditions. Specifically, both models share the exact same coarse prefix up to Scale 4, receive the exact same Gaussian perturbation (σ=0.25\sigma = 0.25) at Scale 5, and continue generation under identical sampling conditions (cfg=1.5, top-k=900, top-p=0.96, fixed random seeds, evaluated across 5,000 sampled images).

We have included the clean/noisy trajectory comparison in figure 11 in the revised manuscript.

To separate cross-scale error correction from general single-scale quality enhancement, we measure the per-scale perceptual divergence (LPIPS) between each model's perturbed trajectory and its own unperturbed (clean) trajectory across subsequent scales (Scales 6–10):

Scale Transition (Post-Perturbation at Scale 5)Scale 6Scale 7Scale 8Scale 9Scale 10 (Final)
Standard VAR (LPIPS(NoisyVAR,CleanVAR)↓\mathrm{LPIPS}(\mathrm{Noisy}_{\mathrm{VAR}}, \mathrm{Clean}_{\mathrm{VAR}})\downarrow)0.32080.41310.52580.57440.5625
AID-VAR (Ours) (LPIPS(NoisyAID,CleanAID)↓\mathrm{LPIPS}(\mathrm{Noisy}_{\mathrm{AID}}, \mathrm{Clean}_{\mathrm{AID}})\downarrow)0.00050.00610.14820.25210.3160
Relative change in LPIPS-99.8%-98.5%-71.8%-56.1%-43.8%

Key Observations:

  1. Immediate Damping vs. Cascading Amplification: Immediately after the Scale 5 perturbation, standard VAR suffers a large trajectory deviation at Scale 6 (LPIPS = 0.3208), which reaches 0.5625 at Scale 10. In contrast, AID-VAR exhibits much smaller clean/noisy divergence at Scales 6 and 7 (LPIPS = 0.0005 and 0.0061).
  2. 43.8% Reduction in Terminal Trajectory Drift: By Scale 10, AID-VAR reduces final trajectory divergence relative to its own unperturbed trajectory from 0.5625 to 0.3160 (-43.8%). Because this metric measures self-consistency against each model's own clean baseline rather than absolute image realism, it supports reduced trajectory sensitivity under this intervention. It does not alone establish improved final-image quality, preservation of diversity, or robustness to other perturbation strengths and locations.
  3. Qualitative illustration: The revised manusript have presented the qualitative comparison alongside these measurements. In the reported examples, standard VAR (VAR-noisy) exhibits noticeable structural degradation and delayed recovery following the Scale 5 noise injection. Conversely, AID-VAR (AID-VAR-noisy) tracks its clean baseline (AID-VAR-clean) much more closely.
AuthorsOpenReview ↗

Response to Reviewer 8iQZ [2/3]

2. Validation of ISCS via Adjacent-Scale Shuffling & Independent Features (W1 & Q2)

To verify that ISCS captures genuine cross-scale dependencies rather than merely reflecting marginal per-scale image quality or DINO training alignment, we conducted a controlled shuffling experiment over 50,000 samples (1,000 classes, 10 scales) using an independent feature extractor (the frozen VAR representation space, decoupled from the DINO discriminator).

While keeping the image collection at each individual scale—and thus all Marginal per-scale FDs—strictly unchanged, we evaluated raw Joint FD (↓\downarrow) under: (1) Original Pairing, (2) Global Random Shuffling (20 independent trials), and (3) Intra-Class Random Shuffling (shuffling pairings only within the same ImageNet class, 20 trials), plus a Real-Data Shuffling Sanity Check:

Scale kkMarginal FD (VAR / AID)Orig. Joint FD (VAR / AID)Global Shuffled Joint FD (VAR / AID, 20 trials)Intra-Class Shuffled Joint FD (VAR / AID, 20 trials)Real Data Shuffled FD (Sanity Check)
10.021789 / 0.0213030.071651 / 0.0707000.072439±6e-50.072439_{\pm 6\text{e-}5} / 0.071633±8e-50.071633_{\pm 8\text{e-}5}0.070689±6e-50.070689_{\pm 6\text{e-}5} / 0.069904±6e-50.069904_{\pm 6\text{e-}5}0.006361±1e-40.006361_{\pm 1\text{e-}4}
20.008535 / 0.0084540.031282 / 0.0306870.035013±4e-50.035013_{\pm 4\text{e-}5} / 0.034470±4e-50.034470_{\pm 4\text{e-}5}0.032041±2e-50.032041_{\pm 2\text{e-}5} / 0.031475±2e-50.031475_{\pm 2\text{e-}5}0.005141±4e-50.005141_{\pm 4\text{e-}5}
30.002727 / 0.0026570.011595 / 0.0114470.012982±2e-50.012982_{\pm 2\text{e-}5} / 0.012832±1e-50.012832_{\pm 1\text{e-}5}0.012123±1e-50.012123_{\pm 1\text{e-}5} / 0.011985±9e-60.011985_{\pm 9\text{e-}6}0.001792±2e-50.001792_{\pm 2\text{e-}5}
40.001341 / 0.0013030.004207 / 0.0040960.004649±8e-60.004649_{\pm 8\text{e-}6} / 0.004542±6e-60.004542_{\pm 6\text{e-}6}0.004373±5e-60.004373_{\pm 5\text{e-}6} / 0.004267±4e-60.004267_{\pm 4\text{e-}6}0.000622±9e-60.000622_{\pm 9\text{e-}6}
50.000644 / 0.0006290.002038 / 0.0019880.002518±4e-60.002518_{\pm 4\text{e-}6} / 0.002463±4e-60.002463_{\pm 4\text{e-}6}0.002443±6e-60.002443_{\pm 6\text{e-}6} / 0.002389±6e-60.002389_{\pm 6\text{e-}6}0.000571±6e-60.000571_{\pm 6\text{e-}6}
60.000128 / 0.0001290.000796 / 0.0007830.000830±1e-60.000830_{\pm 1\text{e-}6} / 0.000817±1e-60.000817_{\pm 1\text{e-}6}0.000817±1e-60.000817_{\pm 1\text{e-}6} / 0.000804±1e-60.000804_{\pm 1\text{e-}6}0.000061±1e-60.000061_{\pm 1\text{e-}6}
70.000186 / 0.0001800.000330 / 0.0003270.000340±1e-60.000340_{\pm 1\text{e-}6} / 0.000335±1e-60.000335_{\pm 1\text{e-}6}0.000335±1e-60.000335_{\pm 1\text{e-}6} / 0.000331±0e-60.000331_{\pm 0\text{e-}6}0.000027±1e-60.000027_{\pm 1\text{e-}6}
80.000087 / 0.0000850.000285 / 0.0002770.000357±1e-60.000357_{\pm 1\text{e-}6} / 0.000349±1e-60.000349_{\pm 1\text{e-}6}0.000340±1e-60.000340_{\pm 1\text{e-}6} / 0.000333±1e-60.000333_{\pm 1\text{e-}6}0.000091±1e-60.000091_{\pm 1\text{e-}6}
90.000014 / 0.0000130.000105 / 0.0001030.000107±0e-60.000107_{\pm 0\text{e-}6} / 0.000105±0e-60.000105_{\pm 0\text{e-}6}0.000106±0e-60.000106_{\pm 0\text{e-}6} / 0.000103±0e-60.000103_{\pm 0\text{e-}6}0.000007±0e-60.000007_{\pm 0\text{e-}6}

(Note: Scale 0 Marginal FD is 0.045130 for VAR and 0.044798 for AID-VAR.)

First-transition qualification. With zero-based scale indexing, transition 1 pairs the 1×1 and 2×2 representations. Intra-class shuffling decreases its joint FD from 0.071651 to 0.070689 for VAR and from 0.070700 to 0.069904 for AID-VAR. Thus, shuffling does not uniformly worsen the metric. A single spatial token provides a different representation of structure from later spatial grids, but the present experiment does not establish the cause of this reversal. The result limits a strictly monotonic interpretation of ISCS as consistency: it measures agreement with reference joint statistics, and disrupting a generated association can sometimes bring those statistics closer to the reference. At transitions 2–8, intra-class shuffling increases FD for both models; at transition 9, the AID difference is unresolved at the displayed precision.

Key Findings:

  1. High Sensitivity to Pairing Destruction: While Marginal FD remains strictly constant under shuffling, breaking adjacent-scale correspondence significantly increases joint FD at many transitions, with the first-transition exception discussed above—e.g., at Scales 5 and 8, Global Shuffling increases Joint FD by +23.6% to +26.0%, and even Intra-Class Shuffling (which preserves class semantics and breaks instance-level pairing) increases Joint FD by +19.3% to +20.2% (the reported deviations describe permutation variability, not independent training-run uncertainty).
  2. Complementary Human Evaluation: A blinded study with 15 human raters (7-point scale) provides complementary perceptual evidence, without directly validating ISCS: raters preferred AID-VAR over VAR (60.0% vs. 33.3%, 6.7% tie), scoring AID-VAR higher in Structural Integrity (6.33 vs. 6.00) and Detail & Texture (6.27 vs. 5.87).
AuthorsOpenReview ↗

Response to Reviewer 8iQZ [3/3]

3. Clarification of Reference-Based Evaluation Protocols (W2)

We have described the reference-based protocols in Appendix E:

  • Pairing & Resizing Protocol: Because unconstrained class-conditional generation lacks a paired pixel-level reference, we use a Prefix-Anchored Rollout protocol on the ImageNet validation set. For each real validation image IGTI_{\text{GT}}, we encode and fix its coarsest k0=2k_0 = 2 ground-truth token maps to anchor the global composition/pose, and then run free-running autoregressive generation (with vs. without AID-VAR) for all subsequent scales under identical random seeds.
  • Resizing: Ground-truth images IGTI_{\text{GT}} undergo shorter-side Bicubic resizing to 256 (antialias=True) followed by a 256×256256 \times 256 center crop. Intermediate generated scales are accumulated onto the 16×1616 \times 16 latent grid via the residual upsampling operator Φj\Phi_j and natively decoded by the VQ-VAE decoder into 256×256256 \times 256 RGB images. Identical pairing and resizing procedures are applied to both VAR and AID-VAR.
  • LPIPS Configuration: Computed using lpips v0.1 with a pretrained VGG-16 backbone (net='vgg') and learned linear calibration layers.
  • CKA Configuration:
  • Feature Layer: Extracted from the output hidden states of the final Transformer block (Layer 16, post-LayerNorm, dimension D=1024D = 1024) of VAR-d16.
  • Spatial Alignment: Intermediate feature maps of shape pk×pk×Dp_k \times p_k \times D are spatially aligned across scales via Global Average Pooling (GAP) into RD\mathbb{R}^D vectors.
  • Kernel & Sample Count: Computed using Linear CKA with the unbiased HSIC estimator over N=5,000N = 5,000 generated trajectories (5 samples per class across all 1,000 classes, fixed seed).

Thank you for the clarifications and additional experiments. I continue to recognize the practical value of adapting a frozen VAR backbone through a lightweight, trajectory-conditioned spatial guidance module. The rebuttal addresses some of my original concerns: the matched-perturbation experiment provides more direct evidence of reduced perturbation sensitivity, while the pairing-shuffle controls and evaluation-protocol clarifications improve the interpretability of the reported results. However, these findings do not fully establish that the improvements are specifically attributable to cross-scale error correction.

I also share Reviewer J9Ki's concern regarding the reliability of the reported empirical results. Combining externally sourced baseline means with standard deviations from the authors' own multi-seed runs makes the baselines appear to have been directly reproduced under a controlled evaluation. This is a serious reporting error that should not have occurred.

Taking the corrected results and additional experiments at face value, they still support the effectiveness of the method under the reported settings. I therefore provisionally maintain my current rating. However, I now view the submission as borderline, given the reporting error and the remaining limitations in attributing the gains to cross-scale error correction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.