acceptodds
Under review as a conference paper at ICLR 2027

Refine or Preserve? Acceptance-Gated Correction for Autoregressive Image Generation

Abstract

Post-hoc refinement can repair degraded autoregressive image samples, but the same intervention can damage outputs that are already well tuned. We therefore ask a prior question: should a generated sample be refined at all? We introduce Acceptance-Gated Refinement (AGR), a two-level correction policy for frozen autoregressive image generators. An image-level router chooses between exact preservation and refinement, while a token-level gate proposes masked-token replacements and commits only edits that pass a refiner-score test. The router is derived from the gate's own initial refiner pass and calibrated per domain using tuned and severely degraded validation samples; no degradation label is required at test time. Across FFHQ, CelebA-HQ, and ImageNet, AGR bypasses at least 98.5% of tuned samples, with observed FID changes of only +0.06, +0.08, and −0.01, while reducing severe-input FID from 102.71 to 47.36, 81.26 to 30.17, and 96.25 to 58.09. Mild inputs excluded from router fitting also improve. A router fitted only on temperature shifts improves unseen 4-bit quantized samples while bypassing top- and top- truncated samples for which forced refinement is harmful. At matched edit rates, random editing degrades mild inputs whereas score-gated editing improves them. Across thirteen settings in three domains, refinement gain increases with the input's score deficit under the refiner. These results show that a refiner can provide not only candidate corrections, but also a practical signal for deciding when to refine and when to preserve.

Primary area:
generative models
Keywords:
autoregressive image generation, test-time refinement, selective correction, token-level gating, pseudo-likelihood, masked generative models, abstention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ogUdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses post-hoc correction of degraded outputs from autoregressive image generators and proposes Acceptance-Gated Refinement (AGR), a two-level framework combining an image-level router with a token-level acceptance gate. The router uses refiner-derived scores to decide whether to preserve an input unchanged or refine it, while the gate selectively accepts proposed token replacements based on their score gains. Experiments on FFHQ, CelebA-HQ, and ImageNet demonstrate improvements in the quality of degraded samples while largely preserving well-tuned outputs.

Critical strengths

1.The proposed method is simple, intuitive, and easy to understand.

  1. The paper provides reasonably comprehensive internal validation on FFHQ, CelebA-HQ, and ImageNet, covering well-tuned samples, degraded samples, and unlabeled mixtures.
Critical weaknesses
  1. The methodological novelty appears limited. Iterative refinement, confidence-based token selection, and quality-guided acceptance have been explored in prior work. The paper does not sufficiently clarify its technical advance beyond combining these established ideas into a two-level decision framework.

  2. The application scope is narrow, with experiments mainly focusing on correcting degraded autoregressive image samples. Many degradations are artificially introduced, and their prevalence in practical generation remains unclear. Comparisons with independent external baselines are also limited. I strongly recommend that the authors include more baselines.

Questions for authors

See the weaknesses.

3 comments

We thank the reviewer for the careful reading, and here are some experiments we results we add for rebuttal.

R3 — W1. Limited methodological novelty

Here is the novelty comparison table, same with R2-1, so based on the table. The contribution is knowing which images to skip, without any label, from 25 samples, and having that judgment carry over to damage and generators it never saw.

Table R3-1. What is new relative to prior work.

componentclosest prior workwhat is new here
image-level routerlikelihood thresholds for OOD detection (Nalisnick et al., 2019)used as a repair switch with exact bypass; fit from 25+25 images; transfers to unseen damage (opens on quantization, stays closed on truncation), to a real quantizer (GPTQ, R3-1), and to a larger generator with the cutoff unchanged (GPT-XL → GPT-XXL)
token-level gateToken-Critic (Lezama et al., 2022); MaskGIT (Chang et al., 2022)accept an edit only if its score gain exceeds τ; a rate-matched random control shows the acceptance rule, not the edit budget, drives the gain on mild input, where random edits hurt (R2-1b); the router also wraps other repair rules (routed SDEdit leaves tuned images untouched and keeps the severe gain, R3-2)
evidence–per-image deficit predicts repair gain within every condition (R1-1); the quantized generator's own likelihood misses its quantization damage, while the refiner, already loaded for repair, catches it with no extra memory (a full-precision copy adds 1478 MB on GPT-XL, R1-3); the 1-pass unmasked score routes like the 16-pass masked score (R2-3)

R3 — W2. Narrow scope; artificial degradations; prevalence

The paper already repairs five frozen backbones from 111M to 1.4B parameters: LlamaGen GPT-B, GPT-L and GPT-XL (384²), a 2021 VQGAN transformer, and the random-order RandAR-L. They cover two tokenizer families, 256² and 384² resolution, raster and random order, and three datasets (FFHQ, CelebA-HQ, ImageNet). The rebuttal adds GPT-XXL (Table G1), a real quantizer (below), and the router wrapping a non-AR repairer (SDEdit, Table R3-2).

And we think the degradations are practical. GPTQ quantization is how people serve large models cheaply. Speculative decoding is how they speed up AR generation. Temperature and guidance are user-facing settings. These aren’t artificial.

Also the router scales with the damage. Fit only on temperature, it opens proportionally on real GPTQ quantization: a lot when GPTQ hurts, barely at all when GPTQ is harmless. Repairing everything when it’s harmless would cost 8–11 FID. The router avoids that. The router also abstains where repair would not help (no CFG, truncation).

Table G1 Full method on stronger generators

generatorinputnno repairalways-onfixed τ (val)AGRrepaired
GPT-XL-384tuned (cfg 2)10k6.7012.48off: 6.706.651.2%
mild (cfg 1, t=1.2)10k57.9854.0645.02 (τ2)50.4166.3%
severe (cfg 1, t=1.5)10k140.1979.9679.96 (τ1)79.9799.4%
tuned + severe10k51.64–29.07 (τ2, all)26.69–
tuned + mild10k20.71–16.72 (τ2, all)17.59–
GPT-XXL-384tuned (cfg 1.75)5k8.9215.99–8.921.6%
severe (cfg 1, t=1.5)5k142.0983.68–83.6399.4%
tuned + severe5k55.4839.47–31.96–

Table R3-1. Deployment settings (FID, lower is better).

settinggeneratornno repairAGRrepaired
relaxed speculative decoding (LANTERN, 2.3x)ImageNet GPT-B10k74.8452.80 (AGR-P τ=1)all
GPTQ 4-bit (full precision 7.39)FFHQ GPT-B10k10.059.80 (always-on 18.64)6.2%
GPTQ 3-bitFFHQ GPT-B10k33.8829.17 (always-on 32.51)29.8%
GPTQ 4-bit (full precision 19.72, same split)GPT-XL-3842k19.0719.13 (always-on 27.20)1.5%
GPTQ 3-bitGPT-XL-3842k21.2521.44 (always-on 32.42)2.8%
4-bit rounding (RTN)FFHQ GPT-B10k33.0231.6810.9%
3-bit rounding (RTN)FFHQ GPT-B10k285.70200.93100%
3-bit rounding (RTN)GPT-XL-3842k131.32123.1935.3%
temperature 1.5GPT-XXL-3845k142.0983.6399.4%
temperature 1.2GPT-XL-38410k57.9850.4166.3%
no classifier-free guidanceImageNet GPT-B10k26.0826.11 (always-on 29.57)1.8%
top-k 16 / top-p 0.80FFHQ GPT-B10k73.26 / 27.48unchanged (always-on 78.79 / 40.19)0% / 0%

Also here are more baseline

We added five, one per row of Table G2:

  • the generator's own likelihood as router, deployed and full precision (Table R1-3);
  • compute-matched best-of-N (Reiviewer-2);
  • a fixed τ chosen per condition (G1);
  • one fixed τ for every image on GPT-XL (G1b);
  • SDEdit with a pretrained DiT-XL/2-256 (R3-2).

They come on top of the rate-matched random gate and the ungated limit in the paper, and the Token-Critic-style confidence rule (R2-1b).

Table R3-2. SDEdit with a pretrained DiT-XL/2-256 vs AGR (ImageNet GPT-B, first 5k test images; s = SDEdit strength; "kept" = mean Inception-feature cosine similarity between input and output; two unrelated same-class images score 0.72).

no repairAGRSDEdit s=0.2s=0.4s=0.6s=0.4 routed by AGRDiT sampled directly
FID tuned (cfg 2)11.1011.0910.689.959.6811.0910.02
FID mild (cfg 1, t=1.2)55.9947.9242.9818.689.0432.1110.02
FID severe (cfg 1, t=1.5)98.6061.5179.3527.2210.2827.4610.02
seconds per edited image–0.1140.1140.2050.2960.2050.686 per sample
kept, tuned1.0000.9990.8920.8040.7450.999–
kept, mild1.0000.8730.7960.6670.6060.823–

Table G1b. GPT-XL mixtures: AGR vs one fixed τ for every image (label-free baseline; 10k per mixture; per-condition columns are the 5k halves used, biased upward vs 10k).

policytuned + severetuned + mildtuned (5k)mild (5k)severe (5k)
no repair51.6420.719.8161.64143.40
one τ=3 for all40.7417.879.8453.62116.59
one τ=2 for all29.0716.7210.3348.3486.28
AGR26.6917.599.7754.3983.09
AGR, τ=2 open branch28.2616.899.8151.6486.33

The authors have addressed some of my concerns, but I still have reservations about the paper’s methodological novelty and scope of applicability. I therefore maintain my original rating.

Reviewer_ZScrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper addresses post-hoc correction of samples from a frozen autoregressive image generator. Since refinement can damage already well-tuned samples, the authors propose AGR: an image-level router thresholds the mean refiner log-score from one forward pass to decide between exact bypass and repair, and a token-level gate resamples low-confidence positions, accepting only edits whose refiner score improves by more than τ. On FFHQ, CelebA-HQ, and ImageNet, AGR bypasses ≥98.5% of tuned samples with negligible FID change and roughly halves the FID of severely degraded ones. A router calibrated only on temperature shifts transfers to unseen quantization (opens) and truncation (stays closed), which the authors attribute to repair gain growing with the input’s refiner-score deficit.

Critical strengths

The paper cleanly separates post-hoc correction into “whether to refine” and “which edits to keep,” and isolates each component with rigorous ablations: exact bypass gives a clear preservation guarantee, a rate-matched random control shows the gain comes from the acceptance criterion rather than the edit budget, held-out interventions are excluded from fitting and unified by the score-deficit relation, and the writing is honest about what each result establishes.

Critical weaknesses

1: Limited novelty. The router is a mean-likelihood OOD threshold; the gate is Token-Critic-style editing (resample low-confidence tokens, accept if the score improves). Both are known; the contribution is the combination and the experiments.

2: Repair vs. resampling is not costed. Repair takes 24 refiner forwards per image. Why not generate one or two more samples and pick the best? The paper’s own Best-of-4 beats AGR on mild FFHQ. And if the refiner score can judge sample quality, why not guide or prune during generation instead of fixing afterwards? The speculative-decoding composition in the appendix reaches only FID 52 vs. a baseline of 8, so that route is unresolved.

3: Not validated on strong models. The full method is tested only on GPT-B (FID 8.18). Larger models get gate-only results with no gain on tuned sampling (GPT-XL 6.70→6.71). SOTA AR models (AliTok, RAR) reach FID 1.3–1.5 on ImageNet 256. By the paper’s own logic, stronger generators leave less to fix, so the method may have no room to apply.

Overall: the motivation is sound, but the method is a no-op on tuned samples, far from regeneration on degraded ones, and untested on competitive generators, so its practical value is unclear.

Questions for authors

1: At matched compute (24 refiner forwards per image vs. a few extra generations), how does AGR compare to best-of-N resampling in FID? 2: Can you report full router+gate results on a strong backbone (e.g., AliTok or RAR, FID ≈ 1.3–1.5)? By the score-deficit argument, what opening rate and gain would you expect? 3: The face refiners are trained only on masked positions, so unmasked-position log-scores are unsupervised outputs. Why are they valid as router and gate scores? 4: If the refiner score can judge sample quality, why not guide or prune during generation? The speculative-decoding composition in Appendix G reaches only FID 52; what is the bottleneck? 5: Router calibration requires running the generator at tuned settings to produce validation samples. If that is possible, why not simply regenerate?

3 comments

We thank the reviewer for the careful review and for recognizing the ablations.

W1. Limited novelty

Here we show a novelty comparison table. The contribution is knowing which images to skip, without any label, from 25 samples, and having that judgment carry over to damage and generators it never saw.

Table R2-1. What is new relative to prior work.

componentclosest priorwhat is new
image-level routerlikelihood OOD thresholds (Nalisnick et al., 2019)repair switch with exact bypass; fit from 25+25 images; transfers to unseen damage (opens on quantization, stays closed on truncation), to a real quantizer (GPTQ, R3-1), and to a larger generator with the cutoff unchanged (GPT-XL → GPT-XXL)
token-level gateToken-Critic (Lezama et al., 2022); MaskGIT (Chang et al., 2022)score-gain threshold τ; rate-matched random control shows the acceptance rule, not the edit budget, drives the gain on mild input where random edits hurt (R2-1b); router wraps other repair rules (confidence ranking; routed SDEdit, R3-2)
evidence–per-image deficit predicts gain within every condition (R1-1); quantized generator's own likelihood misses its damage while the refiner catches it at no extra memory (R1-3); 1-pass unmasked score routes like 16-pass masked score (R2-3)

W2 / Q1. Repair vs resampling at matched compute

AGR is cheap, where detection is nearly free and repair runs only on flagged images. Total overhead is 13–30% of one generation. On damaged input, resampling can’t help, every new sample carries the same damage, so more samples don’t fix it but AGR does. On mild input, resampling wins but it costs 2–4× a generation. At the same budget AGR is still better than doing nothing, more reuslts are here at Table R2-2

**Table R2-2 measures wall-clock on one GPU (batch 25)

methodmild (t=1.2): costFIDsevere (t=1.5): costFID
single sample1.0024.891.00102.71
AGR1.1318.551.3047.36
always-on repair1.2921.621.2947.38
best-of-2, gen. likelihood2.0217.322.0283.59
best-of-2, refiner score2.0221.072.0294.42
best-of-4, gen. likelihood4.0513.334.0570.25
best-of-4, refiner score4.0518.314.0586.93
AGR on best-of-2 (gen. LL)2.1114.702.3339.51

W3 / Q2. Strong generators; expected opening rate and gain

First we have to admit that the during the rebuttal time sensitivity, it is not easy for us to get the strong baseline running with the shared GPU but we are running on it and we will share the results here once we obtained. But one thing we want to clarify for AGR is that on stronger baseline where FID is pretty good enough, there will be no room to refine, so we will not expect the gain from the refine, maybe FID will roughly stay the same. AGR's job is to avoid harm when repair is not needed (5.8–11.2 FID saved) and to repair when deployment damages the model, at under 1% added time.

Q3. Why are unmasked scores valid for the face refiners?

The unmasked score makes the right routing and gating decisions on faces even without supervision, but for the gate specifically, a supervised alternative does better, and here below we report both.

Table R2-3. Unmasked vs masked router (fit on validation; 2k test images per cell; FID diff = unmasked − masked AGR on same images, paired bootstrap SD).

cellAUROC: unmasked / maskedagreerepaired: unm / maskedFID diff ± SDSpearman
FFHQ tuned–98.8%1.3% / 0.2%+0.06±0.040.86
FFHQ t=1.20.942 / 0.95280.5%40.0% / 28.6%−0.62±0.300.79
FFHQ t=1.50.999 / 1.00099.5%99.5% / 99.7%−0.06±0.070.59
FFHQ 4-bit0.833 / 0.88690.0%9.8% / 11.7%+0.20±0.260.81
FFHQ 3-bit1.000 / 1.000100%100% / 100%0.00±0.01−0.07
FFHQ top-k 161.000 / 1.000100%0% / 0%–0.96
FFHQ top-p 0.800.955 / 0.962100%0% / 0%–0.92
CelebA tuned–98.6%1.6% / 0.6%–0.90
CelebA t=1.50.999 / 1.00099.8%99.6% / 99.7%–0.77

(3-bit: every image far past cutoff under both scores, hence near-zero within-cell correlation.)

We hope these additions address the reviewer's concerns and would be happy to discuss further.

Thanks the authors for the detailed response.

W1: Partly settled. But this is not the key of rejection. I believe they do not allow us to reject a paper for the limited novelty.

W2Q1: Partly settled.

W3Q2: Not settled. I do understand that extra experiments on sota AR gen are not easy, however, if there is no result reported on a competitive generator, and the authors acknowledge that little gain is expected where FID is already low, how do we know the practical regime where the method matters?

Q3: Mostly solved.

Q4Q5: Not answered.

Overall, the rebuttal adds valuable analysis, and the paper is more transparent as a result. However, my central concerns about practical value (W3Q2 no gain on strong or realistically deployed generators, Q4Q5the unanswered regeneration question). I will keep the score of 2 and raise Soundness from 1 to 2 for the new experiments.

Here are the Q4 and Q5 results

Q4. Why not guide or prune during generation?

The bottleneck in Appendix G is the drafter, not the refiner (Table R2-4). LANTERN accepts 99.5% of draft tokens and shifts the whole sequence (FID 8.2→74.8). AGR-P recovers ~22 FID but edits only the least confident positions, so it cannot undo sequence-level drift.

Using the refiner score to steer generation is not a drop-in fix either. As a ranker among healthy samples it hurts FID, increasingly with N (Table R2-4 bottom; also refiner-score rows in R2-2). The score is a damage detector (R1-1, R2-3), not a quality ranker. In-loop guidance is future work.

Table R2-4. AGR on speculative decoding (ImageNet GPT-B, 10k).

decodingspeed (batch 1)no repair+ AGR-P (τ=1)
plain decoding1.00×8.18–
LANTERN δ=0.012.28×76.4054.27
LANTERN δ=0.032.30×74.8452.80
LANTERN δ=0.12.32×74.8652.86

Refiner as ranker on tuned FFHQ (best-of-N by refiner pseudo-likelihood, 10k):

N124816
FID7.3910.1515.5221.8729.37

Q5. If calibration needs tuned samples, why not regenerate?

Calibration needs 25 tuned images once (R1-5), not a tuned rerun of every output. The degraded settings we target exist for a reason — speed or memory (low-bit weights, speculative decoding). Regenerating at tuned settings gives that up: for quantization it means keeping the full-precision model in memory. It also requires knowing which outputs were hurt. AGR finds them without labels. At AGR's cost (1.1–1.3×), there is no room for even one extra sample (R2-2).

We will add Tables R2-1 through R2-4 and the strong-model results to the revision, and are glad to discuss further.

Meanwhile we are running one stronger baseline RAR, and we will reprort the results, still we want to clarify here is our method serves more like a safety valve to the model which can save the heavy damaged region and leave the perfect image untouched under the situation where model cannot behave well. So at strong baseline, we will not expect a lot gain from there, instead the FID might roughly stay the same or a marginal gain.

Reviewer_hsGTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Acceptance-Gated Refinement (AGR) is a post-hoc correction policy for samples from frozen autoregressive image generators, made up of an image-level router and a token-level gate. The router applies a per-domain cutoff, calibrated on tuned and severely degraded validation samples, to a statistic from a single unmasked refiner pass; images scoring at or below the cutoff are sent to refinement, and all others are returned unchanged. The gate is a parallel masked-proposal refiner that resamples the lowest-confidence positions and commits a proposed token only if its unmasked score in the all-proposals sequence exceeds the current token’s score in the current sequence by more than a threshold τ. The paper reports that AGR nearly preserves tuned-sample FID while recovering most of the severe-regime repair achieved by label-calibrated gating, without needing any degradation label at test time.

Critical strengths
  • The calibration sample-efficiency study (Table 7) and the end-to-end timing table (Table 11) address practical deployment concerns directly: refitting the cutoff from only 25 tuned and 25 severe validation images opens, on average, on at most 1% of tuned and at least 96% of severe test images in every domain, and on tuned FFHQ inputs AGR adds only 0.1% (batch 1) and 1.7% (batch 25) to generation time versus 2.3% and 27.6% for always-on gating at τ = 2. This largely addresses calibration cost and inference overhead as practical barriers to deployment, although small-sample FID is reported only for FFHQ and timing only on FFHQ with a single GPU.
  • The rate-matched random gate, the ungated (τ → −∞) limit, and AGR-without-bypass each isolate a specific component of the method, and the random-gate result on mild inputs is particularly persuasive: at the calibrated gate’s aggregate edit rate, random acceptance worsens FID relative to no repair (24.89 → 29.23 on FFHQ and 16.00 → 23.56 on CelebA-HQ), whereas the calibrated score gate improves it (to 18.50 and 11.95). This shows the acceptance criterion, not a smaller edit budget, drives the repair on mild inputs.
Critical weaknesses
  • The claim that the router estimates corrective leverage rather than detecting temperature rests entirely on an empirical link between refiner-score deficit and repair gain, since its face statistic is simply the mean refiner log-score, and this interpretation is what justifies applying the router to unseen degradations. That link is shown only across 13 condition-level points (Figure 8), every held-out outcome is equally consistent with a detector of score shift (temperature and quantization lower the score, top-k and top-p truncation raise it), and the 3-bit outlier (largest deficit, less gain than t = 1.5, no routed result reported) is only hedged, not explained, so a per-image deficit-versus-gain analysis or a test where deficit and gain could dissociate is needed.
  • Because tuned/severe discrimination is nearly trivial (AUROC ≥ 0.999), AGR reduces to a near no-op on tuned and to the gate at τ = 1 on severe inputs, while on the informative regimes it is never best on mild inputs, its four mixture wins are margins of 0.01–0.23 FID that the authors themselves do not claim, and it trails the regime-aware reference by 0.58 on ImageNet tuned+mild. Its distinctive benefit is therefore, as the authors acknowledge, exact preservation of bypassed images, a nearly definitional property of any bypass rule, yet no alternative router is compared and the router is evaluated only on GPT-B generators; alternatives include the generator’s own log-likelihood, which is available in all reported experiments, and the three unselected candidate statistics, whose AUROCs are not reported.
Questions for authors

Please address the weaknesses.

3 comments

We thank the reviewer for the careful reading and for pinpointing what the leverage claim needs. We ran the requested per-image analysis and now report a routed 3-bit result.

W1. Corrective leverage or a score-shift detector?

The deficit predicts gain per image, not just per condition. Within every condition, images with a higher deficit get more help from repair. Within t=1.2, repair actually hurts the low-deficit images and helps the high-deficit ones. A detector that only knows the condition can’t make that distinction. The router handles unseen damage correctly. It stays closed on truncation, where repair would hurt. On 4-bit, it picks the 11% of images that actually benefit. It was never trained on either. Also we think 3-bit isn’t an outlier. It has the biggest absolute gain. It looked like an outlier in Fig. 8 because of the scale. The leftover FID is model-wide damage that token edits can’t fully fix.

Table R1-1. Gain of always-on repair (τ=1) by deficit quintile (FFHQ, 2k images per cell; KID ×10³; positive = repair helps; deficit in nats/token).

conditionmean deficit, lowest → highest fifthlowest 20%2nd3rd4thhighest 20%slope per nat
tuned t=1.0−0.42 → +0.43−16.9−14.8−11.1−8.1−2.6+17
t=1.2+0.26 → +0.98−6.8+0.3+5.2+10.2+17.3+34
t=1.5+0.91 → +1.43+44.4+57.1+59.5+64.0+67.8+44
4-bit (RTN)+0.03 → +0.72−9.9−5.0−0.5+3.2+10.3+29
3-bit (RTN)+1.40 → +1.84+82.5+97.4+97.6+99.1+104.3+45

Table R1-2. Router on seven FFHQ conditions (FID, 10k images each; cutoff fit on t=1.0 vs t=1.5 only, so the other five are unseen).

conditionmedian deficitrepairedno repairalways-onAGR
tuned t=1.00.001.5%7.3920.437.45
t=1.20.6341.4%24.8921.6218.55
t=1.51.1799.5%102.7147.3847.36
4-bit (RTN)0.3910.9%33.0234.3531.68
3-bit (RTN)1.61100%285.70200.93200.93
top-k 16−1.790%73.2678.7973.26
top-p 0.80−0.750%27.4840.1927.48

We thank the reviewer for suggesting the generator's own likelihood; it is a natural baseline and clarifies where the refiner score matters.

W2. Alternative router, unselected statistics, generators beyond GPT-B

The refiner score is the only router that is both accurate on quantization damage and free at inference, because the refiner is already loaded for repair. A quantized generator can’t score its own error (Table R1-3); a full-precision copy can but costs +1.5 GB. Among refiner-based statistics, ours is the best (Table R1-4); generator likelihood beats it on mild input but fails on quantization. On GPT-XL the method leaves tuned input alone and fixes severe input (G1/G1b). We concede mild — a hand-picked τ does better — but no single τ wins both regimes, and AGR needs no label. The router’s value is choosing which images to skip, from 25 samples, and having that transfer to unseen damage (R1-2, R3-1) and larger generators (G1). Beyond FFHQ, 25+25 calibration images suffice (R1-5), at 0.6% added time.

Table R1-3. Routers compared (identical fit; deployed = the quantized generator that produced the samples).

refiner (ours)gen. LL, deployedgen. LL, full precision
AUROC, FFHQ tuned vs t=1.2 / vs 4-bit0.940 / 0.8330.971 / 0.5060.971 / 0.968
FFHQ 4-bit AGR FID (no repair 33.02)31.68 (10.9% repaired)32.99 (0.3%)30.14 (29.9%)
AUROC, GPT-XL tuned vs 3-bit0.9140.7580.974
GPT-XL 3-bit AGR FID (no repair 131.32, always-on 122.09)123.19 (35.3%)131.32 (0%)123.10 (43.9%)
GPT-XL weights resident (3-bit generator + refiner + extra)574 MB574 MB2052 MB

Table R1-4. AUROC, tuned vs mild (tuned vs severe: all ≥ 0.93; bold = best refiner statistic).

statisticFFHQCelebA-HQImageNet
mean log-score (face router)0.9400.9290.976
mean confidence (ImageNet router)0.8730.8150.979
fraction of proposals with Δ > 00.8320.8300.778
acceptance rate at τ=00.8320.8310.779
generator log-likelihood (new)0.971not run0.988

Table R1-5. Cutoff fit from 25+25 images (mean ± SD over 200 draws for the repaired fraction, 40 for FID; full fit in parentheses).

generatorrepaired, tuned / severeAGR FID, tunedAGR FID, severe
ImageNet GPT-B0.2 ± 0.5% / 95.4 ± 5.3%8.174 ± 0.002 (8.17)58.49 ± 0.67 (57.93)
GPT-XL-3840.4 ± 0.6% / 95.9 ± 3.7%6.685 ± 0.021 (6.65)79.79 ± 0.26 (79.97)

Table G1. Full method on stronger generators. Cutoff fit on 1k+1k GPT-XL validation images (tuned vs t=1.5) only; GPT-XXL uses the same refiner and cutoff (no retraining, no recalibration). Always-on = AGR-P τ=1 on every image. Fixed τ per condition is chosen on validation and needs the condition label.

generatorinputnno repairalways-onfixed τ (val)AGRrepaired
GPT-XL-384tuned (cfg 2)10k6.7012.48off: 6.706.651.2%
mild (cfg 1, t=1.2)10k57.9854.0645.02 (τ2)50.4166.3%
severe (cfg 1, t=1.5)10k140.1979.9679.96 (τ1)79.9799.4%
tuned + severe10k51.64–29.07 (τ2, all)26.69–
tuned + mild10k20.71–16.72 (τ2, all)17.59–
GPT-XXL-384tuned (cfg 1.75)5k8.9215.99–8.921.6%
severe (cfg 1, t=1.5)5k142.0983.68–83.6399.4%
tuned + severe5k55.4839.47–31.96–

Table G1b. GPT-XL mixtures: AGR vs one fixed τ for every image (label-free baseline; 10k per mixture; per-condition columns are the 5k halves used, biased upward vs 10k).

policytuned + severetuned + mildtuned (5k)mild (5k)severe (5k)
no repair51.6420.719.8161.64143.40
one τ=3 for all40.7417.879.8453.62116.59
one τ=2 for all29.0716.7210.3348.3486.28
AGR26.6917.599.7754.3983.09
AGR, τ=2 open branch28.2616.899.8151.6486.33

We hope these additions address the reviewer's concerns and would be happy to discuss further.

I thank the authors for a thorough rebuttal. The new experiments address several of my requests: (i) AUROCs for the unselected router statistics, (ii) generator likelihood as an alternative router, (iii) full-method results on GPT-XL and GPT-XXL, and (iv) a routed 3-bit result. The held-out routing results also show that the router does more than bypass tuned images.

My main concern is only partly resolved. The per-image deficit–gain covariance (R1-1) is also predicted by a score-shift detector, so it does not separate that reading from corrective leverage. It also shows a matched-deficit mismatch (~0.9–1.0 nats: +44.4 at t=1.5 vs. +17.3 at t=1.2). In R1-3, full-precision generator likelihood matches or exceeds the refiner score on every reported AUROC and AGR FID. The refiner’s advantage is therefore operational, not evidence of measuring corrective leverage.

Overall, I view AGR as a practical, label-free bypass rule with solid preservation behavior. I appreciate the authors’ efforts, but given the remaining concern, I am inclined to keep my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.