acceptodds
Withdrawn from ICLR 2027

SparseGRN: Accelerating Generative Refinement Networks with Adaptive Sparse Queries

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generative Refinement Networks (GRNs) synthesize images by repeatedly revisiting the complete token map, enabling global error correction but incurring substantial inference cost. Recomputing every visual token at every refinement step introduces redundancy as image content stabilizes. We present SparseGRN, a training-free framework that accelerates pre-trained GRNs through refinement-aware adaptive query routing. Our analysis reveals that semantic structure forms before local detail refinement is complete, bit uncertainty identifies tokens that remain sensitive to refinement, and sparse query updates benefit from complete key/value coverage. Building on these observations, SparseGRN matches global uncertainty to offline dense reference statistics to determine a prompt-dependent routing onset. During routed refinement, a budget-adaptive controller combines prompt-conditioned response estimates with an offline dynamic-programming table to allocate queries across the remaining steps. High-uncertainty tokens receive fresh attention and feed-forward updates, while inactive positions reuse cached block outputs and keys and values are recomputed over the complete context. This design preserves GRN's sampling schedule and full-map sampling procedure while reducing per-step computation. On an NVIDIA A100 GPU, SparseGRN achieves a speedup, reducing latency from 20.96 s to 14.34 s with a GenEval score of about 0.77 and a DPG-Bench decrease of only 0.08 points.

Primary area:
generative models
Keywords:
Generative Refinement Networks, Model Acceleration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7egJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SparseGRN, a training-free acceleration framework for Generative Refinement Networks. It combines uncertainty-based routing onset with budget-constrained query allocation using an offline response model and dynamic programming. The method reports a 1.46× speedup on GRN-2B with largely preserved generation quality.

Critical strengths
  • The paper is well motivated and clearly presented, with an overall framework and methodological description that are easy to follow.

  • Selective query updates with full KV coverage and cached output reuse form a coherent design, achieving a reported 1.46× speedup on GRN-2B with largely preserved quality and additional validation on GRN-8B.

  • The internal ablations examine routing onset, allocation policies, query budgets, and reduced-step generation, showing that dynamic allocation improves quality over constant allocation at comparable latency.

Critical weaknesses
  1. Limited significance in the redundancy analysis. Temporal redundancy and heterogeneous token sensitivity are already well-established motivations for acceleration in diffusion and visual autoregressive models, including ToCa and FastVAR. The paper largely confirms these observations in GRN without clearly identifying new insights specific to its refinement process.

  2. Insufficient differentiation from existing caching methods. Selective token recomputation, feature reuse, and adaptive scheduling have substantial precedent, including DP-based cache scheduling in CacheQuant and feature forecasting in TaylorSeer. Although these approaches are not identical to SparseGRN, the paper does not clearly establish which GRN-specific challenge requires its proposed design. Consequently, the technical advance beyond adapting established caching techniques remains insufficiently demonstrated.

  3. The main experimental comparison does not address the paper’s central research question. Table 1 devotes substantial space to unrelated diffusion and AR backbones, whose quality differences cannot be attributed to the proposed acceleration strategy and therefore provide no evidence for its methodological contribution. For a GRN acceleration paper, the essential comparison is against relevant acceleration methods on the same GRN backbone under controlled conditions, yet such comparisons are absent. Comparing only against dense GRN and internal variants establishes that the proposed approach can accelerate GRN, but does not establish an advance over existing caching techniques. This is a fundamental gap in the experimental design, leaving the claimed methodological contribution inadequately substantiated despite the apparent breadth of Table 1.

Questions for authors

see weaknesses

Reviewer_s5wPOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

SparseGRN is a training-free method for accelerating Generative Refinement Networks, which spend 50 steps re-predicting a full binary token map. It observes late-step redundancy, sensitivity of high-uncertainty tokens to noise suppression, and the need for full key/value coverage with sparse queries. It routes computation with an uncertainty-matched onset and a dynamic-programming query budget, and reports a 1.46× speedup over a fixed 50-step dense baseline on a 2B GRN, with GenEval 0.762→0.768 and DPG 86.27→86.19.

Critical strengths

The analysis of GRN refinement dynamics is the part worth keeping. Observation 3, that sparse queries need full KV while sparse KV destroys context, is a concrete constraint that separates GRN from next-token and next-scale acceleration. Table 8 supports it with a large gap: SparseGRN at 1.46× reaches GenEval 0.768, while dense GRN at 1.60× (30 steps) drops to 0.689 and at 1.95× (25 steps) to 0.648. The result shows that keeping T=50 matters.

Critical weaknesses
  1. The central claim is not supported by the experiments. The comparison uses a configuration the released model does not run by default. The original GRN paper reports 2–8 s per 1024² image (4 s on average) and a default complexity-aware sampler that varies T from 20 to 50, with an average 1.25× and a maximum 2.5× speedup over the fixed 50-step baseline. This submission measures 1.46× against that fixed 50-step run (20.96 s on A100). The natural baseline is missing, and the gap between 20.96 s and the reported default latency needs an explanation. The internal ablation is worse for the paper's thesis. Table 4 gives a fixed s=26, constant 15% sparse baseline at 1.45×, GenEval 0.762, DPG 86.011, and the full method at 1.46×, 0.768, 86.193. The GenEval difference is 0.006 and the latency difference is 0.6%. Adaptive onset alone lowers GenEval to 0.757; dynamic allocation alone gives 0.767. The text reports the deltas in three different ways: it says adaptive onset "improves GenEval by 0.005" when the table shows a 0.005 decrease, says dynamic allocation improves by 0.015 when the table shows 0.005, and the introduction claims 0.016 over a "constant-13%" reference while Table 4 uses 15% and shows 0.006. No seeds or confidence intervals appear, and no significance test is reported, though Appendix B mentions a prompt-level paired bootstrap. With 553 prompts and four images each, 0.006 is in the range where variance dominates. The results support a simple top-k schedule at step 26. The calibration and DP machinery does not show a measurable benefit, and the DP surrogate is fit from four calibration prompts while optimizing terminal log uncertainty, with no evidence that this proxy tracks GenEval or DPG.

  2. The cached-output approximation breaks the uncertainty-guided claim.

Inactive positions reuse cached block outputs, and the paper states that "inactive scores may remain unchanged." If a token's features and score are frozen while it is inactive, top-k cannot re-select it because its uncertainty rose; it can re-enter only when the query budget grows. The paper says the original full-map sampling rules are preserved, but the predictions feeding those rules are stale for every inactive position. The paper reports no re-entry rate, no age distribution of cached features, and no ablation that forces re-scoring. The method is closer to a fixed subset of tokens with a dynamic count than to uncertainty-guided routing.

Questions for authors
  1. Table 8 makes a good case that keeping T=50 matters. What I still need is a comparison to GRN's own complexity-aware sampler on the same hardware and implementation, plus the generation resolution, batch size, precision, compilation, warmup, and a per-component latency breakdown. The original paper reports 2–8 s (4 s average) at 1024², while your 50-step dense baseline is 20.96 s on A100. How much of the 1.46× survives against the default sampler?

  2. The component ablation is the key evidence. Please give the unrounded Table 4 numbers with paired bootstrap confidence intervals, and clarify the 0.016/0.005/0.015 deltas and the 13%/15% discrepancy. If the full method is within noise of fixed s=26 with a 15% cap, what do the calibration and DP contribute? Also report all six GenEval submetrics; Table 1 omits Counting and Colors.

  3. The caching design is the part I understand least. For an inactive token, which hidden states are cached, and how are its uncertainty score and output-head prediction computed at the next step? Report the fraction of inactive tokens that re-enter the active set, the age distribution of cached features, and an ablation with forced re-scoring or fresh features for a random subset.

  4. The response model is doing a lot of work. How many calibration prompts and records produce the table? Appendix A.2 mentions four calibration prompts for the late-stage lookup. Please report the fit or held-out error of Δ, the correlation between predicted terminal log U_T and GenEval/DPG, the one-time calibration cost, and the COCO split used for calibration versus COCO2014/2017 evaluation.

  5. The operating point needs a selection protocol. Figure 6 and Table 7 show GenEval between 0.759 and 0.769 while speedup moves from 1.17× to 1.50×, with no monotonic trend, and 15% sits at a local maximum. Were ρ=0.15 and reference step 24 selected on GenEval? Report the sweep with confidence intervals and choose the point on a held-out split. Also report the distribution of onset s across prompts; the gate's range is only 21–29.

Reviewer_u1F7OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SparseGRN is a training-free inference acceleration for a pre-trained GRN (Han et al., 2026). GRN generates an image by 50 refinement steps over the 0/1 latent produced by the HBQ tokenizer: at each step the Transformer reads the current latent, outputs a probability for each of the 256 bits at every position, samples a complete predicted latent, and mixes it bit-wise with the fixed initial random latent at a retention ratio l_t that rises from 0 to 1. Every step is a full forward pass over all L positions; the 2B model takes 20.96 s per image on an A100.

SparseGRN keeps the sampling rules unchanged, all 50 steps, the noise-mixing schedule and the full-map sampling, and splits the trajectory into two phases. Steps 1 to s-1 are computed in full, as in GRN. From step s to step 50, each step ranks positions by the bit uncertainty of the previous prediction, defined as the mean of 4p(1-p) over the 256 bits of a position, passes only the top-k positions through attention and the feed-forward network as queries, and reuses the cached output of each block for the remaining positions; keys and values are recomputed from all positions in every block, and the output head still predicts bits for all positions. The onset s is chosen by matching the online uncertainty against a dense reference curve from 300 COCO prompts; routing starts once the match reaches step 25 or later on two consecutive checks, and no later than step 29. The per-step query ratio q_t is chosen from {0.05, ..., 1.00} by a response estimate from kernel regression over offline sparse rollouts plus a backward dynamic-programming table of future costs, under a total budget of 15% of the positions per remaining step.

Evaluation covers GenEval, DPG-Bench, FID/recall/coverage on COCO2014/2017, and HPSv2.1/PickScore/ImageReward. The 2B model goes from 20.96 s to 14.34 s on an A100 (1.46×), GenEval from 0.762 to 0.768, DPG from 86.27 to 86.19; the 8B model reaches 1.43× on an H200. Table 4 ablates fixed versus adaptive onset and constant versus dynamic allocation in a 2×2 design, Table 5 varies the reference step, Table 6 compares allocation policies, Table 7 and Figure 6 vary the budget, and Table 8 compares against dense GRN run with fewer steps.

Critical strengths

The problem is real. GRN's 50 full-map refinement steps make it an order of magnitude slower than models of the same size: in Table 1 the 2B GRN takes 20.96 s, SD3 Medium 4.4 s, and Infinity 2.2 s. Reducing the per-step cost without retraining and without changing the sampling rules is what this paradigm needs most.

The transfer of token-wise feature caching from diffusion to GRN is done cleanly. Observation 3 and Figure 4 settle by direct experiment which side of attention to sparsify: sparse queries keep the image content, sparse keys and values lose it progressively, and the method follows that result with sparse queries and complete key/value coverage.

Critical weaknesses

Technical soundness

The sparse-update phase after the onset s may introduce unnecessary complexity. Section 3.3 and Appendix A consist mostly of two adaptive components: the matching rule that chooses the onset s, and the offline response estimate with dynamic programming that chooses the per-step query ratio. Table 4 shows that the fixed scheme without either, s=26 with a constant 15% of queries, already reaches 1.45×, GenEval 0.752 and DPG 86.01; with both added the numbers are 1.46×, 0.768 and 86.19. The gaps at equal speed are 0.016 and 0.18, the same order as the gaps between the fixed scheme and dense GRN, 0.010 and 0.26. GenEval is a binary score over 553 prompts with four images each, so 0.016 corresponds to roughly thirty images; Appendix B notes that a prompt-level paired bootstrap could quantify the uncertainty of these differences, and none is reported. At this resolution the two adaptive components cannot be distinguished from the fixed scheme. More fundamentally, the sparse phase as a whole has no comparison against the simplest alternative. By Observation 1 the semantic structure is formed by about step 20 and the DINOv2 similarity approaches 1 thereafter, while the onset s falls between step 21 and step 29. Stopping at s and decoding the current prediction is what GRN's own complexity-aware step selection already does; Section 1 states that GRN supports it, and the paper does not use it afterwards. With s around step 30 this gives about 1.67×, above SparseGRN's 1.46×, with no added mechanism. The reduced-step dense baselines in Table 8 compress the l_t schedule into fewer steps rather than stopping the 50-step schedule early, so they do not serve as this control. Whether the twenty or so sparse steps after s add more quality than stopping at s loses is therefore not measured, and until it is, the necessity of the sparse phase and of its two adaptive components is not established.

Questions for authors

1. The early-stop control at the onset step

Under the 50-step schedule, stop the refinement at the onset s actually chosen by the paper's rule, around step 30, decode the full prediction of that step directly, and report the GenEval, DPG, COCO FID/recall/coverage and preference metrics of Tables 1 to 3 together with latency under the same protocol; report GRN's own complexity-aware step selection on the same benchmarks as well. This control shares the dense prefix with every sparse variant and differs only in whether refinement continues after s. It requires no training and no additional mechanism.

2. The selection score u against a random control, and its weighting over bits

The active set is the Top-K positions by u_{t,i}, the mean of 4p(1-p) over the 256 bits. Two points about this score. First, GRN's own selection map S_t is resampled at random at every step, and the ablation in the GRN paper reports that choosing the coordinates to update by confidence underperforms random choice. SparseGRN has no random control: Table 6 varies how many queries to spend, with Top-K by u fixed throughout, so whether u picks better positions than a uniformly random draw of the same size is not measured. Please report the fixed scheme and the full method with the active set drawn uniformly at random at the same q_t. Second, HBQ encodes each channel with M = 4 coarse-to-fine bits, and u weights all 256 bits equally. Flipping the most significant bit of a channel moves the decoded latent by a multiple of what flipping its least significant bit does, so the uncertainty that matters for the image is not the one u measures. Please report whether weighting the bits by their position in the HBQ hierarchy changes the selected set and the results.

3. What Observation 2 shows

Observation 2 suppresses noise at the high-uncertainty positions from step 25 onward and finds that fidelity degrades, while low-uncertainty positions are robust (Fig. 3). A position being sensitive to having its noise suppressed is not the same as that position needing a new forward computation: the first intervenes on the sampling state, the second on the query computation, and the method applies the second. u_{t,i} is read from the prediction at position i, but that prediction comes from attention over all positions, so a high u at i can come from unsettled neighbors rather than from i itself. Please state what in Fig. 3 supports the step from "sensitive to noise suppression" to "needs recomputation", and how the dependence of u_{t,i} on the neighbors of i is handled in the selection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.