acceptodds
Under review as a conference paper at ICLR 2027

QuadTok: Quad-Tree Visual Tokenizer For Autoregressive Image Generation

Abstract

We introduce QuadTok, a novel framework for visual tokenization and autoregressive image generation. Compared to traditional approaches using 2D grids or 1D token sequences, we propose a hierarchical quadtree structure, bridging the gap between 2D spatial binding and 1D sequence-level flexibility. The QuadTok tokenizer dynamically allocates representational capacity to visually intricate ar- eas while leaving homogeneous regions at a coarse resolution. Compared with a fixed 256-token grid, our ImageNet-trained tokenizer saves approximately **10%** of tokens on ImageNet and **9%** when transferred zero-shot to COCO dataset, while maintaining comparable reconstruction fidelity. Furthermore, the natural causality introduced by the tree structure seamlessly enables autoregressive image generation. Conditioned on a quadtree topology supplied before generation, our 947M GPT-style generative model achieves a **2.08** gFID on the ImageNet 256 × 256 benchmark. Additionally, leveraging the strong spatial correlation preserved by the quadtree structure, the QuadTok generator enables zero-shot spatially controlled image generation capabilities.

Primary area:
generative models
Keywords:
Autoregressive Visual Generation, Visual Tokenization, Hierarchical Image Representation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mHoTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes QuadTok, which organizes discrete visual tokens in a quadtree to allocate representational capacity by region while retaining spatial correspondence. The encoder uses LPIPS reconstruction benefits estimated through complementary probing trees to select regions for refinement. A Kinship Causal Mask restricts information flow among tree nodes, and the decoder reconstructs hierarchically. The generator predicts visual codes in breadth-first order under a topology supplied in advance; it does not predict the tree structure itself. Validation covers ImageNet/COCO reconstruction, class-conditional generation at 256/512 resolution, component ablations, and zero-shot regional control with frozen models. The default reconstruction setting uses an average of 230 tokens and achieves ImageNet rFID of 1.46; the 947M generator achieves gFID of 2.08 (Tables 1 and 2).

Critical strengths
  1. The structural design is well motivated, with direct evidence for its component benefits.

    Section 3 integrates regional position, parent–child dependencies, and hierarchical residual decoding. In Table 4, using the same 697M generator and CFG, adding the mask reduces gFID from 3.17 to 2.23 and increases PSNR from 19.16 to 20.37, although rFID changes slightly from 1.41 to 1.46.

    This control demonstrates the value of structural constraints more directly than rankings across different models and is a principal reason to support acceptance.

  2. Regional allocation has measurable practical benefits.

    The guided tree in Table 5 achieves rFID/PSNR of 1.46/20.37 with 230 tokens, close to or better than the full 320-token tree's 1.50/20.39. Table 11 compares LPIPS, MSE, and their combination at approximately 230 tokens. Table 3 additionally fixes the budget at 192 tokens, freezes the models, and uses random topologies as a control, with improved localization metrics in all four directions.

    Together, reconstruction and layout-control results support the usefulness of explicit spatial representations beyond generation FID alone.

Critical weaknesses
  1. Spatial-control validation mainly supports coarse positional shifts, and the magnitude of control remains insufficiently characterized.

    Table 3 and Appendix F evaluate four half-planes—left, right, top, and bottom—and compare prescribed and random topologies at the same 192-token budget. This is a targeted control. However, Box hit rates improve by only 4.3–7.1 percentage points and Mask rates by 4.9–7.9 points, with some absolute hit rates still below 50%.

    Hit rates establish that more subjects enter the target half-plane, but do not distinguish substantial movement from small boundary crossings by samples already near the boundary, or directly establish localization accuracy for arbitrary target regions. Since missed detections count as failures, 50% should also not be interpreted as a universal chance level here.

    The evidence supports using topology to steer subject position, but the magnitude and applicable granularity of this spatial control should be clearer. Appendix H's discussion of multi-object limitations does not replace analysis of the current single-subject setting.

  2. Dependencies among localization metrics and stability across classes require further examination.

    Grad-CAM gains in Table 3 exceed Box/Mask gains, but the random-topology baseline already has pronounced directional biases: 29.6%/70.4% for left/right and 33.9%/66.1% for top/bottom. This suggests that the values may depend on the classifier's response structure, and their agreement with actual changes in subject position needs further explanation.

    SAM 2 also uses Grounding DINO boxes as prompts, so Box and Mask share a localization source and cannot be treated as fully independent validations. Appendix F covers 50 classes with 20 images per class per condition, totaling 1,000 images. Class-level gains, detection-success breakdowns, and uncertainty are not reported, leaving it unclear whether improvements extend broadly across classes or are concentrated in some readily detected classes.

    This gap affects the robustness and generalizability of the spatial-control results.

  3. Generation quality and efficiency have not yet been connected in a directly comparable joint evaluation.

    Appendix E on pages 17–18 provides timing on the same hardware at matched model scales, including absolute latency with decoding, and Figure 4 on page 6 shows the quality cost of changing token budgets. These are valuable results. However, timing uses randomly initialized LlamaGen weights, the 775M QuadTok model is used only for timing, and the XL quality results in Table 2 come from a 697M model. These comparisons do not directly establish acceleration at matched quality.

    Gains at nearly equal token budgets are only about 1.09–1.20×, which the paper also attributes to implementation differences. Larger speedups mainly occur with fewer tokens and should be paired with the corresponding quality points. Table 10 further shows slower adaptive pretokenization. Although this cost can be amortized through caching and generation requires no image-dependent search, the benefit depends on the use case.

    The paper should therefore clarify the quality levels and generation volumes at which variable representational capacity translates into actual time savings, rather than reporting best quality and speed at specified budgets separately.

Questions for authors
  1. Control magnitude and scope of the conclusions:

    Using the existing generated images from Table 3, could you report distributions of Box centers or Mask centroids along the target direction and displacement relative to random topologies, in addition to half-plane hit rates?

    Do the improvements mainly reflect substantial subject movement toward the target region or changes in hits for samples near the boundary?

    Please use these results to explain the control granularity supported by the evidence and distinguish the conclusions of the prescribed-half-plane experiments from finer layout control.

  2. Metric reliability and stability across classes:

    Could you provide class-level breakdowns of the Box/Mask gains in Table 3, uncertainty estimates obtained by resampling classes, and detection-success rates for each condition?

    Do the larger Grad-CAM gains agree with Box/Mask positional changes, or are they concentrated in a small number of classes?

    Given SAM 2's dependence on DINO boxes, please explain how shared detection errors affect interpretation of the two metrics.

  3. Paired validation of quality and efficiency:

    What topology-expansion probabilities, actual mean token counts, and CFG parameters are used by each QuadTok model in Table 2?

    Could you pair the existing quality points in Figure 4 with generation-plus-decoding latency for the corresponding checkpoints and compare against LlamaGen configurations with quality results, identifying which quality levels actually yield speedups?

    Please explain the comparability of the 775M timing model and the 697M quality model, and distinguish the one-time adaptive pretokenization cost from subsequent generation costs.

  4. Comparison with RAR, PAR, FAR, and BAR:

    Beyond the existing results for VAR, xAR, FlowAR, and other methods, could you add experimental comparisons and methodological discussion of RAR (Q. Yu et al., 2025), PAR (Wang et al., 2025), FAR (H. Yu et al., 2025), and BAR (Tang et al., 2026)? Would like to see additional ImageNet 256×256256\times256 comparisons with all four methods in gFID, parameter count, token count, sampling rounds, and measured generation-plus-decoding latency or throughput. Even if time constraints prevent additional experiments, I would still appreciate a detailed comparative discussion of the methods above.

5 comments

Response to Reviewer mHoT

We thank the reviewer for the detailed requests. We analyzed the per-image data behind Table 3 (8,000 images: 8 conditions × 50 classes × 20 samples). CIs are 95% two-stage bootstrap intervals that resample classes and then images within each class.

W1/Q1. Magnitude and form of the shift. In all four directions, the box-center distribution shifts toward the target (KS p < 0.05), not just samples near the boundary; the mean shift is significant in three of them. Mean signed displacement of the box center into the target half (px on 256-px images; random → prescribed):

TargetBoxΔ [CI]KS p
Left+0.8 → +5.0+4.2 [0.9, 7.7]0.035
Right−0.8 → +5.3+6.1 [2.2, 10.0]2.5e-5
Top−3.7 → −0.7+3.0 [−0.1, 6.2]0.006
Bottom+3.7 → +9.0+5.3 [2.1, 8.6]1e-5

In three of four directions, most gained Box hits land at least one coarse cell (32 px) inside the target half. Two factors cap the Box/Mask hit rate. First, the topology does not change the coarse codes, which set the global layout (response to 6i7B, W1), so it moves box centers only through the fine tokens: 3–6 px on average, i.e., 1–2% of the image width. Second, 29% of box centers lie within 1 px of the vertical midline, because these objects span nearly the full width, so their box center cannot move.

With the coarse codes held fixed (paired experiment in our response to 6i7B, W1), box centers still move 5.3 px [4.2, 6.4] toward the target, matching the unpaired estimate above, and 8.8% of pairs move by at least one coarse cell toward it, while the fine level changes about a third of the subject mask (IoU 0.65). The topology thus places detail and, to a smaller extent, structure in the target half.

These results support half-plane control of where detail goes, with a smaller effect on the subject's extent, not placement of the subject in arbitrary target regions. We also acknowledge that this is not a standard layout-to-image evaluation. QuadTok is trained class-conditionally on ImageNet, which has no layout annotations, so we built a proxy: a prescribed topology refines the target half, and off-the-shelf Grad-CAM, Grounding DINO, and SAM 2 measure where the subject lands. The proxy is meant to test whether the topology carries a usable control signal without any layout training, not to benchmark layout-to-image generation. Within this scope, the evidence is consistent: in the re-evaluation with 10,000 images per condition (W2), every metric improves in every direction with intervals that exclude zero. We therefore believe these results show that the topology enables coarse, zero-shot control of where detail goes, and we will revise Sec. 4.3 and the captions of Fig. 1(c) and Fig. 5 to give a precise statement.

W2/Q2. Metric reliability and class stability.

Thank you for these questions. Because 20 images per class give wide per-direction intervals, we regenerated 200 images per class and condition with new seeds (10,000 per condition) and re-evaluated all three metrics; we will update Table 3 with these results.

TargetGrad-CAMBoxMask
Left+28.7 [24.4, 32.8]+5.6 [3.5, 7.6]+5.8 [3.8, 7.7]
Right+18.4 [15.8, 20.9]+6.2 [4.2, 8.1]+6.2 [4.3, 8.1]
Top+26.6 [22.7, 30.4]+5.3 [3.3, 7.4]+5.2 [3.1, 7.2]
Bottom+20.2 [17.2, 23.2]+6.4 [4.5, 8.4]+6.2 [4.3, 8.2]
Pooled+23.5 [20.9, 25.7]+5.9 [5.0, 6.7]+5.8 [4.9, 6.7]

Every interval excludes zero, including Left Box and Top Mask, whose intervals on the original 20 images per class included zero. The pooled gains agree with those of the original images (+21.3, +5.8, and +6.5 pp).

Class stability. Out of 50 classes, gains are positive in 46–48 per direction for Grad-CAM, 39–44 for Box, and 42–45 for Mask. If the Box or Mask gain were confined to half of the classes, about 37.5 classes per direction would show a positive gain; summed over the four directions, we observe 169/200 for Box and 175/200 for Mask, against about 150 under that alternative. The gains are therefore not confined to a few classes. Grounding DINO detects the target in 99.8% of the new images, so no class is poorly detected, and the gains cannot come from a subset of readily detected classes. We will add per-class gains and per-condition detection counts to the appendix.

Agreement between metrics. All three metrics move toward the target in every direction, by different amounts, because they locate different things. Grad-CAM locates class-discriminative evidence, which refinement affects most directly; Box and Mask locate the detected object, whose global position is set by the coarse codes and so moves less (W1). We therefore claim agreement in the direction of the change, not in its size, and will describe Grad-CAM in Table 3 as locating discriminative evidence rather than the subject.

Grad-CAM baseline asymmetry. The random-topology split (29.6/70.4 and 33.9/66.1 in Table 3; 27.5/72.5 and 32.4/67.6 on the new images) reflects where the center of mass of ResNet-50's 7×7 CAM falls relative to the midline. On the original images, counting the center column of the CAM grid as left, instead of splitting it at its center, flips the left/right split to 68/32, yet the gains stay between +17.5 and +31.3 pp in every direction. Each gain compares prescribed and random topologies under the same classifier, so a fixed classifier bias enters both terms; we therefore interpret Grad-CAM through its gains, not its absolute hit rates.

Box and Mask share one pipeline. SAM 2 is prompted with the Grounding DINO box, so a wrong box propagates to the mask. Box and Mask are therefore two summaries of one localization pipeline, not independent confirmations, and we will revise the paper accordingly to avoid overclaiming. Grad-CAM, in contrast, comes from a separate model, an ImageNet-trained ResNet-50 classifier, and shares no component with the detection pipeline. Because two independently built localizers both show the shift toward the target in every direction, their agreement is a valid cross-check of the direction of the effect.

W3/Q3. Paired quality and efficiency. The main results are in Part 2 of the General Response. Every two-level row of Table 2 uses p = 0.75 (256 tokens in expectation; 255.0 measured), and the three-level row uses p1 = 0.95 and p2 = 0.70 (about 988 tokens); all rows use the power-cosine CFG schedule rising from 1 to 20 (mean 10.5). That part pairs each quality point with generation-plus-decoding latency against LlamaGen at the same scale, and explains how the 775M timing model relates to the 697M quality model. QuadTok is both better and faster than LlamaGen at every scale; at matched quality, QuadTok-L at p = 0.5 is 2.02× faster than LlamaGen-XL (1.50× from the smaller model, 1.35× from fewer tokens).

One-time vs. per-sample cost. Adaptive pretokenization costs 3.3 ms more per image than LlamaGen's VQ (6.19 vs. 2.89 ms), about 70 A100-minutes per pass over ImageNet-1K. It is paid once when codes are cached for training. Sampling under a supplied or predicted tree needs no probe. Against LlamaGen-XL at matched quality, the one-time cost is recovered after about 21K generated images.

Q4. RAR, PAR, FAR, and BAR. As discussed in Part 1 of the General Response, these designs are orthogonal to QuadTok. They change how a fixed token set is ordered, parallelized, or transformed, whereas QuadTok changes the token set itself, so each can be applied on top of QuadTok's token sequence. We deliberately keep a plain tokenizer and a plain AR generator so that the gains over LlamaGen can be attributed to spatial adaptivity. We give method-specific details below.

  • RAR (Yu et al., 2025) trains a raster-order generator on randomly permuted orders, annealed back to raster order, over a fixed 16×16 grid of discrete tokens.
  • PAR (Wang et al., 2025) splits the token grid into regions, generates one initial token per region sequentially, and then generates the tokens at the same position in different regions in parallel (4 or 16 per step). This reduces LlamaGen's 576 steps (24×24 tokens) to 147 or 51.
  • FAR (Yu et al., 2025) generates continuous tokens from low to high frequency; each of its 10 steps predicts a full-image frequency level with a diffusion head.
  • BAR (Tang et al., 2026) treats tokens as projections onto basis vectors and learns an orthogonal linear transform of the token sequence end to end, on top of MAR and xAR with continuous tokens.

RAR and PAR keep a uniform spatial grid and change the order or the number of rounds. FAR and BAR are coarse-to-fine in frequency or in a learned basis, where each token describes the whole image. QuadTok is coarse-to-fine in space and gives more tokens to selected regions. Each method maps onto QuadTok's sequence: RAR-style order randomization within each BFS level, PAR-style parallel decoding of children of different parents (which interact only through their parents in the tokenizer), and frequency or basis reordering within the tokens of each level.

For reference, published ImageNet 256×256 results are listed below, followed by LlamaGen and QuadTok timed by us under one protocol. The tokenizers, token counts, latent types, and speed protocols of the published methods differ, so those rows are system-level references, not matched comparisons:

MethodTokensParamsStepsgFIDReported speed
RAR-L / XL256, discrete461M / 955M2561.70 / 1.50– / 8.3 img/s (A100, FP32, batch 128)
PAR-L-4× / XL-4×576, discrete343M / 775M1473.76 / 2.613.38 / 4.94 s per image (A100, batch 1)
PAR-XXL-16×576, discrete1.4B513.022.28 s per image
FAR-L / Hcontinuous427M / 812M103.45 / 3.21not reported
BAR-H (on xAR-H)continuous1.1B–1.15not reported
LlamaGen-L / XL256, discrete343M / 775M2563.80 / 3.398.51 / 12.72 s per batch of 32 (A100, BF16, CFG)
QuadTok-L, p = 0.75 (0.5)256 (192), discrete343M256 (192)2.93 (3.25)8.50 (6.30) s per batch of 32 (A100, BF16, CFG)
QuadTok-XL, p = 0.75 (0.5)256 (192), discrete697M256 (192)2.23 (3.01)10.36 (7.65) s per batch of 32

Scope. As stated in the Scope paragraph of the General Response, all four methods are orthogonal to QuadTok: they change how a token sequence is generated, not which tokens represent the image, so their techniques transfer to QuadTok's sequence as described above and could further improve its results. The controlled comparison is QuadTok versus LlamaGen, which shares our codebook, architecture, and timing protocol (last rows): QuadTok improves gFID at both scales and, at p = 0.5, is still better while 1.35× faster at L. We will add the four methods to the related work with this discussion.

Reviewer_6oYcOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

QuadTok represents an image as tokens in quad-trees. A ViT encoder feeds region-indexed query tokens through an aggregator (in a TiTok style), and the tokens are vector-quantized and decoded coarse to fine. The authors introduce a Kinship causal mask under which coarse tokens attend only to each other and fine tokens attend to all coarse tokens and to their own siblings. They further introduce region-wise complexity guidance, which picks the cells to split by reconstructing under two complementary probing trees and thresholding the per-region LPIPS gain. The image generator predicts codes in BFS order. Experimental results cover ImageNet and zero-shot COCO reconstruction, and class-conditional ImageNet generation.

Critical strengths
  1. The quad-tree tokenization is novel and interesting. Most AR tokenizers use a fixed 2D grid or an unstructured 1D sequence. QuadTok instead gives each token an explicit place in a spatial hierarchy, which yields a variable-length code whose length tracks where detail is spent.

  2. The mask ablation is convincing. Without the mask, PSNR drops substantially and gFID rises from 2.23 to 3.17 with the same 697M generator. This shows that the token structure improves generation quality.

Critical weaknesses
  1. It seems that the adaptivity that motivates the paper does not carry over to generation. If I understand correctly, at sampling time the tree is random with expansion probability p, independent of content, so the generator never chooses where detail goes. In Fig. 4 the best two-level gFID (2.93, the QuadTok-L value in Table 2) occurs at p=0.75 with 256 tokens, which is the same budget as the LlamaGen grid. The generation results therefore show coarse-to-fine ordering with randomly placed refinements, but not content adaptive allocation.

  2. The efficiency evidence is thin, and some close prior works are missing. The headline saving is 230 versus 256 tokens (about 10%). It costs two extra probing reconstructions plus LPIPS per image, and pretokenization takes 6.19 versus 2.89 ms per image. At similar budgets QuadTok does not beat fixed-budget tokenizers such as GigaTok [1]. Moreover, recent coarse to fine and variable length tokenizers are also not discussed. VAR [2] generates coarse to fine, Semanticist [3] emit 1-256 tokens ordered coarse to fine. SpectralAR [4] and ResTok [5] build hierarchical 1D sequences for AR generation. More recent tokenizers like AliTok [6] and EOSTok [7] achieves low gFID with autoregressive generators. The paper should position QuadTok against these and compare gFID where settings match.

[1] Xiong et al. GigaTok: Scaling visual tokenizers to 3 billion parameters for autoregressive image generation. arXiv:2504.08736, 2025.

[2] Tian et al. Visual autoregressive modeling: Scalable image generation via next-scale prediction. NeurIPS, 2024.

[3] Wen et al. "Principal components" enable a new language of images. arXiv:2503.08685, 2025.

[4] Huang et al. SpectralAR: Spectral autoregressive visual generation. arXiv:2506.10962, 2025.

[5] Zhang et al. ResTok: Learning hierarchical residuals in 1D visual tokenizers for autoregressive image generation. arXiv:2601.03955, 2026.

[6] Wu et al. AliTok: Towards sequence modeling alignment between tokenizer and autoregressive model. arXiv:2506.05289, 2025.

[7] Chu et al. End-to-end autoregressive image generation with 1D semantic tokenizer. arXiv:2605.00503, 2026.

Questions for authors
  1. Which expansion probability, and so what token count, does each row of Table 2 use, including the 3-level results?

  2. Are the generators trained on guided tree codes or on random trees?

  3. Why does increasing the number of tokens (or the expansion probability) eventually hurts the generation quality in Fig.4?

3 comments

Dear Reviewer 6oYc,

Thank you for your thoughtful review. We have tried to address your concerns in our responses above. As the discussion period ends soon, please let us know if you have any remaining questions or concerns. We will respond as soon as possible.

Best regards, Authors

Thank you for the response and for running the additional generation experiments. The new experimental results suggest that adaptive tokenization can modestly improve generation quality while reducing the amortized token budget. The largest improvement is in Inception Score. Given the metric’s sensitivity to fine-grained details, this is consistent with the authors’ claim that QuadTree enables zero-shot control of where detail goes. I understand that the main experiments in the paper are designed as a controlled comparison. If feasible, I would still encourage the authors to evaluate their method with more recent tokenizers to assess how well the gains carry over to state-of-the-art systems. The authors’ response has addressed most of my concerns, so I will raise my score to 3 (weak accept).

Thank you very much for your thoughtful follow-up, constructive feedback throughout the discussion, and for raising your score! We completely agree that scaling the evaluation to more recent, state-of-the-art tokenizers is an important next step. We are actively working on securing additional compute resources to pursue this follow-up investigation and hope to share broader results in future work.

Reviewer_6i7BOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

QuadTok uses a quadtree to allocate visual tokens to regions at different spatial resolutions. A kinship attention mask preserves hierarchical structure within the tokenizer, while complementary reconstruction probes select regions that benefit from refinement. The generator predicts codes in breadth-first order under a topology supplied before generation. On ImageNet at 256 resolution, adaptive reconstruction uses 230 tokens on average, and a separate generation experiment reaches gFID 2.08 with a 947M model. The paper also evaluates COCO reconstruction transfer, deeper trees, 512-resolution synthesis, and zero-shot spatial steering.

Critical strengths
  1. The kinship-mask ablation supports the structural design: removing it worsens gFID from 2.23 to 3.17 with the same 697M generator and CFG. Token perturbations also provide evidence of regional correspondence.
  2. Guided allocation reaches rFID 1.46 with 230 tokens versus 1.50 with 320 fully expanded tokens. Transfer, higher-resolution experiments, and fixed-budget layout evaluation extend the evidence beyond this reconstruction result.
Critical weaknesses
  1. Where layout conditioning takes effect: Section 3.4 provides only current and preceding BFS instructions. For the same class, equal-budget layouts, and matched guidance schedules, the 64 coarse instructions are identical, so coarse predictions cannot directly condition on the locations of future refinements. How does layout information entering through the fine-node instructions affect subject placement during fine-token generation and final decoding in the reported half-plane experiments?
  2. Interpretation of the reconstruction-benefit estimate: Section 3.3 uses one probing tree and its complement to estimate regional LPIPS improvements. Kinship masking limits interactions between child groups, while decoder and LPIPS dependencies may still connect neighboring regions. How should a region's estimated benefit be interpreted when other regions have different refinement states in the two probes? Please explain the relevant decoder and metric dependencies.
  3. Correspondence between quality and timing results: Figure 4 provides a budget-quality sweep, and Table 9 includes decoder latency, but their checkpoint and guidance correspondence is not explicit. The 775M timing model also differs from the 697M XL quality model, as acknowledged in Appendix E. Could the authors map the existing results to their model sizes, token budgets, checkpoints, and guidance settings, identifying directly comparable points where available?
Questions for authors

Please refer to weaknesses.

3 comments

Dear Reviewer 6i7B,

Thank you for your thoughtful review. We have tried to address your concerns in our responses above. As the discussion period ends soon, please let us know if you have any remaining questions or concerns. We will respond as soon as possible.

Best regards, Authors

Dear Reviewer 6i7B,

As the discussion period closes in a few hours, we wanted to thank you once again for your insightful review and constructive questions on information flow and evaluation alignment.

In case it is helpful for a quick final check, here is a brief pointer to our updates in the responses and revised paper:

  • How fine-level layout conditioning acts (W1): Added a paired 1,000-sequence experiment holding the 64 coarse codes fixed, showing that fine tokens reshape both object masks (IoU 0.65, 5.25 px center shift) and class-discriminative detail (Appendix F).
  • Interpreting the benefit estimate brb_r (W2): Separated tokenizer locality (95.8% within-cell pixel change) from LPIPS receptive-field effects, validated brb_r against 5 probe splits and exact single-region counterfactuals (Spearman 0.90–0.91), and clarified its context-averaged interpretation (Sec. 3.3, Appendix G.1).
  • Paired quality–timing correspondence (W3): Unified model sizes (697M quality vs. 775M size-matched timing), token budgets, checkpoints, and CFG settings in a single paired comparison table (Appendix E, Table 8).

Thank you again for your time and helpful feedback, and please let us know if we can clarify anything else in the remaining hours!

Best regards,

The Authors

Thanks for providing the additional results and analysis. I don't have any further requests or questions for the authors. I recommend Weak acceptance.

Comments on the paper

3 comments

General Response (1/2): Scope of QuadTok and relation to other methods

We thank all reviewers for their careful reading. Several questions concern how QuadTok relates to recent tokenizers and generators (6oYc W2, mHoT Q4). We first state the scope of our contribution and then position QuadTok.

Scope. QuadTok's contribution is spatially adaptive tokenization: how many tokens an image receives, and where they go, follow its content. Our aim is to test this idea in a clean setting and to study the behaviors and applications it enables, not to push ImageNet gFID with additional techniques. We therefore kept every other component plain. The tokenizer (about 120M parameters) uses the same 16,384-entry codebook as LlamaGen and standard VQGAN losses, and is trained only for reconstruction. The generator is a LlamaGen-style decoder-only Transformer with a standard causal mask, next-token prediction, and classifier-free guidance. Recent tokenizers improve gFID through tokenizer scaling, semantic regularization, alignment with DINOv2 features or with autoregressive generation, and stronger guidance such as AutoGuidance. While these techniques can improve performance, they are orthogonal to spatial allocation: they modify how the tokenizer or generator is trained or sampled, rather than where tokens are allocated, and can be incorporated into QuadTok without altering its quadtree structure. We excluded them to validate that the improvements over LlamaGen, which shares our codebook and generator architecture, can be attributed to spatial allocation.

Generation-side methods are complementary. RAR changes the generation order, PAR decodes distant tokens in parallel, FAR generates by frequency level, and BAR learns a linear basis for the token sequence. Each changes how a token sequence is generated, not which tokens represent the image, so each can be applied to QuadTok's token sequence. For example, order randomization can act within each BFS level, and children of different parents, which interact only through their parents in the tokenizer, are natural candidates for parallel decoding.

Coarse-to-fine and variable-length tokenizers. The tokenizers raised by 6oYc order tokens from coarse to fine or allow variable length, but none of them allocates tokens according to image content:

MethodCoarse-to-fine axisTokens per imageContent-adaptive allocationAllocation over space
VARscale (each scale covers the full image)fixed (680)nouniform
Semanticistimportance (PCA-like ordering)any prefix up to 256, same for all imagesnono spatial tokens
SpectralARfrequency (DCT bands)fixed (64), global truncationnono spatial tokens
ResTokresidual scale levelsfixed (128), global truncationnono spatial tokens
AliToknone (raster order)fixed (272)nouniform
EOSToknonefixed (256)nono spatial tokens
QuadTokspacevaries per image (10th–90th pct. 176–272)yesper region

Why the quadtree couples the two properties. QuadTok does not attach adaptivity to a coarse-to-fine tokenizer as a separate module. It starts from the observation that image complexity varies across space, so the token budget should vary across space as well. The quadtree serves both purposes at once: coarse tokens always describe the whole image, and finer tokens are added only where a region needs them. This single structure gives two properties that the methods above do not provide together: (i) tokens are spent where the detail is (230 tokens reach rFID 1.46, compared with 1.50 for all 320); and (ii) every token is interpretable as a known region at a known level.

General Response (2/2): Paired quality and latency

All three reviewers raise the efficiency evidence, in particular how latency relates to generation quality, so we answer this shared point here. All other questions are answered in the individual responses.

Paired quality and latency. Latency depends on the architecture, token count, batch size, and whether CFG is used, but not on weight values or on the CFG scale and schedule. Each timing row therefore pairs directly with the trained checkpoint of the same architecture. The table below lists the directly comparable points (A100-80GB, BF16, KV cache, batch 32 with CFG, seconds per batch including image decoding; QuadTok tokens are the expected count 64 + 256p). Every Table 2 two-level row samples random trees with p = 0.75 (64 + 256p = 256 tokens in expectation) under a power-cosine CFG schedule rising from 1 to 20 (mean scale 10.5). We re-measured the timings under this protocol. At p = 0.75, QuadTok uses LlamaGen's token count and has the same latency, as expected for the same architecture; this corrects the near-equal-token speedups (1.09–1.20×) in Tables 8 and 9 and Appendix E, which we will update. The speedups below therefore come from using fewer tokens; the 697M rows also benefit from the smaller model.

ParamsModelTreeTokensgFIDLatency (s)Speedup vs. LlamaGen, same scale
111MLlamaGen-Bgrid2565.465.551.00×
111MQuadTok-Brandom p=0.752564.415.541.00×
111MQuadTok-Brandom p=0.501925.324.151.33×
343MLlamaGen-Lgrid2563.808.511.00×
343MQuadTok-Lrandom p=0.251285.074.291.98×
343MQuadTok-Lrandom p=0.501923.256.301.35×
343MQuadTok-Lrandom p=0.752562.938.501.00×
775MLlamaGen-XLgrid2563.3912.721.00×
697MQuadTok-XLrandom p=0.752562.2310.361.23×
697MQuadTok-XLrandom p=0.501923.017.651.66×
775MQuadTok-XL, LlamaGen-XL architecture (timing only)random p=0.75256–12.721.00×
775MQuadTok-XL, LlamaGen-XL architecture (timing only)random p=0.50192–9.041.41×

What "timing only" means. The two rows marked "timing only" report latency without gFID. Our XL quality model was trained at 697M parameters, which does not exactly match LlamaGen-XL's 775M. To compare speed at identical size, we time a QuadTok generator with exactly LlamaGen-XL's architecture (width 1280, 36 layers) using randomly initialized weights, so these rows have no gFID. Random weights suffice because latency does not depend on weight values. Together, the two XL comparisons give the speedup at matched size (775M: 1.00× / 1.41× at p = 0.75 / 0.5) and at the reported quality (697M: 1.23× / 1.66×; its 1.23× at equal tokens comes from the smaller model).

At p = 0.75, QuadTok uses the same number of tokens as LlamaGen, has the same latency, and reaches better gFID at every scale. At a fixed model size, latency scales with the number of tokens (L: 6.30 vs. 8.50 s at 192 vs. 256 tokens). At p = 0.5, QuadTok uses 25% fewer tokens than LlamaGen (192 vs. 256) and is still better at every scale, and also faster: B 5.32 vs. 5.46 (1.33×), L 3.25 vs. 3.80 (1.35×), and XL 3.01 vs. 3.39 (1.66× for the 697M quality model; 1.41× for the size-matched 775M timing model). At matched quality, QuadTok-L at p = 0.5 (gFID 3.25) beats LlamaGen-XL (3.39) at 2.02× lower latency: 1.50× from the smaller model (2.3× fewer parameters) and 1.35× from fewer tokens.

AuthorsOpenReview ↗

Rebuttal Revision Updated

Revision uploaded: summary of changes

We have uploaded a revised paper with all changes marked in blue. The main text stays within the 10-page limit; most new material is in the appendix (Appendices I and J are new). The main changes are:

  • Efficiency. Appendix E and Table 8 pair quality with latency under a single protocol and replace the previous speedup tables. At equal token count, QuadTok has the same latency as LlamaGen; the speedups come from using fewer tokens.
  • Spatial control. Table 3 now reports the re-evaluation with 10,000 images per condition and 95% intervals. Sec. 4.3 and the captions of Fig. 1(c) and Fig. 5 state the claim as coarse, half-plane control of where detail goes. Appendix F adds the supporting analyses.
  • Reconstruction-benefit estimate. Sec. 3.3 clarifies how b_r should be read, and Appendix G.1 validates it against single-region counterfactuals.
  • Generation. Table 2 lists token counts, and Sec. 4.2 explains why more tokens eventually hurt. Appendix I reports generation with a predicted topology as a preliminary experiment (QuadTok-B only), so the main-text claims are unchanged.
  • Related work. Sec. 2 and Appendix J position QuadTok against the tokenizers and generation methods raised by the reviewers.

We thank the reviewers again and are happy to answer further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.