- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces BudgetPix, a compute-adaptive tokenization framework for pixel-space diffusion transformers. Motivated by the uneven spatial complexity of images and the inflexibility of uniform tokenization, it allocates different-sized patches across regions to support variable inference budgets. The framework consists of:
- Content-adaptive layout: Uses an entropy-guided quadtree to determine patch sizes under a token budget, deriving layouts from clean training images or intermediate clean-image predictions during inference.
- Multi-scale encoder and decoder: A Patch Aggregator consolidates local information into coarse tokens, while a Patch Refiner supplements upsampled predictions with local detail.
- Layout-augmented fine-tuning: Trains with uniform, random, and content-adaptive layouts so that a single checkpoint supports varying token counts.
Experiments on JiT, MiniT2I, PixelDiT, and pixel-space MeanFlow demonstrate improved quality–efficiency tradeoffs over the evaluated token-reduction baselines. At 25% of the original token budget, BudgetPix achieves GenEval scores close to the full-budget MiniT2I-L and PixelDiT models.
Critical strengths
BudgetPix allocates tokens according to local image complexity and uses local information aggregation and detail refinement to mitigate compression losses. This allows a single model to operate across different compute budgets. The qualitative examples are particularly encouraging: even at 10% of the original token budget, the generated images retain coherent structure and good visual quality.
Critical weaknesses
BudgetPix fine-tunes the entire backbone, whereas several token-reduction baselines are evaluated without comparable adaptation. This makes it difficult to determine how much of the reported improvement comes from the proposed design rather than fine-tuning itself. This is my main concern because the paper’s contribution rests on the effectiveness of its adaptive tokenization framework.
Questions for authors
- How much of the gain comes from adaptive tokenization rather than additional fine-tuning?
- If a small object that should appear in the image has not emerged during warm-up, its region may be assigned a coarse patch. Can later denoising still generate the object, or does the coarse representation make it harder for the object to emerge?
5 comments
We are happy to address any concerns that need clarification.
Dear Reviewer jTvM, we would like to kindly remind you that you can find our explanations in the response above. In addition, the revised paper includes the following changes, highlighted in orange:
- Section 4.3 (Figure R1): Analysis of objects that emerge after the warm-up, with examples.
- Appendix D.11 (Tables R6–R7): Controls separating adaptive tokenization from fine-tuning (dense vs. layout fine-tuning; ToMe and FeatSim on fine-tuned weights).
- Appendix D.12 (Table R8): How often late-emerging regions occur, and their PSNR with a fixed layout vs. re-layout.
We would be happy to clarify any remaining concerns.
Dear Reviewer,
We hope that our clarifications and additional experiments above and in the revised draft have adequately addressed your concerns. As the discussion period is approaching, we would greatly appreciate it if you could review our responses and reconsider your assessment if you find the new results and explanations satisfactory.
We would be happy to address any remaining concerns!
Best regards, All authors
Response
REBUTTAL_FIG7, referred to below, is in the supplementary zip.
We thank the reviewer for the thoughtful questions. In short, nearly all of the low-budget gain comes from the adaptive tokenization rather than from extra training: fine-tuning teaches the backbone to process coarse tokens, and fine-tuning the base model (with uniform tokenization) with the same number of steps does not help (W1, Q1). Small objects that appear only after warm-up still form in their coarse token, and re-layout refines them (Q2).
W1 and Q1 · Adaptive tokenization versus fine-tuning
The gain comes from the adaptive tokenization, not from the extra training, as three controls show:
We will add these controls to the paper.
Table 1. JiT-B/16, dense versus layout fine-tuning of equal length, all sampled with our mixed-resolution layouts, FID-10k (lower is better)
Table 2. ToMe and FeatSim on released and fine-tuned weights, against BudgetPix (lower is better)
Q2 · Small objects that have not emerged after warm-up
Yes, later denoising still generates it. Over 500 paired JiT-B/32 generations, regions that look flat after warm-up, receive a coarse cell and later develop structure cover only a small share of the pixels. The coarse token still forms the object, because it carries its cell's full pixel content through the aggregator and refiner; we saw no missing objects. Periodic re-layout detects the emerging detail and splits the cell, recovering most of the local fidelity of regions that were fine throughout (Table 3, REBUTTAL_FIG7). Re-layout also changes subtle details that FID does not capture, such as the ringing around the skydiver in REBUTTAL_FIG7; we will study these qualitative effects further. We will add this analysis to the paper.
Table 3. JiT-B/32, late-emerging regions: how often they occur, and their PSNR in dB against the full-budget image (higher is better)