acceptodds
Under review as a conference paper at ICLR 2027

BudgetPix: Compute-Adaptive Tokenization for Pixel-Space Image Diffusion

Abstract

Most image generation models rely on uniform tokenization, allocating the exact same computational budget to equally-sized image patches. This static paradigm cannot adapt to different resource constraints at inference time, and yields suboptimal quality-cost tradeoff by devoting the same effort to both plain backgrounds and intricate details. We propose BudgetPix, an adaptive tokenization framework that dynamically allocates compute based on visual complexity and spatial layout, enabling flexible computational budgeting at inference time. BudgetPix comprises three key components: (1) an adaptive encoder that maps a fixed-size image to a variable-length token sequence using an entropy-guided quadtree alongside a multi-scale patch embedder; (2) a scale-aware decoder reconstructs fixed-resolution images from multi-scale token sets; and (3) a flexible training and sampling schedule that enables pixel-space denoisers to operate across variable token counts. BudgetPix seamlessly integrates with existing pixel-space diffusion architectures, enabling a single checkpoint to be operated at a wide range of compute budgets. Evaluated on text-to-image generation, BudgetPix matches the fidelity of MiniT2I-L at 512² (GenEval: 0.874 vs. 0.882) and PixelDiT at 1024² (0.725 vs. 0.721) using just 25% of the original compute budget. In class-conditional using a MeanFlow backbone, BudgetPix requires merely 60% of the full compute budget to produce images with near-zero quality degradation, observing a marginal 0.8-point increase in FID (3.46 vs. 2.65). Comprehensive assessments by human and VLM judges confirm that BudgetPix establishes a significantly improved quality-efficiency tradeoff over prior budget-adaptive baselines.

Primary area:
generative models
Keywords:
image generation, pixel-level diffusion, efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jTvMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces BudgetPix, a compute-adaptive tokenization framework for pixel-space diffusion transformers. Motivated by the uneven spatial complexity of images and the inflexibility of uniform tokenization, it allocates different-sized patches across regions to support variable inference budgets. The framework consists of:

  • Content-adaptive layout: Uses an entropy-guided quadtree to determine patch sizes under a token budget, deriving layouts from clean training images or intermediate clean-image predictions during inference.
  • Multi-scale encoder and decoder: A Patch Aggregator consolidates local information into coarse tokens, while a Patch Refiner supplements upsampled predictions with local detail.
  • Layout-augmented fine-tuning: Trains with uniform, random, and content-adaptive layouts so that a single checkpoint supports varying token counts.

Experiments on JiT, MiniT2I, PixelDiT, and pixel-space MeanFlow demonstrate improved quality–efficiency tradeoffs over the evaluated token-reduction baselines. At 25% of the original token budget, BudgetPix achieves GenEval scores close to the full-budget MiniT2I-L and PixelDiT models.

Critical strengths

BudgetPix allocates tokens according to local image complexity and uses local information aggregation and detail refinement to mitigate compression losses. This allows a single model to operate across different compute budgets. The qualitative examples are particularly encouraging: even at 10% of the original token budget, the generated images retain coherent structure and good visual quality.

Critical weaknesses

BudgetPix fine-tunes the entire backbone, whereas several token-reduction baselines are evaluated without comparable adaptation. This makes it difficult to determine how much of the reported improvement comes from the proposed design rather than fine-tuning itself. This is my main concern because the paper’s contribution rests on the effectiveness of its adaptive tokenization framework.

Questions for authors
  1. How much of the gain comes from adaptive tokenization rather than additional fine-tuning?
  2. If a small object that should appear in the image has not emerged during warm-up, its region may be assigned a coarse patch. Can later denoising still generate the object, or does the coarse representation make it harder for the object to emerge?
5 comments

REBUTTAL_FIG7, referred to below, is in the supplementary zip.

We thank the reviewer for the thoughtful questions. In short, nearly all of the low-budget gain comes from the adaptive tokenization rather than from extra training: fine-tuning teaches the backbone to process coarse tokens, and fine-tuning the base model (with uniform tokenization) with the same number of steps does not help (W1, Q1). Small objects that appear only after warm-up still form in their coarse token, and re-layout refines them (Q2).

W1 and Q1 · Adaptive tokenization versus fine-tuning

W1: BudgetPix fine-tunes the entire backbone, whereas several token-reduction baselines are evaluated without comparable adaptation. This makes it difficult to determine how much of the reported improvement comes from the proposed design rather than fine-tuning itself. This is my main concern because the paper’s contribution rests on the effectiveness of its adaptive tokenization framework.

Q1: How much of the gain comes from adaptive tokenization rather than additional fine-tuning?

The gain comes from the adaptive tokenization, not from the extra training, as three controls show:

  • Fine-tuning alone leaves quality essentially unchanged at 100% budget, where BudgetPix uses the full dense grid.
  • Dense fine-tuning of the same length does not help reduced budgets. Layout fine-tuning of the same length recovers nearly all the quality (Table 1).
  • Baselines on our fine-tuned weights still collapse. ToMe and FeatSim stay far behind BudgetPix even on its fine-tuned weights (Table 2).

We will add these controls to the paper.

Table 1. JiT-B/16, dense versus layout fine-tuning of equal length, all sampled with our mixed-resolution layouts, FID-10k (lower is better)

Fine-tuning100%90%80%70%60%50%40%30%25%20%10%
Released weights, no fine-tuning6.026.678.3013.2626.3467.38116.44152.40147.56180.24225.97
Dense fine-tuning, 30 epochs5.936.728.4914.6034.8286.15139.25160.46154.67185.12219.53
Layout fine-tuning, 30 epochs (full BudgetPix)6.186.617.047.878.9610.5312.5114.6316.3423.5752.46

Table 2. ToMe and FeatSim on released and fine-tuned weights, against BudgetPix (lower is better)

Method and weights50%25%10%
Pixel MeanFlow (one step), FID-50k
ToMe, released weights70.62149.41184.48
ToMe, BudgetPix fine-tuned weights59.27143.84183.42
FeatSim, released weights134.35192.94189.40
FeatSim, BudgetPix fine-tuned weights134.23190.25196.25
BudgetPix, random layout9.4920.5537.01
JiT-B/16, FID-10k
ToMe, released weights38.83107.75177.51
ToMe, dense fine-tuning40.98107.76174.63
FeatSim, released weights118.56169.17162.16
FeatSim, dense fine-tuning123.16170.36162.12
BudgetPix, 100 epochs9.6415.7447.92

Q2 · Small objects that have not emerged after warm-up

If a small object that should appear in the image has not emerged during warm-up, its region may be assigned a coarse patch. Can later denoising still generate the object, or does the coarse representation make it harder for the object to emerge?

Yes, later denoising still generates it. Over 500 paired JiT-B/32 generations, regions that look flat after warm-up, receive a coarse cell and later develop structure cover only a small share of the pixels. The coarse token still forms the object, because it carries its cell's full pixel content through the aggregator and refiner; we saw no missing objects. Periodic re-layout detects the emerging detail and splits the cell, recovering most of the local fidelity of regions that were fine throughout (Table 3, REBUTTAL_FIG7). Re-layout also changes subtle details that FID does not capture, such as the ringing around the skydiver in REBUTTAL_FIG7; we will study these qualitative effects further. We will add this analysis to the paper.

Table 3. JiT-B/32, late-emerging regions: how often they occur, and their PSNR in dB against the full-budget image (higher is better)

BudgetShare of pixelsShare of imagesPSNR, layout fixedPSNR, re-layout at every stepPSNR, regions fine throughout (reference)
With five warm-up steps
25%1.6%56.8%19.5321.1621.93
Without warm-up
25%3.9%76.2%17.2017.4616.18
10%0.12%7.0%13.0813.85n/a
AuthorsOpenReview ↗

We are happy to address any concerns that need clarification.

Dear Reviewer jTvM, we would like to kindly remind you that you can find our explanations in the response above. In addition, the revised paper includes the following changes, highlighted in orange:

  • Section 4.3 (Figure R1): Analysis of objects that emerge after the warm-up, with examples.
  • Appendix D.11 (Tables R6–R7): Controls separating adaptive tokenization from fine-tuning (dense vs. layout fine-tuning; ToMe and FeatSim on fine-tuned weights).
  • Appendix D.12 (Table R8): How often late-emerging regions occur, and their PSNR with a fixed layout vs. re-layout.

We would be happy to clarify any remaining concerns.

Dear Reviewer,

We hope that our clarifications and additional experiments above and in the revised draft have adequately addressed your concerns. As the discussion period is approaching, we would greatly appreciate it if you could review our responses and reconsider your assessment if you find the new results and explanations satisfactory.

We would be happy to address any remaining concerns!

Best regards, All authors

Reviewer_kxgyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BudgetPix, a compute-adaptive tokenization framework for pixel-space image diffusion models, aiming to address the computational inefficiency and inflexible inference budgets associated with uniform tokenization. The proposed framework employs an entropy-guided quadtree to construct non-uniform image layouts, allocating fewer tokens to homogeneous regions and more tokens to visually complex regions. It introduces a multi-scale encoder with a Patch Aggregator to preserve local information during token compression, and a scale-aware decoder with a Patch Refiner to reconstruct fixed-resolution images from variable-length token sequences. Additionally, the authors develop a flexible training and sampling strategy that combines layout augmentation, uniform-grid warm-up, and dynamic layout updates to support different computational budgets using a single checkpoint. Experiments are conducted on multiple pixel-space diffusion architectures, including JiT, MiniT2I, PixelDiT, and MeanFlow, covering both class-conditional and text-to-image generation. The reported results show that BudgetPix maintains competitive generation quality under reduced token budgets while improving computational efficiency compared with the evaluated baselines.

Critical strengths
  1. The paper addresses a well-motivated computational bottleneck in pixel-space diffusion models. By exploiting spatial variations in image complexity, BudgetPix reduces the redundancy of uniform tokenization and enables more efficient token allocation.
  2. The proposed method demonstrates promising quality-efficiency trade-offs across both class-conditional and text-to-image generation tasks. At only 25% token budget, BudgetPix achieves GenEval scores of 0.874 and 0.725 on MiniT2I-L and PixelDiT, respectively, closely matching their full-budget performance.
  3. The experimental evaluation is relatively comprehensive, covering generation quality, image faithfulness, human and VLM judgments, wall-clock speedups, and component ablations. These evaluations provide empirical support for the effectiveness of the proposed method.
Critical weaknesses
  1. Although Layout Augmentation is designed to improve robustness across varying token layouts, the paper lacks dedicated ablations validating the contributions of Uniform, Random, and Content-Adaptive layouts. Moreover, the choices of a=0.15 and b=0.1 are not supported by hyperparameter sensitivity studies, and the Random Layout probability is inconsistently reported as 10% in Section 3.5 and 8.5% in Appendix C.4. These issues leave the effectiveness and parameter choices of Layout Augmentation insufficiently justified.
  2. Although Dynamic Re-layout is presented as an important component of the inference strategy, its necessity is not sufficiently justified. Table 27 in Appendix D.6 shows that updating the layout at every denoising step achieves nearly identical FID scores to updating it only once after warm-up. These results raise questions about whether repeated layout updates provide meaningful benefits over a simpler fixed-layout strategy.
  3. Although BudgetPix improves generation quality at reduced token budgets, its practical efficiency gains remain limited and vary across models. Table 6 shows only 1.36× and 1.95× speedups for JiT-B/32 and PixelDiT at 25% token budget, with PixelDiT even slowing down at 90%. Moreover, the abstract conflates Token Budget with Compute Budget, overlooking the overhead of dense warm-up, additional encoding/decoding modules, and full-resolution pixel processing. Together with the cost of full-parameter fine-tuning, these limitations weaken the claimed practical efficiency benefits.
  4. Although BudgetPix performs well on one-step MeanFlow, its token layouts rely on same-class training images rather than the generated samples' own content, leaving content-adaptive token allocation insufficiently validated. Moreover, BudgetPix undergoes 64K fine-tuning steps, whereas the baselines use unadapted weights, making it difficult to distinguish architectural improvements from the benefits of additional training.
Questions for authors

see above weakness

6 comments
AuthorsOpenReview ↗

Response [1/3]

REBUTTAL_FIG2, REBUTTAL_FIG4b and REBUTTAL_FIG6, referred to below, are in the supplementary zip.

We thank the reviewer for the detailed feedback. In short, each layout family matters and BudgetPix is not sensitive to their probabilities – hyper-parameters a & b (W1); re-layout is an optional refinement for low budgets (W2); all measured speed-ups already include the overhead engineering time (e.g. decoding) and increase with model size, and fine-tuning is only a one-time cost (W3); and BudgetPix also works with a one-step model without training-image layouts (W4).

W1.1 · Ablating the layout families, and the choice of a and b

Although Layout Augmentation is designed to improve robustness across varying token layouts, the paper lacks dedicated ablations validating the contributions of Uniform, Random, and Content-Adaptive layouts. Moreover, the choices of a=0.15 and b=0.1 are not supported by hyperparameter sensitivity studies, …

We thank the reviewer for the suggestion. We ran a controlled ablation on JiT-B/16 (FID-10k), sweeping one probability at a time (Table 1), and will add it to the paper. BudgetPix is robust to these probabilities: values around the defaults give similar FID, so a finer sweep would add compute without changing this conclusion.

Main findings:

  • Uniform layouts keep full-budget fidelity. Without them, every budget gets worse.
  • Random layouts matter at the lowest budgets. Without them, 100% to 50% barely changes, but 10% degrades sharply.
  • Content-adaptive layouts carry the main operating range. Without them, 100% and 50% get worse.
  • Choice of the defaults. We set a = 0.15 and b = 0.10 as moderate values before the sweep: uniform and random layouts act as augmentation, so they stay a minority and about three quarters of the training layouts are content-adaptive, the kind used at reduced budgets. The sweep supports this safe middle setting: dropping either family hurts, moderately larger values barely change FID from 100% to 25%, and only removing content-adaptive layouts trades full-budget quality for the lowest budgets.

Table 1. JiT-B/16, layout-family ablation over the uniform probability a and the random probability b, FID-10k (lower is better)

Setting100%50%25%10%50% with random layouts at sampling
Varying a (b = 0.10)
a = 0 (no uniform layouts)8.0613.3719.5857.6420.20
a = 0.15 (default)6.1810.5316.3452.4616.21
a = 0.306.0010.2316.3252.9516.14
Varying b (a = 0.15)
b = 0 (no random layouts)6.1010.5217.0466.2617.00
b = 0.10 (default)6.1810.5316.3452.4616.21
b = 0.256.1810.4115.9648.7716.09
b = 1 (no content-adaptive layouts)6.8011.2916.2043.2816.54
Varying a (b = 0)
a = 0 (content-adaptive layouts only)8.0913.4820.1172.3720.89
a = 0.156.1010.5217.0466.2617.00

W1.2 · Random-layout probability, 10% vs. 8.5%

… and the Random Layout probability is inconsistently reported as 10% in Section 3.5 and 8.5% in Appendix C.4. These issues leave the effectiveness and parameter choices of Layout Augmentation insufficiently justified.

Both numbers describe the same sampler: a uniform layout is drawn with probability a, otherwise a random one with conditional probability b, else a content-adaptive one. Section 3.5 gave the conditional b = 10%; Appendix C.4 gave the unconditional share (1 − a) · b = 8.5%. We will report unconditional shares everywhere: 15% uniform, 8.5% random, 76.5% content-adaptive.

W2 · Necessity of dynamic re-layout

Although Dynamic Re-layout is presented as an important component of the inference strategy, its necessity is not sufficiently justified. Table 27 in Appendix D.6 shows that updating the layout at every denoising step achieves nearly identical FID scores to updating it only once after warm-up. …

The similar FID is expected: diffusion models settle an image's coarse structure early, so the layout fixed after warm-up already fits most of it, and periodic re-layout is an optional refinement. It reuses the sampler's own clean estimate (no extra network evaluations) and helps where the early layout is least reliable: at low budgets, especially without warm-up (Table 2, JiT-B/32). We will make this explicit in the paper.

Table 2. JiT-B/32 with 50 sampling steps, fixed layout versus re-layout: FID-10k (lower is better) and Inception Score (higher is better)

Layout policyFID 50%FID 25%FID 10%IS 50%IS 25%IS 10%
With 5 warm-up steps
Layout fixed after warm-up10.9619.7662.95175.7126.444.0
Re-layout every five steps10.9919.7961.44175128.246.0
Without warm-up
Layout fixed11.6820.8662.46162.8115.938.0
Re-layout every five steps12.4420.4158.23160.7120.241.9
Re-layout at every step13.0420.3257.70152.4117.341.3
AuthorsOpenReview ↗

Response [2/3]

W3 · Practical efficiency

Although BudgetPix improves generation quality at reduced token budgets, its practical efficiency gains remain limited and vary across models.

W3.1 · Speed-ups in the paper's Table 6 and the PixelDiT slowdown at 90%

Table 6 shows only 1.36× and 1.95× speedups for JiT-B/32 and PixelDiT at 25% token budget, with PixelDiT even slowing down at 90%.

  • PixelDiT at 90%. The slowdown comes from switching dense attention to a masked kernel with padding, not from BudgetPix modules: the unadapted base model run the same way slows down identically. PixelDiT also keeps a dense full-resolution pixel transformer tail, which caps its speed-up.
  • JiT-B/32 is small and launch-bound at short sequences, yet BudgetPix is still faster than ToMe and FeatSim at 25% (Table 6 of the paper), and faster still without warm-up (REBUTTAL_FIG4b).
  • Larger backbones gain more. MiniT2I-L reaches the largest speed-up, well above RTI, FeatSim and ToMe on the same model.

We will make the difference between token budget and compute budget explicit throughout the revised paper and report measured speed-up and wall-clock time for every backbone and budget, with all overheads included; the speed-up against token budget is already given in Appendix B.4 (Tables 6–13). We have also added comparisons with other baselines at the latent level (below), which show that BudgetPix keeps its quality best as the speed-up grows. Even a modest speed-up such as 1.36× is valuable when images are generated in volume, because the saving applies to every image while the fine-tuning is paid only once.

Comparison with other acceleration methods. Table 6 of the paper (App. B.4) already compares BudgetPix with the concurrent RTI adapter, ToMe and FeatSim on both MiniT2I backbones from 100% to 25%, and BudgetPix is faster than all three at every budget. On FLUX.2, BudgetPix runs about as fast as SPEED and RALU and faster than Foveated Diffusion at every reduced budget (Table 3), while staying much closer to full-budget FLUX.2 in quality (REBUTTAL_FIG2).

Table 3. FLUX.2-klein-base-4B at 512², measured speed-up by method (higher is better)

Method100%90%80%70%60%50%40%30%25%20%10%
BudgetPix (ours)1.00×1.01×1.05×1.21×1.29×1.54×1.60×1.78×1.85×2.19×2.46×
Foveated Diffusion, random mask1.00×0.98×1.01×1.06×1.16×1.26×1.39×1.47×1.50×n/an/a
Foveated Diffusion, saliency mask0.83×0.82×0.89×1.07×1.01×1.27×1.12×1.48×1.51×n/an/a
SPEED1.00×1.05×1.15×1.23×1.35×1.47×1.64×1.85×1.94×2.04×2.36×
RALU1.00×1.06×1.13×1.25×1.34×1.49×1.61×1.78×1.90×n/an/a

n/a: below the method's lowest setting. Each speed-up is relative to full-budget FLUX.2 on the same hardware.

W3.2 · Token budget versus compute budget: warm-up, added modules, full-resolution processing

Moreover, the abstract conflates Token Budget with Compute Budget, overlooking the overhead of dense warm-up, additional encoding/decoding modules, and full-resolution pixel processing.

We will say token budget throughout the paper, including the abstract, and quantify each overhead there; all of them are included in every timing we report:

  • Added modules: the Patch Aggregator and Patch Refiner add 0.34–1.82% parameters, and at 100% budget the time matches the dense model on FLUX.2, MiniT2I and PixelDiT.
  • Dense warm-up is optional: without it, JiT-L/32 runs faster at a modest FID cost (REBUTTAL_FIG4b); all FLUX.2 results use no warm-up.
  • Full-resolution pixel processing happens only in the lightweight aggregator and refiner, apart from PixelDiT's own pixel tail (W3.1).

(See Tables 6–13 in Appendix B.4.)

W3.3 · Cost of full-parameter fine-tuning

Together with the cost of full-parameter fine-tuning, these limitations weaken the claimed practical efficiency benefits.

Fine-tuning is a one-time adaptation that yields a single checkpoint for the whole budget range, and it is a small fraction of pretraining (Table 4). On MiniT2I-L/16, BudgetPix uses as many fine-tuning steps as the concurrent RTI adapter, the closest competing approach; BudgetPix is comparable to RTI and outperforms it in the user study and in speed. We will add the fine-tuning cost to the paper.

Table 4. Length of the one-time fine-tuning

ModelFine-tuning lengthShare of pretraining
BudgetPix
JiT-B/16, controlled study30 epochs5%
JiT-B/16, released checkpoint100 epochs17%
JiT-B/32100 epochs17%
JiT-L/3250 epochsabout 8%
pMF-L/1664K stepsunder 1.5%
MiniT2I-B/16100K stepsabout 9%
MiniT2I-L/1640K stepsabout 3.5%
PixelDiT-T2I30K stepsabout 0.2%
Concurrent work
RTI adapter on MiniT2I-L/1640K stepsabout 3.5%
AuthorsOpenReview ↗

Response [3/3]

W4.1 · MeanFlow layouts from same-class training images

Although BudgetPix performs well on one-step MeanFlow, its token layouts rely on same-class training images rather than the generated samples' own content, leaving content-adaptive token allocation insufficiently validated.

The MeanFlow experiment is meant to show that BudgetPix also works with a one-step model; content-adaptive allocation itself is validated on the multi-step models, where the layout comes from the sample being generated. To remove the reliance on training images here as well, we added two experiments: (i) random layouts in a single pass, which already beat ToMe and FeatSim at every reduced budget (Table 5), and (ii) a self-preview: from the same noise, the model first generates a rough draft of the sample with only 64 tokens (the whole image as 64 coarse patches), computes the entropy layout from this draft, and then generates the sample at the target budget with that layout. This takes one extra pass at a quarter of the dense, uniform tokens and matches an oracle that takes the layout from the sample's own full-budget output (Table 5). REBUTTAL_FIG6 shows samples. We will report both in the revised paper.

Table 5. Pixel MeanFlow, one step, FID-50k by layout source, with token merging on the released weights (lower is better)

Method100%90%80%70%60%50%40%30%25%20%10%
Token merging on the released weights
ToMe2.543.065.9314.6636.0470.62109.40122.59149.41156.29184.48
FeatSim2.545.7120.4457.37100.26134.35180.26194.43192.94198.96189.40
BudgetPix, by layout source
Random layout, single pass2.652.803.364.666.549.4913.6217.8520.5524.6437.01
Self-preview, 64-token draft2.652.633.023.694.224.684.814.835.017.0929.21
Reference
Oracle: layout from the sample's own full-budget output2.652.603.063.784.284.654.764.805.017.3429.88

W4.2 · 64K fine-tuning steps versus unadapted baselines

Moreover, BudgetPix undergoes 64K fine-tuning steps, whereas the baselines use unadapted weights, making it difficult to distinguish architectural improvements from the benefits of additional training.

The fine-tuning teaches the backbone to process coarse tokens, which reduced budgets need, rather than improving generation in general. Three controls separate the effect of our design from that of extra training:

  • Competitors also train. The concurrent RTI adapter fine-tunes for as many steps as our MiniT2I-L run; Foveated Diffusion and LoT Diffusion train LoRAs.
  • Extra training alone does not help. At 100% budget BudgetPix is the dense grid, and fine-tuning leaves quality essentially unchanged. On JiT-B/16, dense fine-tuning of the same length does not help reduced budgets (Table 6 below).
  • Baselines on our fine-tuned weights still collapse. ToMe and FeatSim stay far behind BudgetPix even on its fine-tuned weights (Table 7 below).

We will add these controls to the paper.

Table 6. JiT-B/16, dense versus layout fine-tuning of equal length, all sampled with our mixed-resolution layouts, FID-10k (lower is better)

Fine-tuning100%90%80%70%60%50%40%30%25%20%10%
Released weights, no fine-tuning6.026.678.3013.2626.3467.38116.44152.40147.56180.24225.97
Dense fine-tuning, 30 epochs5.936.728.4914.6034.8286.15139.25160.46154.67185.12219.53
Layout fine-tuning, 30 epochs (full BudgetPix)6.186.617.047.878.9610.5312.5114.6316.3423.5752.46

Table 7. ToMe and FeatSim on released and fine-tuned weights, against BudgetPix (lower is better)

Method and weights50%25%10%
Pixel MeanFlow (one step), FID-50k
ToMe, released weights70.62149.41184.48
ToMe, BudgetPix fine-tuned weights59.27143.84183.42
FeatSim, released weights134.35192.94189.40
FeatSim, BudgetPix fine-tuned weights134.23190.25196.25
BudgetPix, random layout9.4920.5537.01
JiT-B/16, FID-10k
ToMe, released weights38.83107.75177.51
ToMe, dense fine-tuning40.98107.76174.63
FeatSim, released weights118.56169.17162.16
FeatSim, dense fine-tuning123.16170.36162.12
BudgetPix, 100 epochs9.6415.7447.92
Reviewer_y6FMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BudgetPix fine-tunes pixel-space diffusion transformers to generate images with mixed-size tokens under a chosen token budget. It uses entropy from an estimated clean image to select the layout, with additional encoding and decoding modules, and evaluates quality, content preservation and speed across several backbones.

Critical strengths
  1. It is nice that the content stays coherent while changing the budget. I appreciate this effect, which is supported by paired metrics and a user study beyond the qualitative examples.
  2. The motivation for the approach is clear and appreciated. The evaluation across several pixel-space backbones is useful and shows flexibility of the proposed approach to the different backbones.
Critical weaknesses
  1. Closely related mixed-resolution generation work is missing from both the introduction and related work. Foveated Diffusion (https://arxiv.org/abs/2603.23491) already fine-tunes pretrained transformers to process mixed-resolution tokens; RALU (https://arxiv.org/abs/2507.08422) and CRPA (https://arxiv.org/abs/2511.19778) also directly address mixed-resolution diffusion, while SPEED (https://arxiv.org/abs/2605.18736) studies progressive resolution changes. Although the stated first claim is restricted to pixel space, the broader positioning does not adequately acknowledge this literature or explain the differences. Further more, the approach is demonstrated only in pixel space, and adapting its pixel-entropy layout selection to latent models is non-trivial, potentially introducing decoding overhead.
  2. Important technical qualifications are not sufficiently clear in the main presentation, making the efficiency claims difficult to assess. Token budget is described as compute budget, although it does not translate directly into speedup. The pipeline starts with full-token, finest-grid denoising, while the one-step MeanFlow experiment instead borrows layouts from training images rather than estimating them from the generated sample. Also, timing freezes the layout after warm-up, whereas quality evaluation updates it periodically, and PixelDiT uses different step counts. These distinctions should be made explicit. I would add speedups to the teaser and bring the token-budget versus speedup connection from Figure 35 into the main discussion. The description of dynamic budgeting should also clarify that the shown budget comparisons are separate generations, not progressive refinement of one image.

I am positive about the paper direction and application, but I would like the authors to address these points in the rebuttal.

Questions for authors
  1. How should the speed–quality curves be interpreted given the different timing and quality configurations? Do the reported quality values hold under the timed settings?
  2. Table 27 shows very similar FID with a layout fixed after warm-up. What motivates periodic updates, and when are they beneficial?
  3. At low budgets I notice seams between coarse patches. What causes these artifacts, and how does the method handle consistency across patch boundaries?
7 comments
AuthorsOpenReview ↗

Response [1/2]

The figures referred to below, REBUTTAL_FIG1 to REBUTTAL_FIG7, are in the supplementary zip.

We thank the reviewer for recognizing the strong motivation, thorough evaluation, and adaptable nature of our flexible budget inference framework.

In short, BudgetPix can also be applied to latent-based models and beats the suggested baselines on FLUX.2 (W1), the reported quality holds under the timed settings (W2, Q1), re-layout helps mainly at low budgets (Q2), and a multi-cascade tokenizer reduces seams (Q3).

W1 · Related mixed-resolution work, and extension to latent models

We extend BudgetPix to FLUX.2-klein-base-4B: the layout is computed from the local variation of the predicted clean latent, with no intermediate decoding, and BudgetPix (aggregator and refiner) runs on the latents, adding only 19.0M parameters. It runs fully without warm-up: the layout is adaptive from the first step and updated by re-layout during sampling. REBUTTAL_FIG1 shows generations from the full budget down to 10% of the tokens, from the same noise and prompt.

We thank the reviewer for pointing us to these methods, and we compare BudgetPix with Foveated Diffusion (random and saliency masks), SPEED and RALU on the same base model, prompts and seeds (REBUTTAL_FIG2). As the speed-up increases, BudgetPix stays close to the FLUX.2 baseline at full budget (grey dashed line), declining only gradually up to about 2.5×, while most other methods lose quality beyond 1.2–1.5×. The gap is widest in GenEval position accuracy, which BudgetPix keeps at or above the full-budget baseline while every baseline loses much of it.

REBUTTAL_FIG3 compares their samples across budgets.

We will expand our related work section to discuss these baselines and tone down the claim about the “first” adaptive method for pixel diffusion.

W2 · Technical qualifications behind the efficiency claims

W2.1 · Token budget versus speedup

Token budget is described as compute budget, although it does not translate directly into speedup.

We will make the difference between token budget and compute budget explicit throughout the revised paper and report measured speed-up and wall-clock time for every backbone and budget, with all overheads included; the measured speed-up against token budget is already in Appendix B.4 (Tables 6–13).

W2.2 · Dense warm-up and MeanFlow layouts

The pipeline starts with full-token, finest-grid denoising, while the one-step MeanFlow experiment … borrows layouts from training images …

Without the warm-up, BudgetPix runs at the reduced budget from the first step, which raises the speed-up at a modest cost in quality; on JiT-L/32 at 25%, FID-50k is even lower without warm-up (REBUTTAL_FIG4b). All FLUX.2 results use no warm-up; we will make this explicit and add them to the paper.

The Pixel MeanFlow experiment shows that BudgetPix also works with a one-step model, complementary to step-reduction methods such as distillation; content-adaptive allocation itself is validated on the multi-step models. Even without training-image layouts, random layouts in a single pass beat ToMe and FeatSim at every reduced budget (Table 1, REBUTTAL_FIG6).

Table 1. Pixel MeanFlow, one step, FID-50k: BudgetPix with random layouts against token merging on the released weights (lower is better)

Method100%90%80%70%60%50%40%30%25%20%10%
Token merging on the released weights
ToMe2.543.065.9314.6636.0470.62109.40122.59149.41156.29184.48
FeatSim2.545.7120.4457.37100.26134.35180.26194.43192.94198.96189.40
BudgetPix
Random layout, single pass2.652.803.364.666.549.4913.6217.8520.5524.6437.01

W2.3 · Timing versus quality settings, PixelDiT step counts

Also, timing freezes the layout after warm-up, whereas quality evaluation updates it periodically, and PixelDiT uses different step counts.

We will make both explicit in the paper.

  • Timing versus quality. FID is unchanged under the exact timed protocol (see Q1, REBUTTAL_FIG4a).
  • PixelDiT step counts. GenEval and DPG-Bench use the 25-step DPM-Solver++ schedule recommended by its authors; held-out FID and timing use 50 steps.

W2.4 · Presentation

I would add speedups to the teaser and bring the token-budget versus speedup connection from Figure 35 into the main discussion. … the shown budget comparisons are separate generations, not progressive refinement of one image.

We will make these updates in the revised manuscript.

AuthorsOpenReview ↗

Response [2/2]

Q1 · Reading the speed–quality curves

How should the speed–quality curves be interpreted given the different timing and quality configurations? Do the reported quality values hold under the timed settings?

We read this as asking whether each reported quality value holds for the exact configuration that was timed. It does: FID under the timed protocol matches the paper's protocol at every timed budget (REBUTTAL_FIG4a), so each point pairs a measured speed-up with its own quality, and skipping the warm-up is faster still at a modest FID cost (REBUTTAL_FIG4b). We will re-plot the curves from these matched runs.

Q2 · When periodic re-layout helps

Table 27 shows very similar FID with a layout fixed after warm-up. What motivates periodic updates, and when are they beneficial?

The similar FID is expected: diffusion models settle an image's coarse structure early, so the layout fixed after warm-up already fits most of it, and periodic re-layout is an optional refinement. It reuses the sampler's own clean estimate (no extra network evaluations) and helps where the early layout is least reliable: at low budgets, especially without warm-up (Table 2, JiT-B/32), and locally, when a small object appears only after warm-up and re-layout splits its coarse cell, recovering detail that FID barely reflects (REBUTTAL_FIG7). We will make this explicit in the paper.

Table 2. JiT-B/32 with 50 sampling steps, fixed layout versus re-layout: FID-10k (lower is better) and Inception Score (higher is better)

Layout policyFID 50%FID 25%FID 10%IS 50%IS 25%IS 10%
With 5 warm-up steps
Layout fixed after warm-up10.9619.7662.95175.7126.444.0
Re-layout every five steps10.9919.7961.44175128.246.0
Without warm-up
Layout fixed11.6820.8662.46162.8115.938.0
Re-layout every five steps12.4420.4158.23160.7120.241.9
Re-layout at every step13.0420.3257.70152.4117.341.3

Q3 · Seams between coarse patches at low budgets

At low budgets I notice seams between coarse patches. What causes these artifacts, and how does the method handle consistency across patch boundaries?

Some error is unavoidable at low budgets, as in any compressed representation; what matters is how visible it is. Seams are its most visible form, aliasing along coarse-cell boundaries: the original tokenizer jumped directly between the finest and coarsest patch sizes and refined each coarse token locally, so neighbouring cells could disagree along their shared edge. Our multi-cascade tokenizer replaces each jump with successive 2× down- and upsampling steps that pass features across scales. On FLUX.2 (same loss, prompts and seeds), it lowers both the boundary discontinuity ratio and FID to the full-budget teacher at low budgets (Table 3, REBUTTAL_FIG5). On JiT-B/32, BudgetPix already removes most seams of the unadapted model, and running the last two of fifty steps on the dense grid brings the boundary ratio below one at every budget while matching or lowering FID (Table 4). We will add this analysis to the paper and discuss less regular spatial structures that make the remaining aliasing harder to see.

Table 3. FLUX.2 without warm-up on thirty-two MS-COCO prompts, seams by tokenizer (lower is better)

Tokenizer100%90%80%70%60%50%40%30%20%10%
Boundary discontinuity ratio (one means no seams)
Single-hop1.0220.9931.0131.0311.0130.9690.9801.0341.1101.112
Multi-cascade1.0060.9890.9971.0050.9520.9700.9801.0001.0221.059
FID to the full-budget teacher
Single-hop162.5159.3161.6168.0169.9157.9173.3170.9174.7210.7
Multi-cascade158.9166.0167.0167.8159.3170.2161.1164.3164.5169.4

Table 4. JiT-B/32 with 50 sampling steps, seams and FID-10k by variant (lower is better)

Variant90%80%70%60%50%40%30%25%20%10%
Boundary discontinuity ratio (one means no seams)
Unadapted base model1.041.051.091.161.261.371.361.231.281.68
BudgetPix1.041.041.031.041.041.020.980.941.021.28
BudgetPix, last two steps dense0.860.850.840.830.830.810.780.760.790.89
FID-10k
BudgetPix6.757.198.029.0310.9913.2916.9819.7927.4661.44
BudgetPix, last two steps dense6.757.167.938.8810.4412.5215.5017.5423.4650.53
AuthorsOpenReview ↗

We are happy to address any concerns that need clarification.

Dear Reviewer y6FM, we would like to kindly remind you that you can find our explanations in the response above. In addition, the revised paper includes the following changes, highlighted in blue:

  • Section 1: Toned-down "first" claim in the contributions.
  • Section 2: Discussion of related mixed-resolution generation work (Foveated Diffusion, RALU, phase-aligned RoPE, SPEED).
  • Section 3.5: Clarification on when periodic re-layout helps (low budgets, no warm-up, objects that emerge late).
  • Section 4.2: Clarification on PixelDiT step counts (25 steps for GenEval/DPG-Bench, 50 for FID and timing).
  • Section 4.3: Clarification on token budget vs. measured speed-up, FID under the exact timed protocol, and a summary of the latent-model (FLUX.2) results.
  • Appendix B.4 (Figure R2): FID under the timed protocol vs. the paper's protocol.
  • Appendix B.7 (Figures R3–R4, Table R2): Full FLUX.2 results: setup, quality vs. speed-up against the suggested baselines, speed-ups and samples.
  • Appendix D.6 (Table R4): Fixed layout vs. re-layout, with and without warm-up.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Common Response to AC and Reviewers

We sincerely thank the AC for overseeing our submission and appreciate all reviewers for their insightful and constructive feedback.

Below, we summarize the primary highlights of our rebuttal, focusing exclusively on important questions and responses. Comprehensive details on these key points, as well as answers to minor clarification questions, are available in our individual responses to each reviewer.

To reviewer 1 (y6FM)

Q1. BudgetPix on latent-based diffusion models and comparison with additional baselines

Per reviewer y6FM’s request, we adapted our BudgetPix to a latent-based model (FLUX.2-klein-base-4B) and compared it with additional baselines, suggested by the reviewer: Foveated Diffusion, SPEED and RALU.

The take-away messages are:

(i) BudgetPix is also applicable to latent-based diffusion models (REBUTTAL_FIG1).

(ii) BudgetPix significantly outperforms the baselines (Foveated Diffusion, SPEED and RALU) both quantitatively (REBUTTAL_FIG2) and qualitatively (REBUTTAL_FIG3).

Q2. MeanFlow borrows layouts from training images

We provide experiments (Table 1 of the R1 response) validating that our BudgetPix on MeanFlow also works with random layouts and beats other baselines such as ToMe and FeatSim.

Q3. Seams between coarse patches at low budgets

This is an expected phenomenon at extreme budgets, e.g., 10% of the tokens. We present a further architectural improvement, the multi-cascade tokenizer, which uses multi-step cascaded patch refinement instead of single-hop refinement. Qualitative improvements are presented in REBUTTAL_FIG5.

To reviewer 2 (kxgy)

Q4. Ablation on the layout families

We provide the ablation (Table 1 of the R2 response) and confirm that:

(i) All included layout families are essential; omitting any of them hurts performance.

(ii) The default hyper-parameters a and b already give good performance, and nearby values give similar FID; we did not run a finer sweep initially because its compute cost is high relative to the small differences it reveals.

Q5. Additional fine-tuning vs. unfine-tuned baselines

We provide additional experiments validating that:

(i) Additional fine-tuning of the original dense model does not help, but fine-tuning with BudgetPix's mixed-resolution layouts does (Table 6 of the R2 response).

(ii) Running the other baselines (ToMe, FeatSim) on fine-tuned weights does not help much (Table 7 of the R2 response).

We note that other adaptive methods such as RTI, Foveated Diffusion and LoT Diffusion conduct additional fine-tuning as well.

Q6. MeanFlow uses the layout of training images and does not allow adaptive layout

In Q2, we showed that BudgetPix (with MeanFlow) works with random layouts and thus does not depend on training-image layouts.

We further show that BudgetPix (with MeanFlow) can be used adaptively at inference, in a self-preview mode (Table 5 of the R2 response) with two passes: a quick coarse pass to estimate the layout, then a second pass at the target budget.

To reviewer 3 (jTvM)

Q7. Fine-tuning benefits

We refer to Q5.

Q8. Small objects that have not emerged after warm-up

We provide further analysis for this excellent question and answer:

(i) how often small objects emerge only after warm-up (Table 3 of the R3 response)

(ii) whether re-layout helps refine these small objects (REBUTTAL_FIG7)

We confirm that these objects still emerge and that re-layout helps refine them.

Area_Chair_oLixOpenReview ↗

Reviewers, please help to join the discussion.

Dear reviewers,

The authors have uploaded the rebuttal. Please help to join the discussion and the further process.

Thanks!

Your AC

AuthorsOpenReview ↗

Revision uploaded

We would like to note that we have uploaded a revised PDF with the changes promised in our responses, highlighted in blue for Reviewer y6FM, green for Reviewer kxgy and orange for Reviewer jTvM. New tables and figures are numbered R1, R2, …, so all table, figure and section numbers cited in the reviews are unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.