acceptodds
Under review as a conference paper at ICLR 2027

Pixel Art Is a Palette, Not a Downscale: Rebuilding Targets and Decoding for Sprite Generation

Abstract

Game sprites of 12–24 px are among the smallest images people draw on purpose, and their medium is defined by a handful of flat colours and hard alpha edges. Despite rapid progress in text-to-image generation, existing systems render such sprites as shrunken illustrations with soft shading and around a hundred colours. We identify two root causes: the training targets of sprite corpora are dominated by downscaled artwork rather than native pixel art, and continuous diffusion has no mechanism to express the few-colour structure of the medium. To address these challenges, we propose PaletteDiff, a framework for native-resolution sprite generation. First, we introduce Native-Resolution Target Rebuilding, which removes duplicate leakage, forbids aggressive downscaling and adds 15k permissively licensed sprites, together with a native evaluation protocol that scores against real low-resolution pixel art only. Second, we propose Palette-Projected Sampling, a training-free decoding rule that projects the predicted clean image onto a per-sprite k-colour palette during the late denoising steps, so that the remaining steps repair the quantisation seams. Third, combined with SDEdit-style re-noising, the same rule yields a Palette Refinement module for sprites from generators we did not train. PaletteDiff lowers FD-DINOv2 at 16 px from 167.2 to 31.8 ± 0.5; a per-sprite palette is the decisive ingredient, and imposing it inside the sampler improves on the same quantisation applied afterwards by a further 18%. Under the native protocol, PaletteDiff has the lowest FD among gpt-image-2, FLUX.2 and SDXL pipelines given the same palette post-process; Palette Refinement improves all three external systems by 36–41% while preserving 80% of their silhouettes.

Primary area:
generative models
Keywords:
diffusion models, text-to-image generation, sampling, pixel art, evaluation protocols, dataset curation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_w2CnOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PaletteDiff, a framework for generating native-resolution pixel-art sprites at very small resolutions. The paper identifies two main limitations of existing approaches: low-resolution training buckets are heavily contaminated by downscaled higher-resolution artwork, and standard continuous diffusion models tend to generate many slightly different colors rather than the small set of flat colors characteristic of native pixel art.

The proposed framework has three main components. First, Native-Resolution Target Rebuilding removes exact duplicate leakage, restricts aggressive downscaling, and augments the training set with approximately 15k additional licensed sprites. Second, Palette-Projected Sampling (PPS) estimates a per-sprite k-color palette from the model's current clean-image prediction and projects the prediction onto this palette during the later diffusion steps. The subsequent denoising steps can then repair artifacts introduced by quantization. Third, the same mechanism is combined with SDEdit-style re-noising to refine sprites generated by external models.

The experiments show substantial improvements in native FD, particularly at 16 px, where the full method reaches 31.8 compared with 87.1 without PPS and 167.2 for the original baseline. Ablations investigate the contribution of target rebuilding, guidance, palette size, in-sampler projection versus post-processing, backbone choice, and different resolutions. The authors additionally evaluate refinement of outputs from gpt-image-2, FLUX.2, and SDXL.This paper introduces PaletteDiff, a framework for generating native-resolution pixel-art sprites at very small resolutions. The paper identifies two main limitations of existing approaches: low-resolution training buckets are heavily contaminated by downscaled higher-resolution artwork, and standard continuous diffusion models tend to generate many slightly different colors rather than the small set of flat colors characteristic of native pixel art.

The proposed framework has three main components. First, Native-Resolution Target Rebuilding removes exact duplicate leakage, restricts aggressive downscaling, and augments the training set with approximately 15k additional licensed sprites. Second, Palette-Projected Sampling (PPS) estimates a per-sprite k-color palette from the model's current clean-image prediction and projects the prediction onto this palette during the later diffusion steps. The subsequent denoising steps can then repair artifacts introduced by quantization. Third, the same mechanism is combined with SDEdit-style re-noising to refine sprites generated by external models.

The experiments show substantial improvements in native FD, particularly at 16 px, where the full method reaches 31.8 compared with 87.1 without PPS and 167.2 for the original baseline. Ablations investigate the contribution of target rebuilding, guidance, palette size, in-sampler projection versus post-processing, backbone choice, and different resolutions. The authors additionally evaluate refinement of outputs from gpt-image-2, FLUX.2, and SDXL.

Critical strengths
  1. Strong diagnosis of an overlooked dataset problem. The analysis showing that low-resolution sprite buckets are largely populated by downscaled higher-resolution artwork is convincing and practically important. The authors also quantify duplicate leakage and show that the conventional reference distribution can reward precisely the blurred appearance that native pixel art should avoid.

  2. Palette-Projected Sampling is simple and empirically effective. The distinction between applying palette quantization as a final post-processing step and imposing the same constraint inside the diffusion sampler is well motivated. The ablation showing 41.6 FD for post-hoc quantization versus 34.1 for in-sampler PPS supports the claim that subsequent denoising can repair quantization artifacts.

  3. The empirical analysis is extensive. The paper includes component ablations, palette-size sweeps, schedule analysis, alternative backbones, several resolutions, external generators, alternative feature spaces, training-length checks, and an additional training-excluded evaluation. These experiments make it relatively easy to identify which parts of the method are responsible for the reported gains.

  4. The method is computationally attractive. PPS requires no additional network evaluations or retraining and can also be used to refine outputs from generators that were not trained by the authors. This makes the contribution practically useful even if one does not adopt the complete PaletteDiff training pipeline.

Critical weaknesses
  1. The core algorithmic novelty is somewhat incremental. Projection of the predicted clean image inside a diffusion sampler is already an established technique, palette quantization itself is standard, and the refinement procedure is based on SDEdit. The novel part appears to be the specific combination of on-the-fly per-sprite palette estimation with in-sampler projection and the application to native-resolution pixel art. This is useful, but I view it more as a well-designed specialization and integration of existing ideas than a fundamentally new diffusion methodology.

  2. Hyperparameters are selected using the evaluation metric. Section 4.1 states that the palette size k, projection start f, and guidance weight w were selected on evaluation FD. Although the paper later shows that the selected k transfers between two halves of the captions, this only partially addresses the issue and does not cover all tuned hyperparameters. A separate validation set should ideally be used for all method selection before reporting the final evaluation numbers.

  3. The primary native reference is not fully independent of the training data. The authors explicitly state that the native reference is drawn from the same underlying pool as the training data and is not removed from training. The additional training-excluded experiment is useful and preserves the ordering, but it contains only 124 examples after near-duplicate filtering. I would prefer the main headline results to also be reported against a larger fully training-disjoint native reference.

  4. Perceptual evaluation remains incomplete. Pixel-art quality includes properties such as deliberate pixel placement, readability, palette coherence, and aesthetic quality that may not be fully captured by DINOv2 FD, color counts, or flat-neighbor statistics. The paper itself observes that a VLM judge is unreliable for this task. A small human evaluation by either pixel-art practitioners or ordinary raters would substantially strengthen the claim that lower native FD corresponds to genuinely better sprites.

  5. The comparison to prior academic pixel-art generators could be stronger. The main quantitative comparison focuses on large general-purpose generators used as practitioner baselines. These are useful comparisons, but evaluating at least a smaller subset against the closest specialized generation methods would make it easier to assess progress relative to the research literature rather than primarily relative to zero-shot general-purpose systems.

  6. There are a few minor wording and consistency issues. For example, Section 3.1 and Appendix A describe the starting corpus as containing "37,489 unique sprites" and then state that exact-pixel hashing reveals 4,888 duplicate copies. Presumably "unique" here means extracted sprite instances or file paths rather than pixel-content uniqueness, but this should be reworded. In addition, Section 3.2 defines the generation resolutions as R in {12, 16, 20, 24}, whereas later experiments and Appendix C also report 32 px generation. The intended distinction between the main target range and additional-resolution experiments should be stated more consistently.

Questions for authors
  1. Evaluation-set tuning: Why were k, f, and w selected directly using evaluation FD rather than a separate validation set? Can the authors report the headline result when all hyperparameters are selected on a disjoint validation subset and the evaluation set is used only once?

  2. Training/reference overlap: Can the authors provide the main comparison using a native reference set that is completely excluded from training? The 124-example training-excluded experiment is reassuring, but a larger fully disjoint evaluation would materially increase my confidence in the reported improvement.

  3. Novelty relative to existing projection methods: Beyond estimating the palette per sprite, what technical property of PPS distinguishes it from applying an existing projected-diffusion or projected-x0 framework with k-means quantization as the projection operator? Clarifying this distinction would affect my assessment of the methodological novelty.

  4. Specialized baselines: Is it possible to compare PaletteDiff against the closest specialized pixel-art generation approaches on even a smaller evaluation subset if full-scale evaluation is computationally prohibitive?

  5. Human evaluation: Do human raters, ideally including people familiar with pixel art, prefer PaletteDiff outputs over post-hoc palette quantization and the external baselines? Given that the authors find the VLM judge unreliable, such evidence would materially strengthen the quality claims.

Reviewer_c6ecOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper targets native-resolution pixel-art sprite generation (12–24 px RGBA) and proposes PaletteDiff, which has three parts.

  • Native-Resolution Target Rebuilding. The authors deduplicate the sprite corpus, forbid downscaling a sprite by more than 1.5× into a size bucket, and add 15k permissively licensed sprites. They pair this with a "native" evaluation protocol whose FD reference set contains only sprites originally drawn at the target size.
  • Palette-Projected Sampling (PPS). This is a training-free rule. During the late denoising steps, it projects the predicted clean image onto a per-sprite k-colour palette obtained by k-means, and the remaining steps repair the quantisation seams.
  • Palette Refinement. PPS is combined with SDEdit-style re-noising to refine sprites produced by external generators.

At 16 px, FD-DINOv2 drops from 167.2 to 31.8. PaletteDiff achieves the lowest native FD against gpt-image-2, FLUX.2-klein with a community pixel-art LoRA, and SDXL with Pixel Art XL. In this comparison the baselines are generated at high resolution, box-downscaled, and given the same k-colour post-process. Palette Refinement improves these external systems by 36–41% in FD while keeping silhouette IoU around 0.8.

Critical strengths
  1. A careful diagnosis of corpus contamination. The paper finds that 92.9% of the 16 px training targets are downscaled larger artwork and that 16.2% of the held-out set has a pixel-identical duplicate in training. Showing that a mixed-size reference set can reverse the ranking of methods is a useful caution for anyone evaluating in this setting.
  2. A simple, training-free sampler constraint. PPS adds no extra NFEs and moves the colour statistics of samples toward those of real sprites (Table 5).
Critical weaknesses

W1. The significance of the problem and the validity of its evaluation are not established.

  • A self-defined problem. Native 12–24 px sprite generation is defined largely by the authors themselves, not a problem that the image-generation community broadly recognizes, such as fine-grained or high-fidelity generation. The paper does not show that the setting matters beyond a narrow niche or that its findings transfer to broader generative modeling.
  • FD is questionable at this scale. A 16 px sprite contains very little fine detail. Yet the main metric is FD-DINOv2, computed on 16 px images that are nearest-neighbour upsampled to 224 px. DINOv2 was not trained on such inputs, so it is unclear what its features capture.
  • Too few samples for FD. The external comparison uses only 200 captions (Table 1). FD estimates from 200 samples are known to be unreliable, and most results at other resolutions are single-seed.
  • No human evaluation. No human study is reported, and the authors acknowledge that their VLM judge prefers polished renders over real sprites (Limitations, Appendix G). In the qualitative samples (Figs. 1, 4, 5), the differences between methods are subtle enough that it is hard to reach a consensus on which is better. It therefore remains unverified whether the metric improvements correspond to sprites that practitioners would prefer.

W2. The comparison with general-purpose generators is not fair or informative.

  • Wrong use of the baselines. gpt-image-2, FLUX.2, and SDXL are optimized for high-resolution generation. In this paper they are rendered at about 1024 px, cut out, box-downscaled to 16 px, and then quantised by the authors' own k-colour post-process (Sec. 4).
  • A lossy conversion favors the proposed method. This conversion discards information and imposes a pipeline designed around the proposed method. Box downscaling was the best of the conversions the authors tried (Appendix E), but the comparison still mainly measures how well a 1024 px illustration survives aggressive lossy reduction, not the capability of these models. Without the post-process, their FD is 284.4 or higher, which shows how much the outcome depends on the conversion.
  • Unreviewed LoRA baselines. The pixel-art adapters used as baselines are community LoRAs cited as Hugging Face model pages (nerijs, 2023; Limbicnation, 2026), not peer-reviewed methods. The stronger baselines in the paper are therefore not established reference points.
  • Limited novelty. Methodologically, projecting x^0\hat{x}_0 onto a constraint set during sampling is a standard device from inverse problems, as the paper acknowledges in Sec. 2. PPS applies it with a k-means palette, and Palette Refinement applies SDEdit. The technical contribution is modest, so the paper's value depends on the significance of the task, which is unconvincing as discussed in W1.
Questions for authors
  1. Can you provide a human preference study, for example with pixel-art practitioners, that compares PaletteDiff with the baselines and with real sprites? The current evidence relies on FD from upsampled 16 px images and a VLM judge that you note is biased.
  2. How stable is the external comparison in Table 1 if the 200 captions are expanded to a few thousand and FD is reported over several seeds?
  3. What would the comparison look like if the baselines were used as practitioners actually use them? Examples include prompting the large models to output pixel art at low resolution directly, or using dedicated pixel-art conversion tools, instead of rendering at 1024 px and applying your post-process.
  4. Beyond 12–24 px game sprites, which broader problem do the findings inform? For instance, does PPS help other palette-constrained media such as icons or limited-palette illustrations?
Reviewer_BmpgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets text-to-sprite generation at 12–24 px RGBA. It has three parts:

  • Corpus rebuild. Exact-duplicate removal, a ≤1.5× downscaling limit per size bucket, and 15k added CC-licensed sprites.
  • Native evaluation protocol. FD on DINOv2 features, measured against sprites drawn at the target size.
  • Palette-Projected Sampling (PPS). In the late denoising steps, the predicted clean image is snapped to a per-sprite k-colour palette (k-means), and the remaining steps repair the seams. Combined with SDEdit, PPS also refines sprites from external generators. A 72.5M pixel-space UNet trained from scratch reaches FD 31.8 at 16 px (from 167.2). It beats gpt-image-2, FLUX.2-klein + LoRA and SDXL + LoRA after the same k-means post-process.
Critical strengths
  • Clear data diagnosis. 93% of the 16 px bucket is downscaled artwork, and 16% of the test set has a pixel-identical training twin. The two reference sets reverse sampler rankings. This is useful beyond this paper.
  • The obvious quantisation controls are present. Post-hoc k-means on the authors' own samples gives 41.6, versus 34.1 in-sampler. All external baselines are quantised too. A shared palette (120.1) and a median filter (112.2) are both far worse than PPS.
  • PPS is simple and training-free, adds no network evaluations, and transfers to a pixel-space transformer.
  • Honest reporting and thorough appendices, including results that cut against the method: the VLM judge, quality conditioning, and canvas occupancy.
Critical weaknesses

1. Practical advantage over the tools practitioners actually use is not established

The main evidence is FD to a native pool. The paper's own controls show that FD rewards closeness to the native pool's style, which is not the same as being a better tool (see W4).

The reported VLM judgments do not establish an advantage over external systems. The judge ties PaletteDiff with real sprites (50%) and prefers FLUX.2 and gpt-image-2 over real sprites (72% and 70%). PaletteDiff vs. the external systems head-to-head is not reported. Consider reporting it.

Dedicated pixel-art generators are missing.

  • Retro Diffusion and PixelLab produce grid-aligned sprites at native size (Retro Diffusion from 16×16) with controlled palettes.
  • Retro Diffusion also offers repair of "fake" pixel art (native-grid recovery), which competes directly with Palette Refinement.
  • These are the tools a game developer would compare against. Their documented features establish relevance, not superiority; a reasonable subset of captions would suffice. Request: define one or two concrete use cases and run head-to-head comparisons against the strongest frontier, open and specialist pipelines. Examples:
  • (a) Items for an existing asset pack with a fixed palette (an optional extension of PPS: a supplied palette instead of per-sprite k-means), vs. baselines remapped to that palette.
  • (b) Inventory or item icons displayed at 1× and 2× on a game background. Score each with a rubric-based pairwise VLM judge:
  • Criteria judged separately: prompt match, legibility at display scale, anti-aliasing, orphan pixels, stray dither, outline quality.
  • Randomised order, ties allowed, integer-upscaled display.
  • Calibrated against a blinded study with pixel artists (e.g., a few hundred pairs, with confidence intervals). Per-criterion win rates would show where the method actually helps.

2. Prompt fidelity is missing from the main comparison

  • Table 1 reports only FD.
  • PPS at k=4 lowers R@1 from 22.8% to 19.2% (Table 2). The externals' CLIP and R@1 scores are not given, and they plausibly follow captions better.
  • Palette Refinement. Refined gpt-image-2 at σ=0.6 scores 86.5 FD, worse than PaletteDiff from scratch (53.1), and refinement reduces its retrieval from 22.0% to 18.5%. Comparable prompt-fidelity results should be reported for all systems.
  • Request: report CLIP and R@1 for every row of Tables 1 and 4 on the same 200 captions, and state what refinement offers beyond preserving the input's silhouette (IoU).

3. Baselines are not tuned the way PaletteDiff is, and data vs. method gains are entangled

Unequal tuning.

  • PaletteDiff's k, f and w were selected on the evaluation FD.
  • Baselines get a fixed k (4/8/6) and a single prompt that does not ask for the target size, a limited palette or no anti-aliasing. Conversion chosen on the wrong objective. The conversion method was chosen on the biased reference, without the palette step (App. E).
  • PixelOE is better at 20 and 24 px (282.5 vs 317.6; 358.0 vs 428.5), yet box downscaling is used throughout.
  • Grid-aware conversion is not tried: where a generated image has a recoverable block grid, detect it and take the modal colour per cell. No matched-data baseline. The internal ablations separate several components, but do not establish whether the advantage over external generators persists after adapting one to the same data. One strong baseline would suffice. Candidates:
  • FLUX.2-klein-4B (Apache 2.0, already used in the paper): a relatively small, accessible candidate; the non-distilled base checkpoint suits adaptation.
  • Ideogram 4 (open weights, research licence): palette and layout controls suit this medium.
  • Qwen-Image-Edit: fine-tuned on degraded→native sprite pairs, it is the natural learned competitor to Palette Refinement. These are latent models, so a fair setup is integer upscaling for training, and grid-aware conversion plus the same quantisation and alpha thresholding at output. Applying PPS to such a model (decode, quantise, re-encode) would also test its transfer claim.

PPS's reach is unclear. PPS needs pixel-space x̂₀. Extending it to latent models (e.g., decode, quantise, re-encode at each step, with VAE error and added cost) is not discussed. The transfer claim covers only a 69.7M pixel-space transformer.

4. The evaluation reference is mostly training data, and the metric is not validated for quality

Reference overlaps training. The native reference stays in training except where it overlaps the 5,928 excluded paths (App. C); the exact overlap should be reported. k, f and w were also selected on this FD.

The training-excluded check is small and the gain shrinks there. It uses 124 prompts. The no-projection-to-PPS improvement falls from 87.1→31.8 (headline) to 76.2→61.1, though the magnitudes are not directly comparable because prompt population and sample count also change. The only external evaluated there is unquantised FLUX (Fig. 6).

Prompt and reference distributions differ. Prompts come from mixed-size held-out sprites, whereas the reference contains only sprites drawn at ≤16 px; the size and semantic distributions of the prompt subset should be reported. FD can reward drifting toward the reference's content instead of following the prompt. The object-only protocol only partly addresses this. (This is not a trivial matched-quantisation effect: neither training targets nor the reference are fixed-k quantised.)

Signs that FD tracks "native-pool likeness" rather than quality:

  • k=4 has the best FD while overshooting real sprite statistics (4 colours and 0.54 flat pairs, vs. 6 and 0.46 for real sprites). k=6, which matches the real statistics, scores worse (36.9).
  • Outputs with the top craft tag, described as more finished, score worse (45.6 vs 34.2).
  • Under Inception features, the PPS gain shrinks (23.79 vs 26.27), and k=6 beats k=4.
  • The provenance probe uses the same DINOv2 features. Request:
  • A larger native test split that is disjoint by sheet or pack and excluded from training.
  • Hyperparameters selected on a separate validation split.
  • Prompts and reference drawn from that same held-out native split. W1's evaluation would also validate the metric.

5. Statistics for the external comparison

  • Table 1 uses n=200 against 384-dimensional features.
  • The claim that the small-sample bias applies "identically for every row" is unsupported. FD bias depends on the generated distribution [1].
  • The large 16 px gap is suggestive, but robustness of any ranking or refinement gain cannot be established without uncertainty and sample-count analyses.
  • SDXL and FLUX.2-klein are open models, so a substantially larger caption set is feasible.
  • Request: report FD∞ [1] or KID with confidence intervals, using bootstrap resampling by source sheet.

Minor

  • Novelty of PPS is modest. Projecting x̂₀ inside the sampler is standard, as the paper itself notes. The contribution is the per-sprite k-means palette.
  • The 1.5× cutoff is not isolated. Table 2 bundles deduplication with the bucket policy.
  • Fig. 4 shows unquantised FLUX while Table 1 uses the quantised version. This is disclosed, but please show the quantised outputs.

What would change my assessment

  1. Head-to-head, per-criterion evaluation on a concrete use case against frontier and specialist tools (W1), with prompt-fidelity metrics in all tables (W2).
  2. One matched-data pretrained baseline, plus baselines whose conversion and k are tuned on the native objective (W3).
  3. Main results on a training-disjoint native test split, with uncertainty estimates (W4, W5).

References

[1] M. J. Chong and D. Forsyth. Effectively Unbiased FID and Inception Score and Where to Find Them. CVPR 2020. arXiv:1911.07023. [2] Retro Diffusion, native-resolution pixel-art generation and pixel-art repair. https://www.retrodiffusion.ai [3] PixelLab, pixel-art sprite generation. https://www.pixellab.ai

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.