acceptodds
Under review as a conference paper at ICLR 2027

Can LLMs Understand and Generate Images Pixel by Pixel?

Abstract

Large language models (LLMs) can generate images through text formats such as SVG and CAD, but these representations use high-level primitives rather than explicit raster values. We ask whether LLMs can instead generate pixel images directly with their existing vocabulary, without a learned image tokenizer or decoder. To test this, we define a palette-indexed representation: a model outputs an RGB palette and a grid assigning one palette color to each pixel. Using it, PixelBench evaluates text-to-pixel (T2P) and image-to-pixel (I2P) generation across resolutions, alongside a complementary pixel-understanding task. Models show substantial pixel-generation ability in both tasks. Cross-resolution analysis reveals model-specific scaling: most models improve and then decline with resolution, whereas the strongest produces high-quality, recognizable images and remains robust. Pixel-understanding performance follows a similar model ranking and correlates strongly with generation performance. This connection motivates a second question: can training models to reconstruct pixels improve general visual understanding? To investigate, we introduce Pixel Reconstruction Tuning (PRT), a self-supervised objective that trains VLMs to reconstruct images as pixel grids. PRT improves average performance across 13 visual-centric benchmarks}, indicating that explicit pixel reconstruction can provide useful auxiliary supervision for visual understanding.

Primary area:
datasets and benchmarks
Keywords:
LLM, VLM, Image Generation, Alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3Tz6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PixelBench, which tests whether language models can generate and understand images through a text-based palette and pixel grid, without a learned image tokenizer or decoder. It evaluates text-to-pixel generation, image-to-pixel reconstruction, and pixel understanding at several resolutions. Performance varies by model and often declines at higher resolutions. The authors also propose Pixel Reconstruction Tuning, which improves average results on 13 visual benchmarks for Qwen3-VL models at three sizes.

Critical strengths
  1. PixelBench provides a simple way to study visual generation and understanding through the model's existing text interface.
  2. The paper combines evaluation with a practical training study. Pixel Reconstruction Tuning improves average performance across 13 benchmarks and is tested at multiple model sizes.
Critical weaknesses
  1. The main training setup does not fully isolate gains from additional image exposure and training steps. The matched-data control helps, but comparisons with other auxiliary objectives are missing.
  2. Some conclusions about the link between pixel understanding and generation rely on correlations across only six or seven models. The results are suggestive but do not establish a general relationship.
Questions for authors
  1. Under the matched-data and matched-step setting, how consistent are the gains across all 13 benchmarks and model sizes?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer 3Tz6 (Part 1/3): 13-Benchmark Fixed-Budget Breakdown Across 2B/4B/8B

We thank Reviewer 3Tz6 for the positive assessment and helpful questions. Below we provide the full 13-benchmark breakdown across all three model scales under the fixed budget, our new same-image captioning and token-matched controls, and an expanded 12-model correlation study.

(1) Per-Benchmark Consistency Under Matched-Data and Matched-Step Settings Across 2B, 4B, and 8B (re: Question 1)

In addition to expanding Table 10 into a full 13-benchmark breakdown for all 2B and 4B settings, we completed representative 8B Fixed-Budget (64,000 samples, 1,000 steps) runs (Pixel-16 and Pixel-Mix). Across all three scales (2B, 4B, and 8B), fixed-budget single-resolution and mixed configurations improve the majority of benchmarks (11–12/13 on 2B, 10–12/13 on 4B single-resolution and 7/13 on 4B Mix, and 10/13 on 8B), with gains consistently concentrated on visual perception and spatial reasoning benchmarks (MMVP, BlindTest, CV-Bench, HallusionBench, BabyVision, MMBench, MMStar, PerceptionBench), closely mirroring Table 6:

Benchmark2B Δ162B Δ322B Δ642B ΔMix4B Δ164B Δ324B Δ644B ΔMix†8B Δ16 (new)8B ΔMix (new)
MMVP+14.0+14.0+15.3+15.3+18.7+5.3+12.7+8.0+2.7−0.7
BlindTest+5.3+3.5+4.2+8.5+6.0+4.2+0.4−4.5+3.8+1.5
HallusionBench+7.8+0.4+5.5+5.8+4.3+5.0+5.4+4.1+0.8+0.3
CV-Bench+4.6+3.3+4.1+3.7+2.0+0.5+1.0+0.7+2.7+1.6
BabyVision+4.4+3.1+3.4+0.3+3.9+2.3+0.8+4.4−1.3+0.5
MMBench+4.2+3.1+4.5+3.9+2.3+0.9+1.9−1.5+1.5+0.4
MMStar+2.5+1.7+1.5+0.6+2.1+0.3+0.5−0.4+1.5+2.3
PerceptionBench‡+2.1+0.3+1.5+1.4+1.7+0.3+0.4+0.3+0.4+0.5
V-STAR Bench+4.7+0.5+0.5+3.1−2.6+3.7+3.7−1.6+2.1−2.6
TallyQA*+2.8+1.0+3.5+3.0−0.5+0.5−0.1−0.4+1.1+1.3
Q-Bench+4.5+4.2+4.7+2.8−2.3−0.9−1.9−0.9−0.3+1.1
BLINK−0.9−1.6+0.9−1.1+3.9+1.0+2.7+3.1−1.2−0.1
VLMBias−0.3−1.2−0.9−0.8+1.2+0.6+2.5+1.5+0.1+1.7
# Benchmarks Improved11 / 1311 / 1312 / 1311 / 1310 / 1312 / 1311 / 137 / 1310 / 1310 / 13
Example Avg (N=22,554 / 22,902‡)+3.15+1.53+2.83+2.97+2.27 (+2.21‡)+1.34 (+1.30‡)+1.17 (+1.14‡)−0.01+1.21‡+1.07‡
Benchmark Avg (13)+4.28+2.48+3.74+3.58+3.13 (+3.10‡)+1.83 (+1.82‡)+2.31 (+2.30‡)+0.99+1.07‡+0.59‡

(‡: 2B/4B per-benchmark rows reflect original Table 10 N=22,554 [2,652 single-image PerceptionBench items; 12/13 benchmarks identical to N=22,902], while parenthesized 4B values and 8B columns are evaluated on the full N=22,902 suite [3,000 PerceptionBench items], matching Section (2) below; *: 3k random TallyQA subset. †: Raw-log audit corrects a +1.14 pp Ex / +1.00 pp Bench transcription error in Table 10's printed 4B Pixel-Mix row [pulled down by single-seed BlindTest −4.5, whereas 2B/8B Fixed Mix and 4B Seq-Token-Matched Mix at +2.80/+3.86 are strongly positive], and we will revise Appendix D.3 accordingly; all other 7 Table 10 rows match to 0.001 pp. All summary averages are computed from unrounded benchmark scores.)

AuthorsOpenReview ↗

Author Response to Reviewer 3Tz6 (Part 2/3): Same-Image Detailed Caption & Token-Matched Controls

(2) Comparison with Alternative Auxiliary Objectives and Strict Token-Matched Controls (re: Weakness 1)

To isolate the pixel-grid objective from additional image exposure and training tokens, we ran two new controlled studies on Qwen3-VL-4B (N = 22,902 official units):

  1. Same-image +3k Detailed Caption control: Using the exact same 3,044 source images ("3k") as PRT, we trained a fixed-budget control (64,000 samples, 1,000 steps) whose target combines fluent frontier VLM spatial descriptions with exact 8-color PIA statistics to specify the main subject, 2D spatial layout, and all 8 dominant #RRGGBB hex colors with area shares (~185 supervised tokens/response, ~5.76M total supervised tokens, omitting only the pixel grid). +Detailed Caption yields only +0.54 Example / +0.88 Benchmark, whereas pure-grid +Pixel-16 achieves +2.21 Example / +3.10 Benchmark (over 4× the Example gain).
  2. Strict token-budget-matched controls: When we remove 14,665 or 12,216 FineVision examples so that supervised assistant loss tokens (Sup-Token-Matched +Pixel-16: 49,335 FV + 3,044 Pixel-16 = 52,379 rows, 819 steps, 5.45M supervised tokens—even fewer than +Detailed Caption's ~5.76M—and −18.2% total tokens) or total vision+text sequence tokens (Seq-Token-Matched +Pixel-Mix: 51,784 FV + 3,044 Pixel-Mix = 54,828 rows, 857 steps, 0 token diff) are 100% identical to 64k FV-only, PRT still improves performance by +2.26 to +2.80 Example Avg and +3.63 to +3.86 Benchmark Avg:
Control Setting (Qwen3-VL-4B, Same Source Images)Total SamplesStepsToken Budget vs. 64k FV-onlyExample Avg (N=22,902)Δ Example*Benchmark Avg (13)Δ Benchmark*
Baseline (FV-only)64,0001,0001.00× (5.45M sup / 32.93M seq)53.90—53.63—
Fixed-Budget +3k Detailed Caption (no grid) (new)64,0001,000Same samples & steps (~5.76M sup)54.44+0.5454.51+0.88
Fixed-Budget +3k Pixel-16 (pure grid)64,0001,000Same samples & steps (+0.23% seq)56.10+2.2156.72+3.10
Fixed-Budget +3k Pixel-32 / 64 (pure grid)64,0001,000Same samples & steps55.20 / 55.04+1.30 / +1.1455.44 / 55.92+1.82 / +2.30
Sup-Token-Matched +Pixel-16 (pure grid) (new)52,3798190 diff sup. tokens (−18.2% total seq)56.16+2.2657.26+3.63
Seq-Token-Matched +Pixel-Mix (pure grid) (new)54,8288570 diff total vision+text seq tokens56.70+2.8057.48+3.86
Additive +3k Pixel-16 / Mix (Paper Table 6/7)67,0441,048+4.91% / +19.24% total seq tokens57.03 / 56.84+3.14 / +2.9457.69 / 57.33+4.07 / +3.70

(*: The exact 3,044-image pool was rounded to "3,000" / "67,000" / "1,047 steps" in Sec. 5.1 & App. D.1–D.3 [ceil(67,044 / 64) = 1,048 exact steps], which we will make exact in revision. Deltas are computed from unrounded scores.)

AuthorsOpenReview ↗

Author Response to Reviewer 3Tz6 (Part 3/3): 12-Model Expansion of Generation–Understanding Correlation

(3) Scope and 12-Model Expansion of the Generation–Understanding Correlation (re: Weakness 2)

We agree with the reviewer that correlations across a finite set of models are descriptive rather than a universal law (as noted in Section 3.3). To test whether this relationship holds across a broader model pool and within individual model families, we evaluated 5 additional multimodal models (GPT-6 Sol, GPT-6 Luna, GPT-5.6 Luna, Gemini 3.8 Flash, and Grok 4.7) on the full 1,200-item PixelBench suite under the same no-tool API protocol as Appendix E (Table 12; full hyperparameters and Valid/Flat rates for new models will be added in revision), expanding the pool from 7 models (6 multimodal) to 12 models spanning 7 model families (11 multimodal across 6 families, plus text-only DeepSeek V4 Pro):

  1. Every correlation metric is statistically significant (p≤.0125p \le .0125) across the expanded pool: While adding heterogeneous model families naturally moderates point-estimate correlation magnitudes relative to the initial 7-model sample (e.g., T2P Spearman ρ\rho: 1.000 → .876), statistical significance strengthens substantially—Pearson pp-values tighten 5× on T2P (r=.793,p=.0021r = .793, p = .0021; Spearman p=.0002p = .0002) and 7× on I2P (r=.846,p=.0010r = .846, p = .0010; Kendall τ\tau tightens from p=.0556p = .0556 to p=.0125p = .0125). Even when retaining only a single flagship representative per independent provider across all 7 families (n=7n=7 T2P / n=6n=6 I2P) to eliminate within-provider clustering, correlations remain highly significant (T2P r=.872,p=.0104,ρ=.964,p=.0005r = .872, p = .0104, \rho = .964, p = .0005; I2P r=.909,p=.0121r = .909, p = .0121).
  2. Strict within-family monotonicity under identical tokenization: Within the GPT-6 family (Luna → Sol → Astra), Understanding (29.2% → 44.0% → 67.5%), T2P Agg. (65.72 → 84.56 → 90.83), and I2P Agg. (10.80 → 50.28 → 83.81) increase strictly monotonically; the GPT-5.6 family (5.6 Luna → 5.6 Sol) and full-task generational upgrades (5.6 Luna → 6 Luna, Gemini 3.6 → 3.8 Flash) likewise improve all tasks simultaneously (while 5.6 Sol → 6 Sol improves T2P and Understanding with lower I2P), and all 12/12 models improve when switching from PIA to the Astra-I2P oracle:
ModelT2P Top-1 Avg / Judge Avg / T2P Agg.*I2P Top-1 Avg / SigLIP / DINO / I2P Agg.†Understanding (PIA / Astra-I2P / Overall)
GPT-6 Astra (paper)91.0 / 90.7 / 90.8348.0 / .654 / .175 / 83.8157.3% / 77.7% / 67.5%
GPT-6 Sol (new)87.7 / 81.5 / 84.5645.7 / .579 / .081 / 50.2835.0% / 53.0% / 44.0%
GPT-5.6 Sol (paper)85.3 / 76.7 / 80.9956.0 / .588 / .094 / 64.2232.3% / 46.0% / 39.2%
Gemini 3.8 Flash (new)82.3 / 77.3 / 79.8244.3 / .565 / .071 / 44.3735.7% / 52.3% / 44.0%
Claude Opus 5 (paper)85.7 / 73.1 / 79.3940.7 / .561 / .060 / 38.2224.0% / 46.7% / 35.3%
Grok 4.7 (new)77.3 / 76.2 / 76.7827.7 / .491 / .038 / 7.6822.7% / 29.7% / 26.2%
Kimi K3 (paper)81.7 / 70.3 / 75.9636.7 / .541 / .040 / 26.8918.3% / 36.3% / 27.3%
Gemini 3.6 Flash (paper)79.0 / 69.8 / 74.3930.7 / .539 / .051 / 22.5916.3% / 27.3% / 21.8%
Seed-2.1 Pro (paper)72.0 / 64.2 / 68.0835.0 / .547 / .043 / 26.9913.7% / 18.3% / 16.0%
GPT-6 Luna (new)69.3 / 62.1 / 65.7225.7 / .519 / .034 / 10.8023.0% / 35.3% / 29.2%
GPT-5.6 Luna (new)54.3 / 54.8 / 54.5521.3 / .509 / .029 / 3.6820.0% / 30.3% / 25.2%
DeepSeek V4 Pro (paper)53.0 / 51.5 / 52.22— (text-only)10.3% / 14.7% / 12.5%

(*: T2P Agg. is the mean of the 6 unrounded per-resolution Top-1 and VLM Judge scores. †: I2P Agg. is re-normalized across all 14 rows [11 multimodal models + 3 algorithmic baselines] rather than the 9 rows in Table 2, shifting absolute values while preserving rank order; raw unnormalized Top-1, SigLIP, and DINO correlations with Understanding all remain statistically significant at p≤.025p \le .025.)

Correlation SummaryOriginal Pool (nn)Original rr / ρ\rho / τ\tau (pp-value)Expanded Pool (nn)Expanded Pearson rr (pp-val)Expanded Spearman ρ\rho (pp-val)Expanded Kendall τ\tau (pp-val)
Understanding – T2P7.871 (.0107) / 1.000 (<.0001) / 1.000 (.0004)12.793 (.0021).876 (.0002).748 (.0008)
Understanding – I2P6.930 (.0073) / .829 (.0416) / .733 (.0556)11.846 (.0010).761 (.0065).587 (.0125)

We will incorporate the full multi-scale 13-benchmark breakdown, same-image captioning and token-matched controls, expanded 12-model analyses, and Table 10 updates into the revised manuscript. We thank you again for your positive and constructive feedback, and welcome any further questions during the discussion period.

Thank you for your clarification. I will keep my positive rating unchanged.

Reviewer_H8SuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether existing language models can understand and generate images pixel by pixel through their text vocabularies. PixelBench represents images as palettes of up to eight RGB colors and grids of palette indices, covering text-to-pixel generation, image-to-pixel generation, and pixel understanding, with additional resolution-scaling experiments. The authors compare seven models and several pixelization algorithms, finding that performance varies with model and resolution and that generation and understanding performance are correlated. They further propose Pixel Reconstruction Tuning (PRT), which constructs reconstruction targets using a deterministic algorithm and mixes them into fine-tuning of Qwen3-VL models at the 2B, 4B, and 8B scales. Average performance across 13 visual benchmarks improves at all three scales. The appendices include controls with fixed training-example counts and optimization steps, along with some seed-sensitivity results.

Critical strengths
  1. The shared interface makes capability comparisons concrete and inspectable.

    Section 3 and Appendix G cover both generation and understanding, using deterministic palette-based rendering to avoid mixing the capabilities of a specialized image decoder into the evaluation. Tables 1–3 combine retrieval, image similarity, valid-output rates, and recognition accuracy. Table 2 and Appendix L also provide algorithmic baselines under the same color constraints. This makes the conclusions more convincing than a small collection of generated examples, while the resolution sweeps in Section 4 reveal capability boundaries rather than reporting only the best setting.

  2. Positive PRT results appear across model scales, and the authors address several confounding factors.

    In Table 6, the example-weighted averages improve by 2.0, 2.9, and 2.1 percentage points at the three scales. Appendix D.3 fixes the training set at 64,000 examples and training at 1,000 steps for 2B and 4B, and still obtains positive average gains when replacing some FineVision data. Table 11 also reports three-seed results for one setting.

    These results support pixel reconstruction as an auxiliary form of supervision worth further study. The gains cannot simply all be attributed to additional training steps.

Critical weaknesses
  1. The gains cannot yet be attributed specifically to the pixel-reconstruction objective.

    The main experiment in Section 5.1 retains 64,000 FineVision examples and adds 3,000 pixel examples and 47 training steps. Appendix D.3 controls the number of examples and updates, but still changes image sources, supervision format, and target length, and does not include a corresponding 8B control.

    The positive results with fixed example counts and steps weaken an explanation based solely on more training, but do not separate the pixel objective from additional image exposure or supervision through longer responses. In particular, an alternative auxiliary objective, such as captioning, on the same source images under a comparable budget is missing. Appendix D.4 explicitly acknowledges this limitation, which is an appropriate qualification. Nevertheless, the evidence remains insufficient to establish a distinctive training benefit from explicit spatial reconstruction supervision.

    This affects the mechanistic interpretation of the training contribution, without negating the effectiveness of the current data mixture.

  2. Training stability in the main settings and evaluation uncertainty for key capability differences are insufficiently characterized.

    The main experiments in Appendix D.1 use seed 42. Although Table 11 reports three seeds, it uses the 4B model with 4,000 pixel examples and explicitly follows the original MMVP scoring protocol, which differs from the main tables.

    These results support reasonably stable positive gains in that auxiliary setting, but do not directly characterize variation for the 3,000-example setting in Table 6 or for the other model scales. Average improvements of 2.0, 2.9, and 2.1 percentage points across scales are positive evidence, but agreement in direction across scales should be distinguished from training variance within a setting.

    Separately, PixelBench's model rankings, algorithm comparisons, and resolution trends are based on finite test sets, and the tables do not provide paired uncertainty estimates for key differences.

    It is therefore difficult to assess how much smaller ranking differences and PRT gains exceed training variation or variation in the test instances. These two sources of uncertainty also cannot be characterized by the same three-seed experiment.

  3. The evaluation units and aggregation protocols of the fixed-budget controls and main experiments are not fully reconciled.

    Tables 5 and 6 report 22,902 official evaluation units, whereas Table 10 reports 22,554 after likewise describing MMVP pair-level aggregation and MMBench CircularEval aggregation, a difference of 348 units. Table 10 does not provide enough benchmark-level counts to reconstruct this total, leaving it unclear whether the difference comes from sample filtering, benchmark versions, valid-output handling, or a counting error.

    This discrepancy alone does not invalidate the positive gains within either table. However, if the sample composition or scoring rules changed, the magnitudes of gains across the two tables should not be directly used to judge how much benefit remains under fixed training budgets. For the example-weighted average, benchmark counts also determine the weights.

    The evaluation lists, scoring rules, and corresponding FV baselines should therefore be specified for both experiments so that the fixed-budget controls can be fully interpreted.

Questions for authors
  1. Training attribution and scope of the conclusions:

    Do you have results for another auxiliary objective, such as captioning, using the same pixel-source images under a comparable training budget?

    If so, how much advantage remains for the pixel objective after matching image exposure and training amount? If not, please limit the conclusion to the effectiveness of the current data mixture and training configuration, and identify what existing evidence supports a distinctive contribution from the pixel objective itself.

    Appendices D.3–D.4 already describe differences in target length and the share of training tokens. Please explain how these limit mechanistic attribution and how the absence of a corresponding fixed-budget control for 8B affects conclusions across scales.

  2. Stability and uncertainty validation:

    Could you provide repeated training results for the main settings in Table 6, or explicitly state that these have not been measured?

    Please distinguish variation in the main setting from the three-seed results in Table 11, which use 4B, 4,000 examples, and a different MMVP protocol.

    Using the existing PixelBench outputs, could you also provide paired uncertainty estimates over the same test instances for key model comparisons, algorithm comparisons, and adjacent resolutions?

    The priority is differences that support the main conclusions; a complete set of repeated experiments for every table is unnecessary.

  3. Evaluation protocol and count clarification:

    What are the benchmark-level counts underlying the 22,554 units in Table 10 and the 22,902 units in Tables 5 and 6, and where does the 348-unit difference arise?

    Do the fixed-budget and main experiments use the same samples, scoring rules, valid-output handling, and corresponding FV baselines?

    If protocols or samples differ, could you recompute results from existing outputs on a common evaluation list, or specify which within-table comparisons remain valid and which cross-table gains should not be directly compared?

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer H8Su (Part 1/3): Same-Image Caption, Token-Matched & 8B Controls

We thank Reviewer H8Su for the thorough, constructive review. Below we address all three questions with new controls, multi-seed runs, paired bootstrap intervals, and an exact unit reconciliation.

(1) Same-Image Caption Control, Token-Budget Matching, and 8B Fixed-Budget Control (re: Weakness 1 and Question 1)

  • Same-image Detailed Caption control: Using the exact same 3,044 source images ("3k") as PRT, we trained a +3k Detailed Caption baseline on Qwen3-VL-4B (combining fluent VLM spatial descriptions with exact 8-color PIA statistics specifying the main subject, 2D foreground/background/left/right layout, and all 8 #RRGGBB hex colors with area shares; ~185 supervised tokens/response, omitting only the pixel grid). Under Fixed-Budget (64,000 samples, 1,000 steps; unified N = 22,902), +Detailed Caption yields only +0.54 Example / +0.88 Benchmark, whereas pure-grid +Pixel-16 achieves +2.21 Example / +3.10 Benchmark (>4× the Example gain; net +1.67 Ex / +2.22 Bench advantage over matched-image captioning), leading +Detailed Caption by +13.3 on MMVP, +5.6 on BlindTest, +3.9 on MMBench, +3.2 on MMStar, +2.8 on BLINK, and +2.6 on CV-Bench.
  • Target length, token share, and strict 0-token-difference controls: Because 256×256 Pixel images use only 64 vision tokens (vs. 386.3 average vision tokens per FV image), a Pixel-16 example has 530.9 total sequence tokens (410.3 supervised), closely matching an average FV example (514.6 total, 85.2 supervised). We agree that in 32²/64²/Mix settings, longer grid responses raise the supervised-token share (11.42M in Additive +Pixel-Mix vs. 5.45M in FV-only). To test whether gains rely on extra supervised or total tokens, we ran two strict 0-token-difference controls on Qwen3-VL-4B: (i) Sup-Token-Matched +Pixel-16 (49,335 FV + 3,044 Pixel-16 = 52,379 rows, 819 steps; 0 diff in supervised loss tokens: 5,450,195 vs. 5,450,195—fewer than +Detailed Caption's ~5.76M—and −18.2% total sequence tokens), gaining +2.26 Example / +3.63 Benchmark (moreover, Fixed-Budget gains do not rise with grid length from 16²→32²→64²: +2.21 → +1.30 → +1.14, ruling out target length alone); and (ii) Seq-Token-Matched +Pixel-Mix (51,784 FV + 3,044 Pixel-Mix = 54,828 rows, 857 steps; 0 diff in total vision+prompt+target tokens: 32,933,037 vs. 32,933,037), gaining +2.80 Example / +3.86 Benchmark. While these controls rule out extra updates, image exposure, token volume, and dense captions as sole explanations, changing supervision format alters output token structure and FV mixture ratios; we will retain calibrated mechanistic scope statements in Sec. 5 and App. D.4.
  • 8B Fixed-Budget control and cross-scale synthesis: On Qwen3-VL-8B under Fixed-Budget (64,000 samples, 1,000 steps), both 8B Fixed +Pixel-16 (+1.21 Ex / +1.07 Bench, 10/13 up) and 8B Fixed +Pixel-Mix (+1.07 Ex / +0.59 Bench, 10/13 up) outperform 8B FV-only. Across 2B, 4B, and 8B, fixed-budget +Pixel-16 is the most consistent setting (+3.15 / +2.21 / +1.21 Ex), whereas mixed-resolution replacement under a fixed sample cap has higher single-seed variance (Sec. (3): 4B Fixed +Pixel-Mix is −0.01 Ex / +0.99 Bench in raw logs, whereas 2B/8B Fixed +Pixel-Mix and 4B Seq-Token-Matched +Pixel-Mix are strongly positive):
Protocol (Unified N = 22,902)SettingSamples / StepsToken Budget vs. 64k FV-onlyΔ Ex*Δ Bench (13)*
4B Same-Image Caption (new)Fixed +3k Detailed Caption64,000 / 1,000~0% FLOPs diff (~5.76M sup)+0.54+0.88
4B Fixed-Budget PRTFixed +3k Pixel-1664,000 / 1,000+0.23% seq / ~0% FLOPs diff+2.21+3.10
4B Fixed-Budget PRTFixed +3k Pixel-32 / 6464,000 / 1,0000% step diff+1.30 / +1.14+1.82 / +2.30
4B Token-Matched PRT (new)Sup-Token-Matched +Pixel-1652,379 / 8190 diff sup (5.45M; −18.2% seq)+2.26+3.63
4B Token-Matched PRT (new)Seq-Token-Matched +Pixel-Mix54,828 / 8570 diff total seq (32.93M)+2.80+3.86
4B Additive PRT (Paper)Additive +3k Pixel-16 / Mix67,044 / 1,048+4.91% / +19.24% total seq+3.14 / +2.94+4.07 / +3.70
8B Fixed-Budget PRT (new)Fixed +3k Pixel-1664,000 / 1,000~0% FLOPs diff+1.21+1.07 (10/13 up)
8B Fixed-Budget PRT (new)Fixed +3k Pixel-Mix64,000 / 1,0000% step diff+1.07+0.59 (10/13 up)

(*: The exact 3,044-image pool was rounded to "3,000" / "67,000" / "1,047 steps" in Sec. 5.1 & App. D.1–D.3 [ceil(67,044 / 64) = 1,048 exact steps; 49,335 + 3,044 = 52,379 and 51,784 + 3,044 = 54,828], which we will make exact in revision. Deltas are computed from unrounded scores.)

AuthorsOpenReview ↗

Author Response to Reviewer H8Su (Part 2/3): Main-Setting 3-Seed Stability & Paired Bootstrap CIs

(2) Multi-Seed Training Stability vs. Paired Evaluation Uncertainty (re: Weakness 2 and Question 2)

  • Multi-seed training stability in the main Table 6 setting: Given rebuttal compute limits, we prioritized matched 3-seed paired training on the flagship main Table 6 Qwen3-VL-4B setting (+3k Pixel-Mix, N = 22,902, pair-level MMVP), training both FV-only and FV + 3k Pixel-Mix under Seeds 43 and 44 alongside Seed 42 (6 full runs; multi-seed runs for 2B and 8B in Table 6 have not been measured). All 3/3 seed pairs are positive on both metrics, with non-overlapping distributions across all 6 runs (min +PRT > max Baseline: 55.83 > 54.23 Ex and 56.15 > 55.67 Bench): Paired Δ Example = +2.17 ± 0.66 (SEM = 0.38, 95% CI [+0.53, +3.82], strictly > 0) and Paired Δ Benchmark = +1.94 ± 1.63 (SEM = 0.94, 95% CI [−2.12, +5.99])—where the wider tdf=2t_{df=2} interval on unweighted Benchmark Avg reflects equal 1/13 weighting of small benchmarks (MMVP N=150N=150, V-STAR N=191N=191, baseline SD = 1.02 vs. +PRT SD = 0.64), whereas pooled Example Avg (N=22,902N=22,902) is tightly concentrated. We also separate the auxiliary Table 11 study (4B + 4k Pixel, original 300-item MMVP):
Multi-Seed Study & SettingSeed / SummaryBaseline (Ex / Bench)+PRT (Ex / Bench)Paired Δ Ex (95% CI)*Paired Δ Bench (95% CI)*
Main Table 6: 4B (+3k Mix, N=22,902, pair-MMVP)Seed 4253.90 / 53.6356.84 / 57.33+2.94+3.70
↳ Same Main Table 6 settingSeed 43 (new)54.06 / 54.6955.83 / 56.31+1.77+1.63
↳ Same Main Table 6 settingSeed 44 (new)54.23 / 55.6756.05 / 56.15+1.82+0.48
↳ Main Table 6 (4B +3k Mix)3-Seed Mean ± SD (SEM)54.06±0.17 / 54.66±1.0256.24±0.53 / 56.60±0.64+2.17 ± 0.66 (0.38) [+0.53, +3.82]+1.94 ± 1.63 (0.94) [−2.12, +5.99]
Aux. Table 11: 4B (+4k Pixel, orig. 300-MMVP)3-Seed Mean ± SD (SEM)——+1.58 ± 0.15 (0.09) [+1.21, +1.95]+1.84 ± 0.20 (0.11) [+1.35, +2.33]

(*: Paired deltas, means, and SDs are computed from unrounded per-seed scores; we will report these 3-seed statistics alongside Seed 42 in revised Sec. 5.2.)

  • Paired instance-level uncertainty on PixelBench: From saved per-item PixelBench outputs (100 T2P concepts, 100 I2P images, and 100 image clusters [6 Qs/image: 2 oracles × 3 resolutions, 600 Qs]), we compute paired non-parametric bootstrap 95% CIs (10,000 resamples) and two-sided paired sign-flip permutation pp-values: (1) all headline leadership claims are significant (p<.001p < .001); (2) both sides of the VLM-vs.-algorithm crossover are significant (Sol > PIA at 16²/32² Top-1, p<.001/.0445p < .001 / .0445; PIA & SSIM-opt > Astra at 64² DINO, p<.001p < .001); and (3) adjacent-resolution scaling divergence is clear—Astra improves from 32²→64² (+26.0% Top-1 / +.149 DINO, p<.001p < .001, after +32.0% / +.154 from 16²→32²), whereas Kimi K3 (−16.0%, p=.0053p = .0053) and Seed-2.1 (−12.0%, p=.0122p = .0122) degrade from 32²→64², while T2P 2nd vs. 3rd (Sol vs. Opus 5, p=.2642p = .2642) and Gemini 3.6 Flash's single-step 32²→64² T2P diff (−6.0%, p=.2141p = .2141) are statistical ties:
Claim CategoryPaired Comparison (AA vs. BB)MetricPaired Units (NN)Point Diff (Δ\Delta)95% Paired Bootstrap CIPaired pp-value
T2P LeaderGPT-6 Astra vs. GPT-5.6 SolAggregated / Judge Avg100 concepts+9.84 / +14.01[+7.28, +12.62] / [+11.95, +16.12]<.001 / <.001
T2P 2nd vs. 3rdGPT-5.6 Sol vs. Claude Opus 5Aggregated / Top-1 Avg100 concepts+1.60 / −0.33%[−1.25, +4.38] / [−5.00%, +4.33%].2642 / .8508
I2P LeaderGPT-6 Astra vs. GPT-5.6 SolSigLIP / DINO Avg100 images+.066 / +.081[+.055, +.076] / [+.067, +.097]<.001 / <.001
I2P Low-Res (16²/32²)GPT-5.6 Sol vs. PIA (reduced)100-way Top-1 (16² / 32²)100 images+26.0% / +11.0%[+16.0%, +36.0%] / [+1.0%, +21.0%]<.001 / .0445
I2P High-Res (64²)PIA / SSIM-opt vs. GPT-6 AstraDINO Cos (64²)100 images+.164 / +.217[+.127, +.199] / [+.179, +.256]<.001 / <.001
T2P Resolution (32²→64²)Gemini 3.6 F / Kimi K3 (64² − 32²)500-way Top-1100 concepts−6.0% / −10.0%[−14.0%, +2.0%] / [−18.0%, −2.0%].2141 / .0312
I2P Resolution (32²→64²)GPT-6 Astra (64² − 32²)Top-1 / DINO Cos100 images+26.0% / +.149[+15.0%, +37.0%] / [+.122, +.177]<.001 / <.001
I2P Resolution (32²→64²)Kimi K3 / Seed-2.1 (64² − 32²)100-way Top-1100 images−16.0% / −12.0%[−26.0%, −6.0%] / [−21.0%, −4.0%].0053 / .0122
Pixel UnderstandingGPT-6 Astra vs. GPT-5.6 SolOverall Accuracy100 image clusters (600 Q)+28.33%[+22.17%, +34.50%]<.001
AuthorsOpenReview ↗

Author Response to Reviewer H8Su (Part 3/3): Exact Reconciliation of 22,902 vs. 22,554 Units

(3) Exact Reconciliation of the 22,902 vs. 22,554 Evaluation Units (re: Weakness 3 and Question 3)

  • Source of the 348-unit difference: Across all 13 benchmarks, 12 benchmarks have 100% identical sample lists and unit counts between Tables 5–6 (22,902) and Table 10 (22,554): MMVP (150 pairs), BabyVision (388), CV-Bench (2,638), TallyQA* (3,000), HallusionBench (1,129), V-STAR (191), Q-Bench (1,495), MMStar (1,500), MMBench (1,292 CircularEval groups), BlindTest (3,434), BLINK (1,901), and VLMBias (2,784), totaling 19,902 units. The sole difference is PerceptionBench: earlier Table 10 runs evaluated 2,652 single-image items (19,902 + 2,652 = 22,554), whereas Tables 5, 6, and 9 included the remaining 348 multi-image items for the full 3,000-item benchmark (19,902 + 3,000 = 22,902; 3,000 − 2,652 = 348).
  • Within-table consistency and unified 22,902 recomputation: Both the main and fixed-budget experiments share the exact same 64k, 1,000-step Seed-42 FV-only baseline checkpoint; within each table, FV-only and all +Pixel runs used identical sample lists, scoring rules, and valid-output handling, so all within-table comparisons remain valid. Below we account for every row of Table 10, recomputing the saved 4B single-resolution checkpoints and new 8B runs on the full 22,902-unit list (the 348 multi-image items represent 1.5% of units and shift Δ Example by ≤0.06 pp).
  • Data retention scope & Table 10 erratum: Earlier 2B and 4B Fixed +Pixel-Mix runs saved only the 22,554 prediction files without retaining model weights, so their comparisons are strictly valid within Table 10 (N = 22,554) and should not be directly subtracted against N = 22,902 tables without noting this 1.5% weight difference. In addition, re-auditing the raw prediction logs revealed a +1.14 pp Ex / +1.00 pp Bench transcription error in Table 10's printed 4B Pixel-Mix row (raw log: −0.01 Ex / +0.99 Bench, pulled down by single-seed BlindTest −4.5, whereas our new 0-token-diff Seq-Token-Matched +Pixel-Mix achieves +2.80 Ex / +3.86 Bench), while all other 7 Table 10 rows match to 0.001 pp; we will update Table 10 and revise Appendix D.3 accordingly:
Fixed-Budget Setting (64k Samples, 1,000 Steps)Table 10 Δ Example (N=22,554, PB=2,652)Unified Δ Example (N=22,902, PB=3,000)Table 10 Δ Benchmark (PB=2,652)Unified Δ Benchmark (N=22,902, PB=3,000)
Qwen3-VL-2B Fixed +Pixel-16 / 32 / 64 / Mix+3.15 / +1.53 / +2.83 / +2.97(22,554 predictions saved only)+4.28 / +2.48 / +3.74 / +3.58(within-table valid on 22,554)
Qwen3-VL-4B Fixed +Pixel-16+2.27+2.21+3.13+3.10
Qwen3-VL-4B Fixed +Pixel-32+1.34+1.30+1.83+1.82
Qwen3-VL-4B Fixed +Pixel-64+1.17+1.14+2.31+2.30
Qwen3-VL-4B Fixed +Pixel-Mix−0.01 (printed +1.13 in Tab. 10)(22,554 predictions saved only)+0.99 (printed +1.99 in Tab. 10)(within-table valid on 22,554)
Qwen3-VL-8B Fixed +Pixel-16 (new)—+1.21—+1.07
Qwen3-VL-8B Fixed +Pixel-Mix (new)—+1.07—+0.59

We will incorporate all of the above controls, 3-seed and paired bootstrap intervals, exact sample/step counts, and the Table 10 / Appendix D.3 corrections into the revised manuscript with calibrated scope statements. We thank you again for the rigorous feedback, hope these updates resolve your methodological concerns, and welcome any further questions during the discussion period.

Dear Reviewer,

Just a gentle reminder, as less than a day remains for the rebuttal period. Please feel free to raise any remaining concerns, and we would be happy to discuss them. Thank you!

Reviewer_2ob3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether current generative models can understand and generate images by explicitly specifying pixel values (Image-to-Pixel and Text-to-Pixel), without learning an additional tokenizer or decoder. The authors formulate a task in which a color palette containing 2 to 8 colors is first defined, and models are required to select colors from this palette to fill a pixel grid.

The paper reports that recent powerful generative models, including GPT-6, Claude Opus, and GPT-5.6, can perform direct text-to-pixel generation and achieve better image-to-pixel conversion results than conventional algorithm-based approaches. The authors also provide quantitative evaluations of pixel generation at different resolutions. Furthermore, they propose Pixel Reconstruction Tuning (PRT), a method for training vision-language models (VLMs). Experimental results demonstrate that PRT improves the performance of Qwen3-VL-Instruct across 13 datasets.

Prior work, such as Pixel Art Bench, has already explored pixel-art generation capabilities using 24×24 images and AOMVs. Therefore, the contributions of this paper are primarily incremental, focusing on the evaluation framework and experimental design. Specifically, the paper provides more detailed task definitions and expands the evaluation to cover more models, concepts, and resolutions. It also jointly investigates Text-to-Pixel (T2P), Image-to-Pixel (I2P), and pixel understanding through textual representations.

Critical strengths
  1. Unlike existing approaches that require learning a mapping between text and pixel spaces, this paper explores pixel reconstruction through purely textual representations. This provides valuable insights into the visual understanding capabilities of existing generative models.

  2. The experiments demonstrate that generative models can not only understand image pixel tokens but also exhibit a strong understanding of the relationships between textual descriptions and pixel tokens.

  3. The proposed Pixel Reconstruction Tuning (PRT) offers a promising direction for enhancing model capabilities by encouraging models to learn and exploit visual cues.

  4. Experimental results show that the proposed auxiliary training method consistently improves the performance of models at different scales.

Critical weaknesses
  1. Insufficient analysis of the pixel generation process. Although the paper introduces a benchmark, it does not provide a sufficiently detailed analysis of how individual models perform pixel-by-pixel generation. For example, why does GPT-6 Astra achieve better generation performance? Does it internally invoke specific image recognition or generation algorithms? These aspects require further clarification.

  2. Limited discussion of the computational cost and efficiency of PRT. The paper does not adequately discuss the training costs and efficiency of PRT. In particular, representing images as pixel grids may require substantially more training tokens than conventional visual question-answering approaches.

  3. Insufficient evaluation of pixel-level reconstruction accuracy. Most evaluation metrics focus on semantic recognition or understanding, while the paper lacks sufficiently detailed quantitative metrics for assessing pixel-level reconstruction fidelity.

  4. Limited coverage of the proposed benchmark. The T2P evaluation includes only 100 common concepts, while the I2P evaluation uses only 100 images from Imaginate, with a single image per category. Moreover, most of these images contain little or no background information. Therefore, it remains unclear whether the reported performance generalizes to more complex natural images.

Questions for authors
  1. The paper does not provide a detailed account of the inference process for each evaluated model. Consequently, it is difficult to determine whether these models might internally rely on conventional algorithms or tools for pixel extraction or image generation. Could the authors provide more detailed inference traces or other evidence clarifying how the models perform these tasks, to the extent that such information is accessible?

  2. Could the authors extend the evaluation to images containing more complex backgrounds and provide quantitative assessments of both pixel-level reconstruction fidelity and semantic recognition accuracy?

  3. Are the performance improvements from PRT primarily attributable to the pixel reconstruction objective, or simply to additional training on visual data? For example, consider two alternative training tasks: (1) given an image, the model is required to output a 32×32 pixel grid; and (2) given the same image, the model is required to generate a detailed description of the objects, including their colors and spatial relationships. Would both approaches lead to similar improvements in visual understanding? What makes PRT fundamentally different from conventional visual supervision methods?

7 comments
AuthorsOpenReview ↗

Author Response to Reviewer 2ob3 (Part 1/5): Comparison with Pixel Art Bench & Scope

We thank Reviewer 2ob3 for the constructive feedback and for recognizing our text-based pixel interface and multi-scale PRT experiments. Below we address every concern with new controlled experiments and quantitative analyses.

(1) Comparison with Pixel Art Bench and Scope of Contributions (re: Paper Summary)

We explicitly cite and discuss Pixel Art Bench in Appendix A.1. As summarized below, Pixel Art Bench (Emergent Mind, 2026) is an online leaderboard that shares only the basic palette-indexed JSON serialization for T2P, whereas PixelBench is a systematic multi-task benchmark and training study. (Note: We could not locate the abbreviation "AOMVs" in the public Pixel Art Bench materials; if it refers to a specific method or metric, we would be glad to address it.)

DimensionPixel Art Bench (Emergent Mind, 2026)PixelBench (Ours)
Protocol StatusWeb leaderboard (no paper; judge prompt, pair sampling, and retry rules undisclosed)Reproducible benchmark with released prompts, deterministic metrics, and configs (Appendices E–L)
Tasks EvaluatedT2P only (text →\to pixel grid)Three coupled tasks: T2P generation, I2P image reconstruction, and Pixel Understanding
Dataset Scale7 test items (7 text prompts at 1 resolution)1,200 test items across 3 resolutions (162,322,64216^2, 32^2, 64^2): 300 T2P (100 concepts), 300 I2P (100 ImageNet images), and 600 Understanding (2 sources ×\times 300)
ResolutionsFixed 24×24 grid16², 32², 64² main suite + 10-resolution scaling sweep (16² to 128²)
MetricsSingle LLM pairwise judge (GPT-5.4 Elo)Deterministic metrics + VLM rubric: 100/500-way SigLIP Top-1 & MRR, paired SigLIP & DINOv2 cosine, 6-dim VLM rubric, Valid/Flat rates
BaselinesNone10 palette-matched classical downscaling & pixelization algorithms across 4 tiers (Table 2, Appendix L)
Analysis & TrainingNone (leaderboard only)Cross-model generation–understanding correlation (Table 4) + Pixel Reconstruction Tuning (PRT) across Qwen3-VL (2B, 4B, and 8B across 13 visual-centric benchmarks)
AuthorsOpenReview ↗

Author Response to Reviewer 2ob3 (Part 2/5): Inference Process, No-Tool Evidence & GPT-6 Astra

(2) Inference Process, Absence of External Tools, and Why GPT-6 Astra Excels (re: Weakness 1 and Question 1)

  • Strict no-tool text interface: Every model is queried through its standard API completion endpoint using only the task prompt (Appendix G) and raw input image (for I2P), with full inference parameters in Appendix E (Table 12). No external tools, code interpreters, image-generation APIs, or pixelization scripts are provided or enabled. Models emit UTF-8 JSON tokens (palette and pixels) directly, and rendering is performed offline via deterministic palette lookup.
  • Three observable lines of evidence confirming token-by-token generation rather than classical filtering:
    1. Characteristic high-resolution autoregressive failure modes (Section 4 & Appendix B): At 64²–128², models exhibit effective-resolution collapse into coarse ~8×8 macro-blocks (GPT-5.6 Sol), row-stripe repetition (Claude Opus 5), or subject omission (Gemini 3.6 Flash), whereas any deterministic image-processing tool would sample uniformly at any resolution.
    2. Low-resolution semantic abstraction and opposite foreground/background fidelity gaps (Table 2 & COCO-Clutter-100 below): At 16²–32², leading VLMs outperform classical algorithms on semantic Top-1 (61% vs. 50% at 32²) and exhibit a positive foreground-over-background DINO fidelity gap (+.022 to +.024) on cluttered scenes, whereas all classical algorithms have negative gaps (−.017 to −.021).
    3. Resolution-scaling reasoning tokens and text-based coordinate planning: As shown below for the five multimodal models whose APIs report reasoning/completion token usage without an external 2,500-token cap, internal reasoning token usage increases substantially as the canvas grows from 16² to 64². For models exposing raw thinking traces (e.g., Kimi K3, DeepSeek V4 Pro), both T2P and I2P traces follow a structured text-planning routine—binding palette hex codes to semantic parts, partitioning vertical row ranges, and explicitly counting horizontal character spans so each row sums to WW (in I2P, summarizing subject pose and foreground/background color regions in natural language before estimating row spans rather than reading a raw pixel array). Below is a verbatim excerpt from Kimi K3's raw reasoning trace on 32×32 T2P "mushroom":
Design a mushroom: red cap with white spots, cream stem, background light. Palette: 0 background light blue, 1 red cap, 2 white spots, 3 cream stem, 4 dark outline maybe. Build 32 rows.
Cap: rows 6-17, dome shape. Stem rows 18-27. [...]
Row6: "00000000000011111111000000000000" — count: 12 zeros + 8 ones + 12 zeros =32. Good. [...]
Row9: 7 zeros + cap with spot: positions 7-24 (18 wide). Put spot at 11-12: ones 7-10, 2 at 11-12, ones 13-24.
Row9 string: "0000000"+"1111"+"22"+"111111111111" (12) + "0000000" → 7+4+2+12+7=32 ✓. [...]
Row16: cap bottom row: 4 zeros + 24 ones (4-27) +4 zeros? 4+24+4=32 ✓.
Row17: underside dark: 4 zeros + 24 of "4" +4 zeros.
Row18: stem: 13 zeros + "3"*6 +13 zeros =32 ✓. Add outline? Keep simple.
Rows18-25 stem, widen bottom: rows 24-25: 12 zeros +8 "3"+12 zeros.
Row26: stem base: 11 zeros + 10 "3" + 11 zeros. [...]
Palette: ["#7EC8E3" bg, "#D62828" cap, "#FFFFFF" spots, "#F5E6C8" stem, "#8B4513" underside].

Moreover, more reasoning tokens do not guarantee better spatial structure: GPT-6 Astra achieves the strongest results with compact ~1,000-token reasoning usage (510→1,714 from 16²→64²), whereas Kimi K3 averages 4,976–8,085 reasoning tokens yet still degrades at 64².

  • Why GPT-6 Astra excels: While proprietary model weights and hidden states are not externally accessible, interface-level behavioral analysis across Section 4 (Figures 4–5) and Appendix K (Table 18) shows that as grid size increases from 16² to 128² (256 to 16,384 cells), other models lose 2D coordinate alignment over long sequences, whereas Astra preserves global proportions (leading Table 18 on both Struct. 18.04/20 and Detail 16.38/20 at 64²) and allocates new cells to boundary refinement and internal texture.
Representative ModelT2P Avg. Reasoning Tokens (16² / 32² / 64² / Avg)I2P Avg. Reasoning Tokens (16² / 32² / 64² / Avg)
GPT-6 Astra510 / 1,056 / 1,714 / 1,093604 / 826 / 1,685 / 1,038
Claude Opus 5403 / 736 / 1,528 / 889431 / 761 / 1,684 / 959
GPT-5.6 Sol619 / 1,052 / 2,065 / 1,2452,491 / 4,115 / 3,756 / 3,454
Kimi K32,409 / 5,501 / 7,017 / 4,9766,040 / 7,702 / 10,512 / 8,085
Gemini 3.6 Flash385 / 1,232 / 4,446 / 2,021432 / 1,265 / 4,463 / 2,053
AuthorsOpenReview ↗

Author Response to Reviewer 2ob3 (Part 3/5): Fidelity Metrics & COCO-Clutter-100 Evaluation

(3) Reconstruction Fidelity Metrics and Complex-Background Evaluation (COCO-Clutter-100) (re: Weaknesses 3 & 4 and Question 2)

  • Why paired DINOv2 cosine and 10 classical fidelity baselines evaluate reconstruction fidelity: Our I2P suite pairs semantic recognition (100-way Top-1, SigLIP cosine) with appearance/structural fidelity (paired DINOv2-Large CLS cosine, Section 3.2 & Table 2) and benchmarks models against 10 palette-matched classical algorithms optimizing pixel/structural fidelity (including SSIM-optimal downscaling, DPID, Bicubic/Lanczos, and PIA; Table 19). We avoid raw pixel-wise MSE/PSNR against the 256×256 source as a primary score because 16²–64² 8-color pixel art is a spatial and chromatic abstraction: box averaging minimizes MSE via blurry blocks, whereas a 1-cell shift on a 16×16 grid is pathologically penalized by pixel-wise L2; combined with our 32² foreground/background DINO patch decomposition below, our metrics quantify spatial fidelity on both subject and background.
  • Benchmark scope clarification and new COCO-Clutter-100 benchmark: For T2P, the 100 concepts span five broad semantic categories (Appendix H) and are evaluated against a 500-way fine-grained distractor pool (Appendix J). For I2P, the 100 main inputs are center-cropped natural photos from ImageNet-1K validation (Figure 7). To test multi-instance categories and heavy background clutter, we constructed COCO-Clutter-100 from MS COCO 2017 Panoptic (val2017; n=100n = 100 256×256 crops across 46 categories, >2 cluttered instances/category; mean background area = 77.5%, mean non-primary panoptic segments = 8.9; significantly higher Sobel edge density, CIE-Lab color entropy, and feature congestion than ImageNet-100, Cohen's d=1.01d = 1.01–1.371.37, p<10−13p < 10^{-13}), evaluating models across the top, mid, and lower tiers of Table 2 plus newly released GPT-6 Sol and Gemini 3.8 Flash (full configs will be added to Appendix E) alongside the three classical baselines:
    1. The VLM semantic advantage widens under heavy background clutter: When cluttered backgrounds occupy 77.5% of image area, 8-color classical algorithms suffer severe semantic collapse at 16²–32² (Top-1 drops to 2%–11%) because background textures monopolize the 8-color palette. Frontier VLMs perform figure–ground abstraction, achieving nearly 5× higher Top-1 at 16² (GPT-5.6 Sol 19.0% vs. PIA 4.0%), over 3× at 32² (GPT-6 Astra 35.0% vs. PIA 11.0%), and even at 64² GPT-6 Astra (47.0%) outperforms all classical algorithms on semantic retrieval (≤ 32.0%), while classical methods still lead on dense local feature fidelity at 64² (DINO .408–.476 vs. .259).
    2. Ground-truth panoptic foreground/background patch decomposition explains why: Using COCO panoptic masks to compute 32² DINO patch similarity separately on foreground subject (FG) vs. background (BG), every classical algorithm has a negative FG − BG gap (−.017 to −.021), confirming that semantics-agnostic quantization prioritizes the larger background area. Conversely, leading VLMs maintain a positive FG − BG fidelity gap (GPT-6 Astra +.022, GPT-5.6 Sol +.024), achieve 3.6×–4.0× higher mask-isolated foreground retrieval (FG-Only Top-1: 18.0%–20.0% vs. PIA 5.0%), and remain robust across Moderate-High → High → Extreme clutter tertiles (GPT-6 Astra mean Top-1: 29.3% → 28.4% → 34.3%).
Method (COCO-Clutter-100, 77.5% BG)Top-1 ↑ (16 / 32 / 64 / Avg)SigLIP ↑ (16 / 32 / 64 / Avg)DINO ↑ (16 / 32 / 64 / Avg)32² FG Top-1 ↑32² DINO (FG / BG / Gap*)
GPT-6 Astra10 / 35 / 47 / 30.7.591 / .619 / .655 / .622.026 / .123 / .259 / .13618.0%.239 / .217 / +.022
GPT-5.6 Sol19 / 31 / 19 / 23.0.583 / .597 / .576 / .585.025 / .086 / .069 / .06020.0%.185 / .161 / +.024
Gemini 3.8 Flash13 / 20 / 17 / 16.7.575 / .588 / .554 / .572.028 / .060 / .058 / .0497.0%.158 / .152 / +.006
GPT-6 Sol13 / 17 / 11 / 13.7.572 / .573 / .517 / .554.026 / .052 / .034 / .03710.0%.137 / .131 / +.006
Kimi K39 / 14 / 5 / 9.3.560 / .557 / .566 / .561.026 / .044 / .031 / .0345.0%.114 / .118 / −.005
PIA (reduced, 8c)4 / 11 / 32 / 15.7.580 / .597 / .663 / .613.025 / .089 / .408 / .1745.0%.199 / .215 / −.017
SSIM-opt. + median cut (8c)2 / 4 / 32 / 12.7.568 / .579 / .663 / .603.020 / .071 / .476 / .1893.0%.174 / .191 / −.018
Nearest + median cut (8c)2 / 6 / 26 / 11.3.566 / .574 / .649 / .596.023 / .059 / .425 / .1692.0%.155 / .177 / −.021

(*: Averages and FG−BG gaps are computed from unrounded per-image scores and may differ by .001 from rounded columns.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.