- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PixelBench, which tests whether language models can generate and understand images through a text-based palette and pixel grid, without a learned image tokenizer or decoder. It evaluates text-to-pixel generation, image-to-pixel reconstruction, and pixel understanding at several resolutions. Performance varies by model and often declines at higher resolutions. The authors also propose Pixel Reconstruction Tuning, which improves average results on 13 visual benchmarks for Qwen3-VL models at three sizes.
Critical strengths
- PixelBench provides a simple way to study visual generation and understanding through the model's existing text interface.
- The paper combines evaluation with a practical training study. Pixel Reconstruction Tuning improves average performance across 13 benchmarks and is tested at multiple model sizes.
Critical weaknesses
- The main training setup does not fully isolate gains from additional image exposure and training steps. The matched-data control helps, but comparisons with other auxiliary objectives are missing.
- Some conclusions about the link between pixel understanding and generation rely on correlations across only six or seven models. The results are suggestive but do not establish a general relationship.
Questions for authors
- Under the matched-data and matched-step setting, how consistent are the gains across all 13 benchmarks and model sizes?
4 comments
Author Response to Reviewer 3Tz6 (Part 2/3): Same-Image Detailed Caption & Token-Matched Controls
(2) Comparison with Alternative Auxiliary Objectives and Strict Token-Matched Controls (re: Weakness 1)
To isolate the pixel-grid objective from additional image exposure and training tokens, we ran two new controlled studies on Qwen3-VL-4B (N = 22,902 official units):
- Same-image +3k Detailed Caption control: Using the exact same 3,044 source images ("3k") as PRT, we trained a fixed-budget control (64,000 samples, 1,000 steps) whose target combines fluent frontier VLM spatial descriptions with exact 8-color PIA statistics to specify the main subject, 2D spatial layout, and all 8 dominant
#RRGGBBhex colors with area shares (~185 supervised tokens/response, ~5.76M total supervised tokens, omitting only the pixel grid). +Detailed Caption yields only +0.54 Example / +0.88 Benchmark, whereas pure-grid +Pixel-16 achieves +2.21 Example / +3.10 Benchmark (over 4× the Example gain). - Strict token-budget-matched controls: When we remove 14,665 or 12,216 FineVision examples so that supervised assistant loss tokens (Sup-Token-Matched +Pixel-16: 49,335 FV + 3,044 Pixel-16 = 52,379 rows, 819 steps, 5.45M supervised tokens—even fewer than +Detailed Caption's ~5.76M—and −18.2% total tokens) or total vision+text sequence tokens (Seq-Token-Matched +Pixel-Mix: 51,784 FV + 3,044 Pixel-Mix = 54,828 rows, 857 steps, 0 token diff) are 100% identical to 64k FV-only, PRT still improves performance by +2.26 to +2.80 Example Avg and +3.63 to +3.86 Benchmark Avg:
| Control Setting (Qwen3-VL-4B, Same Source Images) | Total Samples | Steps | Token Budget vs. 64k FV-only | Example Avg (N=22,902) | Δ Example* | Benchmark Avg (13) | Δ Benchmark* |
|---|---|---|---|---|---|---|---|
| Baseline (FV-only) | 64,000 | 1,000 | 1.00× (5.45M sup / 32.93M seq) | 53.90 | — | 53.63 | — |
| Fixed-Budget +3k Detailed Caption (no grid) (new) | 64,000 | 1,000 | Same samples & steps (~5.76M sup) | 54.44 | +0.54 | 54.51 | +0.88 |
| Fixed-Budget +3k Pixel-16 (pure grid) | 64,000 | 1,000 | Same samples & steps (+0.23% seq) | 56.10 | +2.21 | 56.72 | +3.10 |
| Fixed-Budget +3k Pixel-32 / 64 (pure grid) | 64,000 | 1,000 | Same samples & steps | 55.20 / 55.04 | +1.30 / +1.14 | 55.44 / 55.92 | +1.82 / +2.30 |
| Sup-Token-Matched +Pixel-16 (pure grid) (new) | 52,379 | 819 | 0 diff sup. tokens (−18.2% total seq) | 56.16 | +2.26 | 57.26 | +3.63 |
| Seq-Token-Matched +Pixel-Mix (pure grid) (new) | 54,828 | 857 | 0 diff total vision+text seq tokens | 56.70 | +2.80 | 57.48 | +3.86 |
| Additive +3k Pixel-16 / Mix (Paper Table 6/7) | 67,044 | 1,048 | +4.91% / +19.24% total seq tokens | 57.03 / 56.84 | +3.14 / +2.94 | 57.69 / 57.33 | +4.07 / +3.70 |
(*: The exact 3,044-image pool was rounded to "3,000" / "67,000" / "1,047 steps" in Sec. 5.1 & App. D.1–D.3 [ceil(67,044 / 64) = 1,048 exact steps], which we will make exact in revision. Deltas are computed from unrounded scores.)
Author Response to Reviewer 3Tz6 (Part 3/3): 12-Model Expansion of Generation–Understanding Correlation
(3) Scope and 12-Model Expansion of the Generation–Understanding Correlation (re: Weakness 2)
We agree with the reviewer that correlations across a finite set of models are descriptive rather than a universal law (as noted in Section 3.3). To test whether this relationship holds across a broader model pool and within individual model families, we evaluated 5 additional multimodal models (GPT-6 Sol, GPT-6 Luna, GPT-5.6 Luna, Gemini 3.8 Flash, and Grok 4.7) on the full 1,200-item PixelBench suite under the same no-tool API protocol as Appendix E (Table 12; full hyperparameters and Valid/Flat rates for new models will be added in revision), expanding the pool from 7 models (6 multimodal) to 12 models spanning 7 model families (11 multimodal across 6 families, plus text-only DeepSeek V4 Pro):
- Every correlation metric is statistically significant () across the expanded pool: While adding heterogeneous model families naturally moderates point-estimate correlation magnitudes relative to the initial 7-model sample (e.g., T2P Spearman : 1.000 → .876), statistical significance strengthens substantially—Pearson -values tighten 5× on T2P (; Spearman ) and 7× on I2P (; Kendall tightens from to ). Even when retaining only a single flagship representative per independent provider across all 7 families ( T2P / I2P) to eliminate within-provider clustering, correlations remain highly significant (T2P ; I2P ).
- Strict within-family monotonicity under identical tokenization: Within the GPT-6 family (Luna → Sol → Astra), Understanding (29.2% → 44.0% → 67.5%), T2P Agg. (65.72 → 84.56 → 90.83), and I2P Agg. (10.80 → 50.28 → 83.81) increase strictly monotonically; the GPT-5.6 family (5.6 Luna → 5.6 Sol) and full-task generational upgrades (5.6 Luna → 6 Luna, Gemini 3.6 → 3.8 Flash) likewise improve all tasks simultaneously (while 5.6 Sol → 6 Sol improves T2P and Understanding with lower I2P), and all 12/12 models improve when switching from PIA to the Astra-I2P oracle:
| Model | T2P Top-1 Avg / Judge Avg / T2P Agg.* | I2P Top-1 Avg / SigLIP / DINO / I2P Agg.† | Understanding (PIA / Astra-I2P / Overall) |
|---|---|---|---|
| GPT-6 Astra (paper) | 91.0 / 90.7 / 90.83 | 48.0 / .654 / .175 / 83.81 | 57.3% / 77.7% / 67.5% |
| GPT-6 Sol (new) | 87.7 / 81.5 / 84.56 | 45.7 / .579 / .081 / 50.28 | 35.0% / 53.0% / 44.0% |
| GPT-5.6 Sol (paper) | 85.3 / 76.7 / 80.99 | 56.0 / .588 / .094 / 64.22 | 32.3% / 46.0% / 39.2% |
| Gemini 3.8 Flash (new) | 82.3 / 77.3 / 79.82 | 44.3 / .565 / .071 / 44.37 | 35.7% / 52.3% / 44.0% |
| Claude Opus 5 (paper) | 85.7 / 73.1 / 79.39 | 40.7 / .561 / .060 / 38.22 | 24.0% / 46.7% / 35.3% |
| Grok 4.7 (new) | 77.3 / 76.2 / 76.78 | 27.7 / .491 / .038 / 7.68 | 22.7% / 29.7% / 26.2% |
| Kimi K3 (paper) | 81.7 / 70.3 / 75.96 | 36.7 / .541 / .040 / 26.89 | 18.3% / 36.3% / 27.3% |
| Gemini 3.6 Flash (paper) | 79.0 / 69.8 / 74.39 | 30.7 / .539 / .051 / 22.59 | 16.3% / 27.3% / 21.8% |
| Seed-2.1 Pro (paper) | 72.0 / 64.2 / 68.08 | 35.0 / .547 / .043 / 26.99 | 13.7% / 18.3% / 16.0% |
| GPT-6 Luna (new) | 69.3 / 62.1 / 65.72 | 25.7 / .519 / .034 / 10.80 | 23.0% / 35.3% / 29.2% |
| GPT-5.6 Luna (new) | 54.3 / 54.8 / 54.55 | 21.3 / .509 / .029 / 3.68 | 20.0% / 30.3% / 25.2% |
| DeepSeek V4 Pro (paper) | 53.0 / 51.5 / 52.22 | — (text-only) | 10.3% / 14.7% / 12.5% |
(*: T2P Agg. is the mean of the 6 unrounded per-resolution Top-1 and VLM Judge scores. †: I2P Agg. is re-normalized across all 14 rows [11 multimodal models + 3 algorithmic baselines] rather than the 9 rows in Table 2, shifting absolute values while preserving rank order; raw unnormalized Top-1, SigLIP, and DINO correlations with Understanding all remain statistically significant at .)
| Correlation Summary | Original Pool () | Original / / (-value) | Expanded Pool () | Expanded Pearson (-val) | Expanded Spearman (-val) | Expanded Kendall (-val) |
|---|---|---|---|---|---|---|
| Understanding – T2P | 7 | .871 (.0107) / 1.000 (<.0001) / 1.000 (.0004) | 12 | .793 (.0021) | .876 (.0002) | .748 (.0008) |
| Understanding – I2P | 6 | .930 (.0073) / .829 (.0416) / .733 (.0556) | 11 | .846 (.0010) | .761 (.0065) | .587 (.0125) |
We will incorporate the full multi-scale 13-benchmark breakdown, same-image captioning and token-matched controls, expanded 12-model analyses, and Table 10 updates into the revised manuscript. We thank you again for your positive and constructive feedback, and welcome any further questions during the discussion period.
Thank you for your clarification. I will keep my positive rating unchanged.
Author Response to Reviewer 3Tz6 (Part 1/3): 13-Benchmark Fixed-Budget Breakdown Across 2B/4B/8B
We thank Reviewer 3Tz6 for the positive assessment and helpful questions. Below we provide the full 13-benchmark breakdown across all three model scales under the fixed budget, our new same-image captioning and token-matched controls, and an expanded 12-model correlation study.
(1) Per-Benchmark Consistency Under Matched-Data and Matched-Step Settings Across 2B, 4B, and 8B (re: Question 1)
In addition to expanding Table 10 into a full 13-benchmark breakdown for all 2B and 4B settings, we completed representative 8B Fixed-Budget (64,000 samples, 1,000 steps) runs (Pixel-16 and Pixel-Mix). Across all three scales (2B, 4B, and 8B), fixed-budget single-resolution and mixed configurations improve the majority of benchmarks (11–12/13 on 2B, 10–12/13 on 4B single-resolution and 7/13 on 4B Mix, and 10/13 on 8B), with gains consistently concentrated on visual perception and spatial reasoning benchmarks (MMVP, BlindTest, CV-Bench, HallusionBench, BabyVision, MMBench, MMStar, PerceptionBench), closely mirroring Table 6:
(‡: 2B/4B per-benchmark rows reflect original Table 10 N=22,554 [2,652 single-image PerceptionBench items; 12/13 benchmarks identical to N=22,902], while parenthesized 4B values and 8B columns are evaluated on the full N=22,902 suite [3,000 PerceptionBench items], matching Section (2) below; *: 3k random TallyQA subset. †: Raw-log audit corrects a +1.14 pp Ex / +1.00 pp Bench transcription error in Table 10's printed 4B Pixel-Mix row [pulled down by single-seed BlindTest −4.5, whereas 2B/8B Fixed Mix and 4B Seq-Token-Matched Mix at +2.80/+3.86 are strongly positive], and we will revise Appendix D.3 accordingly; all other 7 Table 10 rows match to 0.001 pp. All summary averages are computed from unrounded benchmark scores.)