acceptodds
Under review as a conference paper at ICLR 2027

Making Language the Low-Rate Code of Vision: Successive Refinement from Words to Pixels

Abstract

An ordered visual code lets a short prefix stand in for an image, and the training loss decides what that prefix holds. Under pixel reconstruction the first bits go to the highest-variance directions: we construct a source for which a code that is rate-optimal for pixels at every length carries no caption information in its entire first half. A caption is therefore not the low-rate code of its image unless the code is built to make it one. We cast captioning and text-to-image generation as successive refinement under a distortion measure that changes along the code, from caption log-loss in the early stages to pixel distortion in the last. Putting words first then costs nothing among the caption stages exactly when the captions form a summarization chain, and costs the pixels at most the caption uncertainty left in a pixel-optimal reconstruction, which is large for short prefixes and small at full length. We build RUNG, an ordered code whose prefixes are supervised by a summarization ladder of captions and whose full length reconstructs the image; generation extends the code past the prompt and captioning truncates it. With data and capacity held fixed, ordering by language instead of reconstruction raises prefix–caption information 2.9× and GenEval from 0.76 to 0.83, at a cost of 2.8 dB PSNR at a quarter of the code and 0.1 dB at full length. RUNG-XL (3.4B) reaches 0.88 GenEval and 87.6 DPG-Bench in 32 network evaluations, and 0.91 GenEval when errors are redrawn at the rung where they appear; its fine captions entail 94.1% of the attribute tuples in its coarse ones, and it edits images without inversion.

Primary area:
generative models
Keywords:
multimodal generation, text-to-image generation, representation learning, successive refinement, ordered representations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zYBeOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RUNG, an ordered visual code that places caption-relevant information in early prefixes and reserves later rungs for image detail. It frames this design as successive refinement under a distortion schedule that shifts from caption supervision to pixel reconstruction. The theoretical results characterize when nested caption stages can be rate-optimal, illustrate how pixel-first ordering can omit caption information at low rates, and bound the pixel-rate cost of words-first ordering. Experimentally, the paper reports stronger prefix–caption alignment and text-to-image compositional performance than matched reconstruction-ordered alternatives, with little reconstruction loss at full code length.

Critical strengths
  1. The idea of explicitly choosing what early visual-code prefixes represent is compelling.
  2. The theory motivates the successive-summary caption ladder, and the predicted fidelity trade-off is tested experimentally.
  3. The controlled ordering comparison retrains models under matched architectures, data, and capacity, supporting the reported GenEval gain from 0.76 to 0.83. The evaluation covers generation, caption consistency, reconstruction, editing, and correction.
Critical weaknesses
  1. Appendix J uses a language-model cross-entropy H^(T)\hat H(T) in a Barber–Agakov-style estimate. Since H^(T)\hat H(T) can exceed H(T)H(T), the result is not necessarily a mutual-information lower bound. The relative ranking is supported by the InfoNCE check, but Figure 2a’s absolute values and the 2.9× ratio need more careful qualification.

  2. Proposition 3 concerns optimal information rates, whereas Figure 2c uses learned reconstructions and sums of marginal FSQ entropies. The plotted estimated “lower bound” is not guaranteed to be a lower bound either. These curves are useful diagnostics, but should not be presented as a direct verification of the proposition.

  3. LadderCap deliberately puts counts and spatial relations in short captions—precisely the attributes emphasized by GenEval. Existing ablations test how the ladder is constructed, but not which facts are prioritized. A matched successive-summary ablation with different early-caption priorities would help isolate this choice.

  4. The paper reports latency and compares against best-of-NN with the same verifier, but its network-evaluation counts combine operations with different costs. A verifier-cost breakdown and a latency-matched best-of-NN comparison would strengthen the claim.

Questions for authors
  1. The contribution of the proposed ordering is not sufficiently isolated. In Table 1, the words-first model receives caption supervision, while the pixel-first model does not. The improvement may therefore reflect both the additional language supervision and the ordering of information within the code. Could the authors include a comparison using the same caption data and supervision budget, with the language loss applied only to the full code? This would help clarify the additional benefit of supervising prefixes at different lengths.

  2. The interpretation of the information estimate needs clarification. Appendix J uses a language model’s cross-entropy to estimate caption entropy. Since cross-entropy generally upper-bounds the true entropy, substituting it into the expression does not by itself preserve the mutual-information lower bound. Could the authors clarify the justification for calling this estimate a lower bound?

  3. The effectiveness of prefix-level verification needs further analysis. The correction procedure checks attributes decoded from an intermediate prefix, but the final image is generated after additional blocks are sampled. How often does a prefix pass verification while the final image still violates the prompt? It would be useful to distinguish errors in the attribute checker from changes introduced during later generation. The authors should also clarify whether the 80% agreement threshold can allow important counting or spatial-relation errors to pass.

Reviewer_xAJjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that ordered visual codes trained on pixel reconstruction put high-variance content like illumination first, and proves such a code can carry no caption information in its first half. The authors instead treat captioning and T2I generation as successive refinement, judging early code stages by caption accuracy and later ones by pixel fidelity. They show this costs nothing among the caption stages when each coarser caption summarizes the next finer one, and that the cost to pixels is bounded and fades at full length. Their model, RUNG, places a ladder of summarized captions at the front of the code, so generation extends the code and captioning truncates it. Ordering by language instead of reconstruction raises GenEval from 0.76 to 0.83 at a cost of only 0.1 dB PSNR at full length, and the 3.4B RUNG-XL reaches 0.88 GenEval, or 0.91 when failing blocks are redrawn during generation.

Critical strengths
  1. The paper casts cross-modal generation as successive refinement under a distortion schedule. Each proposition yields a concrete, testable consequence: Proposition 1 becomes a rule for building caption data, Proposition 2 explains why reconstruction-ordered codes leave language out, and Proposition 3 with Corollary 1 bounds the pixel cost of putting words first. The proofs in Appendix A are short and check out, and the authors state their assumptions and the limits of the qualitative corollary openly.

  2. The core experiments that isolates the ordering signal are well-controlled. Table 1 and Figure 2 retrain the tokenizer and ascent model at full scale with only the ordering signal changed, so the gain from words-first ordering (GenEval 0.76 to 0.83, prefix–caption information 0.68 to 1.97 nats per token) is not confounded by architecture, data or capacity.

  3. Since generation extends the code and captioning truncates it, the same model generates, captions at a chosen granularity, corrects errors at the rung where they appear, and edits by swapping prefixes without inversion. Rung-level correction is compute-efficient: it reaches 0.91 GenEval at 1.2 times the network evaluations, while best-of-N with the same verifier reaches only 0.89 at 4 times.

Critical weaknesses
  1. The paper states that ordering accounts for 7 of the 16 GenEval points separating RUNG-L from FlexTok, and that the remaining 9 come from the rest of the design, citing Table 9. However, FlexTok with the same text-to-image head scores 0.67, while RUNG’s own pixel-first variant scores 0.76, and nothing in the paper explains this 9-point difference. Table 9 is run in a different setting, contains only single-factor ablations that do not sum to a decomposition, and never compares FlexTok’s tokenizer architecture against RUNG’s. The authors should either provide a controlled comparison of the two tokenizers or remove the claim.

  2. Section 5.3 claims that the advantage over Lumina-DiMOO on T2I-CompBench++ is concentrated in numeracy (+0.052) and 2D spatial relations (+0.048), and is close to zero for non-spatial relations. This is presented as evidence that the words-first order helps precisely where short prompts are explicit. Yet Table 6, which the text cites, reports only the benchmark mean, and no appendix gives a category breakdown. Since this claim supports one of the central arguments of the paper, the full per-category results should be included.

  3. Several conclusions rest on GenEval differences of one to three points, such as lambda=1 against lambda=3 (0.83 against 0.82), L=32 against L=64, truncated captions against successive summaries (0.76 against 0.79), and different summarizers (0.78 to 0.79). The paper reports seed variation only for rFID and FID. Without standard deviations across seeds or confidence intervals for GenEval, it is not possible to tell which of these differences are meaningful and which are within noise.

  4. Two of the strongest understanding results use metrics introduced by the authors. NCC measures whether a fine caption entails a coarse one, and the paper itself attributes RUNG’s 94.1 to the fact that both captions read prefixes of one shared code, whereas baselines must produce two independent outputs. The metric therefore rewards the architecture by construction rather than measuring captioning quality. GRANBENCH is described in a single sentence, with no information on who annotated the granularity labels, how granularity was defined, or how the score is computed, so its result of 71.3 against at most 44.7 cannot be interpreted.

Questions for authors

What are the per-category T2I-CompBench++ results? Section 5.3 reports gains of +0.052 on numeracy and +0.048 on 2D spatial relations over Lumina-DiMOO, and near-zero gains on non-spatial relations, but Table 6 reports only the mean. Could you provide the full breakdown? This claim is a central piece of evidence that words-first ordering helps where prompts are explicit.

Reviewer_iW6AOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Problem: The authors study progressively decoded image codes, where later chunks add detail to an image reconstructed from earlier chunks. If training focuses on pixel reconstruction, the limited early chunks may encode broad visual appearance before prompt-specific details, such as object counts or positions. As a result, it may produce a plausible-looking image while missing something the prompt specifies.

Authors' idea: Put the prompt's meaning into the early part of the representation, then use later parts to add visual detail.

How RUNG works: RUNG encodes each image as an ordered sequence of code chunks. A code prefix is the first chunks of that sequence, not the first words of a caption. The authors create five caption levels, from about 8 to 180 words, by repeatedly summarizing a detailed caption. Shorter captions are intended to preserve key facts. During training, the image decoder learns to reconstruct the image from prefixes of varying lengths. At five selected prefix lengths, the language decoder is also trained to produce the matching caption level.

Why it matters: The paper suggests a useful strategy for image models: learn the important ideas in the instructions first and add visual detail afterward. This may help the model make and describe the requested image, and fix errors, while preserving the quality of the finished image.

Evaluation Setup: The authors measure several aspects of RUNG:

  • Prompt following and composition measure how well the generated image matches the instructions, including requested objects, counts, colors, and positions. When the code ordering changes from pixel-first to language-first, the GenEval score increases from 0.76 to 0.83.
  • Caption information measures how much of a detailed caption can be recovered from a short code. At one-quarter code length, this estimate increases from 0.68 to 1.97 nats per caption token with language-first ordering.
  • Image similarity and realism tests compare reconstructed images with their originals or compare generated images with real-image sets. PSNR and LPIPS measure reconstruction similarity; rFID, FID, and CMMD measure similarity between image distributions. Language-first ordering lowers PSNR by 2.8 dB for a quarter-length code, while the full-code difference is only 0.1 dB.

These are automated measurements, not human preference judgments; the paper reports no human preference study.

Critical strengths

The paper suggests a useful strategy for image models: learn the important ideas in the instructions first and add visual detail afterward. This may help the model make and describe the requested image, and fix errors, while preserving the quality of the finished image.

Critical weaknesses
  • No human preference study: The paper does not establish whether people find the images visually better overall.

  • Correction-checker validation: The corrected model’s 0.91 GenEval score is promising, but the correction process relies on its own internal attribute checker. Independent tests could show whether that correction method works reliably beyond this evaluation.

  • The paper is difficult for readers outside the field to follow. I encourage the authors to make the abstract and introduction more accessible by clearly stating the problem, proposed approach, and significance in a progressive manner. Key concepts, such as progressive image codes, would also benefit from a brief plain-language explanation or example.

Questions for authors

Can human evaluators independently assess prompt faithfulness and image quality, especially for corrected generations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.